Max level shown:
1 starting March 2, 2024 at 01:05:27 AM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2855957
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.07076578
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.06822688
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start ds: 8820
13 start ds: 8830
14 start access at:127.0.0.1:9998
15 Control access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Make output file at "/tmp/dsc/downstairs-8810.txt"
18 [8810][0] reports Starting
19 Starting downstairs at port 8830
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 Starting downstairs at port 8820
22 Make output file at "/tmp/dsc/downstairs-8820.txt"
23 [8830][2] reports Starting
24 [8820][1] reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1152)
26 [8810][0] reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1153)
28 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1154)
29 Mar 02 01:05:28.478 INFO listening, local_addr: 127.0.0.1:9998
30 [8830][2] reports Running
31 [8820][1] reports Running
32 Running initial fill
332024-03-02T01:05:33.436ZINFOcrucible: Upstairs starts
342024-03-02T01:05:33.436ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "1c1574fb721f98f2df1b23e3fd27d83be018882e", git_commit_timestamp: "2024-03-02T00:46:46.000000000Z", git_branch: "main", rustc_semver: "1.76.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "07dca489ac2d933c78d3c5158e3f43beefeb02ce", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352024-03-02T01:05:33.436ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
362024-03-02T01:05:33.437ZINFOcrucible: Crucible fac59c5c-60c6-447c-ba42-bc1522c76515 has session id: b8307345-8170-418b-8ba3-fa4cd7698a86 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
372024-03-02T01:05:33.437ZINFOcrucible: Upstairs opts: Upstairs UUID: fac59c5c-60c6-447c-ba42-bc1522c76515, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
382024-03-02T01:05:33.437ZINFOcrucible: Crucible stats registered with UUID: fac59c5c-60c6-447c-ba42-bc1522c76515 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
39 Crucible runtime is spawned
402024-03-02T01:05:33.437ZINFOcrucible: The guest has requested activation with gen:1
412024-03-02T01:05:33.437ZINFOcrucible: fac59c5c-60c6-447c-ba42-bc1522c76515 active request set session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
422024-03-02T01:05:33.437ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
432024-03-02T01:05:33.437ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
442024-03-02T01:05:33.437ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
452024-03-02T01:05:33.437ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
462024-03-02T01:05:33.437ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
472024-03-02T01:05:33.437ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
482024-03-02T01:05:33.437ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
492024-03-02T01:05:33.437ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
502024-03-02T01:05:33.438ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
512024-03-02T01:05:33.438ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
522024-03-02T01:05:33.438ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
532024-03-02T01:05:33.438ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
542024-03-02T01:05:33.439ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
552024-03-02T01:05:33.439ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
562024-03-02T01:05:33.439ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
572024-03-02T01:05:33.439ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
582024-03-02T01:05:33.439ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
592024-03-02T01:05:33.439ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
602024-03-02T01:05:33.440ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
612024-03-02T01:05:33.440ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
622024-03-02T01:05:33.440ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
632024-03-02T01:05:33.441ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
642024-03-02T01:05:33.441ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
652024-03-02T01:05:33.441ZINFOcrucible: Waiting for true more clients to be ready session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
662024-03-02T01:05:33.441ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
672024-03-02T01:05:33.441ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
682024-03-02T01:05:33.441ZINFOcrucible: Waiting for true more clients to be ready session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
692024-03-02T01:05:33.441ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
702024-03-02T01:05:33.441ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
712024-03-02T01:05:33.441ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
722024-03-02T01:05:33.441ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
732024-03-02T01:05:33.441ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
742024-03-02T01:05:33.441ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
752024-03-02T01:05:33.441ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
762024-03-02T01:05:33.441ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
772024-03-02T01:05:33.441ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
782024-03-02T01:05:33.441ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
792024-03-02T01:05:33.441ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
802024-03-02T01:05:33.441ZINFOcrucible: Max found gen is 1 = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
812024-03-02T01:05:33.441ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
822024-03-02T01:05:33.441ZINFOcrucible: Next flush: 1 = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
832024-03-02T01:05:33.441ZINFOcrucible: All extents match = downstairs session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
842024-03-02T01:05:33.441ZINFOcrucible: No downstairs reconciliation required session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
85 {"msg":"Transition from WaitQuorum to Active","v":0Activate successful
86 ,"Wait for a query_work_queue command to finish before sending IO
87 name":"crucible","level":30,"time":"2024-03-02T01:05:33.441987611Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1158,"client":"0","":"downstairs","session_id":"b8307345-8170-418b-8ba3-fa4cd7698a86"}
882024-03-02T01:05:33.442ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
892024-03-02T01:05:33.442ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
902024-03-02T01:05:33.442ZINFOcrucible: All required reconciliation work is completed session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
912024-03-02T01:05:33.442ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
922024-03-02T01:05:33.442ZINFOcrucible: fac59c5c-60c6-447c-ba42-bc1522c76515 is now active with session: b8307345-8170-418b-8ba3-fa4cd7698a86 session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
932024-03-02T01:05:33.442ZINFOcrucible: Set Active after no reconciliation session_id = b8307345-8170-418b-8ba3-fa4cd7698a86
942024-03-02T01:05:33.442ZINFOcrucible: The guest has finished waiting for activation with:1
95 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
96 Fill test
97 Read and Verify all blocks (0..5000 range:false)
98 Wrote out file "alan"
99 CLIENT: Tests done. All submitted work has been ACK'd
100 ----------------------------------------------------------------
101 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:9
102 States: Active Active Active
103 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
104 94 Acked 1093 Read 100 Done Done Done false
105 95 Acked 1094 Read 100 Done Done Done false
106 96 Acked 1095 Read 100 Done Done Done false
107 97 Acked 1096 Read 100 Done Done Done false
108 98 Acked 1097 Read 100 Done Done Done false
109 99 Acked 1098 Read 100 Done Done Done false
110 100 Acked 1099 Read 100 Done Done Done false
111 101 Acked 1100 Read 100 Done Done Done false
112 102 Acked 1101 Read 100 Done Done Done false
113 STATES DS:0 DS:1 DS:2 TOTAL
114 New 0 0 0 0
115 Sent 0 0 0 0
116 Done 9 9 9 27
117 Skipped 0 0 0 0
118 Error 0 0 0 0
119 Last Flush: 1092 1092 1092
120 Downstairs last five completed: 1092 1091 1090 1089 1088
121 Upstairs last five completed: 102 101 100 99 98
122 CLIENT: Up:0 ds:9 act:3
123 ----------------------------------------------------------------
124 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
125 Downstairs last five completed: 1102 1101 1100 1099 1098
126 Upstairs last five completed: 103 102 101 100 99
127 CLIENT: Up:0 ds:0 act:3
128 CLIENT: All crucible jobs finished, exiting program
1292024-03-02T01:05:38.542ZWARNcrucible: Guest handle has been dropped
130 Replay loop starts now March 2, 2024 at 01:05:38 AM UTC
1312024-03-02T01:05:38.567ZINFOcrucible: Upstairs starts
1322024-03-02T01:05:38.567ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "1c1574fb721f98f2df1b23e3fd27d83be018882e", git_commit_timestamp: "2024-03-02T00:46:46.000000000Z", git_branch: "main", rustc_semver: "1.76.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "07dca489ac2d933c78d3c5158e3f43beefeb02ce", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1332024-03-02T01:05:38.567ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
134 {"msg":"Crucible e65f42d9-88f3-4704-a1d1-082af8854a14 has session id: b12e3267-1b4e-4723-a1fa-a377ace8dffb","v":0,"name":"crucible","level":30,"time":"Crucible runtime is spawned
135 2024-03-02T01:05:38.567771094Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
1362024-03-02T01:05:38.567ZINFOcrucible: Upstairs opts: Upstairs UUID: e65f42d9-88f3-4704-a1d1-082af8854a14, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1372024-03-02T01:05:38.567ZINFOcrucible: Crucible stats registered with UUID: e65f42d9-88f3-4704-a1d1-082af8854a14 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1382024-03-02T01:05:38.567ZINFOcrucible: The guest has requested activation with gen:2
1392024-03-02T01:05:38.568ZINFOcrucible: e65f42d9-88f3-4704-a1d1-082af8854a14 active request set session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1402024-03-02T01:05:38.568ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1412024-03-02T01:05:38.568ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1422024-03-02T01:05:38.568ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1432024-03-02T01:05:38.568ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1442024-03-02T01:05:38.568ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1452024-03-02T01:05:38.568ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1462024-03-02T01:05:38.568ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1472024-03-02T01:05:38.568ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1482024-03-02T01:05:38.568ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1492024-03-02T01:05:38.568ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1502024-03-02T01:05:38.568ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1512024-03-02T01:05:38.568ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1522024-03-02T01:05:38.569ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1532024-03-02T01:05:38.569ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1542024-03-02T01:05:38.569ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1552024-03-02T01:05:38.569ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1562024-03-02T01:05:38.569ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1572024-03-02T01:05:38.569ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1582024-03-02T01:05:38.569ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1592024-03-02T01:05:38.569ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1602024-03-02T01:05:38.569ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1612024-03-02T01:05:38.570ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1622024-03-02T01:05:38.570ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1632024-03-02T01:05:38.570ZINFOcrucible: Waiting for true more clients to be ready session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1642024-03-02T01:05:38.570ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1652024-03-02T01:05:38.570ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1662024-03-02T01:05:38.570ZINFOcrucible: Waiting for true more clients to be ready session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1672024-03-02T01:05:38.570ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1682024-03-02T01:05:38.570ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1692024-03-02T01:05:38.570ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1702024-03-02T01:05:38.570ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1712024-03-02T01:05:38.570ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1722024-03-02T01:05:38.570ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1732024-03-02T01:05:38.570ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1742024-03-02T01:05:38.570ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1752024-03-02T01:05:38.570ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1762024-03-02T01:05:38.570ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1772024-03-02T01:05:38.570ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1782024-03-02T01:05:38.570ZINFOcrucible: Max found gen is 2 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179 {"msg":"Generation requested: 2 >= found:2","v":0Activate successful
180 ,Wait for a query_work_queue command to finish before sending IO
181 "name":"crucible","level":30,"time":"2024-03-02T01:05:38.570765303Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
1822024-03-02T01:05:38.570ZINFOcrucible: Next flush: 2 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1832024-03-02T01:05:38.570ZINFOcrucible: All extents match = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1842024-03-02T01:05:38.570ZINFOcrucible: No downstairs reconciliation required session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1852024-03-02T01:05:38.570ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1862024-03-02T01:05:38.570ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1872024-03-02T01:05:38.570ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1882024-03-02T01:05:38.570ZINFOcrucible: All required reconciliation work is completed session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1892024-03-02T01:05:38.570ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1902024-03-02T01:05:38.570ZINFOcrucible: e65f42d9-88f3-4704-a1d1-082af8854a14 is now active with session: b12e3267-1b4e-4723-a1fa-a377ace8dffb session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1912024-03-02T01:05:38.570ZINFOcrucible: Set Active after no reconciliation session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1922024-03-02T01:05:38.570ZINFOcrucible: The guest has finished waiting for activation with:2
193 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
194 Loading write count information from file "alan"
195 Mar 02 01:05:38.619 INFO accepted connection, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
196 Main task has work to do, go find it
197 got dsc Stop(0)
198 stop 0
199 [8810] Got stop action so:false kr:false
200 Mar 02 01:05:38.623 INFO request completed, latency_us: 1136, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1f4848a9-3d50-42a1-ae41-8a21f65a129a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
2012024-03-02T01:05:38.623ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2022024-03-02T01:05:38.623ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2032024-03-02T01:05:38.623ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2042024-03-02T01:05:38.623ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2052024-03-02T01:05:38.624ZINFOcrucible: [0] client re-new 0 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2062024-03-02T01:05:38.624ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
207 Mar 02 01:05:38.624 INFO request completed, latency_us: 409, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1580e247-d0cc-4494-9839-7d29892ddd9a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
208 [8810] Exited with: ExitStatus(unix_wait_status(9))
209 [8810][0] reports Exit
2102024-03-02T01:05:39.625ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2112024-03-02T01:05:39.625ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2122024-03-02T01:05:39.625ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2132024-03-02T01:05:39.625ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2142024-03-02T01:05:39.625ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2152024-03-02T01:05:39.625ZINFOcrucible: [0] client re-new 0 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2162024-03-02T01:05:39.625ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2172024-03-02T01:05:40.626ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2182024-03-02T01:05:40.626ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2192024-03-02T01:05:40.626ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2202024-03-02T01:05:40.626ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2212024-03-02T01:05:40.626ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2222024-03-02T01:05:40.626ZINFOcrucible: [0] client re-new 0 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2232024-03-02T01:05:40.626ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2242024-03-02T01:05:41.628ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2252024-03-02T01:05:41.628ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2262024-03-02T01:05:41.628ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2272024-03-02T01:05:41.628ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2282024-03-02T01:05:41.628ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2292024-03-02T01:05:41.628ZINFOcrucible: [0] client re-new 0 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2302024-03-02T01:05:41.628ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
231 Mar 02 01:05:42.627 INFO request completed, latency_us: 318, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3da0b29b-47f7-481b-a810-b4f083e4d843, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
232 001/300 Read block 949 len 28672
2332024-03-02T01:05:42.630ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2342024-03-02T01:05:42.630ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2352024-03-02T01:05:42.630ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2362024-03-02T01:05:42.630ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2372024-03-02T01:05:42.630ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2382024-03-02T01:05:42.630ZINFOcrucible: [0] client re-new 1 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2392024-03-02T01:05:42.630ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
240 002/300 Read block 3374 len 28672
241 003/300 Read block 1820 len 40960
242 004/300 Write block 3611 len 36864 data: 2 2 2 2 2 2 2 2 2
243 005/300 Write block 2265 len 4096 data: 2
244 006/300 Read block 3765 len 4096
245 007/300 Write block 3512 len 20480 data: 2 2 2 2 2
246 008/300 Write block 3116 len 16384 data: 2 2 2 2
247 009/300 Read block 2002 len 28672
248 010/300 Write block 3418 len 36864 data: 2 2 2 2 2 2 2 2 2
249 011/300 Write block 49 len 24576 data: 2 2 2 2 2 2
250 012/300 Read block 612 len 40960
251 013/300 Read block 4947 len 12288
252 014/300 Write block 3965 len 32768 data: 2 2 2 2 2 2 2 2
253 015/300 Write block 1298 len 16384 data: 2 2 2 2
254 016/300 Read block 2772 len 20480
255 017/300 Write block 2521 len 24576 data: 2 2 2 2 2 2
256 018/300 Write block 4452 len 20480 data: 2 2 2 2 2
257 019/300 Write block 4345 len 36864 data: 2 2 2 2 2 2 2 2 2
258 020/300 Write block 1322 len 36864 data: 2 2 2 2 2 2 2 2 2
259 021/300 Write block 2817 len 16384 data: 2 2 2 2
260 022/300 Write block 2502 len 4096 data: 2
261 023/300 Write block 1952 len 20480 data: 2 2 2 2 2
262 024/300 Write block 4619 len 32768 data: 2 2 2 2 2 2 2 2
263 025/300 Read block 3295 len 40960
264 026/300 Write block 3437 len 8192 data: 2 2
265 027/300 Read block 748 len 28672
266 028/300 Read block 102 len 4096
267 029/300 Read block 2791 len 40960
268 030/300 Write block 3857 len 40960 data: 2 2 2 2 2 2 2 2 2 2
269 031/300 Read block 1130 len 12288
270 032/300 Read block 2197 len 4096
271 033/300 Read block 1699 len 32768
272 034/300 Read block 2131 len 36864
273 035/300 Read block 1934 len 16384
274 036/300 Read block 1877 len 28672
275 037/300 Read block 3413 len 20480
276 038/300 Read block 60 len 28672
277 039/300 Read block 1553 len 36864
278 040/300 Write block 1502 len 40960 data: 2 2 2 2 2 2 2 2 2 2
279 041/300 Read block 2575 len 8192
280 042/300 Read block 4070 len 24576
281 043/300 Read block 1857 len 12288
282 044/300 Write block 234 len 32768 data: 2 2 2 2 2 2 2 2
283 045/300 Write block 1085 len 16384 data: 2 2 2 2
284 046/300 Write block 2396 len 4096 data: 2
285 047/300 Write block 2314 len 36864 data: 2 2 2 2 2 2 2 2 2
286 048/300 Read block 3037 len 40960
287 049/300 Write block 2249 len 40960 data: 2 2 2 2 2 2 2 2 2 2
288 050/300 Read block 4593 len 8192
289 051/300 Read block 1589 len 4096
290 052/300 Read block 1906 len 20480
291 053/300 Write block 4586 len 28672 data: 2 2 2 2 2 2 2
292 054/300 Write block 957 len 40960 data: 2 2 2 2 2 2 2 2 2 2
293 055/300 Read block 3121 len 36864
294 056/300 Read block 2698 len 8192
295 057/300 Read block 3263 len 16384
296 058/300 Write block 378 len 4096 data: 2
297 059/300 Write block 1417 len 40960 data: 2 2 2 2 2 2 2 2 2 2
298 060/300 Read block 3148 len 20480
299 061/300 Flush
300 062/300 Read block 3882 len 12288
301 063/300 Write block 3485 len 16384 data: 2 2 2 2
302 064/300 Read block 954 len 8192
303 065/300 Write block 4033 len 16384 data: 2 2 2 2
304 066/300 Read block 4168 len 4096
305 067/300 Read block 1527 len 12288
306 068/300 Read block 1612 len 20480
307 069/300 Read block 3778 len 24576
308 070/300 Write block 3061 len 12288 data: 2 2 2
309 071/300 Write block 4840 len 28672 data: 2 2 2 2 2 2 2
310 072/300 Write block 3821 len 28672 data: 2 2 2 2 2 2 2
311 073/300 Read block 3188 len 24576
312 074/300 Write block 4602 len 12288 data: 2 2 2
313 075/300 Read block 3681 len 8192
314 076/300 Write block 1151 len 32768 data: 2 2 2 2 2 2 2 2
315 077/300 Write block 584 len 8192 data: 2 2
316 078/300 Read block 4583 len 32768
317 079/300 Read block 4814 len 28672
318 080/300 Flush
319 081/300 Write block 295 len 16384 data: 2 2 2 2
320 082/300 Write block 3418 len 16384 data: 3 3 3 3
321 083/300 Read block 4808 len 8192
322 084/300 Read block 2984 len 20480
323 085/300 Read block 4519 len 8192
324 086/300 Write block 2675 len 36864 data: 2 2 2 2 2 2 2 2 2
325 087/300 Read block 2710 len 20480
326 088/300 Read block 4176 len 12288
327 089/300 Read block 1427 len 4096
328 090/300 Flush
329 091/300 Read block 1648 len 28672
330 092/300 Flush
331 093/300 Write block 2791 len 16384 data: 2 2 2 2
332 094/300 Read block 2884 len 28672
333 095/300 Flush
334 096/300 Read block 774 len 4096
335 097/300 Read block 2562 len 28672
336 098/300 Write block 1190 len 32768 data: 2 2 2 2 2 2 2 2
337 099/300 Read block 1067 len 16384
338 100/300 Write block 1244 len 20480 data: 2 2 2 2 2
339 101/300 Write block 3979 len 16384 data: 2 2 2 2
340 102/300 Write block 1726 len 12288 data: 2 2 2
341 103/300 Read block 2414 len 16384
342 104/300 Write block 3903 len 8192 data: 2 2
343 105/300 Read block 649 len 16384
344 106/300 Read block 2578 len 28672
345 107/300 Write block 2842 len 32768 data: 2 2 2 2 2 2 2 2
346 108/300 Read block 4993 len 8192
347 109/300 Write block 3441 len 4096 data: 2
348 110/300 Read block 4488 len 28672
349 111/300 Flush
350 112/300 Read block 2322 len 8192
351 113/300 Flush
352 114/300 Write block 1437 len 8192 data: 2 2
353 115/300 Read block 4278 len 28672
354 116/300 Write block 3414 len 8192 data: 2 2
355 117/300 Read block 4841 len 32768
356 118/300 Write block 1951 len 4096 data: 2
357 119/300 Write block 410 len 36864 data: 2 2 2 2 2 2 2 2 2
358 120/300 Read block 3309 len 20480
359 121/300 Write block 664 len 32768 data: 2 2 2 2 2 2 2 2
360 122/300 Read block 91 len 28672
361 123/300 Read block 3850 len 28672
362 124/300 Write block 295 len 4096 data: 3
363 125/300 Write block 3225 len 4096 data: 2
364 126/300 Write block 1242 len 16384 data: 2 2 3 3
365 127/300 Flush
366 128/300 Read block 1424 len 20480
367 129/300 Write block 2863 len 32768 data: 2 2 2 2 2 2 2 2
368 130/300 Write block 1256 len 28672 data: 2 2 2 2 2 2 2
369 131/300 Write block 3564 len 24576 data: 2 2 2 2 2 2
370 132/300 Write block 3789 len 8192 data: 2 2
371 133/300 Read block 4218 len 12288
372 134/300 Write block 1443 len 24576 data: 2 2 2 2 2 2
373 135/300 Read block 1351 len 20480
374 136/300 Read block 2941 len 36864
375 137/300 Flush
376 138/300 Flush
377 139/300 Read block 3375 len 28672
378 140/300 Write block 3015 len 40960 data: 2 2 2 2 2 2 2 2 2 2
379 141/300 Write block 3552 len 20480 data: 2 2 2 2 2
380 142/300 Flush
381 143/300 Read block 1416 len 20480
382 144/300 Read block 3323 len 28672
383 145/300 Write block 1029 len 24576 data: 2 2 2 2 2 2
384 146/300 Read block 2575 len 40960
385 147/300 Read block 1805 len 12288
386 148/300 Write block 4600 len 20480 data: 2 2 3 3 3
387 149/300 Write block 4583 len 12288 data: 2 2 2
388 150/300 Read block 2700 len 36864
389 151/300 Read block 539 len 28672
390 152/300 Read block 3452 len 20480
391 153/300 Flush
392 154/300 Read block 2373 len 8192
393 155/300 Write block 4332 len 12288 data: 2 2 2
394 156/300 Read block 4785 len 4096
395 157/300 Read block 2969 len 36864
396 158/300 Write block 1833 len 12288 data: 2 2 2
397 159/300 Write block 2551 len 4096 data: 2
398 160/300 Read block 3956 len 8192
399 161/300 Read block 1053 len 16384
400 162/300 Write block 3949 len 16384 data: 2 2 2 2
401 163/300 Read block 1602 len 36864
402 164/300 Write block 3003 len 8192 data: 2 2
403 165/300 Write block 145 len 4096 data: 2
404 166/300 Write block 4265 len 16384 data: 2 2 2 2
405 167/300 Read block 42 len 36864
406 168/300 Read block 1760 len 12288
407 169/300 Write block 2374 len 4096 data: 2
408 170/300 Write block 3974 len 4096 data: 2
409 171/300 Read block 3714 len 24576
410 172/300 Read block 3126 len 36864
411 173/300 Flush
412 174/300 Flush
413 175/300 Read block 1426 len 28672
414 176/300 Read block 553 len 12288
415 177/300 Read block 2387 len 16384
416 178/300 Write block 4872 len 12288 data: 2 2 2
417 179/300 Write block 790 len 36864 data: 2 2 2 2 2 2 2 2 2
418 180/300 Write block 2499 len 8192 data: 2 2
419 181/300 Read block 4397 len 32768
420 182/300 Read block 2442 len 40960
421 183/300 Write block 2021 len 12288 data: 2 2 2
422 184/300 Flush
423 185/300 Write block 267 len 4096 data: 2
424 186/300 Read block 483 len 4096
425 187/300 Write block 2536 len 32768 data: 2 2 2 2 2 2 2 2
426 188/300 Write block 479 len 8192 data: 2 2
427 189/300 Flush
428 190/300 Read block 4026 len 20480
429 191/300 Write block 1986 len 4096 data: 2
430 192/300 Read block 4827 len 24576
431 193/300 Read block 1104 len 32768
432 194/300 Flush
433 195/300 Read block 4443 len 8192
434 196/300 Read block 1941 len 12288
435 197/300 Flush
436 198/300 Flush
437 199/300 Read block 3297 len 16384
438 200/300 Write block 2955 len 32768 data: 2 2 2 2 2 2 2 2
439 201/300 Read block 1043 len 8192
440 202/300 Read block 4218 len 4096
441 203/300 Write block 1815 len 4096 data: 2
442 204/300 Read block 1664 len 16384
443 205/300 Write block 89 len 8192 data: 2 2
444 206/300 Read block 4415 len 8192
445 207/300 Flush
446 208/300 Read block 2904 len 16384
447 209/300 Read block 2228 len 16384
448 210/300 Read block 3186 len 20480
449 211/300 Read block 4631 len 24576
450 212/300 Read block 2784 len 36864
451 213/300 Write block 490 len 4096 data: 2
452 214/300 Read block 3130 len 12288
453 215/300 Read block 1468 len 36864
454 216/300 Write block 1640 len 24576 data: 2 2 2 2 2 2
455 217/300 Write block 2087 len 40960 data: 2 2 2 2 2 2 2 2 2 2
456 218/300 Read block 2722 len 16384
457 219/300 Read block 3049 len 4096
458 220/300 Read block 617 len 40960
459 221/300 Write block 1584 len 28672 data: 2 2 2 2 2 2 2
460 222/300 Read block 2356 len 8192
461 223/300 Write block 4522 len 20480 data: 2 2 2 2 2
462 224/300 Read block 3670 len 36864
463 225/300 Write block 2244 len 36864 data: 2 2 2 2 2 3 3 3 3
464 226/300 Read block 510 len 28672
465 227/300 Write block 3052 len 16384 data: 2 2 2 2
466 228/300 Read block 923 len 16384
467 229/300 Write block 3186 len 36864 data: 2 2 2 2 2 2 2 2 2
468 230/300 Write block 3593 len 28672 data: 2 2 2 2 2 2 2
469 231/300 Read block 2184 len 20480
470 232/300 Write block 926 len 12288 data: 2 2 2
471 233/300 Write block 637 len 28672 data: 2 2 2 2 2 2 2
472 234/300 Flush
473 235/300 Read block 2237 len 16384
474 236/300 Write block 545 len 40960 data: 2 2 2 2 2 2 2 2 2 2
475 237/300 Write block 1230 len 40960 data: 2 2 2 2 2 2 2 2 2 2
476 238/300 Write block 4202 len 12288 data: 2 2 2
477 239/300 Read block 1501 len 24576
478 240/300 Write block 3863 len 4096 data: 3
479 241/300 Flush
480 242/300 Read block 2290 len 4096
481 243/300 Write block 669 len 4096 data: 3
482 244/300 Write block 4178 len 40960 data: 2 2 2 2 2 2 2 2 2 2
483 245/300 Write block 991 len 24576 data: 2 2 2 2 2 2
484 246/300 Read block 3680 len 16384
485 247/300 Write block 4969 len 16384 data: 2 2 2 2
486 248/300 Read block 521 len 12288
487 249/300 Read block 709 len 12288
488 250/300 Write block 3421 len 24576 data: 4 3 3 3 3 3
489 251/300 Write block 2131 len 20480 data: 2 2 2 2 2
490 252/300 Write block 2510 len 4096 data: 2
491 253/300 Write block 4639 len 36864 data: 2 2 2 2 2 2 2 2 2
492 254/300 Write block 1688 len 12288 data: 2 2 2
493 255/300 Read block 3899 len 24576
494 256/300 Write block 1821 len 4096 data: 2
495 257/300 Read block 4827 len 12288
496 258/300 Write block 2480 len 24576 data: 2 2 2 2 2 2
497 259/300 Read block 989 len 16384
498 260/300 Read block 625 len 16384
499 261/300 Read block 1150 len 4096
500 262/300 Read block 4138 len 24576
501 263/300 Write block 247 len 36864 data: 2 2 2 2 2 2 2 2 2
502 264/300 Read block 1335 len 16384
503 265/300 Read block 1980 len 28672
504 266/300 Write block 4297 len 12288 data: 2 2 2
505 267/300 Write block 1331 len 16384 data: 2 2 2 2
506 268/300 Read block 3124 len 16384
507 269/300 Read block 3184 len 12288
508 270/300 Read block 1524 len 28672
509 271/300 Flush
510 272/300 Read block 4614 len 16384
511 273/300 Flush
512 274/300 Write block 3262 len 20480 data: 2 2 2 2 2
513 275/300 Flush
514 276/300 Write block 4788 len 24576 data: 2 2 2 2 2 2
515 277/300 Write block 1260 len 24576 data: 3 3 3 2 2 2
516 278/300 Read block 1035 len 40960
517 279/300 Write block 3819 len 12288 data: 2 2 3
518 280/300 Read block 3788 len 12288
519 281/300 Read block 1601 len 4096
520 282/300 Read block 4515 len 40960
521 283/300 Flush
522 284/300 Write block 1797 len 4096 data: 2
523 285/300 Read block 542 len 8192
524 286/300 Flush
525 287/300 Write block 3817 len 32768 data: 2 2 3 3 4 3 3 3
526 288/300 Write block 4056 len 24576 data: 2 2 2 2 2 2
527 289/300 Read block 422 len 32768
528 290/300 Flush
529 291/300 Read block 203 len 4096
530 292/300 Read block 3642 len 40960
531 293/300 Flush
532 294/300 Read block 120 len 40960
533 295/300 Read block 3498 len 28672
534 296/300 Read block 3117 len 28672
535 297/300 Write block 1627 len 20480 data: 2 2 2 2 2
536 298/300 Read block 661 len 28672
537 299/300 Read block 2042 len 32768
538 300/300 Read block 1077 len 4096
539 Main task has work to do, go find it
540 got dsc Start(0)
541 start 0
542 Mar 02 01:05:43.341 INFO request completed, latency_us: 412, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0c9c85d5-2175-4cf0-a915-889287651002, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
543 Replay: started 0, returned:Ok(())
544 ----------------------------------------------------------------
545 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
546 States: Offline Active Active
547 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
548 1 Acked 1000 Read 7 New Done Done true
549 2 Acked 1001 Read 7 New Done Done false
550 3 Acked 1002 Read 10 New Done Done false
551 4 Acked 1003 Write 9 New Done Done false
552 5 Acked 1004 Write 1 New Done Done false
553 6 Acked 1005 Read 1 New Done Done false
554 7 Acked 1006 Write 5 New Done Done false
555 8 Acked 1007 Write 4 New Done Done false
556 9 Acked 1008 Read 7 New Done Done false
557 10 Acked 1009 Write 9 New Done Done false
558 11 Acked 1010 Write 6 New Done Done false
559 12 Acked 1011 Read 10 New Done Done false
560 13 Acked 1012 Read 3 New Done Done false
561 14 Acked 1013 Write 8 New Done Done false
562 15 Acked 1014 Write 4 New Done Done false
563 16 Acked 1015 Read 5 New Done Done false
564 17 Acked 1016 Write 6 New Done Done false
565 18 Acked 1017 Write 5 New Done Done false
566 19 Acked 1018 Write 9 New Done Done false
567 20 Acked 1019 Write 9 New Done Done false
568 21 Acked 1020 Write 4 New Done Done false
569 22 Acked 1021 Write 1 New Done Done false
570 23 Acked 1022 Write 5 New Done Done false
571 24 Acked 1023 Write 8 New Done Done false
572 25 Acked 1024 Read 10 New Done Done false
573 26 Acked 1025 Write 2 New Done Done false
574 27 Acked 1026 Read 7 New Done Done false
575 28 Acked 1027 Read 1 New Done Done false
576 29 Acked 1028 Read 10 New Done Done false
577 30 Acked 1029 Write 10 New Done Done false
578 31 Acked 1030 Read 3 New Done Done false
579 32 Acked 1031 Read 1 New Done Done false
580 33 Acked 1032 Read 8 New Done Done false
581 34 Acked 1033 Read 9 New Done Done false
582 35 Acked 1034 Read 4 New Done Done false
583 36 Acked 1035 Read 7 New Done Done false
584 37 Acked 1036 Read 5 New Done Done false
585 38 Acked 1037 Read 7 New Done Done false
586 39 Acked 1038 Read 9 New Done Done false
587 40 Acked 1039 Write 10 New Done Done false
588 41 Acked 1040 Read 2 New Done Done false
589 42 Acked 1041 Read 6 New Done Done false
590 43 Acked 1042 Read 3 New Done Done false
591 44 Acked 1043 Write 8 New Done Done false
592 45 Acked 1044 Write 4 New Done Done false
593 46 Acked 1045 Write 1 New Done Done false
594 47 Acked 1046 Write 9 New Done Done false
595 48 Acked 1047 Read 10 New Done Done false
596 49 Acked 1048 Write 10 New Done Done false
597 50 Acked 1049 Read 2 New Done Done false
598 51 Acked 1050 Read 1 New Done Done false
599 52 Acked 1051 Read 5 New Done Done false
600 53 Acked 1052 Write 7 New Done Done false
601 54 Acked 1053 Write 10 New Done Done false
602 55 Acked 1054 Read 9 New Done Done false
603 56 Acked 1055 Read 2 New Done Done false
604 57 Acked 1056 Read 4 New Done Done false
605 58 Acked 1057 Write 1 New Done Done false
606 59 Acked 1058 Write 10 New Done Done false
607 60 Acked 1059 Read 5 New Done Done false
608 61 Acked 1060 Flush 0 New Done Done false
609 62 Acked 1061 Read 3 New Done Done false
610 63 Acked 1062 Write 4 New Done Done false
611 64 Acked 1063 Read 2 New Done Done false
612 65 Acked 1064 Write 4 New Done Done false
613 66 Acked 1065 Read 1 New Done Done false
614 67 Acked 1066 Read 3 New Done Done false
615 68 Acked 1067 Read 5 New Done Done false
616 69 Acked 1068 Read 6 New Done Done false
617 70 Acked 1069 Write 3 New Done Done false
618 71 Acked 1070 Write 7 New Done Done false
619 72 Acked 1071 Write 7 New Done Done false
620 73 Acked 1072 Read 6 New Done Done false
621 74 Acked 1073 Write 3 New Done Done false
622 75 Acked 1074 Read 2 New Done Done false
623 76 Acked 1075 Write 8 New Done Done false
624 77 Acked 1076 Write 2 New Done Done false
625 78 Acked 1077 Read 8 New Done Done false
626 79 Acked 1078 Read 7 New Done Done false
627 80 Acked 1079 Flush 0 New Done Done false
628 81 Acked 1080 Write 4 New Done Done false
629 82 Acked 1081 Write 4 New Done Done false
630 83 Acked 1082 Read 2 New Done Done false
631 84 Acked 1083 Read 5 New Done Done false
632 85 Acked 1084 Read 2 New Done Done false
633 86 Acked 1085 Write 9 New Done Done false
634 87 Acked 1086 Read 5 New Done Done false
635 88 Acked 1087 Read 3 New Done Done false
636 89 Acked 1088 Read 1 New Done Done false
637 90 Acked 1089 Flush 0 New Done Done false
638 91 Acked 1090 Read 7 New Done Done false
639 92 Acked 1091 Flush 0 New Done Done false
640 93 Acked 1092 Write 4 New Done Done false
641 94 Acked 1093 Read 7 New Done Done false
642 95 Acked 1094 Flush 0 New Done Done false
643 96 Acked 1095 Read 1 New Done Done false
644 97 Acked 1096 Read 7 New Done Done false
645 98 Acked 1097 Write 8 New Done Done false
646 99 Acked 1098 Read 4 New Done Done false
647 100 Acked 1099 Write 5 New Done Done false
648 101 Acked 1100 Write 4 New Done Done false
649 102 Acked 1101 Write 3 New Done Done false
650 103 Acked 1102 Read 4 New Done Done false
651 104 Acked 1103 Write 2 New Done Done false
652 105 Acked 1104 Read 4 New Done Done false
653 106 Acked 1105 Read 7 New Done Done false
654 107 Acked 1106 Write 8 New Done Done false
655 108 Acked 1107 Read 2 New Done Done false
656 109 Acked 1108 Write 1 New Done Done false
657 110 Acked 1109 Read 7 New Done Done false
658 111 Acked 1110 Flush 0 New Done Done false
659 112 Acked 1111 Read 2 New Done Done false
660 113 Acked 1112 Flush 0 New Done Done false
661 114 Acked 1113 Write 2 New Done Done false
662 115 Acked 1114 Read 7 New Done Done false
663 116 Acked 1115 Write 2 New Done Done false
664 117 Acked 1116 Read 8 New Done Done false
665 118 Acked 1117 Write 1 New Done Done false
666 119 Acked 1118 Write 9 New Done Done false
667 120 Acked 1119 Read 5 New Done Done false
668 121 Acked 1120 Write 8 New Done Done false
669 122 Acked 1121 Read 7 New Done Done false
670 123 Acked 1122 Read 7 New Done Done false
671 124 Acked 1123 Write 1 New Done Done false
672 125 Acked 1124 Write 1 New Done Done false
673 126 Acked 1125 Write 4 New Done Done false
674 127 Acked 1126 Flush 0 New Done Done false
675 128 Acked 1127 Read 5 New Done Done false
676 129 Acked 1128 Write 8 New Done Done false
677 130 Acked 1129 Write 7 New Done Done false
678 131 Acked 1130 Write 6 New Done Done false
679 132 Acked 1131 Write 2 New Done Done false
680 133 Acked 1132 Read 3 New Done Done false
681 134 Acked 1133 Write 6 New Done Done false
682 135 Acked 1134 Read 5 New Done Done false
683 136 Acked 1135 Read 9 New Done Done false
684 137 Acked 1136 Flush 0 New Done Done false
685 138 Acked 1137 Flush 0 New Done Done false
686 139 Acked 1138 Read 7 New Done Done false
687 140 Acked 1139 Write 10 New Done Done false
688 141 Acked 1140 Write 5 New Done Done false
689 142 Acked 1141 Flush 0 New Done Done false
690 143 Acked 1142 Read 5 New Done Done false
691 144 Acked 1143 Read 7 New Done Done false
692 145 Acked 1144 Write 6 New Done Done false
693 146 Acked 1145 Read 10 New Done Done false
694 147 Acked 1146 Read 3 New Done Done false
695 148 Acked 1147 Write 5 New Done Done false
696 149 Acked 1148 Write 3 New Done Done false
697 150 Acked 1149 Read 9 New Done Done false
698 151 Acked 1150 Read 7 New Done Done false
699 152 Acked 1151 Read 5 New Done Done false
700 153 Acked 1152 Flush 0 New Done Done false
701 154 Acked 1153 Read 2 New Done Done false
702 155 Acked 1154 Write 3 New Done Done false
703 156 Acked 1155 Read 1 New Done Done false
704 157 Acked 1156 Read 9 New Done Done false
705 158 Acked 1157 Write 3 New Done Done false
706 159 Acked 1158 Write 1 New Done Done false
707 160 Acked 1159 Read 2 New Done Done false
708 161 Acked 1160 Read 4 New Done Done false
709 162 Acked 1161 Write 4 New Done Done false
710 163 Acked 1162 Read 9 New Done Done false
711 164 Acked 1163 Write 2 New Done Done false
712 165 Acked 1164 Write 1 New Done Done false
713 166 Acked 1165 Write 4 New Done Done false
714 167 Acked 1166 Read 9 New Done Done false
715 168 Acked 1167 Read 3 New Done Done false
716 169 Acked 1168 Write 1 New Done Done false
717 170 Acked 1169 Write 1 New Done Done false
718 171 Acked 1170 Read 6 New Done Done false
719 172 Acked 1171 Read 9 New Done Done false
720 173 Acked 1172 Flush 0 New Done Done false
721 174 Acked 1173 Flush 0 New Done Done false
722 175 Acked 1174 Read 7 New Done Done false
723 176 Acked 1175 Read 3 New Done Done false
724 177 Acked 1176 Read 4 New Done Done false
725 178 Acked 1177 Write 3 New Done Done false
726 179 Acked 1178 Write 9 New Done Done false
727 180 Acked 1179 Write 2 New Done Done false
728 181 Acked 1180 Read 8 New Done Done false
729 182 Acked 1181 Read 10 New Done Done false
730 183 Acked 1182 Write 3 New Done Done false
731 184 Acked 1183 Flush 0 New Done Done false
732 185 Acked 1184 Write 1 New Done Done false
733 186 Acked 1185 Read 1 New Done Done false
734 187 Acked 1186 Write 8 New Done Done false
735 188 Acked 1187 Write 2 New Done Done false
736 189 Acked 1188 Flush 0 New Done Done false
737 190 Acked 1189 Read 5 New Done Done false
738 191 Acked 1190 Write 1 New Done Done false
739 192 Acked 1191 Read 6 New Done Done false
740 193 Acked 1192 Read 8 New Done Done false
741 194 Acked 1193 Flush 0 New Done Done false
742 195 Acked 1194 Read 2 New Done Done false
743 196 Acked 1195 Read 3 New Done Done false
744 197 Acked 1196 Flush 0 New Done Done false
745 198 Acked 1197 Flush 0 New Done Done false
746 199 Acked 1198 Read 4 New Done Done false
747 200 Acked 1199 Write 8 New Done Done false
748 201 Acked 1200 Read 2 New Done Done false
749 202 Acked 1201 Read 1 New Done Done false
750 203 Acked 1202 Write 1 New Done Done false
751 204 Acked 1203 Read 4 New Done Done false
752 205 Acked 1204 Write 2 New Done Done false
753 206 Acked 1205 Read 2 New Done Done false
754 207 Acked 1206 Flush 0 New Done Done false
755 208 Acked 1207 Read 4 New Done Done false
756 209 Acked 1208 Read 4 New Done Done false
757 210 Acked 1209 Read 5 New Done Done false
758 211 Acked 1210 Read 6 New Done Done false
759 212 Acked 1211 Flush 0 New Done Done false
760 213 Acked 1212 Read 9 New Done Done false
761 214 Acked 1213 Write 1 New Done Done false
762 215 Acked 1214 Read 3 New Done Done false
763 216 Acked 1215 Read 9 New Done Done false
764 217 Acked 1216 Write 6 New Done Done false
765 218 Acked 1217 Write 10 New Done Done false
766 219 Acked 1218 Read 4 New Done Done false
767 220 Acked 1219 Read 1 New Done Done false
768 221 Acked 1220 Read 10 New Done Done false
769 222 Acked 1221 Write 7 New Done Done false
770 223 Acked 1222 Read 2 New Done Done false
771 224 Acked 1223 Write 5 New Done Done false
772 225 Acked 1224 Read 9 New Done Done false
773 226 Acked 1225 Write 9 New Done Done false
774 227 Acked 1226 Read 7 New Done Done false
775 228 Acked 1227 Write 4 New Done Done false
776 229 Acked 1228 Read 4 New Done Done false
777 230 Acked 1229 Write 9 New Done Done false
778 231 Acked 1230 Write 7 New Done Done false
779 232 Acked 1231 Read 5 New Done Done false
780 233 Acked 1232 Write 3 New Done Done false
781 234 Acked 1233 Write 7 New Done Done false
782 235 Acked 1234 Flush 0 New Done Done false
783 236 Acked 1235 Read 4 New Done Done false
784 237 Acked 1236 Write 10 New Done Done false
785 238 Acked 1237 Write 10 New Done Done false
786 239 Acked 1238 Write 3 New Done Done false
787 240 Acked 1239 Read 6 New Done Done false
788 241 Acked 1240 Write 1 New Done Done false
789 242 Acked 1241 Flush 0 New Done Done false
790 243 Acked 1242 Read 1 New Done Done false
791 244 Acked 1243 Write 1 New Done Done false
792 245 Acked 1244 Write 10 New Done Done false
793 246 Acked 1245 Write 6 New Done Done false
794 247 Acked 1246 Read 4 New Done Done false
795 248 Acked 1247 Write 4 New Done Done false
796 249 Acked 1248 Read 3 New Done Done false
797 250 Acked 1249 Read 3 New Done Done false
798 251 Acked 1250 Write 6 New Done Done false
799 252 Acked 1251 Write 5 New Done Done false
800 253 Acked 1252 Write 1 New Done Done false
801 254 Acked 1253 Write 9 New Done Done false
802 255 Acked 1254 Write 3 New Done Done false
803 256 Acked 1255 Read 6 New Done Done false
804 257 Acked 1256 Write 1 New Done Done false
805 258 Acked 1257 Read 3 New Done Done false
806 259 Acked 1258 Write 6 New Done Done false
807 260 Acked 1259 Read 4 New Done Done false
808 261 Acked 1260 Read 4 New Done Done false
809 262 Acked 1261 Read 1 New Done Done false
810 263 Acked 1262 Read 6 New Done Done false
811 264 Acked 1263 Write 9 New Done Done false
812 265 Acked 1264 Read 4 New Done Done false
813 266 Acked 1265 Read 7 New Done Done false
814 267 Acked 1266 Write 3 New Done Done false
815 268 Acked 1267 Write 4 New Done Done false
816 269 Acked 1268 Read 4 New Done Done false
817 270 Acked 1269 Read 3 New Done Done false
818 271 Acked 1270 Read 7 New Done Done false
819 272 Acked 1271 Flush 0 New Done Done false
820 273 Acked 1272 Read 4 New Done Done false
821 274 Acked 1273 Flush 0 New Done Done false
822 275 Acked 1274 Write 5 New Done Done false
823 276 Acked 1275 Flush 0 New Done Done false
824 277 Acked 1276 Write 6 New Done Done false
825 278 Acked 1277 Write 6 New Done Done false
826 279 Acked 1278 Read 10 New Done Done false
827 280 Acked 1279 Write 3 New Done Done false
828 281 Acked 1280 Read 3 New Done Done false
829 282 Acked 1281 Read 1 New Done Done false
830 283 Acked 1282 Read 10 New Done Done false
831 284 Acked 1283 Flush 0 New Done Done false
832 285 Acked 1284 Write 1 New Done Done false
833 286 Acked 1285 Read 2 New Done Done false
834 287 Acked 1286 Flush 0 New Done Done false
835 288 Acked 1287 Write 8 New Done Done false
836 289 Acked 1288 Write 6 New Done Done false
837 290 Acked 1289 Read 8 New Done Done false
838 291 Acked 1290 Flush 0 New Done Done false
839 292 Acked 1291 Read 1 New Done Done false
840 293 Acked 1292 Read 10 New Done Done false
841 294 Acked 1293 Flush 0 New Done Done false
842 295 Acked 1294 Read 10 New Done Done false
843 296 Acked 1295 Read 7 New Done Done false
844 297 Acked 1296 Read 7 New Done Done false
845 298 Acked 1297 Write 5 New Done Done false
846 299 Acked 1298 Read 7 New Done Done false
847 300 Acked 1299 Read 8 New Done Done false
848 301 Acked 1300 Read 1 New Done Done false
849 STATES DS:0 DS:1 DS:2 TOTAL
850 New 301 0 0 301
851 Sent 0 0 0 0
852 Done 0 301 301 602
853 Skipped 0 0 0 0
854 Error 0 0 0 0
855 Last Flush: 0 1293 1293
856 Downstairs last five completed:
857 Upstairs last five completed: 301 300 299 298 297
858 CLIENT: Up:0 ds:301 act:2
8592024-03-02T01:05:43.631ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8602024-03-02T01:05:43.631ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8612024-03-02T01:05:43.631ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8622024-03-02T01:05:43.631ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8632024-03-02T01:05:43.631ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8642024-03-02T01:05:43.631ZINFOcrucible: [0] client re-new 302 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8652024-03-02T01:05:43.632ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
866 [8810] got start action
867 [8810] I am going to restart
868 Starting downstairs at port 8810
869 Make output file at "/tmp/dsc/downstairs-8810.txt"
870 [8810][0] reports Starting
8712024-03-02T01:05:44.632ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8722024-03-02T01:05:44.632ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8732024-03-02T01:05:44.632ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8742024-03-02T01:05:44.633ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8752024-03-02T01:05:44.633ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8762024-03-02T01:05:44.633ZINFOcrucible: [0] client re-new 302 jobs since flush 0 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8772024-03-02T01:05:44.633ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
878 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1163)
879 [8810][0] reports Running
8802024-03-02T01:05:45.633ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8812024-03-02T01:05:45.634ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8822024-03-02T01:05:45.635ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8832024-03-02T01:05:45.635ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8842024-03-02T01:05:45.635ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8852024-03-02T01:05:45.635ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8862024-03-02T01:05:45.635ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8872024-03-02T01:05:45.635ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8882024-03-02T01:05:45.635ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8892024-03-02T01:05:45.635ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
890 ----------------------------------------------------------------
891 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
892 Downstairs last five completed: 1301 1300 1299 1298 1297
893 Upstairs last five completed: 302 301 300 299 298
894 CLIENT: Up:0 ds:0 act:3
895 Replay: All jobs finished, all DS active.
896 Main task has work to do, go find it
897 got dsc Stop(1)
898 stop 1
899 Mar 02 01:05:47.343 INFO request completed, latency_us: 347, response_code[8820] Got stop action so:false kr:false
900 : 204, uri: /stop/cid/1, method: POST, req_id: 8ad9bc1f-6d98-42dd-ae39-2d8e7a9ad0b0, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
901 Mar 02 01:05:47.344 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/1, method: GET, req_id: c910eede-dbee-428e-b0b1-2bde973e8f37, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
9022024-03-02T01:05:47.344ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9032024-03-02T01:05:47.344ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9042024-03-02T01:05:47.344ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9052024-03-02T01:05:47.344ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9062024-03-02T01:05:47.344ZINFOcrucible: [1] client re-new 0 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9072024-03-02T01:05:47.344ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
908 [8820] Exited with: ExitStatus(unix_wait_status(9))
909 [8820][1] reports Exit
9102024-03-02T01:05:48.346ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9112024-03-02T01:05:48.346ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9122024-03-02T01:05:48.346ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9132024-03-02T01:05:48.346ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9142024-03-02T01:05:48.346ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9152024-03-02T01:05:48.346ZINFOcrucible: [1] client re-new 0 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9162024-03-02T01:05:48.346ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9172024-03-02T01:05:49.346ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9182024-03-02T01:05:49.346ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9192024-03-02T01:05:49.346ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9202024-03-02T01:05:49.346ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9212024-03-02T01:05:49.346ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9222024-03-02T01:05:49.346ZINFOcrucible: [1] client re-new 0 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9232024-03-02T01:05:49.346ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9242024-03-02T01:05:50.347ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9252024-03-02T01:05:50.347ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9262024-03-02T01:05:50.347ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9272024-03-02T01:05:50.347ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9282024-03-02T01:05:50.347ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9292024-03-02T01:05:50.347ZINFOcrucible: [1] client re-new 0 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9302024-03-02T01:05:50.348ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
931 Mar 02 01:05:51.346 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/1, method: GET, req_id: 15f2987d-1e06-43af-93a7-dd080a5170d9, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
932 001/300 Flush
933 002/300 Read block 4028 len 40960
9342024-03-02T01:05:51.348ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9352024-03-02T01:05:51.348ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9362024-03-02T01:05:51.348ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9372024-03-02T01:05:51.348ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9382024-03-02T01:05:51.348ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9392024-03-02T01:05:51.348ZINFOcrucible: [1] client re-new 2 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9402024-03-02T01:05:51.348ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
941 003/300 Read block 238 len 12288
942 004/300 Read block 101 len 28672
943 005/300 Read block 1496 len 12288
944 006/300 Read block 2748 len 4096
945 007/300 Write block 1645 len 24576 data: 3 2 2 2 2 2
946 008/300 Write block 1083 len 24576 data: 2 2 3 3 3 3
947 009/300 Read block 43 len 4096
948 010/300 Write block 3559 len 28672 data: 2 2 2 2 2 3 3
949 011/300 Write block 943 len 20480 data: 2 2 2 2 2
950 012/300 Read block 2418 len 40960
951 013/300 Write block 3837 len 24576 data: 2 2 2 2 2 2
952 014/300 Write block 982 len 28672 data: 2 2 2 2 2 2 2
953 015/300 Read block 3827 len 8192
954 016/300 Read block 4887 len 24576
955 017/300 Read block 4080 len 8192
956 018/300 Read block 4859 len 40960
957 019/300 Read block 575 len 40960
958 020/300 Write block 565 len 4096 data: 2
959 021/300 Write block 2210 len 20480 data: 2 2 2 2 2
960 022/300 Write block 4108 len 40960 data: 2 2 2 2 2 2 2 2 2 2
961 023/300 Read block 3907 len 12288
962 024/300 Write block 4588 len 8192 data: 3 3
963 025/300 Read block 4300 len 32768
964 026/300 Flush
965 027/300 Read block 4324 len 28672
966 028/300 Read block 2463 len 28672
967 029/300 Write block 1971 len 28672 data: 2 2 2 2 2 2 2
968 030/300 Read block 3923 len 24576
969 031/300 Read block 2846 len 24576
970 032/300 Flush
971 033/300 Read block 2059 len 24576
972 034/300 Write block 2938 len 20480 data: 2 2 2 2 2
973 035/300 Write block 4287 len 36864 data: 2 2 2 2 2 2 2 2 2
974 036/300 Read block 2035 len 36864
975 037/300 Flush
976 038/300 Flush
977 039/300 Flush
978 040/300 Write block 241 len 36864 data: 3 2 2 2 2 2 3 3 3
979 041/300 Write block 4354 len 24576 data: 2 2 2 2 2 2
980 042/300 Read block 2017 len 24576
981 043/300 Read block 3524 len 4096
982 044/300 Read block 1472 len 20480
983 045/300 Flush
984 046/300 Write block 4946 len 12288 data: 2 2 2
985 047/300 Read block 3359 len 40960
986 048/300 Read block 4965 len 36864
987 049/300 Write block 1063 len 8192 data: 2 2
988 050/300 Read block 2644 len 40960
989 051/300 Read block 1859 len 28672
990 052/300 Read block 1192 len 32768
991 053/300 Write block 2098 len 32768 data: 2 2 2 2 2 2 2 2
992 054/300 Write block 4540 len 32768 data: 2 2 2 2 2 2 2 2
993 055/300 Read block 214 len 28672
994 056/300 Read block 429 len 20480
995 057/300 Write block 4682 len 20480 data: 2 2 2 2 2
996 058/300 Read block 984 len 24576
997 059/300 Write block 81 len 36864 data: 2 2 2 2 2 2 2 2 3
998 060/300 Flush
999 061/300 Write block 2417 len 4096 data: 2
1000 062/300 Write block 1924 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1001 063/300 Write block 4176 len 16384 data: 2 2 3 3
1002 064/300 Read block 4118 len 4096
1003 065/300 Write block 720 len 20480 data: 2 2 2 2 2
1004 066/300 Write block 2825 len 28672 data: 2 2 2 2 2 2 2
1005 067/300 Read block 3456 len 36864
1006 068/300 Write block 2987 len 24576 data: 2 2 2 2 2 2
1007 069/300 Read block 2783 len 24576
1008 070/300 Write block 2062 len 28672 data: 2 2 2 2 2 2 2
1009 071/300 Write block 2602 len 12288 data: 2 2 2
1010 072/300 Write block 3528 len 32768 data: 2 2 2 2 2 2 2 2
1011 073/300 Write block 1250 len 8192 data: 2 2
1012 074/300 Read block 3609 len 4096
1013 075/300 Read block 1049 len 24576
1014 076/300 Write block 1665 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1015 077/300 Read block 4635 len 4096
1016 078/300 Write block 4792 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1017 079/300 Write block 721 len 40960 data: 3 3 3 3 2 2 2 2 2 2
1018 080/300 Read block 4936 len 8192
1019 081/300 Flush
1020 082/300 Read block 2611 len 28672
1021 083/300 Write block 3918 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1022 084/300 Write block 57 len 4096 data: 2
1023 085/300 Write block 4244 len 24576 data: 2 2 2 2 2 2
1024 086/300 Write block 3292 len 20480 data: 2 2 2 2 2
1025 087/300 Read block 4491 len 8192
1026 088/300 Write block 1182 len 12288 data: 2 2 2
1027 089/300 Read block 4261 len 28672
1028 090/300 Write block 4970 len 20480 data: 3 3 3 2 2
1029 091/300 Write block 734 len 8192 data: 2 2
1030 092/300 Read block 1748 len 24576
1031 093/300 Flush
1032 094/300 Read block 4729 len 4096
1033 095/300 Write block 3867 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1034 096/300 Read block 623 len 24576
1035 097/300 Read block 3068 len 12288
1036 098/300 Read block 1377 len 32768
1037 099/300 Read block 1545 len 16384
1038 100/300 Flush
1039 101/300 Read block 748 len 4096
1040 102/300 Read block 2806 len 32768
1041 103/300 Read block 2565 len 12288
1042 104/300 Flush
1043 105/300 Read block 2838 len 36864
1044 106/300 Read block 1598 len 40960
1045 107/300 Read block 2113 len 12288
1046 108/300 Read block 3793 len 4096
1047 109/300 Write block 1543 len 16384 data: 2 2 2 2
1048 110/300 Read block 2028 len 36864
1049 111/300 Read block 3485 len 32768
1050 112/300 Write block 4193 len 24576 data: 2 2 2 2 2 2
1051 113/300 Read block 172 len 24576
1052 114/300 Read block 4762 len 24576
1053 115/300 Read block 4703 len 40960
1054 116/300 Read block 802 len 32768
1055 117/300 Write block 1525 len 16384 data: 2 2 2 2
1056 118/300 Read block 824 len 32768
1057 119/300 Read block 986 len 28672
1058 120/300 Write block 3480 len 8192 data: 2 2
1059 121/300 Write block 1169 len 8192 data: 2 2
1060 122/300 Flush
1061 123/300 Read block 3525 len 40960
1062 124/300 Read block 3649 len 32768
1063 125/300 Read block 503 len 32768
1064 126/300 Write block 760 len 24576 data: 2 2 2 2 2 2
1065 127/300 Read block 2466 len 24576
1066 128/300 Read block 148 len 20480
1067 129/300 Read block 3460 len 16384
1068 130/300 Read block 2657 len 20480
1069 131/300 Read block 952 len 40960
1070 132/300 Read block 4002 len 4096
1071 133/300 Read block 4380 len 20480
1072 134/300 Write block 1343 len 16384 data: 2 2 2 2
1073 135/300 Read block 3130 len 40960
1074 136/300 Read block 3969 len 12288
1075 137/300 Read block 1893 len 4096
1076 138/300 Write block 4445 len 24576 data: 2 2 2 2 2 2
1077 139/300 Flush
1078 140/300 Read block 1199 len 8192
1079 141/300 Write block 4343 len 16384 data: 2 2 3 3
1080 142/300 Write block 4596 len 40960 data: 2 2 2 2 3 3 4 4 4 2
1081 143/300 Write block 4197 len 24576 data: 3 3 2 2 2 3
1082 144/300 Write block 3414 len 12288 data: 3 3 2
1083 145/300 Flush
1084 146/300 Read block 3921 len 40960
1085 147/300 Write block 2994 len 20480 data: 2 2 2 2 2
1086 148/300 Read block 466 len 12288
1087 149/300 Flush
1088 150/300 Write block 3678 len 32768 data: 2 2 2 2 2 2 2 2
1089 151/300 Write block 532 len 20480 data: 2 2 2 2 2
1090 152/300 Read block 1558 len 20480
1091 153/300 Read block 2695 len 20480
1092 154/300 Flush
1093 155/300 Write block 1921 len 32768 data: 2 2 2 3 3 3 3 3
1094 156/300 Flush
1095 157/300 Write block 2754 len 36864 data: 2 2 2 2 2 2 2 2 2
1096 158/300 Flush
1097 159/300 Write block 1618 len 28672 data: 2 2 2 2 2 2 2
1098 160/300 Write block 4734 len 4096 data: 2
1099 161/300 Read block 4221 len 20480
1100 162/300 Write block 3278 len 8192 data: 2 2
1101 163/300 Write block 11 len 4096 data: 2
1102 164/300 Flush
1103 165/300 Write block 676 len 12288 data: 2 2 2
1104 166/300 Read block 1729 len 24576
1105 167/300 Read block 4602 len 12288
1106 168/300 Write block 817 len 20480 data: 2 2 2 2 2
1107 169/300 Write block 1316 len 28672 data: 2 2 2 2 2 2 3
1108 170/300 Read block 4796 len 32768
1109 171/300 Write block 1839 len 36864 data: 2 2 2 2 2 2 2 2 2
1110 172/300 Write block 30 len 36864 data: 2 2 2 2 2 2 2 2 2
1111 173/300 Read block 4210 len 28672
1112 174/300 Write block 2696 len 24576 data: 2 2 2 2 2 2
1113 175/300 Write block 110 len 16384 data: 2 2 2 2
1114 176/300 Read block 1804 len 36864
1115 177/300 Write block 4888 len 4096 data: 2
1116 178/300 Write block 304 len 8192 data: 2 2
1117 179/300 Write block 3859 len 28672 data: 3 3 3 3 4 3 3
1118 180/300 Flush
1119 181/300 Write block 2796 len 24576 data: 2 2 2 2 2 2
1120 182/300 Write block 2808 len 28672 data: 2 2 2 2 2 2 2
1121 183/300 Read block 3374 len 32768
1122 184/300 Read block 310 len 40960
1123 185/300 Flush
1124 186/300 Read block 4369 len 16384
1125 187/300 Flush
1126 188/300 Read block 3431 len 24576
1127 189/300 Read block 2565 len 32768
1128 190/300 Read block 4638 len 32768
1129 191/300 Read block 1307 len 12288
1130 192/300 Read block 4094 len 24576
1131 193/300 Write block 4864 len 12288 data: 2 2 2
1132 194/300 Read block 3227 len 36864
1133 195/300 Write block 1628 len 24576 data: 3 3 3 3 2 2
1134 196/300 Write block 1318 len 20480 data: 3 3 3 3 4
1135 197/300 Write block 2848 len 12288 data: 3 3 2
1136 198/300 Read block 2443 len 20480
1137 199/300 Write block 2364 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1138 200/300 Write block 452 len 12288 data: 2 2 2
1139 201/300 Read block 1129 len 12288
1140 202/300 Write block 2156 len 24576 data: 2 2 2 2 2 2
1141 203/300 Read block 3305 len 24576
1142 204/300 Read block 1450 len 40960
1143 205/300 Read block 350 len 12288
1144 206/300 Read block 2144 len 36864
1145 207/300 Read block 70 len 36864
1146 208/300 Write block 3503 len 20480 data: 2 2 2 2 2
1147 209/300 Read block 1492 len 8192
1148 210/300 Write block 3676 len 32768 data: 2 2 3 3 3 3 3 3
1149 211/300 Flush
1150 212/300 Write block 3801 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1151 213/300 Write block 3519 len 24576 data: 2 2 2 2 2 2
1152 214/300 Write block 2128 len 40960 data: 2 2 2 3 3 3 3 3 2 2
1153 215/300 Write block 4345 len 12288 data: 4 4 3
1154 216/300 Read block 4430 len 12288
1155 217/300 Write block 2129 len 16384 data: 3 3 4 4
1156 218/300 Read block 2881 len 4096
1157 219/300 Write block 3808 len 12288 data: 3 3 3
1158 220/300 Write block 4430 len 12288 data: 2 2 2
1159 221/300 Read block 2908 len 36864
1160 222/300 Read block 3633 len 32768
1161 223/300 Flush
1162 224/300 Write block 2635 len 36864 data: 2 2 2 2 2 2 2 2 2
1163 225/300 Read block 2883 len 28672
1164 226/300 Write block 4684 len 28672 data: 3 3 3 2 2 2 2
1165 227/300 Read block 3321 len 4096
1166 228/300 Write block 2743 len 28672 data: 2 2 2 2 2 2 2
1167 229/300 Read block 1350 len 12288
1168 230/300 Read block 1831 len 24576
1169 231/300 Read block 1920 len 32768
1170 232/300 Write block 2510 len 28672 data: 3 2 2 2 2 2 2
1171 233/300 Read block 44 len 12288
1172 234/300 Read block 419 len 8192
1173 235/300 Flush
1174 236/300 Read block 4424 len 12288
1175 237/300 Write block 67 len 16384 data: 2 2 2 2
1176 238/300 Read block 1551 len 40960
1177 239/300 Write block 1552 len 24576 data: 2 2 2 2 2 2
1178 240/300 Write block 4759 len 36864 data: 2 2 2 2 2 2 2 2 2
1179 241/300 Read block 3807 len 24576
1180 242/300 Write block 4120 len 16384 data: 2 2 2 2
1181 243/300 Read block 1437 len 28672
1182 244/300 Read block 3881 len 40960
1183 245/300 Write block 4024 len 28672 data: 2 2 2 2 2 2 2
1184 246/300 Write block 2555 len 32768 data: 2 2 2 2 2 2 2 2
1185 247/300 Read block 2572 len 36864
1186 248/300 Flush
1187 249/300 Flush
1188 250/300 Write block 2320 len 12288 data: 3 3 3
1189 251/300 Read block 3943 len 16384
1190 252/300 Read block 1515 len 24576
1191 253/300 Write block 3755 len 4096 data: 2
1192 254/300 Read block 3312 len 40960
1193 255/300 Read block 3150 len 20480
1194 256/300 Flush
1195 257/300 Write block 2828 len 20480 data: 3 3 3 3 2
1196 258/300 Write block 1964 len 4096 data: 2
1197 259/300 Read block 1171 len 20480
1198 260/300 Read block 323 len 36864
1199 261/300 Read block 4024 len 4096
1200 262/300 Read block 4166 len 4096
1201 263/300 Read block 1776 len 8192
1202 264/300 Read block 959 len 8192
1203 265/300 Write block 2630 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1204 266/300 Read block 1076 len 12288
1205 267/300 Read block 2181 len 24576
1206 268/300 Write block 3758 len 4096 data: 2
1207 269/300 Read block 2079 len 24576
1208 270/300 Read block 2747 len 8192
1209 271/300 Write block 3934 len 32768 data: 2 2 2 2 2 2 2 2
1210 272/300 Read block 2004 len 20480
1211 273/300 Read block 4511 len 28672
1212 274/300 Read block 3493 len 8192
1213 275/300 Write block 240 len 8192 data: 3 4
1214 276/300 Read block 134 len 12288
1215 277/300 Flush
1216 278/300 Write block 1015 len 8192 data: 2 2
1217 279/300 Read block 2269 len 20480
1218 280/300 Write block 3746 len 16384 data: 2 2 2 2
1219 281/300 Read block 819 len 40960
1220 282/300 Read block 1483 len 36864
1221 283/300 Flush
1222 284/300 Read block 3777 len 40960
1223 285/300 Write block 3021 len 36864 data: 3 3 3 3 2 2 2 2 2
1224 286/300 Write block 1047 len 12288 data: 2 2 2
1225 287/300 Read block 474 len 40960
1226 288/300 Write block 892 len 24576 data: 2 2 2 2 2 2
1227 289/300 Read block 4281 len 24576
1228 290/300 Read block 3797 len 32768
1229 291/300 Read block 1820 len 28672
1230 292/300 Read block 2607 len 24576
1231 293/300 Read block 4445 len 28672
1232 294/300 Write block 2272 len 32768 data: 2 2 2 2 2 2 2 2
1233 295/300 Flush
1234 296/300 Read block 2287 len 40960
1235 297/300 Write block 107 len 32768 data: 2 2 2 3 3 3 3 2
1236 298/300 Read block 2984 len 4096
1237 299/300 Read block 1064 len 32768
1238 300/300 Read block 1030 len 28672
1239 Main task has work to do, go find it
1240 got dsc Start(1)
1241 start 1
1242 Mar 02 01:05:52.191 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 10cc0f94-c332-45f3-bce9-a7cd7370c8a4, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
1243 Replay: started 1, returned:Ok(())
1244 ----------------------------------------------------------------
1245 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
1246 States: Active Offline Active
1247 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1248 303 Acked 1302 Flush 0 Done New Done true
1249 304 Acked 1303 Read 10 Done New Done true
1250 305 Acked 1304 Read 3 Done New Done false
1251 306 Acked 1305 Read 7 Done New Done false
1252 307 Acked 1306 Read 3 Done New Done false
1253 308 Acked 1307 Read 1 Done New Done false
1254 309 Acked 1308 Write 6 Done New Done false
1255 310 Acked 1309 Write 6 Done New Done false
1256 311 Acked 1310 Read 1 Done New Done false
1257 312 Acked 1311 Write 7 Done New Done false
1258 313 Acked 1312 Write 5 Done New Done false
1259 314 Acked 1313 Read 10 Done New Done false
1260 315 Acked 1314 Write 6 Done New Done false
1261 316 Acked 1315 Write 7 Done New Done false
1262 317 Acked 1316 Read 2 Done New Done false
1263 318 Acked 1317 Read 6 Done New Done false
1264 319 Acked 1318 Read 2 Done New Done false
1265 320 Acked 1319 Read 10 Done New Done false
1266 321 Acked 1320 Read 10 Done New Done false
1267 322 Acked 1321 Write 1 Done New Done false
1268 323 Acked 1322 Write 5 Done New Done false
1269 324 Acked 1323 Write 10 Done New Done false
1270 325 Acked 1324 Read 3 Done New Done false
1271 326 Acked 1325 Write 2 Done New Done false
1272 327 Acked 1326 Read 8 Done New Done false
1273 328 Acked 1327 Flush 0 Done New Done false
1274 329 Acked 1328 Read 7 Done New Done false
1275 330 Acked 1329 Read 7 Done New Done false
1276 331 Acked 1330 Write 7 Done New Done false
1277 332 Acked 1331 Read 6 Done New Done false
1278 333 Acked 1332 Read 6 Done New Done false
1279 334 Acked 1333 Flush 0 Done New Done false
1280 335 Acked 1334 Read 6 Done New Done false
1281 336 Acked 1335 Write 5 Done New Done false
1282 337 Acked 1336 Write 9 Done New Done false
1283 338 Acked 1337 Read 9 Done New Done false
1284 339 Acked 1338 Flush 0 Done New Done false
1285 340 Acked 1339 Flush 0 Done New Done false
1286 341 Acked 1340 Flush 0 Done New Done false
1287 342 Acked 1341 Write 9 Done New Done false
1288 343 Acked 1342 Write 6 Done New Done false
1289 344 Acked 1343 Read 6 Done New Done false
1290 345 Acked 1344 Read 1 Done New Done false
1291 346 Acked 1345 Read 5 Done New Done false
1292 347 Acked 1346 Flush 0 Done New Done false
1293 348 Acked 1347 Write 3 Done New Done false
1294 349 Acked 1348 Read 10 Done New Done false
1295 350 Acked 1349 Read 9 Done New Done false
1296 351 Acked 1350 Write 2 Done New Done false
1297 352 Acked 1351 Read 10 Done New Done false
1298 353 Acked 1352 Read 7 Done New Done false
1299 354 Acked 1353 Read 8 Done New Done false
1300 355 Acked 1354 Write 8 Done New Done false
1301 356 Acked 1355 Write 8 Done New Done false
1302 357 Acked 1356 Read 7 Done New Done false
1303 358 Acked 1357 Read 5 Done New Done false
1304 359 Acked 1358 Write 5 Done New Done false
1305 360 Acked 1359 Read 6 Done New Done false
1306 361 Acked 1360 Write 9 Done New Done false
1307 362 Acked 1361 Flush 0 Done New Done false
1308 363 Acked 1362 Write 1 Done New Done false
1309 364 Acked 1363 Write 10 Done New Done false
1310 365 Acked 1364 Write 4 Done New Done false
1311 366 Acked 1365 Read 1 Done New Done false
1312 367 Acked 1366 Write 5 Done New Done false
1313 368 Acked 1367 Write 7 Done New Done false
1314 369 Acked 1368 Read 9 Done New Done false
1315 370 Acked 1369 Write 6 Done New Done false
1316 371 Acked 1370 Read 6 Done New Done false
1317 372 Acked 1371 Write 7 Done New Done false
1318 373 Acked 1372 Write 3 Done New Done false
1319 374 Acked 1373 Write 8 Done New Done false
1320 375 Acked 1374 Write 2 Done New Done false
1321 376 Acked 1375 Read 1 Done New Done false
1322 377 Acked 1376 Read 6 Done New Done false
1323 378 Acked 1377 Write 10 Done New Done false
1324 379 Acked 1378 Read 1 Done New Done false
1325 380 Acked 1379 Write 10 Done New Done false
1326 381 Acked 1380 Write 10 Done New Done false
1327 382 Acked 1381 Read 2 Done New Done false
1328 383 Acked 1382 Flush 0 Done New Done false
1329 384 Acked 1383 Read 7 Done New Done false
1330 385 Acked 1384 Write 10 Done New Done false
1331 386 Acked 1385 Write 1 Done New Done false
1332 387 Acked 1386 Write 6 Done New Done false
1333 388 Acked 1387 Write 5 Done New Done false
1334 389 Acked 1388 Read 2 Done New Done false
1335 390 Acked 1389 Write 3 Done New Done false
1336 391 Acked 1390 Read 7 Done New Done false
1337 392 Acked 1391 Write 5 Done New Done false
1338 393 Acked 1392 Write 2 Done New Done false
1339 394 Acked 1393 Read 6 Done New Done false
1340 395 Acked 1394 Flush 0 Done New Done false
1341 396 Acked 1395 Read 1 Done New Done false
1342 397 Acked 1396 Write 10 Done New Done false
1343 398 Acked 1397 Read 6 Done New Done false
1344 399 Acked 1398 Read 3 Done New Done false
1345 400 Acked 1399 Read 8 Done New Done false
1346 401 Acked 1400 Read 4 Done New Done false
1347 402 Acked 1401 Flush 0 Done New Done false
1348 403 Acked 1402 Read 1 Done New Done false
1349 404 Acked 1403 Read 8 Done New Done false
1350 405 Acked 1404 Read 3 Done New Done false
1351 406 Acked 1405 Flush 0 Done New Done false
1352 407 Acked 1406 Read 9 Done New Done false
1353 408 Acked 1407 Read 10 Done New Done false
1354 409 Acked 1408 Read 3 Done New Done false
1355 410 Acked 1409 Read 1 Done New Done false
1356 411 Acked 1410 Write 4 Done New Done false
1357 412 Acked 1411 Read 9 Done New Done false
1358 413 Acked 1412 Read 8 Done New Done false
1359 414 Acked 1413 Write 6 Done New Done false
1360 415 Acked 1414 Read 6 Done New Done false
1361 416 Acked 1415 Read 6 Done New Done false
1362 417 Acked 1416 Read 10 Done New Done false
1363 418 Acked 1417 Read 8 Done New Done false
1364 419 Acked 1418 Write 4 Done New Done false
1365 420 Acked 1419 Read 8 Done New Done false
1366 421 Acked 1420 Read 7 Done New Done false
1367 422 Acked 1421 Write 2 Done New Done false
1368 423 Acked 1422 Write 2 Done New Done false
1369 424 Acked 1423 Flush 0 Done New Done false
1370 425 Acked 1424 Read 10 Done New Done false
1371 426 Acked 1425 Read 8 Done New Done false
1372 427 Acked 1426 Read 8 Done New Done false
1373 428 Acked 1427 Write 6 Done New Done false
1374 429 Acked 1428 Read 6 Done New Done false
1375 430 Acked 1429 Read 5 Done New Done false
1376 431 Acked 1430 Read 4 Done New Done false
1377 432 Acked 1431 Read 5 Done New Done false
1378 433 Acked 1432 Read 10 Done New Done false
1379 434 Acked 1433 Read 1 Done New Done false
1380 435 Acked 1434 Read 5 Done New Done false
1381 436 Acked 1435 Write 4 Done New Done false
1382 437 Acked 1436 Read 10 Done New Done false
1383 438 Acked 1437 Read 3 Done New Done false
1384 439 Acked 1438 Read 1 Done New Done false
1385 440 Acked 1439 Write 6 Done New Done false
1386 441 Acked 1440 Flush 0 Done New Done false
1387 442 Acked 1441 Read 2 Done New Done false
1388 443 Acked 1442 Write 4 Done New Done false
1389 444 Acked 1443 Write 10 Done New Done false
1390 445 Acked 1444 Write 6 Done New Done false
1391 446 Acked 1445 Write 3 Done New Done false
1392 447 Acked 1446 Flush 0 Done New Done false
1393 448 Acked 1447 Read 10 Done New Done false
1394 449 Acked 1448 Write 5 Done New Done false
1395 450 Acked 1449 Read 3 Done New Done false
1396 451 Acked 1450 Flush 0 Done New Done false
1397 452 Acked 1451 Write 8 Done New Done false
1398 453 Acked 1452 Write 5 Done New Done false
1399 454 Acked 1453 Read 5 Done New Done false
1400 455 Acked 1454 Read 5 Done New Done false
1401 456 Acked 1455 Flush 0 Done New Done false
1402 457 Acked 1456 Write 8 Done New Done false
1403 458 Acked 1457 Flush 0 Done New Done false
1404 459 Acked 1458 Write 9 Done New Done false
1405 460 Acked 1459 Flush 0 Done New Done false
1406 461 Acked 1460 Write 7 Done New Done false
1407 462 Acked 1461 Write 1 Done New Done false
1408 463 Acked 1462 Read 5 Done New Done false
1409 464 Acked 1463 Write 2 Done New Done false
1410 465 Acked 1464 Write 1 Done New Done false
1411 466 Acked 1465 Flush 0 Done New Done false
1412 467 Acked 1466 Write 3 Done New Done false
1413 468 Acked 1467 Read 6 Done New Done false
1414 469 Acked 1468 Read 3 Done New Done false
1415 470 Acked 1469 Write 5 Done New Done false
1416 471 Acked 1470 Write 7 Done New Done false
1417 472 Acked 1471 Read 8 Done New Done false
1418 473 Acked 1472 Write 9 Done New Done false
1419 474 Acked 1473 Write 9 Done New Done false
1420 475 Acked 1474 Read 7 Done New Done false
1421 476 Acked 1475 Write 6 Done New Done false
1422 477 Acked 1476 Write 4 Done New Done false
1423 478 Acked 1477 Read 9 Done New Done false
1424 479 Acked 1478 Write 1 Done New Done false
1425 480 Acked 1479 Write 2 Done New Done false
1426 481 Acked 1480 Write 7 Done New Done false
1427 482 Acked 1481 Flush 0 Done New Done false
1428 483 Acked 1482 Write 6 Done New Done false
1429 484 Acked 1483 Write 7 Done New Done false
1430 485 Acked 1484 Read 8 Done New Done false
1431 486 Acked 1485 Read 10 Done New Done false
1432 487 Acked 1486 Flush 0 Done New Done false
1433 488 Acked 1487 Read 4 Done New Done false
1434 489 Acked 1488 Flush 0 Done New Done false
1435 490 Acked 1489 Read 6 Done New Done false
1436 491 Acked 1490 Read 8 Done New Done false
1437 492 Acked 1491 Read 8 Done New Done false
1438 493 Acked 1492 Read 3 Done New Done false
1439 494 Acked 1493 Read 6 Done New Done false
1440 495 Acked 1494 Write 3 Done New Done false
1441 496 Acked 1495 Read 9 Done New Done false
1442 497 Acked 1496 Write 6 Done New Done false
1443 498 Acked 1497 Write 5 Done New Done false
1444 499 Acked 1498 Write 3 Done New Done false
1445 500 Acked 1499 Read 5 Done New Done false
1446 501 Acked 1500 Write 10 Done New Done false
1447 502 Acked 1501 Write 3 Done New Done false
1448 503 Acked 1502 Read 3 Done New Done false
1449 504 Acked 1503 Write 6 Done New Done false
1450 505 Acked 1504 Read 6 Done New Done false
1451 506 Acked 1505 Read 10 Done New Done false
1452 507 Acked 1506 Read 3 Done New Done false
1453 508 Acked 1507 Read 9 Done New Done false
1454 509 Acked 1508 Read 9 Done New Done false
1455 510 Acked 1509 Write 5 Done New Done false
1456 511 Acked 1510 Read 2 Done New Done false
1457 512 Acked 1511 Write 8 Done New Done false
1458 513 Acked 1512 Flush 0 Done New Done false
1459 514 Acked 1513 Write 10 Done New Done false
1460 515 Acked 1514 Write 6 Done New Done false
1461 516 Acked 1515 Write 10 Done New Done false
1462 517 Acked 1516 Write 3 Done New Done false
1463 518 Acked 1517 Read 3 Done New Done false
1464 519 Acked 1518 Write 4 Done New Done false
1465 520 Acked 1519 Read 1 Done New Done false
1466 521 Acked 1520 Write 3 Done New Done false
1467 522 Acked 1521 Write 3 Done New Done false
1468 523 Acked 1522 Read 9 Done New Done false
1469 524 Acked 1523 Read 8 Done New Done false
1470 525 Acked 1524 Flush 0 Done New Done false
1471 526 Acked 1525 Write 9 Done New Done false
1472 527 Acked 1526 Read 7 Done New Done false
1473 528 Acked 1527 Write 7 Done New Done false
1474 529 Acked 1528 Read 1 Done New Done false
1475 530 Acked 1529 Write 7 Done New Done false
1476 531 Acked 1530 Read 3 Done New Done false
1477 532 Acked 1531 Read 6 Done New Done false
1478 533 Acked 1532 Read 8 Done New Done false
1479 534 Acked 1533 Write 7 Done New Done false
1480 535 Acked 1534 Read 3 Done New Done false
1481 536 Acked 1535 Read 2 Done New Done false
1482 537 Acked 1536 Flush 0 Done New Done false
1483 538 Acked 1537 Read 3 Done New Done false
1484 539 Acked 1538 Write 4 Done New Done false
1485 540 Acked 1539 Read 10 Done New Done false
1486 541 Acked 1540 Write 6 Done New Done false
1487 542 Acked 1541 Write 9 Done New Done false
1488 543 Acked 1542 Read 6 Done New Done false
1489 544 Acked 1543 Write 4 Done New Done false
1490 545 Acked 1544 Read 7 Done New Done false
1491 546 Acked 1545 Read 10 Done New Done false
1492 547 Acked 1546 Write 7 Done New Done false
1493 548 Acked 1547 Write 8 Done New Done false
1494 549 Acked 1548 Read 9 Done New Done false
1495 550 Acked 1549 Flush 0 Done New Done false
1496 551 Acked 1550 Flush 0 Done New Done false
1497 552 Acked 1551 Write 3 Done New Done false
1498 553 Acked 1552 Read 4 Done New Done false
1499 554 Acked 1553 Read 6 Done New Done false
1500 555 Acked 1554 Write 1 Done New Done false
1501 556 Acked 1555 Read 10 Done New Done false
1502 557 Acked 1556 Read 5 Done New Done false
1503 558 Acked 1557 Flush 0 Done New Done false
1504 559 Acked 1558 Write 5 Done New Done false
1505 560 Acked 1559 Write 1 Done New Done false
1506 561 Acked 1560 Read 5 Done New Done false
1507 562 Acked 1561 Read 9 Done New Done false
1508 563 Acked 1562 Read 1 Done New Done false
1509 564 Acked 1563 Read 1 Done New Done false
1510 565 Acked 1564 Read 2 Done New Done false
1511 566 Acked 1565 Read 2 Done New Done false
1512 567 Acked 1566 Write 10 Done New Done false
1513 568 Acked 1567 Read 3 Done New Done false
1514 569 Acked 1568 Read 6 Done New Done false
1515 570 Acked 1569 Write 1 Done New Done false
1516 571 Acked 1570 Read 6 Done New Done false
1517 572 Acked 1571 Read 2 Done New Done false
1518 573 Acked 1572 Write 8 Done New Done false
1519 574 Acked 1573 Read 5 Done New Done false
1520 575 Acked 1574 Read 7 Done New Done false
1521 576 Acked 1575 Read 2 Done New Done false
1522 577 Acked 1576 Write 2 Done New Done false
1523 578 Acked 1577 Read 3 Done New Done false
1524 579 Acked 1578 Flush 0 Done New Done false
1525 580 Acked 1579 Write 2 Done New Done false
1526 581 Acked 1580 Read 5 Done New Done false
1527 582 Acked 1581 Write 4 Done New Done false
1528 583 Acked 1582 Read 10 Done New Done false
1529 584 Acked 1583 Read 9 Done New Done false
1530 585 Acked 1584 Flush 0 Done New Done false
1531 586 Acked 1585 Read 10 Done New Done false
1532 587 Acked 1586 Write 9 Done New Done false
1533 588 Acked 1587 Write 3 Done New Done false
1534 589 Acked 1588 Read 10 Done New Done false
1535 590 Acked 1589 Write 6 Done New Done false
1536 591 Acked 1590 Read 6 Done New Done false
1537 592 Acked 1591 Read 8 Done New Done false
1538 593 Acked 1592 Read 7 Done New Done false
1539 594 Acked 1593 Read 6 Done New Done false
1540 595 Acked 1594 Read 7 Done New Done false
1541 596 Acked 1595 Write 8 Done New Done false
1542 597 Acked 1596 Flush 0 Done New Done false
1543 598 Acked 1597 Read 10 Done New Done false
1544 599 Acked 1598 Write 8 Done New Done false
1545 600 Acked 1599 Read 1 Done New Done false
1546 601 Acked 1600 Read 8 Done New Done false
1547 602 Acked 1601 Read 7 Done New Done false
1548 STATES DS:0 DS:1 DS:2 TOTAL
1549 New 0 300 0 300
1550 Sent 0 0 0 0
1551 Done 300 0 300 600
1552 Skipped 0 0 0 0
1553 Error 0 0 0 0
1554 Last Flush: 1596 1301 1596
1555 Downstairs last five completed: 1301 1300 1299 1298 1297
1556 Upstairs last five completed: 602 601 600 599 598
1557 CLIENT: Up:0 ds:300 act:2
15582024-03-02T01:05:52.349ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15592024-03-02T01:05:52.349ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15602024-03-02T01:05:52.350ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15612024-03-02T01:05:52.350ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15622024-03-02T01:05:52.350ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15632024-03-02T01:05:52.350ZINFOcrucible: [1] client re-new 300 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15642024-03-02T01:05:52.350ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15652024-03-02T01:05:53.351ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15662024-03-02T01:05:53.351ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15672024-03-02T01:05:53.351ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15682024-03-02T01:05:53.351ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15692024-03-02T01:05:53.351ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15702024-03-02T01:05:53.351ZINFOcrucible: [1] client re-new 301 jobs since flush 1301 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1571 [8820] got start action
1572 [8820] I am going to restart
1573 Starting downstairs at port 8820
1574 Make output file at "/tmp/dsc/downstairs-8820.txt"
1575 [8820][1] reports Starting
15762024-03-02T01:05:53.351ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1577 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1164)
1578 [8820][1] reports Running
15792024-03-02T01:05:54.352ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15802024-03-02T01:05:54.353ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15812024-03-02T01:05:54.354ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15822024-03-02T01:05:54.354ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15832024-03-02T01:05:54.354ZINFOcrucible: send last flush ID to this DS: 1301 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15842024-03-02T01:05:54.354ZINFOcrucible: Replied this last flush ID: 1301 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15852024-03-02T01:05:54.354ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15862024-03-02T01:05:54.354ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15872024-03-02T01:05:54.354ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15882024-03-02T01:05:54.354ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1589 ----------------------------------------------------------------
1590 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1591 Downstairs last five completed: 1602 1601 1600 1599 1598
1592 Upstairs last five completed: 603 602 601 600 599
1593 CLIENT: Up:0 ds:0 act:3
1594 Replay: All jobs finished, all DS active.
1595 Main task has work to do, go find it
1596 got dsc Stop(0)
1597 stop 0
1598 Mar 02 01:05:56.194 INFO request completed, latency_us: 332, response_code:[8810] Got stop action so:false kr:false
1599 204, uri: /stop/cid/0, method: POST, req_id: 143f045b-958a-495a-9ab2-f0152e404d96, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
1600 Mar 02 01:05:56.195 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1ac99686-d541-4b25-82a1-7e36c4fe3236, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
16012024-03-02T01:05:56.195ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16022024-03-02T01:05:56.195ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16032024-03-02T01:05:56.195ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16042024-03-02T01:05:56.195ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16052024-03-02T01:05:56.195ZINFOcrucible: [0] client re-new 0 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16062024-03-02T01:05:56.195ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1607 [8810] Exited with: ExitStatus(unix_wait_status(9))
1608 [8810][0] reports Exit
16092024-03-02T01:05:57.196ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16102024-03-02T01:05:57.196ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16112024-03-02T01:05:57.196ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16122024-03-02T01:05:57.196ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16132024-03-02T01:05:57.196ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16142024-03-02T01:05:57.196ZINFOcrucible: [0] client re-new 0 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16152024-03-02T01:05:57.196ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16162024-03-02T01:05:58.198ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16172024-03-02T01:05:58.198ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16182024-03-02T01:05:58.198ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16192024-03-02T01:05:58.198ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16202024-03-02T01:05:58.198ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16212024-03-02T01:05:58.198ZINFOcrucible: [0] client re-new 0 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16222024-03-02T01:05:58.198ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16232024-03-02T01:05:59.200ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16242024-03-02T01:05:59.200ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16252024-03-02T01:05:59.200ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16262024-03-02T01:05:59.200ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16272024-03-02T01:05:59.200ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16282024-03-02T01:05:59.200ZINFOcrucible: [0] client re-new 0 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16292024-03-02T01:05:59.200ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1630 Mar 02 01:06:00.197 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/0, method: GET, req_id: fe67e611-4461-4876-bf79-4fba9a539a63, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
1631 001/300 Read block 4068 len 32768
1632 002/300 Read block 3557 len 4096
1633 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30003/300 Write block 3699 len 8192 data: 2 2
1634 ,"time":"2024-03-02T01:06:00.200659239Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
16352024-03-02T01:06:00.200ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16362024-03-02T01:06:00.200ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16372024-03-02T01:06:00.200ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16382024-03-02T01:06:00.200ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16392024-03-02T01:06:00.200ZINFOcrucible: [0] client re-new 2 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16402024-03-02T01:06:00.200ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
1641 004/300 Write block 2708 len 8192 data: 2 2
1642 005/300 Write block 1506 len 24576 data: 3 3 3 3 3 3
1643 006/300 Write block 620 len 36864 data: 2 2 2 2 2 2 2 2 2
1644 007/300 Write block 3906 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1645 008/300 Write block 2428 len 16384 data: 2 2 2 2
1646 009/300 Read block 535 len 12288
1647 010/300 Write block 440 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1648 011/300 Write block 2296 len 24576 data: 2 2 2 2 2 2
1649 012/300 Write block 41 len 20480 data: 2 2 2 2 2
1650 013/300 Write block 1430 len 8192 data: 2 2
1651 014/300 Read block 2486 len 24576
1652 015/300 Read block 4290 len 8192
1653 016/300 Write block 1038 len 8192 data: 2 2
1654 017/300 Read block 4455 len 36864
1655 018/300 Read block 4447 len 8192
1656 019/300 Write block 1036 len 12288 data: 2 2 3
1657 020/300 Read block 249 len 28672
1658 021/300 Read block 862 len 12288
1659 022/300 Write block 2954 len 32768 data: 2 3 3 3 3 3 3 3
1660 023/300 Read block 2586 len 12288
1661 024/300 Read block 4666 len 24576
1662 025/300 Write block 2439 len 20480 data: 2 2 2 2 2
1663 026/300 Read block 787 len 8192
1664 027/300 Read block 4372 len 12288
1665 028/300 Read block 4230 len 20480
1666 029/300 Write block 1736 len 24576 data: 2 2 2 2 2 2
1667 030/300 Write block 2228 len 12288 data: 2 2 2
1668 031/300 Write block 1858 len 20480 data: 2 2 2 2 2
1669 032/300 Read block 4760 len 32768
1670 033/300 Write block 4230 len 28672 data: 2 2 2 2 2 2 2
1671 034/300 Write block 3940 len 4096 data: 3
1672 035/300 Write block 34 len 4096 data: 3
1673 036/300 Read block 1743 len 20480
1674 037/300 Write block 4642 len 12288 data: 3 3 3
1675 038/300 Write block 3704 len 20480 data: 2 2 2 2 2
1676 039/300 Read block 710 len 4096
1677 040/300 Write block 1902 len 20480 data: 2 2 2 2 2
1678 041/300 Read block 395 len 16384
1679 042/300 Write block 4679 len 36864 data: 2 2 2 3 3 4 4 4 3
1680 043/300 Write block 4835 len 20480 data: 2 2 2 2 2
1681 044/300 Read block 2949 len 36864
1682 045/300 Flush
1683 046/300 Read block 1644 len 28672
1684 047/300 Read block 4336 len 40960
1685 048/300 Read block 3370 len 32768
1686 049/300 Write block 2743 len 12288 data: 3 3 3
1687 050/300 Read block 2861 len 12288
1688 051/300 Write block 158 len 32768 data: 2 2 2 2 2 2 2 2
1689 052/300 Read block 140 len 12288
1690 053/300 Write block 923 len 16384 data: 2 2 2 3
1691 054/300 Write block 1617 len 12288 data: 2 3 3
1692 055/300 Read block 1293 len 24576
1693 056/300 Write block 1491 len 32768 data: 2 2 2 2 2 2 2 2
1694 057/300 Read block 2126 len 32768
1695 058/300 Write block 2839 len 40960 data: 2 2 2 3 3 3 3 3 3 4
1696 059/300 Read block 1322 len 20480
1697 060/300 Read block 177 len 40960
1698 061/300 Read block 4992 len 28672
1699 062/300 Read block 1802 len 8192
1700 063/300 Read block 1422 len 20480
1701 064/300 Write block 336 len 32768 data: 2 2 2 2 2 2 2 2
1702 065/300 Write block 3650 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1703 066/300 Read block 1819 len 16384
1704 067/300 Read block 1672 len 28672
1705 068/300 Write block 2047 len 32768 data: 2 2 2 2 2 2 2 2
1706 069/300 Write block 1711 len 4096 data: 2
1707 070/300 Write block 4036 len 20480 data: 3 2 2 2 2
1708 071/300 Write block 3984 len 36864 data: 2 2 2 2 2 2 2 2 2
1709 072/300 Read block 1904 len 32768
1710 073/300 Write block 1395 len 16384 data: 2 2 2 2
1711 074/300 Read block 3085 len 40960
1712 075/300 Flush
1713 076/300 Write block 1201 len 16384 data: 2 2 2 2
1714 077/300 Write block 3551 len 16384 data: 2 3 3 3
1715 078/300 Write block 1800 len 24576 data: 2 2 2 2 2 2
1716 079/300 Write block 2159 len 4096 data: 3
1717 080/300 Read block 3869 len 40960
1718 081/300 Read block 2061 len 36864
1719 082/300 Write block 3095 len 24576 data: 2 2 2 2 2 2
1720 083/300 Read block 1483 len 8192
1721 084/300 Read block 685 len 40960
1722 085/300 Read block 2176 len 20480
1723 086/300 Read block 758 len 20480
1724 087/300 Write block 4231 len 28672 data: 3 3 3 3 3 3 2
1725 088/300 Write block 1674 len 8192 data: 3 2
1726 089/300 Read block 1248 len 4096
1727 090/300 Write block 2249 len 24576 data: 4 4 4 4 3 3
1728 091/300 Read block 924 len 40960
1729 092/300 Write block 3412 len 20480 data: 2 2 4 4 3
1730 093/300 Read block 3591 len 28672
1731 094/300 Write block 507 len 12288 data: 2 2 2
1732 095/300 Write block 3780 len 28672 data: 2 2 2 2 2 2 2
1733 096/300 Write block 1608 len 32768 data: 2 2 2 2 2 2 2 2
1734 097/300 Write block 530 len 32768 data: 2 2 3 3 3 3 3 2
1735 098/300 Read block 964 len 36864
1736 099/300 Read block 1732 len 36864
1737 100/300 Write block 644 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1738 101/300 Write block 3334 len 4096 data: 2
1739 102/300 Read block 4754 len 20480
1740 103/300 Write block 65 len 40960 data: 2 2 3 3 3 3 2 2 2 2
1741 104/300 Write block 3691 len 24576 data: 2 2 2 2 2 2
1742 105/300 Read block 2047 len 20480
1743 106/300 Write block 2568 len 4096 data: 2
1744 107/300 Read block 3338 len 4096
1745 108/300 Read block 4479 len 16384
1746 109/300 Write block 3283 len 16384 data: 2 2 2 2
1747 110/300 Read block 2965 len 4096
1748 111/300 Flush
1749 112/300 Read block 3895 len 16384
1750 113/300 Flush
1751 114/300 Write block 4713 len 12288 data: 2 2 2
1752 115/300 Write block 2831 len 40960 data: 4 3 2 2 2 2 2 2 3 3
1753 116/300 Read block 1439 len 24576
1754 117/300 Write block 4867 len 4096 data: 2
1755 118/300 Read block 1727 len 40960
1756 119/300 Read block 1257 len 20480
1757 120/300 Write block 3193 len 12288 data: 3 3 2
1758 121/300 Write block 2765 len 12288 data: 2 2 2
1759 122/300 Write block 1180 len 28672 data: 2 2 3 3 3 2 2
1760 123/300 Write block 4646 len 32768 data: 3 3 2 2 2 2 2 2
1761 124/300 Read block 24 len 36864
1762 125/300 Write block 1200 len 28672 data: 2 3 3 3 3 2 2
1763 126/300 Write block 4804 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1764 127/300 Read block 2047 len 40960
1765 128/300 Write block 1528 len 8192 data: 3 2
1766 129/300 Read block 3449 len 40960
1767 130/300 Read block 3362 len 28672
1768 131/300 Flush
1769 132/300 Read block 3592 len 8192
1770 133/300 Write block 1706 len 32768 data: 2 2 2 2 2 3 2 2
1771 134/300 Write block 4045 len 12288 data: 2 2 2
1772 135/300 Read block 2181 len 28672
1773 136/300 Write block 702 len 8192 data: 2 2
1774 137/300 Read block 4972 len 32768
1775 138/300 Write block 3293 len 24576 data: 3 3 3 3 2 2
1776 139/300 Read block 1629 len 8192
1777 140/300 Write block 4720 len 8192 data: 2 2
1778 141/300 Read block 3173 len 24576
1779 142/300 Read block 3139 len 20480
1780 143/300 Read block 2275 len 24576
1781 144/300 Write block 2482 len 28672 data: 3 3 3 3 2 2 2
1782 145/300 Read block 3607 len 16384
1783 146/300 Read block 1214 len 16384
1784 147/300 Read block 3956 len 16384
1785 148/300 Write block 2421 len 16384 data: 2 2 2 2
1786 149/300 Flush
1787 150/300 Write block 4277 len 20480 data: 2 2 2 2 2
1788 151/300 Read block 1353 len 40960
1789 152/300 Write block 2024 len 8192 data: 2 2
1790 153/300 Read block 4593 len 12288
1791 154/300 Write block 1580 len 4096 data: 2
1792 155/300 Write block 614 len 8192 data: 2 2
1793 156/300 Read block 4511 len 36864
1794 157/300 Write block 1932 len 24576 data: 3 3 2 2 2 2
1795 158/300 Read block 3003 len 24576
1796 159/300 Read block 2970 len 40960
1797 160/300 Flush
1798 161/300 Read block 801 len 24576
1799 162/300 Read block 742 len 16384
1800 163/300 Read block 4301 len 36864
1801 164/300 Write block 640 len 36864 data: 3 3 3 3 3 3 3 3 3
1802 165/300 Write block 2993 len 16384 data: 2 3 3 3
1803 166/300 Write block 2358 len 32768 data: 2 2 2 2 2 2 3 3
1804 167/300 Read block 254 len 36864
1805 168/300 Read block 2591 len 16384
1806 169/300 Read block 3028 len 12288
1807 170/300 Write block 644 len 20480 data: 4 4 4 4 4
1808 171/300 Write block 4196 len 12288 data: 3 4 4
1809 172/300 Write block 2561 len 20480 data: 3 3 2 2 2
1810 173/300 Read block 1749 len 24576
1811 174/300 Read block 583 len 28672
1812 175/300 Write block 1925 len 36864 data: 4 4 4 4 3 3 3 4 4
1813 176/300 Read block 1821 len 24576
1814 177/300 Read block 4733 len 28672
1815 178/300 Write block 2173 len 24576 data: 2 2 2 2 2 2
1816 179/300 Read block 3054 len 4096
1817 180/300 Write block 3514 len 8192 data: 3 3
1818 181/300 Read block 773 len 20480
1819 182/300 Flush
1820 183/300 Write block 1889 len 12288 data: 2 2 2
1821 184/300 Write block 4050 len 12288 data: 2 2 2
1822 185/300 Write block 3447 len 20480 data: 2 2 2 2 2
1823 186/300 Read block 797 len 40960
1824 187/300 Read block 596 len 4096
1825 188/300 Read block 1939 len 28672
1826 189/300 Read block 370 len 36864
1827 190/300 Read block 696 len 36864
1828 191/300 Read block 1350 len 8192
1829 192/300 Read block 4989 len 40960
1830 193/300 Write block 2677 len 12288 data: 3 3 3
1831 194/300 Write block 3959 len 36864 data: 2 2 2 2 2 2 3 3 3
1832 195/300 Read block 1350 len 8192
1833 196/300 Write block 2258 len 24576 data: 3 2 2 2 2 2
1834 197/300 Read block 3246 len 12288
1835 198/300 Read block 1558 len 12288
1836 199/300 Flush
1837 200/300 Read block 217 len 4096
1838 201/300 Read block 628 len 36864
1839 202/300 Flush
1840 203/300 Read block 1241 len 40960
1841 204/300 Write block 4885 len 20480 data: 2 2 2 3 2
1842 205/300 Read block 567 len 24576
1843 206/300 Write block 4807 len 12288 data: 3 3 3
1844 207/300 Write block 3206 len 16384 data: 2 2 2 2
1845 208/300 Read block 2539 len 8192
1846 209/300 Read block 3978 len 28672
1847 210/300 Read block 91 len 16384
1848 211/300 Read block 2154 len 16384
1849 212/300 Write block 610 len 32768 data: 2 2 2 2 3 3 2 2
1850 213/300 Write block 895 len 20480 data: 3 3 3 2 2
1851 214/300 Flush
1852 215/300 Read block 3940 len 20480
1853 216/300 Read block 503 len 12288
1854 217/300 Read block 4846 len 16384
1855 218/300 Write block 3186 len 24576 data: 3 3 3 3 3 3
1856 219/300 Flush
1857 220/300 Read block 4692 len 8192
1858 221/300 Write block 4298 len 36864 data: 3 3 2 2 2 2 2 2 2
1859 222/300 Read block 1132 len 28672
1860 223/300 Read block 2039 len 28672
1861 224/300 Write block 4501 len 16384 data: 2 2 2 2
1862 225/300 Read block 4932 len 16384
1863 226/300 Read block 2501 len 4096
1864 227/300 Write block 2778 len 32768 data: 2 2 2 2 2 2 2 2
1865 228/300 Read block 1332 len 4096
1866 229/300 Write block 4976 len 32768 data: 2 2 2 2 2 2 2 2
1867 230/300 Read block 2964 len 12288
1868 231/300 Read block 3215 len 24576
1869 232/300 Flush
1870 233/300 Read block 1816 len 32768
1871 234/300 Read block 488 len 40960
1872 235/300 Read block 2115 len 12288
1873 236/300 Flush
1874 237/300 Write block 827 len 28672 data: 2 2 2 2 2 2 2
1875 238/300 Read block 2769 len 36864
1876 239/300 Read block 4022 len 4096
1877 240/300 Write block 4363 len 16384 data: 2 2 2 2
1878 241/300 Write block 3177 len 28672 data: 2 2 2 2 2 2 2
1879 242/300 Read block 4855 len 16384
1880 243/300 Read block 1998 len 8192
1881 244/300 Read block 4732 len 16384
1882 245/300 Flush
1883 246/300 Write block 4932 len 32768 data: 2 2 2 2 2 2 2 2
1884 247/300 Read block 4381 len 4096
1885 248/300 Read block 2375 len 24576
1886 249/300 Read block 1489 len 8192
1887 250/300 Read block 4443 len 12288
1888 251/300 Read block 946 len 8192
1889 252/300 Read block 2111 len 4096
1890 253/300 Write block 2604 len 8192 data: 3 2
1891 254/300 Flush
1892 255/300 Write block 3151 len 20480 data: 2 2 2 2 2
1893 256/300 Read block 4703 len 20480
1894 257/300 Read block 1797 len 12288
1895 258/300 Read block 486 len 28672
1896 259/300 Read block 3771 len 12288
1897 260/300 Read block 3009 len 32768
1898 261/300 Read block 1850 len 32768
1899 262/300 Write block 3278 len 40960 data: 3 3 2 2 2 3 3 3 3 2
1900 263/300 Read block 379 len 24576
1901 264/300 Flush
1902 265/300 Read block 4340 len 4096
1903 266/300 Write block 2907 len 12288 data: 2 2 2
1904 267/300 Read block 3478 len 24576
1905 268/300 Write block 2210 len 12288 data: 3 3 3
1906 269/300 Read block 3904 len 4096
1907 270/300 Read block 2181 len 28672
1908 271/300 Write block 36 len 40960 data: 3 3 3 2 2 3 3 3 3 3
1909 272/300 Read block 1067 len 4096
1910 273/300 Read block 2035 len 36864
1911 274/300 Write block 4067 len 20480 data: 2 2 2 2 2
1912 275/300 Write block 2313 len 36864 data: 2 3 3 3 3 3 3 4 4
1913 276/300 Read block 981 len 32768
1914 277/300 Read block 2961 len 28672
1915 278/300 Write block 3329 len 40960 data: 2 2 2 2 2 3 2 2 2 2
1916 279/300 Read block 230 len 12288
1917 280/300 Write block 2456 len 4096 data: 2
1918 281/300 Write block 3060 len 20480 data: 2 3 3 3 2
1919 282/300 Read block 2757 len 20480
1920 283/300 Flush
1921 284/300 Write block 4155 len 4096 data: 2
1922 285/300 Read block 2949 len 12288
1923 286/300 Flush
1924 287/300 Flush
1925 288/300 Read block 2247 len 4096
1926 289/300 Read block 4324 len 36864
1927 290/300 Read block 3607 len 4096
1928 291/300 Write block 1466 len 4096 data: 2
1929 292/300 Write block 4438 len 36864 data: 2 2 2 2 2 2 2 3 3
1930 293/300 Read block 1850 len 4096
1931 294/300 Read block 2819 len 20480
1932 295/300 Write block 1800 len 12288 data: 3 3 3
1933 296/300 Read block 628 len 8192
1934 297/300 Write block 749 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1935 298/300 Write block 3777 len 32768 data: 2 2 2 3 3 3 3 3
1936 299/300 Write block 3631 len 24576 data: 2 2 2 2 2 2
1937 300/300 Read block 2027 len 40960
1938 Main task has work to do, go find it
1939 got dsc Start(0)
1940 start 0
1941 Mar 02 01:06:00.777 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3110334a-4600-448e-811c-b261dfd55599, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
1942 Replay: started 0, returned:Ok(())
1943 ----------------------------------------------------------------
1944 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
1945 States: Offline Active Active
1946 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1947 604 Acked 1603 Read 8 New Done Done true
1948 605 Acked 1604 Read 1 New Done Done true
1949 606 Acked 1605 Write 2 New Done Done false
1950 607 Acked 1606 Write 2 New Done Done false
1951 608 Acked 1607 Write 6 New Done Done false
1952 609 Acked 1608 Write 9 New Done Done false
1953 610 Acked 1609 Write 10 New Done Done false
1954 611 Acked 1610 Write 4 New Done Done false
1955 612 Acked 1611 Read 3 New Done Done false
1956 613 Acked 1612 Write 10 New Done Done false
1957 614 Acked 1613 Write 6 New Done Done false
1958 615 Acked 1614 Write 5 New Done Done false
1959 616 Acked 1615 Write 2 New Done Done false
1960 617 Acked 1616 Read 6 New Done Done false
1961 618 Acked 1617 Read 2 New Done Done false
1962 619 Acked 1618 Write 2 New Done Done false
1963 620 Acked 1619 Read 9 New Done Done false
1964 621 Acked 1620 Read 2 New Done Done false
1965 622 Acked 1621 Write 3 New Done Done false
1966 623 Acked 1622 Read 7 New Done Done false
1967 624 Acked 1623 Read 3 New Done Done false
1968 625 Acked 1624 Write 8 New Done Done false
1969 626 Acked 1625 Read 3 New Done Done false
1970 627 Acked 1626 Read 6 New Done Done false
1971 628 Acked 1627 Write 5 New Done Done false
1972 629 Acked 1628 Read 2 New Done Done false
1973 630 Acked 1629 Read 3 New Done Done false
1974 631 Acked 1630 Read 5 New Done Done false
1975 632 Acked 1631 Write 6 New Done Done false
1976 633 Acked 1632 Write 3 New Done Done false
1977 634 Acked 1633 Write 5 New Done Done false
1978 635 Acked 1634 Read 8 New Done Done false
1979 636 Acked 1635 Write 7 New Done Done false
1980 637 Acked 1636 Write 1 New Done Done false
1981 638 Acked 1637 Write 1 New Done Done false
1982 639 Acked 1638 Read 5 New Done Done false
1983 640 Acked 1639 Write 3 New Done Done false
1984 641 Acked 1640 Write 5 New Done Done false
1985 642 Acked 1641 Read 1 New Done Done false
1986 643 Acked 1642 Write 5 New Done Done false
1987 644 Acked 1643 Read 4 New Done Done false
1988 645 Acked 1644 Write 9 New Done Done false
1989 646 Acked 1645 Write 5 New Done Done false
1990 647 Acked 1646 Read 9 New Done Done false
1991 648 Acked 1647 Flush 0 New Done Done false
1992 649 Acked 1648 Read 7 New Done Done false
1993 650 Acked 1649 Read 10 New Done Done false
1994 651 Acked 1650 Read 8 New Done Done false
1995 652 Acked 1651 Write 3 New Done Done false
1996 653 Acked 1652 Read 3 New Done Done false
1997 654 Acked 1653 Write 8 New Done Done false
1998 655 Acked 1654 Read 3 New Done Done false
1999 656 Acked 1655 Write 4 New Done Done false
2000 657 Acked 1656 Write 3 New Done Done false
2001 658 Acked 1657 Read 6 New Done Done false
2002 659 Acked 1658 Write 8 New Done Done false
2003 660 Acked 1659 Read 8 New Done Done false
2004 661 Acked 1660 Write 10 New Done Done false
2005 662 Acked 1661 Read 5 New Done Done false
2006 663 Acked 1662 Read 10 New Done Done false
2007 664 Acked 1663 Read 7 New Done Done false
2008 665 Acked 1664 Read 2 New Done Done false
2009 666 Acked 1665 Read 5 New Done Done false
2010 667 Acked 1666 Write 8 New Done Done false
2011 668 Acked 1667 Write 10 New Done Done false
2012 669 Acked 1668 Read 4 New Done Done false
2013 670 Acked 1669 Read 7 New Done Done false
2014 671 Acked 1670 Write 8 New Done Done false
2015 672 Acked 1671 Write 1 New Done Done false
2016 673 Acked 1672 Write 5 New Done Done false
2017 674 Acked 1673 Write 9 New Done Done false
2018 675 Acked 1674 Read 8 New Done Done false
2019 676 Acked 1675 Write 4 New Done Done false
2020 677 Acked 1676 Read 10 New Done Done false
2021 678 Acked 1677 Flush 0 New Done Done false
2022 679 Acked 1678 Write 4 New Done Done false
2023 680 Acked 1679 Write 4 New Done Done false
2024 681 Acked 1680 Write 6 New Done Done false
2025 682 Acked 1681 Write 1 New Done Done false
2026 683 Acked 1682 Read 10 New Done Done false
2027 684 Acked 1683 Read 9 New Done Done false
2028 685 Acked 1684 Write 6 New Done Done false
2029 686 Acked 1685 Read 2 New Done Done false
2030 687 Acked 1686 Read 10 New Done Done false
2031 688 Acked 1687 Read 5 New Done Done false
2032 689 Acked 1688 Read 5 New Done Done false
2033 690 Acked 1689 Write 7 New Done Done false
2034 691 Acked 1690 Write 2 New Done Done false
2035 692 Acked 1691 Read 1 New Done Done false
2036 693 Acked 1692 Write 6 New Done Done false
2037 694 Acked 1693 Read 10 New Done Done false
2038 695 Acked 1694 Write 5 New Done Done false
2039 696 Acked 1695 Read 7 New Done Done false
2040 697 Acked 1696 Write 3 New Done Done false
2041 698 Acked 1697 Write 7 New Done Done false
2042 699 Acked 1698 Write 8 New Done Done false
2043 700 Acked 1699 Write 8 New Done Done false
2044 701 Acked 1700 Read 9 New Done Done false
2045 702 Acked 1701 Read 9 New Done Done false
2046 703 Acked 1702 Write 10 New Done Done false
2047 704 Acked 1703 Write 1 New Done Done false
2048 705 Acked 1704 Read 5 New Done Done false
2049 706 Acked 1705 Write 10 New Done Done false
2050 707 Acked 1706 Write 6 New Done Done false
2051 708 Acked 1707 Read 5 New Done Done false
2052 709 Acked 1708 Write 1 New Done Done false
2053 710 Acked 1709 Read 1 New Done Done false
2054 711 Acked 1710 Read 4 New Done Done false
2055 712 Acked 1711 Write 4 New Done Done false
2056 713 Acked 1712 Read 1 New Done Done false
2057 714 Acked 1713 Flush 0 New Done Done false
2058 715 Acked 1714 Read 4 New Done Done false
2059 716 Acked 1715 Flush 0 New Done Done false
2060 717 Acked 1716 Write 3 New Done Done false
2061 718 Acked 1717 Write 10 New Done Done false
2062 719 Acked 1718 Read 6 New Done Done false
2063 720 Acked 1719 Write 1 New Done Done false
2064 721 Acked 1720 Read 10 New Done Done false
2065 722 Acked 1721 Read 5 New Done Done false
2066 723 Acked 1722 Write 3 New Done Done false
2067 724 Acked 1723 Write 3 New Done Done false
2068 725 Acked 1724 Write 7 New Done Done false
2069 726 Acked 1725 Write 8 New Done Done false
2070 727 Acked 1726 Read 9 New Done Done false
2071 728 Acked 1727 Write 7 New Done Done false
2072 729 Acked 1728 Write 10 New Done Done false
2073 730 Acked 1729 Read 10 New Done Done false
2074 731 Acked 1730 Write 2 New Done Done false
2075 732 Acked 1731 Read 10 New Done Done false
2076 733 Acked 1732 Read 7 New Done Done false
2077 734 Acked 1733 Flush 0 New Done Done false
2078 735 Acked 1734 Read 2 New Done Done false
2079 736 Acked 1735 Write 8 New Done Done false
2080 737 Acked 1736 Write 3 New Done Done false
2081 738 Acked 1737 Read 7 New Done Done false
2082 739 Acked 1738 Write 2 New Done Done false
2083 740 Acked 1739 Read 8 New Done Done false
2084 741 Acked 1740 Write 6 New Done Done false
2085 742 Acked 1741 Read 2 New Done Done false
2086 743 Acked 1742 Write 2 New Done Done false
2087 744 Acked 1743 Read 6 New Done Done false
2088 745 Acked 1744 Read 5 New Done Done false
2089 746 Acked 1745 Read 6 New Done Done false
2090 747 Acked 1746 Write 7 New Done Done false
2091 748 Acked 1747 Read 4 New Done Done false
2092 749 Acked 1748 Read 4 New Done Done false
2093 750 Acked 1749 Read 4 New Done Done false
2094 751 Acked 1750 Write 4 New Done Done false
2095 752 Acked 1751 Flush 0 New Done Done false
2096 753 Acked 1752 Write 5 New Done Done false
2097 754 Acked 1753 Read 10 New Done Done false
2098 755 Acked 1754 Write 2 New Done Done false
2099 756 Acked 1755 Read 3 New Done Done false
2100 757 Acked 1756 Write 1 New Done Done false
2101 758 Acked 1757 Write 2 New Done Done false
2102 759 Acked 1758 Read 9 New Done Done false
2103 760 Acked 1759 Write 6 New Done Done false
2104 761 Acked 1760 Read 6 New Done Done false
2105 762 Acked 1761 Read 10 New Done Done false
2106 763 Acked 1762 Flush 0 New Done Done false
2107 764 Acked 1763 Read 6 New Done Done false
2108 765 Acked 1764 Read 4 New Done Done false
2109 766 Acked 1765 Read 9 New Done Done false
2110 767 Acked 1766 Write 9 New Done Done false
2111 768 Acked 1767 Write 4 New Done Done false
2112 769 Acked 1768 Write 8 New Done Done false
2113 770 Acked 1769 Read 9 New Done Done false
2114 771 Acked 1770 Read 4 New Done Done false
2115 772 Acked 1771 Read 3 New Done Done false
2116 773 Acked 1772 Write 5 New Done Done false
2117 774 Acked 1773 Write 3 New Done Done false
2118 775 Acked 1774 Write 5 New Done Done false
2119 776 Acked 1775 Read 6 New Done Done false
2120 777 Acked 1776 Read 7 New Done Done false
2121 778 Acked 1777 Write 9 New Done Done false
2122 779 Acked 1778 Read 6 New Done Done false
2123 780 Acked 1779 Read 7 New Done Done false
2124 781 Acked 1780 Write 6 New Done Done false
2125 782 Acked 1781 Read 1 New Done Done false
2126 783 Acked 1782 Write 2 New Done Done false
2127 784 Acked 1783 Read 5 New Done Done false
2128 785 Acked 1784 Flush 0 New Done Done false
2129 786 Acked 1785 Write 3 New Done Done false
2130 787 Acked 1786 Write 3 New Done Done false
2131 788 Acked 1787 Write 5 New Done Done false
2132 789 Acked 1788 Read 10 New Done Done false
2133 790 Acked 1789 Read 1 New Done Done false
2134 791 Acked 1790 Read 7 New Done Done false
2135 792 Acked 1791 Read 9 New Done Done false
2136 793 Acked 1792 Read 9 New Done Done false
2137 794 Acked 1793 Read 2 New Done Done false
2138 795 Acked 1794 Read 10 New Done Done false
2139 796 Acked 1795 Write 3 New Done Done false
2140 797 Acked 1796 Write 9 New Done Done false
2141 798 Acked 1797 Read 2 New Done Done false
2142 799 Acked 1798 Write 6 New Done Done false
2143 800 Acked 1799 Read 3 New Done Done false
2144 801 Acked 1800 Read 3 New Done Done false
2145 802 Acked 1801 Flush 0 New Done Done false
2146 803 Acked 1802 Read 1 New Done Done false
2147 804 Acked 1803 Read 9 New Done Done false
2148 805 Acked 1804 Flush 0 New Done Done false
2149 806 Acked 1805 Read 10 New Done Done false
2150 807 Acked 1806 Write 5 New Done Done false
2151 808 Acked 1807 Read 6 New Done Done false
2152 809 Acked 1808 Write 3 New Done Done false
2153 810 Acked 1809 Write 4 New Done Done false
2154 811 Acked 1810 Read 2 New Done Done false
2155 812 Acked 1811 Read 7 New Done Done false
2156 813 Acked 1812 Read 4 New Done Done false
2157 814 Acked 1813 Read 4 New Done Done false
2158 815 Acked 1814 Write 8 New Done Done false
2159 816 Acked 1815 Write 5 New Done Done false
2160 817 Acked 1816 Flush 0 New Done Done false
2161 818 Acked 1817 Read 5 New Done Done false
2162 819 Acked 1818 Read 3 New Done Done false
2163 820 Acked 1819 Read 4 New Done Done false
2164 821 Acked 1820 Write 6 New Done Done false
2165 822 Acked 1821 Flush 0 New Done Done false
2166 823 Acked 1822 Read 2 New Done Done false
2167 824 Acked 1823 Write 9 New Done Done false
2168 825 Acked 1824 Read 7 New Done Done false
2169 826 Acked 1825 Read 7 New Done Done false
2170 827 Acked 1826 Write 4 New Done Done false
2171 828 Acked 1827 Read 4 New Done Done false
2172 829 Acked 1828 Read 1 New Done Done false
2173 830 Acked 1829 Write 8 New Done Done false
2174 831 Acked 1830 Read 1 New Done Done false
2175 832 Acked 1831 Write 8 New Done Done false
2176 833 Acked 1832 Read 3 New Done Done false
2177 834 Acked 1833 Read 6 New Done Done false
2178 835 Acked 1834 Flush 0 New Done Done false
2179 836 Acked 1835 Read 8 New Done Done false
2180 837 Acked 1836 Read 10 New Done Done false
2181 838 Acked 1837 Read 3 New Done Done false
2182 839 Acked 1838 Flush 0 New Done Done false
2183 840 Acked 1839 Write 7 New Done Done false
2184 841 Acked 1840 Read 9 New Done Done false
2185 842 Acked 1841 Read 1 New Done Done false
2186 843 Acked 1842 Write 4 New Done Done false
2187 844 Acked 1843 Write 7 New Done Done false
2188 845 Acked 1844 Read 4 New Done Done false
2189 846 Acked 1845 Read 2 New Done Done false
2190 847 Acked 1846 Read 4 New Done Done false
2191 848 Acked 1847 Flush 0 New Done Done false
2192 849 Acked 1848 Write 8 New Done Done false
2193 850 Acked 1849 Read 1 New Done Done false
2194 851 Acked 1850 Read 6 New Done Done false
2195 852 Acked 1851 Read 2 New Done Done false
2196 853 Acked 1852 Read 3 New Done Done false
2197 854 Acked 1853 Read 2 New Done Done false
2198 855 Acked 1854 Read 1 New Done Done false
2199 856 Acked 1855 Write 2 New Done Done false
2200 857 Acked 1856 Flush 0 New Done Done false
2201 858 Acked 1857 Write 5 New Done Done false
2202 859 Acked 1858 Read 5 New Done Done false
2203 860 Acked 1859 Read 3 New Done Done false
2204 861 Acked 1860 Read 7 New Done Done false
2205 862 Acked 1861 Read 3 New Done Done false
2206 863 Acked 1862 Read 8 New Done Done false
2207 864 Acked 1863 Read 8 New Done Done false
2208 865 Acked 1864 Write 10 New Done Done false
2209 866 Acked 1865 Read 6 New Done Done false
2210 867 Acked 1866 Flush 0 New Done Done false
2211 868 Acked 1867 Read 1 New Done Done false
2212 869 Acked 1868 Write 3 New Done Done false
2213 870 Acked 1869 Read 6 New Done Done false
2214 871 Acked 1870 Write 3 New Done Done false
2215 872 Acked 1871 Read 1 New Done Done false
2216 873 Acked 1872 Read 7 New Done Done false
2217 874 Acked 1873 Write 10 New Done Done false
2218 875 Acked 1874 Read 1 New Done Done false
2219 876 Acked 1875 Read 9 New Done Done false
2220 877 Acked 1876 Write 5 New Done Done false
2221 878 Acked 1877 Write 9 New Done Done false
2222 879 Acked 1878 Read 8 New Done Done false
2223 880 Acked 1879 Read 7 New Done Done false
2224 881 Acked 1880 Write 10 New Done Done false
2225 882 Acked 1881 Read 3 New Done Done false
2226 883 Acked 1882 Write 1 New Done Done false
2227 884 Acked 1883 Write 5 New Done Done false
2228 885 Acked 1884 Read 5 New Done Done false
2229 886 Acked 1885 Flush 0 New Done Done false
2230 887 Acked 1886 Write 1 New Done Done false
2231 888 Acked 1887 Read 3 New Done Done false
2232 889 Acked 1888 Flush 0 New Done Done false
2233 890 Acked 1889 Flush 0 New Done Done false
2234 891 Acked 1890 Read 1 New Done Done false
2235 892 Acked 1891 Read 9 New Done Done false
2236 893 Acked 1892 Read 1 New Done Done false
2237 894 Acked 1893 Write 1 New Done Done false
2238 895 Acked 1894 Write 9 New Done Done false
2239 896 Acked 1895 Read 1 New Done Done false
2240 897 Acked 1896 Read 5 New Done Done false
2241 898 Acked 1897 Write 3 New Done Done false
2242 899 Acked 1898 Read 2 New Done Done false
2243 900 Acked 1899 Write 10 New Done Done false
2244 901 Acked 1900 Write 8 New Done Done false
2245 902 Acked 1901 Write 6 New Done Done false
2246 903 Acked 1902 Read 10 New Done Done false
2247 STATES DS:0 DS:1 DS:2 TOTAL
2248 New 300 0 0 300
2249 Sent 0 0 0 0
2250 Done 0 300 300 600
2251 Skipped 0 0 0 0
2252 Error 0 0 0 0
2253 Last Flush: 1602 1889 1889
2254 Downstairs last five completed: 1602 1601 1600 1599 1598
2255 Upstairs last five completed: 903 902 901 900 899
2256 CLIENT: Up:0 ds:300 act:2
22572024-03-02T01:06:01.201ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22582024-03-02T01:06:01.201ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22592024-03-02T01:06:01.202ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22602024-03-02T01:06:01.202ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22612024-03-02T01:06:01.202ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22622024-03-02T01:06:01.202ZINFOcrucible: [0] client re-new 301 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22632024-03-02T01:06:01.202ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2264 [8810] got start action
2265 [8810] I am going to restart
2266 Starting downstairs at port 8810
2267 Make output file at "/tmp/dsc/downstairs-8810.txt"
2268 [8810][0] reports Starting
22692024-03-02T01:06:02.203ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22702024-03-02T01:06:02.203ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22712024-03-02T01:06:02.203ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22722024-03-02T01:06:02.203ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22732024-03-02T01:06:02.203ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22742024-03-02T01:06:02.203ZINFOcrucible: [0] client re-new 301 jobs since flush 1602 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22752024-03-02T01:06:02.203ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2276 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1165)
2277 [8810][0] reports Running
22782024-03-02T01:06:03.204ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22792024-03-02T01:06:03.204ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22802024-03-02T01:06:03.206ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22812024-03-02T01:06:03.206ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22822024-03-02T01:06:03.206ZINFOcrucible: send last flush ID to this DS: 1602 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22832024-03-02T01:06:03.206ZINFOcrucible: Replied this last flush ID: 1602 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22842024-03-02T01:06:03.206ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22852024-03-02T01:06:03.206ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22862024-03-02T01:06:03.206ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22872024-03-02T01:06:03.206ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2288 ----------------------------------------------------------------
2289 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2290 Downstairs last five completed: 1903 1902 1901 1900 1899
2291 Upstairs last five completed: 904 903 902 901 900
2292 CLIENT: Up:0 ds:0 act:3
2293 Replay: All jobs finished, all DS active.
2294 Main task has work to do, go find it
2295 got dsc Stop(1)
2296 stop 1
2297 Mar 02 01:06:04.780 INFO request completed, latency_us: 377, [8820] Got stop action so:false kr:false
2298 response_code: 204, uri: /stop/cid/1, method: POST, req_id: 01a8d87e-2862-40d5-9986-1fbb0dda3b07, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
22992024-03-02T01:06:04.781ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23002024-03-02T01:06:04.781ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23012024-03-02T01:06:04.781ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23022024-03-02T01:06:04.781ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23032024-03-02T01:06:04.781ZINFOcrucible: [1] client re-new 0 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23042024-03-02T01:06:04.781ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2305 Mar 02 01:06:04.781 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0300aeae-b3b6-4b48-ac46-67831759d220, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
2306 [8820] Exited with: ExitStatus(unix_wait_status(9))
2307 [8820][1] reports Exit
23082024-03-02T01:06:05.782ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23092024-03-02T01:06:05.782ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23102024-03-02T01:06:05.782ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23112024-03-02T01:06:05.782ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23122024-03-02T01:06:05.783ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23132024-03-02T01:06:05.783ZINFOcrucible: [1] client re-new 0 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23142024-03-02T01:06:05.783ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23152024-03-02T01:06:06.783ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23162024-03-02T01:06:06.783ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23172024-03-02T01:06:06.783ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23182024-03-02T01:06:06.783ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23192024-03-02T01:06:06.783ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23202024-03-02T01:06:06.784ZINFOcrucible: [1] client re-new 0 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23212024-03-02T01:06:06.784ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23222024-03-02T01:06:07.785ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23232024-03-02T01:06:07.785ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23242024-03-02T01:06:07.785ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23252024-03-02T01:06:07.785ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23262024-03-02T01:06:07.785ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23272024-03-02T01:06:07.785ZINFOcrucible: [1] client re-new 0 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23282024-03-02T01:06:07.785ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2329 Mar 02 01:06:08.783 INFO request completed, latency_us: 201, response_code: 200, uri: /state/cid/1, method: GET, req_id: dcc06c03-0a8f-455c-b6fb-1111a2aebfff, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
2330 001/300 Write block 3482 len 12288 data: 2 2 2
2331 002/300 Flush
23322024-03-02T01:06:08.785ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23332024-03-02T01:06:08.785ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23342024-03-02T01:06:08.785ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23352024-03-02T01:06:08.785ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23362024-03-02T01:06:08.785ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23372024-03-02T01:06:08.785ZINFOcrucible: [1] client re-new 2 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23382024-03-02T01:06:08.785ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2339 003/300 Read block 4683 len 4096
2340 004/300 Write block 4836 len 4096 data: 3
2341 005/300 Write block 2914 len 20480 data: 2 2 2 2 2
2342 006/300 Read block 3424 len 40960
2343 007/300 Read block 2236 len 40960
2344 008/300 Write block 2862 len 28672 data: 2 3 3 3 3 3 3
2345 009/300 Write block 2276 len 8192 data: 3 3
2346 010/300 Write block 1844 len 28672 data: 3 3 3 3 2 2 2
2347 011/300 Read block 45 len 12288
2348 012/300 Read block 981 len 4096
2349 013/300 Write block 1837 len 28672 data: 2 2 3 3 3 3 3
2350 014/300 Read block 308 len 24576
2351 015/300 Write block 2444 len 12288 data: 2 2 2
2352 016/300 Write block 1577 len 4096 data: 2
2353 017/300 Write block 123 len 28672 data: 2 2 2 2 2 2 2
2354 018/300 Read block 2829 len 20480
2355 019/300 Read block 379 len 20480
2356 020/300 Read block 2333 len 32768
2357 021/300 Write block 3950 len 4096 data: 3
2358 022/300 Flush
2359 023/300 Write block 1745 len 16384 data: 2 2 2 2
2360 024/300 Flush
2361 025/300 Read block 4206 len 40960
2362 026/300 Read block 4698 len 20480
2363 027/300 Read block 3188 len 36864
2364 028/300 Write block 2580 len 32768 data: 2 2 2 2 2 2 2 2
2365 029/300 Read block 4779 len 28672
2366 030/300 Flush
2367 031/300 Read block 2697 len 40960
2368 032/300 Write block 3657 len 12288 data: 3 3 3
2369 033/300 Read block 2741 len 16384
2370 034/300 Write block 447 len 32768 data: 3 3 3 2 2 3 3 3
2371 035/300 Read block 2186 len 12288
2372 036/300 Write block 27 len 24576 data: 2 2 2 3 3 3
2373 037/300 Flush
2374 038/300 Read block 4788 len 40960
2375 039/300 Read block 3761 len 4096
2376 040/300 Read block 1810 len 32768
2377 041/300 Write block 1943 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2378 042/300 Write block 4571 len 20480 data: 2 2 2 2 2
2379 043/300 Read block 2690 len 40960
2380 044/300 Read block 4285 len 40960
2381 045/300 Write block 3597 len 16384 data: 3 3 3 2
2382 046/300 Read block 698 len 24576
2383 047/300 Flush
2384 048/300 Read block 1190 len 40960
2385 049/300 Write block 2525 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2386 050/300 Flush
2387 051/300 Flush
2388 052/300 Write block 935 len 8192 data: 2 2
2389 053/300 Write block 3818 len 20480 data: 3 4 4 5 4
2390 054/300 Write block 4081 len 12288 data: 2 2 2
2391 055/300 Write block 2575 len 24576 data: 2 2 2 2 2 3
2392 056/300 Read block 1771 len 24576
2393 057/300 Write block 1141 len 16384 data: 2 2 2 2
2394 058/300 Write block 922 len 24576 data: 2 3 3 3 4 3
2395 059/300 Write block 4562 len 40960 data: 2 2 2 2 2 2 2 2 2 3
2396 060/300 Flush
2397 061/300 Read block 2196 len 36864
2398 062/300 Write block 3608 len 32768 data: 2 2 2 3 3 3 3 3
2399 063/300 Write block 2459 len 12288 data: 2 2 2
2400 064/300 Write block 1781 len 32768 data: 2 2 2 2 2 2 2 2
2401 065/300 Read block 3947 len 24576
2402 066/300 Read block 1185 len 16384
2403 067/300 Write block 4279 len 12288 data: 3 3 3
2404 068/300 Write block 4133 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2405 069/300 Read block 1314 len 28672
2406 070/300 Read block 3534 len 8192
2407 071/300 Write block 814 len 24576 data: 2 2 2 3 3 3
2408 072/300 Write block 635 len 28672 data: 2 2 3 3 3 4 4
2409 073/300 Read block 4690 len 40960
2410 074/300 Read block 1901 len 28672
2411 075/300 Read block 4423 len 36864
2412 076/300 Read block 2882 len 4096
2413 077/300 Write block 2212 len 4096 data: 4
2414 078/300 Write block 4326 len 16384 data: 2 2 2 2
2415 079/300 Write block 3004 len 32768 data: 3 2 2 2 2 2 2 2
2416 080/300 Read block 1138 len 24576
2417 081/300 Read block 4629 len 24576
2418 082/300 Write block 3452 len 4096 data: 2
2419 083/300 Read block 1831 len 16384
2420 084/300 Read block 86 len 28672
2421 085/300 Write block 4344 len 16384 data: 3 5 5 4
2422 086/300 Read block 1437 len 40960
2423 087/300 Flush
2424 088/300 Write block 3403 len 24576 data: 2 2 2 2 2 2
2425 089/300 Read block 1933 len 8192
2426 090/300 Read block 3163 len 28672
2427 091/300 Write block 2765 len 24576 data: 3 3 3 2 2 2
2428 092/300 Read block 2047 len 4096
2429 093/300 Read block 3189 len 4096
2430 094/300 Write block 2218 len 4096 data: 2
2431 095/300 Write block 4020 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2432 096/300 Read block 3998 len 36864
2433 097/300 Read block 3308 len 4096
2434 098/300 Read block 382 len 24576
2435 099/300 Write block 3760 len 4096 data: 2
2436 100/300 Write block 1829 len 36864 data: 2 2 2 2 3 3 3 2 3
2437 101/300 Read block 206 len 12288
2438 102/300 Read block 4215 len 4096
2439 103/300 Write block 3342 len 20480 data: 2 2 2 2 2
2440 104/300 Read block 4940 len 4096
2441 105/300 Read block 3346 len 4096
2442 106/300 Write block 192 len 16384 data: 2 2 2 2
2443 107/300 Flush
2444 108/300 Flush
2445 109/300 Write block 69 len 12288 data: 4 4 3
2446 110/300 Read block 4165 len 4096
2447 111/300 Read block 1767 len 28672
2448 112/300 Read block 4808 len 28672
2449 113/300 Write block 2324 len 4096 data: 2
2450 114/300 Flush
2451 115/300 Write block 3802 len 36864 data: 3 3 3 3 3 3 4 4 4
2452 116/300 Read block 882 len 8192
2453 117/300 Read block 2863 len 8192
2454 118/300 Flush
2455 119/300 Read block 4846 len 8192
2456 120/300 Write block 1074 len 24576 data: 2 2 2 2 2 2
2457 121/300 Read block 1918 len 28672
2458 122/300 Read block 815 len 20480
2459 123/300 Write block 192 len 24576 data: 3 3 3 3 2 2
2460 124/300 Write block 3094 len 20480 data: 2 3 3 3 3
2461 125/300 Flush
2462 126/300 Read block 3777 len 8192
2463 127/300 Read block 1283 len 24576
2464 128/300 Write block 2430 len 40960 data: 3 3 2 2 2 2 2 2 2 3
2465 129/300 Flush
2466 130/300 Write block 4752 len 16384 data: 2 2 2 2
2467 131/300 Read block 2101 len 24576
2468 132/300 Read block 3916 len 32768
2469 133/300 Write block 2898 len 36864 data: 2 2 2 2 2 2 2 2 2
2470 134/300 Write block 3832 len 36864 data: 2 2 2 2 2 3 3 3 3
2471 135/300 Write block 4176 len 16384 data: 3 3 4 4
2472 136/300 Read block 2694 len 16384
2473 137/300 Write block 1569 len 4096 data: 2
2474 138/300 Read block 2724 len 16384
2475 139/300 Read block 474 len 28672
2476 140/300 Write block 467 len 32768 data: 2 2 2 2 2 2 2 2
2477 141/300 Flush
2478 142/300 Read block 1605 len 4096
2479 143/300 Write block 4839 len 32768 data: 3 3 3 3 3 3 3 3
2480 144/300 Flush
2481 145/300 Write block 4557 len 40960 data: 2 2 2 2 2 3 3 3 3 3
2482 146/300 Write block 2360 len 40960 data: 3 3 3 3 4 4 3 3 3 3
2483 147/300 Write block 4653 len 12288 data: 3 2 2
2484 148/300 Write block 4688 len 28672 data: 3 3 3 2 2 2 2
2485 149/300 Flush
2486 150/300 Write block 4816 len 20480 data: 2 2 2 2 2
2487 151/300 Read block 4195 len 12288
2488 152/300 Write block 2173 len 12288 data: 3 3 3
2489 153/300 Write block 4862 len 8192 data: 2 2
2490 154/300 Flush
2491 155/300 Read block 3584 len 16384
2492 156/300 Read block 2610 len 36864
2493 157/300 Write block 2592 len 20480 data: 2 2 2 2 2
2494 158/300 Flush
2495 159/300 Read block 4889 len 32768
2496 160/300 Write block 4575 len 24576 data: 3 2 2 2 2 2
2497 161/300 Read block 282 len 32768
2498 162/300 Write block 2810 len 36864 data: 3 3 3 3 3 2 2 3 3
2499 163/300 Write block 2709 len 4096 data: 3
2500 164/300 Write block 2065 len 24576 data: 3 3 3 3 2 2
2501 165/300 Read block 56 len 24576
2502 166/300 Read block 2352 len 8192
2503 167/300 Read block 636 len 12288
2504 168/300 Read block 308 len 8192
2505 169/300 Write block 3398 len 32768 data: 2 2 2 2 2 3 3 3
2506 170/300 Flush
2507 171/300 Write block 4555 len 12288 data: 2 2 3
2508 172/300 Read block 1218 len 20480
2509 173/300 Read block 1136 len 40960
2510 174/300 Write block 4195 len 20480 data: 3 4 5 5 3
2511 175/300 Write block 1734 len 32768 data: 2 2 3 3 3 3 3 3
2512 176/300 Write block 2530 len 28672 data: 3 3 3 3 3 2 3
2513 177/300 Read block 900 len 16384
2514 178/300 Write block 43 len 12288 data: 4 4 4
2515 179/300 Write block 673 len 28672 data: 2 2 2 3 3 3 2
2516 180/300 Write block 1681 len 32768 data: 2 2 2 2 2 2 2 3
2517 181/300 Write block 1007 len 36864 data: 2 2 2 2 2 2 2 2 3
2518 182/300 Write block 4503 len 32768 data: 3 3 2 2 2 2 2 2
2519 183/300 Read block 3368 len 12288
2520 184/300 Read block 1027 len 36864
2521 185/300 Flush
2522 186/300 Write block 4072 len 36864 data: 2 2 2 2 2 2 2 2 2
2523 187/300 Read block 4722 len 12288
2524 188/300 Flush
2525 189/300 Write block 3125 len 8192 data: 2 2
2526 190/300 Read block 1540 len 40960
2527 191/300 Write block 3005 len 24576 data: 3 3 3 3 3 3
2528 192/300 Write block 4175 len 8192 data: 2 4
2529 193/300 Read block 4582 len 24576
2530 194/300 Read block 1899 len 24576
2531 195/300 Write block 2613 len 4096 data: 2
2532 196/300 Read block 2887 len 32768
2533 197/300 Read block 259 len 8192
2534 198/300 Write block 2767 len 20480 data: 4 3 3 3 2
2535 199/300 Flush
2536 200/300 Read block 4550 len 8192
2537 201/300 Write block 2991 len 16384 data: 3 3 3 4
2538 202/300 Read block 430 len 16384
2539 203/300 Write block 2778 len 16384 data: 3 3 3 3
2540 204/300 Read block 3958 len 8192
2541 205/300 Read block 3762 len 24576
2542 206/300 Read block 3465 len 16384
2543 207/300 Write block 4259 len 24576 data: 2 2 2 2 2 2
2544 208/300 Write block 2862 len 12288 data: 3 4 4
2545 209/300 Write block 2194 len 20480 data: 2 2 2 2 2
2546 210/300 Read block 1549 len 32768
2547 211/300 Read block 2937 len 28672
2548 212/300 Read block 3346 len 12288
2549 213/300 Read block 3290 len 24576
2550 214/300 Read block 4193 len 24576
2551 215/300 Read block 1328 len 8192
2552 216/300 Read block 2302 len 40960
2553 217/300 Write block 3755 len 12288 data: 3 2 2
2554 218/300 Read block 2175 len 20480
2555 219/300 Flush
2556 220/300 Write block 2048 len 8192 data: 3 3
2557 221/300 Write block 2247 len 24576 data: 3 3 5 5 5 5
2558 222/300 Read block 3383 len 40960
2559 223/300 Read block 4158 len 32768
2560 224/300 Read block 190 len 28672
2561 225/300 Write block 3848 len 40960 data: 2 2 2 2 2 2 2 2 2 3
2562 226/300 Write block 82 len 40960 data: 3 3 3 3 3 3 3 4 3 2
2563 227/300 Read block 131 len 32768
2564 228/300 Write block 2116 len 20480 data: 2 2 2 2 2
2565 229/300 Read block 4009 len 12288
2566 230/300 Write block 1789 len 40960 data: 2 2 2 2 2 2 2 2 3 2
2567 231/300 Read block 1382 len 36864
2568 232/300 Read block 4519 len 28672
2569 233/300 Write block 0 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2570 234/300 Write block 3900 len 16384 data: 2 2 2 3
2571 235/300 Write block 1514 len 24576 data: 2 2 2 2 2 2
2572 236/300 Read block 927 len 28672
2573 237/300 Write block 123 len 36864 data: 3 3 3 3 3 3 3 2 2
2574 238/300 Read block 3992 len 28672
2575 239/300 Read block 856 len 12288
2576 240/300 Write block 3977 len 24576 data: 2 2 3 3 3 3
2577 241/300 Write block 4348 len 32768 data: 3 3 3 3 3 3 3 3
2578 242/300 Write block 2716 len 20480 data: 2 2 2 2 2
2579 243/300 Read block 1949 len 32768
2580 244/300 Read block 195 len 20480
2581 245/300 Read block 1624 len 20480
2582 246/300 Read block 4062 len 36864
2583 247/300 Flush
2584 248/300 Read block 1863 len 40960
2585 249/300 Read block 3322 len 40960
2586 250/300 Read block 2664 len 32768
2587 251/300 Read block 4184 len 4096
2588 252/300 Read block 2407 len 40960
2589 253/300 Write block 333 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2590 254/300 Write block 474 len 32768 data: 3 2 2 2 2 3 3 2
2591 255/300 Write block 619 len 32768 data: 2 3 3 3 3 3 3 3
2592 256/300 Write block 1868 len 24576 data: 2 2 2 2 2 2
2593 257/300 Read block 2859 len 24576
2594 258/300 Write block 1755 len 4096 data: 2
2595 259/300 Read block 1336 len 40960
2596 260/300 Write block 3446 len 40960 data: 2 3 3 3 3 3 3 2 2 2
2597 261/300 Read block 3117 len 24576
2598 262/300 Read block 1434 len 28672
2599 263/300 Write block 1526 len 32768 data: 3 3 4 3 2 2 2 2
2600 264/300 Read block 408 len 12288
2601 265/300 Read block 4432 len 24576
2602 266/300 Write block 3351 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2603 267/300 Write block 19 len 20480 data: 2 2 2 2 2
2604 268/300 Write block 285 len 28672 data: 2 2 2 2 2 2 2
2605 269/300 Write block 2143 len 20480 data: 2 2 2 2 2
2606 270/300 Read block 2110 len 20480
2607 271/300 Write block 1928 len 16384 data: 5 4 4 4
2608 272/300 Read block 1265 len 20480
2609 273/300 Write block 4644 len 36864 data: 4 3 4 4 3 3 3 3 3
2610 274/300 Read block 1151 len 40960
2611 275/300 Flush
2612 276/300 Read block 1064 len 20480
2613 277/300 Flush
2614 278/300 Write block 4539 len 20480 data: 2 3 3 3 3
2615 279/300 Read block 2759 len 4096
2616 280/300 Read block 3675 len 28672
2617 281/300 Read block 1895 len 36864
2618 282/300 Read block 3298 len 28672
2619 283/300 Read block 3715 len 8192
2620 284/300 Read block 4189 len 8192
2621 285/300 Write block 4122 len 12288 data: 3 3 2
2622 286/300 Write block 4396 len 4096 data: 2
2623 287/300 Read block 4398 len 16384
2624 288/300 Write block 4837 len 24576 data: 3 3 4 4 4 4
2625 289/300 Read block 1141 len 4096
2626 290/300 Read block 1618 len 20480
2627 291/300 Read block 2226 len 20480
2628 292/300 Read block 2305 len 32768
2629 293/300 Read block 747 len 28672
2630 294/300 Read block 3186 len 40960
2631 295/300 Write block 3785 len 16384 data: 3 3 2 2
2632 296/300 Read block 369 len 12288
2633 297/300 Write block 4878 len 4096 data: 2
2634 298/300 Read block 2865 len 20480
2635 299/300 Write block 2191 len 28672 data: 2 2 2 3 3 3 3
2636 300/300 Read block 3508 len 16384
2637 Main task has work to do, go find it
2638 got dsc Start(1)
2639 start 1
2640 Mar 02 01:06:09.568 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/1, method: POST, req_id: 33769391-2f7c-4809-90d0-8eeab3521b52, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
2641 Replay: started 1, returned:Ok(())
2642 ----------------------------------------------------------------
2643 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2644 States: Active Offline Active
2645 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2646 905 Acked 1904 Write 3 Done New Done true
2647 906 Acked 1905 Flush 0 Done New Done true
2648 907 Acked 1906 Read 1 Done New Done false
2649 908 Acked 1907 Write 1 Done New Done false
2650 909 Acked 1908 Write 5 Done New Done false
2651 910 Acked 1909 Read 10 Done New Done false
2652 911 Acked 1910 Read 10 Done New Done false
2653 912 Acked 1911 Write 7 Done New Done false
2654 913 Acked 1912 Write 2 Done New Done false
2655 914 Acked 1913 Write 7 Done New Done false
2656 915 Acked 1914 Read 3 Done New Done false
2657 916 Acked 1915 Read 1 Done New Done false
2658 917 Acked 1916 Write 7 Done New Done false
2659 918 Acked 1917 Read 6 Done New Done false
2660 919 Acked 1918 Write 3 Done New Done false
2661 920 Acked 1919 Write 1 Done New Done false
2662 921 Acked 1920 Write 7 Done New Done false
2663 922 Acked 1921 Read 5 Done New Done false
2664 923 Acked 1922 Read 5 Done New Done false
2665 924 Acked 1923 Read 8 Done New Done false
2666 925 Acked 1924 Write 1 Done New Done false
2667 926 Acked 1925 Flush 0 Done New Done false
2668 927 Acked 1926 Write 4 Done New Done false
2669 928 Acked 1927 Flush 0 Done New Done false
2670 929 Acked 1928 Read 10 Done New Done false
2671 930 Acked 1929 Read 5 Done New Done false
2672 931 Acked 1930 Read 9 Done New Done false
2673 932 Acked 1931 Write 8 Done New Done false
2674 933 Acked 1932 Read 7 Done New Done false
2675 934 Acked 1933 Flush 0 Done New Done false
2676 935 Acked 1934 Read 10 Done New Done false
2677 936 Acked 1935 Write 3 Done New Done false
2678 937 Acked 1936 Read 4 Done New Done false
2679 938 Acked 1937 Write 8 Done New Done false
2680 939 Acked 1938 Read 3 Done New Done false
2681 940 Acked 1939 Write 6 Done New Done false
2682 941 Acked 1940 Flush 0 Done New Done false
2683 942 Acked 1941 Read 10 Done New Done false
2684 943 Acked 1942 Read 1 Done New Done false
2685 944 Acked 1943 Read 8 Done New Done false
2686 945 Acked 1944 Write 10 Done New Done false
2687 946 Acked 1945 Write 5 Done New Done false
2688 947 Acked 1946 Read 10 Done New Done false
2689 948 Acked 1947 Read 10 Done New Done false
2690 949 Acked 1948 Write 4 Done New Done false
2691 950 Acked 1949 Read 6 Done New Done false
2692 951 Acked 1950 Flush 0 Done New Done false
2693 952 Acked 1951 Read 10 Done New Done false
2694 953 Acked 1952 Write 10 Done New Done false
2695 954 Acked 1953 Flush 0 Done New Done false
2696 955 Acked 1954 Flush 0 Done New Done false
2697 956 Acked 1955 Write 2 Done New Done false
2698 957 Acked 1956 Write 5 Done New Done false
2699 958 Acked 1957 Write 3 Done New Done false
2700 959 Acked 1958 Write 6 Done New Done false
2701 960 Acked 1959 Read 6 Done New Done false
2702 961 Acked 1960 Write 4 Done New Done false
2703 962 Acked 1961 Write 6 Done New Done false
2704 963 Acked 1962 Write 10 Done New Done false
2705 964 Acked 1963 Flush 0 Done New Done false
2706 965 Acked 1964 Read 9 Done New Done false
2707 966 Acked 1965 Write 8 Done New Done false
2708 967 Acked 1966 Write 3 Done New Done false
2709 968 Acked 1967 Write 8 Done New Done false
2710 969 Acked 1968 Read 6 Done New Done false
2711 970 Acked 1969 Read 4 Done New Done false
2712 971 Acked 1970 Write 3 Done New Done false
2713 972 Acked 1971 Write 10 Done New Done false
2714 973 Acked 1972 Read 7 Done New Done false
2715 974 Acked 1973 Read 2 Done New Done false
2716 975 Acked 1974 Write 6 Done New Done false
2717 976 Acked 1975 Write 7 Done New Done false
2718 977 Acked 1976 Read 10 Done New Done false
2719 978 Acked 1977 Read 7 Done New Done false
2720 979 Acked 1978 Read 9 Done New Done false
2721 980 Acked 1979 Read 1 Done New Done false
2722 981 Acked 1980 Write 1 Done New Done false
2723 982 Acked 1981 Write 4 Done New Done false
2724 983 Acked 1982 Write 8 Done New Done false
2725 984 Acked 1983 Read 6 Done New Done false
2726 985 Acked 1984 Read 6 Done New Done false
2727 986 Acked 1985 Write 1 Done New Done false
2728 987 Acked 1986 Read 4 Done New Done false
2729 988 Acked 1987 Read 7 Done New Done false
2730 989 Acked 1988 Write 4 Done New Done false
2731 990 Acked 1989 Read 10 Done New Done false
2732 991 Acked 1990 Flush 0 Done New Done false
2733 992 Acked 1991 Write 6 Done New Done false
2734 993 Acked 1992 Read 2 Done New Done false
2735 994 Acked 1993 Read 7 Done New Done false
2736 995 Acked 1994 Write 6 Done New Done false
2737 996 Acked 1995 Read 1 Done New Done false
2738 997 Acked 1996 Read 1 Done New Done false
2739 998 Acked 1997 Write 1 Done New Done false
2740 999 Acked 1998 Write 10 Done New Done false
2741 1000 Acked 1999 Read 9 Done New Done false
2742 1001 Acked 2000 Read 1 Done New Done false
2743 1002 Acked 2001 Read 6 Done New Done false
2744 1003 Acked 2002 Write 1 Done New Done false
2745 1004 Acked 2003 Write 9 Done New Done false
2746 1005 Acked 2004 Read 3 Done New Done false
2747 1006 Acked 2005 Read 1 Done New Done false
2748 1007 Acked 2006 Write 5 Done New Done false
2749 1008 Acked 2007 Read 1 Done New Done false
2750 1009 Acked 2008 Read 1 Done New Done false
2751 1010 Acked 2009 Write 4 Done New Done false
2752 1011 Acked 2010 Flush 0 Done New Done false
2753 1012 Acked 2011 Flush 0 Done New Done false
2754 1013 Acked 2012 Write 3 Done New Done false
2755 1014 Acked 2013 Read 1 Done New Done false
2756 1015 Acked 2014 Read 7 Done New Done false
2757 1016 Acked 2015 Read 7 Done New Done false
2758 1017 Acked 2016 Write 1 Done New Done false
2759 1018 Acked 2017 Flush 0 Done New Done false
2760 1019 Acked 2018 Write 9 Done New Done false
2761 1020 Acked 2019 Read 2 Done New Done false
2762 1021 Acked 2020 Read 2 Done New Done false
2763 1022 Acked 2021 Flush 0 Done New Done false
2764 1023 Acked 2022 Read 2 Done New Done false
2765 1024 Acked 2023 Write 6 Done New Done false
2766 1025 Acked 2024 Read 7 Done New Done false
2767 1026 Acked 2025 Read 5 Done New Done false
2768 1027 Acked 2026 Write 6 Done New Done false
2769 1028 Acked 2027 Write 5 Done New Done false
2770 1029 Acked 2028 Flush 0 Done New Done false
2771 1030 Acked 2029 Read 2 Done New Done false
2772 1031 Acked 2030 Read 6 Done New Done false
2773 1032 Acked 2031 Write 10 Done New Done false
2774 1033 Acked 2032 Flush 0 Done New Done false
2775 1034 Acked 2033 Write 4 Done New Done false
2776 1035 Acked 2034 Read 6 Done New Done false
2777 1036 Acked 2035 Read 8 Done New Done false
2778 1037 Acked 2036 Write 9 Done New Done false
2779 1038 Acked 2037 Write 9 Done New Done false
2780 1039 Acked 2038 Write 4 Done New Done false
2781 1040 Acked 2039 Read 4 Done New Done false
2782 1041 Acked 2040 Write 1 Done New Done false
2783 1042 Acked 2041 Read 4 Done New Done false
2784 1043 Acked 2042 Read 7 Done New Done false
2785 1044 Acked 2043 Flush 0 Done New Done false
2786 1045 Acked 2044 Write 8 Done New Done false
2787 1046 Acked 2045 Flush 0 Done New Done false
2788 1047 Acked 2046 Read 1 Done New Done false
2789 1048 Acked 2047 Write 8 Done New Done false
2790 1049 Acked 2048 Flush 0 Done New Done false
2791 1050 Acked 2049 Write 10 Done New Done false
2792 1051 Acked 2050 Write 10 Done New Done false
2793 1052 Acked 2051 Write 3 Done New Done false
2794 1053 Acked 2052 Write 7 Done New Done false
2795 1054 Acked 2053 Flush 0 Done New Done false
2796 1055 Acked 2054 Write 5 Done New Done false
2797 1056 Acked 2055 Read 3 Done New Done false
2798 1057 Acked 2056 Write 3 Done New Done false
2799 1058 Acked 2057 Write 2 Done New Done false
2800 1059 Acked 2058 Flush 0 Done New Done false
2801 1060 Acked 2059 Read 4 Done New Done false
2802 1061 Acked 2060 Read 9 Done New Done false
2803 1062 Acked 2061 Write 5 Done New Done false
2804 1063 Acked 2062 Flush 0 Done New Done false
2805 1064 Acked 2063 Read 8 Done New Done false
2806 1065 Acked 2064 Write 6 Done New Done false
2807 1066 Acked 2065 Read 8 Done New Done false
2808 1067 Acked 2066 Write 9 Done New Done false
2809 1068 Acked 2067 Write 1 Done New Done false
2810 1069 Acked 2068 Write 6 Done New Done false
2811 1070 Acked 2069 Read 6 Done New Done false
2812 1071 Acked 2070 Read 2 Done New Done false
2813 1072 Acked 2071 Read 3 Done New Done false
2814 1073 Acked 2072 Read 2 Done New Done false
2815 1074 Acked 2073 Write 8 Done New Done false
2816 1075 Acked 2074 Flush 0 Done New Done false
2817 1076 Acked 2075 Write 3 Done New Done false
2818 1077 Acked 2076 Read 5 Done New Done false
2819 1078 Acked 2077 Read 10 Done New Done false
2820 1079 Acked 2078 Write 5 Done New Done false
2821 1080 Acked 2079 Write 8 Done New Done false
2822 1081 Acked 2080 Write 7 Done New Done false
2823 1082 Acked 2081 Read 4 Done New Done false
2824 1083 Acked 2082 Write 3 Done New Done false
2825 1084 Acked 2083 Write 7 Done New Done false
2826 1085 Acked 2084 Write 8 Done New Done false
2827 1086 Acked 2085 Write 9 Done New Done false
2828 1087 Acked 2086 Write 8 Done New Done false
2829 1088 Acked 2087 Read 3 Done New Done false
2830 1089 Acked 2088 Read 9 Done New Done false
2831 1090 Acked 2089 Flush 0 Done New Done false
2832 1091 Acked 2090 Write 9 Done New Done false
2833 1092 Acked 2091 Read 3 Done New Done false
2834 1093 Acked 2092 Flush 0 Done New Done false
2835 1094 Acked 2093 Write 2 Done New Done false
2836 1095 Acked 2094 Read 10 Done New Done false
2837 1096 Acked 2095 Write 6 Done New Done false
2838 1097 Acked 2096 Write 2 Done New Done false
2839 1098 Acked 2097 Read 6 Done New Done false
2840 1099 Acked 2098 Read 6 Done New Done false
2841 1100 Acked 2099 Write 1 Done New Done false
2842 1101 Acked 2100 Read 8 Done New Done false
2843 1102 Acked 2101 Read 2 Done New Done false
2844 1103 Acked 2102 Write 5 Done New Done false
2845 1104 Acked 2103 Flush 0 Done New Done false
2846 1105 Acked 2104 Read 2 Done New Done false
2847 1106 Acked 2105 Write 4 Done New Done false
2848 1107 Acked 2106 Read 4 Done New Done false
2849 1108 Acked 2107 Write 4 Done New Done false
2850 1109 Acked 2108 Read 2 Done New Done false
2851 1110 Acked 2109 Read 6 Done New Done false
2852 1111 Acked 2110 Read 4 Done New Done false
2853 1112 Acked 2111 Write 6 Done New Done false
2854 1113 Acked 2112 Write 3 Done New Done false
2855 1114 Acked 2113 Write 5 Done New Done false
2856 1115 Acked 2114 Read 8 Done New Done false
2857 1116 Acked 2115 Read 7 Done New Done false
2858 1117 Acked 2116 Read 3 Done New Done false
2859 1118 Acked 2117 Read 6 Done New Done false
2860 1119 Acked 2118 Read 6 Done New Done false
2861 1120 Acked 2119 Read 2 Done New Done false
2862 1121 Acked 2120 Read 10 Done New Done false
2863 1122 Acked 2121 Write 3 Done New Done false
2864 1123 Acked 2122 Read 5 Done New Done false
2865 1124 Acked 2123 Flush 0 Done New Done false
2866 1125 Acked 2124 Write 2 Done New Done false
2867 1126 Acked 2125 Write 6 Done New Done false
2868 1127 Acked 2126 Read 10 Done New Done false
2869 1128 Acked 2127 Read 8 Done New Done false
2870 1129 Acked 2128 Read 7 Done New Done false
2871 1130 Acked 2129 Write 10 Done New Done false
2872 1131 Acked 2130 Write 10 Done New Done false
2873 1132 Acked 2131 Read 8 Done New Done false
2874 1133 Acked 2132 Write 5 Done New Done false
2875 1134 Acked 2133 Read 3 Done New Done false
2876 1135 Acked 2134 Write 10 Done New Done false
2877 1136 Acked 2135 Read 9 Done New Done false
2878 1137 Acked 2136 Read 7 Done New Done false
2879 1138 Acked 2137 Write 10 Done New Done false
2880 1139 Acked 2138 Write 4 Done New Done false
2881 1140 Acked 2139 Write 6 Done New Done false
2882 1141 Acked 2140 Read 7 Done New Done false
2883 1142 Acked 2141 Write 9 Done New Done false
2884 1143 Acked 2142 Read 7 Done New Done false
2885 1144 Acked 2143 Read 3 Done New Done false
2886 1145 Acked 2144 Write 6 Done New Done false
2887 1146 Acked 2145 Write 8 Done New Done false
2888 1147 Acked 2146 Write 5 Done New Done false
2889 1148 Acked 2147 Read 8 Done New Done false
2890 1149 Acked 2148 Read 5 Done New Done false
2891 1150 Acked 2149 Read 5 Done New Done false
2892 1151 Acked 2150 Read 9 Done New Done false
2893 1152 Acked 2151 Flush 0 Done New Done false
2894 1153 Acked 2152 Read 10 Done New Done false
2895 1154 Acked 2153 Read 10 Done New Done false
2896 1155 Acked 2154 Read 8 Done New Done false
2897 1156 Acked 2155 Read 1 Done New Done false
2898 1157 Acked 2156 Read 10 Done New Done false
2899 1158 Acked 2157 Write 10 Done New Done false
2900 1159 Acked 2158 Write 8 Done New Done false
2901 1160 Acked 2159 Write 8 Done New Done false
2902 1161 Acked 2160 Write 6 Done New Done false
2903 1162 Acked 2161 Read 6 Done New Done false
2904 1163 Acked 2162 Write 1 Done New Done false
2905 1164 Acked 2163 Read 10 Done New Done false
2906 1165 Acked 2164 Write 10 Done New Done false
2907 1166 Acked 2165 Read 6 Done New Done false
2908 1167 Acked 2166 Read 7 Done New Done false
2909 1168 Acked 2167 Write 8 Done New Done false
2910 1169 Acked 2168 Read 3 Done New Done false
2911 1170 Acked 2169 Read 6 Done New Done false
2912 1171 Acked 2170 Write 10 Done New Done false
2913 1172 Acked 2171 Write 5 Done New Done false
2914 1173 Acked 2172 Write 7 Done New Done false
2915 1174 Acked 2173 Write 5 Done New Done false
2916 1175 Acked 2174 Read 5 Done New Done false
2917 1176 Acked 2175 Write 4 Done New Done false
2918 1177 Acked 2176 Read 5 Done New Done false
2919 1178 Acked 2177 Write 9 Done New Done false
2920 1179 Acked 2178 Read 10 Done New Done false
2921 1180 Acked 2179 Flush 0 Done New Done false
2922 1181 Acked 2180 Read 5 Done New Done false
2923 1182 Acked 2181 Flush 0 Done New Done false
2924 1183 Acked 2182 Write 5 Done New Done false
2925 1184 Acked 2183 Read 1 Done New Done false
2926 1185 Acked 2184 Read 7 Done New Done false
2927 1186 Acked 2185 Read 9 Done New Done false
2928 1187 Acked 2186 Read 7 Done New Done false
2929 1188 Acked 2187 Read 2 Done New Done false
2930 1189 Acked 2188 Read 2 Done New Done false
2931 1190 Acked 2189 Write 3 Done New Done false
2932 1191 Acked 2190 Write 1 Done New Done false
2933 1192 Acked 2191 Read 4 Done New Done false
2934 1193 Acked 2192 Write 6 Done New Done false
2935 1194 Acked 2193 Read 1 Done New Done false
2936 1195 Acked 2194 Read 5 Done New Done false
2937 1196 Acked 2195 Read 5 Done New Done false
2938 1197 Acked 2196 Read 8 Done New Done false
2939 1198 Acked 2197 Read 7 Done New Done false
2940 1199 Acked 2198 Read 10 Done New Done false
2941 1200 Acked 2199 Write 4 Done New Done false
2942 1201 Acked 2200 Read 3 Done New Done false
2943 1202 Acked 2201 Write 1 Done New Done false
2944 1203 Acked 2202 Read 5 Done New Done false
2945 1204 Acked 2203 Write 7 Done New Done false
2946 1205 Acked 2204 Read 4 Done New Done false
2947 STATES DS:0 DS:1 DS:2 TOTAL
2948 New 0 301 0 301
2949 Sent 0 0 0 0
2950 Done 301 0 301 602
2951 Skipped 0 0 0 0
2952 Error 0 0 0 0
2953 Last Flush: 2181 1903 2181
2954 Downstairs last five completed: 1903 1902 1901 1900 1899
2955 Upstairs last five completed: 1205 1204 1203 1202 1201
2956 CLIENT: Up:0 ds:301 act:2
29572024-03-02T01:06:09.787ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29582024-03-02T01:06:09.787ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29592024-03-02T01:06:09.787ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29602024-03-02T01:06:09.787ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29612024-03-02T01:06:09.787ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29622024-03-02T01:06:09.787ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29632024-03-02T01:06:09.787ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29642024-03-02T01:06:10.787ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29652024-03-02T01:06:10.787ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29662024-03-02T01:06:10.788ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29672024-03-02T01:06:10.788ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29682024-03-02T01:06:10.788ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29692024-03-02T01:06:10.788ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29702024-03-02T01:06:10.788ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29712024-03-02T01:06:11.789ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29722024-03-02T01:06:11.789ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29732024-03-02T01:06:11.789ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29742024-03-02T01:06:11.789ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29752024-03-02T01:06:11.789ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29762024-03-02T01:06:11.789ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29772024-03-02T01:06:11.789ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29782024-03-02T01:06:12.790ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29792024-03-02T01:06:12.790ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29802024-03-02T01:06:12.790ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29812024-03-02T01:06:12.790ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29822024-03-02T01:06:12.790ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29832024-03-02T01:06:12.790ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
29842024-03-02T01:06:12.790ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
2985 ----------------------------------------------------------------
2986 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2987 States: Active Offline Active
2988 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2989 905 Acked 1904 Write 3 Done New Done true
2990 906 Acked 1905 Flush 0 Done New Done true
2991 907 Acked 1906 Read 1 Done New Done true
2992 908 Acked 1907 Write 1 Done New Done true
2993 909 Acked 1908 Write 5 Done New Done true
2994 910 Acked 1909 Read 10 Done New Done true
2995 911 Acked 1910 Read 10 Done New Done true
2996 912 Acked 1911 Write 7 Done New Done true
2997 913 Acked 1912 Write 2 Done New Done true
2998 914 Acked 1913 Write 7 Done New Done true
2999 915 Acked 1914 Read 3 Done New Done true
3000 916 Acked 1915 Read 1 Done New Done true
3001 917 Acked 1916 Write 7 Done New Done true
3002 918 Acked 1917 Read 6 Done New Done true
3003 919 Acked 1918 Write 3 Done New Done true
3004 920 Acked 1919 Write 1 Done New Done true
3005 921 Acked 1920 Write 7 Done New Done true
3006 922 Acked 1921 Read 5 Done New Done true
3007 923 Acked 1922 Read 5 Done New Done true
3008 924 Acked 1923 Read 8 Done New Done true
3009 925 Acked 1924 Write 1 Done New Done true
3010 926 Acked 1925 Flush 0 Done New Done true
3011 927 Acked 1926 Write 4 Done New Done true
3012 928 Acked 1927 Flush 0 Done New Done true
3013 929 Acked 1928 Read 10 Done New Done true
3014 930 Acked 1929 Read 5 Done New Done true
3015 931 Acked 1930 Read 9 Done New Done true
3016 932 Acked 1931 Write 8 Done New Done true
3017 933 Acked 1932 Read 7 Done New Done true
3018 934 Acked 1933 Flush 0 Done New Done true
3019 935 Acked 1934 Read 10 Done New Done true
3020 936 Acked 1935 Write 3 Done New Done true
3021 937 Acked 1936 Read 4 Done New Done true
3022 938 Acked 1937 Write 8 Done New Done true
3023 939 Acked 1938 Read 3 Done New Done true
3024 940 Acked 1939 Write 6 Done New Done true
3025 941 Acked 1940 Flush 0 Done New Done true
3026 942 Acked 1941 Read 10 Done New Done true
3027 943 Acked 1942 Read 1 Done New Done true
3028 944 Acked 1943 Read 8 Done New Done true
3029 945 Acked 1944 Write 10 Done New Done true
3030 946 Acked 1945 Write 5 Done New Done true
3031 947 Acked 1946 Read 10 Done New Done true
3032 948 Acked 1947 Read 10 Done New Done true
3033 949 Acked 1948 Write 4 Done New Done true
3034 950 Acked 1949 Read 6 Done New Done true
3035 951 Acked 1950 Flush 0 Done New Done true
3036 952 Acked 1951 Read 10 Done New Done true
3037 953 Acked 1952 Write 10 Done New Done true
3038 954 Acked 1953 Flush 0 Done New Done true
3039 955 Acked 1954 Flush 0 Done New Done true
3040 956 Acked 1955 Write 2 Done New Done true
3041 957 Acked 1956 Write 5 Done New Done true
3042 958 Acked 1957 Write 3 Done New Done true
3043 959 Acked 1958 Write 6 Done New Done true
3044 960 Acked 1959 Read 6 Done New Done true
3045 961 Acked 1960 Write 4 Done New Done true
3046 962 Acked 1961 Write 6 Done New Done true
3047 963 Acked 1962 Write 10 Done New Done true
3048 964 Acked 1963 Flush 0 Done New Done true
3049 965 Acked 1964 Read 9 Done New Done true
3050 966 Acked 1965 Write 8 Done New Done true
3051 967 Acked 1966 Write 3 Done New Done true
3052 968 Acked 1967 Write 8 Done New Done true
3053 969 Acked 1968 Read 6 Done New Done true
3054 970 Acked 1969 Read 4 Done New Done true
3055 971 Acked 1970 Write 3 Done New Done true
3056 972 Acked 1971 Write 10 Done New Done true
3057 973 Acked 1972 Read 7 Done New Done true
3058 974 Acked 1973 Read 2 Done New Done true
3059 975 Acked 1974 Write 6 Done New Done true
3060 976 Acked 1975 Write 7 Done New Done true
3061 977 Acked 1976 Read 10 Done New Done true
3062 978 Acked 1977 Read 7 Done New Done true
3063 979 Acked 1978 Read 9 Done New Done true
3064 980 Acked 1979 Read 1 Done New Done true
3065 981 Acked 1980 Write 1 Done New Done true
3066 982 Acked 1981 Write 4 Done New Done true
3067 983 Acked 1982 Write 8 Done New Done true
3068 984 Acked 1983 Read 6 Done New Done true
3069 985 Acked 1984 Read 6 Done New Done true
3070 986 Acked 1985 Write 1 Done New Done true
3071 987 Acked 1986 Read 4 Done New Done true
3072 988 Acked 1987 Read 7 Done New Done true
3073 989 Acked 1988 Write 4 Done New Done true
3074 990 Acked 1989 Read 10 Done New Done true
3075 991 Acked 1990 Flush 0 Done New Done true
3076 992 Acked 1991 Write 6 Done New Done true
3077 993 Acked 1992 Read 2 Done New Done true
3078 994 Acked 1993 Read 7 Done New Done true
3079 995 Acked 1994 Write 6 Done New Done true
3080 996 Acked 1995 Read 1 Done New Done true
3081 997 Acked 1996 Read 1 Done New Done true
3082 998 Acked 1997 Write 1 Done New Done true
3083 999 Acked 1998 Write 10 Done New Done true
3084 1000 Acked 1999 Read 9 Done New Done true
3085 1001 Acked 2000 Read 1 Done New Done true
3086 1002 Acked 2001 Read 6 Done New Done true
3087 1003 Acked 2002 Write 1 Done New Done true
3088 1004 Acked 2003 Write 9 Done New Done true
3089 1005 Acked 2004 Read 3 Done New Done true
3090 1006 Acked 2005 Read 1 Done New Done true
3091 1007 Acked 2006 Write 5 Done New Done true
3092 1008 Acked 2007 Read 1 Done New Done true
3093 1009 Acked 2008 Read 1 Done New Done true
3094 1010 Acked 2009 Write 4 Done New Done true
3095 1011 Acked 2010 Flush 0 Done New Done true
3096 1012 Acked 2011 Flush 0 Done New Done true
3097 1013 Acked 2012 Write 3 Done New Done true
3098 1014 Acked 2013 Read 1 Done New Done true
3099 1015 Acked 2014 Read 7 Done New Done true
3100 1016 Acked 2015 Read 7 Done New Done true
3101 1017 Acked 2016 Write 1 Done New Done true
3102 1018 Acked 2017 Flush 0 Done New Done true
3103 1019 Acked 2018 Write 9 Done New Done true
3104 1020 Acked 2019 Read 2 Done New Done true
3105 1021 Acked 2020 Read 2 Done New Done true
3106 1022 Acked 2021 Flush 0 Done New Done true
3107 1023 Acked 2022 Read 2 Done New Done true
3108 1024 Acked 2023 Write 6 Done New Done true
3109 1025 Acked 2024 Read 7 Done New Done true
3110 1026 Acked 2025 Read 5 Done New Done true
3111 1027 Acked 2026 Write 6 Done New Done true
3112 1028 Acked 2027 Write 5 Done New Done true
3113 1029 Acked 2028 Flush 0 Done New Done true
3114 1030 Acked 2029 Read 2 Done New Done true
3115 1031 Acked 2030 Read 6 Done New Done true
3116 1032 Acked 2031 Write 10 Done New Done true
3117 1033 Acked 2032 Flush 0 Done New Done true
3118 1034 Acked 2033 Write 4 Done New Done true
3119 1035 Acked 2034 Read 6 Done New Done true
3120 1036 Acked 2035 Read 8 Done New Done true
3121 1037 Acked 2036 Write 9 Done New Done true
3122 1038 Acked 2037 Write 9 Done New Done true
3123 1039 Acked 2038 Write 4 Done New Done true
3124 1040 Acked 2039 Read 4 Done New Done true
3125 1041 Acked 2040 Write 1 Done New Done true
3126 1042 Acked 2041 Read 4 Done New Done true
3127 1043 Acked 2042 Read 7 Done New Done true
3128 1044 Acked 2043 Flush 0 Done New Done true
3129 1045 Acked 2044 Write 8 Done New Done true
3130 1046 Acked 2045 Flush 0 Done New Done true
3131 1047 Acked 2046 Read 1 Done New Done true
3132 1048 Acked 2047 Write 8 Done New Done true
3133 1049 Acked 2048 Flush 0 Done New Done true
3134 1050 Acked 2049 Write 10 Done New Done true
3135 1051 Acked 2050 Write 10 Done New Done true
3136 1052 Acked 2051 Write 3 Done New Done true
3137 1053 Acked 2052 Write 7 Done New Done true
3138 1054 Acked 2053 Flush 0 Done New Done true
3139 1055 Acked 2054 Write 5 Done New Done true
3140 1056 Acked 2055 Read 3 Done New Done true
3141 1057 Acked 2056 Write 3 Done New Done true
3142 1058 Acked 2057 Write 2 Done New Done true
3143 1059 Acked 2058 Flush 0 Done New Done true
3144 1060 Acked 2059 Read 4 Done New Done true
3145 1061 Acked 2060 Read 9 Done New Done true
3146 1062 Acked 2061 Write 5 Done New Done true
3147 1063 Acked 2062 Flush 0 Done New Done true
3148 1064 Acked 2063 Read 8 Done New Done true
3149 1065 Acked 2064 Write 6 Done New Done true
3150 1066 Acked 2065 Read 8 Done New Done true
3151 1067 Acked 2066 Write 9 Done New Done true
3152 1068 Acked 2067 Write 1 Done New Done true
3153 1069 Acked 2068 Write 6 Done New Done true
3154 1070 Acked 2069 Read 6 Done New Done true
3155 1071 Acked 2070 Read 2 Done New Done true
3156 1072 Acked 2071 Read 3 Done New Done true
3157 1073 Acked 2072 Read 2 Done New Done true
3158 1074 Acked 2073 Write 8 Done New Done true
3159 1075 Acked 2074 Flush 0 Done New Done true
3160 1076 Acked 2075 Write 3 Done New Done true
3161 1077 Acked 2076 Read 5 Done New Done true
3162 1078 Acked 2077 Read 10 Done New Done true
3163 1079 Acked 2078 Write 5 Done New Done true
3164 1080 Acked 2079 Write 8 Done New Done true
3165 1081 Acked 2080 Write 7 Done New Done true
3166 1082 Acked 2081 Read 4 Done New Done true
3167 1083 Acked 2082 Write 3 Done New Done true
3168 1084 Acked 2083 Write 7 Done New Done true
3169 1085 Acked 2084 Write 8 Done New Done true
3170 1086 Acked 2085 Write 9 Done New Done true
3171 1087 Acked 2086 Write 8 Done New Done true
3172 1088 Acked 2087 Read 3 Done New Done true
3173 1089 Acked 2088 Read 9 Done New Done true
3174 1090 Acked 2089 Flush 0 Done New Done true
3175 1091 Acked 2090 Write 9 Done New Done true
3176 1092 Acked 2091 Read 3 Done New Done true
3177 1093 Acked 2092 Flush 0 Done New Done true
3178 1094 Acked 2093 Write 2 Done New Done true
3179 1095 Acked 2094 Read 10 Done New Done true
3180 1096 Acked 2095 Write 6 Done New Done true
3181 1097 Acked 2096 Write 2 Done New Done true
3182 1098 Acked 2097 Read 6 Done New Done true
3183 1099 Acked 2098 Read 6 Done New Done true
3184 1100 Acked 2099 Write 1 Done New Done true
3185 1101 Acked 2100 Read 8 Done New Done true
3186 1102 Acked 2101 Read 2 Done New Done true
3187 1103 Acked 2102 Write 5 Done New Done true
3188 1104 Acked 2103 Flush 0 Done New Done true
3189 1105 Acked 2104 Read 2 Done New Done true
3190 1106 Acked 2105 Write 4 Done New Done true
3191 1107 Acked 2106 Read 4 Done New Done true
3192 1108 Acked 2107 Write 4 Done New Done true
3193 1109 Acked 2108 Read 2 Done New Done true
3194 1110 Acked 2109 Read 6 Done New Done true
3195 1111 Acked 2110 Read 4 Done New Done true
3196 1112 Acked 2111 Write 6 Done New Done true
3197 1113 Acked 2112 Write 3 Done New Done true
3198 1114 Acked 2113 Write 5 Done New Done true
3199 1115 Acked 2114 Read 8 Done New Done true
3200 1116 Acked 2115 Read 7 Done New Done true
3201 1117 Acked 2116 Read 3 Done New Done true
3202 1118 Acked 2117 Read 6 Done New Done true
3203 1119 Acked 2118 Read 6 Done New Done true
3204 1120 Acked 2119 Read 2 Done New Done true
3205 1121 Acked 2120 Read 10 Done New Done true
3206 1122 Acked 2121 Write 3 Done New Done true
3207 1123 Acked 2122 Read 5 Done New Done true
3208 1124 Acked 2123 Flush 0 Done New Done true
3209 1125 Acked 2124 Write 2 Done New Done true
3210 1126 Acked 2125 Write 6 Done New Done true
3211 1127 Acked 2126 Read 10 Done New Done true
3212 1128 Acked 2127 Read 8 Done New Done true
3213 1129 Acked 2128 Read 7 Done New Done true
3214 1130 Acked 2129 Write 10 Done New Done true
3215 1131 Acked 2130 Write 10 Done New Done true
3216 1132 Acked 2131 Read 8 Done New Done true
3217 1133 Acked 2132 Write 5 Done New Done true
3218 1134 Acked 2133 Read 3 Done New Done true
3219 1135 Acked 2134 Write 10 Done New Done true
3220 1136 Acked 2135 Read 9 Done New Done true
3221 1137 Acked 2136 Read 7 Done New Done true
3222 1138 Acked 2137 Write 10 Done New Done true
3223 1139 Acked 2138 Write 4 Done New Done true
3224 1140 Acked 2139 Write 6 Done New Done true
3225 1141 Acked 2140 Read 7 Done New Done true
3226 1142 Acked 2141 Write 9 Done New Done true
3227 1143 Acked 2142 Read 7 Done New Done true
3228 1144 Acked 2143 Read 3 Done New Done true
3229 1145 Acked 2144 Write 6 Done New Done true
3230 1146 Acked 2145 Write 8 Done New Done true
3231 1147 Acked 2146 Write 5 Done New Done true
3232 1148 Acked 2147 Read 8 Done New Done true
3233 1149 Acked 2148 Read 5 Done New Done true
3234 1150 Acked 2149 Read 5 Done New Done true
3235 1151 Acked 2150 Read 9 Done New Done true
3236 1152 Acked 2151 Flush 0 Done New Done true
3237 1153 Acked 2152 Read 10 Done New Done true
3238 1154 Acked 2153 Read 10 Done New Done true
3239 1155 Acked 2154 Read 8 Done New Done true
3240 1156 Acked 2155 Read 1 Done New Done true
3241 1157 Acked 2156 Read 10 Done New Done true
3242 1158 Acked 2157 Write 10 Done New Done true
3243 1159 Acked 2158 Write 8 Done New Done true
3244 1160 Acked 2159 Write 8 Done New Done true
3245 1161 Acked 2160 Write 6 Done New Done true
3246 1162 Acked 2161 Read 6 Done New Done true
3247 1163 Acked 2162 Write 1 Done New Done true
3248 1164 Acked 2163 Read 10 Done New Done true
3249 1165 Acked 2164 Write 10 Done New Done true
3250 1166 Acked 2165 Read 6 Done New Done true
3251 1167 Acked 2166 Read 7 Done New Done true
3252 1168 Acked 2167 Write 8 Done New Done true
3253 1169 Acked 2168 Read 3 Done New Done true
3254 1170 Acked 2169 Read 6 Done New Done true
3255 1171 Acked 2170 Write 10 Done New Done true
3256 1172 Acked 2171 Write 5 Done New Done true
3257 1173 Acked 2172 Write 7 Done New Done true
3258 1174 Acked 2173 Write 5 Done New Done true
3259 1175 Acked 2174 Read 5 Done New Done true
3260 1176 Acked 2175 Write 4 Done New Done true
3261 1177 Acked 2176 Read 5 Done New Done true
3262 1178 Acked 2177 Write 9 Done New Done true
3263 1179 Acked 2178 Read 10 Done New Done true
3264 1180 Acked 2179 Flush 0 Done New Done true
3265 1181 Acked 2180 Read 5 Done New Done true
3266 1182 Acked 2181 Flush 0 Done New Done true
3267 1183 Acked 2182 Write 5 Done New Done true
3268 1184 Acked 2183 Read 1 Done New Done true
3269 1185 Acked 2184 Read 7 Done New Done true
3270 1186 Acked 2185 Read 9 Done New Done true
3271 1187 Acked 2186 Read 7 Done New Done true
3272 1188 Acked 2187 Read 2 Done New Done true
3273 1189 Acked 2188 Read 2 Done New Done true
3274 1190 Acked 2189 Write 3 Done New Done true
3275 1191 Acked 2190 Write 1 Done New Done true
3276 1192 Acked 2191 Read 4 Done New Done true
3277 1193 Acked 2192 Write 6 Done New Done true
3278 1194 Acked 2193 Read 1 Done New Done true
3279 1195 Acked 2194 Read 5 Done New Done true
3280 1196 Acked 2195 Read 5 Done New Done true
3281 1197 Acked 2196 Read 8 Done New Done true
3282 1198 Acked 2197 Read 7 Done New Done true
3283 1199 Acked 2198 Read 10 Done New Done true
3284 1200 Acked 2199 Write 4 Done New Done true
3285 1201 Acked 2200 Read 3 Done New Done true
3286 1202 Acked 2201 Write 1 Done New Done true
3287 1203 Acked 2202 Read 5 Done New Done true
3288 1204 Acked 2203 Write 7 Done New Done true
3289 1205 Acked 2204 Read 4 Done New Done true
3290 1206 Acked 2205 Flush 0 Done New Done true
3291 STATES DS:0 DS:1 DS:2 TOTAL
3292 New 0 302 0 302
3293 Sent 0 0 0 0
3294 Done 302 0 302 604
3295 Skipped 0 0 0 0
3296 Error 0 0 0 0
3297 Last Flush: 2205 1903 2205
3298 Downstairs last five completed: 1903 1902 1901 1900 1899
3299 Upstairs last five completed: 1206 1205 1204 1203 1202
3300 CLIENT: Up:0 ds:302 act:2
33012024-03-02T01:06:13.791ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33022024-03-02T01:06:13.791ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33032024-03-02T01:06:13.791ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33042024-03-02T01:06:13.791ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33052024-03-02T01:06:13.791ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33062024-03-02T01:06:13.791ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33072024-03-02T01:06:13.792ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33082024-03-02T01:06:14.792ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33092024-03-02T01:06:14.792ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33102024-03-02T01:06:14.793ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33112024-03-02T01:06:14.793ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33122024-03-02T01:06:14.793ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33132024-03-02T01:06:14.793ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33142024-03-02T01:06:14.793ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33152024-03-02T01:06:15.794ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33162024-03-02T01:06:15.794ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33172024-03-02T01:06:15.794ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33182024-03-02T01:06:15.794ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33192024-03-02T01:06:15.794ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33202024-03-02T01:06:15.794ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33212024-03-02T01:06:15.794ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33222024-03-02T01:06:16.795ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33232024-03-02T01:06:16.795ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33242024-03-02T01:06:16.795ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33252024-03-02T01:06:16.795ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33262024-03-02T01:06:16.795ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33272024-03-02T01:06:16.795ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
33282024-03-02T01:06:16.796ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3329 ----------------------------------------------------------------
3330 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3331 States: Active Offline Active
3332 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3333 905 Acked 1904 Write 3 Done New Done true
3334 906 Acked 1905 Flush 0 Done New Done true
3335 907 Acked 1906 Read 1 Done New Done true
3336 908 Acked 1907 Write 1 Done New Done true
3337 909 Acked 1908 Write 5 Done New Done true
3338 910 Acked 1909 Read 10 Done New Done true
3339 911 Acked 1910 Read 10 Done New Done true
3340 912 Acked 1911 Write 7 Done New Done true
3341 913 Acked 1912 Write 2 Done New Done true
3342 914 Acked 1913 Write 7 Done New Done true
3343 915 Acked 1914 Read 3 Done New Done true
3344 916 Acked 1915 Read 1 Done New Done true
3345 917 Acked 1916 Write 7 Done New Done true
3346 918 Acked 1917 Read 6 Done New Done true
3347 919 Acked 1918 Write 3 Done New Done true
3348 920 Acked 1919 Write 1 Done New Done true
3349 921 Acked 1920 Write 7 Done New Done true
3350 922 Acked 1921 Read 5 Done New Done true
3351 923 Acked 1922 Read 5 Done New Done true
3352 924 Acked 1923 Read 8 Done New Done true
3353 925 Acked 1924 Write 1 Done New Done true
3354 926 Acked 1925 Flush 0 Done New Done true
3355 927 Acked 1926 Write 4 Done New Done true
3356 928 Acked 1927 Flush 0 Done New Done true
3357 929 Acked 1928 Read 10 Done New Done true
3358 930 Acked 1929 Read 5 Done New Done true
3359 931 Acked 1930 Read 9 Done New Done true
3360 932 Acked 1931 Write 8 Done New Done true
3361 933 Acked 1932 Read 7 Done New Done true
3362 934 Acked 1933 Flush 0 Done New Done true
3363 935 Acked 1934 Read 10 Done New Done true
3364 936 Acked 1935 Write 3 Done New Done true
3365 937 Acked 1936 Read 4 Done New Done true
3366 938 Acked 1937 Write 8 Done New Done true
3367 939 Acked 1938 Read 3 Done New Done true
3368 940 Acked 1939 Write 6 Done New Done true
3369 941 Acked 1940 Flush 0 Done New Done true
3370 942 Acked 1941 Read 10 Done New Done true
3371 943 Acked 1942 Read 1 Done New Done true
3372 944 Acked 1943 Read 8 Done New Done true
3373 945 Acked 1944 Write 10 Done New Done true
3374 946 Acked 1945 Write 5 Done New Done true
3375 947 Acked 1946 Read 10 Done New Done true
3376 948 Acked 1947 Read 10 Done New Done true
3377 949 Acked 1948 Write 4 Done New Done true
3378 950 Acked 1949 Read 6 Done New Done true
3379 951 Acked 1950 Flush 0 Done New Done true
3380 952 Acked 1951 Read 10 Done New Done true
3381 953 Acked 1952 Write 10 Done New Done true
3382 954 Acked 1953 Flush 0 Done New Done true
3383 955 Acked 1954 Flush 0 Done New Done true
3384 956 Acked 1955 Write 2 Done New Done true
3385 957 Acked 1956 Write 5 Done New Done true
3386 958 Acked 1957 Write 3 Done New Done true
3387 959 Acked 1958 Write 6 Done New Done true
3388 960 Acked 1959 Read 6 Done New Done true
3389 961 Acked 1960 Write 4 Done New Done true
3390 962 Acked 1961 Write 6 Done New Done true
3391 963 Acked 1962 Write 10 Done New Done true
3392 964 Acked 1963 Flush 0 Done New Done true
3393 965 Acked 1964 Read 9 Done New Done true
3394 966 Acked 1965 Write 8 Done New Done true
3395 967 Acked 1966 Write 3 Done New Done true
3396 968 Acked 1967 Write 8 Done New Done true
3397 969 Acked 1968 Read 6 Done New Done true
3398 970 Acked 1969 Read 4 Done New Done true
3399 971 Acked 1970 Write 3 Done New Done true
3400 972 Acked 1971 Write 10 Done New Done true
3401 973 Acked 1972 Read 7 Done New Done true
3402 974 Acked 1973 Read 2 Done New Done true
3403 975 Acked 1974 Write 6 Done New Done true
3404 976 Acked 1975 Write 7 Done New Done true
3405 977 Acked 1976 Read 10 Done New Done true
3406 978 Acked 1977 Read 7 Done New Done true
3407 979 Acked 1978 Read 9 Done New Done true
3408 980 Acked 1979 Read 1 Done New Done true
3409 981 Acked 1980 Write 1 Done New Done true
3410 982 Acked 1981 Write 4 Done New Done true
3411 983 Acked 1982 Write 8 Done New Done true
3412 984 Acked 1983 Read 6 Done New Done true
3413 985 Acked 1984 Read 6 Done New Done true
3414 986 Acked 1985 Write 1 Done New Done true
3415 987 Acked 1986 Read 4 Done New Done true
3416 988 Acked 1987 Read 7 Done New Done true
3417 989 Acked 1988 Write 4 Done New Done true
3418 990 Acked 1989 Read 10 Done New Done true
3419 991 Acked 1990 Flush 0 Done New Done true
3420 992 Acked 1991 Write 6 Done New Done true
3421 993 Acked 1992 Read 2 Done New Done true
3422 994 Acked 1993 Read 7 Done New Done true
3423 995 Acked 1994 Write 6 Done New Done true
3424 996 Acked 1995 Read 1 Done New Done true
3425 997 Acked 1996 Read 1 Done New Done true
3426 998 Acked 1997 Write 1 Done New Done true
3427 999 Acked 1998 Write 10 Done New Done true
3428 1000 Acked 1999 Read 9 Done New Done true
3429 1001 Acked 2000 Read 1 Done New Done true
3430 1002 Acked 2001 Read 6 Done New Done true
3431 1003 Acked 2002 Write 1 Done New Done true
3432 1004 Acked 2003 Write 9 Done New Done true
3433 1005 Acked 2004 Read 3 Done New Done true
3434 1006 Acked 2005 Read 1 Done New Done true
3435 1007 Acked 2006 Write 5 Done New Done true
3436 1008 Acked 2007 Read 1 Done New Done true
3437 1009 Acked 2008 Read 1 Done New Done true
3438 1010 Acked 2009 Write 4 Done New Done true
3439 1011 Acked 2010 Flush 0 Done New Done true
3440 1012 Acked 2011 Flush 0 Done New Done true
3441 1013 Acked 2012 Write 3 Done New Done true
3442 1014 Acked 2013 Read 1 Done New Done true
3443 1015 Acked 2014 Read 7 Done New Done true
3444 1016 Acked 2015 Read 7 Done New Done true
3445 1017 Acked 2016 Write 1 Done New Done true
3446 1018 Acked 2017 Flush 0 Done New Done true
3447 1019 Acked 2018 Write 9 Done New Done true
3448 1020 Acked 2019 Read 2 Done New Done true
3449 1021 Acked 2020 Read 2 Done New Done true
3450 1022 Acked 2021 Flush 0 Done New Done true
3451 1023 Acked 2022 Read 2 Done New Done true
3452 1024 Acked 2023 Write 6 Done New Done true
3453 1025 Acked 2024 Read 7 Done New Done true
3454 1026 Acked 2025 Read 5 Done New Done true
3455 1027 Acked 2026 Write 6 Done New Done true
3456 1028 Acked 2027 Write 5 Done New Done true
3457 1029 Acked 2028 Flush 0 Done New Done true
3458 1030 Acked 2029 Read 2 Done New Done true
3459 1031 Acked 2030 Read 6 Done New Done true
3460 1032 Acked 2031 Write 10 Done New Done true
3461 1033 Acked 2032 Flush 0 Done New Done true
3462 1034 Acked 2033 Write 4 Done New Done true
3463 1035 Acked 2034 Read 6 Done New Done true
3464 1036 Acked 2035 Read 8 Done New Done true
3465 1037 Acked 2036 Write 9 Done New Done true
3466 1038 Acked 2037 Write 9 Done New Done true
3467 1039 Acked 2038 Write 4 Done New Done true
3468 1040 Acked 2039 Read 4 Done New Done true
3469 1041 Acked 2040 Write 1 Done New Done true
3470 1042 Acked 2041 Read 4 Done New Done true
3471 1043 Acked 2042 Read 7 Done New Done true
3472 1044 Acked 2043 Flush 0 Done New Done true
3473 1045 Acked 2044 Write 8 Done New Done true
3474 1046 Acked 2045 Flush 0 Done New Done true
3475 1047 Acked 2046 Read 1 Done New Done true
3476 1048 Acked 2047 Write 8 Done New Done true
3477 1049 Acked 2048 Flush 0 Done New Done true
3478 1050 Acked 2049 Write 10 Done New Done true
3479 1051 Acked 2050 Write 10 Done New Done true
3480 1052 Acked 2051 Write 3 Done New Done true
3481 1053 Acked 2052 Write 7 Done New Done true
3482 1054 Acked 2053 Flush 0 Done New Done true
3483 1055 Acked 2054 Write 5 Done New Done true
3484 1056 Acked 2055 Read 3 Done New Done true
3485 1057 Acked 2056 Write 3 Done New Done true
3486 1058 Acked 2057 Write 2 Done New Done true
3487 1059 Acked 2058 Flush 0 Done New Done true
3488 1060 Acked 2059 Read 4 Done New Done true
3489 1061 Acked 2060 Read 9 Done New Done true
3490 1062 Acked 2061 Write 5 Done New Done true
3491 1063 Acked 2062 Flush 0 Done New Done true
3492 1064 Acked 2063 Read 8 Done New Done true
3493 1065 Acked 2064 Write 6 Done New Done true
3494 1066 Acked 2065 Read 8 Done New Done true
3495 1067 Acked 2066 Write 9 Done New Done true
3496 1068 Acked 2067 Write 1 Done New Done true
3497 1069 Acked 2068 Write 6 Done New Done true
3498 1070 Acked 2069 Read 6 Done New Done true
3499 1071 Acked 2070 Read 2 Done New Done true
3500 1072 Acked 2071 Read 3 Done New Done true
3501 1073 Acked 2072 Read 2 Done New Done true
3502 1074 Acked 2073 Write 8 Done New Done true
3503 1075 Acked 2074 Flush 0 Done New Done true
3504 1076 Acked 2075 Write 3 Done New Done true
3505 1077 Acked 2076 Read 5 Done New Done true
3506 1078 Acked 2077 Read 10 Done New Done true
3507 1079 Acked 2078 Write 5 Done New Done true
3508 1080 Acked 2079 Write 8 Done New Done true
3509 1081 Acked 2080 Write 7 Done New Done true
3510 1082 Acked 2081 Read 4 Done New Done true
3511 1083 Acked 2082 Write 3 Done New Done true
3512 1084 Acked 2083 Write 7 Done New Done true
3513 1085 Acked 2084 Write 8 Done New Done true
3514 1086 Acked 2085 Write 9 Done New Done true
3515 1087 Acked 2086 Write 8 Done New Done true
3516 1088 Acked 2087 Read 3 Done New Done true
3517 1089 Acked 2088 Read 9 Done New Done true
3518 1090 Acked 2089 Flush 0 Done New Done true
3519 1091 Acked 2090 Write 9 Done New Done true
3520 1092 Acked 2091 Read 3 Done New Done true
3521 1093 Acked 2092 Flush 0 Done New Done true
3522 1094 Acked 2093 Write 2 Done New Done true
3523 1095 Acked 2094 Read 10 Done New Done true
3524 1096 Acked 2095 Write 6 Done New Done true
3525 1097 Acked 2096 Write 2 Done New Done true
3526 1098 Acked 2097 Read 6 Done New Done true
3527 1099 Acked 2098 Read 6 Done New Done true
3528 1100 Acked 2099 Write 1 Done New Done true
3529 1101 Acked 2100 Read 8 Done New Done true
3530 1102 Acked 2101 Read 2 Done New Done true
3531 1103 Acked 2102 Write 5 Done New Done true
3532 1104 Acked 2103 Flush 0 Done New Done true
3533 1105 Acked 2104 Read 2 Done New Done true
3534 1106 Acked 2105 Write 4 Done New Done true
3535 1107 Acked 2106 Read 4 Done New Done true
3536 1108 Acked 2107 Write 4 Done New Done true
3537 1109 Acked 2108 Read 2 Done New Done true
3538 1110 Acked 2109 Read 6 Done New Done true
3539 1111 Acked 2110 Read 4 Done New Done true
3540 1112 Acked 2111 Write 6 Done New Done true
3541 1113 Acked 2112 Write 3 Done New Done true
3542 1114 Acked 2113 Write 5 Done New Done true
3543 1115 Acked 2114 Read 8 Done New Done true
3544 1116 Acked 2115 Read 7 Done New Done true
3545 1117 Acked 2116 Read 3 Done New Done true
3546 1118 Acked 2117 Read 6 Done New Done true
3547 1119 Acked 2118 Read 6 Done New Done true
3548 1120 Acked 2119 Read 2 Done New Done true
3549 1121 Acked 2120 Read 10 Done New Done true
3550 1122 Acked 2121 Write 3 Done New Done true
3551 1123 Acked 2122 Read 5 Done New Done true
3552 1124 Acked 2123 Flush 0 Done New Done true
3553 1125 Acked 2124 Write 2 Done New Done true
3554 1126 Acked 2125 Write 6 Done New Done true
3555 1127 Acked 2126 Read 10 Done New Done true
3556 1128 Acked 2127 Read 8 Done New Done true
3557 1129 Acked 2128 Read 7 Done New Done true
3558 1130 Acked 2129 Write 10 Done New Done true
3559 1131 Acked 2130 Write 10 Done New Done true
3560 1132 Acked 2131 Read 8 Done New Done true
3561 1133 Acked 2132 Write 5 Done New Done true
3562 1134 Acked 2133 Read 3 Done New Done true
3563 1135 Acked 2134 Write 10 Done New Done true
3564 1136 Acked 2135 Read 9 Done New Done true
3565 1137 Acked 2136 Read 7 Done New Done true
3566 1138 Acked 2137 Write 10 Done New Done true
3567 1139 Acked 2138 Write 4 Done New Done true
3568 1140 Acked 2139 Write 6 Done New Done true
3569 1141 Acked 2140 Read 7 Done New Done true
3570 1142 Acked 2141 Write 9 Done New Done true
3571 1143 Acked 2142 Read 7 Done New Done true
3572 1144 Acked 2143 Read 3 Done New Done true
3573 1145 Acked 2144 Write 6 Done New Done true
3574 1146 Acked 2145 Write 8 Done New Done true
3575 1147 Acked 2146 Write 5 Done New Done true
3576 1148 Acked 2147 Read 8 Done New Done true
3577 1149 Acked 2148 Read 5 Done New Done true
3578 1150 Acked 2149 Read 5 Done New Done true
3579 1151 Acked 2150 Read 9 Done New Done true
3580 1152 Acked 2151 Flush 0 Done New Done true
3581 1153 Acked 2152 Read 10 Done New Done true
3582 1154 Acked 2153 Read 10 Done New Done true
3583 1155 Acked 2154 Read 8 Done New Done true
3584 1156 Acked 2155 Read 1 Done New Done true
3585 1157 Acked 2156 Read 10 Done New Done true
3586 1158 Acked 2157 Write 10 Done New Done true
3587 1159 Acked 2158 Write 8 Done New Done true
3588 1160 Acked 2159 Write 8 Done New Done true
3589 1161 Acked 2160 Write 6 Done New Done true
3590 1162 Acked 2161 Read 6 Done New Done true
3591 1163 Acked 2162 Write 1 Done New Done true
3592 1164 Acked 2163 Read 10 Done New Done true
3593 1165 Acked 2164 Write 10 Done New Done true
3594 1166 Acked 2165 Read 6 Done New Done true
3595 1167 Acked 2166 Read 7 Done New Done true
3596 1168 Acked 2167 Write 8 Done New Done true
3597 1169 Acked 2168 Read 3 Done New Done true
3598 1170 Acked 2169 Read 6 Done New Done true
3599 1171 Acked 2170 Write 10 Done New Done true
3600 1172 Acked 2171 Write 5 Done New Done true
3601 1173 Acked 2172 Write 7 Done New Done true
3602 1174 Acked 2173 Write 5 Done New Done true
3603 1175 Acked 2174 Read 5 Done New Done true
3604 1176 Acked 2175 Write 4 Done New Done true
3605 1177 Acked 2176 Read 5 Done New Done true
3606 1178 Acked 2177 Write 9 Done New Done true
3607 1179 Acked 2178 Read 10 Done New Done true
3608 1180 Acked 2179 Flush 0 Done New Done true
3609 1181 Acked 2180 Read 5 Done New Done true
3610 1182 Acked 2181 Flush 0 Done New Done true
3611 1183 Acked 2182 Write 5 Done New Done true
3612 1184 Acked 2183 Read 1 Done New Done true
3613 1185 Acked 2184 Read 7 Done New Done true
3614 1186 Acked 2185 Read 9 Done New Done true
3615 1187 Acked 2186 Read 7 Done New Done true
3616 1188 Acked 2187 Read 2 Done New Done true
3617 1189 Acked 2188 Read 2 Done New Done true
3618 1190 Acked 2189 Write 3 Done New Done true
3619 1191 Acked 2190 Write 1 Done New Done true
3620 1192 Acked 2191 Read 4 Done New Done true
3621 1193 Acked 2192 Write 6 Done New Done true
3622 1194 Acked 2193 Read 1 Done New Done true
3623 1195 Acked 2194 Read 5 Done New Done true
3624 1196 Acked 2195 Read 5 Done New Done true
3625 1197 Acked 2196 Read 8 Done New Done true
3626 1198 Acked 2197 Read 7 Done New Done true
3627 1199 Acked 2198 Read 10 Done New Done true
3628 1200 Acked 2199 Write 4 Done New Done true
3629 1201 Acked 2200 Read 3 Done New Done true
3630 1202 Acked 2201 Write 1 Done New Done true
3631 1203 Acked 2202 Read 5 Done New Done true
3632 1204 Acked 2203 Write 7 Done New Done true
3633 1205 Acked 2204 Read 4 Done New Done true
3634 1206 Acked 2205 Flush 0 Done New Done true
3635 STATES DS:0 DS:1 DS:2 TOTAL
3636 New 0 302 0 302
3637 Sent 0 0 0 0
3638 Done 302 0 302 604
3639 Skipped 0 0 0 0
3640 Error 0 0 0 0
3641 Last Flush: 2205 1903 2205
3642 Downstairs last five completed: 1903 1902 1901 1900 1899
3643 Upstairs last five completed: 1206 1205 1204 1203 1202
3644 CLIENT: Up:0 ds:302 act:2
36452024-03-02T01:06:17.797ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36462024-03-02T01:06:17.797ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36472024-03-02T01:06:17.797ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36482024-03-02T01:06:17.797ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36492024-03-02T01:06:17.797ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36502024-03-02T01:06:17.797ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36512024-03-02T01:06:17.797ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3652 [8820] got start action
3653 [8820] I am going to restart
3654 Starting downstairs at port 8820
3655 Make output file at "/tmp/dsc/downstairs-8820.txt"
3656 [8820][1] reports Starting
3657 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1166)
3658 [8820][1] reports Running
36592024-03-02T01:06:18.798ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36602024-03-02T01:06:18.798ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36612024-03-02T01:06:18.798ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36622024-03-02T01:06:18.798ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36632024-03-02T01:06:18.798ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36642024-03-02T01:06:18.798ZINFOcrucible: [1] client re-new 302 jobs since flush 1903 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36652024-03-02T01:06:18.798ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36662024-03-02T01:06:19.799ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36672024-03-02T01:06:19.799ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36682024-03-02T01:06:19.800ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36692024-03-02T01:06:19.800ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36702024-03-02T01:06:19.800ZINFOcrucible: send last flush ID to this DS: 1903 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36712024-03-02T01:06:19.801ZINFOcrucible: Replied this last flush ID: 1903 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36722024-03-02T01:06:19.801ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36732024-03-02T01:06:19.801ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36742024-03-02T01:06:19.801ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36752024-03-02T01:06:19.801ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3676 ----------------------------------------------------------------
3677 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3678 Downstairs last five completed: 2205 2204 2203 2202 2201
3679 Upstairs last five completed: 1206 1205 1204 1203 1202
3680 CLIENT: Up:0 ds:0 act:3
3681 Replay: All jobs finished, all DS active.
3682 Main task has work to do, go find it
3683 got dsc Stop(0)
3684 stop 0
3685 Mar 02 01:06:21.575 INFO request completed, latency_us: 344, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 01a77d37-c946-4dfc-9974-603b618c2011, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
3686 [8810] Got stop action so:false kr:false
3687 Mar 02 01:06:21.576 INFO request completed, latency_us: 116, response_code{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:06:21.576404534Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
3688 : 200, uri: /state/cid/0, method: GET, req_id: b3cbf5f3-98fc-4eaa-b53d-e390fbc42a4a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:06:21.576483049Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
3689 
36902024-03-02T01:06:21.576ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36912024-03-02T01:06:21.576ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36922024-03-02T01:06:21.576ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36932024-03-02T01:06:21.576ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3694 [8810] Exited with: ExitStatus(unix_wait_status(9))
3695 [8810][0] reports Exit
36962024-03-02T01:06:22.578ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36972024-03-02T01:06:22.578ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36982024-03-02T01:06:22.578ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
36992024-03-02T01:06:22.578ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37002024-03-02T01:06:22.578ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37012024-03-02T01:06:22.578ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37022024-03-02T01:06:22.578ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37032024-03-02T01:06:23.580ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37042024-03-02T01:06:23.580ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37052024-03-02T01:06:23.580ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37062024-03-02T01:06:23.580ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37072024-03-02T01:06:23.580ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37082024-03-02T01:06:23.580ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37092024-03-02T01:06:23.580ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37102024-03-02T01:06:24.582ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37112024-03-02T01:06:24.582ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37122024-03-02T01:06:24.582ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37132024-03-02T01:06:24.582ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37142024-03-02T01:06:24.582ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37152024-03-02T01:06:24.582ZINFOcrucible: [0] client re-new 0 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37162024-03-02T01:06:24.582ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3717 Mar 02 01:06:25.578 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/0, method: GET, req_id: 728de4ec-0102-44cb-9e4f-e94cf87001ea, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
3718 001/300 Read block 1266 len 36864
3719 002/300 Read block 2848 len 36864
3720 003/300 Write block 2309 len 36864 data: 2 2 2 2 3 4 4 4 4
3721 004/300 Read block 3481 len 4096
37222024-03-02T01:06:25.583ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37232024-03-02T01:06:25.583ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37242024-03-02T01:06:25.583ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37252024-03-02T01:06:25.583ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37262024-03-02T01:06:25.583ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37272024-03-02T01:06:25.583ZINFOcrucible: [0] client re-new 4 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
37282024-03-02T01:06:25.583ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
3729 005/300 Write block 3904 len 8192 data: 3 2
3730 006/300 Write block 3459 len 24576 data: 2 2 2 2 2 2
3731 007/300 Write block 1380 len 36864 data: 2 2 2 2 2 2 2 2 2
3732 008/300 Write block 2834 len 8192 data: 3 3
3733 009/300 Read block 78 len 24576
3734 010/300 Read block 705 len 40960
3735 011/300 Write block 2933 len 36864 data: 2 2 2 2 2 3 3 3 3
3736 012/300 Read block 2483 len 16384
3737 013/300 Write block 4952 len 16384 data: 2 2 2 2
3738 014/300 Flush
3739 015/300 Read block 819 len 40960
3740 016/300 Write block 4031 len 12288 data: 2 2 3
3741 017/300 Write block 192 len 36864 data: 4 4 4 4 3 3 2 2 2
3742 018/300 Read block 3351 len 24576
3743 019/300 Flush
3744 020/300 Read block 808 len 4096
3745 021/300 Write block 2638 len 8192 data: 4 4
3746 022/300 Flush
3747 023/300 Write block 251 len 28672 data: 3 3 3 3 3 2 2
3748 024/300 Read block 503 len 8192
3749 025/300 Write block 332 len 40960 data: 2 3 3 3 4 4 4 4 4 4
3750 026/300 Write block 3805 len 4096 data: 4
3751 027/300 Write block 4098 len 28672 data: 2 2 2 2 2 2 2
3752 028/300 Write block 4968 len 32768 data: 2 3 4 4 4 3 3 2
3753 029/300 Read block 4219 len 24576
3754 030/300 Write block 4185 len 28672 data: 3 3 3 2 2 2 2
3755 031/300 Write block 411 len 4096 data: 3
3756 032/300 Write block 357 len 16384 data: 2 2 2 2
3757 033/300 Write block 2166 len 28672 data: 2 2 2 2 2 2 2
3758 034/300 Flush
3759 035/300 Read block 1573 len 32768
3760 036/300 Read block 3477 len 32768
3761 037/300 Write block 3806 len 24576 data: 4 4 5 5 5 2
3762 038/300 Read block 4968 len 12288
3763 039/300 Flush
3764 040/300 Write block 3154 len 32768 data: 3 3 2 2 2 2 2 2
3765 041/300 Write block 2359 len 36864 data: 3 4 4 4 4 5 5 4 4
3766 042/300 Write block 3477 len 16384 data: 2 2 2 3
3767 043/300 Write block 4317 len 24576 data: 2 2 2 2 2 2
3768 044/300 Read block 3006 len 28672
3769 045/300 Write block 303 len 20480 data: 2 3 3 2 2
3770 046/300 Flush
3771 047/300 Read block 1041 len 8192
3772 048/300 Read block 4255 len 8192
3773 049/300 Read block 1288 len 8192
3774 050/300 Write block 3070 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3775 051/300 Write block 332 len 36864 data: 3 4 4 4 5 5 5 5 5
3776 052/300 Write block 3955 len 32768 data: 2 2 2 2 3 3 3 3
3777 053/300 Read block 2156 len 12288
3778 054/300 Write block 3323 len 32768 data: 2 2 2 2 2 2 3 3
3779 055/300 Read block 3869 len 32768
3780 056/300 Read block 110 len 40960
3781 057/300 Read block 803 len 12288
3782 058/300 Write block 4136 len 28672 data: 3 3 3 3 3 3 3
3783 059/300 Flush
3784 060/300 Write block 4799 len 12288 data: 3 3 3
3785 061/300 Write block 3322 len 4096 data: 2
3786 062/300 Read block 3049 len 40960
3787 063/300 Read block 385 len 40960
3788 064/300 Read block 1924 len 28672
3789 065/300 Read block 3806 len 8192
3790 066/300 Read block 1988 len 28672
3791 067/300 Read block 4765 len 28672
3792 068/300 Flush
3793 069/300 Write block 1095 len 32768 data: 2 2 2 2 2 2 2 2
3794 070/300 Flush
3795 071/300 Write block 1647 len 36864 data: 3 3 3 3 2 2 2 2 2
3796 072/300 Write block 953 len 16384 data: 2 2 2 2
3797 073/300 Read block 4279 len 28672
3798 074/300 Write block 2768 len 4096 data: 4
3799 075/300 Flush
3800 076/300 Flush
3801 077/300 Read block 2215 len 36864
3802 078/300 Write block 856 len 32768 data: 2 2 2 2 2 2 2 2
3803 079/300 Flush
3804 080/300 Read block 737 len 40960
3805 081/300 Write block 3482 len 32768 data: 3 3 3 3 3 3 3 2
3806 082/300 Read block 3906 len 28672
3807 083/300 Read block 1053 len 4096
3808 084/300 Read block 4669 len 4096
3809 085/300 Write block 180 len 36864 data: 2 2 2 2 2 2 2 2 2
3810 086/300 Read block 4170 len 20480
3811 087/300 Flush
3812 088/300 Write block 1020 len 20480 data: 2 2 2 2 2
3813 089/300 Flush
3814 090/300 Write block 4643 len 24576 data: 4 5 4 5 5 4
3815 091/300 Write block 3984 len 32768 data: 3 3 3 3 3 3 3 3
3816 092/300 Read block 3414 len 4096
3817 093/300 Write block 3246 len 24576 data: 2 2 2 2 2 2
3818 094/300 Write block 2716 len 32768 data: 3 3 3 3 3 2 2 2
3819 095/300 Read block 4103 len 20480
3820 096/300 Write block 2200 len 36864 data: 2 2 2 2 2 2 2 2 2
3821 097/300 Read block 4924 len 32768
3822 098/300 Write block 3530 len 28672 data: 3 3 3 3 3 3 2
3823 099/300 Flush
3824 100/300 Flush
3825 101/300 Read block 3960 len 36864
3826 102/300 Write block 4118 len 28672 data: 2 2 3 3 4 4 3
3827 103/300 Read block 3149 len 24576
3828 104/300 Read block 2 len 24576
3829 105/300 Read block 2452 len 12288
3830 106/300 Read block 4091 len 4096
3831 107/300 Read block 2224 len 40960
3832 108/300 Read block 3706 len 12288
3833 109/300 Write block 3440 len 36864 data: 2 3 2 2 2 2 3 4 4
3834 110/300 Write block 4425 len 8192 data: 2 2
3835 111/300 Read block 1557 len 36864
3836 112/300 Write block 1299 len 28672 data: 3 3 3 2 2 2 2
3837 113/300 Write block 1732 len 32768 data: 2 2 3 3 4 4 4 4
3838 114/300 Write block 1026 len 40960 data: 2 2 2 3 3 3 3 3 3 2
3839 115/300 Read block 3017 len 4096
3840 116/300 Read block 3361 len 32768
3841 117/300 Write block 662 len 4096 data: 2
3842 118/300 Write block 3462 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3843 119/300 Read block 4410 len 4096
3844 120/300 Read block 2218 len 32768
3845 121/300 Read block 1603 len 36864
3846 122/300 Write block 61 len 8192 data: 2 2
3847 123/300 Read block 4060 len 36864
3848 124/300 Read block 978 len 32768
3849 125/300 Write block 3352 len 12288 data: 3 3 3
3850 126/300 Read block 4943 len 32768
3851 127/300 Read block 2686 len 40960
3852 128/300 Write block 424 len 4096 data: 2
3853 129/300 Write block 3708 len 28672 data: 3 2 2 2 2 2 2
3854 130/300 Write block 996 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3855 131/300 Write block 4069 len 32768 data: 3 3 3 3 3 3 3 3
3856 132/300 Flush
3857 133/300 Read block 4830 len 28672
3858 134/300 Write block 2476 len 40960 data: 2 2 2 2 3 3 4 4 4 4
3859 135/300 Read block 506 len 36864
3860 136/300 Write block 4569 len 40960 data: 3 3 4 3 3 3 4 3 3 3
3861 137/300 Flush
3862 138/300 Read block 3745 len 8192
3863 139/300 Write block 2772 len 16384 data: 2 2 2 2
3864 140/300 Write block 4217 len 32768 data: 2 2 2 2 2 2 2 2
3865 141/300 Write block 350 len 36864 data: 2 2 2 2 2 2 2 3 3
3866 142/300 Read block 130 len 36864
3867 143/300 Read block 2965 len 16384
3868 144/300 Read block 473 len 24576
3869 145/300 Write block 3294 len 24576 data: 4 4 4 3 3 2
3870 146/300 Write block 1569 len 8192 data: 3 2
3871 147/300 Write block 277 len 32768 data: 2 2 2 2 2 2 2 2
3872 148/300 Read block 3172 len 36864
3873 149/300 Read block 3253 len 28672
3874 150/300 Flush
3875 151/300 Read block 2885 len 4096
3876 152/300 Read block 228 len 8192
3877 153/300 Write block 1712 len 16384 data: 3 3 2 2
3878 154/300 Flush
3879 155/300 Read block 604 len 36864
3880 156/300 Read block 477 len 28672
3881 157/300 Read block 3722 len 40960
3882 158/300 Flush
3883 159/300 Write block 375 len 4096 data: 2
3884 160/300 Read block 1597 len 20480
3885 161/300 Write block 263 len 36864 data: 2 2 2 2 3 2 2 2 2
3886 162/300 Read block 3054 len 12288
3887 163/300 Flush
3888 164/300 Read block 289 len 16384
3889 165/300 Write block 4396 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3890 166/300 Write block 2082 len 36864 data: 2 2 2 2 2 3 3 3 3
3891 167/300 Write block 2809 len 12288 data: 3 4 4
3892 168/300 Write block 1656 len 40960 data: 2 2 2 2 2 2 2 2 2 3
3893 169/300 Read block 275 len 28672
3894 170/300 Write block 2038 len 24576 data: 2 2 2 2 2 2
3895 171/300 Write block 190 len 8192 data: 2 2
3896 172/300 Write block 1639 len 20480 data: 2 3 3 3 3
3897 173/300 Read block 4929 len 20480
3898 174/300 Write block 4739 len 28672 data: 2 2 2 2 2 2 2
3899 175/300 Read block 361 len 24576
3900 176/300 Write block 4050 len 16384 data: 3 3 3 2
3901 177/300 Write block 2908 len 32768 data: 3 3 2 2 2 2 3 3
3902 178/300 Read block 4165 len 32768
3903 179/300 Read block 570 len 8192
3904 180/300 Write block 2501 len 16384 data: 2 3 2 2
3905 181/300 Read block 1625 len 4096
3906 182/300 Read block 1088 len 36864
3907 183/300 Read block 4074 len 16384
3908 184/300 Write block 4583 len 32768 data: 3 3 3 3 3 4 4 3
3909 185/300 Read block 988 len 36864
3910 186/300 Read block 2532 len 20480
3911 187/300 Read block 4101 len 40960
3912 188/300 Write block 4910 len 4096 data: 2
3913 189/300 Read block 3547 len 20480
3914 190/300 Write block 4994 len 8192 data: 2 2
3915 191/300 Read block 4333 len 20480
3916 192/300 Read block 4677 len 12288
3917 193/300 Write block 4767 len 32768 data: 3 2 2 2 2 2 2 2
3918 194/300 Write block 3216 len 16384 data: 2 2 2 2
3919 195/300 Read block 1621 len 36864
3920 196/300 Write block 3427 len 4096 data: 2
3921 197/300 Write block 3581 len 28672 data: 2 2 2 2 2 2 2
3922 198/300 Read block 4064 len 4096
3923 199/300 Write block 629 len 8192 data: 2 2
3924 200/300 Read block 2817 len 24576
3925 201/300 Read block 1345 len 20480
3926 202/300 Write block 1297 len 8192 data: 2 3
3927 203/300 Write block 3765 len 36864 data: 2 2 2 2 2 2 2 2 2
3928 204/300 Read block 4451 len 8192
3929 205/300 Read block 3756 len 32768
3930 206/300 Write block 2365 len 20480 data: 6 5 5 4 4
3931 207/300 Read block 2011 len 16384
3932 208/300 Write block 733 len 20480 data: 2 3 3 2 2
3933 209/300 Flush
3934 210/300 Read block 674 len 36864
3935 211/300 Write block 2382 len 16384 data: 2 2 2 2
3936 212/300 Write block 2040 len 32768 data: 3 3 3 3 2 2 2 3
3937 213/300 Read block 948 len 20480
3938 214/300 Write block 923 len 24576 data: 4 4 4 5 4 3
3939 215/300 Read block 3223 len 8192
3940 216/300 Write block 4714 len 20480 data: 3 3 2 2 2
3941 217/300 Read block 4132 len 16384
3942 218/300 Read block 3052 len 12288
3943 219/300 Write block 928 len 40960 data: 4 2 2 2 2 2 2 3 3 2
3944 220/300 Write block 3167 len 32768 data: 2 2 2 2 2 2 2 2
3945 221/300 Read block 4498 len 28672
3946 222/300 Write block 4092 len 8192 data: 2 2
3947 223/300 Write block 3234 len 12288 data: 2 2 2
3948 224/300 Write block 739 len 36864 data: 2 2 2 2 2 2 2 2 2
3949 225/300 Read block 3964 len 32768
3950 226/300 Read block 2364 len 4096
3951 227/300 Read block 4128 len 16384
3952 228/300 Read block 3616 len 40960
3953 229/300 Flush
3954 230/300 Read block 2726 len 24576
3955 231/300 Read block 2800 len 16384
3956 232/300 Write block 4476 len 20480 data: 2 2 2 2 2
3957 233/300 Write block 1623 len 28672 data: 3 3 2 2 3 4 4
3958 234/300 Write block 2682 len 4096 data: 3
3959 235/300 Write block 3488 len 20480 data: 4 3 2 2 2
3960 236/300 Write block 2431 len 28672 data: 4 3 3 3 3 3 3
3961 237/300 Write block 3206 len 32768 data: 3 3 3 3 2 2 2 2
3962 238/300 Write block 2091 len 4096 data: 3
3963 239/300 Write block 3760 len 40960 data: 3 2 2 2 2 3 3 3 3 3
3964 240/300 Write block 2993 len 20480 data: 4 5 4 4 3
3965 241/300 Read block 1084 len 24576
3966 242/300 Read block 3374 len 12288
3967 243/300 Write block 1894 len 36864 data: 2 2 2 2 2 2 2 2 3
3968 244/300 Read block 2042 len 40960
3969 245/300 Write block 3940 len 4096 data: 4
3970 246/300 Write block 3225 len 12288 data: 3 2 2
3971 247/300 Read block 1787 len 36864
3972 248/300 Flush
3973 249/300 Write block 3038 len 4096 data: 2
3974 250/300 Read block 2480 len 32768
3975 251/300 Write block 2469 len 28672 data: 2 2 2 2 2 2 2
3976 252/300 Write block 1182 len 8192 data: 4 4
3977 253/300 Flush
3978 254/300 Write block 2835 len 40960 data: 4 3 3 3 4 4 3 4 4 4
3979 255/300 Read block 2368 len 40960
3980 256/300 Read block 3758 len 4096
3981 257/300 Read block 2865 len 16384
3982 258/300 Write block 4227 len 32768 data: 2 2 2 3 4 4 4 4
3983 259/300 Flush
3984 260/300 Write block 810 len 24576 data: 2 2 2 2 3 3
3985 261/300 Read block 2969 len 40960
3986 262/300 Read block 1243 len 20480
3987 263/300 Write block 481 len 24576 data: 3 2 2 2 2 2
3988 264/300 Write block 1708 len 4096 data: 3
3989 265/300 Write block 2030 len 4096 data: 2
3990 266/300 Read block 1647 len 40960
3991 267/300 Write block 4791 len 32768 data: 3 4 4 3 3 3 3 3
3992 268/300 Read block 4870 len 24576
3993 269/300 Write block 2728 len 4096 data: 2
3994 270/300 Flush
3995 271/300 Write block 4328 len 28672 data: 3 3 2 2 3 3 3
3996 272/300 Read block 605 len 4096
3997 273/300 Read block 1317 len 20480
3998 274/300 Read block 572 len 8192
3999 275/300 Write block 399 len 32768 data: 2 2 2 2 2 2 2 2
4000 276/300 Write block 765 len 40960 data: 3 2 2 2 2 2 2 2 2 2
4001 277/300 Write block 4133 len 28672 data: 3 3 3 4 4 4 4
4002 278/300 Write block 2405 len 20480 data: 2 2 2 2 2
4003 279/300 Write block 519 len 20480 data: 2 2 2 2 2
4004 280/300 Read block 3476 len 20480
4005 281/300 Write block 4229 len 16384 data: 3 4 5 5
4006 282/300 Read block 2601 len 8192
4007 283/300 Read block 2484 len 28672
4008 284/300 Read block 525 len 16384
4009 285/300 Write block 995 len 40960 data: 3 4 3 3 3 3 3 3 3 3
4010 286/300 Write block 4542 len 20480 data: 4 4 3 3 3
4011 287/300 Read block 4716 len 4096
4012 288/300 Read block 2415 len 20480
4013 289/300 Read block 3064 len 16384
4014 290/300 Write block 3592 len 4096 data: 2
4015 291/300 Read block 3404 len 24576
4016 292/300 Write block 2790 len 32768 data: 2 3 3 3 3 2 3 3
4017 293/300 Write block 878 len 8192 data: 2 2
4018 294/300 Write block 4267 len 8192 data: 3 3
4019 295/300 Read block 4704 len 4096
4020 296/300 Write block 4237 len 24576 data: 3 2 2 2 2 2
4021 297/300 Write block 733 len 4096 data: 3
4022 298/300 Write block 2218 len 4096 data: 3
4023 299/300 Write block 4844 len 40960 data: 4 4 4 2 2 2 2 2 2 2
4024 300/300 Read block 3864 len 12288
4025 Main task has work to do, go find it
4026 got dsc Start(0)
4027 start 0
4028 Mar 02 01:06:26.279 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/0, method: POST, req_id: 13ea96b7-8ab6-47a6-ad59-2574e83ff81f, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
4029 Replay: started 0, returned:Ok(())
4030 ----------------------------------------------------------------
4031 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
4032 States: Offline Active Active
4033 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4034 1207 Acked 2206 Read 9 New Done Done true
4035 1208 Acked 2207 Read 9 New Done Done true
4036 1209 Acked 2208 Write 9 New Done Done true
4037 1210 Acked 2209 Read 1 New Done Done true
4038 1211 Acked 2210 Write 2 New Done Done false
4039 1212 Acked 2211 Write 6 New Done Done false
4040 1213 Acked 2212 Write 9 New Done Done false
4041 1214 Acked 2213 Write 2 New Done Done false
4042 1215 Acked 2214 Read 6 New Done Done false
4043 1216 Acked 2215 Read 10 New Done Done false
4044 1217 Acked 2216 Write 9 New Done Done false
4045 1218 Acked 2217 Read 4 New Done Done false
4046 1219 Acked 2218 Write 4 New Done Done false
4047 1220 Acked 2219 Flush 0 New Done Done false
4048 1221 Acked 2220 Read 10 New Done Done false
4049 1222 Acked 2221 Write 3 New Done Done false
4050 1223 Acked 2222 Write 9 New Done Done false
4051 1224 Acked 2223 Read 6 New Done Done false
4052 1225 Acked 2224 Flush 0 New Done Done false
4053 1226 Acked 2225 Read 1 New Done Done false
4054 1227 Acked 2226 Write 2 New Done Done false
4055 1228 Acked 2227 Flush 0 New Done Done false
4056 1229 Acked 2228 Write 7 New Done Done false
4057 1230 Acked 2229 Read 2 New Done Done false
4058 1231 Acked 2230 Write 10 New Done Done false
4059 1232 Acked 2231 Write 1 New Done Done false
4060 1233 Acked 2232 Write 7 New Done Done false
4061 1234 Acked 2233 Write 8 New Done Done false
4062 1235 Acked 2234 Read 6 New Done Done false
4063 1236 Acked 2235 Write 7 New Done Done false
4064 1237 Acked 2236 Write 1 New Done Done false
4065 1238 Acked 2237 Write 4 New Done Done false
4066 1239 Acked 2238 Write 7 New Done Done false
4067 1240 Acked 2239 Flush 0 New Done Done false
4068 1241 Acked 2240 Read 8 New Done Done false
4069 1242 Acked 2241 Read 8 New Done Done false
4070 1243 Acked 2242 Write 6 New Done Done false
4071 1244 Acked 2243 Read 3 New Done Done false
4072 1245 Acked 2244 Flush 0 New Done Done false
4073 1246 Acked 2245 Write 8 New Done Done false
4074 1247 Acked 2246 Write 9 New Done Done false
4075 1248 Acked 2247 Write 4 New Done Done false
4076 1249 Acked 2248 Write 6 New Done Done false
4077 1250 Acked 2249 Read 7 New Done Done false
4078 1251 Acked 2250 Write 5 New Done Done false
4079 1252 Acked 2251 Flush 0 New Done Done false
4080 1253 Acked 2252 Read 2 New Done Done false
4081 1254 Acked 2253 Read 2 New Done Done false
4082 1255 Acked 2254 Read 2 New Done Done false
4083 1256 Acked 2255 Write 10 New Done Done false
4084 1257 Acked 2256 Write 9 New Done Done false
4085 1258 Acked 2257 Write 8 New Done Done false
4086 1259 Acked 2258 Read 3 New Done Done false
4087 1260 Acked 2259 Write 8 New Done Done false
4088 1261 Acked 2260 Read 8 New Done Done false
4089 1262 Acked 2261 Read 10 New Done Done false
4090 1263 Acked 2262 Read 3 New Done Done false
4091 1264 Acked 2263 Write 7 New Done Done false
4092 1265 Acked 2264 Flush 0 New Done Done false
4093 1266 Acked 2265 Write 3 New Done Done false
4094 1267 Acked 2266 Write 1 New Done Done false
4095 1268 Acked 2267 Read 10 New Done Done false
4096 1269 Acked 2268 Read 10 New Done Done false
4097 1270 Acked 2269 Read 7 New Done Done false
4098 1271 Acked 2270 Read 2 New Done Done false
4099 1272 Acked 2271 Read 7 New Done Done false
4100 1273 Acked 2272 Read 7 New Done Done false
4101 1274 Acked 2273 Flush 0 New Done Done false
4102 1275 Acked 2274 Write 8 New Done Done false
4103 1276 Acked 2275 Flush 0 New Done Done false
4104 1277 Acked 2276 Write 9 New Done Done false
4105 1278 Acked 2277 Write 4 New Done Done false
4106 1279 Acked 2278 Read 7 New Done Done false
4107 1280 Acked 2279 Write 1 New Done Done false
4108 1281 Acked 2280 Flush 0 New Done Done false
4109 1282 Acked 2281 Flush 0 New Done Done false
4110 1283 Acked 2282 Read 9 New Done Done false
4111 1284 Acked 2283 Write 8 New Done Done false
4112 1285 Acked 2284 Flush 0 New Done Done false
4113 1286 Acked 2285 Read 10 New Done Done false
4114 1287 Acked 2286 Write 8 New Done Done false
4115 1288 Acked 2287 Read 7 New Done Done false
4116 1289 Acked 2288 Read 1 New Done Done false
4117 1290 Acked 2289 Read 1 New Done Done false
4118 1291 Acked 2290 Write 9 New Done Done false
4119 1292 Acked 2291 Read 5 New Done Done false
4120 1293 Acked 2292 Flush 0 New Done Done false
4121 1294 Acked 2293 Write 5 New Done Done false
4122 1295 Acked 2294 Flush 0 New Done Done false
4123 1296 Acked 2295 Write 6 New Done Done false
4124 1297 Acked 2296 Write 8 New Done Done false
4125 1298 Acked 2297 Read 1 New Done Done false
4126 1299 Acked 2298 Write 6 New Done Done false
4127 1300 Acked 2299 Write 8 New Done Done false
4128 1301 Acked 2300 Read 5 New Done Done false
4129 1302 Acked 2301 Write 9 New Done Done false
4130 1303 Acked 2302 Read 8 New Done Done false
4131 1304 Acked 2303 Write 7 New Done Done false
4132 1305 Acked 2304 Flush 0 New Done Done false
4133 1306 Acked 2305 Flush 0 New Done Done false
4134 1307 Acked 2306 Read 9 New Done Done false
4135 1308 Acked 2307 Write 7 New Done Done false
4136 1309 Acked 2308 Read 6 New Done Done false
4137 1310 Acked 2309 Read 6 New Done Done false
4138 1311 Acked 2310 Read 3 New Done Done false
4139 1312 Acked 2311 Read 1 New Done Done false
4140 1313 Acked 2312 Read 10 New Done Done false
4141 1314 Acked 2313 Read 3 New Done Done false
4142 1315 Acked 2314 Write 9 New Done Done false
4143 1316 Acked 2315 Write 2 New Done Done false
4144 1317 Acked 2316 Read 9 New Done Done false
4145 1318 Acked 2317 Write 7 New Done Done false
4146 1319 Acked 2318 Write 8 New Done Done false
4147 1320 Acked 2319 Write 10 New Done Done false
4148 1321 Acked 2320 Read 1 New Done Done false
4149 1322 Acked 2321 Read 8 New Done Done false
4150 1323 Acked 2322 Write 1 New Done Done false
4151 1324 Acked 2323 Write 10 New Done Done false
4152 1325 Acked 2324 Read 1 New Done Done false
4153 1326 Acked 2325 Read 8 New Done Done false
4154 1327 Acked 2326 Read 9 New Done Done false
4155 1328 Acked 2327 Write 2 New Done Done false
4156 1329 Acked 2328 Read 9 New Done Done false
4157 1330 Acked 2329 Read 8 New Done Done false
4158 1331 Acked 2330 Write 3 New Done Done false
4159 1332 Acked 2331 Read 8 New Done Done false
4160 1333 Acked 2332 Read 10 New Done Done false
4161 1334 Acked 2333 Write 1 New Done Done false
4162 1335 Acked 2334 Write 7 New Done Done false
4163 1336 Acked 2335 Write 10 New Done Done false
4164 1337 Acked 2336 Write 8 New Done Done false
4165 1338 Acked 2337 Flush 0 New Done Done false
4166 1339 Acked 2338 Read 7 New Done Done false
4167 1340 Acked 2339 Write 10 New Done Done false
4168 1341 Acked 2340 Read 9 New Done Done false
4169 1342 Acked 2341 Write 10 New Done Done false
4170 1343 Acked 2342 Flush 0 New Done Done false
4171 1344 Acked 2343 Read 2 New Done Done false
4172 1345 Acked 2344 Write 4 New Done Done false
4173 1346 Acked 2345 Write 8 New Done Done false
4174 1347 Acked 2346 Write 9 New Done Done false
4175 1348 Acked 2347 Read 9 New Done Done false
4176 1349 Acked 2348 Read 4 New Done Done false
4177 1350 Acked 2349 Read 6 New Done Done false
4178 1351 Acked 2350 Write 6 New Done Done false
4179 1352 Acked 2351 Write 2 New Done Done false
4180 1353 Acked 2352 Write 8 New Done Done false
4181 1354 Acked 2353 Read 9 New Done Done false
4182 1355 Acked 2354 Read 7 New Done Done false
4183 1356 Acked 2355 Flush 0 New Done Done false
4184 1357 Acked 2356 Read 1 New Done Done false
4185 1358 Acked 2357 Read 2 New Done Done false
4186 1359 Acked 2358 Write 4 New Done Done false
4187 1360 Acked 2359 Flush 0 New Done Done false
4188 1361 Acked 2360 Read 9 New Done Done false
4189 1362 Acked 2361 Read 7 New Done Done false
4190 1363 Acked 2362 Read 10 New Done Done false
4191 1364 Acked 2363 Flush 0 New Done Done false
4192 1365 Acked 2364 Write 1 New Done Done false
4193 1366 Acked 2365 Read 5 New Done Done false
4194 1367 Acked 2366 Write 9 New Done Done false
4195 1368 Acked 2367 Read 3 New Done Done false
4196 1369 Acked 2368 Flush 0 New Done Done false
4197 1370 Acked 2369 Read 4 New Done Done false
4198 1371 Acked 2370 Write 10 New Done Done false
4199 1372 Acked 2371 Write 9 New Done Done false
4200 1373 Acked 2372 Write 3 New Done Done false
4201 1374 Acked 2373 Write 10 New Done Done false
4202 1375 Acked 2374 Read 7 New Done Done false
4203 1376 Acked 2375 Write 6 New Done Done false
4204 1377 Acked 2376 Write 2 New Done Done false
4205 1378 Acked 2377 Write 5 New Done Done false
4206 1379 Acked 2378 Read 5 New Done Done false
4207 1380 Acked 2379 Write 7 New Done Done false
4208 1381 Acked 2380 Read 6 New Done Done false
4209 1382 Acked 2381 Write 4 New Done Done false
4210 1383 Acked 2382 Write 8 New Done Done false
4211 1384 Acked 2383 Read 8 New Done Done false
4212 1385 Acked 2384 Read 2 New Done Done false
4213 1386 Acked 2385 Write 4 New Done Done false
4214 1387 Acked 2386 Read 1 New Done Done false
4215 1388 Acked 2387 Read 9 New Done Done false
4216 1389 Acked 2388 Read 4 New Done Done false
4217 1390 Acked 2389 Write 8 New Done Done false
4218 1391 Acked 2390 Read 9 New Done Done false
4219 1392 Acked 2391 Read 5 New Done Done false
4220 1393 Acked 2392 Read 10 New Done Done false
4221 1394 Acked 2393 Write 1 New Done Done false
4222 1395 Acked 2394 Read 5 New Done Done false
4223 1396 Acked 2395 Write 2 New Done Done false
4224 1397 Acked 2396 Read 5 New Done Done false
4225 1398 Acked 2397 Read 3 New Done Done false
4226 1399 Acked 2398 Write 8 New Done Done false
4227 1400 Acked 2399 Write 4 New Done Done false
4228 1401 Acked 2400 Read 9 New Done Done false
4229 1402 Acked 2401 Write 1 New Done Done false
4230 1403 Acked 2402 Write 7 New Done Done false
4231 1404 Acked 2403 Read 1 New Done Done false
4232 1405 Acked 2404 Write 2 New Done Done false
4233 1406 Acked 2405 Read 6 New Done Done false
4234 1407 Acked 2406 Read 5 New Done Done false
4235 1408 Acked 2407 Write 2 New Done Done false
4236 1409 Acked 2408 Write 9 New Done Done false
4237 1410 Acked 2409 Read 2 New Done Done false
4238 1411 Acked 2410 Read 8 New Done Done false
4239 1412 Acked 2411 Write 5 New Done Done false
4240 1413 Acked 2412 Read 4 New Done Done false
4241 1414 Acked 2413 Write 5 New Done Done false
4242 1415 Acked 2414 Flush 0 New Done Done false
4243 1416 Acked 2415 Read 9 New Done Done false
4244 1417 Acked 2416 Write 4 New Done Done false
4245 1418 Acked 2417 Write 8 New Done Done false
4246 1419 Acked 2418 Read 5 New Done Done false
4247 1420 Acked 2419 Write 6 New Done Done false
4248 1421 Acked 2420 Read 2 New Done Done false
4249 1422 Acked 2421 Write 5 New Done Done false
4250 1423 Acked 2422 Read 4 New Done Done false
4251 1424 Acked 2423 Read 3 New Done Done false
4252 1425 Acked 2424 Write 10 New Done Done false
4253 1426 Acked 2425 Write 8 New Done Done false
4254 1427 Acked 2426 Read 7 New Done Done false
4255 1428 Acked 2427 Write 2 New Done Done false
4256 1429 Acked 2428 Write 3 New Done Done false
4257 1430 Acked 2429 Write 9 New Done Done false
4258 1431 Acked 2430 Read 8 New Done Done false
4259 1432 Acked 2431 Read 1 New Done Done false
4260 1433 Acked 2432 Read 4 New Done Done false
4261 1434 Acked 2433 Read 10 New Done Done false
4262 1435 Acked 2434 Flush 0 New Done Done false
4263 1436 Acked 2435 Read 6 New Done Done false
4264 1437 Acked 2436 Read 4 New Done Done false
4265 1438 Acked 2437 Write 5 New Done Done false
4266 1439 Acked 2438 Write 7 New Done Done false
4267 1440 Acked 2439 Write 1 New Done Done false
4268 1441 Acked 2440 Write 5 New Done Done false
4269 1442 Acked 2441 Write 7 New Done Done false
4270 1443 Acked 2442 Write 8 New Done Done false
4271 1444 Acked 2443 Write 1 New Done Done false
4272 1445 Acked 2444 Write 10 New Done Done false
4273 1446 Acked 2445 Write 5 New Done Done false
4274 1447 Acked 2446 Read 6 New Done Done false
4275 1448 Acked 2447 Read 3 New Done Done false
4276 1449 Acked 2448 Write 9 New Done Done false
4277 1450 Acked 2449 Read 10 New Done Done false
4278 1451 Acked 2450 Write 1 New Done Done false
4279 1452 Acked 2451 Write 3 New Done Done false
4280 1453 Acked 2452 Read 9 New Done Done false
4281 1454 Acked 2453 Flush 0 New Done Done false
4282 1455 Acked 2454 Write 1 New Done Done false
4283 1456 Acked 2455 Read 8 New Done Done false
4284 1457 Acked 2456 Write 7 New Done Done false
4285 1458 Acked 2457 Write 2 New Done Done false
4286 1459 Acked 2458 Flush 0 New Done Done false
4287 1460 Acked 2459 Write 10 New Done Done false
4288 1461 Acked 2460 Read 10 New Done Done false
4289 1462 Acked 2461 Read 1 New Done Done false
4290 1463 Acked 2462 Read 4 New Done Done false
4291 1464 Acked 2463 Write 8 New Done Done false
4292 1465 Acked 2464 Flush 0 New Done Done false
4293 1466 Acked 2465 Write 6 New Done Done false
4294 1467 Acked 2466 Read 10 New Done Done false
4295 1468 Acked 2467 Read 5 New Done Done false
4296 1469 Acked 2468 Write 6 New Done Done false
4297 1470 Acked 2469 Write 1 New Done Done false
4298 1471 Acked 2470 Write 1 New Done Done false
4299 1472 Acked 2471 Read 10 New Done Done false
4300 1473 Acked 2472 Write 8 New Done Done false
4301 1474 Acked 2473 Read 6 New Done Done false
4302 1475 Acked 2474 Write 1 New Done Done false
4303 1476 Acked 2475 Flush 0 New Done Done false
4304 1477 Acked 2476 Write 7 New Done Done false
4305 1478 Acked 2477 Read 1 New Done Done false
4306 1479 Acked 2478 Read 5 New Done Done false
4307 1480 Acked 2479 Read 2 New Done Done false
4308 1481 Acked 2480 Write 8 New Done Done false
4309 1482 Acked 2481 Write 10 New Done Done false
4310 1483 Acked 2482 Write 7 New Done Done false
4311 1484 Acked 2483 Write 5 New Done Done false
4312 1485 Acked 2484 Write 5 New Done Done false
4313 1486 Acked 2485 Read 5 New Done Done false
4314 1487 Acked 2486 Write 4 New Done Done false
4315 1488 Acked 2487 Read 2 New Done Done false
4316 1489 Acked 2488 Read 7 New Done Done false
4317 1490 Acked 2489 Read 4 New Done Done false
4318 1491 Acked 2490 Write 10 New Done Done false
4319 1492 Acked 2491 Write 5 New Done Done false
4320 1493 Acked 2492 Read 1 New Done Done false
4321 1494 Acked 2493 Read 5 New Done Done false
4322 1495 Acked 2494 Read 4 New Done Done false
4323 1496 Acked 2495 Write 1 New Done Done false
4324 1497 Acked 2496 Read 6 New Done Done false
4325 1498 Acked 2497 Write 8 New Done Done false
4326 1499 Acked 2498 Write 2 New Done Done false
4327 1500 Acked 2499 Write 2 New Done Done false
4328 1501 Acked 2500 Read 1 New Done Done false
4329 1502 Acked 2501 Write 6 New Done Done false
4330 1503 Acked 2502 Write 1 New Done Done false
4331 1504 Acked 2503 Write 1 New Done Done false
4332 1505 Acked 2504 Write 10 New Done Done false
4333 1506 Acked 2505 Read 3 New Done Done false
4334 STATES DS:0 DS:1 DS:2 TOTAL
4335 New 300 0 0 300
4336 Sent 0 0 0 0
4337 Done 0 300 300 600
4338 Skipped 0 0 0 0
4339 Error 0 0 0 0
4340 Last Flush: 2205 2475 2475
4341 Downstairs last five completed: 2205 2204 2203 2202 2201
4342 Upstairs last five completed: 1506 1505 1504 1503 1502
4343 CLIENT: Up:0 ds:300 act:2
43442024-03-02T01:06:26.584ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43452024-03-02T01:06:26.584ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43462024-03-02T01:06:26.584ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43472024-03-02T01:06:26.585ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43482024-03-02T01:06:26.585ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43492024-03-02T01:06:26.585ZINFOcrucible: [0] client re-new 300 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43502024-03-02T01:06:26.585ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4351 [8810] got start action
4352 [8810] I am going to restart
4353 Starting downstairs at port 8810
4354 Make output file at "/tmp/dsc/downstairs-8810.txt"
4355 [8810][0] reports Starting
43562024-03-02T01:06:27.586ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43572024-03-02T01:06:27.586ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43582024-03-02T01:06:27.586ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43592024-03-02T01:06:27.586ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4360 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1167)
43612024-03-02T01:06:27.586ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43622024-03-02T01:06:27.586ZINFOcrucible: [0] client re-new 301 jobs since flush 2205 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4363 [8810][0] reports Running
43642024-03-02T01:06:27.586ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43652024-03-02T01:06:28.588ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43662024-03-02T01:06:28.588ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43672024-03-02T01:06:28.589ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43682024-03-02T01:06:28.589ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43692024-03-02T01:06:28.589ZINFOcrucible: send last flush ID to this DS: 2205 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43702024-03-02T01:06:28.589ZINFOcrucible: Replied this last flush ID: 2205 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43712024-03-02T01:06:28.589ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43722024-03-02T01:06:28.589ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43732024-03-02T01:06:28.589ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43742024-03-02T01:06:28.589ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4375 ----------------------------------------------------------------
4376 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4377 Downstairs last five completed: 2506 2505 2504 2503 2502
4378 Upstairs last five completed: 1507 1506 1505 1504 1503
4379 CLIENT: Up:0 ds:0 act:3
4380 Replay: All jobs finished, all DS active.
4381 Main task has work to do, go find it
4382 got dsc Stop(0)
4383 stop 0
4384 Mar 02 01:06:30.282 INFO request completed, latency_us: 342[8810] Got stop action so:false kr:false
4385 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0ad1dd69-b7da-4f39-954b-3a9bb1d75adb, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
43862024-03-02T01:06:30.283ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43872024-03-02T01:06:30.283ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43882024-03-02T01:06:30.283ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4389 Mar 02 01:06:30.283 INFO request completed{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:06:30.283359072Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
4390 , latency_us: {"msg":"[0] client re-new 0 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:06:30.28339816Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
4391 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: b2502ddc-1ee0-4b9b-a363-890ac0bc9f22, remote_addr: 127.0.0.1:40735, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:06:30.283427558Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
4392 local_addr: 127.0.0.1:9998
4393 [8810] Exited with: ExitStatus(unix_wait_status(9))
4394 [8810][0] reports Exit
43952024-03-02T01:06:31.284ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43962024-03-02T01:06:31.284ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43972024-03-02T01:06:31.284ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43982024-03-02T01:06:31.284ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
43992024-03-02T01:06:31.284ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44002024-03-02T01:06:31.284ZINFOcrucible: [0] client re-new 0 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44012024-03-02T01:06:31.284ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44022024-03-02T01:06:32.286ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44032024-03-02T01:06:32.286ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44042024-03-02T01:06:32.286ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44052024-03-02T01:06:32.286ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44062024-03-02T01:06:32.286ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44072024-03-02T01:06:32.286ZINFOcrucible: [0] client re-new 0 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44082024-03-02T01:06:32.286ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44092024-03-02T01:06:33.288ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44102024-03-02T01:06:33.288ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44112024-03-02T01:06:33.288ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44122024-03-02T01:06:33.288ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44132024-03-02T01:06:33.288ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44142024-03-02T01:06:33.288ZINFOcrucible: [0] client re-new 0 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44152024-03-02T01:06:33.288ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4416 Mar 02 01:06:34.285 INFO request completed, latency_us: 239, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1af97ab7-febb-4207-8a70-8073ea7c60e1, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
4417 001/300 Read block 4899 len 28672
4418 002/300 Write block 2414 len 4096 data: 2
4419 003/300 Read block 4053 len 36864
4420 004/300 Write block 541 len 16384 data: 2 2 2 2
44212024-03-02T01:06:34.290ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44222024-03-02T01:06:34.290ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44232024-03-02T01:06:34.290ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44242024-03-02T01:06:34.290ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44252024-03-02T01:06:34.290ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44262024-03-02T01:06:34.290ZINFOcrucible: [0] client re-new 4 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
44272024-03-02T01:06:34.290ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
4428 005/300 Write block 3279 len 28672 data: 4 3 3 3 4 4 4
4429 006/300 Write block 4950 len 8192 data: 2 2
4430 007/300 Read block 3183 len 24576
4431 008/300 Write block 828 len 20480 data: 3 3 3 3 3
4432 009/300 Read block 4733 len 36864
4433 010/300 Read block 4077 len 12288
4434 011/300 Write block 4298 len 8192 data: 4 4
4435 012/300 Write block 4531 len 36864 data: 2 2 2 2 2 2 2 2 3
4436 013/300 Read block 4460 len 20480
4437 014/300 Write block 379 len 4096 data: 2
4438 015/300 Read block 1801 len 36864
4439 016/300 Read block 1787 len 16384
4440 017/300 Write block 358 len 20480 data: 4 3 3 2 2
4441 018/300 Flush
4442 019/300 Write block 1679 len 36864 data: 2 2 3 3 3 3 3 3 3
4443 020/300 Write block 3513 len 16384 data: 3 4 4 3
4444 021/300 Read block 2253 len 20480
4445 022/300 Read block 71 len 32768
4446 023/300 Write block 2885 len 16384 data: 2 2 2 2
4447 024/300 Write block 3443 len 40960 data: 3 3 3 4 5 5 4 4 4 4
4448 025/300 Flush
4449 026/300 Read block 4816 len 4096
4450 027/300 Read block 3654 len 32768
4451 028/300 Write block 278 len 28672 data: 3 3 3 3 3 3 3
4452 029/300 Read block 1763 len 20480
4453 030/300 Write block 745 len 24576 data: 3 3 3 2 3 3
4454 031/300 Write block 1819 len 28672 data: 2 2 3 2 2 2 2
4455 032/300 Read block 4060 len 36864
4456 033/300 Write block 3495 len 36864 data: 2 2 2 2 2 2 2 2 3
4457 034/300 Flush
4458 035/300 Read block 70 len 36864
4459 036/300 Read block 1562 len 24576
4460 037/300 Flush
4461 038/300 Read block 2984 len 40960
4462 039/300 Read block 2250 len 16384
4463 040/300 Read block 2614 len 32768
4464 041/300 Write block 2238 len 32768 data: 2 2 2 2 2 2 3 3
4465 042/300 Read block 4809 len 40960
4466 043/300 Write block 3390 len 36864 data: 2 2 2 2 2 2 2 2 3
4467 044/300 Write block 2140 len 4096 data: 2
4468 045/300 Read block 3293 len 40960
4469 046/300 Write block 2617 len 16384 data: 2 2 2 2
4470 047/300 Read block 4229 len 32768
4471 048/300 Read block 4982 len 12288
4472 049/300 Write block 137 len 16384 data: 2 2 2 2
4473 050/300 Read block 4252 len 28672
4474 051/300 Flush
4475 052/300 Read block 99 len 28672
4476 053/300 Read block 2908 len 12288
4477 054/300 Read block 860 len 36864
4478 055/300 Write block 1553 len 24576 data: 3 3 3 3 3 2
4479 056/300 Write block 4420 len 20480 data: 2 2 2 2 2
4480 057/300 Read block 3246 len 32768
4481 058/300 Write block 3117 len 20480 data: 3 3 3 2 2
4482 059/300 Read block 3648 len 40960
4483 060/300 Write block 2950 len 32768 data: 2 2 2 2 3 4 4 4
4484 061/300 Read block 3241 len 40960
4485 062/300 Flush
4486 063/300 Flush
4487 064/300 Flush
4488 065/300 Read block 1903 len 24576
4489 066/300 Read block 404 len 32768
4490 067/300 Read block 3056 len 12288
4491 068/300 Write block 1300 len 12288 data: 4 4 3
4492 069/300 Write block 2712 len 28672 data: 2 2 2 2 4 4 4
4493 070/300 Write block 887 len 40960 data: 2 2 2 2 2 3 3 3 4 4
4494 071/300 Read block 416 len 32768
4495 072/300 Read block 2259 len 16384
4496 073/300 Read block 3282 len 40960
4497 074/300 Read block 3589 len 4096
4498 075/300 Flush
4499 076/300 Write block 3453 len 24576 data: 3 3 3 2 2 2
4500 077/300 Read block 4034 len 32768
4501 078/300 Read block 192 len 36864
4502 079/300 Read block 2765 len 12288
4503 080/300 Write block 3030 len 40960 data: 2 2 2 2 2 2 2 2 3 2
4504 081/300 Read block 730 len 32768
4505 082/300 Write block 2079 len 16384 data: 2 2 2 3
4506 083/300 Write block 290 len 8192 data: 3 3
4507 084/300 Write block 4740 len 28672 data: 3 3 3 3 3 3 2
4508 085/300 Read block 3874 len 40960
4509 086/300 Read block 4351 len 36864
4510 087/300 Read block 3919 len 24576
4511 088/300 Write block 3909 len 32768 data: 3 3 3 3 3 3 3 2
4512 089/300 Write block 1871 len 32768 data: 3 3 3 2 2 2 2 2
4513 090/300 Read block 4389 len 32768
4514 091/300 Read block 720 len 24576
4515 092/300 Read block 117 len 16384
4516 093/300 Read block 4248 len 4096
4517 094/300 Write block 3647 len 36864 data: 2 2 2 3 3 3 3 3 3
4518 095/300 Write block 1906 len 20480 data: 3 2 2 2 2
4519 096/300 Read block 4404 len 8192
4520 097/300 Read block 3216 len 4096
4521 098/300 Write block 1572 len 36864 data: 2 2 2 2 2 3 2 2 3
4522 099/300 Read block 551 len 40960
4523 100/300 Write block 1812 len 20480 data: 2 2 2 3 2
4524 101/300 Read block 1754 len 16384
4525 102/300 Read block 3015 len 28672
4526 103/300 Read block 2643 len 8192
4527 104/300 Flush
4528 105/300 Write block 3588 len 40960 data: 2 2 2 2 3 3 3 3 3 4
4529 106/300 Write block 743 len 8192 data: 3 3
4530 107/300 Read block 3235 len 16384
4531 108/300 Write block 4379 len 12288 data: 2 2 2
4532 109/300 Flush
4533 110/300 Read block 530 len 40960
4534 111/300 Read block 4145 len 20480
4535 112/300 Read block 3561 len 28672
4536 113/300 Read block 2799 len 20480
4537 114/300 Read block 579 len 8192
4538 115/300 Write block 1158 len 16384 data: 3 2 2 2
4539 116/300 Read block 3307 len 32768
4540 117/300 Read block 69 len 36864
4541 118/300 Flush
4542 119/300 Read block 2314 len 36864
4543 120/300 Read block 4268 len 24576
4544 121/300 Read block 315 len 28672
4545 122/300 Write block 936 len 8192 data: 4 3
4546 123/300 Write block 1447 len 20480 data: 3 3 2 2 2
4547 124/300 Read block 454 len 32768
4548 125/300 Write block 2007 len 36864 data: 2 2 2 2 2 2 2 2 2
4549 126/300 Read block 3145 len 32768
4550 127/300 Write block 4390 len 32768 data: 2 2 2 2 2 2 4 3
4551 128/300 Read block 1517 len 8192
4552 129/300 Read block 3158 len 40960
4553 130/300 Read block 4092 len 4096
4554 131/300 Write block 4089 len 40960 data: 2 2 2 3 3 2 2 2 2 3
4555 132/300 Write block 4013 len 28672 data: 2 2 2 2 2 2 2
4556 133/300 Read block 2615 len 16384
4557 134/300 Read block 1066 len 40960
4558 135/300 Read block 4219 len 16384
4559 136/300 Write block 2806 len 36864 data: 2 2 3 4 5 5 4 4 4
4560 137/300 Write block 4938 len 20480 data: 3 3 2 2 2
4561 138/300 Write block 4764 len 20480 data: 3 3 3 4 3
4562 139/300 Write block 601 len 24576 data: 2 2 2 2 2 2
4563 140/300 Write block 4643 len 28672 data: 5 6 5 6 6 5 4
4564 141/300 Read block 398 len 8192
4565 142/300 Flush
4566 143/300 Write block 495 len 4096 data: 2
4567 144/300 Write block 124 len 16384 data: 4 4 4 4
4568 145/300 Read block 1329 len 12288
4569 146/300 Write block 1121 len 4096 data: 2
4570 147/300 Write block 3485 len 4096 data: 4
4571 148/300 Read block 1745 len 8192
4572 149/300 Write block 4124 len 4096 data: 4
4573 150/300 Read block 2947 len 16384
4574 151/300 Flush
4575 152/300 Write block 3134 len 32768 data: 2 2 2 2 2 2 2 2
4576 153/300 Read block 842 len 24576
4577 154/300 Read block 4473 len 16384
4578 155/300 Read block 3187 len 36864
4579 156/300 Read block 314 len 32768
4580 157/300 Read block 3743 len 28672
4581 158/300 Write block 609 len 32768 data: 2 3 3 3 3 4 4 3
4582 159/300 Read block 301 len 36864
4583 160/300 Write block 2826 len 36864 data: 3 3 4 4 4 5 4 3 4
4584 161/300 Write block 3522 len 28672 data: 3 3 3 2 2 2 3
4585 162/300 Read block 264 len 4096
4586 163/300 Read block 3834 len 36864
4587 164/300 Read block 4082 len 36864
4588 165/300 Write block 2093 len 20480 data: 3 3 3 3 2
4589 166/300 Read block 1166 len 24576
4590 167/300 Read block 1926 len 12288
4591 168/300 Read block 4788 len 4096
4592 169/300 Write block 1193 len 20480 data: 3 3 3 3 3
4593 170/300 Read block 2461 len 28672
4594 171/300 Read block 758 len 12288
4595 172/300 Read block 4951 len 32768
4596 173/300 Read block 3991 len 36864
4597 174/300 Write block 3910 len 40960 data: 4 4 4 4 4 4 3 2 3 3
4598 175/300 Flush
4599 176/300 Read block 4033 len 32768
4600 177/300 Read block 1631 len 12288
4601 178/300 Write block 3722 len 4096 data: 2
4602 179/300 Write block 4186 len 28672 data: 4 4 3 3 3 3 2
4603 180/300 Read block 3633 len 20480
4604 181/300 Read block 4346 len 8192
4605 182/300 Write block 807 len 32768 data: 2 2 2 3 3 3 3 4
4606 183/300 Write block 149 len 32768 data: 2 2 2 2 2 2 2 2
4607 184/300 Flush
4608 185/300 Read block 2837 len 40960
4609 186/300 Flush
4610 187/300 Read block 3187 len 32768
4611 188/300 Read block 2344 len 36864
4612 189/300 Write block 4711 len 20480 data: 2 2 3 4 4
4613 190/300 Flush
4614 191/300 Write block 3431 len 16384 data: 2 2 2 2
4615 192/300 Read block 3221 len 40960
4616 193/300 Read block 1133 len 20480
4617 194/300 Flush
4618 195/300 Read block 1355 len 4096
4619 196/300 Write block 1443 len 16384 data: 3 3 3 3
4620 197/300 Write block 48 len 24576 data: 2 3 3 3 3 3
4621 198/300 Read block 1353 len 4096
4622 199/300 Write block 3174 len 36864 data: 3 2 2 3 3 3 3 3 3
4623 200/300 Write block 2282 len 4096 data: 2
4624 201/300 Write block 1744 len 28672 data: 2 3 3 3 3 2 2
4625 202/300 Read block 803 len 28672
4626 203/300 Write block 1058 len 12288 data: 2 2 2
4627 204/300 Write block 159 len 24576 data: 3 3 3 3 3 3
4628 205/300 Read block 3381 len 32768
4629 206/300 Flush
4630 207/300 Read block 2861 len 4096
4631 208/300 Flush
4632 209/300 Read block 170 len 40960
4633 210/300 Read block 3479 len 36864
4634 211/300 Write block 2746 len 40960 data: 3 3 3 3 2 2 2 2 3 3
4635 212/300 Read block 1097 len 24576
4636 213/300 Write block 1082 len 32768 data: 2 3 3 4 4 4 4 2
4637 214/300 Write block 2946 len 32768 data: 2 2 2 2 3 3 3 3
4638 215/300 Read block 3209 len 20480
4639 216/300 Write block 1060 len 28672 data: 3 2 2 3 3 2 2
4640 217/300 Write block 1429 len 16384 data: 2 3 3 2
4641 218/300 Write block 2999 len 8192 data: 2 2
4642 219/300 Read block 1291 len 32768
4643 220/300 Read block 2038 len 40960
4644 221/300 Read block 2627 len 8192
4645 222/300 Read block 1225 len 24576
4646 223/300 Write block 321 len 16384 data: 2 2 2 2
4647 224/300 Read block 262 len 32768
4648 225/300 Read block 4732 len 12288
4649 226/300 Read block 76 len 16384
4650 227/300 Read block 3104 len 12288
4651 228/300 Write block 1666 len 8192 data: 3 3
4652 229/300 Write block 1747 len 40960 data: 4 4 3 3 2 2 2 2 3 2
4653 230/300 Read block 616 len 4096
4654 231/300 Read block 4574 len 28672
4655 232/300 Flush
4656 233/300 Write block 2388 len 24576 data: 2 2 2 2 2 2
4657 234/300 Write block 4190 len 4096 data: 4
4658 235/300 Read block 4556 len 36864
4659 236/300 Read block 658 len 8192
4660 237/300 Write block 4756 len 16384 data: 2 2 2 3
4661 238/300 Read block 1836 len 36864
4662 239/300 Read block 3361 len 32768
4663 240/300 Flush
4664 241/300 Write block 4028 len 24576 data: 4 4 3 3 3 4
4665 242/300 Read block 3783 len 40960
4666 243/300 Flush
4667 244/300 Read block 837 len 24576
4668 245/300 Read block 4782 len 4096
4669 246/300 Read block 1095 len 4096
4670 247/300 Write block 1948 len 28672 data: 3 3 3 4 4 3 3
4671 248/300 Read block 3360 len 16384
4672 249/300 Read block 2777 len 8192
4673 250/300 Write block 937 len 36864 data: 4 2 2 2 2 2 3 3 3
4674 251/300 Write block 4210 len 36864 data: 2 2 2 2 2 2 2 3 3
4675 252/300 Read block 894 len 8192
4676 253/300 Read block 1315 len 4096
4677 254/300 Read block 1487 len 20480
4678 255/300 Read block 468 len 4096
4679 256/300 Read block 469 len 28672
4680 257/300 Read block 1846 len 20480
4681 258/300 Flush
4682 259/300 Write block 1946 len 20480 data: 3 3 4 4 4
4683 260/300 Read block 3057 len 28672
4684 261/300 Write block 4367 len 36864 data: 2 2 2 2 2 2 2 2 2
4685 262/300 Read block 4843 len 12288
4686 263/300 Read block 268 len 20480
4687 264/300 Write block 4620 len 8192 data: 3 3
4688 265/300 Write block 3923 len 20480 data: 3 3 3 3 3
4689 266/300 Read block 1819 len 32768
4690 267/300 Read block 1817 len 16384
4691 268/300 Read block 2744 len 36864
4692 269/300 Write block 3227 len 36864 data: 3 2 2 2 2 2 2 3 3
4693 270/300 Read block 3827 len 4096
4694 271/300 Read block 3714 len 36864
4695 272/300 Flush
4696 273/300 Read block 2497 len 8192
4697 274/300 Write block 1251 len 16384 data: 3 2 2 2
4698 275/300 Read block 541 len 16384
4699 276/300 Read block 3511 len 24576
4700 277/300 Write block 4579 len 28672 data: 3 3 2 2 4 4 4
4701 278/300 Read block 3374 len 16384
4702 279/300 Read block 1346 len 16384
4703 280/300 Write block 3713 len 24576 data: 3 3 2 2 2 2
4704 281/300 Read block 945 len 36864
4705 282/300 Write block 570 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4706 283/300 Read block 1449 len 28672
4707 284/300 Flush
4708 285/300 Write block 1358 len 28672 data: 2 2 2 2 2 2 2
4709 286/300 Read block 3514 len 24576
4710 287/300 Read block 1046 len 8192
4711 288/300 Read block 3708 len 8192
4712 289/300 Write block 3370 len 16384 data: 2 2 2 2
4713 290/300 Read block 4748 len 4096
4714 291/300 Read block 4717 len 16384
4715 292/300 Write block 3157 len 20480 data: 3 3 3 3 3
4716 293/300 Read block 1711 len 16384
4717 294/300 Read block 3111 len 40960
4718 295/300 Write block 322 len 40960 data: 3 3 3 2 2 2 2 2 2 2
4719 296/300 Write block 266 len 24576 data: 3 4 3 3 3 3
4720 297/300 Flush
4721 298/300 Write block 3750 len 36864 data: 2 2 2 2 2 4 3 3 3
4722 299/300 Write block 4723 len 8192 data: 2 2
4723 300/300 Write block 1051 len 24576 data: 2 2 2 2 2 2
4724 Main task has work to do, go find it
4725 got dsc Start(0)
4726 start 0
4727 Mar 02 01:06:35.017 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: f6e28ae9-ff65-47a3-95d6-4903952e69e3, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
4728 Replay: started 0, returned:Ok(())
4729 ----------------------------------------------------------------
4730 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
4731 States: Offline Active Active
4732 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4733 1508 Acked 2507 Read 7 New Done Done true
4734 1509 Acked 2508 Write 1 New Done Done true
4735 1510 Acked 2509 Read 9 New Done Done true
4736 1511 Acked 2510 Write 4 New Done Done true
4737 1512 Acked 2511 Write 7 New Done Done false
4738 1513 Acked 2512 Write 2 New Done Done false
4739 1514 Acked 2513 Read 6 New Done Done false
4740 1515 Acked 2514 Write 5 New Done Done false
4741 1516 Acked 2515 Read 9 New Done Done false
4742 1517 Acked 2516 Read 3 New Done Done false
4743 1518 Acked 2517 Write 2 New Done Done false
4744 1519 Acked 2518 Write 9 New Done Done false
4745 1520 Acked 2519 Read 5 New Done Done false
4746 1521 Acked 2520 Write 1 New Done Done false
4747 1522 Acked 2521 Read 9 New Done Done false
4748 1523 Acked 2522 Read 4 New Done Done false
4749 1524 Acked 2523 Write 5 New Done Done false
4750 1525 Acked 2524 Flush 0 New Done Done false
4751 1526 Acked 2525 Write 9 New Done Done false
4752 1527 Acked 2526 Write 4 New Done Done false
4753 1528 Acked 2527 Read 5 New Done Done false
4754 1529 Acked 2528 Read 8 New Done Done false
4755 1530 Acked 2529 Write 4 New Done Done false
4756 1531 Acked 2530 Write 10 New Done Done false
4757 1532 Acked 2531 Flush 0 New Done Done false
4758 1533 Acked 2532 Read 1 New Done Done false
4759 1534 Acked 2533 Read 8 New Done Done false
4760 1535 Acked 2534 Write 7 New Done Done false
4761 1536 Acked 2535 Read 5 New Done Done false
4762 1537 Acked 2536 Write 6 New Done Done false
4763 1538 Acked 2537 Write 7 New Done Done false
4764 1539 Acked 2538 Read 9 New Done Done false
4765 1540 Acked 2539 Write 9 New Done Done false
4766 1541 Acked 2540 Flush 0 New Done Done false
4767 1542 Acked 2541 Read 9 New Done Done false
4768 1543 Acked 2542 Read 6 New Done Done false
4769 1544 Acked 2543 Flush 0 New Done Done false
4770 1545 Acked 2544 Read 10 New Done Done false
4771 1546 Acked 2545 Read 4 New Done Done false
4772 1547 Acked 2546 Read 8 New Done Done false
4773 1548 Acked 2547 Write 8 New Done Done false
4774 1549 Acked 2548 Read 10 New Done Done false
4775 1550 Acked 2549 Write 9 New Done Done false
4776 1551 Acked 2550 Write 1 New Done Done false
4777 1552 Acked 2551 Read 10 New Done Done false
4778 1553 Acked 2552 Write 4 New Done Done false
4779 1554 Acked 2553 Read 8 New Done Done false
4780 1555 Acked 2554 Read 3 New Done Done false
4781 1556 Acked 2555 Write 4 New Done Done false
4782 1557 Acked 2556 Read 7 New Done Done false
4783 1558 Acked 2557 Flush 0 New Done Done false
4784 1559 Acked 2558 Read 7 New Done Done false
4785 1560 Acked 2559 Read 3 New Done Done false
4786 1561 Acked 2560 Read 9 New Done Done false
4787 1562 Acked 2561 Write 6 New Done Done false
4788 1563 Acked 2562 Write 5 New Done Done false
4789 1564 Acked 2563 Read 8 New Done Done false
4790 1565 Acked 2564 Write 5 New Done Done false
4791 1566 Acked 2565 Read 10 New Done Done false
4792 1567 Acked 2566 Write 8 New Done Done false
4793 1568 Acked 2567 Read 10 New Done Done false
4794 1569 Acked 2568 Flush 0 New Done Done false
4795 1570 Acked 2569 Flush 0 New Done Done false
4796 1571 Acked 2570 Flush 0 New Done Done false
4797 1572 Acked 2571 Read 6 New Done Done false
4798 1573 Acked 2572 Read 8 New Done Done false
4799 1574 Acked 2573 Read 3 New Done Done false
4800 1575 Acked 2574 Write 3 New Done Done false
4801 1576 Acked 2575 Write 7 New Done Done false
4802 1577 Acked 2576 Write 10 New Done Done false
4803 1578 Acked 2577 Read 8 New Done Done false
4804 1579 Acked 2578 Read 4 New Done Done false
4805 1580 Acked 2579 Read 10 New Done Done false
4806 1581 Acked 2580 Read 1 New Done Done false
4807 1582 Acked 2581 Flush 0 New Done Done false
4808 1583 Acked 2582 Write 6 New Done Done false
4809 1584 Acked 2583 Read 8 New Done Done false
4810 1585 Acked 2584 Read 9 New Done Done false
4811 1586 Acked 2585 Read 3 New Done Done false
4812 1587 Acked 2586 Write 10 New Done Done false
4813 1588 Acked 2587 Read 8 New Done Done false
4814 1589 Acked 2588 Write 4 New Done Done false
4815 1590 Acked 2589 Write 2 New Done Done false
4816 1591 Acked 2590 Write 7 New Done Done false
4817 1592 Acked 2591 Read 10 New Done Done false
4818 1593 Acked 2592 Read 9 New Done Done false
4819 1594 Acked 2593 Read 6 New Done Done false
4820 1595 Acked 2594 Write 8 New Done Done false
4821 1596 Acked 2595 Write 8 New Done Done false
4822 1597 Acked 2596 Read 8 New Done Done false
4823 1598 Acked 2597 Read 6 New Done Done false
4824 1599 Acked 2598 Read 4 New Done Done false
4825 1600 Acked 2599 Read 1 New Done Done false
4826 1601 Acked 2600 Write 9 New Done Done false
4827 1602 Acked 2601 Write 5 New Done Done false
4828 1603 Acked 2602 Read 2 New Done Done false
4829 1604 Acked 2603 Read 1 New Done Done false
4830 1605 Acked 2604 Write 9 New Done Done false
4831 1606 Acked 2605 Read 10 New Done Done false
4832 1607 Acked 2606 Write 5 New Done Done false
4833 1608 Acked 2607 Read 4 New Done Done false
4834 1609 Acked 2608 Read 7 New Done Done false
4835 1610 Acked 2609 Read 2 New Done Done false
4836 1611 Acked 2610 Flush 0 New Done Done false
4837 1612 Acked 2611 Write 10 New Done Done false
4838 1613 Acked 2612 Write 2 New Done Done false
4839 1614 Acked 2613 Read 4 New Done Done false
4840 1615 Acked 2614 Write 3 New Done Done false
4841 1616 Acked 2615 Flush 0 New Done Done false
4842 1617 Acked 2616 Read 10 New Done Done false
4843 1618 Acked 2617 Read 5 New Done Done false
4844 1619 Acked 2618 Read 7 New Done Done false
4845 1620 Acked 2619 Read 5 New Done Done false
4846 1621 Acked 2620 Read 2 New Done Done false
4847 1622 Acked 2621 Write 4 New Done Done false
4848 1623 Acked 2622 Read 8 New Done Done false
4849 1624 Acked 2623 Read 9 New Done Done false
4850 1625 Acked 2624 Flush 0 New Done Done false
4851 1626 Acked 2625 Read 9 New Done Done false
4852 1627 Acked 2626 Read 6 New Done Done false
4853 1628 Acked 2627 Read 7 New Done Done false
4854 1629 Acked 2628 Write 2 New Done Done false
4855 1630 Acked 2629 Write 5 New Done Done false
4856 1631 Acked 2630 Read 8 New Done Done false
4857 1632 Acked 2631 Write 9 New Done Done false
4858 1633 Acked 2632 Read 8 New Done Done false
4859 1634 Acked 2633 Write 8 New Done Done false
4860 1635 Acked 2634 Read 2 New Done Done false
4861 1636 Acked 2635 Read 10 New Done Done false
4862 1637 Acked 2636 Read 1 New Done Done false
4863 1638 Acked 2637 Write 10 New Done Done false
4864 1639 Acked 2638 Write 7 New Done Done false
4865 1640 Acked 2639 Read 4 New Done Done false
4866 1641 Acked 2640 Read 10 New Done Done false
4867 1642 Acked 2641 Read 4 New Done Done false
4868 1643 Acked 2642 Write 9 New Done Done false
4869 1644 Acked 2643 Write 5 New Done Done false
4870 1645 Acked 2644 Write 5 New Done Done false
4871 1646 Acked 2645 Write 6 New Done Done false
4872 1647 Acked 2646 Write 7 New Done Done false
4873 1648 Acked 2647 Read 2 New Done Done false
4874 1649 Acked 2648 Flush 0 New Done Done false
4875 1650 Acked 2649 Write 1 New Done Done false
4876 1651 Acked 2650 Write 4 New Done Done false
4877 1652 Acked 2651 Read 3 New Done Done false
4878 1653 Acked 2652 Write 1 New Done Done false
4879 1654 Acked 2653 Write 1 New Done Done false
4880 1655 Acked 2654 Read 2 New Done Done false
4881 1656 Acked 2655 Write 1 New Done Done false
4882 1657 Acked 2656 Read 4 New Done Done false
4883 1658 Acked 2657 Flush 0 New Done Done false
4884 1659 Acked 2658 Write 8 New Done Done false
4885 1660 Acked 2659 Read 6 New Done Done false
4886 1661 Acked 2660 Read 4 New Done Done false
4887 1662 Acked 2661 Read 9 New Done Done false
4888 1663 Acked 2662 Read 8 New Done Done false
4889 1664 Acked 2663 Read 7 New Done Done false
4890 1665 Acked 2664 Write 8 New Done Done false
4891 1666 Acked 2665 Read 9 New Done Done false
4892 1667 Acked 2666 Write 9 New Done Done false
4893 1668 Acked 2667 Write 7 New Done Done false
4894 1669 Acked 2668 Read 1 New Done Done false
4895 1670 Acked 2669 Read 9 New Done Done false
4896 1671 Acked 2670 Read 9 New Done Done false
4897 1672 Acked 2671 Write 5 New Done Done false
4898 1673 Acked 2672 Read 6 New Done Done false
4899 1674 Acked 2673 Read 3 New Done Done false
4900 1675 Acked 2674 Read 1 New Done Done false
4901 1676 Acked 2675 Write 5 New Done Done false
4902 1677 Acked 2676 Read 7 New Done Done false
4903 1678 Acked 2677 Read 3 New Done Done false
4904 1679 Acked 2678 Read 8 New Done Done false
4905 1680 Acked 2679 Read 9 New Done Done false
4906 1681 Acked 2680 Write 10 New Done Done false
4907 1682 Acked 2681 Flush 0 New Done Done false
4908 1683 Acked 2682 Read 8 New Done Done false
4909 1684 Acked 2683 Read 3 New Done Done false
4910 1685 Acked 2684 Write 1 New Done Done false
4911 1686 Acked 2685 Write 7 New Done Done false
4912 1687 Acked 2686 Read 5 New Done Done false
4913 1688 Acked 2687 Read 2 New Done Done false
4914 1689 Acked 2688 Write 8 New Done Done false
4915 1690 Acked 2689 Write 8 New Done Done false
4916 1691 Acked 2690 Flush 0 New Done Done false
4917 1692 Acked 2691 Read 10 New Done Done false
4918 1693 Acked 2692 Flush 0 New Done Done false
4919 1694 Acked 2693 Read 8 New Done Done false
4920 1695 Acked 2694 Read 9 New Done Done false
4921 1696 Acked 2695 Write 5 New Done Done false
4922 1697 Acked 2696 Flush 0 New Done Done false
4923 1698 Acked 2697 Write 4 New Done Done false
4924 1699 Acked 2698 Read 10 New Done Done false
4925 1700 Acked 2699 Read 5 New Done Done false
4926 1701 Acked 2700 Flush 0 New Done Done false
4927 1702 Acked 2701 Read 1 New Done Done false
4928 1703 Acked 2702 Write 4 New Done Done false
4929 1704 Acked 2703 Write 6 New Done Done false
4930 1705 Acked 2704 Read 1 New Done Done false
4931 1706 Acked 2705 Write 9 New Done Done false
4932 1707 Acked 2706 Write 1 New Done Done false
4933 1708 Acked 2707 Write 7 New Done Done false
4934 1709 Acked 2708 Read 7 New Done Done false
4935 1710 Acked 2709 Write 3 New Done Done false
4936 1711 Acked 2710 Write 6 New Done Done false
4937 1712 Acked 2711 Read 8 New Done Done false
4938 1713 Acked 2712 Flush 0 New Done Done false
4939 1714 Acked 2713 Read 1 New Done Done false
4940 1715 Acked 2714 Flush 0 New Done Done false
4941 1716 Acked 2715 Read 10 New Done Done false
4942 1717 Acked 2716 Read 9 New Done Done false
4943 1718 Acked 2717 Write 10 New Done Done false
4944 1719 Acked 2718 Read 6 New Done Done false
4945 1720 Acked 2719 Write 8 New Done Done false
4946 1721 Acked 2720 Write 8 New Done Done false
4947 1722 Acked 2721 Read 5 New Done Done false
4948 1723 Acked 2722 Write 7 New Done Done false
4949 1724 Acked 2723 Write 4 New Done Done false
4950 1725 Acked 2724 Write 2 New Done Done false
4951 1726 Acked 2725 Read 8 New Done Done false
4952 1727 Acked 2726 Read 10 New Done Done false
4953 1728 Acked 2727 Read 2 New Done Done false
4954 1729 Acked 2728 Read 6 New Done Done false
4955 1730 Acked 2729 Write 4 New Done Done false
4956 1731 Acked 2730 Read 8 New Done Done false
4957 1732 Acked 2731 Read 3 New Done Done false
4958 1733 Acked 2732 Read 4 New Done Done false
4959 1734 Acked 2733 Read 3 New Done Done false
4960 1735 Acked 2734 Write 2 New Done Done false
4961 1736 Acked 2735 Write 10 New Done Done false
4962 1737 Acked 2736 Read 1 New Done Done false
4963 1738 Acked 2737 Read 7 New Done Done false
4964 1739 Acked 2738 Flush 0 New Done Done false
4965 1740 Acked 2739 Write 6 New Done Done false
4966 1741 Acked 2740 Write 1 New Done Done false
4967 1742 Acked 2741 Read 9 New Done Done false
4968 1743 Acked 2742 Read 2 New Done Done false
4969 1744 Acked 2743 Write 4 New Done Done false
4970 1745 Acked 2744 Read 9 New Done Done false
4971 1746 Acked 2745 Read 8 New Done Done false
4972 1747 Acked 2746 Flush 0 New Done Done false
4973 1748 Acked 2747 Write 6 New Done Done false
4974 1749 Acked 2748 Read 10 New Done Done false
4975 1750 Acked 2749 Flush 0 New Done Done false
4976 1751 Acked 2750 Read 6 New Done Done false
4977 1752 Acked 2751 Read 1 New Done Done false
4978 1753 Acked 2752 Read 1 New Done Done false
4979 1754 Acked 2753 Write 7 New Done Done false
4980 1755 Acked 2754 Read 4 New Done Done false
4981 1756 Acked 2755 Read 2 New Done Done false
4982 1757 Acked 2756 Write 9 New Done Done false
4983 1758 Acked 2757 Write 9 New Done Done false
4984 1759 Acked 2758 Read 2 New Done Done false
4985 1760 Acked 2759 Read 1 New Done Done false
4986 1761 Acked 2760 Read 5 New Done Done false
4987 1762 Acked 2761 Read 1 New Done Done false
4988 1763 Acked 2762 Read 7 New Done Done false
4989 1764 Acked 2763 Read 5 New Done Done false
4990 1765 Acked 2764 Flush 0 New Done Done false
4991 1766 Acked 2765 Write 5 New Done Done false
4992 1767 Acked 2766 Read 7 New Done Done false
4993 1768 Acked 2767 Write 9 New Done Done false
4994 1769 Acked 2768 Read 3 New Done Done false
4995 1770 Acked 2769 Read 5 New Done Done false
4996 1771 Acked 2770 Write 2 New Done Done false
4997 1772 Acked 2771 Write 5 New Done Done false
4998 1773 Acked 2772 Read 8 New Done Done false
4999 1774 Acked 2773 Read 4 New Done Done false
5000 1775 Acked 2774 Read 9 New Done Done false
5001 1776 Acked 2775 Write 9 New Done Done false
5002 1777 Acked 2776 Read 1 New Done Done false
5003 1778 Acked 2777 Read 9 New Done Done false
5004 1779 Acked 2778 Flush 0 New Done Done false
5005 1780 Acked 2779 Read 2 New Done Done false
5006 1781 Acked 2780 Write 4 New Done Done false
5007 1782 Acked 2781 Read 4 New Done Done false
5008 1783 Acked 2782 Read 6 New Done Done false
5009 1784 Acked 2783 Write 7 New Done Done false
5010 1785 Acked 2784 Read 4 New Done Done false
5011 1786 Acked 2785 Read 4 New Done Done false
5012 1787 Acked 2786 Write 6 New Done Done false
5013 1788 Acked 2787 Read 9 New Done Done false
5014 1789 Acked 2788 Write 10 New Done Done false
5015 1790 Acked 2789 Read 7 New Done Done false
5016 1791 Acked 2790 Flush 0 New Done Done false
5017 1792 Acked 2791 Write 7 New Done Done false
5018 1793 Acked 2792 Read 6 New Done Done false
5019 1794 Acked 2793 Read 2 New Done Done false
5020 1795 Acked 2794 Read 2 New Done Done false
5021 1796 Acked 2795 Write 4 New Done Done false
5022 1797 Acked 2796 Read 1 New Done Done false
5023 1798 Acked 2797 Read 4 New Done Done false
5024 1799 Acked 2798 Write 5 New Done Done false
5025 1800 Acked 2799 Read 4 New Done Done false
5026 1801 Acked 2800 Read 10 New Done Done false
5027 1802 Acked 2801 Write 10 New Done Done false
5028 1803 Acked 2802 Write 6 New Done Done false
5029 1804 Acked 2803 Flush 0 New Done Done false
5030 1805 Acked 2804 Write 9 New Done Done false
5031 1806 Acked 2805 Write 2 New Done Done false
5032 1807 Acked 2806 Write 6 New Done Done false
5033 STATES DS:0 DS:1 DS:2 TOTAL
5034 New 300 0 0 300
5035 Sent 0 0 0 0
5036 Done 0 300 300 600
5037 Skipped 0 0 0 0
5038 Error 0 0 0 0
5039 Last Flush: 2506 2803 2803
5040 Downstairs last five completed: 2506 2505 2504 2503 2502
5041 Upstairs last five completed: 1807 1806 1805 1804 1803
5042 CLIENT: Up:0 ds:300 act:2
50432024-03-02T01:06:35.292ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50442024-03-02T01:06:35.292ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50452024-03-02T01:06:35.292ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50462024-03-02T01:06:35.292ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50472024-03-02T01:06:35.292ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50482024-03-02T01:06:35.292ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50492024-03-02T01:06:35.292ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50502024-03-02T01:06:36.293ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50512024-03-02T01:06:36.293ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50522024-03-02T01:06:36.293ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50532024-03-02T01:06:36.293ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50542024-03-02T01:06:36.293ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50552024-03-02T01:06:36.293ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50562024-03-02T01:06:36.294ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50572024-03-02T01:06:37.295ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50582024-03-02T01:06:37.295ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50592024-03-02T01:06:37.295ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50602024-03-02T01:06:37.295ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50612024-03-02T01:06:37.295ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50622024-03-02T01:06:37.295ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50632024-03-02T01:06:37.295ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50642024-03-02T01:06:38.297ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50652024-03-02T01:06:38.297ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50662024-03-02T01:06:38.297ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50672024-03-02T01:06:38.297ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50682024-03-02T01:06:38.297ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50692024-03-02T01:06:38.297ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
50702024-03-02T01:06:38.298ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5071 ----------------------------------------------------------------
5072 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5073 States: Offline Active Active
5074 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5075 1508 Acked 2507 Read 7 New Done Done true
5076 1509 Acked 2508 Write 1 New Done Done true
5077 1510 Acked 2509 Read 9 New Done Done true
5078 1511 Acked 2510 Write 4 New Done Done true
5079 1512 Acked 2511 Write 7 New Done Done true
5080 1513 Acked 2512 Write 2 New Done Done true
5081 1514 Acked 2513 Read 6 New Done Done true
5082 1515 Acked 2514 Write 5 New Done Done true
5083 1516 Acked 2515 Read 9 New Done Done true
5084 1517 Acked 2516 Read 3 New Done Done true
5085 1518 Acked 2517 Write 2 New Done Done true
5086 1519 Acked 2518 Write 9 New Done Done true
5087 1520 Acked 2519 Read 5 New Done Done true
5088 1521 Acked 2520 Write 1 New Done Done true
5089 1522 Acked 2521 Read 9 New Done Done true
5090 1523 Acked 2522 Read 4 New Done Done true
5091 1524 Acked 2523 Write 5 New Done Done true
5092 1525 Acked 2524 Flush 0 New Done Done true
5093 1526 Acked 2525 Write 9 New Done Done true
5094 1527 Acked 2526 Write 4 New Done Done true
5095 1528 Acked 2527 Read 5 New Done Done true
5096 1529 Acked 2528 Read 8 New Done Done true
5097 1530 Acked 2529 Write 4 New Done Done true
5098 1531 Acked 2530 Write 10 New Done Done true
5099 1532 Acked 2531 Flush 0 New Done Done true
5100 1533 Acked 2532 Read 1 New Done Done true
5101 1534 Acked 2533 Read 8 New Done Done true
5102 1535 Acked 2534 Write 7 New Done Done true
5103 1536 Acked 2535 Read 5 New Done Done true
5104 1537 Acked 2536 Write 6 New Done Done true
5105 1538 Acked 2537 Write 7 New Done Done true
5106 1539 Acked 2538 Read 9 New Done Done true
5107 1540 Acked 2539 Write 9 New Done Done true
5108 1541 Acked 2540 Flush 0 New Done Done true
5109 1542 Acked 2541 Read 9 New Done Done true
5110 1543 Acked 2542 Read 6 New Done Done true
5111 1544 Acked 2543 Flush 0 New Done Done true
5112 1545 Acked 2544 Read 10 New Done Done true
5113 1546 Acked 2545 Read 4 New Done Done true
5114 1547 Acked 2546 Read 8 New Done Done true
5115 1548 Acked 2547 Write 8 New Done Done true
5116 1549 Acked 2548 Read 10 New Done Done true
5117 1550 Acked 2549 Write 9 New Done Done true
5118 1551 Acked 2550 Write 1 New Done Done true
5119 1552 Acked 2551 Read 10 New Done Done true
5120 1553 Acked 2552 Write 4 New Done Done true
5121 1554 Acked 2553 Read 8 New Done Done true
5122 1555 Acked 2554 Read 3 New Done Done true
5123 1556 Acked 2555 Write 4 New Done Done true
5124 1557 Acked 2556 Read 7 New Done Done true
5125 1558 Acked 2557 Flush 0 New Done Done true
5126 1559 Acked 2558 Read 7 New Done Done true
5127 1560 Acked 2559 Read 3 New Done Done true
5128 1561 Acked 2560 Read 9 New Done Done true
5129 1562 Acked 2561 Write 6 New Done Done true
5130 1563 Acked 2562 Write 5 New Done Done true
5131 1564 Acked 2563 Read 8 New Done Done true
5132 1565 Acked 2564 Write 5 New Done Done true
5133 1566 Acked 2565 Read 10 New Done Done true
5134 1567 Acked 2566 Write 8 New Done Done true
5135 1568 Acked 2567 Read 10 New Done Done true
5136 1569 Acked 2568 Flush 0 New Done Done true
5137 1570 Acked 2569 Flush 0 New Done Done true
5138 1571 Acked 2570 Flush 0 New Done Done true
5139 1572 Acked 2571 Read 6 New Done Done true
5140 1573 Acked 2572 Read 8 New Done Done true
5141 1574 Acked 2573 Read 3 New Done Done true
5142 1575 Acked 2574 Write 3 New Done Done true
5143 1576 Acked 2575 Write 7 New Done Done true
5144 1577 Acked 2576 Write 10 New Done Done true
5145 1578 Acked 2577 Read 8 New Done Done true
5146 1579 Acked 2578 Read 4 New Done Done true
5147 1580 Acked 2579 Read 10 New Done Done true
5148 1581 Acked 2580 Read 1 New Done Done true
5149 1582 Acked 2581 Flush 0 New Done Done true
5150 1583 Acked 2582 Write 6 New Done Done true
5151 1584 Acked 2583 Read 8 New Done Done true
5152 1585 Acked 2584 Read 9 New Done Done true
5153 1586 Acked 2585 Read 3 New Done Done true
5154 1587 Acked 2586 Write 10 New Done Done true
5155 1588 Acked 2587 Read 8 New Done Done true
5156 1589 Acked 2588 Write 4 New Done Done true
5157 1590 Acked 2589 Write 2 New Done Done true
5158 1591 Acked 2590 Write 7 New Done Done true
5159 1592 Acked 2591 Read 10 New Done Done true
5160 1593 Acked 2592 Read 9 New Done Done true
5161 1594 Acked 2593 Read 6 New Done Done true
5162 1595 Acked 2594 Write 8 New Done Done true
5163 1596 Acked 2595 Write 8 New Done Done true
5164 1597 Acked 2596 Read 8 New Done Done true
5165 1598 Acked 2597 Read 6 New Done Done true
5166 1599 Acked 2598 Read 4 New Done Done true
5167 1600 Acked 2599 Read 1 New Done Done true
5168 1601 Acked 2600 Write 9 New Done Done true
5169 1602 Acked 2601 Write 5 New Done Done true
5170 1603 Acked 2602 Read 2 New Done Done true
5171 1604 Acked 2603 Read 1 New Done Done true
5172 1605 Acked 2604 Write 9 New Done Done true
5173 1606 Acked 2605 Read 10 New Done Done true
5174 1607 Acked 2606 Write 5 New Done Done true
5175 1608 Acked 2607 Read 4 New Done Done true
5176 1609 Acked 2608 Read 7 New Done Done true
5177 1610 Acked 2609 Read 2 New Done Done true
5178 1611 Acked 2610 Flush 0 New Done Done true
5179 1612 Acked 2611 Write 10 New Done Done true
5180 1613 Acked 2612 Write 2 New Done Done true
5181 1614 Acked 2613 Read 4 New Done Done true
5182 1615 Acked 2614 Write 3 New Done Done true
5183 1616 Acked 2615 Flush 0 New Done Done true
5184 1617 Acked 2616 Read 10 New Done Done true
5185 1618 Acked 2617 Read 5 New Done Done true
5186 1619 Acked 2618 Read 7 New Done Done true
5187 1620 Acked 2619 Read 5 New Done Done true
5188 1621 Acked 2620 Read 2 New Done Done true
5189 1622 Acked 2621 Write 4 New Done Done true
5190 1623 Acked 2622 Read 8 New Done Done true
5191 1624 Acked 2623 Read 9 New Done Done true
5192 1625 Acked 2624 Flush 0 New Done Done true
5193 1626 Acked 2625 Read 9 New Done Done true
5194 1627 Acked 2626 Read 6 New Done Done true
5195 1628 Acked 2627 Read 7 New Done Done true
5196 1629 Acked 2628 Write 2 New Done Done true
5197 1630 Acked 2629 Write 5 New Done Done true
5198 1631 Acked 2630 Read 8 New Done Done true
5199 1632 Acked 2631 Write 9 New Done Done true
5200 1633 Acked 2632 Read 8 New Done Done true
5201 1634 Acked 2633 Write 8 New Done Done true
5202 1635 Acked 2634 Read 2 New Done Done true
5203 1636 Acked 2635 Read 10 New Done Done true
5204 1637 Acked 2636 Read 1 New Done Done true
5205 1638 Acked 2637 Write 10 New Done Done true
5206 1639 Acked 2638 Write 7 New Done Done true
5207 1640 Acked 2639 Read 4 New Done Done true
5208 1641 Acked 2640 Read 10 New Done Done true
5209 1642 Acked 2641 Read 4 New Done Done true
5210 1643 Acked 2642 Write 9 New Done Done true
5211 1644 Acked 2643 Write 5 New Done Done true
5212 1645 Acked 2644 Write 5 New Done Done true
5213 1646 Acked 2645 Write 6 New Done Done true
5214 1647 Acked 2646 Write 7 New Done Done true
5215 1648 Acked 2647 Read 2 New Done Done true
5216 1649 Acked 2648 Flush 0 New Done Done true
5217 1650 Acked 2649 Write 1 New Done Done true
5218 1651 Acked 2650 Write 4 New Done Done true
5219 1652 Acked 2651 Read 3 New Done Done true
5220 1653 Acked 2652 Write 1 New Done Done true
5221 1654 Acked 2653 Write 1 New Done Done true
5222 1655 Acked 2654 Read 2 New Done Done true
5223 1656 Acked 2655 Write 1 New Done Done true
5224 1657 Acked 2656 Read 4 New Done Done true
5225 1658 Acked 2657 Flush 0 New Done Done true
5226 1659 Acked 2658 Write 8 New Done Done true
5227 1660 Acked 2659 Read 6 New Done Done true
5228 1661 Acked 2660 Read 4 New Done Done true
5229 1662 Acked 2661 Read 9 New Done Done true
5230 1663 Acked 2662 Read 8 New Done Done true
5231 1664 Acked 2663 Read 7 New Done Done true
5232 1665 Acked 2664 Write 8 New Done Done true
5233 1666 Acked 2665 Read 9 New Done Done true
5234 1667 Acked 2666 Write 9 New Done Done true
5235 1668 Acked 2667 Write 7 New Done Done true
5236 1669 Acked 2668 Read 1 New Done Done true
5237 1670 Acked 2669 Read 9 New Done Done true
5238 1671 Acked 2670 Read 9 New Done Done true
5239 1672 Acked 2671 Write 5 New Done Done true
5240 1673 Acked 2672 Read 6 New Done Done true
5241 1674 Acked 2673 Read 3 New Done Done true
5242 1675 Acked 2674 Read 1 New Done Done true
5243 1676 Acked 2675 Write 5 New Done Done true
5244 1677 Acked 2676 Read 7 New Done Done true
5245 1678 Acked 2677 Read 3 New Done Done true
5246 1679 Acked 2678 Read 8 New Done Done true
5247 1680 Acked 2679 Read 9 New Done Done true
5248 1681 Acked 2680 Write 10 New Done Done true
5249 1682 Acked 2681 Flush 0 New Done Done true
5250 1683 Acked 2682 Read 8 New Done Done true
5251 1684 Acked 2683 Read 3 New Done Done true
5252 1685 Acked 2684 Write 1 New Done Done true
5253 1686 Acked 2685 Write 7 New Done Done true
5254 1687 Acked 2686 Read 5 New Done Done true
5255 1688 Acked 2687 Read 2 New Done Done true
5256 1689 Acked 2688 Write 8 New Done Done true
5257 1690 Acked 2689 Write 8 New Done Done true
5258 1691 Acked 2690 Flush 0 New Done Done true
5259 1692 Acked 2691 Read 10 New Done Done true
5260 1693 Acked 2692 Flush 0 New Done Done true
5261 1694 Acked 2693 Read 8 New Done Done true
5262 1695 Acked 2694 Read 9 New Done Done true
5263 1696 Acked 2695 Write 5 New Done Done true
5264 1697 Acked 2696 Flush 0 New Done Done true
5265 1698 Acked 2697 Write 4 New Done Done true
5266 1699 Acked 2698 Read 10 New Done Done true
5267 1700 Acked 2699 Read 5 New Done Done true
5268 1701 Acked 2700 Flush 0 New Done Done true
5269 1702 Acked 2701 Read 1 New Done Done true
5270 1703 Acked 2702 Write 4 New Done Done true
5271 1704 Acked 2703 Write 6 New Done Done true
5272 1705 Acked 2704 Read 1 New Done Done true
5273 1706 Acked 2705 Write 9 New Done Done true
5274 1707 Acked 2706 Write 1 New Done Done true
5275 1708 Acked 2707 Write 7 New Done Done true
5276 1709 Acked 2708 Read 7 New Done Done true
5277 1710 Acked 2709 Write 3 New Done Done true
5278 1711 Acked 2710 Write 6 New Done Done true
5279 1712 Acked 2711 Read 8 New Done Done true
5280 1713 Acked 2712 Flush 0 New Done Done true
5281 1714 Acked 2713 Read 1 New Done Done true
5282 1715 Acked 2714 Flush 0 New Done Done true
5283 1716 Acked 2715 Read 10 New Done Done true
5284 1717 Acked 2716 Read 9 New Done Done true
5285 1718 Acked 2717 Write 10 New Done Done true
5286 1719 Acked 2718 Read 6 New Done Done true
5287 1720 Acked 2719 Write 8 New Done Done true
5288 1721 Acked 2720 Write 8 New Done Done true
5289 1722 Acked 2721 Read 5 New Done Done true
5290 1723 Acked 2722 Write 7 New Done Done true
5291 1724 Acked 2723 Write 4 New Done Done true
5292 1725 Acked 2724 Write 2 New Done Done true
5293 1726 Acked 2725 Read 8 New Done Done true
5294 1727 Acked 2726 Read 10 New Done Done true
5295 1728 Acked 2727 Read 2 New Done Done true
5296 1729 Acked 2728 Read 6 New Done Done true
5297 1730 Acked 2729 Write 4 New Done Done true
5298 1731 Acked 2730 Read 8 New Done Done true
5299 1732 Acked 2731 Read 3 New Done Done true
5300 1733 Acked 2732 Read 4 New Done Done true
5301 1734 Acked 2733 Read 3 New Done Done true
5302 1735 Acked 2734 Write 2 New Done Done true
5303 1736 Acked 2735 Write 10 New Done Done true
5304 1737 Acked 2736 Read 1 New Done Done true
5305 1738 Acked 2737 Read 7 New Done Done true
5306 1739 Acked 2738 Flush 0 New Done Done true
5307 1740 Acked 2739 Write 6 New Done Done true
5308 1741 Acked 2740 Write 1 New Done Done true
5309 1742 Acked 2741 Read 9 New Done Done true
5310 1743 Acked 2742 Read 2 New Done Done true
5311 1744 Acked 2743 Write 4 New Done Done true
5312 1745 Acked 2744 Read 9 New Done Done true
5313 1746 Acked 2745 Read 8 New Done Done true
5314 1747 Acked 2746 Flush 0 New Done Done true
5315 1748 Acked 2747 Write 6 New Done Done true
5316 1749 Acked 2748 Read 10 New Done Done true
5317 1750 Acked 2749 Flush 0 New Done Done true
5318 1751 Acked 2750 Read 6 New Done Done true
5319 1752 Acked 2751 Read 1 New Done Done true
5320 1753 Acked 2752 Read 1 New Done Done true
5321 1754 Acked 2753 Write 7 New Done Done true
5322 1755 Acked 2754 Read 4 New Done Done true
5323 1756 Acked 2755 Read 2 New Done Done true
5324 1757 Acked 2756 Write 9 New Done Done true
5325 1758 Acked 2757 Write 9 New Done Done true
5326 1759 Acked 2758 Read 2 New Done Done true
5327 1760 Acked 2759 Read 1 New Done Done true
5328 1761 Acked 2760 Read 5 New Done Done true
5329 1762 Acked 2761 Read 1 New Done Done true
5330 1763 Acked 2762 Read 7 New Done Done true
5331 1764 Acked 2763 Read 5 New Done Done true
5332 1765 Acked 2764 Flush 0 New Done Done true
5333 1766 Acked 2765 Write 5 New Done Done true
5334 1767 Acked 2766 Read 7 New Done Done true
5335 1768 Acked 2767 Write 9 New Done Done true
5336 1769 Acked 2768 Read 3 New Done Done true
5337 1770 Acked 2769 Read 5 New Done Done true
5338 1771 Acked 2770 Write 2 New Done Done true
5339 1772 Acked 2771 Write 5 New Done Done true
5340 1773 Acked 2772 Read 8 New Done Done true
5341 1774 Acked 2773 Read 4 New Done Done true
5342 1775 Acked 2774 Read 9 New Done Done true
5343 1776 Acked 2775 Write 9 New Done Done true
5344 1777 Acked 2776 Read 1 New Done Done true
5345 1778 Acked 2777 Read 9 New Done Done true
5346 1779 Acked 2778 Flush 0 New Done Done true
5347 1780 Acked 2779 Read 2 New Done Done true
5348 1781 Acked 2780 Write 4 New Done Done true
5349 1782 Acked 2781 Read 4 New Done Done true
5350 1783 Acked 2782 Read 6 New Done Done true
5351 1784 Acked 2783 Write 7 New Done Done true
5352 1785 Acked 2784 Read 4 New Done Done true
5353 1786 Acked 2785 Read 4 New Done Done true
5354 1787 Acked 2786 Write 6 New Done Done true
5355 1788 Acked 2787 Read 9 New Done Done true
5356 1789 Acked 2788 Write 10 New Done Done true
5357 1790 Acked 2789 Read 7 New Done Done true
5358 1791 Acked 2790 Flush 0 New Done Done true
5359 1792 Acked 2791 Write 7 New Done Done true
5360 1793 Acked 2792 Read 6 New Done Done true
5361 1794 Acked 2793 Read 2 New Done Done true
5362 1795 Acked 2794 Read 2 New Done Done true
5363 1796 Acked 2795 Write 4 New Done Done true
5364 1797 Acked 2796 Read 1 New Done Done true
5365 1798 Acked 2797 Read 4 New Done Done true
5366 1799 Acked 2798 Write 5 New Done Done true
5367 1800 Acked 2799 Read 4 New Done Done true
5368 1801 Acked 2800 Read 10 New Done Done true
5369 1802 Acked 2801 Write 10 New Done Done true
5370 1803 Acked 2802 Write 6 New Done Done true
5371 1804 Acked 2803 Flush 0 New Done Done true
5372 1805 Acked 2804 Write 9 New Done Done true
5373 1806 Acked 2805 Write 2 New Done Done true
5374 1807 Acked 2806 Write 6 New Done Done true
5375 1808 Acked 2807 Flush 0 New Done Done true
5376 STATES DS:0 DS:1 DS:2 TOTAL
5377 New 301 0 0 301
5378 Sent 0 0 0 0
5379 Done 0 301 301 602
5380 Skipped 0 0 0 0
5381 Error 0 0 0 0
5382 Last Flush: 2506 2807 2807
5383 Downstairs last five completed: 2506 2505 2504 2503 2502
5384 Upstairs last five completed: 1808 1807 1806 1805 1804
5385 CLIENT: Up:0 ds:301 act:2
53862024-03-02T01:06:39.298ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53872024-03-02T01:06:39.298ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53882024-03-02T01:06:39.298ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53892024-03-02T01:06:39.298ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53902024-03-02T01:06:39.298ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53912024-03-02T01:06:39.298ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53922024-03-02T01:06:39.298ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53932024-03-02T01:06:40.300ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53942024-03-02T01:06:40.300ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53952024-03-02T01:06:40.300ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53962024-03-02T01:06:40.300ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53972024-03-02T01:06:40.300ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53982024-03-02T01:06:40.300ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
53992024-03-02T01:06:40.300ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54002024-03-02T01:06:41.302ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54012024-03-02T01:06:41.302ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54022024-03-02T01:06:41.302ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54032024-03-02T01:06:41.302ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54042024-03-02T01:06:41.302ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54052024-03-02T01:06:41.302ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54062024-03-02T01:06:41.302ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54072024-03-02T01:06:42.303ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54082024-03-02T01:06:42.303ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54092024-03-02T01:06:42.303ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54102024-03-02T01:06:42.303ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54112024-03-02T01:06:42.303ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54122024-03-02T01:06:42.303ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
54132024-03-02T01:06:42.303ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5414 ----------------------------------------------------------------
5415 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5416 States: Offline Active Active
5417 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5418 1508 Acked 2507 Read 7 New Done Done true
5419 1509 Acked 2508 Write 1 New Done Done true
5420 1510 Acked 2509 Read 9 New Done Done true
5421 1511 Acked 2510 Write 4 New Done Done true
5422 1512 Acked 2511 Write 7 New Done Done true
5423 1513 Acked 2512 Write 2 New Done Done true
5424 1514 Acked 2513 Read 6 New Done Done true
5425 1515 Acked 2514 Write 5 New Done Done true
5426 1516 Acked 2515 Read 9 New Done Done true
5427 1517 Acked 2516 Read 3 New Done Done true
5428 1518 Acked 2517 Write 2 New Done Done true
5429 1519 Acked 2518 Write 9 New Done Done true
5430 1520 Acked 2519 Read 5 New Done Done true
5431 1521 Acked 2520 Write 1 New Done Done true
5432 1522 Acked 2521 Read 9 New Done Done true
5433 1523 Acked 2522 Read 4 New Done Done true
5434 1524 Acked 2523 Write 5 New Done Done true
5435 1525 Acked 2524 Flush 0 New Done Done true
5436 1526 Acked 2525 Write 9 New Done Done true
5437 1527 Acked 2526 Write 4 New Done Done true
5438 1528 Acked 2527 Read 5 New Done Done true
5439 1529 Acked 2528 Read 8 New Done Done true
5440 1530 Acked 2529 Write 4 New Done Done true
5441 1531 Acked 2530 Write 10 New Done Done true
5442 1532 Acked 2531 Flush 0 New Done Done true
5443 1533 Acked 2532 Read 1 New Done Done true
5444 1534 Acked 2533 Read 8 New Done Done true
5445 1535 Acked 2534 Write 7 New Done Done true
5446 1536 Acked 2535 Read 5 New Done Done true
5447 1537 Acked 2536 Write 6 New Done Done true
5448 1538 Acked 2537 Write 7 New Done Done true
5449 1539 Acked 2538 Read 9 New Done Done true
5450 1540 Acked 2539 Write 9 New Done Done true
5451 1541 Acked 2540 Flush 0 New Done Done true
5452 1542 Acked 2541 Read 9 New Done Done true
5453 1543 Acked 2542 Read 6 New Done Done true
5454 1544 Acked 2543 Flush 0 New Done Done true
5455 1545 Acked 2544 Read 10 New Done Done true
5456 1546 Acked 2545 Read 4 New Done Done true
5457 1547 Acked 2546 Read 8 New Done Done true
5458 1548 Acked 2547 Write 8 New Done Done true
5459 1549 Acked 2548 Read 10 New Done Done true
5460 1550 Acked 2549 Write 9 New Done Done true
5461 1551 Acked 2550 Write 1 New Done Done true
5462 1552 Acked 2551 Read 10 New Done Done true
5463 1553 Acked 2552 Write 4 New Done Done true
5464 1554 Acked 2553 Read 8 New Done Done true
5465 1555 Acked 2554 Read 3 New Done Done true
5466 1556 Acked 2555 Write 4 New Done Done true
5467 1557 Acked 2556 Read 7 New Done Done true
5468 1558 Acked 2557 Flush 0 New Done Done true
5469 1559 Acked 2558 Read 7 New Done Done true
5470 1560 Acked 2559 Read 3 New Done Done true
5471 1561 Acked 2560 Read 9 New Done Done true
5472 1562 Acked 2561 Write 6 New Done Done true
5473 1563 Acked 2562 Write 5 New Done Done true
5474 1564 Acked 2563 Read 8 New Done Done true
5475 1565 Acked 2564 Write 5 New Done Done true
5476 1566 Acked 2565 Read 10 New Done Done true
5477 1567 Acked 2566 Write 8 New Done Done true
5478 1568 Acked 2567 Read 10 New Done Done true
5479 1569 Acked 2568 Flush 0 New Done Done true
5480 1570 Acked 2569 Flush 0 New Done Done true
5481 1571 Acked 2570 Flush 0 New Done Done true
5482 1572 Acked 2571 Read 6 New Done Done true
5483 1573 Acked 2572 Read 8 New Done Done true
5484 1574 Acked 2573 Read 3 New Done Done true
5485 1575 Acked 2574 Write 3 New Done Done true
5486 1576 Acked 2575 Write 7 New Done Done true
5487 1577 Acked 2576 Write 10 New Done Done true
5488 1578 Acked 2577 Read 8 New Done Done true
5489 1579 Acked 2578 Read 4 New Done Done true
5490 1580 Acked 2579 Read 10 New Done Done true
5491 1581 Acked 2580 Read 1 New Done Done true
5492 1582 Acked 2581 Flush 0 New Done Done true
5493 1583 Acked 2582 Write 6 New Done Done true
5494 1584 Acked 2583 Read 8 New Done Done true
5495 1585 Acked 2584 Read 9 New Done Done true
5496 1586 Acked 2585 Read 3 New Done Done true
5497 1587 Acked 2586 Write 10 New Done Done true
5498 1588 Acked 2587 Read 8 New Done Done true
5499 1589 Acked 2588 Write 4 New Done Done true
5500 1590 Acked 2589 Write 2 New Done Done true
5501 1591 Acked 2590 Write 7 New Done Done true
5502 1592 Acked 2591 Read 10 New Done Done true
5503 1593 Acked 2592 Read 9 New Done Done true
5504 1594 Acked 2593 Read 6 New Done Done true
5505 1595 Acked 2594 Write 8 New Done Done true
5506 1596 Acked 2595 Write 8 New Done Done true
5507 1597 Acked 2596 Read 8 New Done Done true
5508 1598 Acked 2597 Read 6 New Done Done true
5509 1599 Acked 2598 Read 4 New Done Done true
5510 1600 Acked 2599 Read 1 New Done Done true
5511 1601 Acked 2600 Write 9 New Done Done true
5512 1602 Acked 2601 Write 5 New Done Done true
5513 1603 Acked 2602 Read 2 New Done Done true
5514 1604 Acked 2603 Read 1 New Done Done true
5515 1605 Acked 2604 Write 9 New Done Done true
5516 1606 Acked 2605 Read 10 New Done Done true
5517 1607 Acked 2606 Write 5 New Done Done true
5518 1608 Acked 2607 Read 4 New Done Done true
5519 1609 Acked 2608 Read 7 New Done Done true
5520 1610 Acked 2609 Read 2 New Done Done true
5521 1611 Acked 2610 Flush 0 New Done Done true
5522 1612 Acked 2611 Write 10 New Done Done true
5523 1613 Acked 2612 Write 2 New Done Done true
5524 1614 Acked 2613 Read 4 New Done Done true
5525 1615 Acked 2614 Write 3 New Done Done true
5526 1616 Acked 2615 Flush 0 New Done Done true
5527 1617 Acked 2616 Read 10 New Done Done true
5528 1618 Acked 2617 Read 5 New Done Done true
5529 1619 Acked 2618 Read 7 New Done Done true
5530 1620 Acked 2619 Read 5 New Done Done true
5531 1621 Acked 2620 Read 2 New Done Done true
5532 1622 Acked 2621 Write 4 New Done Done true
5533 1623 Acked 2622 Read 8 New Done Done true
5534 1624 Acked 2623 Read 9 New Done Done true
5535 1625 Acked 2624 Flush 0 New Done Done true
5536 1626 Acked 2625 Read 9 New Done Done true
5537 1627 Acked 2626 Read 6 New Done Done true
5538 1628 Acked 2627 Read 7 New Done Done true
5539 1629 Acked 2628 Write 2 New Done Done true
5540 1630 Acked 2629 Write 5 New Done Done true
5541 1631 Acked 2630 Read 8 New Done Done true
5542 1632 Acked 2631 Write 9 New Done Done true
5543 1633 Acked 2632 Read 8 New Done Done true
5544 1634 Acked 2633 Write 8 New Done Done true
5545 1635 Acked 2634 Read 2 New Done Done true
5546 1636 Acked 2635 Read 10 New Done Done true
5547 1637 Acked 2636 Read 1 New Done Done true
5548 1638 Acked 2637 Write 10 New Done Done true
5549 1639 Acked 2638 Write 7 New Done Done true
5550 1640 Acked 2639 Read 4 New Done Done true
5551 1641 Acked 2640 Read 10 New Done Done true
5552 1642 Acked 2641 Read 4 New Done Done true
5553 1643 Acked 2642 Write 9 New Done Done true
5554 1644 Acked 2643 Write 5 New Done Done true
5555 1645 Acked 2644 Write 5 New Done Done true
5556 1646 Acked 2645 Write 6 New Done Done true
5557 1647 Acked 2646 Write 7 New Done Done true
5558 1648 Acked 2647 Read 2 New Done Done true
5559 1649 Acked 2648 Flush 0 New Done Done true
5560 1650 Acked 2649 Write 1 New Done Done true
5561 1651 Acked 2650 Write 4 New Done Done true
5562 1652 Acked 2651 Read 3 New Done Done true
5563 1653 Acked 2652 Write 1 New Done Done true
5564 1654 Acked 2653 Write 1 New Done Done true
5565 1655 Acked 2654 Read 2 New Done Done true
5566 1656 Acked 2655 Write 1 New Done Done true
5567 1657 Acked 2656 Read 4 New Done Done true
5568 1658 Acked 2657 Flush 0 New Done Done true
5569 1659 Acked 2658 Write 8 New Done Done true
5570 1660 Acked 2659 Read 6 New Done Done true
5571 1661 Acked 2660 Read 4 New Done Done true
5572 1662 Acked 2661 Read 9 New Done Done true
5573 1663 Acked 2662 Read 8 New Done Done true
5574 1664 Acked 2663 Read 7 New Done Done true
5575 1665 Acked 2664 Write 8 New Done Done true
5576 1666 Acked 2665 Read 9 New Done Done true
5577 1667 Acked 2666 Write 9 New Done Done true
5578 1668 Acked 2667 Write 7 New Done Done true
5579 1669 Acked 2668 Read 1 New Done Done true
5580 1670 Acked 2669 Read 9 New Done Done true
5581 1671 Acked 2670 Read 9 New Done Done true
5582 1672 Acked 2671 Write 5 New Done Done true
5583 1673 Acked 2672 Read 6 New Done Done true
5584 1674 Acked 2673 Read 3 New Done Done true
5585 1675 Acked 2674 Read 1 New Done Done true
5586 1676 Acked 2675 Write 5 New Done Done true
5587 1677 Acked 2676 Read 7 New Done Done true
5588 1678 Acked 2677 Read 3 New Done Done true
5589 1679 Acked 2678 Read 8 New Done Done true
5590 1680 Acked 2679 Read 9 New Done Done true
5591 1681 Acked 2680 Write 10 New Done Done true
5592 1682 Acked 2681 Flush 0 New Done Done true
5593 1683 Acked 2682 Read 8 New Done Done true
5594 1684 Acked 2683 Read 3 New Done Done true
5595 1685 Acked 2684 Write 1 New Done Done true
5596 1686 Acked 2685 Write 7 New Done Done true
5597 1687 Acked 2686 Read 5 New Done Done true
5598 1688 Acked 2687 Read 2 New Done Done true
5599 1689 Acked 2688 Write 8 New Done Done true
5600 1690 Acked 2689 Write 8 New Done Done true
5601 1691 Acked 2690 Flush 0 New Done Done true
5602 1692 Acked 2691 Read 10 New Done Done true
5603 1693 Acked 2692 Flush 0 New Done Done true
5604 1694 Acked 2693 Read 8 New Done Done true
5605 1695 Acked 2694 Read 9 New Done Done true
5606 1696 Acked 2695 Write 5 New Done Done true
5607 1697 Acked 2696 Flush 0 New Done Done true
5608 1698 Acked 2697 Write 4 New Done Done true
5609 1699 Acked 2698 Read 10 New Done Done true
5610 1700 Acked 2699 Read 5 New Done Done true
5611 1701 Acked 2700 Flush 0 New Done Done true
5612 1702 Acked 2701 Read 1 New Done Done true
5613 1703 Acked 2702 Write 4 New Done Done true
5614 1704 Acked 2703 Write 6 New Done Done true
5615 1705 Acked 2704 Read 1 New Done Done true
5616 1706 Acked 2705 Write 9 New Done Done true
5617 1707 Acked 2706 Write 1 New Done Done true
5618 1708 Acked 2707 Write 7 New Done Done true
5619 1709 Acked 2708 Read 7 New Done Done true
5620 1710 Acked 2709 Write 3 New Done Done true
5621 1711 Acked 2710 Write 6 New Done Done true
5622 1712 Acked 2711 Read 8 New Done Done true
5623 1713 Acked 2712 Flush 0 New Done Done true
5624 1714 Acked 2713 Read 1 New Done Done true
5625 1715 Acked 2714 Flush 0 New Done Done true
5626 1716 Acked 2715 Read 10 New Done Done true
5627 1717 Acked 2716 Read 9 New Done Done true
5628 1718 Acked 2717 Write 10 New Done Done true
5629 1719 Acked 2718 Read 6 New Done Done true
5630 1720 Acked 2719 Write 8 New Done Done true
5631 1721 Acked 2720 Write 8 New Done Done true
5632 1722 Acked 2721 Read 5 New Done Done true
5633 1723 Acked 2722 Write 7 New Done Done true
5634 1724 Acked 2723 Write 4 New Done Done true
5635 1725 Acked 2724 Write 2 New Done Done true
5636 1726 Acked 2725 Read 8 New Done Done true
5637 1727 Acked 2726 Read 10 New Done Done true
5638 1728 Acked 2727 Read 2 New Done Done true
5639 1729 Acked 2728 Read 6 New Done Done true
5640 1730 Acked 2729 Write 4 New Done Done true
5641 1731 Acked 2730 Read 8 New Done Done true
5642 1732 Acked 2731 Read 3 New Done Done true
5643 1733 Acked 2732 Read 4 New Done Done true
5644 1734 Acked 2733 Read 3 New Done Done true
5645 1735 Acked 2734 Write 2 New Done Done true
5646 1736 Acked 2735 Write 10 New Done Done true
5647 1737 Acked 2736 Read 1 New Done Done true
5648 1738 Acked 2737 Read 7 New Done Done true
5649 1739 Acked 2738 Flush 0 New Done Done true
5650 1740 Acked 2739 Write 6 New Done Done true
5651 1741 Acked 2740 Write 1 New Done Done true
5652 1742 Acked 2741 Read 9 New Done Done true
5653 1743 Acked 2742 Read 2 New Done Done true
5654 1744 Acked 2743 Write 4 New Done Done true
5655 1745 Acked 2744 Read 9 New Done Done true
5656 1746 Acked 2745 Read 8 New Done Done true
5657 1747 Acked 2746 Flush 0 New Done Done true
5658 1748 Acked 2747 Write 6 New Done Done true
5659 1749 Acked 2748 Read 10 New Done Done true
5660 1750 Acked 2749 Flush 0 New Done Done true
5661 1751 Acked 2750 Read 6 New Done Done true
5662 1752 Acked 2751 Read 1 New Done Done true
5663 1753 Acked 2752 Read 1 New Done Done true
5664 1754 Acked 2753 Write 7 New Done Done true
5665 1755 Acked 2754 Read 4 New Done Done true
5666 1756 Acked 2755 Read 2 New Done Done true
5667 1757 Acked 2756 Write 9 New Done Done true
5668 1758 Acked 2757 Write 9 New Done Done true
5669 1759 Acked 2758 Read 2 New Done Done true
5670 1760 Acked 2759 Read 1 New Done Done true
5671 1761 Acked 2760 Read 5 New Done Done true
5672 1762 Acked 2761 Read 1 New Done Done true
5673 1763 Acked 2762 Read 7 New Done Done true
5674 1764 Acked 2763 Read 5 New Done Done true
5675 1765 Acked 2764 Flush 0 New Done Done true
5676 1766 Acked 2765 Write 5 New Done Done true
5677 1767 Acked 2766 Read 7 New Done Done true
5678 1768 Acked 2767 Write 9 New Done Done true
5679 1769 Acked 2768 Read 3 New Done Done true
5680 1770 Acked 2769 Read 5 New Done Done true
5681 1771 Acked 2770 Write 2 New Done Done true
5682 1772 Acked 2771 Write 5 New Done Done true
5683 1773 Acked 2772 Read 8 New Done Done true
5684 1774 Acked 2773 Read 4 New Done Done true
5685 1775 Acked 2774 Read 9 New Done Done true
5686 1776 Acked 2775 Write 9 New Done Done true
5687 1777 Acked 2776 Read 1 New Done Done true
5688 1778 Acked 2777 Read 9 New Done Done true
5689 1779 Acked 2778 Flush 0 New Done Done true
5690 1780 Acked 2779 Read 2 New Done Done true
5691 1781 Acked 2780 Write 4 New Done Done true
5692 1782 Acked 2781 Read 4 New Done Done true
5693 1783 Acked 2782 Read 6 New Done Done true
5694 1784 Acked 2783 Write 7 New Done Done true
5695 1785 Acked 2784 Read 4 New Done Done true
5696 1786 Acked 2785 Read 4 New Done Done true
5697 1787 Acked 2786 Write 6 New Done Done true
5698 1788 Acked 2787 Read 9 New Done Done true
5699 1789 Acked 2788 Write 10 New Done Done true
5700 1790 Acked 2789 Read 7 New Done Done true
5701 1791 Acked 2790 Flush 0 New Done Done true
5702 1792 Acked 2791 Write 7 New Done Done true
5703 1793 Acked 2792 Read 6 New Done Done true
5704 1794 Acked 2793 Read 2 New Done Done true
5705 1795 Acked 2794 Read 2 New Done Done true
5706 1796 Acked 2795 Write 4 New Done Done true
5707 1797 Acked 2796 Read 1 New Done Done true
5708 1798 Acked 2797 Read 4 New Done Done true
5709 1799 Acked 2798 Write 5 New Done Done true
5710 1800 Acked 2799 Read 4 New Done Done true
5711 1801 Acked 2800 Read 10 New Done Done true
5712 1802 Acked 2801 Write 10 New Done Done true
5713 1803 Acked 2802 Write 6 New Done Done true
5714 1804 Acked 2803 Flush 0 New Done Done true
5715 1805 Acked 2804 Write 9 New Done Done true
5716 1806 Acked 2805 Write 2 New Done Done true
5717 1807 Acked 2806 Write 6 New Done Done true
5718 1808 Acked 2807 Flush 0 New Done Done true
5719 STATES DS:0 DS:1 DS:2 TOTAL
5720 New 301 0 0 301
5721 Sent 0 0 0 0
5722 Done 0 301 301 602
5723 Skipped 0 0 0 0
5724 Error 0 0 0 0
5725 Last Flush: 2506 2807 2807
5726 Downstairs last five completed: 2506 2505 2504 2503 2502
5727 Upstairs last five completed: 1808 1807 1806 1805 1804
5728 CLIENT: Up:0 ds:301 act:2
57292024-03-02T01:06:43.305ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57302024-03-02T01:06:43.305ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57312024-03-02T01:06:43.305ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57322024-03-02T01:06:43.305ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57332024-03-02T01:06:43.305ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57342024-03-02T01:06:43.305ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57352024-03-02T01:06:43.305ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5736 [8810] got start action
5737 [8810] I am going to restart
5738 Starting downstairs at port 8810
5739 Make output file at "/tmp/dsc/downstairs-8810.txt"
5740 [8810][0] reports Starting
5741 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1168)
5742 [8810][0] reports Running
57432024-03-02T01:06:44.307ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57442024-03-02T01:06:44.307ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57452024-03-02T01:06:44.307ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57462024-03-02T01:06:44.307ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57472024-03-02T01:06:44.307ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57482024-03-02T01:06:44.307ZINFOcrucible: [0] client re-new 301 jobs since flush 2506 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57492024-03-02T01:06:44.307ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57502024-03-02T01:06:45.307ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57512024-03-02T01:06:45.307ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57522024-03-02T01:06:45.309ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57532024-03-02T01:06:45.309ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57542024-03-02T01:06:45.309ZINFOcrucible: send last flush ID to this DS: 2506 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57552024-03-02T01:06:45.309ZINFOcrucible: Replied this last flush ID: 2506 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57562024-03-02T01:06:45.309ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57572024-03-02T01:06:45.309ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57582024-03-02T01:06:45.309ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57592024-03-02T01:06:45.309ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5760 ----------------------------------------------------------------
5761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5762 Downstairs last five completed: 2807 2806 2805 2804 2803
5763 Upstairs last five completed: 1808 1807 1806 1805 1804
5764 CLIENT: Up:0 ds:0 act:3
5765 Replay: All jobs finished, all DS active.
5766 Main task has work to do, go find it
5767 got dsc Stop(1)
5768 stop 1
5769 Mar 02 01:06:47.025 INFO request completed, latency_us: 362, response_code: [8820] Got stop action so:false kr:false
5770 204, uri: /stop/cid/1, method: POST, req_id: 76fa145d-e647-471b-a9c3-dc17a0d2eb4a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
57712024-03-02T01:06:47.026ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57722024-03-02T01:06:47.026ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57732024-03-02T01:06:47.026ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57742024-03-02T01:06:47.026ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57752024-03-02T01:06:47.026ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57762024-03-02T01:06:47.026ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5777 Mar 02 01:06:47.026 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/1, method: GET, req_id: 75bada01-d559-4fff-95f7-0dea1786e89e, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
5778 [8820] Exited with: ExitStatus(unix_wait_status(9))
5779 [8820][1] reports Exit
57802024-03-02T01:06:48.026ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57812024-03-02T01:06:48.026ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57822024-03-02T01:06:48.026ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57832024-03-02T01:06:48.026ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57842024-03-02T01:06:48.026ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57852024-03-02T01:06:48.026ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57862024-03-02T01:06:48.026ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57872024-03-02T01:06:49.028ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57882024-03-02T01:06:49.028ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57892024-03-02T01:06:49.028ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57902024-03-02T01:06:49.028ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57912024-03-02T01:06:49.028ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57922024-03-02T01:06:49.028ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57932024-03-02T01:06:49.028ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57942024-03-02T01:06:50.029ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57952024-03-02T01:06:50.029ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57962024-03-02T01:06:50.029ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57972024-03-02T01:06:50.029ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57982024-03-02T01:06:50.029ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
57992024-03-02T01:06:50.029ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58002024-03-02T01:06:50.030ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5801 Mar 02 01:06:51.029 INFO request completed, latency_us: 345, response_code: 200, uri: /state/cid/1, method: GET, req_id: 053e642a-9eb6-4085-8e35-ed951ab8b456, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
5802 001/300 Flush
5803 002/300 Write block 1528 len 32768 data: 5 4 3 3 3 3 2 2
5804 003/300 Read block 1915 len 4096
58052024-03-02T01:06:51.031ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58062024-03-02T01:06:51.031ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58072024-03-02T01:06:51.031ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58082024-03-02T01:06:51.031ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58092024-03-02T01:06:51.031ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58102024-03-02T01:06:51.031ZINFOcrucible: [1] client re-new 3 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
58112024-03-02T01:06:51.031ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
5812 004/300 Flush
5813 005/300 Read block 4884 len 16384
5814 006/300 Write block 4354 len 20480 data: 4 4 3 3 3
5815 007/300 Write block 3362 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5816 008/300 Write block 2585 len 40960 data: 3 3 3 2 2 2 2 3 3 3
5817 009/300 Read block 4040 len 36864
5818 010/300 Flush
5819 011/300 Write block 1733 len 12288 data: 3 4 4
5820 012/300 Write block 4372 len 4096 data: 3
5821 013/300 Write block 224 len 36864 data: 2 2 2 2 2 2 2 2 2
5822 014/300 Read block 2141 len 12288
5823 015/300 Read block 2261 len 12288
5824 016/300 Read block 3788 len 36864
5825 017/300 Write block 4294 len 36864 data: 3 3 2 3 5 5 3 3 3
5826 018/300 Write block 3919 len 36864 data: 4 3 3 3 4 4 4 4 4
5827 019/300 Write block 3372 len 4096 data: 3
5828 020/300 Read block 101 len 40960
5829 021/300 Read block 2606 len 16384
5830 022/300 Write block 4564 len 4096 data: 4
5831 023/300 Flush
5832 024/300 Write block 2013 len 24576 data: 3 3 3 2 2 2
5833 025/300 Write block 103 len 40960 data: 2 2 2 2 3 3 3 4 4 4
5834 026/300 Read block 4257 len 24576
5835 027/300 Read block 899 len 12288
5836 028/300 Write block 4311 len 8192 data: 2 2
5837 029/300 Read block 3211 len 12288
5838 030/300 Read block 4626 len 40960
5839 031/300 Read block 3496 len 12288
5840 032/300 Write block 4775 len 16384 data: 2 2 2 2
5841 033/300 Read block 2751 len 32768
5842 034/300 Read block 3713 len 8192
5843 035/300 Write block 859 len 12288 data: 3 3 3
5844 036/300 Read block 1297 len 32768
5845 037/300 Write block 732 len 12288 data: 2 4 4
5846 038/300 Read block 1791 len 28672
5847 039/300 Read block 357 len 28672
5848 040/300 Read block 3142 len 40960
5849 041/300 Write block 1287 len 32768 data: 2 2 2 2 2 2 2 2
5850 042/300 Read block 3043 len 4096
5851 043/300 Read block 3350 len 12288
5852 044/300 Read block 2006 len 16384
5853 045/300 Read block 1274 len 16384
5854 046/300 Write block 3104 len 24576 data: 2 2 2 2 2 2
5855 047/300 Flush
5856 048/300 Read block 4548 len 8192
5857 049/300 Read block 3733 len 16384
5858 050/300 Read block 2495 len 4096
5859 051/300 Write block 2723 len 28672 data: 3 2 2 2 2 3 2
5860 052/300 Read block 3050 len 4096
5861 053/300 Read block 4670 len 24576
5862 054/300 Read block 3058 len 16384
5863 055/300 Write block 3480 len 4096 data: 4
5864 056/300 Write block 3879 len 8192 data: 2 2
5865 057/300 Write block 1047 len 40960 data: 3 3 3 2 3 3 3 3 3 3
5866 058/300 Read block 2086 len 32768
5867 059/300 Write block 2171 len 24576 data: 3 3 4 4 4 3
5868 060/300 Write block 3851 len 12288 data: 3 3 3
5869 061/300 Flush
5870 062/300 Write block 2268 len 20480 data: 2 2 2 2 3
5871 063/300 Write block 2239 len 16384 data: 3 3 3 3
5872 064/300 Write block 1884 len 12288 data: 2 2 2
5873 065/300 Flush
5874 066/300 Read block 1665 len 20480
5875 067/300 Read block 4007 len 28672
5876 068/300 Read block 4886 len 32768
5877 069/300 Write block 466 len 32768 data: 2 3 3 3 3 3 3 3
5878 070/300 Write block 3643 len 36864 data: 2 2 2 2 3 3 3 4 4
5879 071/300 Read block 1719 len 40960
5880 072/300 Write block 2559 len 12288 data: 3 3 4
5881 073/300 Write block 3968 len 20480 data: 3 3 3 3 3
5882 074/300 Write block 1373 len 12288 data: 2 2 2
5883 075/300 Write block 4773 len 20480 data: 3 3 3 3 3
5884 076/300 Read block 1525 len 4096
5885 077/300 Read block 3452 len 16384
5886 078/300 Read block 1308 len 24576
5887 079/300 Read block 4567 len 32768
5888 080/300 Read block 1099 len 32768
5889 081/300 Write block 1914 len 36864 data: 2 2 2 2 2 2 2 3 3
5890 082/300 Write block 3052 len 28672 data: 3 3 3 3 2 2 2
5891 083/300 Write block 371 len 16384 data: 2 2 2 2
5892 084/300 Write block 4969 len 12288 data: 4 5 5
5893 085/300 Write block 1529 len 40960 data: 5 4 4 4 4 3 3 2 2 2
5894 086/300 Read block 3415 len 12288
5895 087/300 Read block 4933 len 36864
5896 088/300 Read block 4361 len 8192
5897 089/300 Write block 3153 len 20480 data: 3 4 4 3 4
5898 090/300 Read block 2669 len 32768
5899 091/300 Write block 3626 len 28672 data: 2 2 2 2 2 3 3
5900 092/300 Write block 1000 len 8192 data: 4 4
5901 093/300 Write block 4654 len 36864 data: 3 3 2 2 2 2 2 2 2
5902 094/300 Read block 2748 len 36864
5903 095/300 Write block 594 len 8192 data: 2 2
5904 096/300 Read block 1084 len 4096
5905 097/300 Read block 1188 len 12288
5906 098/300 Read block 2676 len 20480
5907 099/300 Write block 885 len 28672 data: 2 2 3 3 3 3 3
5908 100/300 Read block 464 len 40960
5909 101/300 Write block 308 len 24576 data: 2 2 2 2 2 2
5910 102/300 Write block 2897 len 4096 data: 2
5911 103/300 Read block 781 len 24576
5912 104/300 Read block 413 len 8192
5913 105/300 Write block 363 len 12288 data: 2 2 2
5914 106/300 Read block 4837 len 12288
5915 107/300 Read block 3430 len 20480
5916 108/300 Read block 4616 len 20480
5917 109/300 Write block 812 len 28672 data: 4 4 5 4 3 4 4
5918 110/300 Read block 3740 len 36864
5919 111/300 Read block 2824 len 24576
5920 112/300 Write block 2442 len 36864 data: 3 3 3 3 3 2 2 2 2
5921 113/300 Write block 122 len 8192 data: 2 4
5922 114/300 Write block 4410 len 8192 data: 2 2
5923 115/300 Read block 406 len 8192
5924 116/300 Read block 2311 len 4096
5925 117/300 Write block 211 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5926 118/300 Read block 2255 len 36864
5927 119/300 Read block 1017 len 32768
5928 120/300 Flush
5929 121/300 Write block 3590 len 20480 data: 3 3 4 4 4
5930 122/300 Read block 4976 len 24576
5931 123/300 Flush
5932 124/300 Write block 4432 len 8192 data: 3 2
5933 125/300 Read block 3326 len 16384
5934 126/300 Read block 4682 len 4096
5935 127/300 Write block 1042 len 4096 data: 2
5936 128/300 Read block 4117 len 24576
5937 129/300 Write block 1552 len 28672 data: 3 4 4 4 4 4 3
5938 130/300 Read block 3363 len 40960
5939 131/300 Read block 4006 len 20480
5940 132/300 Read block 956 len 40960
5941 133/300 Flush
5942 134/300 Write block 2372 len 32768 data: 3 3 3 2 2 2 2 2
5943 135/300 Read block 1313 len 12288
5944 136/300 Write block 4451 len 12288 data: 2 3 3
5945 137/300 Read block 497 len 24576
5946 138/300 Read block 4340 len 24576
5947 139/300 Read block 1113 len 12288
5948 140/300 Write block 1710 len 32768 data: 3 4 4 4 3 3 2 2
5949 141/300 Write block 1603 len 28672 data: 2 2 2 2 2 3 3
5950 142/300 Write block 1048 len 4096 data: 4
5951 143/300 Write block 1356 len 16384 data: 2 2 3 3
5952 144/300 Flush
5953 145/300 Read block 739 len 8192
5954 146/300 Write block 1756 len 16384 data: 3 2 2 2
5955 147/300 Write block 4377 len 28672 data: 2 2 3 3 3 2 2
5956 148/300 Read block 2580 len 16384
5957 149/300 Write block 2556 len 36864 data: 3 3 3 4 4 5 4 3 3
5958 150/300 Read block 3124 len 20480
5959 151/300 Read block 4874 len 28672
5960 152/300 Read block 3592 len 20480
5961 153/300 Read block 1770 len 24576
5962 154/300 Read block 2116 len 20480
5963 155/300 Flush
5964 156/300 Read block 1279 len 24576
5965 157/300 Read block 3154 len 8192
5966 158/300 Write block 1205 len 8192 data: 3 3
5967 159/300 Read block 854 len 16384
5968 160/300 Write block 948 len 32768 data: 2 2 2 2 2 3 3 3
5969 161/300 Write block 2720 len 24576 data: 4 3 3 4 3 3
5970 162/300 Write block 2876 len 12288 data: 2 2 2
5971 163/300 Flush
5972 164/300 Read block 727 len 20480
5973 165/300 Write block 813 len 40960 data: 5 6 5 4 5 5 4 3 3 2
5974 166/300 Write block 1042 len 16384 data: 3 2 2 2
5975 167/300 Read block 2341 len 12288
5976 168/300 Write block 3348 len 24576 data: 2 2 2 3 4 4
5977 169/300 Read block 45 len 8192
5978 170/300 Read block 801 len 4096
5979 171/300 Write block 836 len 24576 data: 2 2 2 2 2 2
5980 172/300 Read block 3079 len 40960
5981 173/300 Write block 2120 len 28672 data: 3 2 2 2 2 2 2
5982 174/300 Read block 4512 len 36864
5983 175/300 Write block 4457 len 12288 data: 2 2 2
5984 176/300 Read block 4163 len 40960
5985 177/300 Write block 4291 len 8192 data: 3 3
5986 178/300 Read block 1277 len 20480
5987 179/300 Read block 1457 len 36864
5988 180/300 Write block 161 len 32768 data: 4 4 4 4 3 2 2 2
5989 181/300 Read block 3265 len 12288
5990 182/300 Write block 1197 len 32768 data: 4 2 2 3 4 4 4 4
5991 183/300 Read block 1935 len 4096
5992 184/300 Read block 1246 len 16384
5993 185/300 Read block 3126 len 4096
5994 186/300 Write block 532 len 32768 data: 4 4 4 4 4 3 2 2
5995 187/300 Read block 3002 len 36864
5996 188/300 Write block 4859 len 4096 data: 2
5997 189/300 Read block 1249 len 20480
5998 190/300 Read block 1147 len 12288
5999 191/300 Flush
6000 192/300 Write block 1580 len 28672 data: 4 2 2 2 3 3 3
6001 193/300 Write block 2201 len 12288 data: 3 3 3
6002 194/300 Read block 4435 len 20480
6003 195/300 Read block 3376 len 36864
6004 196/300 Write block 4335 len 28672 data: 2 2 2 2 2 2 2
6005 197/300 Read block 4358 len 8192
6006 198/300 Read block 4384 len 12288
6007 199/300 Write block 2922 len 4096 data: 2
6008 200/300 Flush
6009 201/300 Read block 3662 len 4096
6010 202/300 Read block 3723 len 40960
6011 203/300 Write block 127 len 12288 data: 5 4 4
6012 204/300 Read block 2911 len 12288
6013 205/300 Read block 3362 len 12288
6014 206/300 Read block 4063 len 16384
6015 207/300 Flush
6016 208/300 Write block 3003 len 28672 data: 3 4 4 4 4 4 4
6017 209/300 Read block 8 len 40960
6018 210/300 Read block 520 len 8192
6019 211/300 Read block 3260 len 4096
6020 212/300 Read block 269 len 16384
6021 213/300 Read block 1496 len 28672
6022 214/300 Read block 2287 len 40960
6023 215/300 Write block 2261 len 16384 data: 3 3 3 2
6024 216/300 Read block 4448 len 40960
6025 217/300 Write block 4330 len 36864 data: 3 3 4 4 4 3 3 3 3
6026 218/300 Write block 3023 len 12288 data: 4 4 3
6027 219/300 Write block 3366 len 40960 data: 3 3 3 3 4 4 4 3 2 2
6028 220/300 Flush
6029 221/300 Read block 4388 len 12288
6030 222/300 Write block 2838 len 40960 data: 4 5 5 4 5 5 5 4 4 4
6031 223/300 Write block 1398 len 40960 data: 3 2 2 2 2 2 2 2 2 2
6032 224/300 Write block 863 len 32768 data: 3 2 2 2 2 2 2 2
6033 225/300 Read block 4032 len 8192
6034 226/300 Flush
6035 227/300 Read block 2822 len 4096
6036 228/300 Read block 3384 len 12288
6037 229/300 Read block 1539 len 40960
6038 230/300 Flush
6039 231/300 Write block 2546 len 12288 data: 2 2 2
6040 232/300 Write block 3015 len 16384 data: 3 3 3 3
6041 233/300 Read block 1289 len 24576
6042 234/300 Write block 1418 len 24576 data: 3 3 3 3 3 3
6043 235/300 Write block 698 len 40960 data: 2 2 2 2 3 3 2 2 2 2
6044 236/300 Read block 4793 len 20480
6045 237/300 Write block 4659 len 12288 data: 3 3 3
6046 238/300 Read block 4030 len 28672
6047 239/300 Write block 2893 len 40960 data: 2 2 2 2 3 3 3 3 3 3
6048 240/300 Read block 4607 len 36864
6049 241/300 Write block 1608 len 8192 data: 4 4
6050 242/300 Read block 1107 len 12288
6051 243/300 Read block 3724 len 4096
6052 244/300 Flush
6053 245/300 Read block 1580 len 12288
6054 246/300 Write block 3776 len 28672 data: 2 3 3 3 4 4 4
6055 247/300 Write block 4447 len 24576 data: 3 3 3 3 3 4
6056 248/300 Write block 2453 len 32768 data: 2 2 2 3 2 2 3 3
6057 249/300 Read block 4668 len 12288
6058 250/300 Read block 712 len 32768
6059 251/300 Write block 751 len 4096 data: 3
6060 252/300 Read block 992 len 36864
6061 253/300 Read block 4461 len 8192
6062 254/300 Write block 2330 len 32768 data: 2 2 2 2 2 2 2 2
6063 255/300 Read block 28 len 12288
6064 256/300 Write block 3617 len 32768 data: 3 3 3 2 2 2 2 2
6065 257/300 Write block 604 len 40960 data: 3 3 3 2 2 3 4 4 4 4
6066 258/300 Write block 1664 len 20480 data: 3 4 4 4 3
6067 259/300 Write block 1893 len 12288 data: 2 3 3
6068 260/300 Read block 2622 len 20480
6069 261/300 Read block 3794 len 20480
6070 262/300 Write block 1095 len 28672 data: 3 3 3 3 3 3 3
6071 263/300 Flush
6072 264/300 Read block 4152 len 4096
6073 265/300 Read block 1391 len 12288
6074 266/300 Write block 3142 len 12288 data: 2 2 2
6075 267/300 Write block 4957 len 8192 data: 2 2
6076 268/300 Write block 939 len 40960 data: 3 3 3 3 4 4 4 3 3 3
6077 269/300 Read block 2026 len 16384
6078 270/300 Write block 130 len 12288 data: 3 3 2
6079 271/300 Read block 936 len 32768
6080 272/300 Write block 1295 len 24576 data: 2 2 3 4 4 5
6081 273/300 Read block 1456 len 24576
6082 274/300 Flush
6083 275/300 Write block 488 len 16384 data: 2 2 3 2
6084 276/300 Read block 2194 len 32768
6085 277/300 Read block 564 len 32768
6086 278/300 Read block 4791 len 8192
6087 279/300 Write block 3196 len 8192 data: 2 2
6088 280/300 Write block 1411 len 40960 data: 2 2 2 2 2 2 3 4 4 4
6089 281/300 Read block 3888 len 16384
6090 282/300 Read block 3374 len 28672
6091 283/300 Read block 1309 len 8192
6092 284/300 Read block 1673 len 4096
6093 285/300 Write block 1578 len 28672 data: 3 3 5 3 3 3 4
6094 286/300 Write block 3709 len 16384 data: 3 3 3 3
6095 287/300 Read block 4290 len 32768
6096 288/300 Write block 3546 len 28672 data: 2 2 2 2 2 3 4
6097 289/300 Write block 3699 len 36864 data: 3 3 2 2 2 3 3 3 3
6098 290/300 Write block 838 len 28672 data: 3 3 3 3 2 2 2
6099 291/300 Read block 4482 len 32768
6100 292/300 Read block 1974 len 12288
6101 293/300 Write block 2114 len 32768 data: 2 2 3 3 3 3 4 3
6102 294/300 Read block 393 len 4096
6103 295/300 Read block 990 len 12288
6104 296/300 Write block 2584 len 8192 data: 3 4
6105 297/300 Write block 2855 len 12288 data: 2 2 2
6106 298/300 Write block 1692 len 32768 data: 2 2 2 2 2 2 2 2
6107 299/300 Read block 726 len 8192
6108 300/300 Write block 4952 len 16384 data: 3 3 3 3
6109 Main task has work to do, go find it
6110 got dsc Start(1)
6111 start 1
6112 Mar 02 01:06:51.717 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: 0cee7516-9acb-444f-8125-3320179d6716, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
6113 Replay: started 1, returned:Ok(())
6114 ----------------------------------------------------------------
6115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
6116 States: Active Offline Active
6117 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6118 1809 Acked 2808 Flush 0 Done New Done true
6119 1810 Acked 2809 Write 8 Done New Done true
6120 1811 Acked 2810 Read 1 Done New Done true
6121 1812 Acked 2811 Flush 0 Done New Done false
6122 1813 Acked 2812 Read 4 Done New Done false
6123 1814 Acked 2813 Write 5 Done New Done false
6124 1815 Acked 2814 Write 10 Done New Done false
6125 1816 Acked 2815 Write 10 Done New Done false
6126 1817 Acked 2816 Read 9 Done New Done false
6127 1818 Acked 2817 Flush 0 Done New Done false
6128 1819 Acked 2818 Write 3 Done New Done false
6129 1820 Acked 2819 Write 1 Done New Done false
6130 1821 Acked 2820 Write 9 Done New Done false
6131 1822 Acked 2821 Read 3 Done New Done false
6132 1823 Acked 2822 Read 3 Done New Done false
6133 1824 Acked 2823 Read 9 Done New Done false
6134 1825 Acked 2824 Write 9 Done New Done false
6135 1826 Acked 2825 Write 9 Done New Done false
6136 1827 Acked 2826 Write 1 Done New Done false
6137 1828 Acked 2827 Read 10 Done New Done false
6138 1829 Acked 2828 Read 4 Done New Done false
6139 1830 Acked 2829 Write 1 Done New Done false
6140 1831 Acked 2830 Flush 0 Done New Done false
6141 1832 Acked 2831 Write 6 Done New Done false
6142 1833 Acked 2832 Write 10 Done New Done false
6143 1834 Acked 2833 Read 6 Done New Done false
6144 1835 Acked 2834 Read 3 Done New Done false
6145 1836 Acked 2835 Write 2 Done New Done false
6146 1837 Acked 2836 Read 3 Done New Done false
6147 1838 Acked 2837 Read 10 Done New Done false
6148 1839 Acked 2838 Read 3 Done New Done false
6149 1840 Acked 2839 Write 4 Done New Done false
6150 1841 Acked 2840 Read 8 Done New Done false
6151 1842 Acked 2841 Read 2 Done New Done false
6152 1843 Acked 2842 Write 3 Done New Done false
6153 1844 Acked 2843 Read 8 Done New Done false
6154 1845 Acked 2844 Write 3 Done New Done false
6155 1846 Acked 2845 Read 7 Done New Done false
6156 1847 Acked 2846 Read 7 Done New Done false
6157 1848 Acked 2847 Read 10 Done New Done false
6158 1849 Acked 2848 Write 8 Done New Done false
6159 1850 Acked 2849 Read 1 Done New Done false
6160 1851 Acked 2850 Read 3 Done New Done false
6161 1852 Acked 2851 Read 4 Done New Done false
6162 1853 Acked 2852 Read 4 Done New Done false
6163 1854 Acked 2853 Write 6 Done New Done false
6164 1855 Acked 2854 Flush 0 Done New Done false
6165 1856 Acked 2855 Read 2 Done New Done false
6166 1857 Acked 2856 Read 4 Done New Done false
6167 1858 Acked 2857 Read 1 Done New Done false
6168 1859 Acked 2858 Write 7 Done New Done false
6169 1860 Acked 2859 Read 1 Done New Done false
6170 1861 Acked 2860 Read 6 Done New Done false
6171 1862 Acked 2861 Read 4 Done New Done false
6172 1863 Acked 2862 Write 1 Done New Done false
6173 1864 Acked 2863 Write 2 Done New Done false
6174 1865 Acked 2864 Write 10 Done New Done false
6175 1866 Acked 2865 Read 8 Done New Done false
6176 1867 Acked 2866 Write 6 Done New Done false
6177 1868 Acked 2867 Write 3 Done New Done false
6178 1869 Acked 2868 Flush 0 Done New Done false
6179 1870 Acked 2869 Write 5 Done New Done false
6180 1871 Acked 2870 Write 4 Done New Done false
6181 1872 Acked 2871 Write 3 Done New Done false
6182 1873 Acked 2872 Flush 0 Done New Done false
6183 1874 Acked 2873 Read 5 Done New Done false
6184 1875 Acked 2874 Read 7 Done New Done false
6185 1876 Acked 2875 Read 8 Done New Done false
6186 1877 Acked 2876 Write 8 Done New Done false
6187 1878 Acked 2877 Write 9 Done New Done false
6188 1879 Acked 2878 Read 10 Done New Done false
6189 1880 Acked 2879 Write 3 Done New Done false
6190 1881 Acked 2880 Write 5 Done New Done false
6191 1882 Acked 2881 Write 3 Done New Done false
6192 1883 Acked 2882 Write 5 Done New Done false
6193 1884 Acked 2883 Read 1 Done New Done false
6194 1885 Acked 2884 Read 4 Done New Done false
6195 1886 Acked 2885 Read 6 Done New Done false
6196 1887 Acked 2886 Read 8 Done New Done false
6197 1888 Acked 2887 Read 8 Done New Done false
6198 1889 Acked 2888 Write 9 Done New Done false
6199 1890 Acked 2889 Write 7 Done New Done false
6200 1891 Acked 2890 Write 4 Done New Done false
6201 1892 Acked 2891 Write 3 Done New Done false
6202 1893 Acked 2892 Write 10 Done New Done false
6203 1894 Acked 2893 Read 3 Done New Done false
6204 1895 Acked 2894 Read 9 Done New Done false
6205 1896 Acked 2895 Read 2 Done New Done false
6206 1897 Acked 2896 Write 5 Done New Done false
6207 1898 Acked 2897 Read 8 Done New Done false
6208 1899 Acked 2898 Write 7 Done New Done false
6209 1900 Acked 2899 Write 2 Done New Done false
6210 1901 Acked 2900 Write 9 Done New Done false
6211 1902 Acked 2901 Read 9 Done New Done false
6212 1903 Acked 2902 Write 2 Done New Done false
6213 1904 Acked 2903 Read 1 Done New Done false
6214 1905 Acked 2904 Read 3 Done New Done false
6215 1906 Acked 2905 Read 5 Done New Done false
6216 1907 Acked 2906 Write 7 Done New Done false
6217 1908 Acked 2907 Read 10 Done New Done false
6218 1909 Acked 2908 Write 6 Done New Done false
6219 1910 Acked 2909 Write 1 Done New Done false
6220 1911 Acked 2910 Read 6 Done New Done false
6221 1912 Acked 2911 Read 2 Done New Done false
6222 1913 Acked 2912 Write 3 Done New Done false
6223 1914 Acked 2913 Read 3 Done New Done false
6224 1915 Acked 2914 Read 5 Done New Done false
6225 1916 Acked 2915 Read 5 Done New Done false
6226 1917 Acked 2916 Write 7 Done New Done false
6227 1918 Acked 2917 Read 9 Done New Done false
6228 1919 Acked 2918 Read 6 Done New Done false
6229 1920 Acked 2919 Write 9 Done New Done false
6230 1921 Acked 2920 Write 2 Done New Done false
6231 1922 Acked 2921 Write 2 Done New Done false
6232 1923 Acked 2922 Read 2 Done New Done false
6233 1924 Acked 2923 Read 1 Done New Done false
6234 1925 Acked 2924 Write 10 Done New Done false
6235 1926 Acked 2925 Read 9 Done New Done false
6236 1927 Acked 2926 Read 8 Done New Done false
6237 1928 Acked 2927 Flush 0 Done New Done false
6238 1929 Acked 2928 Write 5 Done New Done false
6239 1930 Acked 2929 Read 6 Done New Done false
6240 1931 Acked 2930 Flush 0 Done New Done false
6241 1932 Acked 2931 Write 2 Done New Done false
6242 1933 Acked 2932 Read 4 Done New Done false
6243 1934 Acked 2933 Read 1 Done New Done false
6244 1935 Acked 2934 Write 1 Done New Done false
6245 1936 Acked 2935 Read 6 Done New Done false
6246 1937 Acked 2936 Write 7 Done New Done false
6247 1938 Acked 2937 Read 10 Done New Done false
6248 1939 Acked 2938 Read 5 Done New Done false
6249 1940 Acked 2939 Read 10 Done New Done false
6250 1941 Acked 2940 Flush 0 Done New Done false
6251 1942 Acked 2941 Write 8 Done New Done false
6252 1943 Acked 2942 Read 3 Done New Done false
6253 1944 Acked 2943 Write 3 Done New Done false
6254 1945 Acked 2944 Read 6 Done New Done false
6255 1946 Acked 2945 Read 6 Done New Done false
6256 1947 Acked 2946 Read 3 Done New Done false
6257 1948 Acked 2947 Write 8 Done New Done false
6258 1949 Acked 2948 Write 7 Done New Done false
6259 1950 Acked 2949 Write 1 Done New Done false
6260 1951 Acked 2950 Write 4 Done New Done false
6261 1952 Acked 2951 Flush 0 Done New Done false
6262 1953 Acked 2952 Read 2 Done New Done false
6263 1954 Acked 2953 Write 4 Done New Done false
6264 1955 Acked 2954 Write 7 Done New Done false
6265 1956 Acked 2955 Read 4 Done New Done false
6266 1957 Acked 2956 Write 9 Done New Done false
6267 1958 Acked 2957 Read 5 Done New Done false
6268 1959 Acked 2958 Read 7 Done New Done false
6269 1960 Acked 2959 Read 5 Done New Done false
6270 1961 Acked 2960 Read 6 Done New Done false
6271 1962 Acked 2961 Read 5 Done New Done false
6272 1963 Acked 2962 Flush 0 Done New Done false
6273 1964 Acked 2963 Read 6 Done New Done false
6274 1965 Acked 2964 Read 2 Done New Done false
6275 1966 Acked 2965 Write 2 Done New Done false
6276 1967 Acked 2966 Read 4 Done New Done false
6277 1968 Acked 2967 Write 8 Done New Done false
6278 1969 Acked 2968 Write 6 Done New Done false
6279 1970 Acked 2969 Write 3 Done New Done false
6280 1971 Acked 2970 Flush 0 Done New Done false
6281 1972 Acked 2971 Read 5 Done New Done false
6282 1973 Acked 2972 Write 10 Done New Done false
6283 1974 Acked 2973 Write 4 Done New Done false
6284 1975 Acked 2974 Read 3 Done New Done false
6285 1976 Acked 2975 Write 6 Done New Done false
6286 1977 Acked 2976 Read 2 Done New Done false
6287 1978 Acked 2977 Read 1 Done New Done false
6288 1979 Acked 2978 Write 6 Done New Done false
6289 1980 Acked 2979 Read 10 Done New Done false
6290 1981 Acked 2980 Write 7 Done New Done false
6291 1982 Acked 2981 Read 9 Done New Done false
6292 1983 Acked 2982 Write 3 Done New Done false
6293 1984 Acked 2983 Read 10 Done New Done false
6294 1985 Acked 2984 Write 2 Done New Done false
6295 1986 Acked 2985 Read 5 Done New Done false
6296 1987 Acked 2986 Read 9 Done New Done false
6297 1988 Acked 2987 Write 8 Done New Done false
6298 1989 Acked 2988 Read 3 Done New Done false
6299 1990 Acked 2989 Write 8 Done New Done false
6300 1991 Acked 2990 Read 1 Done New Done false
6301 1992 Acked 2991 Read 4 Done New Done false
6302 1993 Acked 2992 Read 1 Done New Done false
6303 1994 Acked 2993 Write 8 Done New Done false
6304 1995 Acked 2994 Read 9 Done New Done false
6305 1996 Acked 2995 Write 1 Done New Done false
6306 1997 Acked 2996 Read 5 Done New Done false
6307 1998 Acked 2997 Read 3 Done New Done false
6308 1999 Acked 2998 Flush 0 Done New Done false
6309 2000 Acked 2999 Write 7 Done New Done false
6310 2001 Acked 3000 Write 3 Done New Done false
6311 2002 Acked 3001 Read 5 Done New Done false
6312 2003 Acked 3002 Read 9 Done New Done false
6313 2004 Acked 3003 Write 7 Done New Done false
6314 2005 Acked 3004 Read 2 Done New Done false
6315 2006 Acked 3005 Read 3 Done New Done false
6316 2007 Acked 3006 Write 1 Done New Done false
6317 2008 Acked 3007 Flush 0 Done New Done false
6318 2009 Acked 3008 Read 1 Done New Done false
6319 2010 Acked 3009 Read 10 Done New Done false
6320 2011 Acked 3010 Write 3 Done New Done false
6321 2012 Acked 3011 Read 3 Done New Done false
6322 2013 Acked 3012 Read 3 Done New Done false
6323 2014 Acked 3013 Read 4 Done New Done false
6324 2015 Acked 3014 Flush 0 Done New Done false
6325 2016 Acked 3015 Write 7 Done New Done false
6326 2017 Acked 3016 Read 10 Done New Done false
6327 2018 Acked 3017 Read 2 Done New Done false
6328 2019 Acked 3018 Read 1 Done New Done false
6329 2020 Acked 3019 Read 4 Done New Done false
6330 2021 Acked 3020 Read 7 Done New Done false
6331 2022 Acked 3021 Read 10 Done New Done false
6332 2023 Acked 3022 Write 4 Done New Done false
6333 2024 Acked 3023 Read 10 Done New Done false
6334 2025 Acked 3024 Write 9 Done New Done false
6335 2026 Acked 3025 Write 3 Done New Done false
6336 2027 Acked 3026 Write 10 Done New Done false
6337 2028 Acked 3027 Flush 0 Done New Done false
6338 2029 Acked 3028 Read 3 Done New Done false
6339 2030 Acked 3029 Write 10 Done New Done false
6340 2031 Acked 3030 Write 10 Done New Done false
6341 2032 Acked 3031 Write 8 Done New Done false
6342 2033 Acked 3032 Read 2 Done New Done false
6343 2034 Acked 3033 Flush 0 Done New Done false
6344 2035 Acked 3034 Read 1 Done New Done false
6345 2036 Acked 3035 Read 3 Done New Done false
6346 2037 Acked 3036 Read 10 Done New Done false
6347 2038 Acked 3037 Flush 0 Done New Done false
6348 2039 Acked 3038 Write 3 Done New Done false
6349 2040 Acked 3039 Write 4 Done New Done false
6350 2041 Acked 3040 Read 6 Done New Done false
6351 2042 Acked 3041 Write 6 Done New Done false
6352 2043 Acked 3042 Write 10 Done New Done false
6353 2044 Acked 3043 Read 5 Done New Done false
6354 2045 Acked 3044 Write 3 Done New Done false
6355 2046 Acked 3045 Read 7 Done New Done false
6356 2047 Acked 3046 Write 10 Done New Done false
6357 2048 Acked 3047 Read 9 Done New Done false
6358 2049 Acked 3048 Write 2 Done New Done false
6359 2050 Acked 3049 Read 3 Done New Done false
6360 2051 Acked 3050 Read 1 Done New Done false
6361 2052 Acked 3051 Flush 0 Done New Done false
6362 2053 Acked 3052 Read 3 Done New Done false
6363 2054 Acked 3053 Write 7 Done New Done false
6364 2055 Acked 3054 Write 6 Done New Done false
6365 2056 Acked 3055 Write 8 Done New Done false
6366 2057 Acked 3056 Read 3 Done New Done false
6367 2058 Acked 3057 Read 8 Done New Done false
6368 2059 Acked 3058 Write 1 Done New Done false
6369 2060 Acked 3059 Read 9 Done New Done false
6370 2061 Acked 3060 Read 2 Done New Done false
6371 2062 Acked 3061 Write 8 Done New Done false
6372 2063 Acked 3062 Read 3 Done New Done false
6373 2064 Acked 3063 Write 8 Done New Done false
6374 2065 Acked 3064 Write 10 Done New Done false
6375 2066 Acked 3065 Write 5 Done New Done false
6376 2067 Acked 3066 Write 3 Done New Done false
6377 2068 Acked 3067 Read 5 Done New Done false
6378 2069 Acked 3068 Read 5 Done New Done false
6379 2070 Acked 3069 Write 7 Done New Done false
6380 2071 Acked 3070 Flush 0 Done New Done false
6381 2072 Acked 3071 Read 1 Done New Done false
6382 2073 Acked 3072 Read 3 Done New Done false
6383 2074 Acked 3073 Write 3 Done New Done false
6384 2075 Acked 3074 Write 2 Done New Done false
6385 2076 Acked 3075 Write 10 Done New Done false
6386 2077 Acked 3076 Read 4 Done New Done false
6387 2078 Acked 3077 Write 3 Done New Done false
6388 2079 Acked 3078 Read 8 Done New Done false
6389 2080 Acked 3079 Write 6 Done New Done false
6390 2081 Acked 3080 Read 6 Done New Done false
6391 2082 Acked 3081 Flush 0 Done New Done false
6392 2083 Acked 3082 Write 4 Done New Done false
6393 2084 Acked 3083 Read 8 Done New Done false
6394 2085 Acked 3084 Read 8 Done New Done false
6395 2086 Acked 3085 Read 2 Done New Done false
6396 2087 Acked 3086 Write 2 Done New Done false
6397 2088 Acked 3087 Write 10 Done New Done false
6398 2089 Acked 3088 Read 4 Done New Done false
6399 2090 Acked 3089 Read 7 Done New Done false
6400 2091 Acked 3090 Read 2 Done New Done false
6401 2092 Acked 3091 Read 1 Done New Done false
6402 2093 Acked 3092 Write 7 Done New Done false
6403 2094 Acked 3093 Write 4 Done New Done false
6404 2095 Acked 3094 Read 8 Done New Done false
6405 2096 Acked 3095 Write 7 Done New Done false
6406 2097 Acked 3096 Write 9 Done New Done false
6407 2098 Acked 3097 Write 7 Done New Done false
6408 2099 Acked 3098 Read 8 Done New Done false
6409 2100 Acked 3099 Read 3 Done New Done false
6410 2101 Acked 3100 Write 8 Done New Done false
6411 2102 Acked 3101 Read 1 Done New Done false
6412 2103 Acked 3102 Read 3 Done New Done false
6413 2104 Acked 3103 Write 2 Done New Done false
6414 2105 Acked 3104 Write 3 Done New Done false
6415 2106 Acked 3105 Write 8 Done New Done false
6416 2107 Acked 3106 Read 2 Done New Done false
6417 2108 Acked 3107 Write 4 Done New Done false
6418 STATES DS:0 DS:1 DS:2 TOTAL
6419 New 0 300 0 300
6420 Sent 0 0 0 0
6421 Done 300 0 300 600
6422 Skipped 0 0 0 0
6423 Error 0 0 0 0
6424 Last Flush: 3081 2807 3081
6425 Downstairs last five completed: 2807 2806 2805 2804 2803
6426 Upstairs last five completed: 2108 2107 2106 2105 2104
6427 CLIENT: Up:0 ds:300 act:2
64282024-03-02T01:06:52.032ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64292024-03-02T01:06:52.032ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64302024-03-02T01:06:52.032ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64312024-03-02T01:06:52.032ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64322024-03-02T01:06:52.032ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64332024-03-02T01:06:52.032ZINFOcrucible: [1] client re-new 301 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64342024-03-02T01:06:52.032ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64352024-03-02T01:06:53.033ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64362024-03-02T01:06:53.033ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64372024-03-02T01:06:53.033ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64382024-03-02T01:06:53.033ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64392024-03-02T01:06:53.033ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64402024-03-02T01:06:53.033ZINFOcrucible: [1] client re-new 301 jobs since flush 2807 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64412024-03-02T01:06:53.033ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6442 [8820] got start action
6443 [8820] I am going to restart
6444 Starting downstairs at port 8820
6445 Make output file at "/tmp/dsc/downstairs-8820.txt"
6446 [8820][1] reports Starting
6447 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1169)
6448 [8820][1] reports Running
64492024-03-02T01:06:54.035ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64502024-03-02T01:06:54.035ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64512024-03-02T01:06:54.036ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64522024-03-02T01:06:54.036ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64532024-03-02T01:06:54.036ZINFOcrucible: send last flush ID to this DS: 2807 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64542024-03-02T01:06:54.036ZINFOcrucible: Replied this last flush ID: 2807 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64552024-03-02T01:06:54.037ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64562024-03-02T01:06:54.037ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64572024-03-02T01:06:54.037ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64582024-03-02T01:06:54.037ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6459 ----------------------------------------------------------------
6460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6461 Downstairs last five completed: 3108 3107 3106 3105 3104
6462 Upstairs last five completed: 2109 2108 2107 2106 2105
6463 CLIENT: Up:0 ds:0 act:3
6464 Replay: All jobs finished, all DS active.
6465 Main task has work to do, go find it
6466 got dsc Stop(1)
6467 stop 1
6468 Mar 02 01:06:55.721 INFO request completed, latency_us: 355[8820] Got stop action so:false kr:false
6469 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1ad54a7f-a16e-48a4-93db-894ddefeddeb, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
64702024-03-02T01:06:55.722ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6471 Mar 02 01:06:55.722 INFO request completed, latency_us: 134, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:06:55.722283584Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"1","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
6472 200, uri: /state/cid/1, method: GET, req_id: 70382442-0312-45b2-a44a-b2acb3d558a1, remote_addr: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:06:55.722328861Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
6473 127.0.0.1:40735, local_addr: 127.0.0.1:9998{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:06:55.722358989Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"1","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
6474 
64752024-03-02T01:06:55.722ZINFOcrucible: [1] client re-new 0 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64762024-03-02T01:06:55.722ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6477 [8820] Exited with: ExitStatus(unix_wait_status(9))
6478 [8820][1] reports Exit
64792024-03-02T01:06:56.723ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64802024-03-02T01:06:56.723ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64812024-03-02T01:06:56.723ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64822024-03-02T01:06:56.723ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64832024-03-02T01:06:56.723ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64842024-03-02T01:06:56.723ZINFOcrucible: [1] client re-new 0 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64852024-03-02T01:06:56.723ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64862024-03-02T01:06:57.724ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64872024-03-02T01:06:57.724ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64882024-03-02T01:06:57.724ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64892024-03-02T01:06:57.724ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64902024-03-02T01:06:57.724ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64912024-03-02T01:06:57.724ZINFOcrucible: [1] client re-new 0 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64922024-03-02T01:06:57.724ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64932024-03-02T01:06:58.726ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64942024-03-02T01:06:58.726ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64952024-03-02T01:06:58.726ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64962024-03-02T01:06:58.726ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64972024-03-02T01:06:58.726ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64982024-03-02T01:06:58.726ZINFOcrucible: [1] client re-new 0 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
64992024-03-02T01:06:58.726ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6500 Mar 02 01:06:59.725 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1210c7bd-b7b8-4c3a-a0ab-9f90fd288be5, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
6501 001/300 Read block 3413 len 32768
6502 002/300 Write block 4041 len 16384 data: 2 2 2 2
65032024-03-02T01:06:59.727ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
65042024-03-02T01:06:59.727ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
65052024-03-02T01:06:59.727ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6506 003/300 Read block 4889 len 8192
65072024-03-02T01:06:59.728ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
65082024-03-02T01:06:59.728ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
65092024-03-02T01:06:59.728ZINFOcrucible: [1] client re-new 2 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
65102024-03-02T01:06:59.728ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
6511 004/300 Write block 947 len 16384 data: 4 4 3 3
6512 005/300 Write block 1874 len 16384 data: 3 3 3 3
6513 006/300 Write block 3930 len 28672 data: 2 2 2 2 3 3 3
6514 007/300 Write block 4159 len 24576 data: 2 2 2 2 2 2
6515 008/300 Read block 1681 len 40960
6516 009/300 Read block 2547 len 4096
6517 010/300 Read block 2486 len 36864
6518 011/300 Write block 2434 len 8192 data: 4 4
6519 012/300 Read block 4173 len 12288
6520 013/300 Write block 757 len 12288 data: 3 3 2
6521 014/300 Write block 2058 len 24576 data: 2 2 2 2 3 3
6522 015/300 Write block 2943 len 32768 data: 2 2 2 3 3 3 3 4
6523 016/300 Read block 2280 len 36864
6524 017/300 Read block 4204 len 4096
6525 018/300 Read block 4046 len 32768
6526 019/300 Write block 1581 len 12288 data: 4 4 4
6527 020/300 Read block 4620 len 20480
6528 021/300 Write block 3949 len 20480 data: 3 4 3 3 2
6529 022/300 Flush
6530 023/300 Write block 4657 len 24576 data: 3 3 4 4 4 3
6531 024/300 Flush
6532 025/300 Read block 2216 len 28672
6533 026/300 Write block 1090 len 4096 data: 2
6534 027/300 Read block 1108 len 24576
6535 028/300 Write block 239 len 32768 data: 3 4 5 3 3 3 3 3
6536 029/300 Flush
6537 030/300 Flush
6538 031/300 Write block 2089 len 16384 data: 4 4 4 3
6539 032/300 Write block 3956 len 8192 data: 3 3
6540 033/300 Write block 427 len 28672 data: 2 2 2 2 2 2 2
6541 034/300 Read block 2568 len 8192
6542 035/300 Write block 4805 len 20480 data: 3 3 4 4 4
6543 036/300 Read block 498 len 20480
6544 037/300 Read block 2778 len 12288
6545 038/300 Write block 812 len 12288 data: 5 6 7
6546 039/300 Write block 2197 len 12288 data: 4 3 2
6547 040/300 Write block 4800 len 40960 data: 4 4 2 2 3 4 4 5 5 5
6548 041/300 Read block 3566 len 36864
6549 042/300 Flush
6550 043/300 Write block 4186 len 40960 data: 5 5 4 4 5 4 3 3 3 4
6551 044/300 Read block 1045 len 28672
6552 045/300 Flush
6553 046/300 Read block 1193 len 28672
6554 047/300 Write block 2803 len 8192 data: 2 2
6555 048/300 Read block 703 len 40960
6556 049/300 Write block 2184 len 16384 data: 2 2 2 2
6557 050/300 Write block 4233 len 24576 data: 5 5 4 4 4 3
6558 051/300 Write block 4009 len 12288 data: 2 2 2
6559 052/300 Read block 2478 len 28672
6560 053/300 Read block 114 len 12288
6561 054/300 Read block 2529 len 20480
6562 055/300 Write block 2341 len 20480 data: 2 2 2 2 2
6563 056/300 Write block 4038 len 24576 data: 3 3 3 3 3 3
6564 057/300 Write block 4781 len 4096 data: 2
6565 058/300 Write block 3587 len 12288 data: 3 3 3
6566 059/300 Flush
6567 060/300 Read block 578 len 36864
6568 061/300 Write block 3423 len 4096 data: 4
6569 062/300 Read block 4430 len 32768
6570 063/300 Read block 3603 len 24576
6571 064/300 Flush
6572 065/300 Write block 489 len 28672 data: 3 4 3 2 2 2 3
6573 066/300 Read block 4206 len 12288
6574 067/300 Read block 533 len 32768
6575 068/300 Read block 725 len 16384
6576 069/300 Flush
6577 070/300 Write block 2803 len 16384 data: 3 3 2 3
6578 071/300 Write block 4915 len 8192 data: 2 2
6579 072/300 Flush
6580 073/300 Read block 1616 len 40960
6581 074/300 Read block 4678 len 32768
6582 075/300 Read block 4682 len 28672
6583 076/300 Read block 2909 len 40960
6584 077/300 Read block 767 len 12288
6585 078/300 Write block 2039 len 20480 data: 3 4 4 4 4
6586 079/300 Read block 4052 len 32768
6587 080/300 Write block 3387 len 16384 data: 2 2 2 3
6588 081/300 Write block 293 len 24576 data: 2 2 4 3 3 3
6589 082/300 Write block 1399 len 36864 data: 3 3 3 3 3 3 3 3 3
6590 083/300 Write block 78 len 8192 data: 2 2
6591 084/300 Read block 9 len 12288
6592 085/300 Read block 2512 len 36864
6593 086/300 Read block 460 len 36864
6594 087/300 Flush
6595 088/300 Write block 4152 len 28672 data: 2 2 2 3 2 2 2
6596 089/300 Write block 825 len 12288 data: 2 2 3
6597 090/300 Read block 3364 len 24576
6598 091/300 Read block 3457 len 40960
6599 092/300 Write block 1920 len 12288 data: 3 4 4
6600 093/300 Write block 1954 len 28672 data: 4 3 3 2 2 2 2
6601 094/300 Write block 3978 len 8192 data: 3 4
6602 095/300 Read block 4352 len 40960
6603 096/300 Read block 2447 len 16384
6604 097/300 Write block 3261 len 20480 data: 2 3 3 3 3
6605 098/300 Write block 66 len 40960 data: 3 4 4 5 5 4 3 3 3 2
6606 099/300 Read block 3832 len 20480
6607 100/300 Flush
6608 101/300 Read block 4294 len 24576
6609 102/300 Write block 1356 len 36864 data: 3 3 4 4 3 3 3 3 3
6610 103/300 Read block 1572 len 28672
6611 104/300 Read block 296 len 36864
6612 105/300 Read block 3615 len 4096
6613 106/300 Read block 2973 len 40960
6614 107/300 Read block 3809 len 32768
6615 108/300 Read block 1386 len 32768
6616 109/300 Read block 4672 len 28672
6617 110/300 Write block 4091 len 12288 data: 3 4 4
6618 111/300 Write block 4092 len 24576 data: 5 5 3 3 3 3
6619 112/300 Read block 1802 len 36864
6620 113/300 Write block 1461 len 20480 data: 2 2 2 2 2
6621 114/300 Read block 420 len 8192
6622 115/300 Read block 3695 len 40960
6623 116/300 Read block 680 len 20480
6624 117/300 Write block 4096 len 28672 data: 4 4 4 3 3 3 3
6625 118/300 Read block 1178 len 12288
6626 119/300 Write block 577 len 4096 data: 3
6627 120/300 Read block 959 len 8192
6628 121/300 Read block 169 len 36864
6629 122/300 Write block 1276 len 36864 data: 2 2 2 2 2 2 2 2 2
6630 123/300 Read block 77 len 40960
6631 124/300 Read block 851 len 16384
6632 125/300 Read block 2888 len 16384
6633 126/300 Read block 2305 len 4096
6634 127/300 Read block 3254 len 40960
6635 128/300 Read block 1019 len 20480
6636 129/300 Read block 2304 len 16384
6637 130/300 Write block 3695 len 36864 data: 3 3 2 2 4 4 3 3 3
6638 131/300 Read block 1942 len 4096
6639 132/300 Write block 4918 len 24576 data: 2 2 2 2 2 2
6640 133/300 Write block 4827 len 28672 data: 2 2 2 2 2 2 2
6641 134/300 Read block 2187 len 16384
6642 135/300 Read block 1873 len 40960
6643 136/300 Read block 4571 len 32768
6644 137/300 Write block 3033 len 16384 data: 3 3 3 3
6645 138/300 Write block 4241 len 12288 data: 3 3 2
6646 139/300 Flush
6647 140/300 Write block 4576 len 28672 data: 4 4 4 4 4 3 3
6648 141/300 Read block 1480 len 16384
6649 142/300 Read block 715 len 28672
6650 143/300 Read block 1644 len 8192
6651 144/300 Write block 3112 len 24576 data: 2 2 2 2 3 4
6652 145/300 Write block 1158 len 28672 data: 4 3 3 3 2 2 2
6653 146/300 Write block 2511 len 12288 data: 3 3 3
6654 147/300 Read block 2774 len 40960
6655 148/300 Write block 4725 len 36864 data: 2 2 2 2 2 2 2 2 2
6656 149/300 Read block 4507 len 28672
6657 150/300 Flush
6658 151/300 Write block 1437 len 8192 data: 3 3
6659 152/300 Read block 3539 len 32768
6660 153/300 Flush
6661 154/300 Write block 3816 len 12288 data: 2 3 4
6662 155/300 Read block 3196 len 20480
6663 156/300 Read block 381 len 16384
6664 157/300 Write block 295 len 12288 data: 5 4 4
6665 158/300 Write block 2390 len 24576 data: 3 3 3 3 2 2
6666 159/300 Read block 310 len 28672
6667 160/300 Flush
6668 161/300 Read block 3429 len 16384
6669 162/300 Write block 2440 len 40960 data: 3 3 4 4 4 4 4 3 3 3
6670 163/300 Write block 3648 len 24576 data: 4 4 5 5 4 4
6671 164/300 Read block 4312 len 20480
6672 165/300 Write block 1237 len 20480 data: 3 3 3 2 2
6673 166/300 Read block 2428 len 16384
6674 167/300 Read block 1097 len 8192
6675 168/300 Read block 2142 len 20480
6676 169/300 Read block 1037 len 28672
6677 170/300 Read block 2159 len 36864
6678 171/300 Read block 3124 len 28672
6679 172/300 Write block 443 len 8192 data: 3 3
6680 173/300 Write block 3594 len 12288 data: 5 4 4
6681 174/300 Read block 2208 len 20480
6682 175/300 Read block 3961 len 8192
6683 176/300 Write block 4261 len 20480 data: 3 3 3 3 3
6684 177/300 Read block 2212 len 36864
6685 178/300 Write block 1113 len 28672 data: 2 2 2 2 2 2 2
6686 179/300 Write block 1759 len 20480 data: 3 2 2 2 2
6687 180/300 Write block 1359 len 24576 data: 5 4 4 4 4 4
6688 181/300 Write block 339 len 12288 data: 6 6 5
6689 182/300 Write block 772 len 32768 data: 3 3 3 2 2 2 2 2
6690 183/300 Write block 4755 len 40960 data: 3 3 3 3 4 3 3 3 3 4
6691 184/300 Read block 2560 len 24576
6692 185/300 Read block 2678 len 4096
6693 186/300 Write block 3138 len 4096 data: 3
6694 187/300 Write block 661 len 20480 data: 2 3 2 3 3
6695 188/300 Read block 2815 len 20480
6696 189/300 Write block 3604 len 4096 data: 2
6697 190/300 Write block 3555 len 16384 data: 3 3 2 2
6698 191/300 Read block 363 len 28672
6699 192/300 Flush
6700 193/300 Read block 653 len 32768
6701 194/300 Flush
6702 195/300 Write block 2735 len 24576 data: 2 2 2 2 2 2
6703 196/300 Read block 2309 len 20480
6704 197/300 Read block 1001 len 36864
6705 198/300 Read block 1267 len 20480
6706 199/300 Read block 2676 len 4096
6707 200/300 Write block 4196 len 4096 data: 5
6708 201/300 Write block 2429 len 32768 data: 3 4 5 4 4 5 5 4
6709 202/300 Read block 4488 len 16384
6710 203/300 Read block 2392 len 4096
6711 204/300 Write block 4905 len 40960 data: 2 2 2 2 2 3 2 2 2 2
6712 205/300 Write block 605 len 8192 data: 4 4
6713 206/300 Write block 4688 len 8192 data: 4 4
6714 207/300 Read block 3042 len 24576
6715 208/300 Read block 1848 len 20480
6716 209/300 Read block 443 len 36864
6717 210/300 Write block 4683 len 4096 data: 4
6718 211/300 Read block 1620 len 32768
6719 212/300 Write block 3779 len 12288 data: 4 5 5
6720 213/300 Flush
6721 214/300 Read block 4884 len 20480
6722 215/300 Flush
6723 216/300 Read block 2313 len 32768
6724 217/300 Write block 868 len 20480 data: 3 3 3 2 2
6725 218/300 Read block 2430 len 8192
6726 219/300 Read block 2473 len 28672
6727 220/300 Read block 1430 len 36864
6728 221/300 Read block 1444 len 4096
6729 222/300 Write block 1846 len 16384 data: 4 4 3 3
6730 223/300 Write block 4191 len 36864 data: 5 4 4 4 5 6 6 6 4
6731 224/300 Read block 552 len 20480
6732 225/300 Read block 3750 len 12288
6733 226/300 Write block 2666 len 8192 data: 2 2
6734 227/300 Read block 272 len 36864
6735 228/300 Write block 4574 len 4096 data: 4
6736 229/300 Write block 1038 len 12288 data: 4 3 2
6737 230/300 Flush
6738 231/300 Read block 280 len 8192
6739 232/300 Write block 3224 len 8192 data: 2 4
6740 233/300 Read block 366 len 8192
6741 234/300 Read block 2005 len 20480
6742 235/300 Read block 4029 len 8192
6743 236/300 Read block 4273 len 28672
6744 237/300 Read block 4345 len 20480
6745 238/300 Flush
6746 239/300 Write block 558 len 40960 data: 2 2 2 2 2 2 2 3 2 2
6747 240/300 Read block 1015 len 12288
6748 241/300 Write block 1770 len 20480 data: 2 2 2 2 2
6749 242/300 Write block 2013 len 28672 data: 4 4 4 3 3 3 2
6750 243/300 Write block 965 len 4096 data: 3
6751 244/300 Write block 944 len 8192 data: 5 5
6752 245/300 Write block 4988 len 12288 data: 2 2 2
6753 246/300 Read block 1645 len 8192
6754 247/300 Flush
6755 248/300 Write block 282 len 32768 data: 4 4 4 3 3 3 3 3
6756 249/300 Write block 1303 len 16384 data: 3 3 3 2
6757 250/300 Write block 1095 len 12288 data: 4 4 4
6758 251/300 Write block 2648 len 28672 data: 2 2 2 2 2 2 2
6759 252/300 Write block 3342 len 32768 data: 3 3 3 3 3 2 3 3
6760 253/300 Flush
6761 254/300 Read block 541 len 12288
6762 255/300 Write block 4587 len 32768 data: 4 5 5 4 3 3 2 2
6763 256/300 Write block 1855 len 28672 data: 2 2 2 3 3 3 3
6764 257/300 Read block 531 len 32768
6765 258/300 Write block 3168 len 32768 data: 3 3 3 3 3 3 4 3
6766 259/300 Flush
6767 260/300 Read block 1608 len 12288
6768 261/300 Read block 715 len 8192
6769 262/300 Write block 1871 len 4096 data: 4
6770 263/300 Read block 3474 len 40960
6771 264/300 Read block 1134 len 24576
6772 265/300 Read block 3603 len 24576
6773 266/300 Write block 4334 len 16384 data: 5 4 4 4
6774 267/300 Read block 1673 len 40960
6775 268/300 Write block 4679 len 20480 data: 3 3 3 4 5
6776 269/300 Read block 1554 len 12288
6777 270/300 Read block 3877 len 20480
6778 271/300 Write block 385 len 28672 data: 2 2 2 2 2 2 2
6779 272/300 Read block 2691 len 40960
6780 273/300 Flush
6781 274/300 Write block 743 len 4096 data: 4
6782 275/300 Read block 3823 len 28672
6783 276/300 Flush
6784 277/300 Write block 2817 len 16384 data: 4 4 3 3
6785 278/300 Write block 404 len 36864 data: 3 3 3 2 2 2 3 4 3
6786 279/300 Flush
6787 280/300 Read block 4432 len 12288
6788 281/300 Read block 1831 len 20480
6789 282/300 Read block 3033 len 32768
6790 283/300 Write block 4930 len 20480 data: 2 2 3 3 3
6791 284/300 Flush
6792 285/300 Write block 569 len 24576 data: 2 3 3 3 3 3
6793 286/300 Read block 2173 len 4096
6794 287/300 Read block 902 len 28672
6795 288/300 Write block 2313 len 20480 data: 4 5 5 5 5
6796 289/300 Write block 3961 len 16384 data: 4 4 3 3
6797 290/300 Read block 2788 len 20480
6798 291/300 Write block 1609 len 32768 data: 5 3 3 3 3 3 3 2
6799 292/300 Read block 4728 len 20480
6800 293/300 Write block 2076 len 12288 data: 2 2 2
6801 294/300 Read block 3323 len 24576
6802 295/300 Write block 890 len 40960 data: 4 4 4 4 4 5 5 4 3 3
6803 296/300 Flush
6804 297/300 Write block 3356 len 32768 data: 3 3 3 3 3 2 3 3
6805 298/300 Read block 3041 len 24576
6806 299/300 Flush
6807 300/300 Write block 588 len 28672 data: 2 2 2 2 2 2 3
6808 Main task has work to do, go find it
6809 got dsc Start(1)
6810 start 1
6811 Mar 02 01:07:00.620 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7f65ff97-8177-445f-b838-09d9a2abfc7a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
6812 Replay: started 1, returned:Ok(())
6813 ----------------------------------------------------------------
6814 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6815 States: Active Offline Active
6816 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6817 2110 Acked 3109 Read 8 Done New Done true
6818 2111 Acked 3110 Write 4 Done New Done true
6819 2112 Acked 3111 Read 2 Done New Done false
6820 2113 Acked 3112 Write 4 Done New Done false
6821 2114 Acked 3113 Write 4 Done New Done false
6822 2115 Acked 3114 Write 7 Done New Done false
6823 2116 Acked 3115 Write 6 Done New Done false
6824 2117 Acked 3116 Read 10 Done New Done false
6825 2118 Acked 3117 Read 1 Done New Done false
6826 2119 Acked 3118 Read 9 Done New Done false
6827 2120 Acked 3119 Write 2 Done New Done false
6828 2121 Acked 3120 Read 3 Done New Done false
6829 2122 Acked 3121 Write 3 Done New Done false
6830 2123 Acked 3122 Write 6 Done New Done false
6831 2124 Acked 3123 Write 8 Done New Done false
6832 2125 Acked 3124 Read 9 Done New Done false
6833 2126 Acked 3125 Flush 0 Done New Done false
6834 2127 Acked 3126 Read 1 Done New Done false
6835 2128 Acked 3127 Read 8 Done New Done false
6836 2129 Acked 3128 Write 3 Done New Done false
6837 2130 Acked 3129 Read 5 Done New Done false
6838 2131 Acked 3130 Write 5 Done New Done false
6839 2132 Acked 3131 Flush 0 Done New Done false
6840 2133 Acked 3132 Write 6 Done New Done false
6841 2134 Acked 3133 Flush 0 Done New Done false
6842 2135 Acked 3134 Read 7 Done New Done false
6843 2136 Acked 3135 Write 1 Done New Done false
6844 2137 Acked 3136 Read 6 Done New Done false
6845 2138 Acked 3137 Write 8 Done New Done false
6846 2139 Acked 3138 Flush 0 Done New Done false
6847 2140 Acked 3139 Flush 0 Done New Done false
6848 2141 Acked 3140 Write 4 Done New Done false
6849 2142 Acked 3141 Write 2 Done New Done false
6850 2143 Acked 3142 Write 7 Done New Done false
6851 2144 Acked 3143 Read 2 Done New Done false
6852 2145 Acked 3144 Write 5 Done New Done false
6853 2146 Acked 3145 Read 5 Done New Done false
6854 2147 Acked 3146 Read 3 Done New Done false
6855 2148 Acked 3147 Write 3 Done New Done false
6856 2149 Acked 3148 Write 3 Done New Done false
6857 2150 Acked 3149 Write 10 Done New Done false
6858 2151 Acked 3150 Read 9 Done New Done false
6859 2152 Acked 3151 Flush 0 Done New Done false
6860 2153 Acked 3152 Write 10 Done New Done false
6861 2154 Acked 3153 Read 7 Done New Done false
6862 2155 Acked 3154 Flush 0 Done New Done false
6863 2156 Acked 3155 Read 7 Done New Done false
6864 2157 Acked 3156 Write 2 Done New Done false
6865 2158 Acked 3157 Read 10 Done New Done false
6866 2159 Acked 3158 Write 4 Done New Done false
6867 2160 Acked 3159 Write 6 Done New Done false
6868 2161 Acked 3160 Write 3 Done New Done false
6869 2162 Acked 3161 Read 7 Done New Done false
6870 2163 Acked 3162 Read 3 Done New Done false
6871 2164 Acked 3163 Read 5 Done New Done false
6872 2165 Acked 3164 Write 5 Done New Done false
6873 2166 Acked 3165 Write 6 Done New Done false
6874 2167 Acked 3166 Write 1 Done New Done false
6875 2168 Acked 3167 Write 3 Done New Done false
6876 2169 Acked 3168 Flush 0 Done New Done false
6877 2170 Acked 3169 Read 9 Done New Done false
6878 2171 Acked 3170 Write 1 Done New Done false
6879 2172 Acked 3171 Read 8 Done New Done false
6880 2173 Acked 3172 Read 6 Done New Done false
6881 2174 Acked 3173 Flush 0 Done New Done false
6882 2175 Acked 3174 Write 7 Done New Done false
6883 2176 Acked 3175 Read 3 Done New Done false
6884 2177 Acked 3176 Read 8 Done New Done false
6885 2178 Acked 3177 Read 4 Done New Done false
6886 2179 Acked 3178 Flush 0 Done New Done false
6887 2180 Acked 3179 Write 4 Done New Done false
6888 2181 Acked 3180 Write 2 Done New Done false
6889 2182 Acked 3181 Flush 0 Done New Done false
6890 2183 Acked 3182 Read 10 Done New Done false
6891 2184 Acked 3183 Read 8 Done New Done false
6892 2185 Acked 3184 Read 7 Done New Done false
6893 2186 Acked 3185 Read 10 Done New Done false
6894 2187 Acked 3186 Read 3 Done New Done false
6895 2188 Acked 3187 Write 5 Done New Done false
6896 2189 Acked 3188 Read 8 Done New Done false
6897 2190 Acked 3189 Write 4 Done New Done false
6898 2191 Acked 3190 Write 6 Done New Done false
6899 2192 Acked 3191 Write 9 Done New Done false
6900 2193 Acked 3192 Write 2 Done New Done false
6901 2194 Acked 3193 Read 3 Done New Done false
6902 2195 Acked 3194 Read 9 Done New Done false
6903 2196 Acked 3195 Read 9 Done New Done false
6904 2197 Acked 3196 Flush 0 Done New Done false
6905 2198 Acked 3197 Write 7 Done New Done false
6906 2199 Acked 3198 Write 3 Done New Done false
6907 2200 Acked 3199 Read 6 Done New Done false
6908 2201 Acked 3200 Read 10 Done New Done false
6909 2202 Acked 3201 Write 3 Done New Done false
6910 2203 Acked 3202 Write 7 Done New Done false
6911 2204 Acked 3203 Write 2 Done New Done false
6912 2205 Acked 3204 Read 10 Done New Done false
6913 2206 Acked 3205 Read 4 Done New Done false
6914 2207 Acked 3206 Write 5 Done New Done false
6915 2208 Acked 3207 Write 10 Done New Done false
6916 2209 Acked 3208 Read 5 Done New Done false
6917 2210 Acked 3209 Flush 0 Done New Done false
6918 2211 Acked 3210 Read 6 Done New Done false
6919 2212 Acked 3211 Write 9 Done New Done false
6920 2213 Acked 3212 Read 7 Done New Done false
6921 2214 Acked 3213 Read 9 Done New Done false
6922 2215 Acked 3214 Read 1 Done New Done false
6923 2216 Acked 3215 Read 10 Done New Done false
6924 2217 Acked 3216 Read 8 Done New Done false
6925 2218 Acked 3217 Read 8 Done New Done false
6926 2219 Acked 3218 Read 7 Done New Done false
6927 2220 Acked 3219 Write 3 Done New Done false
6928 2221 Acked 3220 Write 6 Done New Done false
6929 2222 Acked 3221 Read 9 Done New Done false
6930 2223 Acked 3222 Write 5 Done New Done false
6931 2224 Acked 3223 Read 2 Done New Done false
6932 2225 Acked 3224 Read 10 Done New Done false
6933 2226 Acked 3225 Read 5 Done New Done false
6934 2227 Acked 3226 Write 7 Done New Done false
6935 2228 Acked 3227 Read 3 Done New Done false
6936 2229 Acked 3228 Write 1 Done New Done false
6937 2230 Acked 3229 Read 2 Done New Done false
6938 2231 Acked 3230 Read 9 Done New Done false
6939 2232 Acked 3231 Write 9 Done New Done false
6940 2233 Acked 3232 Read 10 Done New Done false
6941 2234 Acked 3233 Read 4 Done New Done false
6942 2235 Acked 3234 Read 4 Done New Done false
6943 2236 Acked 3235 Read 1 Done New Done false
6944 2237 Acked 3236 Read 10 Done New Done false
6945 2238 Acked 3237 Read 5 Done New Done false
6946 2239 Acked 3238 Read 4 Done New Done false
6947 2240 Acked 3239 Write 9 Done New Done false
6948 2241 Acked 3240 Read 1 Done New Done false
6949 2242 Acked 3241 Write 6 Done New Done false
6950 2243 Acked 3242 Write 7 Done New Done false
6951 2244 Acked 3243 Read 4 Done New Done false
6952 2245 Acked 3244 Read 10 Done New Done false
6953 2246 Acked 3245 Read 8 Done New Done false
6954 2247 Acked 3246 Write 4 Done New Done false
6955 2248 Acked 3247 Write 3 Done New Done false
6956 2249 Acked 3248 Flush 0 Done New Done false
6957 2250 Acked 3249 Write 7 Done New Done false
6958 2251 Acked 3250 Read 4 Done New Done false
6959 2252 Acked 3251 Read 7 Done New Done false
6960 2253 Acked 3252 Read 2 Done New Done false
6961 2254 Acked 3253 Write 6 Done New Done false
6962 2255 Acked 3254 Write 7 Done New Done false
6963 2256 Acked 3255 Write 3 Done New Done false
6964 2257 Acked 3256 Read 10 Done New Done false
6965 2258 Acked 3257 Write 9 Done New Done false
6966 2259 Acked 3258 Read 7 Done New Done false
6967 2260 Acked 3259 Flush 0 Done New Done false
6968 2261 Acked 3260 Write 2 Done New Done false
6969 2262 Acked 3261 Read 8 Done New Done false
6970 2263 Acked 3262 Flush 0 Done New Done false
6971 2264 Acked 3263 Write 3 Done New Done false
6972 2265 Acked 3264 Read 5 Done New Done false
6973 2266 Acked 3265 Read 4 Done New Done false
6974 2267 Acked 3266 Write 3 Done New Done false
6975 2268 Acked 3267 Write 6 Done New Done false
6976 2269 Acked 3268 Read 7 Done New Done false
6977 2270 Acked 3269 Flush 0 Done New Done false
6978 2271 Acked 3270 Read 4 Done New Done false
6979 2272 Acked 3271 Write 10 Done New Done false
6980 2273 Acked 3272 Write 6 Done New Done false
6981 2274 Acked 3273 Read 5 Done New Done false
6982 2275 Acked 3274 Write 5 Done New Done false
6983 2276 Acked 3275 Read 4 Done New Done false
6984 2277 Acked 3276 Read 2 Done New Done false
6985 2278 Acked 3277 Read 5 Done New Done false
6986 2279 Acked 3278 Read 7 Done New Done false
6987 2280 Acked 3279 Read 9 Done New Done false
6988 2281 Acked 3280 Read 7 Done New Done false
6989 2282 Acked 3281 Write 2 Done New Done false
6990 2283 Acked 3282 Write 3 Done New Done false
6991 2284 Acked 3283 Read 5 Done New Done false
6992 2285 Acked 3284 Read 2 Done New Done false
6993 2286 Acked 3285 Write 5 Done New Done false
6994 2287 Acked 3286 Read 9 Done New Done false
6995 2288 Acked 3287 Write 7 Done New Done false
6996 2289 Acked 3288 Write 5 Done New Done false
6997 2290 Acked 3289 Write 6 Done New Done false
6998 2291 Acked 3290 Write 3 Done New Done false
6999 2292 Acked 3291 Write 8 Done New Done false
7000 2293 Acked 3292 Write 10 Done New Done false
7001 2294 Acked 3293 Read 6 Done New Done false
7002 2295 Acked 3294 Read 1 Done New Done false
7003 2296 Acked 3295 Write 1 Done New Done false
7004 2297 Acked 3296 Write 5 Done New Done false
7005 2298 Acked 3297 Read 5 Done New Done false
7006 2299 Acked 3298 Write 1 Done New Done false
7007 2300 Acked 3299 Write 4 Done New Done false
7008 2301 Acked 3300 Read 7 Done New Done false
7009 2302 Acked 3301 Flush 0 Done New Done false
7010 2303 Acked 3302 Read 8 Done New Done false
7011 2304 Acked 3303 Flush 0 Done New Done false
7012 2305 Acked 3304 Write 6 Done New Done false
7013 2306 Acked 3305 Read 5 Done New Done false
7014 2307 Acked 3306 Read 9 Done New Done false
7015 2308 Acked 3307 Read 5 Done New Done false
7016 2309 Acked 3308 Read 1 Done New Done false
7017 2310 Acked 3309 Write 1 Done New Done false
7018 2311 Acked 3310 Write 8 Done New Done false
7019 2312 Acked 3311 Read 4 Done New Done false
7020 2313 Acked 3312 Read 1 Done New Done false
7021 2314 Acked 3313 Write 10 Done New Done false
7022 2315 Acked 3314 Write 2 Done New Done false
7023 2316 Acked 3315 Write 2 Done New Done false
7024 2317 Acked 3316 Read 6 Done New Done false
7025 2318 Acked 3317 Read 5 Done New Done false
7026 2319 Acked 3318 Read 9 Done New Done false
7027 2320 Acked 3319 Write 1 Done New Done false
7028 2321 Acked 3320 Read 8 Done New Done false
7029 2322 Acked 3321 Write 3 Done New Done false
7030 2323 Acked 3322 Flush 0 Done New Done false
7031 2324 Acked 3323 Read 5 Done New Done false
7032 2325 Acked 3324 Flush 0 Done New Done false
7033 2326 Acked 3325 Read 8 Done New Done false
7034 2327 Acked 3326 Write 5 Done New Done false
7035 2328 Acked 3327 Read 2 Done New Done false
7036 2329 Acked 3328 Read 7 Done New Done false
7037 2330 Acked 3329 Read 9 Done New Done false
7038 2331 Acked 3330 Read 1 Done New Done false
7039 2332 Acked 3331 Write 4 Done New Done false
7040 2333 Acked 3332 Write 9 Done New Done false
7041 2334 Acked 3333 Read 5 Done New Done false
7042 2335 Acked 3334 Read 3 Done New Done false
7043 2336 Acked 3335 Write 2 Done New Done false
7044 2337 Acked 3336 Read 9 Done New Done false
7045 2338 Acked 3337 Write 1 Done New Done false
7046 2339 Acked 3338 Write 3 Done New Done false
7047 2340 Acked 3339 Flush 0 Done New Done false
7048 2341 Acked 3340 Read 2 Done New Done false
7049 2342 Acked 3341 Write 2 Done New Done false
7050 2343 Acked 3342 Read 2 Done New Done false
7051 2344 Acked 3343 Read 5 Done New Done false
7052 2345 Acked 3344 Read 2 Done New Done false
7053 2346 Acked 3345 Read 7 Done New Done false
7054 2347 Acked 3346 Read 5 Done New Done false
7055 2348 Acked 3347 Flush 0 Done New Done false
7056 2349 Acked 3348 Write 10 Done New Done false
7057 2350 Acked 3349 Read 3 Done New Done false
7058 2351 Acked 3350 Write 5 Done New Done false
7059 2352 Acked 3351 Write 7 Done New Done false
7060 2353 Acked 3352 Write 1 Done New Done false
7061 2354 Acked 3353 Write 2 Done New Done false
7062 2355 Acked 3354 Write 3 Done New Done false
7063 2356 Acked 3355 Read 2 Done New Done false
7064 2357 Acked 3356 Flush 0 Done New Done false
7065 2358 Acked 3357 Write 8 Done New Done false
7066 2359 Acked 3358 Write 4 Done New Done false
7067 2360 Acked 3359 Write 3 Done New Done false
7068 2361 Acked 3360 Write 7 Done New Done false
7069 2362 Acked 3361 Write 8 Done New Done false
7070 2363 Acked 3362 Flush 0 Done New Done false
7071 2364 Acked 3363 Read 3 Done New Done false
7072 2365 Acked 3364 Write 8 Done New Done false
7073 2366 Acked 3365 Write 7 Done New Done false
7074 2367 Acked 3366 Read 8 Done New Done false
7075 2368 Acked 3367 Write 8 Done New Done false
7076 2369 Acked 3368 Flush 0 Done New Done false
7077 2370 Acked 3369 Read 3 Done New Done false
7078 2371 Acked 3370 Read 2 Done New Done false
7079 2372 Acked 3371 Write 1 Done New Done false
7080 2373 Acked 3372 Read 10 Done New Done false
7081 2374 Acked 3373 Read 6 Done New Done false
7082 2375 Acked 3374 Read 6 Done New Done false
7083 2376 Acked 3375 Write 4 Done New Done false
7084 2377 Acked 3376 Read 10 Done New Done false
7085 2378 Acked 3377 Write 5 Done New Done false
7086 2379 Acked 3378 Read 3 Done New Done false
7087 2380 Acked 3379 Read 5 Done New Done false
7088 2381 Acked 3380 Write 7 Done New Done false
7089 2382 Acked 3381 Read 10 Done New Done false
7090 2383 Acked 3382 Flush 0 Done New Done false
7091 2384 Acked 3383 Write 1 Done New Done false
7092 2385 Acked 3384 Read 7 Done New Done false
7093 2386 Acked 3385 Flush 0 Done New Done false
7094 2387 Acked 3386 Write 4 Done New Done false
7095 2388 Acked 3387 Write 9 Done New Done false
7096 2389 Acked 3388 Flush 0 Done New Done false
7097 2390 Acked 3389 Read 3 Done New Done false
7098 2391 Acked 3390 Read 5 Done New Done false
7099 2392 Acked 3391 Read 8 Done New Done false
7100 2393 Acked 3392 Write 5 Done New Done false
7101 2394 Acked 3393 Flush 0 Done New Done false
7102 2395 Acked 3394 Write 6 Done New Done false
7103 2396 Acked 3395 Read 1 Done New Done false
7104 2397 Acked 3396 Read 7 Done New Done false
7105 2398 Acked 3397 Write 5 Done New Done false
7106 2399 Acked 3398 Write 4 Done New Done false
7107 2400 Acked 3399 Read 5 Done New Done false
7108 2401 Acked 3400 Write 8 Done New Done false
7109 2402 Acked 3401 Read 5 Done New Done false
7110 2403 Acked 3402 Write 3 Done New Done false
7111 2404 Acked 3403 Read 6 Done New Done false
7112 2405 Acked 3404 Write 10 Done New Done false
7113 2406 Acked 3405 Flush 0 Done New Done false
7114 2407 Acked 3406 Write 8 Done New Done false
7115 2408 Acked 3407 Read 6 Done New Done false
7116 2409 Acked 3408 Flush 0 Done New Done false
7117 2410 Acked 3409 Write 7 Done New Done false
7118 STATES DS:0 DS:1 DS:2 TOTAL
7119 New 0 301 0 301
7120 Sent 0 0 0 0
7121 Done 301 0 301 602
7122 Skipped 0 0 0 0
7123 Error 0 0 0 0
7124 Last Flush: 3408 3108 3408
7125 Downstairs last five completed: 3108 3107 3106 3105 3104
7126 Upstairs last five completed: 2410 2409 2408 2407 2406
7127 CLIENT: Up:0 ds:301 act:2
71282024-03-02T01:07:00.729ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71292024-03-02T01:07:00.729ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71302024-03-02T01:07:00.729ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71312024-03-02T01:07:00.729ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71322024-03-02T01:07:00.729ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71332024-03-02T01:07:00.729ZINFOcrucible: [1] client re-new 301 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71342024-03-02T01:07:00.729ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7135 [8820] got start action
7136 [8820] I am going to restart
7137 Starting downstairs at port 8820
7138 Make output file at "/tmp/dsc/downstairs-8820.txt"
7139 [8820][1] reports Starting
71402024-03-02T01:07:01.730ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71412024-03-02T01:07:01.731ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71422024-03-02T01:07:01.731ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71432024-03-02T01:07:01.731ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71442024-03-02T01:07:01.731ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71452024-03-02T01:07:01.731ZINFOcrucible: [1] client re-new 302 jobs since flush 3108 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71462024-03-02T01:07:01.731ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7147 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1170)
7148 [8820][1] reports Running
71492024-03-02T01:07:02.732ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71502024-03-02T01:07:02.732ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71512024-03-02T01:07:02.733ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71522024-03-02T01:07:02.733ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71532024-03-02T01:07:02.733ZINFOcrucible: send last flush ID to this DS: 3108 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71542024-03-02T01:07:02.734ZINFOcrucible: Replied this last flush ID: 3108 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71552024-03-02T01:07:02.734ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71562024-03-02T01:07:02.734ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71572024-03-02T01:07:02.734ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71582024-03-02T01:07:02.734ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7159 ----------------------------------------------------------------
7160 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7161 Downstairs last five completed: 3410 3409 3408 3407 3406
7162 Upstairs last five completed: 2411 2410 2409 2408 2407
7163 CLIENT: Up:0 ds:0 act:3
7164 Replay: All jobs finished, all DS active.
7165 Main task has work to do, go find it
7166 got dsc Stop(2)
7167 stop 2
7168 Mar 02 01:07:04.624 INFO request completed, latency_us: 374, response_code: 204, uri[8830] Got stop action so:false kr:false
7169 : /stop/cid/2, method: POST, req_id: 17eaadad-a048-4bcd-aa06-758e5aae9b72, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
7170 Mar 02 01:07:04.625 INFO request completed, latency_us: 136, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1c5e4a4c-e0d2-46a1-aa76-fd1301915503, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
71712024-03-02T01:07:04.625ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71722024-03-02T01:07:04.625ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71732024-03-02T01:07:04.625ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71742024-03-02T01:07:04.625ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71752024-03-02T01:07:04.625ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71762024-03-02T01:07:04.625ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7177 [8830] Exited with: ExitStatus(unix_wait_status(9))
7178 [8830][2] reports Exit
71792024-03-02T01:07:05.627ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71802024-03-02T01:07:05.627ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71812024-03-02T01:07:05.627ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71822024-03-02T01:07:05.627ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71832024-03-02T01:07:05.627ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71842024-03-02T01:07:05.627ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71852024-03-02T01:07:05.627ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71862024-03-02T01:07:06.629ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71872024-03-02T01:07:06.629ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71882024-03-02T01:07:06.629ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71892024-03-02T01:07:06.629ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71902024-03-02T01:07:06.629ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71912024-03-02T01:07:06.629ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71922024-03-02T01:07:06.629ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71932024-03-02T01:07:07.631ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71942024-03-02T01:07:07.631ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71952024-03-02T01:07:07.631ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71962024-03-02T01:07:07.631ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71972024-03-02T01:07:07.631ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71982024-03-02T01:07:07.631ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
71992024-03-02T01:07:07.631ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7200 Mar 02 01:07:08.627 INFO request completed, latency_us: 201, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8f461b35-59de-4b69-8531-09d97fe3a876, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
7201 001/300 Write block 1576 len 8192 data: 3 4
7202 002/300 Read block 4021 len 4096
7203 003/300 Write block 4270 len 32768 data: 2 2 2 2 2 2 2 3
7204 004/300 Write block 242 len 32768 data: 4 4 4 4 4 4 4 4
7205 005/300 Read block 4040 len 24576
72062024-03-02T01:07:08.631ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72072024-03-02T01:07:08.631ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72082024-03-02T01:07:08.631ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72092024-03-02T01:07:08.631ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72102024-03-02T01:07:08.631ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72112024-03-02T01:07:08.631ZINFOcrucible: [2] client re-new 5 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
72122024-03-02T01:07:08.631ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7213 006/300 Read block 2393 len 16384
7214 007/300 Write block 322 len 36864 data: 4 4 4 3 3 3 3 3 3
7215 008/300 Read block 4175 len 16384
7216 009/300 Read block 4889 len 32768
7217 010/300 Read block 4441 len 8192
7218 011/300 Read block 2370 len 36864
7219 012/300 Read block 3138 len 20480
7220 013/300 Read block 4967 len 24576
7221 014/300 Read block 2227 len 36864
7222 015/300 Write block 2069 len 20480 data: 3 3 2 2 2
7223 016/300 Write block 1160 len 8192 data: 4 4
7224 017/300 Read block 3554 len 28672
7225 018/300 Read block 3908 len 32768
7226 019/300 Write block 3771 len 24576 data: 3 3 3 2 2 3
7227 020/300 Flush
7228 021/300 Read block 2813 len 32768
7229 022/300 Read block 1003 len 32768
7230 023/300 Read block 2179 len 28672
7231 024/300 Read block 4424 len 4096
7232 025/300 Write block 2090 len 8192 data: 5 5
7233 026/300 Read block 4554 len 28672
7234 027/300 Read block 2882 len 8192
7235 028/300 Read block 4164 len 20480
7236 029/300 Read block 550 len 36864
7237 030/300 Read block 3458 len 20480
7238 031/300 Read block 2057 len 36864
7239 032/300 Read block 4704 len 4096
7240 033/300 Write block 2465 len 4096 data: 2
7241 034/300 Write block 2746 len 36864 data: 4 4 4 4 3 3 3 3 4
7242 035/300 Read block 494 len 36864
7243 036/300 Write block 1577 len 16384 data: 5 4 4 6
7244 037/300 Write block 2874 len 28672 data: 2 2 3 3 3 2 2
7245 038/300 Read block 4689 len 28672
7246 039/300 Read block 1270 len 28672
7247 040/300 Write block 3145 len 24576 data: 2 2 2 2 2 2
7248 041/300 Write block 334 len 40960 data: 5 5 6 6 6 7 7 6 4 3
7249 042/300 Write block 750 len 12288 data: 4 4 3
7250 043/300 Read block 2783 len 20480
7251 044/300 Flush
7252 045/300 Read block 1342 len 4096
7253 046/300 Read block 3795 len 36864
7254 047/300 Read block 3040 len 32768
7255 048/300 Flush
7256 049/300 Read block 4973 len 20480
7257 050/300 Read block 3817 len 16384
7258 051/300 Flush
7259 052/300 Read block 4973 len 8192
7260 053/300 Write block 2837 len 36864 data: 4 5 6 6 5 6 6 6 5
7261 054/300 Write block 1018 len 40960 data: 2 2 3 3 3 3 3 2 3 3
7262 055/300 Read block 3509 len 40960
7263 056/300 Read block 3947 len 8192
7264 057/300 Read block 4468 len 16384
7265 058/300 Write block 535 len 12288 data: 5 5 4
7266 059/300 Read block 3487 len 16384
7267 060/300 Read block 1646 len 16384
7268 061/300 Write block 1059 len 20480 data: 3 4 3 3 4
7269 062/300 Write block 1346 len 8192 data: 3 2
7270 063/300 Read block 1075 len 24576
7271 064/300 Read block 1222 len 4096
7272 065/300 Read block 4936 len 40960
7273 066/300 Read block 1899 len 36864
7274 067/300 Write block 971 len 4096 data: 2
7275 068/300 Write block 2748 len 32768 data: 5 5 4 4 4 4 5 4
7276 069/300 Flush
7277 070/300 Read block 3383 len 24576
7278 071/300 Read block 786 len 8192
7279 072/300 Write block 3685 len 12288 data: 3 2 2
7280 073/300 Read block 3294 len 20480
7281 074/300 Read block 4340 len 32768
7282 075/300 Read block 3507 len 36864
7283 076/300 Read block 1902 len 8192
7284 077/300 Write block 1426 len 40960 data: 3 2 2 3 4 4 3 2 2 2
7285 078/300 Read block 1503 len 16384
7286 079/300 Read block 636 len 4096
7287 080/300 Flush
7288 081/300 Flush
7289 082/300 Write block 1560 len 28672 data: 2 2 2 2 2 2 2
7290 083/300 Write block 2922 len 8192 data: 3 2
7291 084/300 Read block 1934 len 40960
7292 085/300 Read block 1117 len 12288
7293 086/300 Read block 2935 len 16384
7294 087/300 Write block 4978 len 36864 data: 3 3 3 3 3 3 2 2 2
7295 088/300 Write block 721 len 24576 data: 4 4 4 4 3 3
7296 089/300 Read block 1497 len 36864
7297 090/300 Write block 3187 len 8192 data: 4 4
7298 091/300 Flush
7299 092/300 Write block 1057 len 12288 data: 2 3 4
7300 093/300 Write block 1161 len 8192 data: 5 3
7301 094/300 Write block 2407 len 16384 data: 3 3 3 2
7302 095/300 Write block 1922 len 40960 data: 5 3 4 5 5 5 6 5 5 5
7303 096/300 Read block 2566 len 4096
7304 097/300 Read block 29 len 36864
7305 098/300 Read block 2892 len 20480
7306 099/300 Write block 2429 len 36864 data: 4 5 6 5 5 6 6 5 4
7307 100/300 Read block 4744 len 40960
7308 101/300 Write block 611 len 32768 data: 5 5 5 5 5 4 3 2
7309 102/300 Write block 4754 len 40960 data: 3 4 4 4 4 5 4 4 4 4
7310 103/300 Read block 2847 len 24576
7311 104/300 Write block 2456 len 32768 data: 4 3 3 4 4 3 2 2
7312 105/300 Read block 2906 len 40960
7313 106/300 Read block 4642 len 36864
7314 107/300 Read block 3713 len 12288
7315 108/300 Read block 1135 len 16384
7316 109/300 Read block 1935 len 32768
7317 110/300 Write block 377 len 12288 data: 2 3 3
7318 111/300 Flush
7319 112/300 Write block 447 len 36864 data: 4 4 4 3 3 4 4 4 2
7320 113/300 Write block 4986 len 4096 data: 3
7321 114/300 Read block 3539 len 24576
7322 115/300 Read block 4345 len 32768
7323 116/300 Read block 3400 len 28672
7324 117/300 Write block 1119 len 4096 data: 3
7325 118/300 Read block 262 len 40960
7326 119/300 Read block 279 len 32768
7327 120/300 Read block 2109 len 8192
7328 121/300 Read block 1487 len 8192
7329 122/300 Flush
7330 123/300 Write block 3057 len 32768 data: 3 3 2 3 4 4 4 3
7331 124/300 Read block 2629 len 36864
7332 125/300 Read block 1890 len 24576
7333 126/300 Write block 278 len 28672 data: 4 4 4 4 5 5 5
7334 127/300 Write block 388 len 24576 data: 3 3 3 3 2 2
7335 128/300 Read block 4555 len 32768
7336 129/300 Read block 4792 len 32768
7337 130/300 Read block 4614 len 16384
7338 131/300 Read block 1088 len 16384
7339 132/300 Read block 4676 len 4096
7340 133/300 Read block 4658 len 24576
7341 134/300 Read block 3274 len 8192
7342 135/300 Read block 3054 len 16384
7343 136/300 Read block 2932 len 20480
7344 137/300 Write block 1337 len 20480 data: 2 2 2 2 2
7345 138/300 Write block 3400 len 12288 data: 3 3 3
7346 139/300 Flush
7347 140/300 Write block 448 len 12288 data: 5 5 4
7348 141/300 Flush
7349 142/300 Read block 2995 len 40960
7350 143/300 Flush
7351 144/300 Write block 1416 len 32768 data: 3 4 5 5 5 4 4 4
7352 145/300 Write block 4541 len 8192 data: 4 5
7353 146/300 Read block 1811 len 36864
7354 147/300 Write block 2083 len 12288 data: 3 3 3
7355 148/300 Read block 2009 len 8192
7356 149/300 Read block 1366 len 4096
7357 150/300 Write block 2025 len 36864 data: 3 2 2 2 2 3 2 2 2
7358 151/300 Write block 661 len 8192 data: 3 4
7359 152/300 Write block 4825 len 16384 data: 2 2 3 3
7360 153/300 Write block 527 len 24576 data: 2 2 2 3 3 5
7361 154/300 Read block 2778 len 40960
7362 155/300 Write block 221 len 12288 data: 2 2 2
7363 156/300 Write block 1754 len 32768 data: 3 4 4 3 3 4 3 3
7364 157/300 Write block 2577 len 8192 data: 3 3
7365 158/300 Read block 2328 len 20480
7366 159/300 Read block 2491 len 4096
7367 160/300 Write block 413 len 8192 data: 3 3
7368 161/300 Read block 2166 len 8192
7369 162/300 Write block 368 len 12288 data: 2 2 2
7370 163/300 Read block 419 len 24576
7371 164/300 Write block 2927 len 8192 data: 2 2
7372 165/300 Read block 2262 len 8192
7373 166/300 Read block 4244 len 20480
7374 167/300 Write block 271 len 28672 data: 4 2 2 2 2 2 3
7375 168/300 Write block 1731 len 16384 data: 2 3 4 5
7376 169/300 Write block 3557 len 8192 data: 3 3
7377 170/300 Write block 805 len 4096 data: 2
7378 171/300 Write block 1050 len 16384 data: 3 4 4 4
7379 172/300 Flush
7380 173/300 Read block 2868 len 36864
7381 174/300 Write block 3369 len 36864 data: 4 5 5 5 4 3 3 2 2
7382 175/300 Read block 236 len 12288
7383 176/300 Flush
7384 177/300 Write block 2169 len 20480 data: 3 3 4 4 5
7385 178/300 Write block 4515 len 24576 data: 2 2 2 2 2 2
7386 179/300 Write block 2560 len 4096 data: 5
7387 180/300 Read block 4856 len 12288
7388 181/300 Read block 2971 len 36864
7389 182/300 Flush
7390 183/300 Write block 3523 len 24576 data: 4 4 3 3 3 4
7391 184/300 Read block 3980 len 28672
7392 185/300 Write block 3353 len 20480 data: 5 4 3 4 4
7393 186/300 Write block 637 len 8192 data: 4 4
7394 187/300 Write block 1112 len 28672 data: 2 3 3 3 3 3 3
7395 188/300 Read block 1627 len 32768
7396 189/300 Flush
7397 190/300 Read block 3138 len 8192
7398 191/300 Write block 1980 len 20480 data: 2 2 2 2 2
7399 192/300 Write block 523 len 8192 data: 3 2
7400 193/300 Read block 4411 len 12288
7401 194/300 Write block 15 len 12288 data: 2 2 2
7402 195/300 Read block 3582 len 8192
7403 196/300 Read block 3228 len 4096
7404 197/300 Read block 1473 len 16384
7405 198/300 Write block 4358 len 40960 data: 4 3 2 2 2 3 3 3 3 3
7406 199/300 Read block 1815 len 40960
7407 200/300 Read block 4754 len 40960
7408 201/300 Read block 3557 len 32768
7409 202/300 Read block 4864 len 20480
7410 203/300 Read block 3254 len 20480
7411 204/300 Write block 4792 len 12288 data: 5 5 4
7412 205/300 Write block 6 len 20480 data: 3 3 3 3 2
7413 206/300 Write block 692 len 8192 data: 2 2
7414 207/300 Write block 1209 len 36864 data: 2 2 2 2 2 2 2 2 2
7415 208/300 Read block 3103 len 24576
7416 209/300 Write block 1507 len 20480 data: 4 4 4 4 4
7417 210/300 Read block 815 len 24576
7418 211/300 Read block 1130 len 32768
7419 212/300 Flush
7420 213/300 Write block 1549 len 12288 data: 2 2 2
7421 214/300 Read block 1683 len 12288
7422 215/300 Read block 82 len 32768
7423 216/300 Flush
7424 217/300 Flush
7425 218/300 Read block 3098 len 36864
7426 219/300 Read block 4834 len 8192
7427 220/300 Read block 2609 len 12288
7428 221/300 Write block 295 len 40960 data: 6 5 5 4 2 2 2 2 3 4
7429 222/300 Read block 1299 len 4096
7430 223/300 Write block 2063 len 36864 data: 4 3 4 4 4 4 4 4 3
7431 224/300 Write block 1097 len 4096 data: 5
7432 225/300 Flush
7433 226/300 Flush
7434 227/300 Write block 918 len 28672 data: 2 2 2 2 3 5 5
7435 228/300 Write block 3282 len 4096 data: 4
7436 229/300 Write block 2588 len 12288 data: 3 3 3
7437 230/300 Read block 3737 len 40960
7438 231/300 Read block 4966 len 8192
7439 232/300 Flush
7440 233/300 Read block 439 len 20480
7441 234/300 Read block 1546 len 40960
7442 235/300 Write block 2001 len 36864 data: 2 2 2 2 2 2 3 3 3
7443 236/300 Read block 1442 len 16384
7444 237/300 Read block 1066 len 16384
7445 238/300 Write block 4085 len 20480 data: 2 2 2 2 3
7446 239/300 Read block 2758 len 36864
7447 240/300 Write block 843 len 24576 data: 3 3 2 2 2 2
7448 241/300 Read block 4955 len 8192
7449 242/300 Read block 4380 len 16384
7450 243/300 Write block 2689 len 36864 data: 2 2 2 2 2 2 2 3 3
7451 244/300 Write block 3650 len 24576 data: 6 6 5 5 4 4
7452 245/300 Write block 4716 len 4096 data: 3
7453 246/300 Read block 1426 len 12288
7454 247/300 Write block 2048 len 4096 data: 4
7455 248/300 Read block 4458 len 12288
7456 249/300 Read block 1505 len 40960
7457 250/300 Read block 58 len 28672
7458 251/300 Flush
7459 252/300 Flush
7460 253/300 Write block 2035 len 4096 data: 2
7461 254/300 Read block 2631 len 36864
7462 255/300 Write block 2886 len 40960 data: 3 3 3 2 2 2 2 3 3 3
7463 256/300 Read block 4063 len 8192
7464 257/300 Write block 4072 len 32768 data: 4 4 4 4 4 3 3 3
7465 258/300 Write block 1915 len 24576 data: 3 3 3 3 3 4
7466 259/300 Read block 4233 len 8192
7467 260/300 Write block 4402 len 16384 data: 3 3 3 3
7468 261/300 Read block 2020 len 36864
7469 262/300 Write block 1683 len 8192 data: 4 4
7470 263/300 Flush
7471 264/300 Read block 4343 len 16384
7472 265/300 Write block 4990 len 16384 data: 3 2 2 2
7473 266/300 Read block 3754 len 40960
7474 267/300 Write block 4162 len 4096 data: 3
7475 268/300 Flush
7476 269/300 Write block 1520 len 12288 data: 2 2 2
7477 270/300 Flush
7478 271/300 Flush
7479 272/300 Read block 483 len 28672
7480 273/300 Read block 4954 len 8192
7481 274/300 Write block 1944 len 8192 data: 3 3
7482 275/300 Read block 503 len 12288
7483 276/300 Write block 4359 len 8192 data: 4 3
7484 277/300 Write block 2270 len 8192 data: 3 3
7485 278/300 Write block 4454 len 16384 data: 3 3 3 3
7486 279/300 Write block 211 len 4096 data: 3
7487 280/300 Write block 3507 len 40960 data: 3 2 2 2 2 3 4 5 5 4
7488 281/300 Read block 3207 len 20480
7489 282/300 Write block 4701 len 12288 data: 2 2 2
7490 283/300 Read block 4827 len 28672
7491 284/300 Write block 4159 len 24576 data: 3 3 3 4 3 3
7492 285/300 Write block 772 len 4096 data: 4
7493 286/300 Flush
7494 287/300 Write block 1523 len 16384 data: 2 2 3 4
7495 288/300 Read block 3055 len 36864
7496 289/300 Read block 1548 len 16384
7497 290/300 Read block 3694 len 40960
7498 291/300 Read block 137 len 8192
7499 292/300 Flush
7500 293/300 Write block 323 len 40960 data: 5 5 4 4 4 4 4 4 3 4
7501 294/300 Write block 4702 len 24576 data: 3 3 2 2 2 2
7502 295/300 Write block 2251 len 36864 data: 6 6 4 4 3 3 3 4 3
7503 296/300 Write block 284 len 40960 data: 6 4 4 4 4 4 4 4 2 3
7504 297/300 Flush
7505 298/300 Write block 1098 len 28672 data: 4 4 4 4 3 2 2
7506 299/300 Read block 4275 len 20480
7507 300/300 Read block 1652 len 8192
7508 Main task has work to do, go find it
7509 got dsc Start(2)
7510 start 2
7511 Mar 02 01:07:08.961 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1d9429a8-7e0d-4dbb-9377-f23f104071e2, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
7512 Replay: started 2, returned:Ok(())
7513 ----------------------------------------------------------------
7514 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7515 States: Active Active Offline
7516 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7517 2412 Acked 3411 Write 2 Done Done New true
7518 2413 Acked 3412 Read 1 Done Done New true
7519 2414 Acked 3413 Write 8 Done Done New true
7520 2415 Acked 3414 Write 8 Done Done New true
7521 2416 Acked 3415 Read 6 Done Done New true
7522 2417 Acked 3416 Read 4 Done Done New false
7523 2418 Acked 3417 Write 9 Done Done New false
7524 2419 Acked 3418 Read 4 Done Done New false
7525 2420 Acked 3419 Read 8 Done Done New false
7526 2421 Acked 3420 Read 2 Done Done New false
7527 2422 Acked 3421 Read 9 Done Done New false
7528 2423 Acked 3422 Read 5 Done Done New false
7529 2424 Acked 3423 Read 6 Done Done New false
7530 2425 Acked 3424 Read 9 Done Done New false
7531 2426 Acked 3425 Write 5 Done Done New false
7532 2427 Acked 3426 Write 2 Done Done New false
7533 2428 Acked 3427 Read 7 Done Done New false
7534 2429 Acked 3428 Read 8 Done Done New false
7535 2430 Acked 3429 Write 6 Done Done New false
7536 2431 Acked 3430 Flush 0 Done Done New false
7537 2432 Acked 3431 Read 8 Done Done New false
7538 2433 Acked 3432 Read 8 Done Done New false
7539 2434 Acked 3433 Read 7 Done Done New false
7540 2435 Acked 3434 Read 1 Done Done New false
7541 2436 Acked 3435 Write 2 Done Done New false
7542 2437 Acked 3436 Read 7 Done Done New false
7543 2438 Acked 3437 Read 2 Done Done New false
7544 2439 Acked 3438 Read 5 Done Done New false
7545 2440 Acked 3439 Read 9 Done Done New false
7546 2441 Acked 3440 Read 5 Done Done New false
7547 2442 Acked 3441 Read 9 Done Done New false
7548 2443 Acked 3442 Read 1 Done Done New false
7549 2444 Acked 3443 Write 1 Done Done New false
7550 2445 Acked 3444 Write 9 Done Done New false
7551 2446 Acked 3445 Read 9 Done Done New false
7552 2447 Acked 3446 Write 4 Done Done New false
7553 2448 Acked 3447 Write 7 Done Done New false
7554 2449 Acked 3448 Read 7 Done Done New false
7555 2450 Acked 3449 Read 7 Done Done New false
7556 2451 Acked 3450 Write 6 Done Done New false
7557 2452 Acked 3451 Write 10 Done Done New false
7558 2453 Acked 3452 Write 3 Done Done New false
7559 2454 Acked 3453 Read 5 Done Done New false
7560 2455 Acked 3454 Flush 0 Done Done New false
7561 2456 Acked 3455 Read 1 Done Done New false
7562 2457 Acked 3456 Read 9 Done Done New false
7563 2458 Acked 3457 Read 8 Done Done New false
7564 2459 Acked 3458 Flush 0 Done Done New false
7565 2460 Acked 3459 Read 5 Done Done New false
7566 2461 Acked 3460 Read 4 Done Done New false
7567 2462 Acked 3461 Flush 0 Done Done New false
7568 2463 Acked 3462 Read 2 Done Done New false
7569 2464 Acked 3463 Write 9 Done Done New false
7570 2465 Acked 3464 Write 10 Done Done New false
7571 2466 Acked 3465 Read 10 Done Done New false
7572 2467 Acked 3466 Read 2 Done Done New false
7573 2468 Acked 3467 Read 4 Done Done New false
7574 2469 Acked 3468 Write 3 Done Done New false
7575 2470 Acked 3469 Read 4 Done Done New false
7576 2471 Acked 3470 Read 4 Done Done New false
7577 2472 Acked 3471 Write 5 Done Done New false
7578 2473 Acked 3472 Write 2 Done Done New false
7579 2474 Acked 3473 Read 6 Done Done New false
7580 2475 Acked 3474 Read 1 Done Done New false
7581 2476 Acked 3475 Read 10 Done Done New false
7582 2477 Acked 3476 Read 9 Done Done New false
7583 2478 Acked 3477 Write 1 Done Done New false
7584 2479 Acked 3478 Write 8 Done Done New false
7585 2480 Acked 3479 Flush 0 Done Done New false
7586 2481 Acked 3480 Read 6 Done Done New false
7587 2482 Acked 3481 Read 2 Done Done New false
7588 2483 Acked 3482 Write 3 Done Done New false
7589 2484 Acked 3483 Read 5 Done Done New false
7590 2485 Acked 3484 Read 8 Done Done New false
7591 2486 Acked 3485 Read 9 Done Done New false
7592 2487 Acked 3486 Read 2 Done Done New false
7593 2488 Acked 3487 Write 10 Done Done New false
7594 2489 Acked 3488 Read 4 Done Done New false
7595 2490 Acked 3489 Read 1 Done Done New false
7596 2491 Acked 3490 Flush 0 Done Done New false
7597 2492 Acked 3491 Flush 0 Done Done New false
7598 2493 Acked 3492 Write 7 Done Done New false
7599 2494 Acked 3493 Write 2 Done Done New false
7600 2495 Acked 3494 Read 10 Done Done New false
7601 2496 Acked 3495 Read 3 Done Done New false
7602 2497 Acked 3496 Read 4 Done Done New false
7603 2498 Acked 3497 Write 9 Done Done New false
7604 2499 Acked 3498 Write 6 Done Done New false
7605 2500 Acked 3499 Read 9 Done Done New false
7606 2501 Acked 3500 Write 2 Done Done New false
7607 2502 Acked 3501 Flush 0 Done Done New false
7608 2503 Acked 3502 Write 3 Done Done New false
7609 2504 Acked 3503 Write 2 Done Done New false
7610 2505 Acked 3504 Write 4 Done Done New false
7611 2506 Acked 3505 Write 10 Done Done New false
7612 2507 Acked 3506 Read 1 Done Done New false
7613 2508 Acked 3507 Read 9 Done Done New false
7614 2509 Acked 3508 Read 5 Done Done New false
7615 2510 Acked 3509 Write 9 Done Done New false
7616 2511 Acked 3510 Read 10 Done Done New false
7617 2512 Acked 3511 Write 8 Done Done New false
7618 2513 Acked 3512 Write 10 Done Done New false
7619 2514 Acked 3513 Read 6 Done Done New false
7620 2515 Acked 3514 Write 8 Done Done New false
7621 2516 Acked 3515 Read 10 Done Done New false
7622 2517 Acked 3516 Read 9 Done Done New false
7623 2518 Acked 3517 Read 3 Done Done New false
7624 2519 Acked 3518 Read 4 Done Done New false
7625 2520 Acked 3519 Read 8 Done Done New false
7626 2521 Acked 3520 Write 3 Done Done New false
7627 2522 Acked 3521 Flush 0 Done Done New false
7628 2523 Acked 3522 Write 9 Done Done New false
7629 2524 Acked 3523 Write 1 Done Done New false
7630 2525 Acked 3524 Read 6 Done Done New false
7631 2526 Acked 3525 Read 8 Done Done New false
7632 2527 Acked 3526 Read 7 Done Done New false
7633 2528 Acked 3527 Flush 0 Done Done New false
7634 2529 Acked 3528 Write 1 Done Done New false
7635 2530 Acked 3529 Read 10 Done Done New false
7636 2531 Acked 3530 Read 8 Done Done New false
7637 2532 Acked 3531 Read 2 Done Done New false
7638 2533 Acked 3532 Read 2 Done Done New false
7639 2534 Acked 3533 Flush 0 Done Done New false
7640 2535 Acked 3534 Write 8 Done Done New false
7641 2536 Acked 3535 Read 9 Done Done New false
7642 2537 Acked 3536 Read 6 Done Done New false
7643 2538 Acked 3537 Write 7 Done Done New false
7644 2539 Acked 3538 Write 6 Done Done New false
7645 2540 Acked 3539 Read 8 Done Done New false
7646 2541 Acked 3540 Read 8 Done Done New false
7647 2542 Acked 3541 Read 4 Done Done New false
7648 2543 Acked 3542 Read 4 Done Done New false
7649 2544 Acked 3543 Read 1 Done Done New false
7650 2545 Acked 3544 Read 6 Done Done New false
7651 2546 Acked 3545 Read 2 Done Done New false
7652 2547 Acked 3546 Read 4 Done Done New false
7653 2548 Acked 3547 Read 5 Done Done New false
7654 2549 Acked 3548 Write 5 Done Done New false
7655 2550 Acked 3549 Write 3 Done Done New false
7656 2551 Acked 3550 Flush 0 Done Done New false
7657 2552 Acked 3551 Write 3 Done Done New false
7658 2553 Acked 3552 Flush 0 Done Done New false
7659 2554 Acked 3553 Read 10 Done Done New false
7660 2555 Acked 3554 Flush 0 Done Done New false
7661 2556 Acked 3555 Write 8 Done Done New false
7662 2557 Acked 3556 Write 2 Done Done New false
7663 2558 Acked 3557 Read 9 Done Done New false
7664 2559 Acked 3558 Write 3 Done Done New false
7665 2560 Acked 3559 Read 2 Done Done New false
7666 2561 Acked 3560 Read 1 Done Done New false
7667 2562 Acked 3561 Write 9 Done Done New false
7668 2563 Acked 3562 Write 2 Done Done New false
7669 2564 Acked 3563 Write 4 Done Done New false
7670 2565 Acked 3564 Write 6 Done Done New false
7671 2566 Acked 3565 Read 10 Done Done New false
7672 2567 Acked 3566 Write 3 Done Done New false
7673 2568 Acked 3567 Write 8 Done Done New false
7674 2569 Acked 3568 Write 2 Done Done New false
7675 2570 Acked 3569 Read 5 Done Done New false
7676 2571 Acked 3570 Read 1 Done Done New false
7677 2572 Acked 3571 Write 2 Done Done New false
7678 2573 Acked 3572 Read 2 Done Done New false
7679 2574 Acked 3573 Write 3 Done Done New false
7680 2575 Acked 3574 Read 6 Done Done New false
7681 2576 Acked 3575 Write 2 Done Done New false
7682 2577 Acked 3576 Read 2 Done Done New false
7683 2578 Acked 3577 Read 5 Done Done New false
7684 2579 Acked 3578 Write 7 Done Done New false
7685 2580 Acked 3579 Write 4 Done Done New false
7686 2581 Acked 3580 Write 2 Done Done New false
7687 2582 Acked 3581 Write 1 Done Done New false
7688 2583 Acked 3582 Write 4 Done Done New false
7689 2584 Acked 3583 Flush 0 Done Done New false
7690 2585 Acked 3584 Read 9 Done Done New false
7691 2586 Acked 3585 Write 9 Done Done New false
7692 2587 Acked 3586 Read 3 Done Done New false
7693 2588 Acked 3587 Flush 0 Done Done New false
7694 2589 Acked 3588 Write 5 Done Done New false
7695 2590 Acked 3589 Write 6 Done Done New false
7696 2591 Acked 3590 Write 1 Done Done New false
7697 2592 Acked 3591 Read 3 Done Done New false
7698 2593 Acked 3592 Read 9 Done Done New false
7699 2594 Acked 3593 Flush 0 Done Done New false
7700 2595 Acked 3594 Write 6 Done Done New false
7701 2596 Acked 3595 Read 7 Done Done New false
7702 2597 Acked 3596 Write 5 Done Done New false
7703 2598 Acked 3597 Write 2 Done Done New false
7704 2599 Acked 3598 Write 7 Done Done New false
7705 2600 Acked 3599 Read 8 Done Done New false
7706 2601 Acked 3600 Flush 0 Done Done New false
7707 2602 Acked 3601 Read 2 Done Done New false
7708 2603 Acked 3602 Write 5 Done Done New false
7709 2604 Acked 3603 Write 2 Done Done New false
7710 2605 Acked 3604 Read 3 Done Done New false
7711 2606 Acked 3605 Write 3 Done Done New false
7712 2607 Acked 3606 Read 2 Done Done New false
7713 2608 Acked 3607 Read 1 Done Done New false
7714 2609 Acked 3608 Read 4 Done Done New false
7715 2610 Acked 3609 Write 10 Done Done New false
7716 2611 Acked 3610 Read 10 Done Done New false
7717 2612 Acked 3611 Read 10 Done Done New false
7718 2613 Acked 3612 Read 8 Done Done New false
7719 2614 Acked 3613 Read 5 Done Done New false
7720 2615 Acked 3614 Read 5 Done Done New false
7721 2616 Acked 3615 Write 3 Done Done New false
7722 2617 Acked 3616 Write 5 Done Done New false
7723 2618 Acked 3617 Write 2 Done Done New false
7724 2619 Acked 3618 Write 9 Done Done New false
7725 2620 Acked 3619 Read 6 Done Done New false
7726 2621 Acked 3620 Write 5 Done Done New false
7727 2622 Acked 3621 Read 6 Done Done New false
7728 2623 Acked 3622 Read 8 Done Done New false
7729 2624 Acked 3623 Flush 0 Done Done New false
7730 2625 Acked 3624 Write 3 Done Done New false
7731 2626 Acked 3625 Read 3 Done Done New false
7732 2627 Acked 3626 Read 8 Done Done New false
7733 2628 Acked 3627 Flush 0 Done Done New false
7734 2629 Acked 3628 Flush 0 Done Done New false
7735 2630 Acked 3629 Read 9 Done Done New false
7736 2631 Acked 3630 Read 2 Done Done New false
7737 2632 Acked 3631 Read 3 Done Done New false
7738 2633 Acked 3632 Write 10 Done Done New false
7739 2634 Acked 3633 Read 1 Done Done New false
7740 2635 Acked 3634 Write 9 Done Done New false
7741 2636 Acked 3635 Write 1 Done Done New false
7742 2637 Acked 3636 Flush 0 Done Done New false
7743 2638 Acked 3637 Flush 0 Done Done New false
7744 2639 Acked 3638 Write 7 Done Done New false
7745 2640 Acked 3639 Write 1 Done Done New false
7746 2641 Acked 3640 Write 3 Done Done New false
7747 2642 Acked 3641 Read 10 Done Done New false
7748 2643 Acked 3642 Read 2 Done Done New false
7749 2644 Acked 3643 Flush 0 Done Done New false
7750 2645 Acked 3644 Read 5 Done Done New false
7751 2646 Acked 3645 Read 10 Done Done New false
7752 2647 Acked 3646 Write 9 Done Done New false
7753 2648 Acked 3647 Read 4 Done Done New false
7754 2649 Acked 3648 Read 4 Done Done New false
7755 2650 Acked 3649 Write 5 Done Done New false
7756 2651 Acked 3650 Read 9 Done Done New false
7757 2652 Acked 3651 Write 6 Done Done New false
7758 2653 Acked 3652 Read 2 Done Done New false
7759 2654 Acked 3653 Read 4 Done Done New false
7760 2655 Acked 3654 Write 9 Done Done New false
7761 2656 Acked 3655 Write 6 Done Done New false
7762 2657 Acked 3656 Write 1 Done Done New false
7763 2658 Acked 3657 Read 3 Done Done New false
7764 2659 Acked 3658 Write 1 Done Done New false
7765 2660 Acked 3659 Read 3 Done Done New false
7766 2661 Acked 3660 Read 10 Done Done New false
7767 2662 Acked 3661 Read 7 Done Done New false
7768 2663 Acked 3662 Flush 0 Done Done New false
7769 2664 Acked 3663 Flush 0 Done Done New false
7770 2665 Acked 3664 Write 1 Done Done New false
7771 2666 Acked 3665 Read 9 Done Done New false
7772 2667 Acked 3666 Write 10 Done Done New false
7773 2668 Acked 3667 Read 2 Done Done New false
7774 2669 Acked 3668 Write 8 Done Done New false
7775 2670 Acked 3669 Write 6 Done Done New false
7776 2671 Acked 3670 Read 2 Done Done New false
7777 2672 Acked 3671 Write 4 Done Done New false
7778 2673 Acked 3672 Read 9 Done Done New false
7779 2674 Acked 3673 Write 2 Done Done New false
7780 2675 Acked 3674 Flush 0 Done Done New false
7781 2676 Acked 3675 Read 4 Done Done New false
7782 2677 Acked 3676 Write 4 Done Done New false
7783 2678 Acked 3677 Read 10 Done Done New false
7784 2679 Acked 3678 Write 1 Done Done New false
7785 2680 Acked 3679 Flush 0 Done Done New false
7786 2681 Acked 3680 Write 3 Done Done New false
7787 2682 Acked 3681 Flush 0 Done Done New false
7788 2683 Acked 3682 Flush 0 Done Done New false
7789 2684 Acked 3683 Read 7 Done Done New false
7790 2685 Acked 3684 Read 2 Done Done New false
7791 2686 Acked 3685 Write 2 Done Done New false
7792 2687 Acked 3686 Read 3 Done Done New false
7793 2688 Acked 3687 Write 2 Done Done New false
7794 2689 Acked 3688 Write 2 Done Done New false
7795 2690 Acked 3689 Write 4 Done Done New false
7796 2691 Acked 3690 Write 1 Done Done New false
7797 2692 Acked 3691 Write 10 Done Done New false
7798 2693 Acked 3692 Read 5 Done Done New false
7799 2694 Acked 3693 Write 3 Done Done New false
7800 2695 Acked 3694 Read 7 Done Done New false
7801 2696 Acked 3695 Write 6 Done Done New false
7802 2697 Acked 3696 Write 1 Done Done New false
7803 2698 Acked 3697 Flush 0 Done Done New false
7804 2699 Acked 3698 Write 4 Done Done New false
7805 2700 Acked 3699 Read 9 Done Done New false
7806 2701 Acked 3700 Read 4 Done Done New false
7807 2702 Acked 3701 Read 10 Done Done New false
7808 2703 Acked 3702 Read 2 Done Done New false
7809 2704 Acked 3703 Flush 0 Done Done New false
7810 2705 Acked 3704 Write 10 Done Done New false
7811 2706 Acked 3705 Write 6 Done Done New false
7812 2707 Acked 3706 Write 9 Done Done New false
7813 2708 Acked 3707 Write 10 Done Done New false
7814 2709 Acked 3708 Flush 0 Done Done New false
7815 2710 Acked 3709 Write 7 Done Done New false
7816 2711 Acked 3710 Read 5 Done Done New false
7817 2712 Acked 3711 Read 2 Done Done New false
7818 STATES DS:0 DS:1 DS:2 TOTAL
7819 New 0 0 301 301
7820 Sent 0 0 0 0
7821 Done 301 301 0 602
7822 Skipped 0 0 0 0
7823 Error 0 0 0 0
7824 Last Flush: 3708 3708 3410
7825 Downstairs last five completed: 3410 3409 3408 3407 3406
7826 Upstairs last five completed: 2712 2711 2710 2709 2708
7827 CLIENT: Up:0 ds:301 act:2
78282024-03-02T01:07:09.633ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78292024-03-02T01:07:09.633ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78302024-03-02T01:07:09.633ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78312024-03-02T01:07:09.633ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78322024-03-02T01:07:09.633ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78332024-03-02T01:07:09.633ZINFOcrucible: [2] client re-new 302 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78342024-03-02T01:07:09.633ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7835 [8830] got start action
7836 [8830] I am going to restart
7837 Starting downstairs at port 8830
7838 Make output file at "/tmp/dsc/downstairs-8830.txt"
7839 [8830][2] reports Starting
7840 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1171)
7841 [8830][2] reports Running
78422024-03-02T01:07:10.635ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78432024-03-02T01:07:10.635ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78442024-03-02T01:07:10.635ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78452024-03-02T01:07:10.635ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78462024-03-02T01:07:10.635ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78472024-03-02T01:07:10.635ZINFOcrucible: [2] client re-new 302 jobs since flush 3410 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78482024-03-02T01:07:10.635ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78492024-03-02T01:07:11.636ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78502024-03-02T01:07:11.637ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78512024-03-02T01:07:11.638ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78522024-03-02T01:07:11.638ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78532024-03-02T01:07:11.638ZINFOcrucible: send last flush ID to this DS: 3410 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78542024-03-02T01:07:11.638ZINFOcrucible: Replied this last flush ID: 3410 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78552024-03-02T01:07:11.638ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78562024-03-02T01:07:11.638ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78572024-03-02T01:07:11.638ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78582024-03-02T01:07:11.638ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7859 ----------------------------------------------------------------
7860 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7861 Downstairs last five completed: 3712 3711 3710 3709 3708
7862 Upstairs last five completed: 2713 2712 2711 2710 2709
7863 CLIENT: Up:0 ds:0 act:3
7864 Replay: All jobs finished, all DS active.
7865 Main task has work to do, go find it
7866 got dsc Stop(2)
7867 stop 2
7868 Mar 02 01:07:12.964 INFO request completed, latency_us: 348, [8830] Got stop action so:false kr:false
7869 response_code: 204, uri: /stop/cid/2, method: POST, req_id: 18c0e3f6-3a4b-44e5-a186-05d654a6fd73, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
7870 Mar 02 01:07:12.965 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5a300d97-e6c5-485f-ac39-94ea3f22c1d4, remote_addr: 127.0.0.1:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:07:12.965685463Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
7871 40735, local_addr: 127.0.0.1:9998
78722024-03-02T01:07:12.965ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78732024-03-02T01:07:12.965ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78742024-03-02T01:07:12.965ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78752024-03-02T01:07:12.965ZINFOcrucible: [2] client re-new 0 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78762024-03-02T01:07:12.965ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7877 [8830] Exited with: ExitStatus(unix_wait_status(9))
7878 [8830][2] reports Exit
78792024-03-02T01:07:13.967ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78802024-03-02T01:07:13.967ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78812024-03-02T01:07:13.967ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78822024-03-02T01:07:13.967ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78832024-03-02T01:07:13.967ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78842024-03-02T01:07:13.967ZINFOcrucible: [2] client re-new 0 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78852024-03-02T01:07:13.967ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78862024-03-02T01:07:14.968ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78872024-03-02T01:07:14.968ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78882024-03-02T01:07:14.968ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78892024-03-02T01:07:14.968ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78902024-03-02T01:07:14.968ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78912024-03-02T01:07:14.968ZINFOcrucible: [2] client re-new 0 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78922024-03-02T01:07:14.968ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78932024-03-02T01:07:15.970ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78942024-03-02T01:07:15.970ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78952024-03-02T01:07:15.970ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78962024-03-02T01:07:15.970ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78972024-03-02T01:07:15.970ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78982024-03-02T01:07:15.970ZINFOcrucible: [2] client re-new 0 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
78992024-03-02T01:07:15.970ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7900 Mar 02 01:07:16.968 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: aaae4b1a-ae55-415b-825a-e48f9a1a96da, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
7901 001/300 Read block 2870 len 4096
7902 002/300 Read block 342 len 24576
7903 003/300 Read block 4485 len 4096
79042024-03-02T01:07:16.970ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79052024-03-02T01:07:16.970ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79062024-03-02T01:07:16.971ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79072024-03-02T01:07:16.971ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79082024-03-02T01:07:16.971ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79092024-03-02T01:07:16.971ZINFOcrucible: [2] client re-new 3 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
79102024-03-02T01:07:16.971ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
7911 004/300 Write block 3731 len 28672 data: 2 2 2 2 2 2 2
7912 005/300 Write block 1383 len 40960 data: 3 3 3 3 3 3 2 2 2 2
7913 006/300 Read block 1446 len 40960
7914 007/300 Read block 37 len 12288
7915 008/300 Write block 4160 len 8192 data: 4 4
7916 009/300 Read block 256 len 36864
7917 010/300 Read block 4763 len 20480
7918 011/300 Write block 746 len 24576 data: 4 4 3 4 5 5
7919 012/300 Read block 2867 len 12288
7920 013/300 Write block 450 len 4096 data: 5
7921 014/300 Read block 4689 len 28672
7922 015/300 Write block 4670 len 12288 data: 2 2 2
7923 016/300 Write block 3575 len 12288 data: 2 2 2
7924 017/300 Write block 3520 len 20480 data: 3 3 4 5 5
7925 018/300 Write block 1833 len 40960 data: 4 4 4 3 4 3 4 4 4 4
7926 019/300 Read block 4805 len 16384
7927 020/300 Read block 6 len 20480
7928 021/300 Flush
7929 022/300 Write block 1053 len 20480 data: 5 4 4 4 3
7930 023/300 Read block 4450 len 40960
7931 024/300 Write block 3277 len 8192 data: 2 4
7932 025/300 Read block 982 len 12288
7933 026/300 Read block 3885 len 28672
7934 027/300 Write block 1988 len 4096 data: 2
7935 028/300 Read block 3338 len 16384
7936 029/300 Write block 1210 len 20480 data: 3 3 3 3 3
7937 030/300 Flush
7938 031/300 Read block 974 len 32768
7939 032/300 Read block 3588 len 4096
7940 033/300 Read block 247 len 36864
7941 034/300 Write block 3115 len 12288 data: 3 4 5
7942 035/300 Read block 540 len 8192
7943 036/300 Read block 4939 len 32768
7944 037/300 Write block 4414 len 16384 data: 2 2 2 2
7945 038/300 Read block 1107 len 28672
7946 039/300 Read block 4541 len 28672
7947 040/300 Flush
7948 041/300 Read block 4718 len 40960
7949 042/300 Read block 1273 len 28672
7950 043/300 Flush
7951 044/300 Write block 4628 len 28672 data: 2 2 2 2 2 2 2
7952 045/300 Flush
7953 046/300 Write block 4688 len 40960 data: 5 5 4 3 3 3 3 2 2 2
7954 047/300 Read block 258 len 40960
7955 048/300 Read block 3313 len 8192
7956 049/300 Read block 3302 len 20480
7957 050/300 Read block 1534 len 24576
7958 051/300 Write block 661 len 20480 data: 4 5 3 4 4
7959 052/300 Flush
7960 053/300 Read block 3930 len 16384
7961 054/300 Write block 4868 len 20480 data: 2 2 2 2 3
7962 055/300 Flush
7963 056/300 Flush
7964 057/300 Write block 3566 len 20480 data: 3 3 3 3 2
7965 058/300 Read block 4584 len 4096
7966 059/300 Write block 3858 len 36864 data: 3 4 4 4 4 5 4 4 3
7967 060/300 Write block 1203 len 32768 data: 5 5 4 4 2 2 3 4
7968 061/300 Read block 2669 len 16384
7969 062/300 Read block 1480 len 16384
7970 063/300 Write block 3171 len 4096 data: 4
7971 064/300 Read block 2061 len 32768
7972 065/300 Read block 1203 len 8192
7973 066/300 Flush
7974 067/300 Read block 938 len 4096
7975 068/300 Write block 1816 len 8192 data: 3 2
7976 069/300 Write block 1493 len 24576 data: 3 3 3 3 3 3
7977 070/300 Flush
7978 071/300 Write block 4222 len 8192 data: 3 3
7979 072/300 Read block 3354 len 32768
7980 073/300 Read block 4641 len 12288
7981 074/300 Write block 2079 len 24576 data: 3 3 3 4 4 4
7982 075/300 Write block 4678 len 40960 data: 2 4 4 4 5 6 5 5 5 4
7983 076/300 Write block 3550 len 32768 data: 3 4 5 4 4 4 4 4
7984 077/300 Write block 691 len 28672 data: 2 3 3 2 2 2 2
7985 078/300 Write block 3014 len 28672 data: 2 4 4 4 4 3 3
7986 079/300 Read block 985 len 4096
7987 080/300 Read block 4955 len 20480
7988 081/300 Write block 604 len 40960 data: 4 5 5 3 3 4 5 6 6 6
7989 082/300 Write block 1014 len 40960 data: 3 4 3 2 3 3 4 4 4 4
7990 083/300 Flush
7991 084/300 Read block 1136 len 32768
7992 085/300 Write block 394 len 8192 data: 2 2
7993 086/300 Read block 950 len 24576
7994 087/300 Write block 3407 len 32768 data: 3 3 2 2 2 3 3 5
7995 088/300 Flush
7996 089/300 Write block 2040 len 16384 data: 5 5 5 5
7997 090/300 Read block 1680 len 40960
7998 091/300 Write block 834 len 8192 data: 2 2
7999 092/300 Read block 793 len 32768
8000 093/300 Read block 3682 len 32768
8001 094/300 Read block 356 len 8192
8002 095/300 Write block 1320 len 20480 data: 4 4 5 3 3
8003 096/300 Read block 3630 len 32768
8004 097/300 Read block 4153 len 24576
8005 098/300 Read block 986 len 20480
8006 099/300 Write block 2718 len 28672 data: 5 4 5 4 4 5 4
8007 100/300 Read block 3469 len 16384
8008 101/300 Write block 101 len 4096 data: 2
8009 102/300 Read block 198 len 32768
8010 103/300 Read block 882 len 28672
8011 104/300 Write block 167 len 12288 data: 3 3 2
8012 105/300 Read block 3656 len 36864
8013 106/300 Read block 3996 len 32768
8014 107/300 Read block 3329 len 20480
8015 108/300 Write block 144 len 16384 data: 2 3 2 2
8016 109/300 Read block 2910 len 28672
8017 110/300 Read block 4783 len 40960
8018 111/300 Write block 4922 len 36864 data: 3 3 2 2 2 2 2 2 3
8019 112/300 Read block 4892 len 28672
8020 113/300 Read block 858 len 40960
8021 114/300 Flush
8022 115/300 Write block 1939 len 36864 data: 2 2 2 2 3 4 4 4 4
8023 116/300 Read block 1051 len 36864
8024 117/300 Read block 3311 len 32768
8025 118/300 Flush
8026 119/300 Read block 559 len 20480
8027 120/300 Flush
8028 121/300 Write block 3888 len 36864 data: 2 2 2 2 2 2 2 2 2
8029 122/300 Read block 1082 len 4096
8030 123/300 Write block 2713 len 36864 data: 3 3 3 5 5 6 5 6 5
8031 124/300 Write block 3764 len 8192 data: 3 4
8032 125/300 Read block 4835 len 28672
8033 126/300 Write block 1890 len 16384 data: 3 3 2 3
8034 127/300 Read block 4741 len 12288
8035 128/300 Write block 3678 len 16384 data: 4 4 4 4
8036 129/300 Write block 361 len 16384 data: 3 3 3 3
8037 130/300 Write block 3724 len 8192 data: 2 2
8038 131/300 Write block 3675 len 36864 data: 2 3 3 5 5 5 5 4 4
8039 132/300 Read block 3264 len 24576
8040 133/300 Read block 1028 len 40960
8041 134/300 Read block 1991 len 40960
8042 135/300 Write block 1691 len 32768 data: 2 3 3 3 3 3 3 3
8043 136/300 Read block 967 len 32768
8044 137/300 Read block 2765 len 12288
8045 138/300 Write block 2580 len 24576 data: 4 3 3 3 4 5
8046 139/300 Read block 1023 len 24576
8047 140/300 Read block 3398 len 20480
8048 141/300 Write block 587 len 24576 data: 2 3 3 3 3 3
8049 142/300 Read block 1206 len 4096
8050 143/300 Read block 3785 len 16384
8051 144/300 Write block 1741 len 8192 data: 4 2
8052 145/300 Read block 4691 len 12288
8053 146/300 Read block 1248 len 40960
8054 147/300 Flush
8055 148/300 Read block 132 len 4096
8056 149/300 Write block 414 len 40960 data: 4 3 3 3 3 2 2 2 2 2
8057 150/300 Write block 3712 len 28672 data: 4 4 4 3 3 3 3
8058 151/300 Write block 3707 len 32768 data: 4 4 4 4 4 5 5 5
8059 152/300 Read block 1436 len 32768
8060 153/300 Read block 2587 len 36864
8061 154/300 Read block 930 len 32768
8062 155/300 Read block 2017 len 32768
8063 156/300 Read block 3279 len 12288
8064 157/300 Write block 3802 len 40960 data: 4 4 4 5 5 5 6 6 6 3
8065 158/300 Write block 3365 len 28672 data: 3 4 4 4 5 6 6
8066 159/300 Write block 3293 len 4096 data: 4
8067 160/300 Read block 1626 len 20480
8068 161/300 Read block 4698 len 40960
8069 162/300 Read block 4471 len 16384
8070 163/300 Read block 4030 len 4096
8071 164/300 Write block 1264 len 12288 data: 3 3 2
8072 165/300 Read block 752 len 32768
8073 166/300 Write block 326 len 4096 data: 5
8074 167/300 Write block 2594 len 20480 data: 4 3 3 2 2
8075 168/300 Write block 4030 len 12288 data: 4 4 4
8076 169/300 Read block 2881 len 28672
8077 170/300 Write block 222 len 36864 data: 3 3 3 3 3 3 3 3 3
8078 171/300 Read block 2184 len 28672
8079 172/300 Read block 3425 len 12288
8080 173/300 Write block 3522 len 12288 data: 5 6 6
8081 174/300 Read block 761 len 24576
8082 175/300 Flush
8083 176/300 Read block 2204 len 24576
8084 177/300 Read block 145 len 4096
8085 178/300 Write block 3227 len 28672 data: 4 3 3 3 3 3 3
8086 179/300 Write block 2657 len 16384 data: 2 2 2 2
8087 180/300 Write block 4337 len 12288 data: 5 4 3
8088 181/300 Write block 3431 len 16384 data: 3 3 3 3
8089 182/300 Read block 3101 len 32768
8090 183/300 Read block 4775 len 40960
8091 184/300 Write block 378 len 24576 data: 4 4 2 2 2 2
8092 185/300 Write block 2993 len 12288 data: 5 6 5
8093 186/300 Read block 4395 len 32768
8094 187/300 Write block 1698 len 4096 data: 4
8095 188/300 Read block 2717 len 28672
8096 189/300 Read block 2676 len 32768
8097 190/300 Read block 3895 len 16384
8098 191/300 Write block 2793 len 20480 data: 4 4 3 4 4
8099 192/300 Flush
8100 193/300 Write block 2824 len 16384 data: 2 3 4 4
8101 194/300 Read block 1864 len 8192
8102 195/300 Read block 2191 len 36864
8103 196/300 Read block 4254 len 24576
8104 197/300 Read block 4268 len 8192
8105 198/300 Read block 3494 len 8192
8106 199/300 Write block 4156 len 12288 data: 3 3 3
8107 200/300 Read block 3565 len 8192
8108 201/300 Write block 3777 len 16384 data: 4 4 5 6
8109 202/300 Read block 2203 len 20480
8110 203/300 Write block 912 len 16384 data: 2 2 2 2
8111 204/300 Flush
8112 205/300 Write block 3000 len 4096 data: 3
8113 206/300 Read block 925 len 8192
8114 207/300 Write block 4022 len 4096 data: 3
8115 208/300 Write block 3375 len 24576 data: 4 3 3 2 2 2
8116 209/300 Read block 3758 len 8192
8117 210/300 Read block 3798 len 20480
8118 211/300 Read block 4861 len 36864
8119 212/300 Read block 4332 len 4096
8120 213/300 Write block 831 len 24576 data: 4 4 3 3 3 3
8121 214/300 Write block 3285 len 24576 data: 5 4 3 2 2 2
8122 215/300 Write block 2583 len 12288 data: 4 5 6
8123 216/300 Read block 4281 len 24576
8124 217/300 Read block 3338 len 20480
8125 218/300 Read block 3572 len 40960
8126 219/300 Read block 3991 len 20480
8127 220/300 Write block 368 len 36864 data: 3 3 3 3 3 3 3 3 2
8128 221/300 Read block 3626 len 12288
8129 222/300 Read block 607 len 24576
8130 223/300 Read block 224 len 20480
8131 224/300 Write block 3496 len 16384 data: 3 3 3 3
8132 225/300 Read block 2926 len 4096
8133 226/300 Read block 182 len 28672
8134 227/300 Write block 2271 len 40960 data: 4 4 3 3 3 4 4 3 3 2
8135 228/300 Read block 494 len 40960
8136 229/300 Read block 514 len 32768
8137 230/300 Write block 4723 len 40960 data: 3 3 3 3 3 3 3 3 3 3
8138 231/300 Read block 1043 len 8192
8139 232/300 Write block 1977 len 24576 data: 3 2 2 3 3 3
8140 233/300 Write block 269 len 36864 data: 4 4 5 3 3 3 3 3 4
8141 234/300 Write block 4792 len 24576 data: 6 6 5 4 4 4
8142 235/300 Flush
8143 236/300 Read block 3309 len 16384
8144 237/300 Write block 2819 len 16384 data: 4 4 2 2
8145 238/300 Read block 3763 len 28672
8146 239/300 Read block 3108 len 4096
8147 240/300 Read block 193 len 36864
8148 241/300 Write block 321 len 28672 data: 3 5 6 6 5 6 5
8149 242/300 Flush
8150 243/300 Read block 2706 len 40960
8151 244/300 Read block 3709 len 32768
8152 245/300 Read block 2526 len 8192
8153 246/300 Read block 1494 len 24576
8154 247/300 Read block 2360 len 8192
8155 248/300 Flush
8156 249/300 Write block 4812 len 20480 data: 3 3 2 2 3
8157 250/300 Read block 1937 len 8192
8158 251/300 Write block 57 len 20480 data: 3 2 2 2 3
8159 252/300 Read block 1653 len 4096
8160 253/300 Write block 4292 len 4096 data: 4
8161 254/300 Read block 2371 len 32768
8162 255/300 Write block 2099 len 40960 data: 3 3 3 3 3 3 3 2 2 2
8163 256/300 Read block 4105 len 32768
8164 257/300 Write block 4385 len 28672 data: 2 2 2 2 2 3 3
8165 258/300 Read block 959 len 8192
8166 259/300 Flush
8167 260/300 Flush
8168 261/300 Read block 1997 len 36864
8169 262/300 Write block 205 len 12288 data: 2 2 2
8170 263/300 Read block 391 len 40960
8171 264/300 Read block 4298 len 40960
8172 265/300 Read block 812 len 4096
8173 266/300 Flush
8174 267/300 Write block 2404 len 12288 data: 2 3 3
8175 268/300 Write block 1230 len 32768 data: 3 3 3 3 3 3 3 4
8176 269/300 Write block 3234 len 24576 data: 4 4 3 2 2 2
8177 270/300 Flush
8178 271/300 Read block 3040 len 20480
8179 272/300 Write block 1061 len 40960 data: 4 4 5 4 3 3 2 2 2 2
8180 273/300 Write block 4593 len 16384 data: 3 3 2 3
8181 274/300 Read block 2188 len 24576
8182 275/300 Write block 2689 len 16384 data: 3 3 3 3
8183 276/300 Read block 746 len 12288
8184 277/300 Write block 1895 len 24576 data: 4 3 3 3 3 3
8185 278/300 Read block 1049 len 28672
8186 279/300 Write block 147 len 8192 data: 3 2
8187 280/300 Flush
8188 281/300 Read block 1734 len 20480
8189 282/300 Read block 2074 len 36864
8190 283/300 Read block 2139 len 28672
8191 284/300 Read block 837 len 20480
8192 285/300 Read block 1568 len 8192
8193 286/300 Read block 1062 len 12288
8194 287/300 Write block 1689 len 28672 data: 3 3 3 4 4 4 4
8195 288/300 Read block 3553 len 8192
8196 289/300 Read block 357 len 20480
8197 290/300 Read block 4015 len 16384
8198 291/300 Read block 278 len 8192
8199 292/300 Read block 3126 len 4096
8200 293/300 Write block 1714 len 12288 data: 4 4 3
8201 294/300 Write block 2066 len 32768 data: 5 5 5 5 5 4 3 3
8202 295/300 Flush
8203 296/300 Write block 1155 len 12288 data: 3 3 3
8204 297/300 Read block 1993 len 36864
8205 298/300 Write block 3149 len 20480 data: 3 3 3 3 4
8206 299/300 Read block 768 len 20480
8207 300/300 Write block 3329 len 12288 data: 4 4 3
8208 Main task has work to do, go find it
8209 got dsc Start(2)
8210 start 2
8211 Mar 02 01:07:17.316 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: e1511257-ec01-40a0-be47-3f02924664d1, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
8212 Replay: started 2, returned:Ok(())
8213 ----------------------------------------------------------------
8214 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8215 States: Active Active Offline
8216 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8217 2714 Acked 3713 Read 1 Done Done New true
8218 2715 Acked 3714 Read 6 Done Done New true
8219 2716 Acked 3715 Read 1 Done Done New true
8220 2717 Acked 3716 Write 7 Done Done New false
8221 2718 Acked 3717 Write 10 Done Done New false
8222 2719 Acked 3718 Read 10 Done Done New false
8223 2720 Acked 3719 Read 3 Done Done New false
8224 2721 Acked 3720 Write 2 Done Done New false
8225 2722 Acked 3721 Read 9 Done Done New false
8226 2723 Acked 3722 Read 5 Done Done New false
8227 2724 Acked 3723 Write 6 Done Done New false
8228 2725 Acked 3724 Read 3 Done Done New false
8229 2726 Acked 3725 Write 1 Done Done New false
8230 2727 Acked 3726 Read 7 Done Done New false
8231 2728 Acked 3727 Write 3 Done Done New false
8232 2729 Acked 3728 Write 3 Done Done New false
8233 2730 Acked 3729 Write 5 Done Done New false
8234 2731 Acked 3730 Write 10 Done Done New false
8235 2732 Acked 3731 Read 4 Done Done New false
8236 2733 Acked 3732 Read 5 Done Done New false
8237 2734 Acked 3733 Flush 0 Done Done New false
8238 2735 Acked 3734 Write 5 Done Done New false
8239 2736 Acked 3735 Read 10 Done Done New false
8240 2737 Acked 3736 Write 2 Done Done New false
8241 2738 Acked 3737 Read 3 Done Done New false
8242 2739 Acked 3738 Read 7 Done Done New false
8243 2740 Acked 3739 Write 1 Done Done New false
8244 2741 Acked 3740 Read 4 Done Done New false
8245 2742 Acked 3741 Write 5 Done Done New false
8246 2743 Acked 3742 Flush 0 Done Done New false
8247 2744 Acked 3743 Read 8 Done Done New false
8248 2745 Acked 3744 Read 1 Done Done New false
8249 2746 Acked 3745 Read 9 Done Done New false
8250 2747 Acked 3746 Write 3 Done Done New false
8251 2748 Acked 3747 Read 2 Done Done New false
8252 2749 Acked 3748 Read 8 Done Done New false
8253 2750 Acked 3749 Write 4 Done Done New false
8254 2751 Acked 3750 Read 7 Done Done New false
8255 2752 Acked 3751 Read 7 Done Done New false
8256 2753 Acked 3752 Flush 0 Done Done New false
8257 2754 Acked 3753 Read 10 Done Done New false
8258 2755 Acked 3754 Read 7 Done Done New false
8259 2756 Acked 3755 Flush 0 Done Done New false
8260 2757 Acked 3756 Write 7 Done Done New false
8261 2758 Acked 3757 Flush 0 Done Done New false
8262 2759 Acked 3758 Write 10 Done Done New false
8263 2760 Acked 3759 Read 10 Done Done New false
8264 2761 Acked 3760 Read 2 Done Done New false
8265 2762 Acked 3761 Read 5 Done Done New false
8266 2763 Acked 3762 Read 6 Done Done New false
8267 2764 Acked 3763 Write 5 Done Done New false
8268 2765 Acked 3764 Flush 0 Done Done New false
8269 2766 Acked 3765 Read 4 Done Done New false
8270 2767 Acked 3766 Write 5 Done Done New false
8271 2768 Acked 3767 Flush 0 Done Done New false
8272 2769 Acked 3768 Flush 0 Done Done New false
8273 2770 Acked 3769 Write 5 Done Done New false
8274 2771 Acked 3770 Read 1 Done Done New false
8275 2772 Acked 3771 Write 9 Done Done New false
8276 2773 Acked 3772 Write 8 Done Done New false
8277 2774 Acked 3773 Read 4 Done Done New false
8278 2775 Acked 3774 Read 4 Done Done New false
8279 2776 Acked 3775 Write 1 Done Done New false
8280 2777 Acked 3776 Read 8 Done Done New false
8281 2778 Acked 3777 Read 2 Done Done New false
8282 2779 Acked 3778 Flush 0 Done Done New false
8283 2780 Acked 3779 Read 1 Done Done New false
8284 2781 Acked 3780 Write 2 Done Done New false
8285 2782 Acked 3781 Write 6 Done Done New false
8286 2783 Acked 3782 Flush 0 Done Done New false
8287 2784 Acked 3783 Write 2 Done Done New false
8288 2785 Acked 3784 Read 8 Done Done New false
8289 2786 Acked 3785 Read 3 Done Done New false
8290 2787 Acked 3786 Write 6 Done Done New false
8291 2788 Acked 3787 Write 10 Done Done New false
8292 2789 Acked 3788 Write 8 Done Done New false
8293 2790 Acked 3789 Write 7 Done Done New false
8294 2791 Acked 3790 Write 7 Done Done New false
8295 2792 Acked 3791 Read 1 Done Done New false
8296 2793 Acked 3792 Read 5 Done Done New false
8297 2794 Acked 3793 Write 10 Done Done New false
8298 2795 Acked 3794 Write 10 Done Done New false
8299 2796 Acked 3795 Flush 0 Done Done New false
8300 2797 Acked 3796 Read 8 Done Done New false
8301 2798 Acked 3797 Write 2 Done Done New false
8302 2799 Acked 3798 Read 6 Done Done New false
8303 2800 Acked 3799 Write 8 Done Done New false
8304 2801 Acked 3800 Flush 0 Done Done New false
8305 2802 Acked 3801 Write 4 Done Done New false
8306 2803 Acked 3802 Read 10 Done Done New false
8307 2804 Acked 3803 Write 2 Done Done New false
8308 2805 Acked 3804 Read 8 Done Done New false
8309 2806 Acked 3805 Read 8 Done Done New false
8310 2807 Acked 3806 Read 2 Done Done New false
8311 2808 Acked 3807 Write 5 Done Done New false
8312 2809 Acked 3808 Read 8 Done Done New false
8313 2810 Acked 3809 Read 6 Done Done New false
8314 2811 Acked 3810 Read 5 Done Done New false
8315 2812 Acked 3811 Write 7 Done Done New false
8316 2813 Acked 3812 Read 4 Done Done New false
8317 2814 Acked 3813 Write 1 Done Done New false
8318 2815 Acked 3814 Read 8 Done Done New false
8319 2816 Acked 3815 Read 7 Done Done New false
8320 2817 Acked 3816 Write 3 Done Done New false
8321 2818 Acked 3817 Read 9 Done Done New false
8322 2819 Acked 3818 Read 8 Done Done New false
8323 2820 Acked 3819 Read 5 Done Done New false
8324 2821 Acked 3820 Write 4 Done Done New false
8325 2822 Acked 3821 Read 7 Done Done New false
8326 2823 Acked 3822 Read 10 Done Done New false
8327 2824 Acked 3823 Write 9 Done Done New false
8328 2825 Acked 3824 Read 7 Done Done New false
8329 2826 Acked 3825 Read 10 Done Done New false
8330 2827 Acked 3826 Flush 0 Done Done New false
8331 2828 Acked 3827 Write 9 Done Done New false
8332 2829 Acked 3828 Read 9 Done Done New false
8333 2830 Acked 3829 Read 8 Done Done New false
8334 2831 Acked 3830 Flush 0 Done Done New false
8335 2832 Acked 3831 Read 5 Done Done New false
8336 2833 Acked 3832 Flush 0 Done Done New false
8337 2834 Acked 3833 Write 9 Done Done New false
8338 2835 Acked 3834 Read 1 Done Done New false
8339 2836 Acked 3835 Write 9 Done Done New false
8340 2837 Acked 3836 Write 2 Done Done New false
8341 2838 Acked 3837 Read 7 Done Done New false
8342 2839 Acked 3838 Write 4 Done Done New false
8343 2840 Acked 3839 Read 3 Done Done New false
8344 2841 Acked 3840 Write 4 Done Done New false
8345 2842 Acked 3841 Write 4 Done Done New false
8346 2843 Acked 3842 Write 2 Done Done New false
8347 2844 Acked 3843 Write 9 Done Done New false
8348 2845 Acked 3844 Read 6 Done Done New false
8349 2846 Acked 3845 Read 10 Done Done New false
8350 2847 Acked 3846 Read 10 Done Done New false
8351 2848 Acked 3847 Write 8 Done Done New false
8352 2849 Acked 3848 Read 8 Done Done New false
8353 2850 Acked 3849 Read 3 Done Done New false
8354 2851 Acked 3850 Write 6 Done Done New false
8355 2852 Acked 3851 Read 6 Done Done New false
8356 2853 Acked 3852 Read 5 Done Done New false
8357 2854 Acked 3853 Write 6 Done Done New false
8358 2855 Acked 3854 Read 1 Done Done New false
8359 2856 Acked 3855 Read 4 Done Done New false
8360 2857 Acked 3856 Write 2 Done Done New false
8361 2858 Acked 3857 Read 3 Done Done New false
8362 2859 Acked 3858 Read 10 Done Done New false
8363 2860 Acked 3859 Flush 0 Done Done New false
8364 2861 Acked 3860 Read 1 Done Done New false
8365 2862 Acked 3861 Write 10 Done Done New false
8366 2863 Acked 3862 Write 7 Done Done New false
8367 2864 Acked 3863 Write 8 Done Done New false
8368 2865 Acked 3864 Read 8 Done Done New false
8369 2866 Acked 3865 Read 9 Done Done New false
8370 2867 Acked 3866 Read 8 Done Done New false
8371 2868 Acked 3867 Read 8 Done Done New false
8372 2869 Acked 3868 Read 3 Done Done New false
8373 2870 Acked 3869 Write 10 Done Done New false
8374 2871 Acked 3870 Write 7 Done Done New false
8375 2872 Acked 3871 Write 1 Done Done New false
8376 2873 Acked 3872 Read 5 Done Done New false
8377 2874 Acked 3873 Read 10 Done Done New false
8378 2875 Acked 3874 Read 4 Done Done New false
8379 2876 Acked 3875 Read 1 Done Done New false
8380 2877 Acked 3876 Write 3 Done Done New false
8381 2878 Acked 3877 Read 8 Done Done New false
8382 2879 Acked 3878 Write 1 Done Done New false
8383 2880 Acked 3879 Write 5 Done Done New false
8384 2881 Acked 3880 Write 3 Done Done New false
8385 2882 Acked 3881 Read 7 Done Done New false
8386 2883 Acked 3882 Write 9 Done Done New false
8387 2884 Acked 3883 Read 7 Done Done New false
8388 2885 Acked 3884 Read 3 Done Done New false
8389 2886 Acked 3885 Write 3 Done Done New false
8390 2887 Acked 3886 Read 6 Done Done New false
8391 2888 Acked 3887 Flush 0 Done Done New false
8392 2889 Acked 3888 Read 6 Done Done New false
8393 2890 Acked 3889 Read 1 Done Done New false
8394 2891 Acked 3890 Write 7 Done Done New false
8395 2892 Acked 3891 Write 4 Done Done New false
8396 2893 Acked 3892 Write 3 Done Done New false
8397 2894 Acked 3893 Write 4 Done Done New false
8398 2895 Acked 3894 Read 8 Done Done New false
8399 2896 Acked 3895 Read 10 Done Done New false
8400 2897 Acked 3896 Write 6 Done Done New false
8401 2898 Acked 3897 Write 3 Done Done New false
8402 2899 Acked 3898 Read 8 Done Done New false
8403 2900 Acked 3899 Write 1 Done Done New false
8404 2901 Acked 3900 Read 7 Done Done New false
8405 2902 Acked 3901 Read 8 Done Done New false
8406 2903 Acked 3902 Read 4 Done Done New false
8407 2904 Acked 3903 Write 5 Done Done New false
8408 2905 Acked 3904 Flush 0 Done Done New false
8409 2906 Acked 3905 Write 4 Done Done New false
8410 2907 Acked 3906 Read 2 Done Done New false
8411 2908 Acked 3907 Read 9 Done Done New false
8412 2909 Acked 3908 Read 6 Done Done New false
8413 2910 Acked 3909 Read 2 Done Done New false
8414 2911 Acked 3910 Read 2 Done Done New false
8415 2912 Acked 3911 Write 3 Done Done New false
8416 2913 Acked 3912 Read 2 Done Done New false
8417 2914 Acked 3913 Write 4 Done Done New false
8418 2915 Acked 3914 Read 5 Done Done New false
8419 2916 Acked 3915 Write 4 Done Done New false
8420 2917 Acked 3916 Flush 0 Done Done New false
8421 2918 Acked 3917 Write 1 Done Done New false
8422 2919 Acked 3918 Read 2 Done Done New false
8423 2920 Acked 3919 Write 1 Done Done New false
8424 2921 Acked 3920 Write 6 Done Done New false
8425 2922 Acked 3921 Read 2 Done Done New false
8426 2923 Acked 3922 Read 5 Done Done New false
8427 2924 Acked 3923 Read 9 Done Done New false
8428 2925 Acked 3924 Read 1 Done Done New false
8429 2926 Acked 3925 Write 6 Done Done New false
8430 2927 Acked 3926 Write 6 Done Done New false
8431 2928 Acked 3927 Write 3 Done Done New false
8432 2929 Acked 3928 Read 6 Done Done New false
8433 2930 Acked 3929 Read 5 Done Done New false
8434 2931 Acked 3930 Read 10 Done Done New false
8435 2932 Acked 3931 Read 5 Done Done New false
8436 2933 Acked 3932 Write 9 Done Done New false
8437 2934 Acked 3933 Read 3 Done Done New false
8438 2935 Acked 3934 Read 6 Done Done New false
8439 2936 Acked 3935 Read 5 Done Done New false
8440 2937 Acked 3936 Write 4 Done Done New false
8441 2938 Acked 3937 Read 1 Done Done New false
8442 2939 Acked 3938 Read 7 Done Done New false
8443 2940 Acked 3939 Write 10 Done Done New false
8444 2941 Acked 3940 Read 10 Done Done New false
8445 2942 Acked 3941 Read 8 Done Done New false
8446 2943 Acked 3942 Write 10 Done Done New false
8447 2944 Acked 3943 Read 2 Done Done New false
8448 2945 Acked 3944 Write 6 Done Done New false
8449 2946 Acked 3945 Write 9 Done Done New false
8450 2947 Acked 3946 Write 6 Done Done New false
8451 2948 Acked 3947 Flush 0 Done Done New false
8452 2949 Acked 3948 Read 4 Done Done New false
8453 2950 Acked 3949 Write 4 Done Done New false
8454 2951 Acked 3950 Read 7 Done Done New false
8455 2952 Acked 3951 Read 1 Done Done New false
8456 2953 Acked 3952 Read 9 Done Done New false
8457 2954 Acked 3953 Write 7 Done Done New false
8458 2955 Acked 3954 Flush 0 Done Done New false
8459 2956 Acked 3955 Read 10 Done Done New false
8460 2957 Acked 3956 Read 8 Done Done New false
8461 2958 Acked 3957 Read 2 Done Done New false
8462 2959 Acked 3958 Read 6 Done Done New false
8463 2960 Acked 3959 Read 2 Done Done New false
8464 2961 Acked 3960 Flush 0 Done Done New false
8465 2962 Acked 3961 Write 5 Done Done New false
8466 2963 Acked 3962 Read 2 Done Done New false
8467 2964 Acked 3963 Write 5 Done Done New false
8468 2965 Acked 3964 Read 1 Done Done New false
8469 2966 Acked 3965 Write 1 Done Done New false
8470 2967 Acked 3966 Read 8 Done Done New false
8471 2968 Acked 3967 Write 10 Done Done New false
8472 2969 Acked 3968 Read 8 Done Done New false
8473 2970 Acked 3969 Write 7 Done Done New false
8474 2971 Acked 3970 Read 2 Done Done New false
8475 2972 Acked 3971 Flush 0 Done Done New false
8476 2973 Acked 3972 Flush 0 Done Done New false
8477 2974 Acked 3973 Read 9 Done Done New false
8478 2975 Acked 3974 Write 3 Done Done New false
8479 2976 Acked 3975 Read 10 Done Done New false
8480 2977 Acked 3976 Read 10 Done Done New false
8481 2978 Acked 3977 Read 1 Done Done New false
8482 2979 Acked 3978 Flush 0 Done Done New false
8483 2980 Acked 3979 Write 3 Done Done New false
8484 2981 Acked 3980 Flush 0 Done Done New false
8485 2982 Acked 3981 Write 8 Done Done New false
8486 2983 Acked 3982 Write 6 Done Done New false
8487 2984 Acked 3983 Flush 0 Done Done New false
8488 2985 Acked 3984 Read 5 Done Done New false
8489 2986 Acked 3985 Write 10 Done Done New false
8490 2987 Acked 3986 Write 4 Done Done New false
8491 2988 Acked 3987 Read 6 Done Done New false
8492 2989 Acked 3988 Write 4 Done Done New false
8493 2990 Acked 3989 Read 3 Done Done New false
8494 2991 Acked 3990 Write 6 Done Done New false
8495 2992 Acked 3991 Read 7 Done Done New false
8496 2993 Acked 3992 Write 2 Done Done New false
8497 2994 Acked 3993 Flush 0 Done Done New false
8498 2995 Acked 3994 Read 5 Done Done New false
8499 2996 Acked 3995 Read 9 Done Done New false
8500 2997 Acked 3996 Read 7 Done Done New false
8501 2998 Acked 3997 Read 5 Done Done New false
8502 2999 Acked 3998 Read 2 Done Done New false
8503 3000 Acked 3999 Read 3 Done Done New false
8504 3001 Acked 4000 Write 7 Done Done New false
8505 3002 Acked 4001 Read 2 Done Done New false
8506 3003 Acked 4002 Read 5 Done Done New false
8507 3004 Acked 4003 Read 4 Done Done New false
8508 3005 Acked 4004 Read 2 Done Done New false
8509 3006 Acked 4005 Read 1 Done Done New false
8510 3007 Acked 4006 Write 3 Done Done New false
8511 3008 Acked 4007 Write 8 Done Done New false
8512 3009 Acked 4008 Flush 0 Done Done New false
8513 3010 Acked 4009 Write 3 Done Done New false
8514 3011 Acked 4010 Read 9 Done Done New false
8515 3012 Acked 4011 Write 5 Done Done New false
8516 3013 Acked 4012 Read 5 Done Done New false
8517 3014 Acked 4013 Write 3 Done Done New false
8518 STATES DS:0 DS:1 DS:2 TOTAL
8519 New 0 0 301 301
8520 Sent 0 0 0 0
8521 Done 301 301 0 602
8522 Skipped 0 0 0 0
8523 Error 0 0 0 0
8524 Last Flush: 4008 4008 3712
8525 Downstairs last five completed: 3712 3711 3710 3709 3708
8526 Upstairs last five completed: 3014 3013 3012 3011 3010
8527 CLIENT: Up:0 ds:301 act:2
85282024-03-02T01:07:17.972ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85292024-03-02T01:07:17.972ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85302024-03-02T01:07:17.972ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85312024-03-02T01:07:17.972ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85322024-03-02T01:07:17.972ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85332024-03-02T01:07:17.972ZINFOcrucible: [2] client re-new 302 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85342024-03-02T01:07:17.972ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8535 [8830] got start action
8536 [8830] I am going to restart
8537 Starting downstairs at port 8830
8538 Make output file at "/tmp/dsc/downstairs-8830.txt"
8539 [8830][2] reports Starting
85402024-03-02T01:07:18.973ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85412024-03-02T01:07:18.973ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85422024-03-02T01:07:18.973ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85432024-03-02T01:07:18.973ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85442024-03-02T01:07:18.973ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85452024-03-02T01:07:18.973ZINFOcrucible: [2] client re-new 302 jobs since flush 3712 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85462024-03-02T01:07:18.974ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8547 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1172)
8548 [8830][2] reports Running
85492024-03-02T01:07:19.974ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85502024-03-02T01:07:19.975ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85512024-03-02T01:07:19.976ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85522024-03-02T01:07:19.976ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85532024-03-02T01:07:19.976ZINFOcrucible: send last flush ID to this DS: 3712 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85542024-03-02T01:07:19.976ZINFOcrucible: Replied this last flush ID: 3712 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85552024-03-02T01:07:19.976ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85562024-03-02T01:07:19.976ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85572024-03-02T01:07:19.976ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85582024-03-02T01:07:19.976ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8559 ----------------------------------------------------------------
8560 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8561 Downstairs last five completed: 4014 4013 4012 4011 4010
8562 Upstairs last five completed: 3015 3014 3013 3012 3011
8563 CLIENT: Up:0 ds:0 act:3
8564 Replay: All jobs finished, all DS active.
8565 Main task has work to do, go find it
8566 got dsc Stop(1)
8567 stop 1
8568 Mar 02 01:07:21.319 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/1, method[8820] Got stop action so:false kr:false
8569 : POST, req_id: af17a57a-8f48-4148-a5ed-60badb02517f, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
85702024-03-02T01:07:21.320ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85712024-03-02T01:07:21.320ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85722024-03-02T01:07:21.320ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85732024-03-02T01:07:21.320ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8574 Mar 02 01:07:21.320 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4681f517-a4b8-4fa8-82eb-b3fbc1d146e5, remote_addr: {"msg":"[1] client re-new 0 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:21.320249223Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
8575 127.0.0.1:40735, local_addr: 127.0.0.1:9998
85762024-03-02T01:07:21.320ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8577 [8820] Exited with: ExitStatus(unix_wait_status(9))
8578 [8820][1] reports Exit
85792024-03-02T01:07:22.320ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85802024-03-02T01:07:22.320ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85812024-03-02T01:07:22.320ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85822024-03-02T01:07:22.320ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85832024-03-02T01:07:22.320ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85842024-03-02T01:07:22.320ZINFOcrucible: [1] client re-new 0 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85852024-03-02T01:07:22.320ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85862024-03-02T01:07:23.321ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85872024-03-02T01:07:23.321ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85882024-03-02T01:07:23.321ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85892024-03-02T01:07:23.321ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85902024-03-02T01:07:23.321ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85912024-03-02T01:07:23.321ZINFOcrucible: [1] client re-new 0 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85922024-03-02T01:07:23.321ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85932024-03-02T01:07:24.322ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85942024-03-02T01:07:24.322ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85952024-03-02T01:07:24.323ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85962024-03-02T01:07:24.323ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85972024-03-02T01:07:24.323ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85982024-03-02T01:07:24.323ZINFOcrucible: [1] client re-new 0 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
85992024-03-02T01:07:24.323ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8600 Mar 02 01:07:25.323 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/1, method: GET, req_id: e681105e-d336-43bd-ba1f-fbcbfa5ebd72, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
8601 001/300 Read block 4759 len 4096
86022024-03-02T01:07:25.323ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86032024-03-02T01:07:25.323ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86042024-03-02T01:07:25.323ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86052024-03-02T01:07:25.323ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86062024-03-02T01:07:25.323ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86072024-03-02T01:07:25.323ZINFOcrucible: [1] client re-new 1 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
86082024-03-02T01:07:25.323ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
8609 002/300 Write block 2114 len 16384 data: 3 3 4 4
8610 003/300 Read block 2029 len 36864
8611 004/300 Read block 183 len 28672
8612 005/300 Write block 4712 len 8192 data: 3 4
8613 006/300 Read block 947 len 40960
8614 007/300 Write block 3101 len 8192 data: 2 2
8615 008/300 Read block 1170 len 8192
8616 009/300 Read block 3973 len 12288
8617 010/300 Read block 2463 len 36864
8618 011/300 Write block 813 len 40960 data: 7 8 6 5 6 6 5 4 4 3
8619 012/300 Flush
8620 013/300 Read block 3399 len 28672
8621 014/300 Write block 444 len 36864 data: 4 3 3 5 6 6 6 4 5
8622 015/300 Flush
8623 016/300 Write block 1591 len 8192 data: 2 2
8624 017/300 Write block 697 len 12288 data: 3 3 3
8625 018/300 Read block 372 len 20480
8626 019/300 Read block 2796 len 8192
8627 020/300 Write block 2335 len 12288 data: 3 3 3
8628 021/300 Read block 3340 len 20480
8629 022/300 Write block 3780 len 16384 data: 7 6 5 4
8630 023/300 Flush
8631 024/300 Read block 4771 len 4096
8632 025/300 Write block 1220 len 8192 data: 2 2
8633 026/300 Write block 3864 len 24576 data: 5 5 4 3 3 3
8634 027/300 Read block 4047 len 4096
8635 028/300 Write block 1855 len 32768 data: 3 3 3 4 4 4 4 3
8636 029/300 Write block 2649 len 36864 data: 3 3 3 3 3 3 2 2 3
8637 030/300 Write block 4107 len 16384 data: 2 3 3 3
8638 031/300 Write block 4729 len 4096 data: 4
8639 032/300 Read block 4017 len 24576
8640 033/300 Read block 1598 len 24576
8641 034/300 Read block 2016 len 36864
8642 035/300 Write block 4977 len 8192 data: 3 4
8643 036/300 Write block 202 len 12288 data: 2 2 2
8644 037/300 Write block 3340 len 24576 data: 2 2 4 4 4 4
8645 038/300 Read block 4108 len 8192
8646 039/300 Write block 3783 len 32768 data: 5 4 4 4 3 3 3 3
8647 040/300 Read block 313 len 40960
8648 041/300 Read block 832 len 24576
8649 042/300 Read block 1054 len 24576
8650 043/300 Read block 4790 len 16384
8651 044/300 Write block 3180 len 40960 data: 4 4 4 3 2 2 4 5 5 4
8652 045/300 Write block 851 len 12288 data: 2 2 2
8653 046/300 Flush
8654 047/300 Write block 2145 len 20480 data: 3 3 3 2 2
8655 048/300 Write block 2405 len 12288 data: 4 4 4
8656 049/300 Write block 733 len 8192 data: 5 5
8657 050/300 Read block 3842 len 16384
8658 051/300 Flush
8659 052/300 Read block 3698 len 8192
8660 053/300 Write block 891 len 24576 data: 5 5 5 5 6 6
8661 054/300 Read block 1603 len 16384
8662 055/300 Write block 2216 len 24576 data: 2 2 4 2 2 2
8663 056/300 Read block 3460 len 28672
8664 057/300 Read block 4229 len 8192
8665 058/300 Read block 1981 len 36864
8666 059/300 Read block 53 len 16384
8667 060/300 Write block 3135 len 36864 data: 3 3 3 4 3 3 3 3 3
8668 061/300 Write block 3715 len 36864 data: 4 4 4 4 2 2 2 3 2
8669 062/300 Read block 1722 len 20480
8670 063/300 Write block 1383 len 40960 data: 4 4 4 4 4 4 3 3 3 3
8671 064/300 Read block 1522 len 24576
8672 065/300 Read block 336 len 12288
8673 066/300 Write block 1974 len 16384 data: 3 3 3 4
8674 067/300 Write block 266 len 4096 data: 4
8675 068/300 Write block 3158 len 16384 data: 4 4 4 4
8676 069/300 Read block 3565 len 16384
8677 070/300 Flush
8678 071/300 Read block 633 len 28672
8679 072/300 Write block 3725 len 4096 data: 3
8680 073/300 Write block 1324 len 40960 data: 4 3 3 3 3 3 3 3 3 3
8681 074/300 Write block 4276 len 4096 data: 3
8682 075/300 Read block 3412 len 20480
8683 076/300 Write block 2497 len 28672 data: 2 2 3 3 3 4 3
8684 077/300 Read block 1690 len 12288
8685 078/300 Write block 201 len 8192 data: 2 3
8686 079/300 Read block 4035 len 16384
8687 080/300 Read block 2639 len 28672
8688 081/300 Write block 4280 len 4096 data: 4
8689 082/300 Write block 3309 len 8192 data: 2 2
8690 083/300 Flush
8691 084/300 Read block 271 len 24576
8692 085/300 Read block 2910 len 28672
8693 086/300 Write block 3848 len 20480 data: 3 3 3 4 4
8694 087/300 Read block 3747 len 16384
8695 088/300 Flush
8696 089/300 Read block 2645 len 4096
8697 090/300 Write block 1410 len 40960 data: 2 3 3 3 3 3 4 5 6 6
8698 091/300 Read block 386 len 28672
8699 092/300 Write block 1980 len 32768 data: 4 4 4 3 3 2 3 2
8700 093/300 Read block 1307 len 28672
8701 094/300 Write block 171 len 20480 data: 2 2 2 2 2
8702 095/300 Read block 127 len 4096
8703 096/300 Read block 2910 len 28672
8704 097/300 Flush
8705 098/300 Read block 376 len 28672
8706 099/300 Write block 3063 len 12288 data: 5 4 2
8707 100/300 Read block 4583 len 32768
8708 101/300 Read block 1949 len 12288
8709 102/300 Read block 2031 len 8192
8710 103/300 Write block 961 len 12288 data: 3 3 3
8711 104/300 Write block 4732 len 24576 data: 4 3 3 2 2 2
8712 105/300 Read block 2229 len 36864
8713 106/300 Write block 856 len 20480 data: 3 3 3 4 4
8714 107/300 Write block 4770 len 24576 data: 3 3 3 4 4 4
8715 108/300 Read block 4734 len 20480
8716 109/300 Write block 1017 len 16384 data: 3 4 4 5
8717 110/300 Read block 2883 len 4096
8718 111/300 Read block 2373 len 20480
8719 112/300 Write block 998 len 20480 data: 4 4 5 5 4
8720 113/300 Write block 3336 len 20480 data: 3 3 3 2 3
8721 114/300 Write block 3899 len 28672 data: 2 3 3 3 4 4 3
8722 115/300 Write block 4444 len 36864 data: 3 4 4 4 4 4 4 4 5
8723 116/300 Write block 4174 len 24576 data: 2 3 5 4 5 5
8724 117/300 Read block 1352 len 16384
8725 118/300 Read block 4456 len 20480
8726 119/300 Write block 702 len 12288 data: 4 4 3
8727 120/300 Write block 1945 len 4096 data: 5
8728 121/300 Write block 2524 len 8192 data: 3 4
8729 122/300 Write block 4783 len 40960 data: 2 2 2 2 2 3 3 3 4 7
8730 123/300 Write block 334 len 20480 data: 6 6 7 7 7
8731 124/300 Read block 431 len 16384
8732 125/300 Read block 2984 len 16384
8733 126/300 Write block 4400 len 8192 data: 3 3
8734 127/300 Read block 4328 len 28672
8735 128/300 Write block 3481 len 20480 data: 3 4 4 4 5
8736 129/300 Read block 2786 len 32768
8737 130/300 Read block 158 len 20480
8738 131/300 Flush
8739 132/300 Write block 4075 len 12288 data: 5 5 4
8740 133/300 Read block 2877 len 24576
8741 134/300 Read block 1614 len 16384
8742 135/300 Write block 1971 len 20480 data: 3 3 3 4 4
8743 136/300 Write block 4794 len 12288 data: 6 5 5
8744 137/300 Write block 2734 len 12288 data: 2 3 3
8745 138/300 Read block 1011 len 20480
8746 139/300 Read block 4616 len 40960
8747 140/300 Write block 317 len 32768 data: 2 2 2 2 4 6 7 7
8748 141/300 Read block 4359 len 12288
8749 142/300 Flush
8750 143/300 Write block 905 len 24576 data: 2 2 2 2 2 2
8751 144/300 Read block 1101 len 24576
8752 145/300 Write block 1441 len 40960 data: 2 2 4 4 4 4 4 4 3 3
8753 146/300 Read block 4378 len 8192
8754 147/300 Read block 2866 len 12288
8755 148/300 Read block 1557 len 4096
8756 149/300 Read block 2694 len 40960
8757 150/300 Read block 4411 len 24576
8758 151/300 Read block 3809 len 40960
8759 152/300 Write block 2910 len 32768 data: 3 3 3 3 4 4 3 3
8760 153/300 Read block 311 len 20480
8761 154/300 Read block 3167 len 40960
8762 155/300 Flush
8763 156/300 Read block 860 len 36864
8764 157/300 Read block 3544 len 40960
8765 158/300 Write block 1231 len 24576 data: 4 4 4 4 4 4
8766 159/300 Read block 3934 len 16384
8767 160/300 Read block 1978 len 40960
8768 161/300 Flush
8769 162/300 Write block 681 len 24576 data: 2 2 2 2 2 2
8770 163/300 Write block 3267 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8771 164/300 Read block 1192 len 12288
8772 165/300 Write block 3208 len 40960 data: 4 4 3 3 3 3 2 2 3 3
8773 166/300 Flush
8774 167/300 Read block 4798 len 40960
8775 168/300 Write block 2614 len 8192 data: 2 2
8776 169/300 Read block 4195 len 20480
8777 170/300 Read block 2662 len 40960
8778 171/300 Read block 2020 len 28672
8779 172/300 Read block 4333 len 20480
8780 173/300 Read block 97 len 16384
8781 174/300 Write block 2684 len 28672 data: 2 2 2 2 2 4 4
8782 175/300 Read block 2204 len 12288
8783 176/300 Read block 4327 len 36864
8784 177/300 Read block 424 len 36864
8785 178/300 Write block 2284 len 12288 data: 2 2 2
8786 179/300 Write block 911 len 12288 data: 2 3 3
8787 180/300 Read block 3140 len 8192
8788 181/300 Read block 2721 len 32768
8789 182/300 Read block 304 len 12288
8790 183/300 Write block 999 len 4096 data: 5
8791 184/300 Read block 2429 len 32768
8792 185/300 Write block 2884 len 12288 data: 2 3 4
8793 186/300 Read block 3530 len 28672
8794 187/300 Write block 342 len 36864 data: 5 4 2 2 2 2 2 2 3
8795 188/300 Read block 1688 len 16384
8796 189/300 Flush
8797 190/300 Write block 622 len 16384 data: 4 4 4 4
8798 191/300 Flush
8799 192/300 Read block 3798 len 20480
8800 193/300 Write block 821 len 32768 data: 5 4 2 2 3 3 4 4
8801 194/300 Write block 3719 len 28672 data: 3 3 3 4 3 3 4
8802 195/300 Write block 2420 len 24576 data: 2 3 3 3 3 2
8803 196/300 Write block 2652 len 8192 data: 4 4
8804 197/300 Read block 4691 len 36864
8805 198/300 Write block 1933 len 40960 data: 5 3 3 3 3 2 3 3 3 3
8806 199/300 Read block 4175 len 40960
8807 200/300 Read block 3019 len 36864
8808 201/300 Read block 1546 len 12288
8809 202/300 Read block 2365 len 40960
8810 203/300 Flush
8811 204/300 Write block 3383 len 28672 data: 2 2 2 2 3 3 3
8812 205/300 Write block 2286 len 12288 data: 3 2 2
8813 206/300 Read block 2566 len 36864
8814 207/300 Write block 317 len 4096 data: 3
8815 208/300 Write block 3245 len 24576 data: 2 3 3 3 3 3
8816 209/300 Flush
8817 210/300 Write block 4403 len 28672 data: 4 4 4 2 2 2 2
8818 211/300 Read block 4958 len 36864
8819 212/300 Read block 4881 len 28672
8820 213/300 Write block 2208 len 4096 data: 3
8821 214/300 Write block 2222 len 20480 data: 2 2 2 2 2
8822 215/300 Read block 945 len 28672
8823 216/300 Write block 1560 len 4096 data: 3
8824 217/300 Write block 1961 len 8192 data: 2 2
8825 218/300 Write block 1975 len 28672 data: 5 4 5 3 3 5 5
8826 219/300 Write block 1446 len 8192 data: 5 5
8827 220/300 Read block 1167 len 36864
8828 221/300 Flush
8829 222/300 Read block 3782 len 12288
8830 223/300 Flush
8831 224/300 Flush
8832 225/300 Read block 1874 len 8192
8833 226/300 Write block 227 len 40960 data: 4 4 4 4 3 3 2 3 3 3
8834 227/300 Write block 4786 len 12288 data: 3 3 4
8835 228/300 Write block 304 len 32768 data: 5 4 3 3 3 3 3 3
8836 229/300 Write block 1866 len 8192 data: 2 2
8837 230/300 Flush
8838 231/300 Read block 457 len 16384
8839 232/300 Read block 4773 len 36864
8840 233/300 Read block 2789 len 8192
8841 234/300 Read block 3171 len 28672
8842 235/300 Write block 1898 len 4096 data: 4
8843 236/300 Write block 4308 len 40960 data: 2 2 2 3 3 2 2 2 2 3
8844 237/300 Read block 776 len 8192
8845 238/300 Read block 3818 len 16384
8846 239/300 Write block 3639 len 20480 data: 2 2 2 2 3
8847 240/300 Write block 271 len 8192 data: 6 4
8848 241/300 Write block 2566 len 24576 data: 2 2 3 2 2 2
8849 242/300 Read block 1120 len 40960
8850 243/300 Write block 2854 len 24576 data: 2 3 3 3 2 2
8851 244/300 Write block 1421 len 4096 data: 5
8852 245/300 Read block 1264 len 36864
8853 246/300 Write block 4820 len 32768 data: 3 2 2 2 2 3 3 4
8854 247/300 Read block 3063 len 8192
8855 248/300 Read block 2749 len 32768
8856 249/300 Read block 1297 len 4096
8857 250/300 Read block 3155 len 12288
8858 251/300 Read block 2734 len 32768
8859 252/300 Read block 2626 len 4096
8860 253/300 Write block 4984 len 36864 data: 3 3 4 2 3 3 4 3 3
8861 254/300 Write block 2398 len 28672 data: 2 2 2 2 2 2 3
8862 255/300 Write block 609 len 28672 data: 5 6 7 7 7 6 6
8863 256/300 Read block 3723 len 24576
8864 257/300 Read block 1394 len 24576
8865 258/300 Write block 3347 len 16384 data: 3 4 4 3
8866 259/300 Write block 1027 len 40960 data: 4 3 4 4 4 4 4 4 3 3
8867 260/300 Read block 4989 len 24576
8868 261/300 Write block 3572 len 16384 data: 2 2 2 3
8869 262/300 Read block 1712 len 28672
8870 263/300 Read block 3806 len 12288
8871 264/300 Read block 242 len 28672
8872 265/300 Write block 1852 len 20480 data: 2 2 2 4 4
8873 266/300 Write block 4155 len 32768 data: 4 4 4 4 4 5 5 5
8874 267/300 Write block 2684 len 12288 data: 3 3 3
8875 268/300 Read block 3191 len 24576
8876 269/300 Read block 254 len 32768
8877 270/300 Write block 1219 len 24576 data: 2 3 3 2 2 2
8878 271/300 Write block 4263 len 28672 data: 4 4 4 3 4 4 2
8879 272/300 Flush
8880 273/300 Flush
8881 274/300 Write block 2454 len 20480 data: 3 3 5 4 4
8882 275/300 Write block 57 len 36864 data: 4 3 3 3 4 3 2 2 3
8883 276/300 Write block 4901 len 20480 data: 2 2 2 2 3
8884 277/300 Write block 1690 len 40960 data: 4 4 5 5 5 5 4 4 5 3
8885 278/300 Flush
8886 279/300 Read block 3620 len 28672
8887 280/300 Write block 2598 len 12288 data: 3 2 2
8888 281/300 Read block 4954 len 20480
8889 282/300 Read block 570 len 12288
8890 283/300 Flush
8891 284/300 Read block 2560 len 4096
8892 285/300 Write block 4852 len 20480 data: 3 3 2 2 2
8893 286/300 Write block 680 len 40960 data: 2 3 3 3 3 3 3 2 2 2
8894 287/300 Write block 3505 len 28672 data: 3 3 4 3 3 3 3
8895 288/300 Read block 2856 len 36864
8896 289/300 Read block 3461 len 4096
8897 290/300 Write block 4995 len 20480 data: 3 2 2 2 2
8898 291/300 Flush
8899 292/300 Write block 533 len 40960 data: 5 5 6 6 5 3 3 2 3 3
8900 293/300 Read block 71 len 40960
8901 294/300 Read block 1255 len 8192
8902 295/300 Write block 700 len 8192 data: 3 3
8903 296/300 Flush
8904 297/300 Write block 2878 len 4096 data: 4
8905 298/300 Read block 3427 len 20480
8906 299/300 Flush
8907 300/300 Write block 662 len 40960 data: 6 4 5 5 3 3 3 4 3 3
8908 Main task has work to do, go find it
8909 got dsc Start(1)
8910 start 1
8911 Mar 02 01:07:25.659 INFO request completed, latency_us: 229, response_code: 204, uri: /start/cid/1, method: POST, req_id: 62b10f9b-6556-4632-b8fa-c1e25d7ecbe1, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
8912 Replay: started 1, returned:Ok(())
8913 ----------------------------------------------------------------
8914 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
8915 States: Active Offline Active
8916 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8917 3016 Acked 4015 Read 1 Done New Done true
8918 3017 Acked 4016 Write 4 Done New Done false
8919 3018 Acked 4017 Read 9 Done New Done false
8920 3019 Acked 4018 Read 7 Done New Done false
8921 3020 Acked 4019 Write 2 Done New Done false
8922 3021 Acked 4020 Read 10 Done New Done false
8923 3022 Acked 4021 Write 2 Done New Done false
8924 3023 Acked 4022 Read 2 Done New Done false
8925 3024 Acked 4023 Read 3 Done New Done false
8926 3025 Acked 4024 Read 9 Done New Done false
8927 3026 Acked 4025 Write 10 Done New Done false
8928 3027 Acked 4026 Flush 0 Done New Done false
8929 3028 Acked 4027 Read 7 Done New Done false
8930 3029 Acked 4028 Write 9 Done New Done false
8931 3030 Acked 4029 Flush 0 Done New Done false
8932 3031 Acked 4030 Write 2 Done New Done false
8933 3032 Acked 4031 Write 3 Done New Done false
8934 3033 Acked 4032 Read 5 Done New Done false
8935 3034 Acked 4033 Read 2 Done New Done false
8936 3035 Acked 4034 Write 3 Done New Done false
8937 3036 Acked 4035 Read 5 Done New Done false
8938 3037 Acked 4036 Write 4 Done New Done false
8939 3038 Acked 4037 Flush 0 Done New Done false
8940 3039 Acked 4038 Read 1 Done New Done false
8941 3040 Acked 4039 Write 2 Done New Done false
8942 3041 Acked 4040 Write 6 Done New Done false
8943 3042 Acked 4041 Read 1 Done New Done false
8944 3043 Acked 4042 Write 8 Done New Done false
8945 3044 Acked 4043 Write 9 Done New Done false
8946 3045 Acked 4044 Write 4 Done New Done false
8947 3046 Acked 4045 Write 1 Done New Done false
8948 3047 Acked 4046 Read 6 Done New Done false
8949 3048 Acked 4047 Read 6 Done New Done false
8950 3049 Acked 4048 Read 9 Done New Done false
8951 3050 Acked 4049 Write 2 Done New Done false
8952 3051 Acked 4050 Write 3 Done New Done false
8953 3052 Acked 4051 Write 6 Done New Done false
8954 3053 Acked 4052 Read 2 Done New Done false
8955 3054 Acked 4053 Write 8 Done New Done false
8956 3055 Acked 4054 Read 10 Done New Done false
8957 3056 Acked 4055 Read 6 Done New Done false
8958 3057 Acked 4056 Read 6 Done New Done false
8959 3058 Acked 4057 Read 4 Done New Done false
8960 3059 Acked 4058 Write 10 Done New Done false
8961 3060 Acked 4059 Write 3 Done New Done false
8962 3061 Acked 4060 Flush 0 Done New Done false
8963 3062 Acked 4061 Write 5 Done New Done false
8964 3063 Acked 4062 Write 3 Done New Done false
8965 3064 Acked 4063 Write 2 Done New Done false
8966 3065 Acked 4064 Read 4 Done New Done false
8967 3066 Acked 4065 Flush 0 Done New Done false
8968 3067 Acked 4066 Read 2 Done New Done false
8969 3068 Acked 4067 Write 6 Done New Done false
8970 3069 Acked 4068 Read 4 Done New Done false
8971 3070 Acked 4069 Write 6 Done New Done false
8972 3071 Acked 4070 Read 7 Done New Done false
8973 3072 Acked 4071 Read 2 Done New Done false
8974 3073 Acked 4072 Read 9 Done New Done false
8975 3074 Acked 4073 Read 4 Done New Done false
8976 3075 Acked 4074 Write 9 Done New Done false
8977 3076 Acked 4075 Write 9 Done New Done false
8978 3077 Acked 4076 Read 5 Done New Done false
8979 3078 Acked 4077 Write 10 Done New Done false
8980 3079 Acked 4078 Read 6 Done New Done false
8981 3080 Acked 4079 Read 3 Done New Done false
8982 3081 Acked 4080 Write 4 Done New Done false
8983 3082 Acked 4081 Write 1 Done New Done false
8984 3083 Acked 4082 Write 4 Done New Done false
8985 3084 Acked 4083 Read 4 Done New Done false
8986 3085 Acked 4084 Flush 0 Done New Done false
8987 3086 Acked 4085 Read 7 Done New Done false
8988 3087 Acked 4086 Write 1 Done New Done false
8989 3088 Acked 4087 Write 10 Done New Done false
8990 3089 Acked 4088 Write 1 Done New Done false
8991 3090 Acked 4089 Read 5 Done New Done false
8992 3091 Acked 4090 Write 7 Done New Done false
8993 3092 Acked 4091 Read 3 Done New Done false
8994 3093 Acked 4092 Write 2 Done New Done false
8995 3094 Acked 4093 Read 4 Done New Done false
8996 3095 Acked 4094 Read 7 Done New Done false
8997 3096 Acked 4095 Write 1 Done New Done false
8998 3097 Acked 4096 Write 2 Done New Done false
8999 3098 Acked 4097 Flush 0 Done New Done false
9000 3099 Acked 4098 Read 6 Done New Done false
9001 3100 Acked 4099 Read 7 Done New Done false
9002 3101 Acked 4100 Write 5 Done New Done false
9003 3102 Acked 4101 Read 4 Done New Done false
9004 3103 Acked 4102 Flush 0 Done New Done false
9005 3104 Acked 4103 Read 1 Done New Done false
9006 3105 Acked 4104 Write 10 Done New Done false
9007 3106 Acked 4105 Read 7 Done New Done false
9008 3107 Acked 4106 Write 8 Done New Done false
9009 3108 Acked 4107 Read 7 Done New Done false
9010 3109 Acked 4108 Write 5 Done New Done false
9011 3110 Acked 4109 Read 1 Done New Done false
9012 3111 Acked 4110 Read 7 Done New Done false
9013 3112 Acked 4111 Flush 0 Done New Done false
9014 3113 Acked 4112 Read 7 Done New Done false
9015 3114 Acked 4113 Write 3 Done New Done false
9016 3115 Acked 4114 Read 8 Done New Done false
9017 3116 Acked 4115 Read 3 Done New Done false
9018 3117 Acked 4116 Read 2 Done New Done false
9019 3118 Acked 4117 Write 3 Done New Done false
9020 3119 Acked 4118 Write 6 Done New Done false
9021 3120 Acked 4119 Read 9 Done New Done false
9022 3121 Acked 4120 Write 5 Done New Done false
9023 3122 Acked 4121 Write 6 Done New Done false
9024 3123 Acked 4122 Read 5 Done New Done false
9025 3124 Acked 4123 Write 4 Done New Done false
9026 3125 Acked 4124 Read 1 Done New Done false
9027 3126 Acked 4125 Read 5 Done New Done false
9028 3127 Acked 4126 Write 5 Done New Done false
9029 3128 Acked 4127 Write 5 Done New Done false
9030 3129 Acked 4128 Write 7 Done New Done false
9031 3130 Acked 4129 Write 9 Done New Done false
9032 3131 Acked 4130 Write 6 Done New Done false
9033 3132 Acked 4131 Read 4 Done New Done false
9034 3133 Acked 4132 Read 5 Done New Done false
9035 3134 Acked 4133 Write 3 Done New Done false
9036 3135 Acked 4134 Write 1 Done New Done false
9037 3136 Acked 4135 Write 2 Done New Done false
9038 3137 Acked 4136 Write 10 Done New Done false
9039 3138 Acked 4137 Write 5 Done New Done false
9040 3139 Acked 4138 Read 4 Done New Done false
9041 3140 Acked 4139 Read 4 Done New Done false
9042 3141 Acked 4140 Write 2 Done New Done false
9043 3142 Acked 4141 Read 7 Done New Done false
9044 3143 Acked 4142 Write 5 Done New Done false
9045 3144 Acked 4143 Read 8 Done New Done false
9046 3145 Acked 4144 Read 5 Done New Done false
9047 3146 Acked 4145 Flush 0 Done New Done false
9048 3147 Acked 4146 Write 3 Done New Done false
9049 3148 Acked 4147 Read 6 Done New Done false
9050 3149 Acked 4148 Read 4 Done New Done false
9051 3150 Acked 4149 Write 5 Done New Done false
9052 3151 Acked 4150 Write 3 Done New Done false
9053 3152 Acked 4151 Write 3 Done New Done false
9054 3153 Acked 4152 Read 5 Done New Done false
9055 3154 Acked 4153 Read 10 Done New Done false
9056 3155 Acked 4154 Write 8 Done New Done false
9057 3156 Acked 4155 Read 3 Done New Done false
9058 3157 Acked 4156 Flush 0 Done New Done false
9059 3158 Acked 4157 Write 6 Done New Done false
9060 3159 Acked 4158 Read 6 Done New Done false
9061 3160 Acked 4159 Write 10 Done New Done false
9062 3161 Acked 4160 Read 2 Done New Done false
9063 3162 Acked 4161 Read 3 Done New Done false
9064 3163 Acked 4162 Read 1 Done New Done false
9065 3164 Acked 4163 Read 10 Done New Done false
9066 3165 Acked 4164 Read 6 Done New Done false
9067 3166 Acked 4165 Read 10 Done New Done false
9068 3167 Acked 4166 Write 8 Done New Done false
9069 3168 Acked 4167 Read 5 Done New Done false
9070 3169 Acked 4168 Read 10 Done New Done false
9071 3170 Acked 4169 Flush 0 Done New Done false
9072 3171 Acked 4170 Read 9 Done New Done false
9073 3172 Acked 4171 Read 10 Done New Done false
9074 3173 Acked 4172 Write 6 Done New Done false
9075 3174 Acked 4173 Read 4 Done New Done false
9076 3175 Acked 4174 Read 10 Done New Done false
9077 3176 Acked 4175 Flush 0 Done New Done false
9078 3177 Acked 4176 Write 6 Done New Done false
9079 3178 Acked 4177 Write 10 Done New Done false
9080 3179 Acked 4178 Read 3 Done New Done false
9081 3180 Acked 4179 Write 10 Done New Done false
9082 3181 Acked 4180 Flush 0 Done New Done false
9083 3182 Acked 4181 Read 10 Done New Done false
9084 3183 Acked 4182 Write 2 Done New Done false
9085 3184 Acked 4183 Read 5 Done New Done false
9086 3185 Acked 4184 Read 10 Done New Done false
9087 3186 Acked 4185 Read 7 Done New Done false
9088 3187 Acked 4186 Read 5 Done New Done false
9089 3188 Acked 4187 Read 4 Done New Done false
9090 3189 Acked 4188 Write 7 Done New Done false
9091 3190 Acked 4189 Read 3 Done New Done false
9092 3191 Acked 4190 Read 9 Done New Done false
9093 3192 Acked 4191 Read 9 Done New Done false
9094 3193 Acked 4192 Write 3 Done New Done false
9095 3194 Acked 4193 Write 3 Done New Done false
9096 3195 Acked 4194 Read 2 Done New Done false
9097 3196 Acked 4195 Read 8 Done New Done false
9098 3197 Acked 4196 Read 3 Done New Done false
9099 3198 Acked 4197 Write 1 Done New Done false
9100 3199 Acked 4198 Read 8 Done New Done false
9101 3200 Acked 4199 Write 3 Done New Done false
9102 3201 Acked 4200 Read 7 Done New Done false
9103 3202 Acked 4201 Write 9 Done New Done false
9104 3203 Acked 4202 Read 4 Done New Done false
9105 3204 Acked 4203 Flush 0 Done New Done false
9106 3205 Acked 4204 Write 4 Done New Done false
9107 3206 Acked 4205 Flush 0 Done New Done false
9108 3207 Acked 4206 Read 5 Done New Done false
9109 3208 Acked 4207 Write 8 Done New Done false
9110 3209 Acked 4208 Write 7 Done New Done false
9111 3210 Acked 4209 Write 6 Done New Done false
9112 3211 Acked 4210 Write 2 Done New Done false
9113 3212 Acked 4211 Read 9 Done New Done false
9114 3213 Acked 4212 Write 10 Done New Done false
9115 3214 Acked 4213 Read 10 Done New Done false
9116 3215 Acked 4214 Read 9 Done New Done false
9117 3216 Acked 4215 Read 3 Done New Done false
9118 3217 Acked 4216 Read 10 Done New Done false
9119 3218 Acked 4217 Flush 0 Done New Done false
9120 3219 Acked 4218 Write 7 Done New Done false
9121 3220 Acked 4219 Write 3 Done New Done false
9122 3221 Acked 4220 Read 9 Done New Done false
9123 3222 Acked 4221 Write 1 Done New Done false
9124 3223 Acked 4222 Write 6 Done New Done false
9125 3224 Acked 4223 Flush 0 Done New Done false
9126 3225 Acked 4224 Write 7 Done New Done false
9127 3226 Acked 4225 Read 9 Done New Done false
9128 3227 Acked 4226 Read 7 Done New Done false
9129 3228 Acked 4227 Write 1 Done New Done false
9130 3229 Acked 4228 Write 5 Done New Done false
9131 3230 Acked 4229 Read 7 Done New Done false
9132 3231 Acked 4230 Write 1 Done New Done false
9133 3232 Acked 4231 Write 2 Done New Done false
9134 3233 Acked 4232 Write 7 Done New Done false
9135 3234 Acked 4233 Write 2 Done New Done false
9136 3235 Acked 4234 Read 9 Done New Done false
9137 3236 Acked 4235 Flush 0 Done New Done false
9138 3237 Acked 4236 Read 3 Done New Done false
9139 3238 Acked 4237 Flush 0 Done New Done false
9140 3239 Acked 4238 Flush 0 Done New Done false
9141 3240 Acked 4239 Read 2 Done New Done false
9142 3241 Acked 4240 Write 10 Done New Done false
9143 3242 Acked 4241 Write 3 Done New Done false
9144 3243 Acked 4242 Write 8 Done New Done false
9145 3244 Acked 4243 Write 2 Done New Done false
9146 3245 Acked 4244 Flush 0 Done New Done false
9147 3246 Acked 4245 Read 4 Done New Done false
9148 3247 Acked 4246 Read 9 Done New Done false
9149 3248 Acked 4247 Read 2 Done New Done false
9150 3249 Acked 4248 Read 7 Done New Done false
9151 3250 Acked 4249 Write 1 Done New Done false
9152 3251 Acked 4250 Write 10 Done New Done false
9153 3252 Acked 4251 Read 2 Done New Done false
9154 3253 Acked 4252 Read 4 Done New Done false
9155 3254 Acked 4253 Write 5 Done New Done false
9156 3255 Acked 4254 Write 2 Done New Done false
9157 3256 Acked 4255 Write 6 Done New Done false
9158 3257 Acked 4256 Read 10 Done New Done false
9159 3258 Acked 4257 Write 6 Done New Done false
9160 3259 Acked 4258 Write 1 Done New Done false
9161 3260 Acked 4259 Read 9 Done New Done false
9162 3261 Acked 4260 Write 8 Done New Done false
9163 3262 Acked 4261 Read 2 Done New Done false
9164 3263 Acked 4262 Read 8 Done New Done false
9165 3264 Acked 4263 Read 1 Done New Done false
9166 3265 Acked 4264 Read 3 Done New Done false
9167 3266 Acked 4265 Read 8 Done New Done false
9168 3267 Acked 4266 Read 1 Done New Done false
9169 3268 Acked 4267 Write 9 Done New Done false
9170 3269 Acked 4268 Write 7 Done New Done false
9171 3270 Acked 4269 Write 7 Done New Done false
9172 3271 Acked 4270 Read 6 Done New Done false
9173 3272 Acked 4271 Read 6 Done New Done false
9174 3273 Acked 4272 Write 4 Done New Done false
9175 3274 Acked 4273 Write 10 Done New Done false
9176 3275 Acked 4274 Read 6 Done New Done false
9177 3276 Acked 4275 Write 4 Done New Done false
9178 3277 Acked 4276 Read 7 Done New Done false
9179 3278 Acked 4277 Read 3 Done New Done false
9180 3279 Acked 4278 Read 7 Done New Done false
9181 3280 Acked 4279 Write 5 Done New Done false
9182 3281 Acked 4280 Write 8 Done New Done false
9183 3282 Acked 4281 Write 3 Done New Done false
9184 3283 Acked 4282 Read 6 Done New Done false
9185 3284 Acked 4283 Read 8 Done New Done false
9186 3285 Acked 4284 Write 6 Done New Done false
9187 3286 Acked 4285 Write 7 Done New Done false
9188 3287 Acked 4286 Flush 0 Done New Done false
9189 3288 Acked 4287 Flush 0 Done New Done false
9190 3289 Acked 4288 Write 5 Done New Done false
9191 3290 Acked 4289 Write 9 Done New Done false
9192 3291 Acked 4290 Write 5 Done New Done false
9193 3292 Acked 4291 Write 10 Done New Done false
9194 3293 Acked 4292 Flush 0 Done New Done false
9195 3294 Acked 4293 Read 7 Done New Done false
9196 3295 Acked 4294 Write 3 Done New Done false
9197 3296 Acked 4295 Read 5 Done New Done false
9198 3297 Acked 4296 Read 3 Done New Done false
9199 3298 Acked 4297 Flush 0 Done New Done false
9200 3299 Acked 4298 Read 1 Done New Done false
9201 3300 Acked 4299 Write 5 Done New Done false
9202 3301 Acked 4300 Write 10 Done New Done false
9203 3302 Acked 4301 Write 7 Done New Done false
9204 3303 Acked 4302 Read 9 Done New Done false
9205 3304 Acked 4303 Read 1 Done New Done false
9206 3305 Acked 4304 Write 5 Done New Done false
9207 3306 Acked 4305 Flush 0 Done New Done false
9208 3307 Acked 4306 Write 10 Done New Done false
9209 3308 Acked 4307 Read 10 Done New Done false
9210 3309 Acked 4308 Read 2 Done New Done false
9211 3310 Acked 4309 Write 2 Done New Done false
9212 3311 Acked 4310 Flush 0 Done New Done false
9213 3312 Acked 4311 Write 1 Done New Done false
9214 3313 Acked 4312 Read 5 Done New Done false
9215 3314 Acked 4313 Flush 0 Done New Done false
9216 3315 Acked 4314 Write 10 Done New Done false
9217 STATES DS:0 DS:1 DS:2 TOTAL
9218 New 0 300 0 300
9219 Sent 0 0 0 0
9220 Done 300 0 300 600
9221 Skipped 0 0 0 0
9222 Error 0 0 0 0
9223 Last Flush: 4313 4014 4313
9224 Downstairs last five completed: 4014 4013 4012 4011 4010
9225 Upstairs last five completed: 3315 3314 3313 3312 3311
9226 CLIENT: Up:0 ds:300 act:2
92272024-03-02T01:07:26.325ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92282024-03-02T01:07:26.325ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92292024-03-02T01:07:26.325ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92302024-03-02T01:07:26.325ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92312024-03-02T01:07:26.325ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92322024-03-02T01:07:26.325ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92332024-03-02T01:07:26.325ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92342024-03-02T01:07:27.326ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92352024-03-02T01:07:27.326ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92362024-03-02T01:07:27.326ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92372024-03-02T01:07:27.326ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92382024-03-02T01:07:27.326ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92392024-03-02T01:07:27.326ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92402024-03-02T01:07:27.326ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92412024-03-02T01:07:28.327ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92422024-03-02T01:07:28.327ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92432024-03-02T01:07:28.328ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92442024-03-02T01:07:28.328ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92452024-03-02T01:07:28.328ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92462024-03-02T01:07:28.328ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92472024-03-02T01:07:28.328ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92482024-03-02T01:07:29.329ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92492024-03-02T01:07:29.329ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92502024-03-02T01:07:29.329ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92512024-03-02T01:07:29.329ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92522024-03-02T01:07:29.329ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92532024-03-02T01:07:29.329ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
92542024-03-02T01:07:29.329ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9255 ----------------------------------------------------------------
9256 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9257 States: Active Offline Active
9258 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9259 3016 Acked 4015 Read 1 Done New Done true
9260 3017 Acked 4016 Write 4 Done New Done true
9261 3018 Acked 4017 Read 9 Done New Done true
9262 3019 Acked 4018 Read 7 Done New Done true
9263 3020 Acked 4019 Write 2 Done New Done true
9264 3021 Acked 4020 Read 10 Done New Done true
9265 3022 Acked 4021 Write 2 Done New Done true
9266 3023 Acked 4022 Read 2 Done New Done true
9267 3024 Acked 4023 Read 3 Done New Done true
9268 3025 Acked 4024 Read 9 Done New Done true
9269 3026 Acked 4025 Write 10 Done New Done true
9270 3027 Acked 4026 Flush 0 Done New Done true
9271 3028 Acked 4027 Read 7 Done New Done true
9272 3029 Acked 4028 Write 9 Done New Done true
9273 3030 Acked 4029 Flush 0 Done New Done true
9274 3031 Acked 4030 Write 2 Done New Done true
9275 3032 Acked 4031 Write 3 Done New Done true
9276 3033 Acked 4032 Read 5 Done New Done true
9277 3034 Acked 4033 Read 2 Done New Done true
9278 3035 Acked 4034 Write 3 Done New Done true
9279 3036 Acked 4035 Read 5 Done New Done true
9280 3037 Acked 4036 Write 4 Done New Done true
9281 3038 Acked 4037 Flush 0 Done New Done true
9282 3039 Acked 4038 Read 1 Done New Done true
9283 3040 Acked 4039 Write 2 Done New Done true
9284 3041 Acked 4040 Write 6 Done New Done true
9285 3042 Acked 4041 Read 1 Done New Done true
9286 3043 Acked 4042 Write 8 Done New Done true
9287 3044 Acked 4043 Write 9 Done New Done true
9288 3045 Acked 4044 Write 4 Done New Done true
9289 3046 Acked 4045 Write 1 Done New Done true
9290 3047 Acked 4046 Read 6 Done New Done true
9291 3048 Acked 4047 Read 6 Done New Done true
9292 3049 Acked 4048 Read 9 Done New Done true
9293 3050 Acked 4049 Write 2 Done New Done true
9294 3051 Acked 4050 Write 3 Done New Done true
9295 3052 Acked 4051 Write 6 Done New Done true
9296 3053 Acked 4052 Read 2 Done New Done true
9297 3054 Acked 4053 Write 8 Done New Done true
9298 3055 Acked 4054 Read 10 Done New Done true
9299 3056 Acked 4055 Read 6 Done New Done true
9300 3057 Acked 4056 Read 6 Done New Done true
9301 3058 Acked 4057 Read 4 Done New Done true
9302 3059 Acked 4058 Write 10 Done New Done true
9303 3060 Acked 4059 Write 3 Done New Done true
9304 3061 Acked 4060 Flush 0 Done New Done true
9305 3062 Acked 4061 Write 5 Done New Done true
9306 3063 Acked 4062 Write 3 Done New Done true
9307 3064 Acked 4063 Write 2 Done New Done true
9308 3065 Acked 4064 Read 4 Done New Done true
9309 3066 Acked 4065 Flush 0 Done New Done true
9310 3067 Acked 4066 Read 2 Done New Done true
9311 3068 Acked 4067 Write 6 Done New Done true
9312 3069 Acked 4068 Read 4 Done New Done true
9313 3070 Acked 4069 Write 6 Done New Done true
9314 3071 Acked 4070 Read 7 Done New Done true
9315 3072 Acked 4071 Read 2 Done New Done true
9316 3073 Acked 4072 Read 9 Done New Done true
9317 3074 Acked 4073 Read 4 Done New Done true
9318 3075 Acked 4074 Write 9 Done New Done true
9319 3076 Acked 4075 Write 9 Done New Done true
9320 3077 Acked 4076 Read 5 Done New Done true
9321 3078 Acked 4077 Write 10 Done New Done true
9322 3079 Acked 4078 Read 6 Done New Done true
9323 3080 Acked 4079 Read 3 Done New Done true
9324 3081 Acked 4080 Write 4 Done New Done true
9325 3082 Acked 4081 Write 1 Done New Done true
9326 3083 Acked 4082 Write 4 Done New Done true
9327 3084 Acked 4083 Read 4 Done New Done true
9328 3085 Acked 4084 Flush 0 Done New Done true
9329 3086 Acked 4085 Read 7 Done New Done true
9330 3087 Acked 4086 Write 1 Done New Done true
9331 3088 Acked 4087 Write 10 Done New Done true
9332 3089 Acked 4088 Write 1 Done New Done true
9333 3090 Acked 4089 Read 5 Done New Done true
9334 3091 Acked 4090 Write 7 Done New Done true
9335 3092 Acked 4091 Read 3 Done New Done true
9336 3093 Acked 4092 Write 2 Done New Done true
9337 3094 Acked 4093 Read 4 Done New Done true
9338 3095 Acked 4094 Read 7 Done New Done true
9339 3096 Acked 4095 Write 1 Done New Done true
9340 3097 Acked 4096 Write 2 Done New Done true
9341 3098 Acked 4097 Flush 0 Done New Done true
9342 3099 Acked 4098 Read 6 Done New Done true
9343 3100 Acked 4099 Read 7 Done New Done true
9344 3101 Acked 4100 Write 5 Done New Done true
9345 3102 Acked 4101 Read 4 Done New Done true
9346 3103 Acked 4102 Flush 0 Done New Done true
9347 3104 Acked 4103 Read 1 Done New Done true
9348 3105 Acked 4104 Write 10 Done New Done true
9349 3106 Acked 4105 Read 7 Done New Done true
9350 3107 Acked 4106 Write 8 Done New Done true
9351 3108 Acked 4107 Read 7 Done New Done true
9352 3109 Acked 4108 Write 5 Done New Done true
9353 3110 Acked 4109 Read 1 Done New Done true
9354 3111 Acked 4110 Read 7 Done New Done true
9355 3112 Acked 4111 Flush 0 Done New Done true
9356 3113 Acked 4112 Read 7 Done New Done true
9357 3114 Acked 4113 Write 3 Done New Done true
9358 3115 Acked 4114 Read 8 Done New Done true
9359 3116 Acked 4115 Read 3 Done New Done true
9360 3117 Acked 4116 Read 2 Done New Done true
9361 3118 Acked 4117 Write 3 Done New Done true
9362 3119 Acked 4118 Write 6 Done New Done true
9363 3120 Acked 4119 Read 9 Done New Done true
9364 3121 Acked 4120 Write 5 Done New Done true
9365 3122 Acked 4121 Write 6 Done New Done true
9366 3123 Acked 4122 Read 5 Done New Done true
9367 3124 Acked 4123 Write 4 Done New Done true
9368 3125 Acked 4124 Read 1 Done New Done true
9369 3126 Acked 4125 Read 5 Done New Done true
9370 3127 Acked 4126 Write 5 Done New Done true
9371 3128 Acked 4127 Write 5 Done New Done true
9372 3129 Acked 4128 Write 7 Done New Done true
9373 3130 Acked 4129 Write 9 Done New Done true
9374 3131 Acked 4130 Write 6 Done New Done true
9375 3132 Acked 4131 Read 4 Done New Done true
9376 3133 Acked 4132 Read 5 Done New Done true
9377 3134 Acked 4133 Write 3 Done New Done true
9378 3135 Acked 4134 Write 1 Done New Done true
9379 3136 Acked 4135 Write 2 Done New Done true
9380 3137 Acked 4136 Write 10 Done New Done true
9381 3138 Acked 4137 Write 5 Done New Done true
9382 3139 Acked 4138 Read 4 Done New Done true
9383 3140 Acked 4139 Read 4 Done New Done true
9384 3141 Acked 4140 Write 2 Done New Done true
9385 3142 Acked 4141 Read 7 Done New Done true
9386 3143 Acked 4142 Write 5 Done New Done true
9387 3144 Acked 4143 Read 8 Done New Done true
9388 3145 Acked 4144 Read 5 Done New Done true
9389 3146 Acked 4145 Flush 0 Done New Done true
9390 3147 Acked 4146 Write 3 Done New Done true
9391 3148 Acked 4147 Read 6 Done New Done true
9392 3149 Acked 4148 Read 4 Done New Done true
9393 3150 Acked 4149 Write 5 Done New Done true
9394 3151 Acked 4150 Write 3 Done New Done true
9395 3152 Acked 4151 Write 3 Done New Done true
9396 3153 Acked 4152 Read 5 Done New Done true
9397 3154 Acked 4153 Read 10 Done New Done true
9398 3155 Acked 4154 Write 8 Done New Done true
9399 3156 Acked 4155 Read 3 Done New Done true
9400 3157 Acked 4156 Flush 0 Done New Done true
9401 3158 Acked 4157 Write 6 Done New Done true
9402 3159 Acked 4158 Read 6 Done New Done true
9403 3160 Acked 4159 Write 10 Done New Done true
9404 3161 Acked 4160 Read 2 Done New Done true
9405 3162 Acked 4161 Read 3 Done New Done true
9406 3163 Acked 4162 Read 1 Done New Done true
9407 3164 Acked 4163 Read 10 Done New Done true
9408 3165 Acked 4164 Read 6 Done New Done true
9409 3166 Acked 4165 Read 10 Done New Done true
9410 3167 Acked 4166 Write 8 Done New Done true
9411 3168 Acked 4167 Read 5 Done New Done true
9412 3169 Acked 4168 Read 10 Done New Done true
9413 3170 Acked 4169 Flush 0 Done New Done true
9414 3171 Acked 4170 Read 9 Done New Done true
9415 3172 Acked 4171 Read 10 Done New Done true
9416 3173 Acked 4172 Write 6 Done New Done true
9417 3174 Acked 4173 Read 4 Done New Done true
9418 3175 Acked 4174 Read 10 Done New Done true
9419 3176 Acked 4175 Flush 0 Done New Done true
9420 3177 Acked 4176 Write 6 Done New Done true
9421 3178 Acked 4177 Write 10 Done New Done true
9422 3179 Acked 4178 Read 3 Done New Done true
9423 3180 Acked 4179 Write 10 Done New Done true
9424 3181 Acked 4180 Flush 0 Done New Done true
9425 3182 Acked 4181 Read 10 Done New Done true
9426 3183 Acked 4182 Write 2 Done New Done true
9427 3184 Acked 4183 Read 5 Done New Done true
9428 3185 Acked 4184 Read 10 Done New Done true
9429 3186 Acked 4185 Read 7 Done New Done true
9430 3187 Acked 4186 Read 5 Done New Done true
9431 3188 Acked 4187 Read 4 Done New Done true
9432 3189 Acked 4188 Write 7 Done New Done true
9433 3190 Acked 4189 Read 3 Done New Done true
9434 3191 Acked 4190 Read 9 Done New Done true
9435 3192 Acked 4191 Read 9 Done New Done true
9436 3193 Acked 4192 Write 3 Done New Done true
9437 3194 Acked 4193 Write 3 Done New Done true
9438 3195 Acked 4194 Read 2 Done New Done true
9439 3196 Acked 4195 Read 8 Done New Done true
9440 3197 Acked 4196 Read 3 Done New Done true
9441 3198 Acked 4197 Write 1 Done New Done true
9442 3199 Acked 4198 Read 8 Done New Done true
9443 3200 Acked 4199 Write 3 Done New Done true
9444 3201 Acked 4200 Read 7 Done New Done true
9445 3202 Acked 4201 Write 9 Done New Done true
9446 3203 Acked 4202 Read 4 Done New Done true
9447 3204 Acked 4203 Flush 0 Done New Done true
9448 3205 Acked 4204 Write 4 Done New Done true
9449 3206 Acked 4205 Flush 0 Done New Done true
9450 3207 Acked 4206 Read 5 Done New Done true
9451 3208 Acked 4207 Write 8 Done New Done true
9452 3209 Acked 4208 Write 7 Done New Done true
9453 3210 Acked 4209 Write 6 Done New Done true
9454 3211 Acked 4210 Write 2 Done New Done true
9455 3212 Acked 4211 Read 9 Done New Done true
9456 3213 Acked 4212 Write 10 Done New Done true
9457 3214 Acked 4213 Read 10 Done New Done true
9458 3215 Acked 4214 Read 9 Done New Done true
9459 3216 Acked 4215 Read 3 Done New Done true
9460 3217 Acked 4216 Read 10 Done New Done true
9461 3218 Acked 4217 Flush 0 Done New Done true
9462 3219 Acked 4218 Write 7 Done New Done true
9463 3220 Acked 4219 Write 3 Done New Done true
9464 3221 Acked 4220 Read 9 Done New Done true
9465 3222 Acked 4221 Write 1 Done New Done true
9466 3223 Acked 4222 Write 6 Done New Done true
9467 3224 Acked 4223 Flush 0 Done New Done true
9468 3225 Acked 4224 Write 7 Done New Done true
9469 3226 Acked 4225 Read 9 Done New Done true
9470 3227 Acked 4226 Read 7 Done New Done true
9471 3228 Acked 4227 Write 1 Done New Done true
9472 3229 Acked 4228 Write 5 Done New Done true
9473 3230 Acked 4229 Read 7 Done New Done true
9474 3231 Acked 4230 Write 1 Done New Done true
9475 3232 Acked 4231 Write 2 Done New Done true
9476 3233 Acked 4232 Write 7 Done New Done true
9477 3234 Acked 4233 Write 2 Done New Done true
9478 3235 Acked 4234 Read 9 Done New Done true
9479 3236 Acked 4235 Flush 0 Done New Done true
9480 3237 Acked 4236 Read 3 Done New Done true
9481 3238 Acked 4237 Flush 0 Done New Done true
9482 3239 Acked 4238 Flush 0 Done New Done true
9483 3240 Acked 4239 Read 2 Done New Done true
9484 3241 Acked 4240 Write 10 Done New Done true
9485 3242 Acked 4241 Write 3 Done New Done true
9486 3243 Acked 4242 Write 8 Done New Done true
9487 3244 Acked 4243 Write 2 Done New Done true
9488 3245 Acked 4244 Flush 0 Done New Done true
9489 3246 Acked 4245 Read 4 Done New Done true
9490 3247 Acked 4246 Read 9 Done New Done true
9491 3248 Acked 4247 Read 2 Done New Done true
9492 3249 Acked 4248 Read 7 Done New Done true
9493 3250 Acked 4249 Write 1 Done New Done true
9494 3251 Acked 4250 Write 10 Done New Done true
9495 3252 Acked 4251 Read 2 Done New Done true
9496 3253 Acked 4252 Read 4 Done New Done true
9497 3254 Acked 4253 Write 5 Done New Done true
9498 3255 Acked 4254 Write 2 Done New Done true
9499 3256 Acked 4255 Write 6 Done New Done true
9500 3257 Acked 4256 Read 10 Done New Done true
9501 3258 Acked 4257 Write 6 Done New Done true
9502 3259 Acked 4258 Write 1 Done New Done true
9503 3260 Acked 4259 Read 9 Done New Done true
9504 3261 Acked 4260 Write 8 Done New Done true
9505 3262 Acked 4261 Read 2 Done New Done true
9506 3263 Acked 4262 Read 8 Done New Done true
9507 3264 Acked 4263 Read 1 Done New Done true
9508 3265 Acked 4264 Read 3 Done New Done true
9509 3266 Acked 4265 Read 8 Done New Done true
9510 3267 Acked 4266 Read 1 Done New Done true
9511 3268 Acked 4267 Write 9 Done New Done true
9512 3269 Acked 4268 Write 7 Done New Done true
9513 3270 Acked 4269 Write 7 Done New Done true
9514 3271 Acked 4270 Read 6 Done New Done true
9515 3272 Acked 4271 Read 6 Done New Done true
9516 3273 Acked 4272 Write 4 Done New Done true
9517 3274 Acked 4273 Write 10 Done New Done true
9518 3275 Acked 4274 Read 6 Done New Done true
9519 3276 Acked 4275 Write 4 Done New Done true
9520 3277 Acked 4276 Read 7 Done New Done true
9521 3278 Acked 4277 Read 3 Done New Done true
9522 3279 Acked 4278 Read 7 Done New Done true
9523 3280 Acked 4279 Write 5 Done New Done true
9524 3281 Acked 4280 Write 8 Done New Done true
9525 3282 Acked 4281 Write 3 Done New Done true
9526 3283 Acked 4282 Read 6 Done New Done true
9527 3284 Acked 4283 Read 8 Done New Done true
9528 3285 Acked 4284 Write 6 Done New Done true
9529 3286 Acked 4285 Write 7 Done New Done true
9530 3287 Acked 4286 Flush 0 Done New Done true
9531 3288 Acked 4287 Flush 0 Done New Done true
9532 3289 Acked 4288 Write 5 Done New Done true
9533 3290 Acked 4289 Write 9 Done New Done true
9534 3291 Acked 4290 Write 5 Done New Done true
9535 3292 Acked 4291 Write 10 Done New Done true
9536 3293 Acked 4292 Flush 0 Done New Done true
9537 3294 Acked 4293 Read 7 Done New Done true
9538 3295 Acked 4294 Write 3 Done New Done true
9539 3296 Acked 4295 Read 5 Done New Done true
9540 3297 Acked 4296 Read 3 Done New Done true
9541 3298 Acked 4297 Flush 0 Done New Done true
9542 3299 Acked 4298 Read 1 Done New Done true
9543 3300 Acked 4299 Write 5 Done New Done true
9544 3301 Acked 4300 Write 10 Done New Done true
9545 3302 Acked 4301 Write 7 Done New Done true
9546 3303 Acked 4302 Read 9 Done New Done true
9547 3304 Acked 4303 Read 1 Done New Done true
9548 3305 Acked 4304 Write 5 Done New Done true
9549 3306 Acked 4305 Flush 0 Done New Done true
9550 3307 Acked 4306 Write 10 Done New Done true
9551 3308 Acked 4307 Read 10 Done New Done true
9552 3309 Acked 4308 Read 2 Done New Done true
9553 3310 Acked 4309 Write 2 Done New Done true
9554 3311 Acked 4310 Flush 0 Done New Done true
9555 3312 Acked 4311 Write 1 Done New Done true
9556 3313 Acked 4312 Read 5 Done New Done true
9557 3314 Acked 4313 Flush 0 Done New Done true
9558 3315 Acked 4314 Write 10 Done New Done true
9559 3316 Acked 4315 Flush 0 Done New Done true
9560 STATES DS:0 DS:1 DS:2 TOTAL
9561 New 0 301 0 301
9562 Sent 0 0 0 0
9563 Done 301 0 301 602
9564 Skipped 0 0 0 0
9565 Error 0 0 0 0
9566 Last Flush: 4315 4014 4315
9567 Downstairs last five completed: 4014 4013 4012 4011 4010
9568 Upstairs last five completed: 3316 3315 3314 3313 3312
9569 CLIENT: Up:0 ds:301 act:2
95702024-03-02T01:07:30.331ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95712024-03-02T01:07:30.331ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95722024-03-02T01:07:30.331ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95732024-03-02T01:07:30.331ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95742024-03-02T01:07:30.331ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95752024-03-02T01:07:30.331ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95762024-03-02T01:07:30.331ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9577 [8820] got start action
9578 [8820] I am going to restart
9579 Starting downstairs at port 8820
9580 Make output file at "/tmp/dsc/downstairs-8820.txt"
9581 [8820][1] reports Starting
95822024-03-02T01:07:31.332ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95832024-03-02T01:07:31.332ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95842024-03-02T01:07:31.333ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95852024-03-02T01:07:31.333ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95862024-03-02T01:07:31.333ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95872024-03-02T01:07:31.333ZINFOcrucible: [1] client re-new 301 jobs since flush 4014 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9588 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1173)
95892024-03-02T01:07:31.333ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9590 [8820][1] reports Running
95912024-03-02T01:07:32.334ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95922024-03-02T01:07:32.334ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95932024-03-02T01:07:32.335ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95942024-03-02T01:07:32.335ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95952024-03-02T01:07:32.335ZINFOcrucible: send last flush ID to this DS: 4014 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95962024-03-02T01:07:32.336ZINFOcrucible: Replied this last flush ID: 4014 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95972024-03-02T01:07:32.336ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95982024-03-02T01:07:32.336ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
95992024-03-02T01:07:32.336ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96002024-03-02T01:07:32.336ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9601 ----------------------------------------------------------------
9602 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9603 Downstairs last five completed: 4315 4314 4313 4312 4311
9604 Upstairs last five completed: 3316 3315 3314 3313 3312
9605 CLIENT: Up:0 ds:0 act:3
9606 Replay: All jobs finished, all DS active.
9607 Main task has work to do, go find it
9608 got dsc Stop(2)
9609 stop 2
9610 Mar 02 01:07:33.663 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
9611 : POST, req_id: 4e6136eb-8d9d-4318-a356-5b9690c64720, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
96122024-03-02T01:07:33.664ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96132024-03-02T01:07:33.664ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96142024-03-02T01:07:33.664ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9615 Mar 02 01:07:33.664{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:33.66482381Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
9616 INFO request completed, latency_us: 114, response_code: 200, uri{"msg":"[2] client re-new 0 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:33.664843009Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
9617 : /state/cid/2, method: GET, req_id: 5a92e4f5-8e27-4120-b1de-3212cfb30fc5, remote_addr: 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:33.664875127Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
9618 1:40735, local_addr: 127.0.0.1:9998
9619 [8830] Exited with: ExitStatus(unix_wait_status(9))
9620 [8830][2] reports Exit
96212024-03-02T01:07:34.666ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96222024-03-02T01:07:34.666ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96232024-03-02T01:07:34.666ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96242024-03-02T01:07:34.666ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96252024-03-02T01:07:34.666ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96262024-03-02T01:07:34.666ZINFOcrucible: [2] client re-new 0 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96272024-03-02T01:07:34.666ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96282024-03-02T01:07:35.666ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96292024-03-02T01:07:35.666ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96302024-03-02T01:07:35.666ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96312024-03-02T01:07:35.666ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96322024-03-02T01:07:35.666ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96332024-03-02T01:07:35.667ZINFOcrucible: [2] client re-new 0 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96342024-03-02T01:07:35.667ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96352024-03-02T01:07:36.667ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96362024-03-02T01:07:36.668ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96372024-03-02T01:07:36.668ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96382024-03-02T01:07:36.668ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96392024-03-02T01:07:36.668ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96402024-03-02T01:07:36.668ZINFOcrucible: [2] client re-new 0 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96412024-03-02T01:07:36.668ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9642 Mar 02 01:07:37.667 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: e0404a45-c166-4792-9785-1e579445a257, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
9643 001/300 Write block 2732 len 12288 data: 2 2 3
9644 002/300 Read block 4836 len 20480
96452024-03-02T01:07:37.669ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96462024-03-02T01:07:37.669ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96472024-03-02T01:07:37.669ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96482024-03-02T01:07:37.669ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96492024-03-02T01:07:37.669ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96502024-03-02T01:07:37.669ZINFOcrucible: [2] client re-new 2 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
96512024-03-02T01:07:37.669ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
9652 003/300 Write block 1531 len 24576 data: 5 5 5 4 4 3
9653 004/300 Write block 74 len 28672 data: 4 3 2 2 3 3 2
9654 005/300 Read block 983 len 32768
9655 006/300 Write block 4903 len 16384 data: 3 3 4 3
9656 007/300 Read block 169 len 36864
9657 008/300 Write block 3373 len 8192 data: 5 4
9658 009/300 Read block 320 len 16384
9659 010/300 Write block 864 len 20480 data: 3 3 3 3 4
9660 011/300 Write block 198 len 24576 data: 3 3 3 3 4 3
9661 012/300 Read block 3480 len 28672
9662 013/300 Write block 4174 len 8192 data: 3 4
9663 014/300 Write block 4400 len 24576 data: 4 4 4 5 5 5
9664 015/300 Write block 2744 len 16384 data: 4 4 5 5
9665 016/300 Read block 3532 len 8192
9666 017/300 Write block 2304 len 16384 data: 2 2 2 2
9667 018/300 Read block 1747 len 12288
9668 019/300 Read block 4260 len 36864
9669 020/300 Write block 4725 len 28672 data: 4 4 4 4 5 4 4
9670 021/300 Read block 3821 len 20480
9671 022/300 Read block 2319 len 36864
9672 023/300 Write block 4234 len 24576 data: 6 5 5 5 4 3
9673 024/300 Flush
9674 025/300 Write block 2283 len 28672 data: 2 3 3 4 3 3 2
9675 026/300 Flush
9676 027/300 Write block 372 len 16384 data: 4 4 4 4
9677 028/300 Write block 2991 len 32768 data: 4 4 6 7 6 5 4 3
9678 029/300 Read block 2953 len 20480
9679 030/300 Read block 6 len 20480
9680 031/300 Write block 1401 len 4096 data: 4
9681 032/300 Flush
9682 033/300 Read block 4106 len 12288
9683 034/300 Write block 4391 len 16384 data: 4 3 3 3
9684 035/300 Read block 3468 len 32768
9685 036/300 Write block 3775 len 20480 data: 3 4 5 5 6
9686 037/300 Flush
9687 038/300 Read block 4951 len 24576
9688 039/300 Write block 869 len 24576 data: 4 4 3 3 2 2
9689 040/300 Read block 1782 len 40960
9690 041/300 Write block 3487 len 36864 data: 4 5 4 3 3 3 2 2 3
9691 042/300 Write block 3760 len 12288 data: 4 3 3
9692 043/300 Write block 2865 len 8192 data: 4 4
9693 044/300 Write block 2628 len 4096 data: 2
9694 045/300 Flush
9695 046/300 Read block 2764 len 28672
9696 047/300 Write block 764 len 8192 data: 3 4
9697 048/300 Read block 4002 len 36864
9698 049/300 Write block 1529 len 12288 data: 6 5 6
9699 050/300 Write block 3564 len 40960 data: 4 4 4 4 4 4 3 2 3 3
9700 051/300 Read block 3599 len 36864
9701 052/300 Write block 1869 len 8192 data: 3 3
9702 053/300 Write block 4719 len 12288 data: 2 3 3
9703 054/300 Read block 758 len 12288
9704 055/300 Read block 2039 len 28672
9705 056/300 Write block 2362 len 8192 data: 5 5
9706 057/300 Write block 3330 len 36864 data: 5 4 3 3 4 3 4 4 4
9707 058/300 Read block 3364 len 32768
9708 059/300 Write block 1038 len 8192 data: 5 4
9709 060/300 Flush
9710 061/300 Read block 2889 len 16384
9711 062/300 Read block 533 len 32768
9712 063/300 Flush
9713 064/300 Write block 4496 len 24576 data: 2 2 2 2 2 3
9714 065/300 Read block 1611 len 4096
9715 066/300 Write block 942 len 20480 data: 4 5 6 6 4
9716 067/300 Read block 1561 len 36864
9717 068/300 Write block 691 len 24576 data: 3 4 4 3 3 3
9718 069/300 Write block 3340 len 20480 data: 4 3 5 5 5
9719 070/300 Read block 4863 len 36864
9720 071/300 Flush
9721 072/300 Write block 3351 len 40960 data: 4 5 6 5 4 5 5 4 4 4
9722 073/300 Read block 3025 len 24576
9723 074/300 Write block 2426 len 4096 data: 2
9724 075/300 Flush
9725 076/300 Read block 4558 len 28672
9726 077/300 Read block 561 len 24576
9727 078/300 Read block 3705 len 36864
9728 079/300 Write block 2622 len 4096 data: 2
9729 080/300 Write block 4762 len 28672 data: 5 5 5 4 4 5 4
9730 081/300 Write block 241 len 24576 data: 6 5 5 5 5 5
9731 082/300 Write block 3692 len 12288 data: 3 3 3
9732 083/300 Write block 2018 len 8192 data: 4 3
9733 084/300 Write block 1442 len 4096 data: 3
9734 085/300 Read block 1342 len 16384
9735 086/300 Read block 1726 len 40960
9736 087/300 Read block 3832 len 32768
9737 088/300 Write block 468 len 20480 data: 4 4 4 4 4
9738 089/300 Read block 3474 len 36864
9739 090/300 Write block 4039 len 12288 data: 4 4 4
9740 091/300 Read block 4501 len 12288
9741 092/300 Read block 363 len 16384
9742 093/300 Read block 2416 len 28672
9743 094/300 Read block 244 len 20480
9744 095/300 Read block 4361 len 8192
9745 096/300 Read block 4787 len 40960
9746 097/300 Read block 3225 len 12288
9747 098/300 Read block 3279 len 8192
9748 099/300 Write block 978 len 32768 data: 2 2 2 2 3 3 3 3
9749 100/300 Read block 2323 len 40960
9750 101/300 Write block 1271 len 20480 data: 2 2 2 2 2
9751 102/300 Read block 1937 len 36864
9752 103/300 Read block 342 len 36864
9753 104/300 Write block 1435 len 4096 data: 3
9754 105/300 Write block 3994 len 24576 data: 2 2 2 2 2 2
9755 106/300 Write block 2354 len 4096 data: 2
9756 107/300 Write block 4601 len 12288 data: 4 5 5
9757 108/300 Flush
9758 109/300 Read block 4250 len 24576
9759 110/300 Write block 1572 len 20480 data: 3 3 3 3 4
9760 111/300 Write block 43 len 12288 data: 5 5 5
9761 112/300 Write block 2088 len 32768 data: 4 5 6 6 4 4 4 4
9762 113/300 Read block 1969 len 24576
9763 114/300 Read block 4908 len 24576
9764 115/300 Flush
9765 116/300 Write block 3529 len 32768 data: 3 4 4 4 4 4 4 3
9766 117/300 Write block 2049 len 32768 data: 4 3 3 3 3 3 2 2
9767 118/300 Write block 613 len 28672 data: 8 7 7 5 4 3 3
9768 119/300 Read block 2445 len 16384
9769 120/300 Read block 2373 len 36864
9770 121/300 Read block 4387 len 12288
9771 122/300 Write block 2149 len 28672 data: 3 2 2 2 2 2 2
9772 123/300 Read block 4660 len 8192
9773 124/300 Write block 3781 len 20480 data: 7 6 6 5 5
9774 125/300 Read block 2359 len 36864
9775 126/300 Write block 712 len 28672 data: 2 2 2 2 2 2 2
9776 127/300 Flush
9777 128/300 Write block 2586 len 20480 data: 4 4 4 4 4
9778 129/300 Read block 907 len 4096
9779 130/300 Read block 2794 len 8192
9780 131/300 Write block 3991 len 8192 data: 4 3
9781 132/300 Read block 4952 len 36864
9782 133/300 Write block 4876 len 28672 data: 2 2 3 2 2 2 2
9783 134/300 Read block 4681 len 24576
9784 135/300 Read block 1507 len 4096
9785 136/300 Write block 3719 len 28672 data: 4 4 4 5 4 4 5
9786 137/300 Flush
9787 138/300 Write block 2923 len 24576 data: 3 2 2 2 3 3
9788 139/300 Write block 3542 len 24576 data: 2 2 2 2 3 3
9789 140/300 Flush
9790 141/300 Read block 4718 len 12288
9791 142/300 Write block 966 len 8192 data: 3 2
9792 143/300 Flush
9793 144/300 Write block 314 len 12288 data: 2 2 2
9794 145/300 Write block 2037 len 36864 data: 2 3 4 6 6 6 6 3 3
9795 146/300 Write block 1309 len 28672 data: 2 2 2 2 2 2 2
9796 147/300 Write block 3319 len 20480 data: 2 2 2 3 3
9797 148/300 Read block 1437 len 40960
9798 149/300 Read block 3489 len 20480
9799 150/300 Read block 4076 len 36864
9800 151/300 Write block 4432 len 16384 data: 4 3 2 2
9801 152/300 Read block 621 len 40960
9802 153/300 Read block 1980 len 24576
9803 154/300 Read block 1182 len 24576
9804 155/300 Read block 298 len 28672
9805 156/300 Write block 2607 len 8192 data: 2 2
9806 157/300 Write block 4643 len 20480 data: 6 7 6 7 7
9807 158/300 Read block 4264 len 8192
9808 159/300 Read block 2198 len 12288
9809 160/300 Write block 905 len 12288 data: 3 3 3
9810 161/300 Read block 3182 len 16384
9811 162/300 Read block 4921 len 4096
9812 163/300 Read block 836 len 8192
9813 164/300 Write block 3922 len 36864 data: 4 5 5 5 5 5 2 2 3
9814 165/300 Flush
9815 166/300 Flush
9816 167/300 Read block 3719 len 32768
9817 168/300 Read block 567 len 4096
9818 169/300 Write block 4353 len 4096 data: 4
9819 170/300 Write block 903 len 24576 data: 2 2 4 4 4 3
9820 171/300 Read block 4212 len 16384
9821 172/300 Write block 1565 len 40960 data: 3 3 2 2 4 3 2 4 4 4
9822 173/300 Read block 3565 len 28672
9823 174/300 Flush
9824 175/300 Read block 4030 len 12288
9825 176/300 Read block 4257 len 16384
9826 177/300 Read block 1037 len 8192
9827 178/300 Write block 870 len 8192 data: 5 4
9828 179/300 Read block 3343 len 12288
9829 180/300 Flush
9830 181/300 Write block 2438 len 16384 data: 3 4 4 4
9831 182/300 Read block 716 len 24576
9832 183/300 Read block 2494 len 24576
9833 184/300 Write block 752 len 40960 data: 4 3 3 3 3 4 4 3 3 3
9834 185/300 Write block 263 len 40960 data: 3 3 3 5 5 4 5 5 7 5
9835 186/300 Write block 4362 len 24576 data: 3 4 4 4 4 4
9836 187/300 Read block 3211 len 32768
9837 188/300 Read block 2685 len 24576
9838 189/300 Write block 3642 len 8192 data: 3 4
9839 190/300 Write block 4073 len 8192 data: 5 5
9840 191/300 Write block 4051 len 28672 data: 4 4 3 2 2 3 3
9841 192/300 Read block 2426 len 24576
9842 193/300 Write block 3449 len 28672 data: 5 5 5 5 4 4 4
9843 194/300 Write block 2686 len 8192 data: 4 3
9844 195/300 Read block 920 len 4096
9845 196/300 Read block 2934 len 40960
9846 197/300 Write block 3946 len 36864 data: 2 2 2 4 5 4 4 3 2
9847 198/300 Write block 3578 len 8192 data: 2 2
9848 199/300 Read block 2094 len 24576
9849 200/300 Read block 878 len 24576
9850 201/300 Write block 2471 len 32768 data: 3 3 3 3 3 3 3 3
9851 202/300 Read block 850 len 40960
9852 203/300 Read block 1257 len 28672
9853 204/300 Read block 261 len 28672
9854 205/300 Read block 3670 len 8192
9855 206/300 Read block 483 len 12288
9856 207/300 Read block 4510 len 40960
9857 208/300 Read block 4147 len 16384
9858 209/300 Flush
9859 210/300 Flush
9860 211/300 Write block 3885 len 12288 data: 2 2 2
9861 212/300 Read block 4666 len 40960
9862 213/300 Flush
9863 214/300 Write block 2359 len 20480 data: 4 5 5 6 6
9864 215/300 Read block 4038 len 12288
9865 216/300 Read block 669 len 20480
9866 217/300 Read block 418 len 40960
9867 218/300 Read block 1962 len 36864
9868 219/300 Read block 3153 len 20480
9869 220/300 Write block 2016 len 24576 data: 4 4 5 4 2 3
9870 221/300 Read block 156 len 20480
9871 222/300 Read block 776 len 8192
9872 223/300 Read block 2522 len 4096
9873 224/300 Write block 4832 len 20480 data: 3 3 2 3 4
9874 225/300 Write block 1348 len 12288 data: 2 2 2
9875 226/300 Write block 434 len 20480 data: 2 2 2 2 2
9876 227/300 Write block 4304 len 32768 data: 3 3 3 2 3 3 3 4
9877 228/300 Write block 1649 len 28672 data: 4 4 3 3 3 3 3
9878 229/300 Read block 2675 len 32768
9879 230/300 Write block 4946 len 40960 data: 3 3 3 2 3 3 4 4 4 4
9880 231/300 Write block 467 len 16384 data: 4 5 5 5
9881 232/300 Read block 3667 len 36864
9882 233/300 Read block 217 len 8192
9883 234/300 Write block 2411 len 36864 data: 2 2 2 3 2 2 3 2 2
9884 235/300 Write block 4394 len 4096 data: 4
9885 236/300 Read block 711 len 16384
9886 237/300 Read block 3681 len 4096
9887 238/300 Flush
9888 239/300 Write block 4984 len 28672 data: 4 4 5 3 4 4 5
9889 240/300 Read block 1258 len 40960
9890 241/300 Write block 3221 len 20480 data: 2 2 2 3 5
9891 242/300 Read block 1241 len 32768
9892 243/300 Read block 2924 len 28672
9893 244/300 Read block 1469 len 8192
9894 245/300 Read block 3417 len 28672
9895 246/300 Write block 4400 len 20480 data: 5 5 5 6 6
9896 247/300 Read block 761 len 32768
9897 248/300 Write block 2761 len 8192 data: 3 3
9898 249/300 Flush
9899 250/300 Read block 2682 len 16384
9900 251/300 Write block 4897 len 28672 data: 2 2 2 2 3 3 4
9901 252/300 Write block 2400 len 20480 data: 3 3 3 3 4
9902 253/300 Read block 1990 len 40960
9903 254/300 Read block 3836 len 28672
9904 255/300 Flush
9905 256/300 Read block 4753 len 12288
9906 257/300 Flush
9907 258/300 Read block 867 len 16384
9908 259/300 Read block 4575 len 36864
9909 260/300 Read block 1985 len 16384
9910 261/300 Read block 1059 len 4096
9911 262/300 Read block 1381 len 24576
9912 263/300 Write block 639 len 20480 data: 4 5 5 4 4
9913 264/300 Flush
9914 265/300 Read block 250 len 32768
9915 266/300 Write block 1450 len 24576 data: 4 3 2 2 2 2
9916 267/300 Write block 645 len 12288 data: 5 5 5
9917 268/300 Write block 529 len 40960 data: 3 4 4 6 6 6 7 7 6 4
9918 269/300 Read block 1574 len 12288
9919 270/300 Read block 2183 len 40960
9920 271/300 Write block 1473 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9921 272/300 Read block 2842 len 28672
9922 273/300 Write block 4298 len 28672 data: 6 6 4 4 4 3 4
9923 274/300 Read block 1709 len 24576
9924 275/300 Flush
9925 276/300 Write block 292 len 24576 data: 3 4 3 7 6 6
9926 277/300 Write block 3298 len 12288 data: 4 3 2
9927 278/300 Write block 2927 len 28672 data: 4 4 2 2 2 2 3
9928 279/300 Read block 2499 len 16384
9929 280/300 Read block 1693 len 8192
9930 281/300 Write block 390 len 20480 data: 4 4 3 3 3
9931 282/300 Write block 197 len 24576 data: 4 4 4 4 4 5
9932 283/300 Read block 4918 len 16384
9933 284/300 Write block 3194 len 4096 data: 4
9934 285/300 Write block 811 len 8192 data: 4 6
9935 286/300 Flush
9936 287/300 Write block 2751 len 12288 data: 5 5 5
9937 288/300 Read block 1899 len 36864
9938 289/300 Read block 1903 len 24576
9939 290/300 Read block 2494 len 20480
9940 291/300 Flush
9941 292/300 Write block 1759 len 8192 data: 5 4
9942 293/300 Read block 3595 len 40960
9943 294/300 Read block 1773 len 24576
9944 295/300 Read block 3884 len 40960
9945 296/300 Read block 4348 len 24576
9946 297/300 Read block 3928 len 32768
9947 298/300 Write block 2626 len 16384 data: 2 2 3 2
9948 299/300 Read block 1697 len 32768
9949 300/300 Write block 2661 len 20480 data: 2 2 2 2 2
9950 Main task has work to do, go find it
9951 got dsc Start(2)
9952 start 2
9953 Mar 02 01:07:38.050 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/2, method: POST, req_id: 507da0c3-eb15-49bf-a63b-61cd1dce0cf7, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
9954 Replay: started 2, returned:Ok(())
9955 ----------------------------------------------------------------
9956 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9957 States: Active Active Offline
9958 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9959 3317 Acked 4316 Write 3 Done Done New true
9960 3318 Acked 4317 Read 5 Done Done New true
9961 3319 Acked 4318 Write 6 Done Done New false
9962 3320 Acked 4319 Write 7 Done Done New false
9963 3321 Acked 4320 Read 8 Done Done New false
9964 3322 Acked 4321 Write 4 Done Done New false
9965 3323 Acked 4322 Read 9 Done Done New false
9966 3324 Acked 4323 Write 2 Done Done New false
9967 3325 Acked 4324 Read 4 Done Done New false
9968 3326 Acked 4325 Write 5 Done Done New false
9969 3327 Acked 4326 Write 6 Done Done New false
9970 3328 Acked 4327 Read 7 Done Done New false
9971 3329 Acked 4328 Write 2 Done Done New false
9972 3330 Acked 4329 Write 6 Done Done New false
9973 3331 Acked 4330 Write 4 Done Done New false
9974 3332 Acked 4331 Read 2 Done Done New false
9975 3333 Acked 4332 Write 4 Done Done New false
9976 3334 Acked 4333 Read 3 Done Done New false
9977 3335 Acked 4334 Read 9 Done Done New false
9978 3336 Acked 4335 Write 7 Done Done New false
9979 3337 Acked 4336 Read 5 Done Done New false
9980 3338 Acked 4337 Read 9 Done Done New false
9981 3339 Acked 4338 Write 6 Done Done New false
9982 3340 Acked 4339 Flush 0 Done Done New false
9983 3341 Acked 4340 Write 7 Done Done New false
9984 3342 Acked 4341 Flush 0 Done Done New false
9985 3343 Acked 4342 Write 4 Done Done New false
9986 3344 Acked 4343 Write 8 Done Done New false
9987 3345 Acked 4344 Read 5 Done Done New false
9988 3346 Acked 4345 Read 5 Done Done New false
9989 3347 Acked 4346 Write 1 Done Done New false
9990 3348 Acked 4347 Flush 0 Done Done New false
9991 3349 Acked 4348 Read 3 Done Done New false
9992 3350 Acked 4349 Write 4 Done Done New false
9993 3351 Acked 4350 Read 8 Done Done New false
9994 3352 Acked 4351 Write 5 Done Done New false
9995 3353 Acked 4352 Flush 0 Done Done New false
9996 3354 Acked 4353 Read 6 Done Done New false
9997 3355 Acked 4354 Write 6 Done Done New false
9998 3356 Acked 4355 Read 10 Done Done New false
9999 3357 Acked 4356 Write 9 Done Done New false
10000 3358 Acked 4357 Write 3 Done Done New false
10001 3359 Acked 4358 Write 2 Done Done New false
10002 3360 Acked 4359 Write 1 Done Done New false
10003 3361 Acked 4360 Flush 0 Done Done New false
10004 3362 Acked 4361 Read 7 Done Done New false
10005 3363 Acked 4362 Write 2 Done Done New false
10006 3364 Acked 4363 Read 9 Done Done New false
10007 3365 Acked 4364 Write 3 Done Done New false
10008 3366 Acked 4365 Write 10 Done Done New false
10009 3367 Acked 4366 Read 9 Done Done New false
10010 3368 Acked 4367 Write 2 Done Done New false
10011 3369 Acked 4368 Write 3 Done Done New false
10012 3370 Acked 4369 Read 3 Done Done New false
10013 3371 Acked 4370 Read 7 Done Done New false
10014 3372 Acked 4371 Write 2 Done Done New false
10015 3373 Acked 4372 Write 9 Done Done New false
10016 3374 Acked 4373 Read 8 Done Done New false
10017 3375 Acked 4374 Write 2 Done Done New false
10018 3376 Acked 4375 Flush 0 Done Done New false
10019 3377 Acked 4376 Read 4 Done Done New false
10020 3378 Acked 4377 Read 8 Done Done New false
10021 3379 Acked 4378 Flush 0 Done Done New false
10022 3380 Acked 4379 Write 6 Done Done New false
10023 3381 Acked 4380 Read 1 Done Done New false
10024 3382 Acked 4381 Write 5 Done Done New false
10025 3383 Acked 4382 Read 9 Done Done New false
10026 3384 Acked 4383 Write 6 Done Done New false
10027 3385 Acked 4384 Write 5 Done Done New false
10028 3386 Acked 4385 Read 9 Done Done New false
10029 3387 Acked 4386 Flush 0 Done Done New false
10030 3388 Acked 4387 Write 10 Done Done New false
10031 3389 Acked 4388 Read 6 Done Done New false
10032 3390 Acked 4389 Write 1 Done Done New false
10033 3391 Acked 4390 Flush 0 Done Done New false
10034 3392 Acked 4391 Read 7 Done Done New false
10035 3393 Acked 4392 Read 6 Done Done New false
10036 3394 Acked 4393 Read 9 Done Done New false
10037 3395 Acked 4394 Write 1 Done Done New false
10038 3396 Acked 4395 Write 7 Done Done New false
10039 3397 Acked 4396 Write 6 Done Done New false
10040 3398 Acked 4397 Write 3 Done Done New false
10041 3399 Acked 4398 Write 2 Done Done New false
10042 3400 Acked 4399 Write 1 Done Done New false
10043 3401 Acked 4400 Read 4 Done Done New false
10044 3402 Acked 4401 Read 10 Done Done New false
10045 3403 Acked 4402 Read 8 Done Done New false
10046 3404 Acked 4403 Write 5 Done Done New false
10047 3405 Acked 4404 Read 9 Done Done New false
10048 3406 Acked 4405 Write 3 Done Done New false
10049 3407 Acked 4406 Read 3 Done Done New false
10050 3408 Acked 4407 Read 4 Done Done New false
10051 3409 Acked 4408 Read 7 Done Done New false
10052 3410 Acked 4409 Read 5 Done Done New false
10053 3411 Acked 4410 Read 2 Done Done New false
10054 3412 Acked 4411 Read 10 Done Done New false
10055 3413 Acked 4412 Read 3 Done Done New false
10056 3414 Acked 4413 Read 2 Done Done New false
10057 3415 Acked 4414 Write 8 Done Done New false
10058 3416 Acked 4415 Read 10 Done Done New false
10059 3417 Acked 4416 Write 5 Done Done New false
10060 3418 Acked 4417 Read 9 Done Done New false
10061 3419 Acked 4418 Read 9 Done Done New false
10062 3420 Acked 4419 Write 1 Done Done New false
10063 3421 Acked 4420 Write 6 Done Done New false
10064 3422 Acked 4421 Write 1 Done Done New false
10065 3423 Acked 4422 Write 3 Done Done New false
10066 3424 Acked 4423 Flush 0 Done Done New false
10067 3425 Acked 4424 Read 6 Done Done New false
10068 3426 Acked 4425 Write 5 Done Done New false
10069 3427 Acked 4426 Write 3 Done Done New false
10070 3428 Acked 4427 Write 8 Done Done New false
10071 3429 Acked 4428 Read 6 Done Done New false
10072 3430 Acked 4429 Read 6 Done Done New false
10073 3431 Acked 4430 Flush 0 Done Done New false
10074 3432 Acked 4431 Write 8 Done Done New false
10075 3433 Acked 4432 Write 8 Done Done New false
10076 3434 Acked 4433 Write 7 Done Done New false
10077 3435 Acked 4434 Read 4 Done Done New false
10078 3436 Acked 4435 Read 9 Done Done New false
10079 3437 Acked 4436 Read 3 Done Done New false
10080 3438 Acked 4437 Write 7 Done Done New false
10081 3439 Acked 4438 Read 2 Done Done New false
10082 3440 Acked 4439 Write 5 Done Done New false
10083 3441 Acked 4440 Read 9 Done Done New false
10084 3442 Acked 4441 Write 7 Done Done New false
10085 3443 Acked 4442 Flush 0 Done Done New false
10086 3444 Acked 4443 Write 5 Done Done New false
10087 3445 Acked 4444 Read 1 Done Done New false
10088 3446 Acked 4445 Read 2 Done Done New false
10089 3447 Acked 4446 Write 2 Done Done New false
10090 3448 Acked 4447 Read 9 Done Done New false
10091 3449 Acked 4448 Flush 0 Done Done New false
10092 3450 Acked 4449 Write 7 Done Done New false
10093 3451 Acked 4450 Read 6 Done Done New false
10094 3452 Acked 4451 Read 1 Done Done New false
10095 3453 Acked 4452 Write 7 Done Done New false
10096 3454 Acked 4453 Flush 0 Done Done New false
10097 3455 Acked 4454 Write 6 Done Done New false
10098 3456 Acked 4455 Write 6 Done Done New false
10099 3457 Acked 4456 Flush 0 Done Done New false
10100 3458 Acked 4457 Read 3 Done Done New false
10101 3459 Acked 4458 Write 2 Done Done New false
10102 3460 Acked 4459 Flush 0 Done Done New false
10103 3461 Acked 4460 Write 3 Done Done New false
10104 3462 Acked 4461 Write 9 Done Done New false
10105 3463 Acked 4462 Write 7 Done Done New false
10106 3464 Acked 4463 Write 5 Done Done New false
10107 3465 Acked 4464 Read 10 Done Done New false
10108 3466 Acked 4465 Read 5 Done Done New false
10109 3467 Acked 4466 Read 9 Done Done New false
10110 3468 Acked 4467 Write 4 Done Done New false
10111 3469 Acked 4468 Read 10 Done Done New false
10112 3470 Acked 4469 Read 6 Done Done New false
10113 3471 Acked 4470 Read 6 Done Done New false
10114 3472 Acked 4471 Read 7 Done Done New false
10115 3473 Acked 4472 Write 2 Done Done New false
10116 3474 Acked 4473 Write 5 Done Done New false
10117 3475 Acked 4474 Read 2 Done Done New false
10118 3476 Acked 4475 Read 3 Done Done New false
10119 3477 Acked 4476 Write 3 Done Done New false
10120 3478 Acked 4477 Read 4 Done Done New false
10121 3479 Acked 4478 Read 1 Done Done New false
10122 3480 Acked 4479 Read 2 Done Done New false
10123 3481 Acked 4480 Write 9 Done Done New false
10124 3482 Acked 4481 Flush 0 Done Done New false
10125 3483 Acked 4482 Flush 0 Done Done New false
10126 3484 Acked 4483 Read 8 Done Done New false
10127 3485 Acked 4484 Read 1 Done Done New false
10128 3486 Acked 4485 Write 1 Done Done New false
10129 3487 Acked 4486 Write 6 Done Done New false
10130 3488 Acked 4487 Read 4 Done Done New false
10131 3489 Acked 4488 Write 10 Done Done New false
10132 3490 Acked 4489 Read 7 Done Done New false
10133 3491 Acked 4490 Flush 0 Done Done New false
10134 3492 Acked 4491 Read 3 Done Done New false
10135 3493 Acked 4492 Read 4 Done Done New false
10136 3494 Acked 4493 Read 2 Done Done New false
10137 3495 Acked 4494 Write 2 Done Done New false
10138 3496 Acked 4495 Read 3 Done Done New false
10139 3497 Acked 4496 Flush 0 Done Done New false
10140 3498 Acked 4497 Write 4 Done Done New false
10141 3499 Acked 4498 Read 6 Done Done New false
10142 3500 Acked 4499 Read 6 Done Done New false
10143 3501 Acked 4500 Write 10 Done Done New false
10144 3502 Acked 4501 Write 10 Done Done New false
10145 3503 Acked 4502 Write 6 Done Done New false
10146 3504 Acked 4503 Read 8 Done Done New false
10147 3505 Acked 4504 Read 6 Done Done New false
10148 3506 Acked 4505 Write 2 Done Done New false
10149 3507 Acked 4506 Write 2 Done Done New false
10150 3508 Acked 4507 Write 7 Done Done New false
10151 3509 Acked 4508 Read 6 Done Done New false
10152 3510 Acked 4509 Write 7 Done Done New false
10153 3511 Acked 4510 Write 2 Done Done New false
10154 3512 Acked 4511 Read 1 Done Done New false
10155 3513 Acked 4512 Read 10 Done Done New false
10156 3514 Acked 4513 Write 9 Done Done New false
10157 3515 Acked 4514 Write 2 Done Done New false
10158 3516 Acked 4515 Read 6 Done Done New false
10159 3517 Acked 4516 Read 6 Done Done New false
10160 3518 Acked 4517 Write 8 Done Done New false
10161 3519 Acked 4518 Read 10 Done Done New false
10162 3520 Acked 4519 Read 7 Done Done New false
10163 3521 Acked 4520 Read 7 Done Done New false
10164 3522 Acked 4521 Read 2 Done Done New false
10165 3523 Acked 4522 Read 3 Done Done New false
10166 3524 Acked 4523 Read 10 Done Done New false
10167 3525 Acked 4524 Read 4 Done Done New false
10168 3526 Acked 4525 Flush 0 Done Done New false
10169 3527 Acked 4526 Flush 0 Done Done New false
10170 3528 Acked 4527 Write 3 Done Done New false
10171 3529 Acked 4528 Read 10 Done Done New false
10172 3530 Acked 4529 Flush 0 Done Done New false
10173 3531 Acked 4530 Write 5 Done Done New false
10174 3532 Acked 4531 Read 3 Done Done New false
10175 3533 Acked 4532 Read 5 Done Done New false
10176 3534 Acked 4533 Read 10 Done Done New false
10177 3535 Acked 4534 Read 9 Done Done New false
10178 3536 Acked 4535 Read 5 Done Done New false
10179 3537 Acked 4536 Write 6 Done Done New false
10180 3538 Acked 4537 Read 5 Done Done New false
10181 3539 Acked 4538 Read 2 Done Done New false
10182 3540 Acked 4539 Read 1 Done Done New false
10183 3541 Acked 4540 Write 5 Done Done New false
10184 3542 Acked 4541 Write 3 Done Done New false
10185 3543 Acked 4542 Write 5 Done Done New false
10186 3544 Acked 4543 Write 8 Done Done New false
10187 3545 Acked 4544 Write 7 Done Done New false
10188 3546 Acked 4545 Read 8 Done Done New false
10189 3547 Acked 4546 Write 10 Done Done New false
10190 3548 Acked 4547 Write 4 Done Done New false
10191 3549 Acked 4548 Read 9 Done Done New false
10192 3550 Acked 4549 Read 2 Done Done New false
10193 3551 Acked 4550 Write 9 Done Done New false
10194 3552 Acked 4551 Write 1 Done Done New false
10195 3553 Acked 4552 Read 4 Done Done New false
10196 3554 Acked 4553 Read 1 Done Done New false
10197 3555 Acked 4554 Flush 0 Done Done New false
10198 3556 Acked 4555 Write 7 Done Done New false
10199 3557 Acked 4556 Read 10 Done Done New false
10200 3558 Acked 4557 Write 5 Done Done New false
10201 3559 Acked 4558 Read 8 Done Done New false
10202 3560 Acked 4559 Read 7 Done Done New false
10203 3561 Acked 4560 Read 2 Done Done New false
10204 3562 Acked 4561 Read 7 Done Done New false
10205 3563 Acked 4562 Write 5 Done Done New false
10206 3564 Acked 4563 Read 8 Done Done New false
10207 3565 Acked 4564 Write 2 Done Done New false
10208 3566 Acked 4565 Flush 0 Done Done New false
10209 3567 Acked 4566 Read 4 Done Done New false
10210 3568 Acked 4567 Write 7 Done Done New false
10211 3569 Acked 4568 Write 5 Done Done New false
10212 3570 Acked 4569 Read 10 Done Done New false
10213 3571 Acked 4570 Read 7 Done Done New false
10214 3572 Acked 4571 Flush 0 Done Done New false
10215 3573 Acked 4572 Read 3 Done Done New false
10216 3574 Acked 4573 Flush 0 Done Done New false
10217 3575 Acked 4574 Read 4 Done Done New false
10218 3576 Acked 4575 Read 9 Done Done New false
10219 3577 Acked 4576 Read 4 Done Done New false
10220 3578 Acked 4577 Read 1 Done Done New false
10221 3579 Acked 4578 Read 6 Done Done New false
10222 3580 Acked 4579 Write 5 Done Done New false
10223 3581 Acked 4580 Flush 0 Done Done New false
10224 3582 Acked 4581 Read 8 Done Done New false
10225 3583 Acked 4582 Write 6 Done Done New false
10226 3584 Acked 4583 Write 3 Done Done New false
10227 3585 Acked 4584 Write 10 Done Done New false
10228 3586 Acked 4585 Read 3 Done Done New false
10229 3587 Acked 4586 Read 10 Done Done New false
10230 3588 Acked 4587 Write 10 Done Done New false
10231 3589 Acked 4588 Read 7 Done Done New false
10232 3590 Acked 4589 Write 7 Done Done New false
10233 3591 Acked 4590 Read 6 Done Done New false
10234 3592 Acked 4591 Flush 0 Done Done New false
10235 3593 Acked 4592 Write 6 Done Done New false
10236 3594 Acked 4593 Write 3 Done Done New false
10237 3595 Acked 4594 Write 7 Done Done New false
10238 3596 Acked 4595 Read 4 Done Done New false
10239 3597 Acked 4596 Read 2 Done Done New false
10240 3598 Acked 4597 Write 5 Done Done New false
10241 3599 Acked 4598 Write 6 Done Done New false
10242 3600 Acked 4599 Read 4 Done Done New false
10243 3601 Acked 4600 Write 1 Done Done New false
10244 3602 Acked 4601 Write 2 Done Done New false
10245 3603 Acked 4602 Flush 0 Done Done New false
10246 3604 Acked 4603 Write 3 Done Done New false
10247 3605 Acked 4604 Read 9 Done Done New false
10248 3606 Acked 4605 Read 6 Done Done New false
10249 3607 Acked 4606 Read 5 Done Done New false
10250 3608 Acked 4607 Flush 0 Done Done New false
10251 3609 Acked 4608 Write 2 Done Done New false
10252 3610 Acked 4609 Read 10 Done Done New false
10253 3611 Acked 4610 Read 6 Done Done New false
10254 3612 Acked 4611 Read 10 Done Done New false
10255 3613 Acked 4612 Read 6 Done Done New false
10256 3614 Acked 4613 Read 8 Done Done New false
10257 3615 Acked 4614 Write 4 Done Done New false
10258 3616 Acked 4615 Read 8 Done Done New false
10259 3617 Acked 4616 Write 5 Done Done New false
10260 STATES DS:0 DS:1 DS:2 TOTAL
10261 New 0 0 301 301
10262 Sent 0 0 0 0
10263 Done 301 301 0 602
10264 Skipped 0 0 0 0
10265 Error 0 0 0 0
10266 Last Flush: 4607 4607 4315
10267 Downstairs last five completed: 4315 4314 4313 4312 4311
10268 Upstairs last five completed: 3617 3616 3615 3614 3613
10269 CLIENT: Up:0 ds:301 act:2
102702024-03-02T01:07:38.670ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102712024-03-02T01:07:38.670ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102722024-03-02T01:07:38.670ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102732024-03-02T01:07:38.670ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102742024-03-02T01:07:38.670ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102752024-03-02T01:07:38.670ZINFOcrucible: [2] client re-new 302 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102762024-03-02T01:07:38.670ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102772024-03-02T01:07:39.672ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102782024-03-02T01:07:39.672ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102792024-03-02T01:07:39.672ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102802024-03-02T01:07:39.672ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102812024-03-02T01:07:39.672ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102822024-03-02T01:07:39.672ZINFOcrucible: [2] client re-new 302 jobs since flush 4315 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102832024-03-02T01:07:39.672ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10284 [8830] got start action
10285 [8830] I am going to restart
10286 Starting downstairs at port 8830
10287 Make output file at "/tmp/dsc/downstairs-8830.txt"
10288 [8830][2] reports Starting
10289 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1174)
10290 [8830][2] reports Running
102912024-03-02T01:07:40.674ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102922024-03-02T01:07:40.674ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102932024-03-02T01:07:40.675ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102942024-03-02T01:07:40.675ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102952024-03-02T01:07:40.675ZINFOcrucible: send last flush ID to this DS: 4315 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102962024-03-02T01:07:40.675ZINFOcrucible: Replied this last flush ID: 4315 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102972024-03-02T01:07:40.675ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102982024-03-02T01:07:40.675ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
102992024-03-02T01:07:40.675ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103002024-03-02T01:07:40.675ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10301 ----------------------------------------------------------------
10302 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10303 Downstairs last five completed: 4617 4616 4615 4614 4613
10304 Upstairs last five completed: 3618 3617 3616 3615 3614
10305 CLIENT: Up:0 ds:0 act:3
10306 Replay: All jobs finished, all DS active.
10307 Main task has work to do, go find it
10308 got dsc Stop(0)
10309 stop 0
10310 Mar 02 01:07:42.053 INFO request completed, latency_us: 324, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1b0170f7-b5d5-4878-ae96-94458cf2d854, remote_addr:[8810] Got stop action so:false kr:false
10311 127.0.0.1:40735, local_addr: 127.0.0.1:9998
103122024-03-02T01:07:42.054ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103132024-03-02T01:07:42.054ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103142024-03-02T01:07:42.054ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103152024-03-02T01:07:42.054ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103162024-03-02T01:07:42.054ZINFOcrucible: [0] client re-new 0 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103172024-03-02T01:07:42.054ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10318 Mar 02 01:07:42.054 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/0, method: GET, req_id: bb10e1c2-d890-4c20-8eb0-22ad3adf7bae, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
10319 [8810] Exited with: ExitStatus(unix_wait_status(9))
10320 [8810][0] reports Exit
103212024-03-02T01:07:43.055ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103222024-03-02T01:07:43.055ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103232024-03-02T01:07:43.055ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103242024-03-02T01:07:43.055ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103252024-03-02T01:07:43.055ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103262024-03-02T01:07:43.055ZINFOcrucible: [0] client re-new 0 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103272024-03-02T01:07:43.056ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103282024-03-02T01:07:44.057ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103292024-03-02T01:07:44.057ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103302024-03-02T01:07:44.057ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103312024-03-02T01:07:44.057ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103322024-03-02T01:07:44.057ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103332024-03-02T01:07:44.057ZINFOcrucible: [0] client re-new 0 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103342024-03-02T01:07:44.057ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103352024-03-02T01:07:45.058ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103362024-03-02T01:07:45.058ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103372024-03-02T01:07:45.058ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103382024-03-02T01:07:45.058ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103392024-03-02T01:07:45.058ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103402024-03-02T01:07:45.058ZINFOcrucible: [0] client re-new 0 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103412024-03-02T01:07:45.058ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10342 Mar 02 01:07:46.057 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/0, method: GET, req_id: 998792f2-2726-4587-8c44-cba9677096fb, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
10343 001/300 Read block 854 len 40960
10344 002/300 Write block 375 len 4096 data: 5
103452024-03-02T01:07:46.060ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103462024-03-02T01:07:46.060ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103472024-03-02T01:07:46.060ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103482024-03-02T01:07:46.060ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10349 {"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:46.060215676Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"0","":"downstairs","session_id":"003/300b12e3267-1b4e-4723-a1fa-a377ace8dffb" Write block 2670 len 24576 data:} 2 2
10350 2 2 2 3
103512024-03-02T01:07:46.060ZINFOcrucible: [0] client re-new 2 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
103522024-03-02T01:07:46.060ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10353 004/300 Read block 4208 len 16384
10354 005/300 Write block 3458 len 4096 data: 3
10355 006/300 Write block 3780 len 20480 data: 8 8 7 7 6
10356 007/300 Write block 1661 len 32768 data: 3 3 3 4 5 5 5 4
10357 008/300 Write block 3678 len 12288 data: 6 6 6
10358 009/300 Read block 582 len 8192
10359 010/300 Write block 1166 len 24576 data: 2 2 2 3 3 2
10360 011/300 Read block 3610 len 36864
10361 012/300 Write block 4341 len 32768 data: 3 2 3 4 6 6 5 4
10362 013/300 Write block 2490 len 32768 data: 2 2 2 2 2 2 2 3
10363 014/300 Read block 834 len 32768
10364 015/300 Write block 493 len 32768 data: 3 3 4 2 2 2 2 2
10365 016/300 Read block 1711 len 20480
10366 017/300 Flush
10367 018/300 Read block 2343 len 40960
10368 019/300 Write block 738 len 40960 data: 2 3 3 3 3 5 4 4 5 5
10369 020/300 Write block 1679 len 40960 data: 3 3 4 4 5 5 4 4 4 4
10370 021/300 Read block 2848 len 20480
10371 022/300 Write block 1618 len 20480 data: 4 4 3 3 3
10372 023/300 Flush
10373 024/300 Read block 3408 len 24576
10374 025/300 Write block 4000 len 40960 data: 2 2 2 2 2 2 2 2 2 3
10375 026/300 Flush
10376 027/300 Write block 2357 len 12288 data: 2 3 5
10377 028/300 Flush
10378 029/300 Write block 4433 len 16384 data: 4 3 3 2
10379 030/300 Write block 1234 len 20480 data: 5 5 5 5 4
10380 031/300 Read block 3222 len 8192
10381 032/300 Read block 2894 len 36864
10382 033/300 Read block 1615 len 4096
10383 034/300 Read block 1277 len 12288
10384 035/300 Write block 1068 len 24576 data: 3 3 3 2 2 2
10385 036/300 Read block 2314 len 16384
10386 037/300 Write block 3348 len 36864 data: 5 5 4 5 6 7 6 5 6
10387 038/300 Read block 205 len 36864
10388 039/300 Read block 142 len 8192
10389 040/300 Read block 3352 len 24576
10390 041/300 Write block 3879 len 20480 data: 3 3 2 2 2
10391 042/300 Write block 973 len 20480 data: 2 2 2 2 2
10392 043/300 Write block 3752 len 16384 data: 3 3 3 5
10393 044/300 Write block 1463 len 40960 data: 3 3 3 3 2 2 2 2 2 2
10394 045/300 Write block 3095 len 24576 data: 4 4 4 4 3 3
10395 046/300 Read block 3932 len 36864
10396 047/300 Write block 493 len 4096 data: 4
10397 048/300 Write block 1528 len 4096 data: 6
10398 049/300 Write block 2774 len 28672 data: 3 3 2 2 4 4 4
10399 050/300 Flush
10400 051/300 Read block 1464 len 16384
10401 052/300 Flush
10402 053/300 Flush
10403 054/300 Write block 2153 len 16384 data: 3 3 3 3
10404 055/300 Write block 214 len 8192 data: 3 3
10405 056/300 Write block 2825 len 16384 data: 4 5 5 5
10406 057/300 Read block 4970 len 24576
10407 058/300 Flush
10408 059/300 Flush
10409 060/300 Read block 1514 len 36864
10410 061/300 Write block 413 len 28672 data: 4 5 4 4 4 4 3
10411 062/300 Write block 2301 len 24576 data: 3 2 2 3 3 3
10412 063/300 Read block 496 len 24576
10413 064/300 Read block 3532 len 28672
10414 065/300 Write block 26 len 40960 data: 2 3 3 3 4 4 4 3 4 3
10415 066/300 Read block 2081 len 28672
10416 067/300 Write block 729 len 16384 data: 3 3 2 3
10417 068/300 Write block 390 len 8192 data: 5 5
10418 069/300 Read block 1543 len 16384
10419 070/300 Write block 1187 len 28672 data: 2 2 2 3 3 3 4
10420 071/300 Flush
10421 072/300 Write block 3818 len 16384 data: 5 5 5 6
10422 073/300 Write block 4734 len 40960 data: 4 3 3 3 2 3 4 4 4 4
10423 074/300 Write block 4686 len 28672 data: 6 5 6 6 5 4 4
10424 075/300 Read block 3002 len 16384
10425 076/300 Write block 939 len 16384 data: 4 4 4 5
10426 077/300 Read block 3772 len 36864
10427 078/300 Write block 103 len 16384 data: 3 3 3 3
10428 079/300 Write block 4803 len 40960 data: 3 4 5 5 6 6 6 3 3 4
10429 080/300 Read block 2840 len 36864
10430 081/300 Write block 1084 len 8192 data: 4 5
10431 082/300 Read block 4746 len 24576
10432 083/300 Read block 4413 len 20480
10433 084/300 Write block 4833 len 36864 data: 4 3 4 5 4 4 5 5 5
10434 085/300 Read block 2278 len 4096
10435 086/300 Write block 523 len 40960 data: 4 3 2 2 3 3 4 5 5 7
10436 087/300 Write block 4395 len 32768 data: 3 5 4 3 3 6 6 6
10437 088/300 Write block 460 len 40960 data: 2 2 2 2 2 2 3 5 6 6
10438 089/300 Read block 4988 len 20480
10439 090/300 Flush
10440 091/300 Read block 22 len 32768
10441 092/300 Write block 2226 len 12288 data: 3 2 3
10442 093/300 Write block 2119 len 24576 data: 4 5 4 3 3 3
10443 094/300 Write block 2558 len 4096 data: 4
10444 095/300 Write block 1990 len 24576 data: 2 2 2 2 2 2
10445 096/300 Flush
10446 097/300 Write block 3746 len 28672 data: 3 3 3 3 3 3 4
10447 098/300 Read block 465 len 36864
10448 099/300 Write block 4669 len 32768 data: 2 3 3 3 2 2 2 2
10449 100/300 Flush
10450 101/300 Read block 1844 len 20480
10451 102/300 Write block 4097 len 36864 data: 5 5 4 4 4 4 3 3 2
10452 103/300 Read block 1066 len 36864
10453 104/300 Flush
10454 105/300 Read block 3716 len 24576
10455 106/300 Read block 216 len 40960
10456 107/300 Read block 57 len 8192
10457 108/300 Flush
10458 109/300 Write block 1705 len 4096 data: 2
10459 110/300 Write block 1698 len 16384 data: 6 4 2 2
10460 111/300 Read block 3851 len 16384
10461 112/300 Write block 4411 len 36864 data: 3 2 2 3 3 3 3 2 2
10462 113/300 Read block 3531 len 28672
10463 114/300 Read block 820 len 8192
10464 115/300 Write block 1937 len 32768 data: 4 3 4 4 4 4 4 5
10465 116/300 Read block 3166 len 20480
10466 117/300 Write block 68 len 16384 data: 5 6 6 5
10467 118/300 Flush
10468 119/300 Read block 1900 len 20480
10469 120/300 Read block 3719 len 24576
10470 121/300 Write block 2414 len 8192 data: 4 3
10471 122/300 Write block 3265 len 36864 data: 4 3 3 3 3 3 3 3 3
10472 123/300 Write block 4090 len 40960 data: 3 4 6 6 4 4 5 6 6 5
10473 124/300 Read block 3174 len 36864
10474 125/300 Write block 932 len 8192 data: 3 3
10475 126/300 Write block 419 len 8192 data: 4 3
10476 127/300 Read block 4540 len 28672
10477 128/300 Read block 666 len 4096
10478 129/300 Read block 3894 len 4096
10479 130/300 Read block 3547 len 16384
10480 131/300 Write block 792 len 4096 data: 3
10481 132/300 Flush
10482 133/300 Read block 4976 len 4096
10483 134/300 Read block 3257 len 12288
10484 135/300 Write block 3748 len 12288 data: 4 4 4
10485 136/300 Read block 890 len 32768
10486 137/300 Flush
10487 138/300 Read block 30 len 36864
10488 139/300 Read block 2337 len 36864
10489 140/300 Read block 3121 len 28672
10490 141/300 Write block 3892 len 28672 data: 3 3 3 3 3 2 2
10491 142/300 Read block 4611 len 20480
10492 143/300 Flush
10493 144/300 Write block 1674 len 16384 data: 4 3 2 2
10494 145/300 Read block 2781 len 40960
10495 146/300 Write block 1567 len 20480 data: 3 3 5 4 3
10496 147/300 Write block 2298 len 12288 data: 3 3 3
10497 148/300 Write block 981 len 24576 data: 3 4 4 4 4 3
10498 149/300 Write block 1498 len 40960 data: 4 2 2 2 3 3 3 3 4 5
10499 150/300 Read block 4985 len 28672
10500 151/300 Read block 2636 len 36864
10501 152/300 Read block 2576 len 24576
10502 153/300 Write block 222 len 4096 data: 4
10503 154/300 Write block 3807 len 8192 data: 6 7
10504 155/300 Read block 803 len 28672
10505 156/300 Read block 1438 len 20480
10506 157/300 Flush
10507 158/300 Write block 1275 len 28672 data: 3 3 3 3 3 3 3
10508 159/300 Flush
10509 160/300 Read block 3940 len 4096
10510 161/300 Read block 1633 len 16384
10511 162/300 Write block 840 len 16384 data: 4 4 3 4
10512 163/300 Read block 2541 len 4096
10513 164/300 Read block 3125 len 16384
10514 165/300 Write block 2871 len 24576 data: 2 2 2 3 3 4
10515 166/300 Read block 231 len 32768
10516 167/300 Write block 4621 len 24576 data: 4 3 3 3 3 3
10517 168/300 Write block 2922 len 12288 data: 4 4 3
10518 169/300 Read block 3310 len 12288
10519 170/300 Write block 689 len 4096 data: 3
10520 171/300 Write block 3145 len 24576 data: 3 3 3 3 4 4
10521 172/300 Flush
10522 173/300 Read block 3865 len 36864
10523 174/300 Read block 1630 len 4096
10524 175/300 Read block 1107 len 28672
10525 176/300 Write block 1366 len 12288 data: 2 2 2
10526 177/300 Write block 3909 len 28672 data: 4 5 5 5 5 5 5
10527 178/300 Read block 1745 len 24576
10528 179/300 Write block 4026 len 32768 data: 4 4 5 5 5 5 5 5
10529 180/300 Write block 1571 len 24576 data: 4 5 5 5 4 5
10530 181/300 Read block 306 len 16384
10531 182/300 Flush
10532 183/300 Write block 4964 len 12288 data: 2 2 2
10533 184/300 Write block 1121 len 4096 data: 3
10534 185/300 Write block 1532 len 12288 data: 6 6 5
10535 186/300 Write block 1300 len 8192 data: 6 5
10536 187/300 Read block 2752 len 28672
10537 188/300 Write block 3356 len 16384 data: 7 6 5 5
10538 189/300 Flush
10539 190/300 Read block 2985 len 36864
10540 191/300 Read block 4844 len 32768
10541 192/300 Write block 3089 len 36864 data: 2 2 2 2 2 3 5 5 5
10542 193/300 Write block 4051 len 20480 data: 5 5 4 3 3
10543 194/300 Read block 427 len 24576
10544 195/300 Write block 3403 len 32768 data: 4 4 4 3 4 4 3 3
10545 196/300 Write block 4169 len 20480 data: 2 2 2 2 2
10546 197/300 Write block 3463 len 32768 data: 4 4 3 3 3 3 3 3
10547 198/300 Write block 3931 len 8192 data: 3 3
10548 199/300 Write block 149 len 16384 data: 3 3 3 3
10549 200/300 Write block 3430 len 36864 data: 2 4 4 4 4 2 2 3 3
10550 201/300 Read block 4831 len 20480
10551 202/300 Read block 2781 len 4096
10552 203/300 Write block 812 len 40960 data: 7 8 9 7 6 7 7 6 5 6
10553 204/300 Read block 700 len 20480
10554 205/300 Read block 1670 len 8192
10555 206/300 Read block 4404 len 20480
10556 207/300 Read block 3438 len 20480
10557 208/300 Write block 3564 len 24576 data: 5 5 5 5 5 5
10558 209/300 Write block 1403 len 16384 data: 4 4 4 4
10559 210/300 Write block 2444 len 32768 data: 5 5 5 4 4 4 3 2
10560 211/300 Read block 1360 len 12288
10561 212/300 Write block 1570 len 8192 data: 5 5
10562 213/300 Write block 3039 len 20480 data: 3 2 2 2 2
10563 214/300 Read block 3257 len 16384
10564 215/300 Read block 1196 len 16384
10565 216/300 Read block 4974 len 8192
10566 217/300 Flush
10567 218/300 Write block 2563 len 8192 data: 4 4
10568 219/300 Flush
10569 220/300 Read block 1756 len 32768
10570 221/300 Write block 1864 len 28672 data: 2 2 3 3 3 4 4
10571 222/300 Write block 447 len 32768 data: 6 7 7 7 5 6 5 5
10572 223/300 Read block 1635 len 24576
10573 224/300 Write block 4265 len 4096 data: 5
10574 225/300 Flush
10575 226/300 Write block 4250 len 16384 data: 2 2 2 2
10576 227/300 Flush
10577 228/300 Read block 2284 len 8192
10578 229/300 Write block 1377 len 8192 data: 2 2
10579 230/300 Read block 1225 len 40960
10580 231/300 Read block 2747 len 28672
10581 232/300 Read block 2832 len 28672
10582 233/300 Flush
10583 234/300 Write block 3246 len 24576 data: 4 4 4 4 4 3
10584 235/300 Flush
10585 236/300 Write block 1437 len 16384 data: 4 4 2 2
10586 237/300 Write block 2552 len 20480 data: 2 2 2 3 4
10587 238/300 Read block 4664 len 8192
10588 239/300 Write block 4983 len 40960 data: 4 5 5 6 4 5 5 6 4 4
10589 240/300 Write block 715 len 8192 data: 3 3
10590 241/300 Read block 2691 len 8192
10591 242/300 Read block 4134 len 36864
10592 243/300 Read block 3163 len 28672
10593 244/300 Write block 725 len 12288 data: 4 4 3
10594 245/300 Write block 3813 len 8192 data: 2 2
10595 246/300 Read block 2248 len 16384
10596 247/300 Read block 1074 len 4096
10597 248/300 Write block 3926 len 8192 data: 6 6
10598 249/300 Read block 1264 len 24576
10599 250/300 Read block 53 len 16384
10600 251/300 Write block 1683 len 28672 data: 6 6 5 5 5 5 4
10601 252/300 Read block 4808 len 8192
10602 253/300 Write block 3845 len 4096 data: 2
10603 254/300 Read block 2897 len 40960
10604 255/300 Read block 1200 len 20480
10605 256/300 Write block 2267 len 28672 data: 2 3 3 4 5 5 4
10606 257/300 Write block 4956 len 24576 data: 2 3 3 2 2 2
10607 258/300 Read block 1403 len 24576
10608 259/300 Write block 1952 len 40960 data: 5 4 5 4 4 3 3 3 3 3
10609 260/300 Write block 4366 len 28672 data: 5 5 3 3 3 3 4
10610 261/300 Read block 4684 len 4096
10611 262/300 Read block 1818 len 32768
10612 263/300 Read block 793 len 8192
10613 264/300 Write block 2252 len 4096 data: 7
10614 265/300 Write block 3856 len 32768 data: 3 4 4 5 5 5 5 6
10615 266/300 Write block 2923 len 36864 data: 5 4 3 3 5 5 3 3 3
10616 267/300 Write block 2595 len 4096 data: 4
10617 268/300 Flush
10618 269/300 Flush
10619 270/300 Read block 3591 len 40960
10620 271/300 Read block 4960 len 8192
10621 272/300 Read block 91 len 8192
10622 273/300 Write block 4404 len 24576 data: 7 6 3 3 3 3
10623 274/300 Write block 2776 len 28672 data: 3 3 5 5 5 4 3
10624 275/300 Write block 3977 len 40960 data: 3 4 5 4 4 4 2 4 4 4
10625 276/300 Write block 1442 len 20480 data: 4 5 5 5 6
10626 277/300 Write block 1992 len 28672 data: 3 3 3 3 2 2 2
10627 278/300 Flush
10628 279/300 Write block 3528 len 20480 data: 5 4 5 5 5
10629 280/300 Read block 3470 len 24576
10630 281/300 Read block 1863 len 36864
10631 282/300 Read block 1815 len 20480
10632 283/300 Read block 255 len 8192
10633 284/300 Write block 97 len 4096 data: 2
10634 285/300 Read block 3381 len 24576
10635 286/300 Read block 4005 len 24576
10636 287/300 Read block 2658 len 40960
10637 288/300 Write block 1201 len 4096 data: 5
10638 289/300 Read block 2863 len 24576
10639 290/300 Flush
10640 291/300 Write block 1040 len 16384 data: 3 2 4 3
10641 292/300 Write block 3743 len 40960 data: 2 2 2 4 4 5 5 5 4 5
10642 293/300 Flush
10643 294/300 Flush
10644 295/300 Read block 4124 len 20480
10645 296/300 Read block 3570 len 12288
10646 297/300 Read block 2459 len 40960
10647 298/300 Write block 4559 len 12288 data: 3 3 3
10648 299/300 Write block 4185 len 8192 data: 4 6
10649 300/300 Write block 2350 len 32768 data: 2 2 2 2 3 2 2 3
10650 Main task has work to do, go find it
10651 got dsc Start(0)
10652 start 0
10653 Mar 02 01:07:46.395 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: bd51142c-c156-40b3-8349-2238fc2946aa, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
10654 Replay: started 0, returned:Ok(())
10655 ----------------------------------------------------------------
10656 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10657 States: Offline Active Active
10658 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10659 3619 Acked 4618 Read 10 New Done Done true
10660 3620 Acked 4619 Write 1 New Done Done true
10661 3621 Acked 4620 Write 6 New Done Done false
10662 3622 Acked 4621 Read 4 New Done Done false
10663 3623 Acked 4622 Write 1 New Done Done false
10664 3624 Acked 4623 Write 5 New Done Done false
10665 3625 Acked 4624 Write 8 New Done Done false
10666 3626 Acked 4625 Write 3 New Done Done false
10667 3627 Acked 4626 Read 2 New Done Done false
10668 3628 Acked 4627 Write 6 New Done Done false
10669 3629 Acked 4628 Read 9 New Done Done false
10670 3630 Acked 4629 Write 8 New Done Done false
10671 3631 Acked 4630 Write 8 New Done Done false
10672 3632 Acked 4631 Read 8 New Done Done false
10673 3633 Acked 4632 Write 8 New Done Done false
10674 3634 Acked 4633 Read 5 New Done Done false
10675 3635 Acked 4634 Flush 0 New Done Done false
10676 3636 Acked 4635 Read 10 New Done Done false
10677 3637 Acked 4636 Write 10 New Done Done false
10678 3638 Acked 4637 Write 10 New Done Done false
10679 3639 Acked 4638 Read 5 New Done Done false
10680 3640 Acked 4639 Write 5 New Done Done false
10681 3641 Acked 4640 Flush 0 New Done Done false
10682 3642 Acked 4641 Read 6 New Done Done false
10683 3643 Acked 4642 Write 10 New Done Done false
10684 3644 Acked 4643 Flush 0 New Done Done false
10685 3645 Acked 4644 Write 3 New Done Done false
10686 3646 Acked 4645 Flush 0 New Done Done false
10687 3647 Acked 4646 Write 4 New Done Done false
10688 3648 Acked 4647 Write 5 New Done Done false
10689 3649 Acked 4648 Read 2 New Done Done false
10690 3650 Acked 4649 Read 9 New Done Done false
10691 3651 Acked 4650 Read 1 New Done Done false
10692 3652 Acked 4651 Read 3 New Done Done false
10693 3653 Acked 4652 Write 6 New Done Done false
10694 3654 Acked 4653 Read 4 New Done Done false
10695 3655 Acked 4654 Write 9 New Done Done false
10696 3656 Acked 4655 Read 9 New Done Done false
10697 3657 Acked 4656 Read 2 New Done Done false
10698 3658 Acked 4657 Read 6 New Done Done false
10699 3659 Acked 4658 Write 5 New Done Done false
10700 3660 Acked 4659 Write 5 New Done Done false
10701 3661 Acked 4660 Write 4 New Done Done false
10702 3662 Acked 4661 Write 10 New Done Done false
10703 3663 Acked 4662 Write 6 New Done Done false
10704 3664 Acked 4663 Read 9 New Done Done false
10705 3665 Acked 4664 Write 1 New Done Done false
10706 3666 Acked 4665 Write 1 New Done Done false
10707 3667 Acked 4666 Write 7 New Done Done false
10708 3668 Acked 4667 Flush 0 New Done Done false
10709 3669 Acked 4668 Read 4 New Done Done false
10710 3670 Acked 4669 Flush 0 New Done Done false
10711 3671 Acked 4670 Flush 0 New Done Done false
10712 3672 Acked 4671 Write 4 New Done Done false
10713 3673 Acked 4672 Write 2 New Done Done false
10714 3674 Acked 4673 Write 4 New Done Done false
10715 3675 Acked 4674 Read 6 New Done Done false
10716 3676 Acked 4675 Flush 0 New Done Done false
10717 3677 Acked 4676 Flush 0 New Done Done false
10718 3678 Acked 4677 Read 9 New Done Done false
10719 3679 Acked 4678 Write 7 New Done Done false
10720 3680 Acked 4679 Write 6 New Done Done false
10721 3681 Acked 4680 Read 6 New Done Done false
10722 3682 Acked 4681 Read 7 New Done Done false
10723 3683 Acked 4682 Write 10 New Done Done false
10724 3684 Acked 4683 Read 7 New Done Done false
10725 3685 Acked 4684 Write 4 New Done Done false
10726 3686 Acked 4685 Write 2 New Done Done false
10727 3687 Acked 4686 Read 4 New Done Done false
10728 3688 Acked 4687 Write 7 New Done Done false
10729 3689 Acked 4688 Flush 0 New Done Done false
10730 3690 Acked 4689 Write 4 New Done Done false
10731 3691 Acked 4690 Write 10 New Done Done false
10732 3692 Acked 4691 Write 7 New Done Done false
10733 3693 Acked 4692 Read 4 New Done Done false
10734 3694 Acked 4693 Write 4 New Done Done false
10735 3695 Acked 4694 Read 9 New Done Done false
10736 3696 Acked 4695 Write 4 New Done Done false
10737 3697 Acked 4696 Write 10 New Done Done false
10738 3698 Acked 4697 Read 9 New Done Done false
10739 3699 Acked 4698 Write 2 New Done Done false
10740 3700 Acked 4699 Read 6 New Done Done false
10741 3701 Acked 4700 Read 5 New Done Done false
10742 3702 Acked 4701 Write 9 New Done Done false
10743 3703 Acked 4702 Read 1 New Done Done false
10744 3704 Acked 4703 Write 10 New Done Done false
10745 3705 Acked 4704 Write 8 New Done Done false
10746 3706 Acked 4705 Write 10 New Done Done false
10747 3707 Acked 4706 Read 5 New Done Done false
10748 3708 Acked 4707 Flush 0 New Done Done false
10749 3709 Acked 4708 Read 8 New Done Done false
10750 3710 Acked 4709 Write 3 New Done Done false
10751 3711 Acked 4710 Write 6 New Done Done false
10752 3712 Acked 4711 Write 1 New Done Done false
10753 3713 Acked 4712 Write 6 New Done Done false
10754 3714 Acked 4713 Flush 0 New Done Done false
10755 3715 Acked 4714 Write 7 New Done Done false
10756 3716 Acked 4715 Read 9 New Done Done false
10757 3717 Acked 4716 Write 8 New Done Done false
10758 3718 Acked 4717 Flush 0 New Done Done false
10759 3719 Acked 4718 Read 5 New Done Done false
10760 3720 Acked 4719 Write 9 New Done Done false
10761 3721 Acked 4720 Read 9 New Done Done false
10762 3722 Acked 4721 Flush 0 New Done Done false
10763 3723 Acked 4722 Read 6 New Done Done false
10764 3724 Acked 4723 Read 10 New Done Done false
10765 3725 Acked 4724 Read 2 New Done Done false
10766 3726 Acked 4725 Flush 0 New Done Done false
10767 3727 Acked 4726 Write 1 New Done Done false
10768 3728 Acked 4727 Write 4 New Done Done false
10769 3729 Acked 4728 Read 4 New Done Done false
10770 3730 Acked 4729 Write 9 New Done Done false
10771 3731 Acked 4730 Read 7 New Done Done false
10772 3732 Acked 4731 Read 2 New Done Done false
10773 3733 Acked 4732 Write 8 New Done Done false
10774 3734 Acked 4733 Read 5 New Done Done false
10775 3735 Acked 4734 Write 4 New Done Done false
10776 3736 Acked 4735 Flush 0 New Done Done false
10777 3737 Acked 4736 Read 5 New Done Done false
10778 3738 Acked 4737 Read 6 New Done Done false
10779 3739 Acked 4738 Write 2 New Done Done false
10780 3740 Acked 4739 Write 9 New Done Done false
10781 3741 Acked 4740 Write 10 New Done Done false
10782 3742 Acked 4741 Read 9 New Done Done false
10783 3743 Acked 4742 Write 2 New Done Done false
10784 3744 Acked 4743 Write 2 New Done Done false
10785 3745 Acked 4744 Read 7 New Done Done false
10786 3746 Acked 4745 Read 1 New Done Done false
10787 3747 Acked 4746 Read 1 New Done Done false
10788 3748 Acked 4747 Read 4 New Done Done false
10789 3749 Acked 4748 Write 1 New Done Done false
10790 3750 Acked 4749 Flush 0 New Done Done false
10791 3751 Acked 4750 Read 1 New Done Done false
10792 3752 Acked 4751 Read 3 New Done Done false
10793 3753 Acked 4752 Write 3 New Done Done false
10794 3754 Acked 4753 Read 8 New Done Done false
10795 3755 Acked 4754 Flush 0 New Done Done false
10796 3756 Acked 4755 Read 9 New Done Done false
10797 3757 Acked 4756 Read 9 New Done Done false
10798 3758 Acked 4757 Read 7 New Done Done false
10799 3759 Acked 4758 Write 7 New Done Done false
10800 3760 Acked 4759 Read 5 New Done Done false
10801 3761 Acked 4760 Flush 0 New Done Done false
10802 3762 Acked 4761 Write 4 New Done Done false
10803 3763 Acked 4762 Read 10 New Done Done false
10804 3764 Acked 4763 Write 5 New Done Done false
10805 3765 Acked 4764 Write 3 New Done Done false
10806 3766 Acked 4765 Write 6 New Done Done false
10807 3767 Acked 4766 Write 10 New Done Done false
10808 3768 Acked 4767 Read 7 New Done Done false
10809 3769 Acked 4768 Read 9 New Done Done false
10810 3770 Acked 4769 Read 6 New Done Done false
10811 3771 Acked 4770 Write 1 New Done Done false
10812 3772 Acked 4771 Write 2 New Done Done false
10813 3773 Acked 4772 Read 7 New Done Done false
10814 3774 Acked 4773 Read 5 New Done Done false
10815 3775 Acked 4774 Flush 0 New Done Done false
10816 3776 Acked 4775 Write 7 New Done Done false
10817 3777 Acked 4776 Flush 0 New Done Done false
10818 3778 Acked 4777 Read 1 New Done Done false
10819 3779 Acked 4778 Read 4 New Done Done false
10820 3780 Acked 4779 Write 4 New Done Done false
10821 3781 Acked 4780 Read 1 New Done Done false
10822 3782 Acked 4781 Read 4 New Done Done false
10823 3783 Acked 4782 Write 6 New Done Done false
10824 3784 Acked 4783 Read 8 New Done Done false
10825 3785 Acked 4784 Write 6 New Done Done false
10826 3786 Acked 4785 Write 3 New Done Done false
10827 3787 Acked 4786 Read 3 New Done Done false
10828 3788 Acked 4787 Write 1 New Done Done false
10829 3789 Acked 4788 Write 6 New Done Done false
10830 3790 Acked 4789 Flush 0 New Done Done false
10831 3791 Acked 4790 Read 9 New Done Done false
10832 3792 Acked 4791 Read 1 New Done Done false
10833 3793 Acked 4792 Read 7 New Done Done false
10834 3794 Acked 4793 Write 3 New Done Done false
10835 3795 Acked 4794 Write 7 New Done Done false
10836 3796 Acked 4795 Read 6 New Done Done false
10837 3797 Acked 4796 Write 8 New Done Done false
10838 3798 Acked 4797 Write 6 New Done Done false
10839 3799 Acked 4798 Read 4 New Done Done false
10840 3800 Acked 4799 Flush 0 New Done Done false
10841 3801 Acked 4800 Write 3 New Done Done false
10842 3802 Acked 4801 Write 1 New Done Done false
10843 3803 Acked 4802 Write 3 New Done Done false
10844 3804 Acked 4803 Write 2 New Done Done false
10845 3805 Acked 4804 Read 7 New Done Done false
10846 3806 Acked 4805 Write 4 New Done Done false
10847 3807 Acked 4806 Flush 0 New Done Done false
10848 3808 Acked 4807 Read 9 New Done Done false
10849 3809 Acked 4808 Read 8 New Done Done false
10850 3810 Acked 4809 Write 9 New Done Done false
10851 3811 Acked 4810 Write 5 New Done Done false
10852 3812 Acked 4811 Read 6 New Done Done false
10853 3813 Acked 4812 Write 8 New Done Done false
10854 3814 Acked 4813 Write 5 New Done Done false
10855 3815 Acked 4814 Write 8 New Done Done false
10856 3816 Acked 4815 Write 2 New Done Done false
10857 3817 Acked 4816 Write 4 New Done Done false
10858 3818 Acked 4817 Write 9 New Done Done false
10859 3819 Acked 4818 Read 5 New Done Done false
10860 3820 Acked 4819 Read 1 New Done Done false
10861 3821 Acked 4820 Write 10 New Done Done false
10862 3822 Acked 4821 Read 5 New Done Done false
10863 3823 Acked 4822 Read 2 New Done Done false
10864 3824 Acked 4823 Read 5 New Done Done false
10865 3825 Acked 4824 Read 5 New Done Done false
10866 3826 Acked 4825 Write 6 New Done Done false
10867 3827 Acked 4826 Write 4 New Done Done false
10868 3828 Acked 4827 Write 8 New Done Done false
10869 3829 Acked 4828 Read 3 New Done Done false
10870 3830 Acked 4829 Write 2 New Done Done false
10871 3831 Acked 4830 Write 5 New Done Done false
10872 3832 Acked 4831 Read 4 New Done Done false
10873 3833 Acked 4832 Read 4 New Done Done false
10874 3834 Acked 4833 Read 2 New Done Done false
10875 3835 Acked 4834 Flush 0 New Done Done false
10876 3836 Acked 4835 Write 2 New Done Done false
10877 3837 Acked 4836 Flush 0 New Done Done false
10878 3838 Acked 4837 Read 8 New Done Done false
10879 3839 Acked 4838 Write 7 New Done Done false
10880 3840 Acked 4839 Write 8 New Done Done false
10881 3841 Acked 4840 Read 6 New Done Done false
10882 3842 Acked 4841 Write 1 New Done Done false
10883 3843 Acked 4842 Flush 0 New Done Done false
10884 3844 Acked 4843 Write 4 New Done Done false
10885 3845 Acked 4844 Flush 0 New Done Done false
10886 3846 Acked 4845 Read 2 New Done Done false
10887 3847 Acked 4846 Write 2 New Done Done false
10888 3848 Acked 4847 Read 10 New Done Done false
10889 3849 Acked 4848 Read 7 New Done Done false
10890 3850 Acked 4849 Read 7 New Done Done false
10891 3851 Acked 4850 Flush 0 New Done Done false
10892 3852 Acked 4851 Write 6 New Done Done false
10893 3853 Acked 4852 Flush 0 New Done Done false
10894 3854 Acked 4853 Write 4 New Done Done false
10895 3855 Acked 4854 Write 5 New Done Done false
10896 3856 Acked 4855 Read 2 New Done Done false
10897 3857 Acked 4856 Write 10 New Done Done false
10898 3858 Acked 4857 Write 2 New Done Done false
10899 3859 Acked 4858 Read 2 New Done Done false
10900 3860 Acked 4859 Read 9 New Done Done false
10901 3861 Acked 4860 Read 7 New Done Done false
10902 3862 Acked 4861 Write 3 New Done Done false
10903 3863 Acked 4862 Write 2 New Done Done false
10904 3864 Acked 4863 Read 4 New Done Done false
10905 3865 Acked 4864 Read 1 New Done Done false
10906 3866 Acked 4865 Write 2 New Done Done false
10907 3867 Acked 4866 Read 6 New Done Done false
10908 3868 Acked 4867 Read 4 New Done Done false
10909 3869 Acked 4868 Write 7 New Done Done false
10910 3870 Acked 4869 Flush 0 New Done Done false
10911 3871 Acked 4870 Read 2 New Done Done false
10912 3872 Acked 4871 Write 1 New Done Done false
10913 3873 Acked 4872 Read 10 New Done Done false
10914 3874 Acked 4873 Read 5 New Done Done false
10915 3875 Acked 4874 Write 7 New Done Done false
10916 3876 Acked 4875 Write 6 New Done Done false
10917 3877 Acked 4876 Read 6 New Done Done false
10918 3878 Acked 4877 Write 10 New Done Done false
10919 3879 Acked 4878 Write 7 New Done Done false
10920 3880 Acked 4879 Read 1 New Done Done false
10921 3881 Acked 4880 Read 8 New Done Done false
10922 3882 Acked 4881 Read 2 New Done Done false
10923 3883 Acked 4882 Write 1 New Done Done false
10924 3884 Acked 4883 Write 8 New Done Done false
10925 3885 Acked 4884 Write 9 New Done Done false
10926 3886 Acked 4885 Write 1 New Done Done false
10927 3887 Acked 4886 Flush 0 New Done Done false
10928 3888 Acked 4887 Flush 0 New Done Done false
10929 3889 Acked 4888 Read 10 New Done Done false
10930 3890 Acked 4889 Read 2 New Done Done false
10931 3891 Acked 4890 Read 2 New Done Done false
10932 3892 Acked 4891 Write 6 New Done Done false
10933 3893 Acked 4892 Write 7 New Done Done false
10934 3894 Acked 4893 Write 10 New Done Done false
10935 3895 Acked 4894 Write 5 New Done Done false
10936 3896 Acked 4895 Write 7 New Done Done false
10937 3897 Acked 4896 Flush 0 New Done Done false
10938 3898 Acked 4897 Write 5 New Done Done false
10939 3899 Acked 4898 Read 6 New Done Done false
10940 3900 Acked 4899 Read 9 New Done Done false
10941 3901 Acked 4900 Read 5 New Done Done false
10942 3902 Acked 4901 Read 2 New Done Done false
10943 3903 Acked 4902 Write 1 New Done Done false
10944 3904 Acked 4903 Read 6 New Done Done false
10945 3905 Acked 4904 Read 6 New Done Done false
10946 3906 Acked 4905 Read 10 New Done Done false
10947 3907 Acked 4906 Write 1 New Done Done false
10948 3908 Acked 4907 Read 6 New Done Done false
10949 3909 Acked 4908 Flush 0 New Done Done false
10950 3910 Acked 4909 Write 4 New Done Done false
10951 3911 Acked 4910 Write 10 New Done Done false
10952 3912 Acked 4911 Flush 0 New Done Done false
10953 3913 Acked 4912 Flush 0 New Done Done false
10954 3914 Acked 4913 Read 5 New Done Done false
10955 3915 Acked 4914 Read 3 New Done Done false
10956 3916 Acked 4915 Read 10 New Done Done false
10957 3917 Acked 4916 Write 3 New Done Done false
10958 3918 Acked 4917 Write 2 New Done Done false
10959 3919 Acked 4918 Write 8 New Sent Sent false
10960 STATES DS:0 DS:1 DS:2 TOTAL
10961 New 301 0 0 301
10962 Sent 0 1 1 2
10963 Done 0 300 300 600
10964 Skipped 0 0 0 0
10965 Error 0 0 0 0
10966 Last Flush: 4617 4912 4912
10967 Downstairs last five completed: 4617 4616 4615 4614 4613
10968 Upstairs last five completed: 3919 3918 3917 3916 3915
10969 CLIENT: Up:0 ds:301 act:2
109702024-03-02T01:07:47.061ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109712024-03-02T01:07:47.061ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109722024-03-02T01:07:47.061ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109732024-03-02T01:07:47.061ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109742024-03-02T01:07:47.061ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109752024-03-02T01:07:47.061ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109762024-03-02T01:07:47.061ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109772024-03-02T01:07:48.063ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109782024-03-02T01:07:48.063ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109792024-03-02T01:07:48.063ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109802024-03-02T01:07:48.063ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109812024-03-02T01:07:48.063ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109822024-03-02T01:07:48.063ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109832024-03-02T01:07:48.063ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109842024-03-02T01:07:49.064ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109852024-03-02T01:07:49.064ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109862024-03-02T01:07:49.064ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109872024-03-02T01:07:49.064ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109882024-03-02T01:07:49.064ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109892024-03-02T01:07:49.064ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109902024-03-02T01:07:49.064ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109912024-03-02T01:07:50.065ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109922024-03-02T01:07:50.065ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109932024-03-02T01:07:50.065ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109942024-03-02T01:07:50.065ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109952024-03-02T01:07:50.065ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109962024-03-02T01:07:50.065ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
109972024-03-02T01:07:50.065ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
10998 ----------------------------------------------------------------
10999 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11000 States: Offline Active Active
11001 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11002 3619 Acked 4618 Read 10 New Done Done true
11003 3620 Acked 4619 Write 1 New Done Done true
11004 3621 Acked 4620 Write 6 New Done Done true
11005 3622 Acked 4621 Read 4 New Done Done true
11006 3623 Acked 4622 Write 1 New Done Done true
11007 3624 Acked 4623 Write 5 New Done Done true
11008 3625 Acked 4624 Write 8 New Done Done true
11009 3626 Acked 4625 Write 3 New Done Done true
11010 3627 Acked 4626 Read 2 New Done Done true
11011 3628 Acked 4627 Write 6 New Done Done true
11012 3629 Acked 4628 Read 9 New Done Done true
11013 3630 Acked 4629 Write 8 New Done Done true
11014 3631 Acked 4630 Write 8 New Done Done true
11015 3632 Acked 4631 Read 8 New Done Done true
11016 3633 Acked 4632 Write 8 New Done Done true
11017 3634 Acked 4633 Read 5 New Done Done true
11018 3635 Acked 4634 Flush 0 New Done Done true
11019 3636 Acked 4635 Read 10 New Done Done true
11020 3637 Acked 4636 Write 10 New Done Done true
11021 3638 Acked 4637 Write 10 New Done Done true
11022 3639 Acked 4638 Read 5 New Done Done true
11023 3640 Acked 4639 Write 5 New Done Done true
11024 3641 Acked 4640 Flush 0 New Done Done true
11025 3642 Acked 4641 Read 6 New Done Done true
11026 3643 Acked 4642 Write 10 New Done Done true
11027 3644 Acked 4643 Flush 0 New Done Done true
11028 3645 Acked 4644 Write 3 New Done Done true
11029 3646 Acked 4645 Flush 0 New Done Done true
11030 3647 Acked 4646 Write 4 New Done Done true
11031 3648 Acked 4647 Write 5 New Done Done true
11032 3649 Acked 4648 Read 2 New Done Done true
11033 3650 Acked 4649 Read 9 New Done Done true
11034 3651 Acked 4650 Read 1 New Done Done true
11035 3652 Acked 4651 Read 3 New Done Done true
11036 3653 Acked 4652 Write 6 New Done Done true
11037 3654 Acked 4653 Read 4 New Done Done true
11038 3655 Acked 4654 Write 9 New Done Done true
11039 3656 Acked 4655 Read 9 New Done Done true
11040 3657 Acked 4656 Read 2 New Done Done true
11041 3658 Acked 4657 Read 6 New Done Done true
11042 3659 Acked 4658 Write 5 New Done Done true
11043 3660 Acked 4659 Write 5 New Done Done true
11044 3661 Acked 4660 Write 4 New Done Done true
11045 3662 Acked 4661 Write 10 New Done Done true
11046 3663 Acked 4662 Write 6 New Done Done true
11047 3664 Acked 4663 Read 9 New Done Done true
11048 3665 Acked 4664 Write 1 New Done Done true
11049 3666 Acked 4665 Write 1 New Done Done true
11050 3667 Acked 4666 Write 7 New Done Done true
11051 3668 Acked 4667 Flush 0 New Done Done true
11052 3669 Acked 4668 Read 4 New Done Done true
11053 3670 Acked 4669 Flush 0 New Done Done true
11054 3671 Acked 4670 Flush 0 New Done Done true
11055 3672 Acked 4671 Write 4 New Done Done true
11056 3673 Acked 4672 Write 2 New Done Done true
11057 3674 Acked 4673 Write 4 New Done Done true
11058 3675 Acked 4674 Read 6 New Done Done true
11059 3676 Acked 4675 Flush 0 New Done Done true
11060 3677 Acked 4676 Flush 0 New Done Done true
11061 3678 Acked 4677 Read 9 New Done Done true
11062 3679 Acked 4678 Write 7 New Done Done true
11063 3680 Acked 4679 Write 6 New Done Done true
11064 3681 Acked 4680 Read 6 New Done Done true
11065 3682 Acked 4681 Read 7 New Done Done true
11066 3683 Acked 4682 Write 10 New Done Done true
11067 3684 Acked 4683 Read 7 New Done Done true
11068 3685 Acked 4684 Write 4 New Done Done true
11069 3686 Acked 4685 Write 2 New Done Done true
11070 3687 Acked 4686 Read 4 New Done Done true
11071 3688 Acked 4687 Write 7 New Done Done true
11072 3689 Acked 4688 Flush 0 New Done Done true
11073 3690 Acked 4689 Write 4 New Done Done true
11074 3691 Acked 4690 Write 10 New Done Done true
11075 3692 Acked 4691 Write 7 New Done Done true
11076 3693 Acked 4692 Read 4 New Done Done true
11077 3694 Acked 4693 Write 4 New Done Done true
11078 3695 Acked 4694 Read 9 New Done Done true
11079 3696 Acked 4695 Write 4 New Done Done true
11080 3697 Acked 4696 Write 10 New Done Done true
11081 3698 Acked 4697 Read 9 New Done Done true
11082 3699 Acked 4698 Write 2 New Done Done true
11083 3700 Acked 4699 Read 6 New Done Done true
11084 3701 Acked 4700 Read 5 New Done Done true
11085 3702 Acked 4701 Write 9 New Done Done true
11086 3703 Acked 4702 Read 1 New Done Done true
11087 3704 Acked 4703 Write 10 New Done Done true
11088 3705 Acked 4704 Write 8 New Done Done true
11089 3706 Acked 4705 Write 10 New Done Done true
11090 3707 Acked 4706 Read 5 New Done Done true
11091 3708 Acked 4707 Flush 0 New Done Done true
11092 3709 Acked 4708 Read 8 New Done Done true
11093 3710 Acked 4709 Write 3 New Done Done true
11094 3711 Acked 4710 Write 6 New Done Done true
11095 3712 Acked 4711 Write 1 New Done Done true
11096 3713 Acked 4712 Write 6 New Done Done true
11097 3714 Acked 4713 Flush 0 New Done Done true
11098 3715 Acked 4714 Write 7 New Done Done true
11099 3716 Acked 4715 Read 9 New Done Done true
11100 3717 Acked 4716 Write 8 New Done Done true
11101 3718 Acked 4717 Flush 0 New Done Done true
11102 3719 Acked 4718 Read 5 New Done Done true
11103 3720 Acked 4719 Write 9 New Done Done true
11104 3721 Acked 4720 Read 9 New Done Done true
11105 3722 Acked 4721 Flush 0 New Done Done true
11106 3723 Acked 4722 Read 6 New Done Done true
11107 3724 Acked 4723 Read 10 New Done Done true
11108 3725 Acked 4724 Read 2 New Done Done true
11109 3726 Acked 4725 Flush 0 New Done Done true
11110 3727 Acked 4726 Write 1 New Done Done true
11111 3728 Acked 4727 Write 4 New Done Done true
11112 3729 Acked 4728 Read 4 New Done Done true
11113 3730 Acked 4729 Write 9 New Done Done true
11114 3731 Acked 4730 Read 7 New Done Done true
11115 3732 Acked 4731 Read 2 New Done Done true
11116 3733 Acked 4732 Write 8 New Done Done true
11117 3734 Acked 4733 Read 5 New Done Done true
11118 3735 Acked 4734 Write 4 New Done Done true
11119 3736 Acked 4735 Flush 0 New Done Done true
11120 3737 Acked 4736 Read 5 New Done Done true
11121 3738 Acked 4737 Read 6 New Done Done true
11122 3739 Acked 4738 Write 2 New Done Done true
11123 3740 Acked 4739 Write 9 New Done Done true
11124 3741 Acked 4740 Write 10 New Done Done true
11125 3742 Acked 4741 Read 9 New Done Done true
11126 3743 Acked 4742 Write 2 New Done Done true
11127 3744 Acked 4743 Write 2 New Done Done true
11128 3745 Acked 4744 Read 7 New Done Done true
11129 3746 Acked 4745 Read 1 New Done Done true
11130 3747 Acked 4746 Read 1 New Done Done true
11131 3748 Acked 4747 Read 4 New Done Done true
11132 3749 Acked 4748 Write 1 New Done Done true
11133 3750 Acked 4749 Flush 0 New Done Done true
11134 3751 Acked 4750 Read 1 New Done Done true
11135 3752 Acked 4751 Read 3 New Done Done true
11136 3753 Acked 4752 Write 3 New Done Done true
11137 3754 Acked 4753 Read 8 New Done Done true
11138 3755 Acked 4754 Flush 0 New Done Done true
11139 3756 Acked 4755 Read 9 New Done Done true
11140 3757 Acked 4756 Read 9 New Done Done true
11141 3758 Acked 4757 Read 7 New Done Done true
11142 3759 Acked 4758 Write 7 New Done Done true
11143 3760 Acked 4759 Read 5 New Done Done true
11144 3761 Acked 4760 Flush 0 New Done Done true
11145 3762 Acked 4761 Write 4 New Done Done true
11146 3763 Acked 4762 Read 10 New Done Done true
11147 3764 Acked 4763 Write 5 New Done Done true
11148 3765 Acked 4764 Write 3 New Done Done true
11149 3766 Acked 4765 Write 6 New Done Done true
11150 3767 Acked 4766 Write 10 New Done Done true
11151 3768 Acked 4767 Read 7 New Done Done true
11152 3769 Acked 4768 Read 9 New Done Done true
11153 3770 Acked 4769 Read 6 New Done Done true
11154 3771 Acked 4770 Write 1 New Done Done true
11155 3772 Acked 4771 Write 2 New Done Done true
11156 3773 Acked 4772 Read 7 New Done Done true
11157 3774 Acked 4773 Read 5 New Done Done true
11158 3775 Acked 4774 Flush 0 New Done Done true
11159 3776 Acked 4775 Write 7 New Done Done true
11160 3777 Acked 4776 Flush 0 New Done Done true
11161 3778 Acked 4777 Read 1 New Done Done true
11162 3779 Acked 4778 Read 4 New Done Done true
11163 3780 Acked 4779 Write 4 New Done Done true
11164 3781 Acked 4780 Read 1 New Done Done true
11165 3782 Acked 4781 Read 4 New Done Done true
11166 3783 Acked 4782 Write 6 New Done Done true
11167 3784 Acked 4783 Read 8 New Done Done true
11168 3785 Acked 4784 Write 6 New Done Done true
11169 3786 Acked 4785 Write 3 New Done Done true
11170 3787 Acked 4786 Read 3 New Done Done true
11171 3788 Acked 4787 Write 1 New Done Done true
11172 3789 Acked 4788 Write 6 New Done Done true
11173 3790 Acked 4789 Flush 0 New Done Done true
11174 3791 Acked 4790 Read 9 New Done Done true
11175 3792 Acked 4791 Read 1 New Done Done true
11176 3793 Acked 4792 Read 7 New Done Done true
11177 3794 Acked 4793 Write 3 New Done Done true
11178 3795 Acked 4794 Write 7 New Done Done true
11179 3796 Acked 4795 Read 6 New Done Done true
11180 3797 Acked 4796 Write 8 New Done Done true
11181 3798 Acked 4797 Write 6 New Done Done true
11182 3799 Acked 4798 Read 4 New Done Done true
11183 3800 Acked 4799 Flush 0 New Done Done true
11184 3801 Acked 4800 Write 3 New Done Done true
11185 3802 Acked 4801 Write 1 New Done Done true
11186 3803 Acked 4802 Write 3 New Done Done true
11187 3804 Acked 4803 Write 2 New Done Done true
11188 3805 Acked 4804 Read 7 New Done Done true
11189 3806 Acked 4805 Write 4 New Done Done true
11190 3807 Acked 4806 Flush 0 New Done Done true
11191 3808 Acked 4807 Read 9 New Done Done true
11192 3809 Acked 4808 Read 8 New Done Done true
11193 3810 Acked 4809 Write 9 New Done Done true
11194 3811 Acked 4810 Write 5 New Done Done true
11195 3812 Acked 4811 Read 6 New Done Done true
11196 3813 Acked 4812 Write 8 New Done Done true
11197 3814 Acked 4813 Write 5 New Done Done true
11198 3815 Acked 4814 Write 8 New Done Done true
11199 3816 Acked 4815 Write 2 New Done Done true
11200 3817 Acked 4816 Write 4 New Done Done true
11201 3818 Acked 4817 Write 9 New Done Done true
11202 3819 Acked 4818 Read 5 New Done Done true
11203 3820 Acked 4819 Read 1 New Done Done true
11204 3821 Acked 4820 Write 10 New Done Done true
11205 3822 Acked 4821 Read 5 New Done Done true
11206 3823 Acked 4822 Read 2 New Done Done true
11207 3824 Acked 4823 Read 5 New Done Done true
11208 3825 Acked 4824 Read 5 New Done Done true
11209 3826 Acked 4825 Write 6 New Done Done true
11210 3827 Acked 4826 Write 4 New Done Done true
11211 3828 Acked 4827 Write 8 New Done Done true
11212 3829 Acked 4828 Read 3 New Done Done true
11213 3830 Acked 4829 Write 2 New Done Done true
11214 3831 Acked 4830 Write 5 New Done Done true
11215 3832 Acked 4831 Read 4 New Done Done true
11216 3833 Acked 4832 Read 4 New Done Done true
11217 3834 Acked 4833 Read 2 New Done Done true
11218 3835 Acked 4834 Flush 0 New Done Done true
11219 3836 Acked 4835 Write 2 New Done Done true
11220 3837 Acked 4836 Flush 0 New Done Done true
11221 3838 Acked 4837 Read 8 New Done Done true
11222 3839 Acked 4838 Write 7 New Done Done true
11223 3840 Acked 4839 Write 8 New Done Done true
11224 3841 Acked 4840 Read 6 New Done Done true
11225 3842 Acked 4841 Write 1 New Done Done true
11226 3843 Acked 4842 Flush 0 New Done Done true
11227 3844 Acked 4843 Write 4 New Done Done true
11228 3845 Acked 4844 Flush 0 New Done Done true
11229 3846 Acked 4845 Read 2 New Done Done true
11230 3847 Acked 4846 Write 2 New Done Done true
11231 3848 Acked 4847 Read 10 New Done Done true
11232 3849 Acked 4848 Read 7 New Done Done true
11233 3850 Acked 4849 Read 7 New Done Done true
11234 3851 Acked 4850 Flush 0 New Done Done true
11235 3852 Acked 4851 Write 6 New Done Done true
11236 3853 Acked 4852 Flush 0 New Done Done true
11237 3854 Acked 4853 Write 4 New Done Done true
11238 3855 Acked 4854 Write 5 New Done Done true
11239 3856 Acked 4855 Read 2 New Done Done true
11240 3857 Acked 4856 Write 10 New Done Done true
11241 3858 Acked 4857 Write 2 New Done Done true
11242 3859 Acked 4858 Read 2 New Done Done true
11243 3860 Acked 4859 Read 9 New Done Done true
11244 3861 Acked 4860 Read 7 New Done Done true
11245 3862 Acked 4861 Write 3 New Done Done true
11246 3863 Acked 4862 Write 2 New Done Done true
11247 3864 Acked 4863 Read 4 New Done Done true
11248 3865 Acked 4864 Read 1 New Done Done true
11249 3866 Acked 4865 Write 2 New Done Done true
11250 3867 Acked 4866 Read 6 New Done Done true
11251 3868 Acked 4867 Read 4 New Done Done true
11252 3869 Acked 4868 Write 7 New Done Done true
11253 3870 Acked 4869 Flush 0 New Done Done true
11254 3871 Acked 4870 Read 2 New Done Done true
11255 3872 Acked 4871 Write 1 New Done Done true
11256 3873 Acked 4872 Read 10 New Done Done true
11257 3874 Acked 4873 Read 5 New Done Done true
11258 3875 Acked 4874 Write 7 New Done Done true
11259 3876 Acked 4875 Write 6 New Done Done true
11260 3877 Acked 4876 Read 6 New Done Done true
11261 3878 Acked 4877 Write 10 New Done Done true
11262 3879 Acked 4878 Write 7 New Done Done true
11263 3880 Acked 4879 Read 1 New Done Done true
11264 3881 Acked 4880 Read 8 New Done Done true
11265 3882 Acked 4881 Read 2 New Done Done true
11266 3883 Acked 4882 Write 1 New Done Done true
11267 3884 Acked 4883 Write 8 New Done Done true
11268 3885 Acked 4884 Write 9 New Done Done true
11269 3886 Acked 4885 Write 1 New Done Done true
11270 3887 Acked 4886 Flush 0 New Done Done true
11271 3888 Acked 4887 Flush 0 New Done Done true
11272 3889 Acked 4888 Read 10 New Done Done true
11273 3890 Acked 4889 Read 2 New Done Done true
11274 3891 Acked 4890 Read 2 New Done Done true
11275 3892 Acked 4891 Write 6 New Done Done true
11276 3893 Acked 4892 Write 7 New Done Done true
11277 3894 Acked 4893 Write 10 New Done Done true
11278 3895 Acked 4894 Write 5 New Done Done true
11279 3896 Acked 4895 Write 7 New Done Done true
11280 3897 Acked 4896 Flush 0 New Done Done true
11281 3898 Acked 4897 Write 5 New Done Done true
11282 3899 Acked 4898 Read 6 New Done Done true
11283 3900 Acked 4899 Read 9 New Done Done true
11284 3901 Acked 4900 Read 5 New Done Done true
11285 3902 Acked 4901 Read 2 New Done Done true
11286 3903 Acked 4902 Write 1 New Done Done true
11287 3904 Acked 4903 Read 6 New Done Done true
11288 3905 Acked 4904 Read 6 New Done Done true
11289 3906 Acked 4905 Read 10 New Done Done true
11290 3907 Acked 4906 Write 1 New Done Done true
11291 3908 Acked 4907 Read 6 New Done Done true
11292 3909 Acked 4908 Flush 0 New Done Done true
11293 3910 Acked 4909 Write 4 New Done Done true
11294 3911 Acked 4910 Write 10 New Done Done true
11295 3912 Acked 4911 Flush 0 New Done Done true
11296 3913 Acked 4912 Flush 0 New Done Done true
11297 3914 Acked 4913 Read 5 New Done Done true
11298 3915 Acked 4914 Read 3 New Done Done true
11299 3916 Acked 4915 Read 10 New Done Done true
11300 3917 Acked 4916 Write 3 New Done Done true
11301 3918 Acked 4917 Write 2 New Done Done true
11302 3919 Acked 4918 Write 8 New Done Done true
11303 3920 Acked 4919 Flush 0 New Done Done true
11304 STATES DS:0 DS:1 DS:2 TOTAL
11305 New 302 0 0 302
11306 Sent 0 0 0 0
11307 Done 0 302 302 604
11308 Skipped 0 0 0 0
11309 Error 0 0 0 0
11310 Last Flush: 4617 4919 4919
11311 Downstairs last five completed: 4617 4616 4615 4614 4613
11312 Upstairs last five completed: 3920 3919 3918 3917 3916
11313 CLIENT: Up:0 ds:302 act:2
113142024-03-02T01:07:51.067ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113152024-03-02T01:07:51.067ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113162024-03-02T01:07:51.067ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113172024-03-02T01:07:51.067ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113182024-03-02T01:07:51.067ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113192024-03-02T01:07:51.067ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113202024-03-02T01:07:51.067ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113212024-03-02T01:07:52.068ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113222024-03-02T01:07:52.068ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113232024-03-02T01:07:52.068ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113242024-03-02T01:07:52.068ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113252024-03-02T01:07:52.068ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113262024-03-02T01:07:52.068ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113272024-03-02T01:07:52.069ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113282024-03-02T01:07:53.070ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113292024-03-02T01:07:53.070ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113302024-03-02T01:07:53.070ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113312024-03-02T01:07:53.070ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113322024-03-02T01:07:53.070ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113332024-03-02T01:07:53.070ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113342024-03-02T01:07:53.070ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11335 [8810] got start action
11336 [8810] I am going to restart
11337 Starting downstairs at port 8810
11338 Make output file at "/tmp/dsc/downstairs-8810.txt"
11339 [8810][0] reports Starting
11340 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1175)
11341 [8810][0] reports Running
113422024-03-02T01:07:54.071ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113432024-03-02T01:07:54.071ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113442024-03-02T01:07:54.071ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113452024-03-02T01:07:54.071ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113462024-03-02T01:07:54.071ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113472024-03-02T01:07:54.071ZINFOcrucible: [0] client re-new 302 jobs since flush 4617 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
113482024-03-02T01:07:54.072ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11349 ----------------------------------------------------------------
11350 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11351 States: Offline Active Active
11352 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11353 3619 Acked 4618 Read 10 New Done Done true
11354 3620 Acked 4619 Write 1 New Done Done true
11355 3621 Acked 4620 Write 6 New Done Done true
11356 3622 Acked 4621 Read 4 New Done Done true
11357 3623 Acked 4622 Write 1 New Done Done true
11358 3624 Acked 4623 Write 5 New Done Done true
11359 3625 Acked 4624 Write 8 New Done Done true
11360 3626 Acked 4625 Write 3 New Done Done true
11361 3627 Acked 4626 Read 2 New Done Done true
11362 3628 Acked 4627 Write 6 New Done Done true
11363 3629 Acked 4628 Read 9 New Done Done true
11364 3630 Acked 4629 Write 8 New Done Done true
11365 3631 Acked 4630 Write 8 New Done Done true
11366 3632 Acked 4631 Read 8 New Done Done true
11367 3633 Acked 4632 Write 8 New Done Done true
11368 3634 Acked 4633 Read 5 New Done Done true
11369 3635 Acked 4634 Flush 0 New Done Done true
11370 3636 Acked 4635 Read 10 New Done Done true
11371 3637 Acked 4636 Write 10 New Done Done true
11372 3638 Acked 4637 Write 10 New Done Done true
11373 3639 Acked 4638 Read 5 New Done Done true
11374 3640 Acked 4639 Write 5 New Done Done true
11375 3641 Acked 4640 Flush 0 New Done Done true
11376 3642 Acked 4641 Read 6 New Done Done true
11377 3643 Acked 4642 Write 10 New Done Done true
11378 3644 Acked 4643 Flush 0 New Done Done true
11379 3645 Acked 4644 Write 3 New Done Done true
11380 3646 Acked 4645 Flush 0 New Done Done true
11381 3647 Acked 4646 Write 4 New Done Done true
11382 3648 Acked 4647 Write 5 New Done Done true
11383 3649 Acked 4648 Read 2 New Done Done true
11384 3650 Acked 4649 Read 9 New Done Done true
11385 3651 Acked 4650 Read 1 New Done Done true
11386 3652 Acked 4651 Read 3 New Done Done true
11387 3653 Acked 4652 Write 6 New Done Done true
11388 3654 Acked 4653 Read 4 New Done Done true
11389 3655 Acked 4654 Write 9 New Done Done true
11390 3656 Acked 4655 Read 9 New Done Done true
11391 3657 Acked 4656 Read 2 New Done Done true
11392 3658 Acked 4657 Read 6 New Done Done true
11393 3659 Acked 4658 Write 5 New Done Done true
11394 3660 Acked 4659 Write 5 New Done Done true
11395 3661 Acked 4660 Write 4 New Done Done true
11396 3662 Acked 4661 Write 10 New Done Done true
11397 3663 Acked 4662 Write 6 New Done Done true
11398 3664 Acked 4663 Read 9 New Done Done true
11399 3665 Acked 4664 Write 1 New Done Done true
11400 3666 Acked 4665 Write 1 New Done Done true
11401 3667 Acked 4666 Write 7 New Done Done true
11402 3668 Acked 4667 Flush 0 New Done Done true
11403 3669 Acked 4668 Read 4 New Done Done true
11404 3670 Acked 4669 Flush 0 New Done Done true
11405 3671 Acked 4670 Flush 0 New Done Done true
11406 3672 Acked 4671 Write 4 New Done Done true
11407 3673 Acked 4672 Write 2 New Done Done true
11408 3674 Acked 4673 Write 4 New Done Done true
11409 3675 Acked 4674 Read 6 New Done Done true
11410 3676 Acked 4675 Flush 0 New Done Done true
11411 3677 Acked 4676 Flush 0 New Done Done true
11412 3678 Acked 4677 Read 9 New Done Done true
11413 3679 Acked 4678 Write 7 New Done Done true
11414 3680 Acked 4679 Write 6 New Done Done true
11415 3681 Acked 4680 Read 6 New Done Done true
11416 3682 Acked 4681 Read 7 New Done Done true
11417 3683 Acked 4682 Write 10 New Done Done true
11418 3684 Acked 4683 Read 7 New Done Done true
11419 3685 Acked 4684 Write 4 New Done Done true
11420 3686 Acked 4685 Write 2 New Done Done true
11421 3687 Acked 4686 Read 4 New Done Done true
11422 3688 Acked 4687 Write 7 New Done Done true
11423 3689 Acked 4688 Flush 0 New Done Done true
11424 3690 Acked 4689 Write 4 New Done Done true
11425 3691 Acked 4690 Write 10 New Done Done true
11426 3692 Acked 4691 Write 7 New Done Done true
11427 3693 Acked 4692 Read 4 New Done Done true
11428 3694 Acked 4693 Write 4 New Done Done true
11429 3695 Acked 4694 Read 9 New Done Done true
11430 3696 Acked 4695 Write 4 New Done Done true
11431 3697 Acked 4696 Write 10 New Done Done true
11432 3698 Acked 4697 Read 9 New Done Done true
11433 3699 Acked 4698 Write 2 New Done Done true
11434 3700 Acked 4699 Read 6 New Done Done true
11435 3701 Acked 4700 Read 5 New Done Done true
11436 3702 Acked 4701 Write 9 New Done Done true
11437 3703 Acked 4702 Read 1 New Done Done true
11438 3704 Acked 4703 Write 10 New Done Done true
11439 3705 Acked 4704 Write 8 New Done Done true
11440 3706 Acked 4705 Write 10 New Done Done true
11441 3707 Acked 4706 Read 5 New Done Done true
11442 3708 Acked 4707 Flush 0 New Done Done true
11443 3709 Acked 4708 Read 8 New Done Done true
11444 3710 Acked 4709 Write 3 New Done Done true
11445 3711 Acked 4710 Write 6 New Done Done true
11446 3712 Acked 4711 Write 1 New Done Done true
11447 3713 Acked 4712 Write 6 New Done Done true
11448 3714 Acked 4713 Flush 0 New Done Done true
11449 3715 Acked 4714 Write 7 New Done Done true
11450 3716 Acked 4715 Read 9 New Done Done true
11451 3717 Acked 4716 Write 8 New Done Done true
11452 3718 Acked 4717 Flush 0 New Done Done true
11453 3719 Acked 4718 Read 5 New Done Done true
11454 3720 Acked 4719 Write 9 New Done Done true
11455 3721 Acked 4720 Read 9 New Done Done true
11456 3722 Acked 4721 Flush 0 New Done Done true
11457 3723 Acked 4722 Read 6 New Done Done true
11458 3724 Acked 4723 Read 10 New Done Done true
11459 3725 Acked 4724 Read 2 New Done Done true
11460 3726 Acked 4725 Flush 0 New Done Done true
11461 3727 Acked 4726 Write 1 New Done Done true
11462 3728 Acked 4727 Write 4 New Done Done true
11463 3729 Acked 4728 Read 4 New Done Done true
11464 3730 Acked 4729 Write 9 New Done Done true
11465 3731 Acked 4730 Read 7 New Done Done true
11466 3732 Acked 4731 Read 2 New Done Done true
11467 3733 Acked 4732 Write 8 New Done Done true
11468 3734 Acked 4733 Read 5 New Done Done true
11469 3735 Acked 4734 Write 4 New Done Done true
11470 3736 Acked 4735 Flush 0 New Done Done true
11471 3737 Acked 4736 Read 5 New Done Done true
11472 3738 Acked 4737 Read 6 New Done Done true
11473 3739 Acked 4738 Write 2 New Done Done true
11474 3740 Acked 4739 Write 9 New Done Done true
11475 3741 Acked 4740 Write 10 New Done Done true
11476 3742 Acked 4741 Read 9 New Done Done true
11477 3743 Acked 4742 Write 2 New Done Done true
11478 3744 Acked 4743 Write 2 New Done Done true
11479 3745 Acked 4744 Read 7 New Done Done true
11480 3746 Acked 4745 Read 1 New Done Done true
11481 3747 Acked 4746 Read 1 New Done Done true
11482 3748 Acked 4747 Read 4 New Done Done true
11483 3749 Acked 4748 Write 1 New Done Done true
11484 3750 Acked 4749 Flush 0 New Done Done true
11485 3751 Acked 4750 Read 1 New Done Done true
11486 3752 Acked 4751 Read 3 New Done Done true
11487 3753 Acked 4752 Write 3 New Done Done true
11488 3754 Acked 4753 Read 8 New Done Done true
11489 3755 Acked 4754 Flush 0 New Done Done true
11490 3756 Acked 4755 Read 9 New Done Done true
11491 3757 Acked 4756 Read 9 New Done Done true
11492 3758 Acked 4757 Read 7 New Done Done true
11493 3759 Acked 4758 Write 7 New Done Done true
11494 3760 Acked 4759 Read 5 New Done Done true
11495 3761 Acked 4760 Flush 0 New Done Done true
11496 3762 Acked 4761 Write 4 New Done Done true
11497 3763 Acked 4762 Read 10 New Done Done true
11498 3764 Acked 4763 Write 5 New Done Done true
11499 3765 Acked 4764 Write 3 New Done Done true
11500 3766 Acked 4765 Write 6 New Done Done true
11501 3767 Acked 4766 Write 10 New Done Done true
11502 3768 Acked 4767 Read 7 New Done Done true
11503 3769 Acked 4768 Read 9 New Done Done true
11504 3770 Acked 4769 Read 6 New Done Done true
11505 3771 Acked 4770 Write 1 New Done Done true
11506 3772 Acked 4771 Write 2 New Done Done true
11507 3773 Acked 4772 Read 7 New Done Done true
11508 3774 Acked 4773 Read 5 New Done Done true
11509 3775 Acked 4774 Flush 0 New Done Done true
11510 3776 Acked 4775 Write 7 New Done Done true
11511 3777 Acked 4776 Flush 0 New Done Done true
11512 3778 Acked 4777 Read 1 New Done Done true
11513 3779 Acked 4778 Read 4 New Done Done true
11514 3780 Acked 4779 Write 4 New Done Done true
11515 3781 Acked 4780 Read 1 New Done Done true
11516 3782 Acked 4781 Read 4 New Done Done true
11517 3783 Acked 4782 Write 6 New Done Done true
11518 3784 Acked 4783 Read 8 New Done Done true
11519 3785 Acked 4784 Write 6 New Done Done true
11520 3786 Acked 4785 Write 3 New Done Done true
11521 3787 Acked 4786 Read 3 New Done Done true
11522 3788 Acked 4787 Write 1 New Done Done true
11523 3789 Acked 4788 Write 6 New Done Done true
11524 3790 Acked 4789 Flush 0 New Done Done true
11525 3791 Acked 4790 Read 9 New Done Done true
11526 3792 Acked 4791 Read 1 New Done Done true
11527 3793 Acked 4792 Read 7 New Done Done true
11528 3794 Acked 4793 Write 3 New Done Done true
11529 3795 Acked 4794 Write 7 New Done Done true
11530 3796 Acked 4795 Read 6 New Done Done true
11531 3797 Acked 4796 Write 8 New Done Done true
11532 3798 Acked 4797 Write 6 New Done Done true
11533 3799 Acked 4798 Read 4 New Done Done true
11534 3800 Acked 4799 Flush 0 New Done Done true
11535 3801 Acked 4800 Write 3 New Done Done true
11536 3802 Acked 4801 Write 1 New Done Done true
11537 3803 Acked 4802 Write 3 New Done Done true
11538 3804 Acked 4803 Write 2 New Done Done true
11539 3805 Acked 4804 Read 7 New Done Done true
11540 3806 Acked 4805 Write 4 New Done Done true
11541 3807 Acked 4806 Flush 0 New Done Done true
11542 3808 Acked 4807 Read 9 New Done Done true
11543 3809 Acked 4808 Read 8 New Done Done true
11544 3810 Acked 4809 Write 9 New Done Done true
11545 3811 Acked 4810 Write 5 New Done Done true
11546 3812 Acked 4811 Read 6 New Done Done true
11547 3813 Acked 4812 Write 8 New Done Done true
11548 3814 Acked 4813 Write 5 New Done Done true
11549 3815 Acked 4814 Write 8 New Done Done true
11550 3816 Acked 4815 Write 2 New Done Done true
11551 3817 Acked 4816 Write 4 New Done Done true
11552 3818 Acked 4817 Write 9 New Done Done true
11553 3819 Acked 4818 Read 5 New Done Done true
11554 3820 Acked 4819 Read 1 New Done Done true
11555 3821 Acked 4820 Write 10 New Done Done true
11556 3822 Acked 4821 Read 5 New Done Done true
11557 3823 Acked 4822 Read 2 New Done Done true
11558 3824 Acked 4823 Read 5 New Done Done true
11559 3825 Acked 4824 Read 5 New Done Done true
11560 3826 Acked 4825 Write 6 New Done Done true
11561 3827 Acked 4826 Write 4 New Done Done true
11562 3828 Acked 4827 Write 8 New Done Done true
11563 3829 Acked 4828 Read 3 New Done Done true
11564 3830 Acked 4829 Write 2 New Done Done true
11565 3831 Acked 4830 Write 5 New Done Done true
11566 3832 Acked 4831 Read 4 New Done Done true
11567 3833 Acked 4832 Read 4 New Done Done true
11568 3834 Acked 4833 Read 2 New Done Done true
11569 3835 Acked 4834 Flush 0 New Done Done true
11570 3836 Acked 4835 Write 2 New Done Done true
11571 3837 Acked 4836 Flush 0 New Done Done true
11572 3838 Acked 4837 Read 8 New Done Done true
11573 3839 Acked 4838 Write 7 New Done Done true
11574 3840 Acked 4839 Write 8 New Done Done true
11575 3841 Acked 4840 Read 6 New Done Done true
11576 3842 Acked 4841 Write 1 New Done Done true
11577 3843 Acked 4842 Flush 0 New Done Done true
11578 3844 Acked 4843 Write 4 New Done Done true
11579 3845 Acked 4844 Flush 0 New Done Done true
11580 3846 Acked 4845 Read 2 New Done Done true
11581 3847 Acked 4846 Write 2 New Done Done true
11582 3848 Acked 4847 Read 10 New Done Done true
11583 3849 Acked 4848 Read 7 New Done Done true
11584 3850 Acked 4849 Read 7 New Done Done true
11585 3851 Acked 4850 Flush 0 New Done Done true
11586 3852 Acked 4851 Write 6 New Done Done true
11587 3853 Acked 4852 Flush 0 New Done Done true
11588 3854 Acked 4853 Write 4 New Done Done true
11589 3855 Acked 4854 Write 5 New Done Done true
11590 3856 Acked 4855 Read 2 New Done Done true
11591 3857 Acked 4856 Write 10 New Done Done true
11592 3858 Acked 4857 Write 2 New Done Done true
11593 3859 Acked 4858 Read 2 New Done Done true
11594 3860 Acked 4859 Read 9 New Done Done true
11595 3861 Acked 4860 Read 7 New Done Done true
11596 3862 Acked 4861 Write 3 New Done Done true
11597 3863 Acked 4862 Write 2 New Done Done true
11598 3864 Acked 4863 Read 4 New Done Done true
11599 3865 Acked 4864 Read 1 New Done Done true
11600 3866 Acked 4865 Write 2 New Done Done true
11601 3867 Acked 4866 Read 6 New Done Done true
11602 3868 Acked 4867 Read 4 New Done Done true
11603 3869 Acked 4868 Write 7 New Done Done true
11604 3870 Acked 4869 Flush 0 New Done Done true
11605 3871 Acked 4870 Read 2 New Done Done true
11606 3872 Acked 4871 Write 1 New Done Done true
11607 3873 Acked 4872 Read 10 New Done Done true
11608 3874 Acked 4873 Read 5 New Done Done true
11609 3875 Acked 4874 Write 7 New Done Done true
11610 3876 Acked 4875 Write 6 New Done Done true
11611 3877 Acked 4876 Read 6 New Done Done true
11612 3878 Acked 4877 Write 10 New Done Done true
11613 3879 Acked 4878 Write 7 New Done Done true
11614 3880 Acked 4879 Read 1 New Done Done true
11615 3881 Acked 4880 Read 8 New Done Done true
11616 3882 Acked 4881 Read 2 New Done Done true
11617 3883 Acked 4882 Write 1 New Done Done true
11618 3884 Acked 4883 Write 8 New Done Done true
11619 3885 Acked 4884 Write 9 New Done Done true
11620 3886 Acked 4885 Write 1 New Done Done true
11621 3887 Acked 4886 Flush 0 New Done Done true
11622 3888 Acked 4887 Flush 0 New Done Done true
11623 3889 Acked 4888 Read 10 New Done Done true
11624 3890 Acked 4889 Read 2 New Done Done true
11625 3891 Acked 4890 Read 2 New Done Done true
11626 3892 Acked 4891 Write 6 New Done Done true
11627 3893 Acked 4892 Write 7 New Done Done true
11628 3894 Acked 4893 Write 10 New Done Done true
11629 3895 Acked 4894 Write 5 New Done Done true
11630 3896 Acked 4895 Write 7 New Done Done true
11631 3897 Acked 4896 Flush 0 New Done Done true
11632 3898 Acked 4897 Write 5 New Done Done true
11633 3899 Acked 4898 Read 6 New Done Done true
11634 3900 Acked 4899 Read 9 New Done Done true
11635 3901 Acked 4900 Read 5 New Done Done true
11636 3902 Acked 4901 Read 2 New Done Done true
11637 3903 Acked 4902 Write 1 New Done Done true
11638 3904 Acked 4903 Read 6 New Done Done true
11639 3905 Acked 4904 Read 6 New Done Done true
11640 3906 Acked 4905 Read 10 New Done Done true
11641 3907 Acked 4906 Write 1 New Done Done true
11642 3908 Acked 4907 Read 6 New Done Done true
11643 3909 Acked 4908 Flush 0 New Done Done true
11644 3910 Acked 4909 Write 4 New Done Done true
11645 3911 Acked 4910 Write 10 New Done Done true
11646 3912 Acked 4911 Flush 0 New Done Done true
11647 3913 Acked 4912 Flush 0 New Done Done true
11648 3914 Acked 4913 Read 5 New Done Done true
11649 3915 Acked 4914 Read 3 New Done Done true
11650 3916 Acked 4915 Read 10 New Done Done true
11651 3917 Acked 4916 Write 3 New Done Done true
11652 3918 Acked 4917 Write 2 New Done Done true
11653 3919 Acked 4918 Write 8 New Done Done true
11654 3920 Acked 4919 Flush 0 New Done Done true
11655 STATES DS:0 DS:1 DS:2 TOTAL
11656 New 302 0 0 302
11657 Sent 0 0 0 0
11658 Done 0 302 302 604
11659 Skipped 0 0 0 0
11660 Error 0 0 0 0
11661 Last Flush: 4617 4919 4919
11662 Downstairs last five completed: 4617 4616 4615 4614 4613
11663 Upstairs last five completed: 3920 3919 3918 3917 3916
11664 CLIENT: Up:0 ds:302 act:2
116652024-03-02T01:07:55.073ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116662024-03-02T01:07:55.073ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116672024-03-02T01:07:55.074ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116682024-03-02T01:07:55.074ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116692024-03-02T01:07:55.074ZINFOcrucible: send last flush ID to this DS: 4617 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116702024-03-02T01:07:55.075ZINFOcrucible: Replied this last flush ID: 4617 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116712024-03-02T01:07:55.075ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116722024-03-02T01:07:55.075ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116732024-03-02T01:07:55.075ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116742024-03-02T01:07:55.075ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11675 ----------------------------------------------------------------
11676 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11677 Downstairs last five completed: 4919 4918 4917 4916 4915
11678 Upstairs last five completed: 3920 3919 3918 3917 3916
11679 CLIENT: Up:0 ds:0 act:3
11680 Replay: All jobs finished, all DS active.
11681 Main task has work to do, go find it
11682 got dsc Stop(2)
11683 stop 2
11684 Mar 02 01:07:58.404 INFO request completed, latency_us: 377, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 177e9094-632a-44fe-a9e4-90c2f580259b, remote_addr: 127.0.0[8830] Got stop action so:false kr:false
11685 .1:40735, local_addr: 127.0.0.1:9998
116862024-03-02T01:07:58.404ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116872024-03-02T01:07:58.404ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11688 Mar 02 01:07:58.404 INFO request completed{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:07:58.404943555Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
11689 , latency_us: 147, response_code: 200, uri: /state/cid/2, method: GET, {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:58.404963244Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
11690 req_id: bebda230-e996-4da8-89d5-4d5fac5869d3, remote_addr: 127.0.0{"msg":"[2] client re-new 0 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:07:58.404985293Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
11691 .1:40735, local_addr: 127.0.0.1:9998
116922024-03-02T01:07:58.405ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11693 [8830] Exited with: ExitStatus(unix_wait_status(9))
11694 [8830][2] reports Exit
116952024-03-02T01:07:59.406ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116962024-03-02T01:07:59.406ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116972024-03-02T01:07:59.406ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116982024-03-02T01:07:59.406ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
116992024-03-02T01:07:59.406ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117002024-03-02T01:07:59.406ZINFOcrucible: [2] client re-new 0 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117012024-03-02T01:07:59.406ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117022024-03-02T01:08:00.406ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117032024-03-02T01:08:00.406ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117042024-03-02T01:08:00.407ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117052024-03-02T01:08:00.407ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117062024-03-02T01:08:00.407ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117072024-03-02T01:08:00.407ZINFOcrucible: [2] client re-new 0 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117082024-03-02T01:08:00.407ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117092024-03-02T01:08:01.408ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117102024-03-02T01:08:01.408ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117112024-03-02T01:08:01.408ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117122024-03-02T01:08:01.408ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117132024-03-02T01:08:01.408ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117142024-03-02T01:08:01.408ZINFOcrucible: [2] client re-new 0 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117152024-03-02T01:08:01.408ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11716 Mar 02 01:08:02.407 INFO request completed, latency_us: 254, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2fda5beb-2b29-4363-9156-607dd417ad6f, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
11717 001/300 Read block 3835 len 32768
117182024-03-02T01:08:02.409ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117192024-03-02T01:08:02.409ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117202024-03-02T01:08:02.409ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11721 {"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:08:02.409593138Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}002/300
11722 Write block 2474 len 20480 data: 4 4 4 4{ 4
11723 "msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:08:02.409636555Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
117242024-03-02T01:08:02.409ZINFOcrucible: [2] client re-new 1 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
117252024-03-02T01:08:02.409ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
11726 003/300 Read block 2661 len 28672
11727 004/300 Read block 1318 len 4096
11728 005/300 Read block 3967 len 8192
11729 006/300 Flush
11730 007/300 Flush
11731 008/300 Read block 4077 len 8192
11732 009/300 Write block 677 len 40960 data: 4 4 3 3 4 4 4 4 4 4
11733 010/300 Write block 2102 len 8192 data: 4 4
11734 011/300 Write block 4811 len 20480 data: 4 5 4 3 3
11735 012/300 Read block 3866 len 8192
11736 013/300 Write block 2394 len 8192 data: 3 3
11737 014/300 Read block 2391 len 24576
11738 015/300 Flush
11739 016/300 Read block 1591 len 24576
11740 017/300 Flush
11741 018/300 Read block 4275 len 16384
11742 019/300 Write block 2278 len 16384 data: 4 4 3 2
11743 020/300 Write block 3368 len 24576 data: 5 6 7 7 6 6
11744 021/300 Write block 3644 len 16384 data: 3 3 3 4
11745 022/300 Write block 1439 len 12288 data: 3 3 3
11746 023/300 Read block 2621 len 24576
11747 024/300 Read block 571 len 24576
11748 025/300 Read block 2217 len 8192
11749 026/300 Read block 4844 len 8192
11750 027/300 Read block 894 len 20480
11751 028/300 Write block 703 len 24576 data: 5 4 3 3 3 2
11752 029/300 Flush
11753 030/300 Read block 4138 len 28672
11754 031/300 Read block 3438 len 36864
11755 032/300 Read block 1637 len 16384
11756 033/300 Write block 949 len 28672 data: 4 4 3 3 4 4 4
11757 034/300 Read block 2977 len 24576
11758 035/300 Read block 2681 len 24576
11759 036/300 Read block 1494 len 40960
11760 037/300 Read block 3280 len 20480
11761 038/300 Write block 1892 len 32768 data: 3 4 4 5 4 4 5 4
11762 039/300 Write block 4901 len 12288 data: 4 4 5
11763 040/300 Read block 3897 len 24576
11764 041/300 Read block 219 len 32768
11765 042/300 Write block 4171 len 32768 data: 3 3 3 4 5 6 5 6
11766 043/300 Write block 1902 len 8192 data: 4 3
11767 044/300 Write block 1189 len 20480 data: 3 4 4 4 5
11768 045/300 Write block 4612 len 8192 data: 2 2
11769 046/300 Read block 2208 len 36864
11770 047/300 Write block 696 len 12288 data: 4 4 4
11771 048/300 Read block 3130 len 12288
11772 049/300 Read block 3233 len 32768
11773 050/300 Read block 385 len 36864
11774 051/300 Write block 3990 len 36864 data: 4 5 4 2 3 3 3 3 3
11775 052/300 Flush
11776 053/300 Write block 1 len 28672 data: 3 3 3 3 3 4 4
11777 054/300 Read block 3667 len 16384
11778 055/300 Read block 1922 len 16384
11779 056/300 Write block 1935 len 40960 data: 4 4 5 4 5 5 5 5 5 6
11780 057/300 Read block 1250 len 8192
11781 058/300 Read block 1775 len 8192
11782 059/300 Write block 988 len 36864 data: 3 2 2 3 3 3 3 4 5
11783 060/300 Read block 4541 len 8192
11784 061/300 Read block 1362 len 32768
11785 062/300 Write block 1602 len 8192 data: 2 3
11786 063/300 Write block 4675 len 40960 data: 3 3 2 3 5 5 5 6 7 6
11787 064/300 Read block 2885 len 40960
11788 065/300 Write block 4126 len 8192 data: 2 2
11789 066/300 Write block 76 len 32768 data: 3 3 4 4 3 3 4 4
11790 067/300 Read block 3946 len 4096
11791 068/300 Read block 1434 len 32768
11792 069/300 Read block 4043 len 16384
11793 070/300 Write block 1031 len 20480 data: 5 5 5 5 4
11794 071/300 Write block 1982 len 24576 data: 5 4 4 3 4 3
11795 072/300 Read block 1837 len 24576
11796 073/300 Write block 4736 len 12288 data: 4 4 3
11797 074/300 Read block 2564 len 24576
11798 075/300 Read block 2330 len 16384
11799 076/300 Write block 4617 len 40960 data: 2 2 3 4 5 4 4 4 4 4
11800 077/300 Read block 6 len 32768
11801 078/300 Write block 1276 len 24576 data: 4 4 4 4 4 4
11802 079/300 Read block 1715 len 16384
11803 080/300 Write block 4176 len 16384 data: 7 6 7 6
11804 081/300 Write block 3950 len 4096 data: 6
11805 082/300 Read block 410 len 16384
11806 083/300 Read block 2780 len 16384
11807 084/300 Write block 172 len 12288 data: 3 3 3
11808 085/300 Read block 3374 len 4096
11809 086/300 Write block 3544 len 8192 data: 3 3
11810 087/300 Read block 282 len 36864
11811 088/300 Write block 4782 len 32768 data: 2 3 3 3 4 4 5 4
11812 089/300 Read block 2417 len 20480
11813 090/300 Write block 1809 len 32768 data: 2 2 2 3 3 3 4 4
11814 091/300 Read block 1094 len 32768
11815 092/300 Read block 4482 len 8192
11816 093/300 Read block 567 len 12288
11817 094/300 Read block 4580 len 40960
11818 095/300 Write block 3986 len 20480 data: 5 4 4 4 5
11819 096/300 Write block 1605 len 24576 data: 3 3 3 5 6 4
11820 097/300 Read block 3144 len 40960
11821 098/300 Read block 4806 len 32768
11822 099/300 Read block 2589 len 4096
11823 100/300 Write block 2120 len 4096 data: 6
11824 101/300 Read block 1365 len 12288
11825 102/300 Flush
11826 103/300 Flush
11827 104/300 Write block 4210 len 32768 data: 3 3 3 3 3 3 3 4
11828 105/300 Flush
11829 106/300 Read block 1511 len 20480
11830 107/300 Write block 1934 len 12288 data: 4 5 5
11831 108/300 Read block 2052 len 12288
11832 109/300 Write block 2043 len 40960 data: 7 4 4 3 4 5 5 4 4 4
11833 110/300 Read block 806 len 28672
11834 111/300 Read block 515 len 40960
11835 112/300 Write block 1724 len 32768 data: 2 2 3 3 3 2 2 3
11836 113/300 Flush
11837 114/300 Read block 2949 len 16384
11838 115/300 Read block 133 len 4096
11839 116/300 Write block 4072 len 36864 data: 5 6 6 6 6 5 4 4 3
11840 117/300 Read block 261 len 12288
11841 118/300 Read block 1472 len 8192
11842 119/300 Write block 4783 len 16384 data: 4 4 4 5
11843 120/300 Write block 2760 len 8192 data: 3 4
11844 121/300 Write block 1042 len 12288 data: 5 4 3
11845 122/300 Read block 2592 len 32768
11846 123/300 Read block 3710 len 32768
11847 124/300 Read block 4916 len 8192
11848 125/300 Flush
11849 126/300 Read block 1714 len 16384
11850 127/300 Flush
11851 128/300 Read block 298 len 8192
11852 129/300 Read block 2360 len 8192
11853 130/300 Write block 4980 len 16384 data: 4 4 4 5
11854 131/300 Write block 3286 len 24576 data: 5 4 3 3 3 2
11855 132/300 Write block 1920 len 20480 data: 5 5 6 4 5
11856 133/300 Write block 3411 len 20480 data: 3 4 4 6 5
11857 134/300 Read block 4727 len 32768
11858 135/300 Write block 4275 len 16384 data: 3 4 4 3
11859 136/300 Read block 1770 len 20480
11860 137/300 Read block 3056 len 32768
11861 138/300 Read block 633 len 16384
11862 139/300 Read block 374 len 28672
11863 140/300 Read block 1231 len 28672
11864 141/300 Read block 3348 len 24576
11865 142/300 Write block 3464 len 20480 data: 5 4 4 4 4
11866 143/300 Write block 4636 len 32768 data: 2 2 2 3 3 3 4 7
11867 144/300 Read block 4816 len 8192
11868 145/300 Read block 3072 len 8192
11869 146/300 Read block 2167 len 32768
11870 147/300 Write block 3852 len 12288 data: 5 4 3
11871 148/300 Write block 616 len 20480 data: 6 5 4 4 4
11872 149/300 Write block 3015 len 28672 data: 5 5 5 5 4 4 4
11873 150/300 Read block 2096 len 28672
11874 151/300 Read block 921 len 8192
11875 152/300 Write block 665 len 20480 data: 6 4 4 4 5
11876 153/300 Write block 3825 len 36864 data: 3 3 3 2 2 2 2 3 3
11877 154/300 Write block 287 len 28672 data: 5 5 5 5 5 4 5
11878 155/300 Write block 1147 len 4096 data: 2
11879 156/300 Read block 2048 len 4096
11880 157/300 Read block 117 len 12288
11881 158/300 Flush
11882 159/300 Write block 3705 len 36864 data: 4 4 5 5 5 5 5 6 6
11883 160/300 Read block 167 len 20480
11884 161/300 Write block 3787 len 12288 data: 4 4 4
11885 162/300 Write block 3856 len 12288 data: 4 5 5
11886 163/300 Read block 1249 len 32768
11887 164/300 Write block 276 len 4096 data: 4
11888 165/300 Read block 1988 len 12288
11889 166/300 Write block 3647 len 8192 data: 5 5
11890 167/300 Read block 4560 len 8192
11891 168/300 Flush
11892 169/300 Write block 582 len 28672 data: 2 2 3 3 2 3 4
11893 170/300 Read block 2004 len 32768
11894 171/300 Read block 4287 len 36864
11895 172/300 Read block 1213 len 12288
11896 173/300 Write block 876 len 8192 data: 2 2
11897 174/300 Read block 1126 len 20480
11898 175/300 Write block 2200 len 12288 data: 3 4 4
11899 176/300 Read block 3316 len 28672
11900 177/300 Read block 338 len 36864
11901 178/300 Write block 2560 len 12288 data: 6 6 5
11902 179/300 Read block 2055 len 28672
11903 180/300 Write block 2930 len 40960 data: 4 4 3 4 3 3 3 3 4 4
11904 181/300 Flush
11905 182/300 Write block 1079 len 12288 data: 3 2 2
11906 183/300 Write block 1598 len 28672 data: 2 2 2 2 3 4 3
11907 184/300 Write block 2563 len 28672 data: 5 5 3 3 3 4 3
11908 185/300 Read block 1746 len 12288
11909 186/300 Write block 1771 len 4096 data: 3
11910 187/300 Flush
11911 188/300 Flush
11912 189/300 Write block 3524 len 40960 data: 7 4 4 4 6 5 6 6 6 5
11913 190/300 Write block 3779 len 24576 data: 7 9 9 8 8 7
11914 191/300 Read block 3428 len 32768
11915 192/300 Write block 4419 len 28672 data: 3 3 3 3 3 3 3
11916 193/300 Write block 3115 len 20480 data: 4 5 6 4 4
11917 194/300 Read block 4123 len 16384
11918 195/300 Read block 3957 len 28672
11919 196/300 Flush
11920 197/300 Read block 3419 len 28672
11921 198/300 Flush
11922 199/300 Write block 2161 len 24576 data: 3 2 2 2 2 3
11923 200/300 Write block 4501 len 28672 data: 4 3 4 4 3 3 3
11924 201/300 Write block 3235 len 32768 data: 5 4 3 3 3 2 2 2
11925 202/300 Write block 2553 len 32768 data: 3 3 4 5 4 5 5 7
11926 203/300 Read block 36 len 28672
11927 204/300 Write block 3355 len 36864 data: 6 8 7 6 6 5 3 4 4
11928 205/300 Read block 15 len 28672
11929 206/300 Read block 2461 len 24576
11930 207/300 Read block 4248 len 20480
11931 208/300 Read block 1718 len 36864
11932 209/300 Read block 873 len 40960
11933 210/300 Write block 1569 len 12288 data: 6 6 6
11934 211/300 Read block 4028 len 24576
11935 212/300 Flush
11936 213/300 Read block 4533 len 40960
11937 214/300 Write block 332 len 32768 data: 5 5 7 7 8 8 8 8
11938 215/300 Write block 2114 len 24576 data: 4 4 5 5 4 5
11939 216/300 Write block 4220 len 36864 data: 3 3 4 4 3 2 2 3 3
11940 217/300 Read block 362 len 24576
11941 218/300 Write block 1944 len 32768 data: 7 6 5 5 5 5 5 5
11942 219/300 Read block 4027 len 40960
11943 220/300 Flush
11944 221/300 Read block 3513 len 12288
11945 222/300 Write block 3626 len 24576 data: 3 3 3 3 3 4
11946 223/300 Read block 4989 len 28672
11947 224/300 Write block 3870 len 28672 data: 3 3 3 3 3 3 3
11948 225/300 Write block 656 len 16384 data: 2 2 2 2
11949 226/300 Read block 2400 len 32768
11950 227/300 Read block 1362 len 8192
11951 228/300 Read block 274 len 32768
11952 229/300 Write block 525 len 32768 data: 3 3 4 4 5 6 6 8
11953 230/300 Read block 2879 len 24576
11954 231/300 Write block 637 len 40960 data: 5 5 5 6 6 5 5 5 6 6
11955 232/300 Read block 4450 len 16384
11956 233/300 Read block 2755 len 32768
11957 234/300 Write block 4916 len 12288 data: 3 2 3
11958 235/300 Read block 2801 len 12288
11959 236/300 Write block 4290 len 20480 data: 3 4 5 3 4
11960 237/300 Write block 4142 len 16384 data: 4 2 2 2
11961 238/300 Write block 1677 len 32768 data: 3 2 4 4 5 5 7 7
11962 239/300 Read block 3964 len 32768
11963 240/300 Write block 4158 len 12288 data: 5 5 6
11964 241/300 Flush
11965 242/300 Read block 3339 len 16384
11966 243/300 Read block 3470 len 8192
11967 244/300 Read block 3197 len 28672
11968 245/300 Read block 216 len 32768
11969 246/300 Flush
11970 247/300 Write block 1450 len 20480 data: 5 4 3 3 3
11971 248/300 Write block 4490 len 16384 data: 2 2 2 2
11972 249/300 Write block 1466 len 16384 data: 4 3 3 3
11973 250/300 Read block 1708 len 16384
11974 251/300 Write block 2702 len 40960 data: 2 2 2 2 2 2 3 4 2 2
11975 252/300 Write block 426 len 28672 data: 2 3 3 3 3 3 3
11976 253/300 Read block 91 len 36864
11977 254/300 Read block 1345 len 4096
11978 255/300 Read block 1576 len 36864
11979 256/300 Read block 1341 len 36864
11980 257/300 Write block 4944 len 32768 data: 2 2 4 4 4 3 4 4
11981 258/300 Read block 3956 len 8192
11982 259/300 Read block 163 len 24576
11983 260/300 Read block 690 len 8192
11984 261/300 Write block 2766 len 36864 data: 4 5 5 4 4 3 3 3 4
11985 262/300 Write block 935 len 36864 data: 4 5 5 3 5 5 5 6 6
11986 263/300 Write block 2012 len 20480 data: 3 5 5 5 5
11987 264/300 Read block 2878 len 24576
11988 265/300 Flush
11989 266/300 Read block 1846 len 4096
11990 267/300 Write block 1584 len 36864 data: 5 4 4 3 3 3 3 3 3
11991 268/300 Read block 1195 len 28672
11992 269/300 Read block 524 len 12288
11993 270/300 Write block 3707 len 24576 data: 6 6 6 6 6 7
11994 271/300 Read block 1137 len 32768
11995 272/300 Read block 4107 len 36864
11996 273/300 Write block 824 len 40960 data: 3 4 4 5 5 4 4 5 5 4
11997 274/300 Write block 3241 len 28672 data: 3 3 2 2 3 5 5
11998 275/300 Write block 4073 len 32768 data: 7 7 7 7 6 5 5 4
11999 276/300 Read block 3476 len 36864
12000 277/300 Write block 334 len 32768 data: 8 8 9 9 9 9 8 7
12001 278/300 Read block 4146 len 8192
12002 279/300 Read block 4152 len 24576
12003 280/300 Read block 1143 len 16384
12004 281/300 Write block 1042 len 4096 data: 6
12005 282/300 Write block 4056 len 16384 data: 4 4 3 3
12006 283/300 Write block 2096 len 4096 data: 4
12007 284/300 Read block 704 len 4096
12008 285/300 Write block 1207 len 36864 data: 3 3 4 5 4 4 4 4 3
12009 286/300 Read block 4527 len 24576
12010 287/300 Read block 939 len 4096
12011 288/300 Write block 4655 len 28672 data: 4 3 4 4 5 5 5
12012 289/300 Write block 4181 len 20480 data: 3 3 3 3 5
12013 290/300 Flush
12014 291/300 Write block 334 len 28672 data: 9 9 10 10 10 10 9
12015 292/300 Read block 310 len 28672
12016 293/300 Flush
12017 294/300 Read block 3130 len 20480
12018 295/300 Read block 2626 len 12288
12019 296/300 Read block 3252 len 36864
12020 297/300 Read block 4023 len 24576
12021 298/300 Read block 2127 len 24576
12022 299/300 Write block 287 len 32768 data: 6 6 6 6 6 5 6 4
12023 300/300 Read block 454 len 4096
12024 Main task has work to do, go find it
12025 got dsc Start(2)
12026 start 2
12027 Mar 02 01:08:02.748 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 209a053d-d410-4cd7-99ce-17fa17606311, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
12028 Replay: started 2, returned:Ok(())
12029 ----------------------------------------------------------------
12030 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
12031 States: Active Active Offline
12032 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12033 3921 Acked 4920 Read 8 Done Done New true
12034 3922 Acked 4921 Write 5 Done Done New false
12035 3923 Acked 4922 Read 7 Done Done New false
12036 3924 Acked 4923 Read 1 Done Done New false
12037 3925 Acked 4924 Read 2 Done Done New false
12038 3926 Acked 4925 Flush 0 Done Done New false
12039 3927 Acked 4926 Flush 0 Done Done New false
12040 3928 Acked 4927 Read 2 Done Done New false
12041 3929 Acked 4928 Write 10 Done Done New false
12042 3930 Acked 4929 Write 2 Done Done New false
12043 3931 Acked 4930 Write 5 Done Done New false
12044 3932 Acked 4931 Read 2 Done Done New false
12045 3933 Acked 4932 Write 2 Done Done New false
12046 3934 Acked 4933 Read 6 Done Done New false
12047 3935 Acked 4934 Flush 0 Done Done New false
12048 3936 Acked 4935 Read 6 Done Done New false
12049 3937 Acked 4936 Flush 0 Done Done New false
12050 3938 Acked 4937 Read 4 Done Done New false
12051 3939 Acked 4938 Write 4 Done Done New false
12052 3940 Acked 4939 Write 6 Done Done New false
12053 3941 Acked 4940 Write 4 Done Done New false
12054 3942 Acked 4941 Write 3 Done Done New false
12055 3943 Acked 4942 Read 6 Done Done New false
12056 3944 Acked 4943 Read 6 Done Done New false
12057 3945 Acked 4944 Read 2 Done Done New false
12058 3946 Acked 4945 Read 2 Done Done New false
12059 3947 Acked 4946 Read 5 Done Done New false
12060 3948 Acked 4947 Write 6 Done Done New false
12061 3949 Acked 4948 Flush 0 Done Done New false
12062 3950 Acked 4949 Read 7 Done Done New false
12063 3951 Acked 4950 Read 9 Done Done New false
12064 3952 Acked 4951 Read 4 Done Done New false
12065 3953 Acked 4952 Write 7 Done Done New false
12066 3954 Acked 4953 Read 6 Done Done New false
12067 3955 Acked 4954 Read 6 Done Done New false
12068 3956 Acked 4955 Read 10 Done Done New false
12069 3957 Acked 4956 Read 5 Done Done New false
12070 3958 Acked 4957 Write 8 Done Done New false
12071 3959 Acked 4958 Write 3 Done Done New false
12072 3960 Acked 4959 Read 6 Done Done New false
12073 3961 Acked 4960 Read 8 Done Done New false
12074 3962 Acked 4961 Write 8 Done Done New false
12075 3963 Acked 4962 Write 2 Done Done New false
12076 3964 Acked 4963 Write 5 Done Done New false
12077 3965 Acked 4964 Write 2 Done Done New false
12078 3966 Acked 4965 Read 9 Done Done New false
12079 3967 Acked 4966 Write 3 Done Done New false
12080 3968 Acked 4967 Read 3 Done Done New false
12081 3969 Acked 4968 Read 8 Done Done New false
12082 3970 Acked 4969 Read 9 Done Done New false
12083 3971 Acked 4970 Write 9 Done Done New false
12084 3972 Acked 4971 Flush 0 Done Done New false
12085 3973 Acked 4972 Write 7 Done Done New false
12086 3974 Acked 4973 Read 4 Done Done New false
12087 3975 Acked 4974 Read 4 Done Done New false
12088 3976 Acked 4975 Write 10 Done Done New false
12089 3977 Acked 4976 Read 2 Done Done New false
12090 3978 Acked 4977 Read 2 Done Done New false
12091 3979 Acked 4978 Write 9 Done Done New false
12092 3980 Acked 4979 Read 2 Done Done New false
12093 3981 Acked 4980 Read 8 Done Done New false
12094 3982 Acked 4981 Write 2 Done Done New false
12095 3983 Acked 4982 Write 10 Done Done New false
12096 3984 Acked 4983 Read 10 Done Done New false
12097 3985 Acked 4984 Write 2 Done Done New false
12098 3986 Acked 4985 Write 8 Done Done New false
12099 3987 Acked 4986 Read 1 Done Done New false
12100 3988 Acked 4987 Read 8 Done Done New false
12101 3989 Acked 4988 Read 4 Done Done New false
12102 3990 Acked 4989 Write 5 Done Done New false
12103 3991 Acked 4990 Write 6 Done Done New false
12104 3992 Acked 4991 Read 6 Done Done New false
12105 3993 Acked 4992 Write 3 Done Done New false
12106 3994 Acked 4993 Read 6 Done Done New false
12107 3995 Acked 4994 Read 4 Done Done New false
12108 3996 Acked 4995 Write 10 Done Done New false
12109 3997 Acked 4996 Read 8 Done Done New false
12110 3998 Acked 4997 Write 6 Done Done New false
12111 3999 Acked 4998 Read 4 Done Done New false
12112 4000 Acked 4999 Write 4 Done Done New false
12113 4001 Acked 5000 Write 1 Done Done New false
12114 4002 Acked 5001 Read 4 Done Done New false
12115 4003 Acked 5002 Read 4 Done Done New false
12116 4004 Acked 5003 Write 3 Done Done New false
12117 4005 Acked 5004 Read 1 Done Done New false
12118 4006 Acked 5005 Write 2 Done Done New false
12119 4007 Acked 5006 Read 9 Done Done New false
12120 4008 Acked 5007 Write 8 Done Done New false
12121 4009 Acked 5008 Read 5 Done Done New false
12122 4010 Acked 5009 Write 8 Done Done New false
12123 4011 Acked 5010 Read 8 Done Done New false
12124 4012 Acked 5011 Read 2 Done Done New false
12125 4013 Acked 5012 Read 3 Done Done New false
12126 4014 Acked 5013 Read 10 Done Done New false
12127 4015 Acked 5014 Write 5 Done Done New false
12128 4016 Acked 5015 Write 6 Done Done New false
12129 4017 Acked 5016 Read 10 Done Done New false
12130 4018 Acked 5017 Read 8 Done Done New false
12131 4019 Acked 5018 Read 1 Done Done New false
12132 4020 Acked 5019 Write 1 Done Done New false
12133 4021 Acked 5020 Read 3 Done Done New false
12134 4022 Acked 5021 Flush 0 Done Done New false
12135 4023 Acked 5022 Flush 0 Done Done New false
12136 4024 Acked 5023 Write 8 Done Done New false
12137 4025 Acked 5024 Flush 0 Done Done New false
12138 4026 Acked 5025 Read 5 Done Done New false
12139 4027 Acked 5026 Write 3 Done Done New false
12140 4028 Acked 5027 Read 3 Done Done New false
12141 4029 Acked 5028 Write 10 Done Done New false
12142 4030 Acked 5029 Read 7 Done Done New false
12143 4031 Acked 5030 Read 10 Done Done New false
12144 4032 Acked 5031 Write 8 Done Done New false
12145 4033 Acked 5032 Flush 0 Done Done New false
12146 4034 Acked 5033 Read 4 Done Done New false
12147 4035 Acked 5034 Read 1 Done Done New false
12148 4036 Acked 5035 Write 9 Done Done New false
12149 4037 Acked 5036 Read 3 Done Done New false
12150 4038 Acked 5037 Read 2 Done Done New false
12151 4039 Acked 5038 Write 4 Done Done New false
12152 4040 Acked 5039 Write 2 Done Done New false
12153 4041 Acked 5040 Write 3 Done Done New false
12154 4042 Acked 5041 Read 8 Done Done New false
12155 4043 Acked 5042 Read 8 Done Done New false
12156 4044 Acked 5043 Read 2 Done Done New false
12157 4045 Acked 5044 Flush 0 Done Done New false
12158 4046 Acked 5045 Read 4 Done Done New false
12159 4047 Acked 5046 Flush 0 Done Done New false
12160 4048 Acked 5047 Read 2 Done Done New false
12161 4049 Acked 5048 Read 2 Done Done New false
12162 4050 Acked 5049 Write 4 Done Done New false
12163 4051 Acked 5050 Write 6 Done Done New false
12164 4052 Acked 5051 Write 5 Done Done New false
12165 4053 Acked 5052 Write 5 Done Done New false
12166 4054 Acked 5053 Read 8 Done Done New false
12167 4055 Acked 5054 Write 4 Done Done New false
12168 4056 Acked 5055 Read 5 Done Done New false
12169 4057 Acked 5056 Read 8 Done Done New false
12170 4058 Acked 5057 Read 4 Done Done New false
12171 4059 Acked 5058 Read 7 Done Done New false
12172 4060 Acked 5059 Read 7 Done Done New false
12173 4061 Acked 5060 Read 6 Done Done New false
12174 4062 Acked 5061 Write 5 Done Done New false
12175 4063 Acked 5062 Write 8 Done Done New false
12176 4064 Acked 5063 Read 2 Done Done New false
12177 4065 Acked 5064 Read 2 Done Done New false
12178 4066 Acked 5065 Read 8 Done Done New false
12179 4067 Acked 5066 Write 3 Done Done New false
12180 4068 Acked 5067 Write 5 Done Done New false
12181 4069 Acked 5068 Write 7 Done Done New false
12182 4070 Acked 5069 Read 7 Done Done New false
12183 4071 Acked 5070 Read 2 Done Done New false
12184 4072 Acked 5071 Write 5 Done Done New false
12185 4073 Acked 5072 Write 9 Done Done New false
12186 4074 Acked 5073 Write 7 Done Done New false
12187 4075 Acked 5074 Write 1 Done Done New false
12188 4076 Acked 5075 Read 1 Done Done New false
12189 4077 Acked 5076 Read 3 Done Done New false
12190 4078 Acked 5077 Flush 0 Done Done New false
12191 4079 Acked 5078 Write 9 Done Done New false
12192 4080 Acked 5079 Read 5 Done Done New false
12193 4081 Acked 5080 Write 3 Done Done New false
12194 4082 Acked 5081 Write 3 Done Done New false
12195 4083 Acked 5082 Read 8 Done Done New false
12196 4084 Acked 5083 Write 1 Done Done New false
12197 4085 Acked 5084 Read 3 Done Done New false
12198 4086 Acked 5085 Write 2 Done Done New false
12199 4087 Acked 5086 Read 2 Done Done New false
12200 4088 Acked 5087 Flush 0 Done Done New false
12201 4089 Acked 5088 Write 7 Done Done New false
12202 4090 Acked 5089 Read 8 Done Done New false
12203 4091 Acked 5090 Read 9 Done Done New false
12204 4092 Acked 5091 Read 3 Done Done New false
12205 4093 Acked 5092 Write 2 Done Done New false
12206 4094 Acked 5093 Read 5 Done Done New false
12207 4095 Acked 5094 Write 3 Done Done New false
12208 4096 Acked 5095 Read 7 Done Done New false
12209 4097 Acked 5096 Read 9 Done Done New false
12210 4098 Acked 5097 Write 3 Done Done New false
12211 4099 Acked 5098 Read 7 Done Done New false
12212 4100 Acked 5099 Write 10 Done Done New false
12213 4101 Acked 5100 Flush 0 Done Done New false
12214 4102 Acked 5101 Write 3 Done Done New false
12215 4103 Acked 5102 Write 7 Done Done New false
12216 4104 Acked 5103 Write 7 Done Done New false
12217 4105 Acked 5104 Read 3 Done Done New false
12218 4106 Acked 5105 Write 1 Done Done New false
12219 4107 Acked 5106 Flush 0 Done Done New false
12220 4108 Acked 5107 Flush 0 Done Done New false
12221 4109 Acked 5108 Write 10 Done Done New false
12222 4110 Acked 5109 Write 6 Done Done New false
12223 4111 Acked 5110 Read 8 Done Done New false
12224 4112 Acked 5111 Write 7 Done Done New false
12225 4113 Acked 5112 Write 5 Done Done New false
12226 4114 Acked 5113 Read 4 Done Done New false
12227 4115 Acked 5114 Read 7 Done Done New false
12228 4116 Acked 5115 Flush 0 Done Done New false
12229 4117 Acked 5116 Read 7 Done Done New false
12230 4118 Acked 5117 Flush 0 Done Done New false
12231 4119 Acked 5118 Write 6 Done Done New false
12232 4120 Acked 5119 Write 7 Done Done New false
12233 4121 Acked 5120 Write 8 Done Done New false
12234 4122 Acked 5121 Write 8 Done Done New false
12235 4123 Acked 5122 Read 7 Done Done New false
12236 4124 Acked 5123 Write 9 Done Done New false
12237 4125 Acked 5124 Read 7 Done Done New false
12238 4126 Acked 5125 Read 6 Done Done New false
12239 4127 Acked 5126 Read 5 Done Done New false
12240 4128 Acked 5127 Read 9 Done Done New false
12241 4129 Acked 5128 Read 10 Done Done New false
12242 4130 Acked 5129 Write 3 Done Done New false
12243 4131 Acked 5130 Read 6 Done Done New false
12244 4132 Acked 5131 Flush 0 Done Done New false
12245 4133 Acked 5132 Read 10 Done Done New false
12246 4134 Acked 5133 Write 8 Done Done New false
12247 4135 Acked 5134 Write 6 Done Done New false
12248 4136 Acked 5135 Write 9 Done Done New false
12249 4137 Acked 5136 Read 6 Done Done New false
12250 4138 Acked 5137 Write 8 Done Done New false
12251 4139 Acked 5138 Read 10 Done Done New false
12252 4140 Acked 5139 Flush 0 Done Done New false
12253 4141 Acked 5140 Read 3 Done Done New false
12254 4142 Acked 5141 Write 6 Done Done New false
12255 4143 Acked 5142 Read 7 Done Done New false
12256 4144 Acked 5143 Write 7 Done Done New false
12257 4145 Acked 5144 Write 4 Done Done New false
12258 4146 Acked 5145 Read 8 Done Done New false
12259 4147 Acked 5146 Read 2 Done Done New false
12260 4148 Acked 5147 Read 8 Done Done New false
12261 4149 Acked 5148 Write 8 Done Done New false
12262 4150 Acked 5149 Read 6 Done Done New false
12263 4151 Acked 5150 Write 10 Done Done New false
12264 4152 Acked 5151 Read 4 Done Done New false
12265 4153 Acked 5152 Read 8 Done Done New false
12266 4154 Acked 5153 Write 3 Done Done New false
12267 4155 Acked 5154 Read 3 Done Done New false
12268 4156 Acked 5155 Write 5 Done Done New false
12269 4157 Acked 5156 Write 4 Done Done New false
12270 4158 Acked 5157 Write 8 Done Done New false
12271 4159 Acked 5158 Read 8 Done Done New false
12272 4160 Acked 5159 Write 3 Done Done New false
12273 4161 Acked 5160 Flush 0 Done Done New false
12274 4162 Acked 5161 Read 4 Done Done New false
12275 4163 Acked 5162 Read 2 Done Done New false
12276 4164 Acked 5163 Read 7 Done Done New false
12277 4165 Acked 5164 Read 8 Done Done New false
12278 4166 Acked 5165 Flush 0 Done Done New false
12279 4167 Acked 5166 Write 5 Done Done New false
12280 4168 Acked 5167 Write 4 Done Done New false
12281 4169 Acked 5168 Write 4 Done Done New false
12282 4170 Acked 5169 Read 4 Done Done New false
12283 4171 Acked 5170 Write 10 Done Done New false
12284 4172 Acked 5171 Write 7 Done Done New false
12285 4173 Acked 5172 Read 9 Done Done New false
12286 4174 Acked 5173 Read 1 Done Done New false
12287 4175 Acked 5174 Read 9 Done Done New false
12288 4176 Acked 5175 Read 9 Done Done New false
12289 4177 Acked 5176 Write 8 Done Done New false
12290 4178 Acked 5177 Read 2 Done Done New false
12291 4179 Acked 5178 Read 6 Done Done New false
12292 4180 Acked 5179 Read 2 Done Done New false
12293 4181 Acked 5180 Write 9 Done Done New false
12294 4182 Acked 5181 Write 9 Done Done New false
12295 4183 Acked 5182 Write 5 Done Done New false
12296 4184 Acked 5183 Read 6 Done Done New false
12297 4185 Acked 5184 Flush 0 Done Done New false
12298 4186 Acked 5185 Read 1 Done Done New false
12299 4187 Acked 5186 Write 9 Done Done New false
12300 4188 Acked 5187 Read 7 Done Done New false
12301 4189 Acked 5188 Read 3 Done Done New false
12302 4190 Acked 5189 Write 6 Done Done New false
12303 4191 Acked 5190 Read 8 Done Done New false
12304 4192 Acked 5191 Read 9 Done Done New false
12305 4193 Acked 5192 Write 10 Done Done New false
12306 4194 Acked 5193 Write 7 Done Done New false
12307 4195 Acked 5194 Write 8 Done Done New false
12308 4196 Acked 5195 Read 9 Done Done New false
12309 4197 Acked 5196 Write 8 Done Done New false
12310 4198 Acked 5197 Read 2 Done Done New false
12311 4199 Acked 5198 Read 6 Done Done New false
12312 4200 Acked 5199 Read 4 Done Done New false
12313 4201 Acked 5200 Write 1 Done Done New false
12314 4202 Acked 5201 Write 4 Done Done New false
12315 4203 Acked 5202 Write 1 Done Done New false
12316 4204 Acked 5203 Read 1 Done Done New false
12317 4205 Acked 5204 Write 9 Done Done New false
12318 4206 Acked 5205 Read 6 Done Done New false
12319 4207 Acked 5206 Read 1 Done Done New false
12320 4208 Acked 5207 Write 7 Done Done New false
12321 4209 Acked 5208 Write 5 Done Done New false
12322 4210 Acked 5209 Flush 0 Done Done New false
12323 4211 Acked 5210 Write 7 Done Done New false
12324 4212 Acked 5211 Read 7 Done Done New false
12325 4213 Acked 5212 Flush 0 Done Done New false
12326 4214 Acked 5213 Read 5 Done Done New false
12327 4215 Acked 5214 Read 3 Done Done New false
12328 4216 Acked 5215 Read 9 Done Done New false
12329 4217 Acked 5216 Read 6 Done Done New false
12330 4218 Acked 5217 Read 6 Done Done New false
12331 4219 Acked 5218 Write 8 Done Done New false
12332 4220 Acked 5219 Read 1 Done Done New false
12333 STATES DS:0 DS:1 DS:2 TOTAL
12334 New 0 0 300 300
12335 Sent 0 0 0 0
12336 Done 300 300 0 600
12337 Skipped 0 0 0 0
12338 Error 0 0 0 0
12339 Last Flush: 5212 5212 4919
12340 Downstairs last five completed: 4919 4918 4917 4916 4915
12341 Upstairs last five completed: 4220 4219 4218 4217 4216
12342 CLIENT: Up:0 ds:300 act:2
123432024-03-02T01:08:03.411ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123442024-03-02T01:08:03.411ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123452024-03-02T01:08:03.411ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123462024-03-02T01:08:03.411ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123472024-03-02T01:08:03.411ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123482024-03-02T01:08:03.411ZINFOcrucible: [2] client re-new 301 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123492024-03-02T01:08:03.411ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123502024-03-02T01:08:04.412ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123512024-03-02T01:08:04.412ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123522024-03-02T01:08:04.412ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123532024-03-02T01:08:04.412ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123542024-03-02T01:08:04.412ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123552024-03-02T01:08:04.412ZINFOcrucible: [2] client re-new 301 jobs since flush 4919 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123562024-03-02T01:08:04.412ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
12357 [8830] got start action
12358 [8830] I am going to restart
12359 Starting downstairs at port 8830
12360 Make output file at "/tmp/dsc/downstairs-8830.txt"
12361 [8830][2] reports Starting
12362 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1176)
12363 [8830][2] reports Running
123642024-03-02T01:08:05.414ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123652024-03-02T01:08:05.414ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123662024-03-02T01:08:05.415ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123672024-03-02T01:08:05.415ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123682024-03-02T01:08:05.415ZINFOcrucible: send last flush ID to this DS: 4919 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123692024-03-02T01:08:05.415ZINFOcrucible: Replied this last flush ID: 4919 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123702024-03-02T01:08:05.415ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123712024-03-02T01:08:05.415ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123722024-03-02T01:08:05.415ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123732024-03-02T01:08:05.415ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
12374 ----------------------------------------------------------------
12375 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12376 Downstairs last five completed: 5220 5219 5218 5217 5216
12377 Upstairs last five completed: 4221 4220 4219 4218 4217
12378 CLIENT: Up:0 ds:0 act:3
12379 Replay: All jobs finished, all DS active.
12380 Main task has work to do, go find it
12381 got dsc Stop(2)
12382 stop 2
12383 Mar 02 01:08:06.751 INFO request completed, latency_us: 376, response_code: 204[8830] Got stop action so:false kr:false
12384 , uri: /stop/cid/2, method: POST, req_id: b6e301e4-589a-4aae-b7a3-2a5e52daf5bb, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
12385 Mar 02 01:08:06.752 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/2, method: GET, req_id: bb1c8ea3-5ec1-489c-89ab-321098825f2e, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
123862024-03-02T01:08:06.752ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123872024-03-02T01:08:06.752ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123882024-03-02T01:08:06.752ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123892024-03-02T01:08:06.752ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123902024-03-02T01:08:06.752ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123912024-03-02T01:08:06.752ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
12392 [8830] Exited with: ExitStatus(unix_wait_status(9))
12393 [8830][2] reports Exit
123942024-03-02T01:08:07.754ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123952024-03-02T01:08:07.754ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123962024-03-02T01:08:07.754ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123972024-03-02T01:08:07.754ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123982024-03-02T01:08:07.754ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
123992024-03-02T01:08:07.754ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124002024-03-02T01:08:07.754ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124012024-03-02T01:08:08.755ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124022024-03-02T01:08:08.755ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124032024-03-02T01:08:08.755ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124042024-03-02T01:08:08.755ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124052024-03-02T01:08:08.755ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124062024-03-02T01:08:08.755ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124072024-03-02T01:08:08.756ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124082024-03-02T01:08:09.757ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124092024-03-02T01:08:09.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124102024-03-02T01:08:09.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124112024-03-02T01:08:09.757ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124122024-03-02T01:08:09.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124132024-03-02T01:08:09.757ZINFOcrucible: [2] client re-new 0 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124142024-03-02T01:08:09.757ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
12415 Mar 02 01:08:10.754 INFO request completed, latency_us: 343, response_code: 200, uri: /state/cid/2, method: GET, req_id: eff1eb5a-b760-4135-bb83-78a8222f2c9b, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
12416 001/300 Flush
12417 002/300 Read block 1192 len 36864
12418 003/300 Read block 3200 len 40960
124192024-03-02T01:08:10.757ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124202024-03-02T01:08:10.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124212024-03-02T01:08:10.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124222024-03-02T01:08:10.757ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124232024-03-02T01:08:10.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124242024-03-02T01:08:10.757ZINFOcrucible: [2] client re-new 3 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
124252024-03-02T01:08:10.757ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
12426 004/300 Read block 2206 len 12288
12427 005/300 Write block 1377 len 20480 data: 3 3 2 3 3
12428 006/300 Write block 2581 len 32768 data: 4 4 5 6 7 5 5 5
12429 007/300 Read block 2449 len 4096
12430 008/300 Read block 4819 len 8192
12431 009/300 Write block 4848 len 36864 data: 3 3 3 3 4 4 3 3 3
12432 010/300 Read block 2858 len 20480
12433 011/300 Write block 936 len 20480 data: 6 6 4 6 6
12434 012/300 Read block 640 len 4096
12435 013/300 Read block 2030 len 24576
12436 014/300 Write block 595 len 32768 data: 3 2 2 2 2 2 3 3
12437 015/300 Write block 506 len 12288 data: 2 3 3
12438 016/300 Read block 1470 len 28672
12439 017/300 Write block 2898 len 24576 data: 4 4 4 4 4 3
12440 018/300 Write block 2722 len 28672 data: 5 6 5 4 3 3 4
12441 019/300 Read block 4651 len 12288
12442 020/300 Write block 3686 len 4096 data: 3
12443 021/300 Read block 933 len 16384
12444 022/300 Read block 3183 len 32768
12445 023/300 Read block 4675 len 28672
12446 024/300 Write block 4695 len 16384 data: 3 3 3 2
12447 025/300 Write block 73 len 8192 data: 4 5
12448 026/300 Write block 2965 len 4096 data: 2
12449 027/300 Flush
12450 028/300 Read block 4305 len 4096
12451 029/300 Write block 463 len 12288 data: 3 3 3
12452 030/300 Read block 2742 len 40960
12453 031/300 Read block 4330 len 4096
12454 032/300 Read block 2221 len 36864
12455 033/300 Read block 918 len 28672
12456 034/300 Flush
12457 035/300 Read block 1731 len 28672
12458 036/300 Read block 2802 len 28672
12459 037/300 Write block 1590 len 12288 data: 4 4 4
12460 038/300 Read block 1084 len 40960
12461 039/300 Read block 3284 len 36864
12462 040/300 Read block 234 len 40960
12463 041/300 Write block 3406 len 28672 data: 4 5 5 4 4 4 5
12464 042/300 Read block 4312 len 24576
12465 043/300 Read block 3709 len 32768
12466 044/300 Write block 723 len 16384 data: 5 5 5 5
12467 045/300 Write block 1830 len 24576 data: 3 3 3 5 5 5
12468 046/300 Flush
12469 047/300 Read block 4714 len 24576
12470 048/300 Read block 667 len 24576
12471 049/300 Write block 3439 len 32768 data: 2 3 4 3 4 4 4 5
12472 050/300 Read block 2392 len 32768
12473 051/300 Read block 950 len 8192
12474 052/300 Write block 1817 len 40960 data: 3 2 3 3 4 3 3 3 3 2
12475 053/300 Read block 2007 len 36864
12476 054/300 Read block 4922 len 16384
12477 055/300 Read block 4862 len 28672
12478 056/300 Write block 2062 len 4096 data: 4
12479 057/300 Read block 1798 len 20480
12480 058/300 Write block 3274 len 28672 data: 3 3 3 3 5 5 4
12481 059/300 Flush
12482 060/300 Write block 1459 len 24576 data: 2 2 3 3 4 4
12483 061/300 Write block 2173 len 28672 data: 6 5 5 4 3 3 2
12484 062/300 Read block 964 len 4096
12485 063/300 Read block 3744 len 8192
12486 064/300 Flush
12487 065/300 Read block 1576 len 40960
12488 066/300 Write block 186 len 16384 data: 3 3 3 2
12489 067/300 Read block 4469 len 4096
12490 068/300 Read block 3938 len 40960
12491 069/300 Write block 3276 len 4096 data: 4
12492 070/300 Write block 4764 len 28672 data: 6 5 5 6 5 3 4
12493 071/300 Flush
12494 072/300 Read block 3206 len 40960
12495 073/300 Write block 2275 len 4096 data: 4
12496 074/300 Read block 4650 len 32768
12497 075/300 Flush
12498 076/300 Flush
12499 077/300 Read block 4397 len 12288
12500 078/300 Write block 3936 len 4096 data: 4
12501 079/300 Read block 2090 len 4096
12502 080/300 Write block 636 len 20480 data: 3 6 6 6 7
12503 081/300 Write block 3608 len 32768 data: 3 3 3 4 4 4 4 4
12504 082/300 Read block 2572 len 12288
12505 083/300 Read block 592 len 8192
12506 084/300 Flush
12507 085/300 Write block 3595 len 24576 data: 5 5 5 4 4 3
12508 086/300 Write block 3237 len 24576 data: 4 4 4 3 4 4
12509 087/300 Write block 182 len 32768 data: 3 3 3 3 4 4 4 3
12510 088/300 Write block 4006 len 4096 data: 3
12511 089/300 Read block 4482 len 24576
12512 090/300 Read block 2548 len 40960
12513 091/300 Flush
12514 092/300 Write block 941 len 20480 data: 6 7 7 7 7
12515 093/300 Read block 423 len 36864
12516 094/300 Read block 4681 len 40960
12517 095/300 Flush
12518 096/300 Write block 635 len 20480 data: 3 4 7 7 7
12519 097/300 Write block 3306 len 16384 data: 2 2 2 3
12520 098/300 Write block 2456 len 4096 data: 6
12521 099/300 Read block 943 len 12288
12522 100/300 Read block 1926 len 8192
12523 101/300 Read block 1761 len 12288
12524 102/300 Read block 114 len 40960
12525 103/300 Read block 3812 len 28672
12526 104/300 Read block 654 len 20480
12527 105/300 Flush
12528 106/300 Read block 4889 len 36864
12529 107/300 Write block 3928 len 20480 data: 3 3 4 4 4
12530 108/300 Read block 4013 len 8192
12531 109/300 Read block 3342 len 24576
12532 110/300 Read block 872 len 36864
12533 111/300 Write block 4713 len 20480 data: 5 5 5 4 3
12534 112/300 Write block 2410 len 12288 data: 3 3 3
12535 113/300 Read block 2081 len 36864
12536 114/300 Write block 3072 len 4096 data: 3
12537 115/300 Write block 239 len 28672 data: 4 5 7 6 6 6 6
12538 116/300 Read block 1252 len 8192
12539 117/300 Read block 543 len 16384
12540 118/300 Read block 2391 len 16384
12541 119/300 Read block 285 len 8192
12542 120/300 Read block 2858 len 40960
12543 121/300 Read block 1588 len 28672
12544 122/300 Write block 4428 len 28672 data: 2 2 3 3 5 5 4
12545 123/300 Read block 1836 len 32768
12546 124/300 Read block 3827 len 12288
12547 125/300 Write block 4913 len 16384 data: 3 3 3 4
12548 126/300 Flush
12549 127/300 Read block 4632 len 28672
12550 128/300 Read block 3779 len 36864
12551 129/300 Write block 1327 len 8192 data: 4 4
12552 130/300 Write block 2663 len 28672 data: 3 3 3 3 3 2 2
12553 131/300 Read block 3921 len 12288
12554 132/300 Read block 738 len 12288
12555 133/300 Read block 2709 len 16384
12556 134/300 Read block 1753 len 36864
12557 135/300 Write block 2675 len 32768 data: 4 3 4 4 4 3 3 4
12558 136/300 Write block 4659 len 40960 data: 6 6 6 4 2 2 2 2 2 2
12559 137/300 Write block 2267 len 20480 data: 3 4 4 5 6
12560 138/300 Read block 4007 len 28672
12561 139/300 Write block 507 len 12288 data: 4 4 3
12562 140/300 Read block 1168 len 32768
12563 141/300 Write block 156 len 16384 data: 3 2 3 4
12564 142/300 Flush
12565 143/300 Read block 4224 len 4096
12566 144/300 Read block 4462 len 36864
12567 145/300 Write block 2612 len 32768 data: 2 3 3 3 2 3 3 3
12568 146/300 Write block 3492 len 36864 data: 4 3 3 4 4 4 4 4 3
12569 147/300 Read block 2966 len 24576
12570 148/300 Read block 1829 len 36864
12571 149/300 Write block 3338 len 8192 data: 5 3
12572 150/300 Read block 604 len 4096
12573 151/300 Flush
12574 152/300 Read block 2998 len 32768
12575 153/300 Read block 3468 len 28672
12576 154/300 Read block 127 len 12288
12577 155/300 Write block 1645 len 8192 data: 4 3
12578 156/300 Read block 1406 len 24576
12579 157/300 Write block 4536 len 32768 data: 3 3 3 4 4 5 6 5
12580 158/300 Read block 2720 len 28672
12581 159/300 Flush
12582 160/300 Read block 763 len 8192
12583 161/300 Read block 154 len 8192
12584 162/300 Write block 620 len 12288 data: 5 4 5
12585 163/300 Read block 1429 len 4096
12586 164/300 Read block 454 len 8192
12587 165/300 Write block 4734 len 16384 data: 5 4 5 5
12588 166/300 Write block 339 len 32768 data: 11 10 8 6 5 3 3 3
12589 167/300 Read block 1826 len 12288
12590 168/300 Write block 683 len 36864 data: 5 5 5 5 3 3 4 2 4
12591 169/300 Read block 3275 len 36864
12592 170/300 Read block 2174 len 28672
12593 171/300 Write block 4913 len 12288 data: 4 4 4
12594 172/300 Read block 4428 len 32768
12595 173/300 Write block 3623 len 8192 data: 3 3
12596 174/300 Write block 2016 len 16384 data: 6 5 6 5
12597 175/300 Read block 1726 len 12288
12598 176/300 Write block 3639 len 40960 data: 3 3 3 4 5 4 4 4 6 6
12599 177/300 Write block 3992 len 36864 data: 5 3 4 4 4 4 4 3 3
12600 178/300 Read block 599 len 28672
12601 179/300 Write block 4795 len 28672 data: 6 6 5 4 4 5 5
12602 180/300 Read block 619 len 20480
12603 181/300 Read block 1293 len 32768
12604 182/300 Read block 877 len 20480
12605 183/300 Read block 3716 len 20480
12606 184/300 Write block 3859 len 24576 data: 6 6 6 6 7 6
12607 185/300 Read block 2565 len 28672
12608 186/300 Read block 3560 len 4096
12609 187/300 Read block 3085 len 40960
12610 188/300 Read block 2147 len 12288
12611 189/300 Write block 1808 len 36864 data: 2 3 3 3 4 4 4 5 5
12612 190/300 Flush
12613 191/300 Write block 3013 len 8192 data: 2 3
12614 192/300 Read block 4297 len 8192
12615 193/300 Read block 103 len 24576
12616 194/300 Read block 921 len 8192
12617 195/300 Read block 3010 len 16384
12618 196/300 Flush
12619 197/300 Read block 1482 len 20480
12620 198/300 Read block 2937 len 8192
12621 199/300 Read block 2812 len 16384
12622 200/300 Write block 111 len 24576 data: 5 5 4 3 2 2
12623 201/300 Write block 1862 len 12288 data: 4 2 3
12624 202/300 Read block 223 len 28672
12625 203/300 Write block 692 len 28672 data: 5 5 4 4 5 5 5
12626 204/300 Flush
12627 205/300 Read block 1584 len 40960
12628 206/300 Read block 568 len 32768
12629 207/300 Read block 1697 len 32768
12630 208/300 Read block 293 len 28672
12631 209/300 Read block 2428 len 4096
12632 210/300 Read block 4680 len 20480
12633 211/300 Read block 4192 len 4096
12634 212/300 Write block 1477 len 16384 data: 3 3 3 3
12635 213/300 Write block 3700 len 40960 data: 5 4 4 4 4 5 5 7 7 7
12636 214/300 Write block 1693 len 24576 data: 6 6 6 5 5 7
12637 215/300 Write block 4965 len 24576 data: 3 3 2 3 5 6
12638 216/300 Read block 1398 len 8192
12639 217/300 Write block 4361 len 4096 data: 3
12640 218/300 Write block 2165 len 40960 data: 3 4 3 3 4 4 5 5 7 6
12641 219/300 Read block 1870 len 8192
12642 220/300 Write block 2816 len 24576 data: 3 5 5 5 5 3
12643 221/300 Write block 2660 len 8192 data: 3 3
12644 222/300 Read block 2139 len 40960
12645 223/300 Write block 654 len 16384 data: 2 2 3 3
12646 224/300 Read block 518 len 36864
12647 225/300 Read block 1401 len 32768
12648 226/300 Read block 4697 len 28672
12649 227/300 Read block 311 len 24576
12650 228/300 Flush
12651 229/300 Read block 2332 len 16384
12652 230/300 Read block 160 len 40960
12653 231/300 Flush
12654 232/300 Read block 935 len 32768
12655 233/300 Write block 49 len 16384 data: 4 4 4 4
12656 234/300 Read block 3903 len 16384
12657 235/300 Read block 1580 len 36864
12658 236/300 Write block 414 len 24576 data: 6 5 5 5 5 5
12659 237/300 Read block 3770 len 20480
12660 238/300 Read block 1925 len 40960
12661 239/300 Read block 3195 len 28672
12662 240/300 Read block 2379 len 20480
12663 241/300 Write block 3052 len 16384 data: 4 4 4 4
12664 242/300 Read block 2687 len 4096
12665 243/300 Read block 372 len 40960
12666 244/300 Read block 3969 len 24576
12667 245/300 Flush
12668 246/300 Write block 3136 len 16384 data: 4 4 5 4
12669 247/300 Read block 3275 len 20480
12670 248/300 Write block 701 len 16384 data: 4 5 6 5
12671 249/300 Read block 3629 len 40960
12672 250/300 Write block 1048 len 28672 data: 5 4 4 5 5 6 5
12673 251/300 Read block 1653 len 32768
12674 252/300 Read block 3812 len 8192
12675 253/300 Read block 2047 len 28672
12676 254/300 Write block 4437 len 20480 data: 2 3 3 3 3
12677 255/300 Write block 648 len 8192 data: 5 3
12678 256/300 Read block 2306 len 20480
12679 257/300 Read block 4847 len 36864
12680 258/300 Write block 771 len 28672 data: 3 5 4 4 3 3 3
12681 259/300 Write block 3202 len 36864 data: 2 2 2 2 4 4 5 5 4
12682 260/300 Flush
12683 261/300 Read block 1987 len 16384
12684 262/300 Read block 1076 len 16384
12685 263/300 Read block 2542 len 16384
12686 264/300 Write block 4656 len 28672 data: 4 5 5 7 7 7 5
12687 265/300 Write block 1017 len 36864 data: 4 5 5 6 5 5 5 4 3
12688 266/300 Write block 627 len 28672 data: 3 3 3 3 2 2 2
12689 267/300 Write block 3161 len 8192 data: 5 2
12690 268/300 Write block 3513 len 28672 data: 5 6 6 5 2 2 3
12691 269/300 Write block 4515 len 8192 data: 3 3
12692 270/300 Write block 3047 len 4096 data: 2
12693 271/300 Read block 3063 len 28672
12694 272/300 Write block 1777 len 24576 data: 2 2 2 2 3 3
12695 273/300 Write block 1504 len 24576 data: 4 4 5 6 5 5
12696 274/300 Flush
12697 275/300 Write block 2325 len 12288 data: 2 2 2
12698 276/300 Read block 3761 len 12288
12699 277/300 Read block 4853 len 20480
12700 278/300 Write block 4599 len 40960 data: 3 4 5 6 6 5 3 2 2 2
12701 279/300 Flush
12702 280/300 Write block 3512 len 24576 data: 4 6 7 7 6 3
12703 281/300 Write block 2848 len 20480 data: 5 4 3 2 2
12704 282/300 Read block 1431 len 12288
12705 283/300 Write block 2734 len 8192 data: 4 4
12706 284/300 Read block 3824 len 24576
12707 285/300 Read block 3205 len 32768
12708 286/300 Write block 4194 len 4096 data: 5
12709 287/300 Write block 623 len 32768 data: 5 5 5 4 4 4 4 4
12710 288/300 Write block 2469 len 16384 data: 3 3 4 4
12711 289/300 Read block 2034 len 28672
12712 290/300 Flush
12713 291/300 Flush
12714 292/300 Read block 280 len 24576
12715 293/300 Read block 289 len 8192
12716 294/300 Read block 2119 len 28672
12717 295/300 Read block 543 len 12288
12718 296/300 Read block 4933 len 12288
12719 297/300 Read block 4138 len 24576
12720 298/300 Read block 4298 len 36864
12721 299/300 Write block 3714 len 36864 data: 6 5 5 5 5 5 5 5 6
12722 300/300 Read block 3984 len 4096
12723 Main task has work to do, go find it
12724 got dsc Start(2)
12725 start 2
12726 Mar 02 01:08:11.102 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/2, method: POST, req_id: d5418b96-ad71-4135-91a4-0027e39eed23, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
12727 Replay: started 2, returned:Ok(())
12728 ----------------------------------------------------------------
12729 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12730 States: Active Active Offline
12731 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12732 4222 Acked 5221 Flush 0 Done Done New true
12733 4223 Acked 5222 Read 9 Done Done New true
12734 4224 Acked 5223 Read 10 Done Done New true
12735 4225 Acked 5224 Read 3 Done Done New false
12736 4226 Acked 5225 Write 5 Done Done New false
12737 4227 Acked 5226 Write 8 Done Done New false
12738 4228 Acked 5227 Read 1 Done Done New false
12739 4229 Acked 5228 Read 2 Done Done New false
12740 4230 Acked 5229 Write 9 Done Done New false
12741 4231 Acked 5230 Read 5 Done Done New false
12742 4232 Acked 5231 Write 5 Done Done New false
12743 4233 Acked 5232 Read 1 Done Done New false
12744 4234 Acked 5233 Read 6 Done Done New false
12745 4235 Acked 5234 Write 8 Done Done New false
12746 4236 Acked 5235 Write 3 Done Done New false
12747 4237 Acked 5236 Read 7 Done Done New false
12748 4238 Acked 5237 Write 6 Done Done New false
12749 4239 Acked 5238 Write 7 Done Done New false
12750 4240 Acked 5239 Read 3 Done Done New false
12751 4241 Acked 5240 Write 1 Done Done New false
12752 4242 Acked 5241 Read 4 Done Done New false
12753 4243 Acked 5242 Read 8 Done Done New false
12754 4244 Acked 5243 Read 7 Done Done New false
12755 4245 Acked 5244 Write 4 Done Done New false
12756 4246 Acked 5245 Write 2 Done Done New false
12757 4247 Acked 5246 Write 1 Done Done New false
12758 4248 Acked 5247 Flush 0 Done Done New false
12759 4249 Acked 5248 Read 1 Done Done New false
12760 4250 Acked 5249 Write 3 Done Done New false
12761 4251 Acked 5250 Read 10 Done Done New false
12762 4252 Acked 5251 Read 1 Done Done New false
12763 4253 Acked 5252 Read 9 Done Done New false
12764 4254 Acked 5253 Read 7 Done Done New false
12765 4255 Acked 5254 Flush 0 Done Done New false
12766 4256 Acked 5255 Read 7 Done Done New false
12767 4257 Acked 5256 Read 7 Done Done New false
12768 4258 Acked 5257 Write 3 Done Done New false
12769 4259 Acked 5258 Read 10 Done Done New false
12770 4260 Acked 5259 Read 9 Done Done New false
12771 4261 Acked 5260 Read 10 Done Done New false
12772 4262 Acked 5261 Write 7 Done Done New false
12773 4263 Acked 5262 Read 6 Done Done New false
12774 4264 Acked 5263 Read 8 Done Done New false
12775 4265 Acked 5264 Write 4 Done Done New false
12776 4266 Acked 5265 Write 6 Done Done New false
12777 4267 Acked 5266 Flush 0 Done Done New false
12778 4268 Acked 5267 Read 6 Done Done New false
12779 4269 Acked 5268 Read 6 Done Done New false
12780 4270 Acked 5269 Write 8 Done Done New false
12781 4271 Acked 5270 Read 8 Done Done New false
12782 4272 Acked 5271 Read 2 Done Done New false
12783 4273 Acked 5272 Write 10 Done Done New false
12784 4274 Acked 5273 Read 9 Done Done New false
12785 4275 Acked 5274 Read 4 Done Done New false
12786 4276 Acked 5275 Read 7 Done Done New false
12787 4277 Acked 5276 Write 1 Done Done New false
12788 4278 Acked 5277 Read 5 Done Done New false
12789 4279 Acked 5278 Write 7 Done Done New false
12790 4280 Acked 5279 Flush 0 Done Done New false
12791 4281 Acked 5280 Write 6 Done Done New false
12792 4282 Acked 5281 Write 7 Done Done New false
12793 4283 Acked 5282 Read 1 Done Done New false
12794 4284 Acked 5283 Read 2 Done Done New false
12795 4285 Acked 5284 Flush 0 Done Done New false
12796 4286 Acked 5285 Read 10 Done Done New false
12797 4287 Acked 5286 Write 4 Done Done New false
12798 4288 Acked 5287 Read 1 Done Done New false
12799 4289 Acked 5288 Read 10 Done Done New false
12800 4290 Acked 5289 Write 1 Done Done New false
12801 4291 Acked 5290 Write 7 Done Done New false
12802 4292 Acked 5291 Flush 0 Done Done New false
12803 4293 Acked 5292 Read 10 Done Done New false
12804 4294 Acked 5293 Write 1 Done Done New false
12805 4295 Acked 5294 Read 8 Done Done New false
12806 4296 Acked 5295 Flush 0 Done Done New false
12807 4297 Acked 5296 Flush 0 Done Done New false
12808 4298 Acked 5297 Read 3 Done Done New false
12809 4299 Acked 5298 Write 1 Done Done New false
12810 4300 Acked 5299 Read 1 Done Done New false
12811 4301 Acked 5300 Write 5 Done Done New false
12812 4302 Acked 5301 Write 8 Done Done New false
12813 4303 Acked 5302 Read 3 Done Done New false
12814 4304 Acked 5303 Read 2 Done Done New false
12815 4305 Acked 5304 Flush 0 Done Done New false
12816 4306 Acked 5305 Write 6 Done Done New false
12817 4307 Acked 5306 Write 6 Done Done New false
12818 4308 Acked 5307 Write 8 Done Done New false
12819 4309 Acked 5308 Write 1 Done Done New false
12820 4310 Acked 5309 Read 6 Done Done New false
12821 4311 Acked 5310 Read 10 Done Done New false
12822 4312 Acked 5311 Flush 0 Done Done New false
12823 4313 Acked 5312 Write 5 Done Done New false
12824 4314 Acked 5313 Read 9 Done Done New false
12825 4315 Acked 5314 Read 10 Done Done New false
12826 4316 Acked 5315 Flush 0 Done Done New false
12827 4317 Acked 5316 Write 5 Done Done New false
12828 4318 Acked 5317 Write 4 Done Done New false
12829 4319 Acked 5318 Write 1 Done Done New false
12830 4320 Acked 5319 Read 3 Done Done New false
12831 4321 Acked 5320 Read 2 Done Done New false
12832 4322 Acked 5321 Read 3 Done Done New false
12833 4323 Acked 5322 Read 10 Done Done New false
12834 4324 Acked 5323 Read 7 Done Done New false
12835 4325 Acked 5324 Read 5 Done Done New false
12836 4326 Acked 5325 Flush 0 Done Done New false
12837 4327 Acked 5326 Read 9 Done Done New false
12838 4328 Acked 5327 Write 5 Done Done New false
12839 4329 Acked 5328 Read 2 Done Done New false
12840 4330 Acked 5329 Read 6 Done Done New false
12841 4331 Acked 5330 Read 9 Done Done New false
12842 4332 Acked 5331 Write 5 Done Done New false
12843 4333 Acked 5332 Write 3 Done Done New false
12844 4334 Acked 5333 Read 9 Done Done New false
12845 4335 Acked 5334 Write 1 Done Done New false
12846 4336 Acked 5335 Write 7 Done Done New false
12847 4337 Acked 5336 Read 2 Done Done New false
12848 4338 Acked 5337 Read 4 Done Done New false
12849 4339 Acked 5338 Read 4 Done Done New false
12850 4340 Acked 5339 Read 2 Done Done New false
12851 4341 Acked 5340 Read 10 Done Done New false
12852 4342 Acked 5341 Read 7 Done Done New false
12853 4343 Acked 5342 Flush 0 Done Done New false
12854 4344 Acked 5343 Write 7 Done Done New false
12855 4345 Acked 5344 Read 8 Done Done New false
12856 4346 Acked 5345 Read 3 Done Done New false
12857 4347 Acked 5346 Write 4 Done Done New false
12858 4348 Acked 5347 Flush 0 Done Done New false
12859 4349 Acked 5348 Read 7 Done Done New false
12860 4350 Acked 5349 Read 9 Done Done New false
12861 4351 Acked 5350 Write 2 Done Done New false
12862 4352 Acked 5351 Write 7 Done Done New false
12863 4353 Acked 5352 Read 3 Done Done New false
12864 4354 Acked 5353 Read 3 Done Done New false
12865 4355 Acked 5354 Read 4 Done Done New false
12866 4356 Acked 5355 Read 9 Done Done New false
12867 4357 Acked 5356 Write 8 Done Done New false
12868 4358 Acked 5357 Write 10 Done Done New false
12869 4359 Acked 5358 Write 5 Done Done New false
12870 4360 Acked 5359 Read 7 Done Done New false
12871 4361 Acked 5360 Write 3 Done Done New false
12872 4362 Acked 5361 Read 8 Done Done New false
12873 4363 Acked 5362 Write 4 Done Done New false
12874 4364 Acked 5363 Flush 0 Done Done New false
12875 4365 Acked 5364 Read 1 Done Done New false
12876 4366 Acked 5365 Read 9 Done Done New false
12877 4367 Acked 5366 Write 8 Done Done New false
12878 4368 Acked 5367 Write 9 Done Done New false
12879 4369 Acked 5368 Read 6 Done Done New false
12880 4370 Acked 5369 Read 9 Done Done New false
12881 4371 Acked 5370 Write 2 Done Done New false
12882 4372 Acked 5371 Read 1 Done Done New false
12883 4373 Acked 5372 Flush 0 Done Done New false
12884 4374 Acked 5373 Read 8 Done Done New false
12885 4375 Acked 5374 Read 7 Done Done New false
12886 4376 Acked 5375 Read 3 Done Done New false
12887 4377 Acked 5376 Write 2 Done Done New false
12888 4378 Acked 5377 Read 6 Done Done New false
12889 4379 Acked 5378 Write 8 Done Done New false
12890 4380 Acked 5379 Read 7 Done Done New false
12891 4381 Acked 5380 Flush 0 Done Done New false
12892 4382 Acked 5381 Read 2 Done Done New false
12893 4383 Acked 5382 Read 2 Done Done New false
12894 4384 Acked 5383 Write 3 Done Done New false
12895 4385 Acked 5384 Read 1 Done Done New false
12896 4386 Acked 5385 Read 2 Done Done New false
12897 4387 Acked 5386 Write 4 Done Done New false
12898 4388 Acked 5387 Write 8 Done Done New false
12899 4389 Acked 5388 Read 3 Done Done New false
12900 4390 Acked 5389 Write 9 Done Done New false
12901 4391 Acked 5390 Read 9 Done Done New false
12902 4392 Acked 5391 Read 7 Done Done New false
12903 4393 Acked 5392 Write 3 Done Done New false
12904 4394 Acked 5393 Read 8 Done Done New false
12905 4395 Acked 5394 Write 2 Done Done New false
12906 4396 Acked 5395 Write 4 Done Done New false
12907 4397 Acked 5396 Read 3 Done Done New false
12908 4398 Acked 5397 Write 10 Done Done New false
12909 4399 Acked 5398 Write 9 Done Done New false
12910 4400 Acked 5399 Read 7 Done Done New false
12911 4401 Acked 5400 Write 7 Done Done New false
12912 4402 Acked 5401 Read 5 Done Done New false
12913 4403 Acked 5402 Read 8 Done Done New false
12914 4404 Acked 5403 Read 5 Done Done New false
12915 4405 Acked 5404 Read 5 Done Done New false
12916 4406 Acked 5405 Write 6 Done Done New false
12917 4407 Acked 5406 Read 7 Done Done New false
12918 4408 Acked 5407 Read 1 Done Done New false
12919 4409 Acked 5408 Read 10 Done Done New false
12920 4410 Acked 5409 Read 3 Done Done New false
12921 4411 Acked 5410 Write 9 Done Done New false
12922 4412 Acked 5411 Flush 0 Done Done New false
12923 4413 Acked 5412 Write 2 Done Done New false
12924 4414 Acked 5413 Read 2 Done Done New false
12925 4415 Acked 5414 Read 6 Done Done New false
12926 4416 Acked 5415 Read 2 Done Done New false
12927 4417 Acked 5416 Read 4 Done Done New false
12928 4418 Acked 5417 Flush 0 Done Done New false
12929 4419 Acked 5418 Read 5 Done Done New false
12930 4420 Acked 5419 Read 2 Done Done New false
12931 4421 Acked 5420 Read 4 Done Done New false
12932 4422 Acked 5421 Write 6 Done Done New false
12933 4423 Acked 5422 Write 3 Done Done New false
12934 4424 Acked 5423 Read 7 Done Done New false
12935 4425 Acked 5424 Write 7 Done Done New false
12936 4426 Acked 5425 Flush 0 Done Done New false
12937 4427 Acked 5426 Read 10 Done Done New false
12938 4428 Acked 5427 Read 8 Done Done New false
12939 4429 Acked 5428 Read 8 Done Done New false
12940 4430 Acked 5429 Read 7 Done Done New false
12941 4431 Acked 5430 Read 1 Done Done New false
12942 4432 Acked 5431 Read 5 Done Done New false
12943 4433 Acked 5432 Read 1 Done Done New false
12944 4434 Acked 5433 Write 4 Done Done New false
12945 4435 Acked 5434 Write 10 Done Done New false
12946 4436 Acked 5435 Write 6 Done Done New false
12947 4437 Acked 5436 Write 6 Done Done New false
12948 4438 Acked 5437 Read 2 Done Done New false
12949 4439 Acked 5438 Write 1 Done Done New false
12950 4440 Acked 5439 Write 10 Done Done New false
12951 4441 Acked 5440 Read 2 Done Done New false
12952 4442 Acked 5441 Write 6 Done Done New false
12953 4443 Acked 5442 Write 2 Done Done New false
12954 4444 Acked 5443 Read 10 Done Done New false
12955 4445 Acked 5444 Write 4 Done Done New false
12956 4446 Acked 5445 Read 9 Done Done New false
12957 4447 Acked 5446 Read 8 Done Done New false
12958 4448 Acked 5447 Read 7 Done Done New false
12959 4449 Acked 5448 Read 6 Done Done New false
12960 4450 Acked 5449 Flush 0 Done Done New false
12961 4451 Acked 5450 Read 4 Done Done New false
12962 4452 Acked 5451 Read 10 Done Done New false
12963 4453 Acked 5452 Flush 0 Done Done New false
12964 4454 Acked 5453 Read 8 Done Done New false
12965 4455 Acked 5454 Write 4 Done Done New false
12966 4456 Acked 5455 Read 4 Done Done New false
12967 4457 Acked 5456 Read 9 Done Done New false
12968 4458 Acked 5457 Write 6 Done Done New false
12969 4459 Acked 5458 Read 5 Done Done New false
12970 4460 Acked 5459 Read 10 Done Done New false
12971 4461 Acked 5460 Read 7 Done Done New false
12972 4462 Acked 5461 Read 5 Done Done New false
12973 4463 Acked 5462 Write 4 Done Done New false
12974 4464 Acked 5463 Read 1 Done Done New false
12975 4465 Acked 5464 Read 10 Done Done New false
12976 4466 Acked 5465 Read 6 Done Done New false
12977 4467 Acked 5466 Flush 0 Done Done New false
12978 4468 Acked 5467 Write 4 Done Done New false
12979 4469 Acked 5468 Read 5 Done Done New false
12980 4470 Acked 5469 Write 4 Done Done New false
12981 4471 Acked 5470 Read 10 Done Done New false
12982 4472 Acked 5471 Write 7 Done Done New false
12983 4473 Acked 5472 Read 8 Done Done New false
12984 4474 Acked 5473 Read 2 Done Done New false
12985 4475 Acked 5474 Read 7 Done Done New false
12986 4476 Acked 5475 Write 5 Done Done New false
12987 4477 Acked 5476 Write 2 Done Done New false
12988 4478 Acked 5477 Read 5 Done Done New false
12989 4479 Acked 5478 Read 9 Done Done New false
12990 4480 Acked 5479 Write 7 Done Done New false
12991 4481 Acked 5480 Write 9 Done Done New false
12992 4482 Acked 5481 Flush 0 Done Done New false
12993 4483 Acked 5482 Read 4 Done Done New false
12994 4484 Acked 5483 Read 4 Done Done New false
12995 4485 Acked 5484 Read 4 Done Done New false
12996 4486 Acked 5485 Write 7 Done Done New false
12997 4487 Acked 5486 Write 9 Done Done New false
12998 4488 Acked 5487 Write 7 Done Done New false
12999 4489 Acked 5488 Write 2 Done Done New false
13000 4490 Acked 5489 Write 7 Done Done New false
13001 4491 Acked 5490 Write 2 Done Done New false
13002 4492 Acked 5491 Write 1 Done Done New false
13003 4493 Acked 5492 Read 7 Done Done New false
13004 4494 Acked 5493 Write 6 Done Done New false
13005 4495 Acked 5494 Write 6 Done Done New false
13006 4496 Acked 5495 Flush 0 Done Done New false
13007 4497 Acked 5496 Write 3 Done Done New false
13008 4498 Acked 5497 Read 3 Done Done New false
13009 4499 Acked 5498 Read 5 Done Done New false
13010 4500 Acked 5499 Write 10 Done Done New false
13011 4501 Acked 5500 Flush 0 Done Done New false
13012 4502 Acked 5501 Write 6 Done Done New false
13013 4503 Acked 5502 Write 5 Done Done New false
13014 4504 Acked 5503 Read 3 Done Done New false
13015 4505 Acked 5504 Write 2 Done Done New false
13016 4506 Acked 5505 Read 6 Done Done New false
13017 4507 Acked 5506 Read 8 Done Done New false
13018 4508 Acked 5507 Write 1 Done Done New false
13019 4509 Acked 5508 Write 8 Done Done New false
13020 4510 Acked 5509 Write 4 Done Done New false
13021 4511 Acked 5510 Read 7 Done Done New false
13022 4512 Acked 5511 Flush 0 Done Done New false
13023 4513 Acked 5512 Flush 0 Done Done New false
13024 4514 Acked 5513 Read 6 Done Done New false
13025 4515 Acked 5514 Read 2 Done Done New false
13026 4516 Acked 5515 Read 7 Done Done New false
13027 4517 Acked 5516 Read 3 Done Done New false
13028 4518 Acked 5517 Read 3 Done Done New false
13029 4519 Acked 5518 Read 6 Done Done New false
13030 4520 Acked 5519 Read 9 Done Done New false
13031 4521 Acked 5520 Write 9 Done Done New false
13032 4522 Acked 5521 Read 1 Done Done New false
13033 STATES DS:0 DS:1 DS:2 TOTAL
13034 New 0 0 301 301
13035 Sent 0 0 0 0
13036 Done 301 301 0 602
13037 Skipped 0 0 0 0
13038 Error 0 0 0 0
13039 Last Flush: 5512 5512 5220
13040 Downstairs last five completed: 5220 5219 5218 5217 5216
13041 Upstairs last five completed: 4522 4521 4520 4519 4518
13042 CLIENT: Up:0 ds:301 act:2
130432024-03-02T01:08:11.758ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130442024-03-02T01:08:11.758ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130452024-03-02T01:08:11.758ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130462024-03-02T01:08:11.758ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130472024-03-02T01:08:11.758ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130482024-03-02T01:08:11.758ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130492024-03-02T01:08:11.758ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130502024-03-02T01:08:12.760ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130512024-03-02T01:08:12.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130522024-03-02T01:08:12.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130532024-03-02T01:08:12.760ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130542024-03-02T01:08:12.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130552024-03-02T01:08:12.760ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130562024-03-02T01:08:12.760ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130572024-03-02T01:08:13.761ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130582024-03-02T01:08:13.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130592024-03-02T01:08:13.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130602024-03-02T01:08:13.761ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130612024-03-02T01:08:13.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130622024-03-02T01:08:13.761ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130632024-03-02T01:08:13.761ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13064 [8830] got start action
13065 [8830] I am going to restart
13066 Starting downstairs at port 8830
13067 Make output file at "/tmp/dsc/downstairs-8830.txt"
13068 [8830][2] reports Starting
130692024-03-02T01:08:14.763ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130702024-03-02T01:08:14.763ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130712024-03-02T01:08:14.763ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130722024-03-02T01:08:14.763ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130732024-03-02T01:08:14.763ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130742024-03-02T01:08:14.763ZINFOcrucible: [2] client re-new 302 jobs since flush 5220 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
130752024-03-02T01:08:14.763ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13076 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1177)
13077 [8830][2] reports Running
13078 ----------------------------------------------------------------
13079 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13080 States: Active Active Offline
13081 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13082 4222 Acked 5221 Flush 0 Done Done New true
13083 4223 Acked 5222 Read 9 Done Done New true
13084 4224 Acked 5223 Read 10 Done Done New true
13085 4225 Acked 5224 Read 3 Done Done New true
13086 4226 Acked 5225 Write 5 Done Done New true
13087 4227 Acked 5226 Write 8 Done Done New true
13088 4228 Acked 5227 Read 1 Done Done New true
13089 4229 Acked 5228 Read 2 Done Done New true
13090 4230 Acked 5229 Write 9 Done Done New true
13091 4231 Acked 5230 Read 5 Done Done New true
13092 4232 Acked 5231 Write 5 Done Done New true
13093 4233 Acked 5232 Read 1 Done Done New true
13094 4234 Acked 5233 Read 6 Done Done New true
13095 4235 Acked 5234 Write 8 Done Done New true
13096 4236 Acked 5235 Write 3 Done Done New true
13097 4237 Acked 5236 Read 7 Done Done New true
13098 4238 Acked 5237 Write 6 Done Done New true
13099 4239 Acked 5238 Write 7 Done Done New true
13100 4240 Acked 5239 Read 3 Done Done New true
13101 4241 Acked 5240 Write 1 Done Done New true
13102 4242 Acked 5241 Read 4 Done Done New true
13103 4243 Acked 5242 Read 8 Done Done New true
13104 4244 Acked 5243 Read 7 Done Done New true
13105 4245 Acked 5244 Write 4 Done Done New true
13106 4246 Acked 5245 Write 2 Done Done New true
13107 4247 Acked 5246 Write 1 Done Done New true
13108 4248 Acked 5247 Flush 0 Done Done New true
13109 4249 Acked 5248 Read 1 Done Done New true
13110 4250 Acked 5249 Write 3 Done Done New true
13111 4251 Acked 5250 Read 10 Done Done New true
13112 4252 Acked 5251 Read 1 Done Done New true
13113 4253 Acked 5252 Read 9 Done Done New true
13114 4254 Acked 5253 Read 7 Done Done New true
13115 4255 Acked 5254 Flush 0 Done Done New true
13116 4256 Acked 5255 Read 7 Done Done New true
13117 4257 Acked 5256 Read 7 Done Done New true
13118 4258 Acked 5257 Write 3 Done Done New true
13119 4259 Acked 5258 Read 10 Done Done New true
13120 4260 Acked 5259 Read 9 Done Done New true
13121 4261 Acked 5260 Read 10 Done Done New true
13122 4262 Acked 5261 Write 7 Done Done New true
13123 4263 Acked 5262 Read 6 Done Done New true
13124 4264 Acked 5263 Read 8 Done Done New true
13125 4265 Acked 5264 Write 4 Done Done New true
13126 4266 Acked 5265 Write 6 Done Done New true
13127 4267 Acked 5266 Flush 0 Done Done New true
13128 4268 Acked 5267 Read 6 Done Done New true
13129 4269 Acked 5268 Read 6 Done Done New true
13130 4270 Acked 5269 Write 8 Done Done New true
13131 4271 Acked 5270 Read 8 Done Done New true
13132 4272 Acked 5271 Read 2 Done Done New true
13133 4273 Acked 5272 Write 10 Done Done New true
13134 4274 Acked 5273 Read 9 Done Done New true
13135 4275 Acked 5274 Read 4 Done Done New true
13136 4276 Acked 5275 Read 7 Done Done New true
13137 4277 Acked 5276 Write 1 Done Done New true
13138 4278 Acked 5277 Read 5 Done Done New true
13139 4279 Acked 5278 Write 7 Done Done New true
13140 4280 Acked 5279 Flush 0 Done Done New true
13141 4281 Acked 5280 Write 6 Done Done New true
13142 4282 Acked 5281 Write 7 Done Done New true
13143 4283 Acked 5282 Read 1 Done Done New true
13144 4284 Acked 5283 Read 2 Done Done New true
13145 4285 Acked 5284 Flush 0 Done Done New true
13146 4286 Acked 5285 Read 10 Done Done New true
13147 4287 Acked 5286 Write 4 Done Done New true
13148 4288 Acked 5287 Read 1 Done Done New true
13149 4289 Acked 5288 Read 10 Done Done New true
13150 4290 Acked 5289 Write 1 Done Done New true
13151 4291 Acked 5290 Write 7 Done Done New true
13152 4292 Acked 5291 Flush 0 Done Done New true
13153 4293 Acked 5292 Read 10 Done Done New true
13154 4294 Acked 5293 Write 1 Done Done New true
13155 4295 Acked 5294 Read 8 Done Done New true
13156 4296 Acked 5295 Flush 0 Done Done New true
13157 4297 Acked 5296 Flush 0 Done Done New true
13158 4298 Acked 5297 Read 3 Done Done New true
13159 4299 Acked 5298 Write 1 Done Done New true
13160 4300 Acked 5299 Read 1 Done Done New true
13161 4301 Acked 5300 Write 5 Done Done New true
13162 4302 Acked 5301 Write 8 Done Done New true
13163 4303 Acked 5302 Read 3 Done Done New true
13164 4304 Acked 5303 Read 2 Done Done New true
13165 4305 Acked 5304 Flush 0 Done Done New true
13166 4306 Acked 5305 Write 6 Done Done New true
13167 4307 Acked 5306 Write 6 Done Done New true
13168 4308 Acked 5307 Write 8 Done Done New true
13169 4309 Acked 5308 Write 1 Done Done New true
13170 4310 Acked 5309 Read 6 Done Done New true
13171 4311 Acked 5310 Read 10 Done Done New true
13172 4312 Acked 5311 Flush 0 Done Done New true
13173 4313 Acked 5312 Write 5 Done Done New true
13174 4314 Acked 5313 Read 9 Done Done New true
13175 4315 Acked 5314 Read 10 Done Done New true
13176 4316 Acked 5315 Flush 0 Done Done New true
13177 4317 Acked 5316 Write 5 Done Done New true
13178 4318 Acked 5317 Write 4 Done Done New true
13179 4319 Acked 5318 Write 1 Done Done New true
13180 4320 Acked 5319 Read 3 Done Done New true
13181 4321 Acked 5320 Read 2 Done Done New true
13182 4322 Acked 5321 Read 3 Done Done New true
13183 4323 Acked 5322 Read 10 Done Done New true
13184 4324 Acked 5323 Read 7 Done Done New true
13185 4325 Acked 5324 Read 5 Done Done New true
13186 4326 Acked 5325 Flush 0 Done Done New true
13187 4327 Acked 5326 Read 9 Done Done New true
13188 4328 Acked 5327 Write 5 Done Done New true
13189 4329 Acked 5328 Read 2 Done Done New true
13190 4330 Acked 5329 Read 6 Done Done New true
13191 4331 Acked 5330 Read 9 Done Done New true
13192 4332 Acked 5331 Write 5 Done Done New true
13193 4333 Acked 5332 Write 3 Done Done New true
13194 4334 Acked 5333 Read 9 Done Done New true
13195 4335 Acked 5334 Write 1 Done Done New true
13196 4336 Acked 5335 Write 7 Done Done New true
13197 4337 Acked 5336 Read 2 Done Done New true
13198 4338 Acked 5337 Read 4 Done Done New true
13199 4339 Acked 5338 Read 4 Done Done New true
13200 4340 Acked 5339 Read 2 Done Done New true
13201 4341 Acked 5340 Read 10 Done Done New true
13202 4342 Acked 5341 Read 7 Done Done New true
13203 4343 Acked 5342 Flush 0 Done Done New true
13204 4344 Acked 5343 Write 7 Done Done New true
13205 4345 Acked 5344 Read 8 Done Done New true
13206 4346 Acked 5345 Read 3 Done Done New true
13207 4347 Acked 5346 Write 4 Done Done New true
13208 4348 Acked 5347 Flush 0 Done Done New true
13209 4349 Acked 5348 Read 7 Done Done New true
13210 4350 Acked 5349 Read 9 Done Done New true
13211 4351 Acked 5350 Write 2 Done Done New true
13212 4352 Acked 5351 Write 7 Done Done New true
13213 4353 Acked 5352 Read 3 Done Done New true
13214 4354 Acked 5353 Read 3 Done Done New true
13215 4355 Acked 5354 Read 4 Done Done New true
13216 4356 Acked 5355 Read 9 Done Done New true
13217 4357 Acked 5356 Write 8 Done Done New true
13218 4358 Acked 5357 Write 10 Done Done New true
13219 4359 Acked 5358 Write 5 Done Done New true
13220 4360 Acked 5359 Read 7 Done Done New true
13221 4361 Acked 5360 Write 3 Done Done New true
13222 4362 Acked 5361 Read 8 Done Done New true
13223 4363 Acked 5362 Write 4 Done Done New true
13224 4364 Acked 5363 Flush 0 Done Done New true
13225 4365 Acked 5364 Read 1 Done Done New true
13226 4366 Acked 5365 Read 9 Done Done New true
13227 4367 Acked 5366 Write 8 Done Done New true
13228 4368 Acked 5367 Write 9 Done Done New true
13229 4369 Acked 5368 Read 6 Done Done New true
13230 4370 Acked 5369 Read 9 Done Done New true
13231 4371 Acked 5370 Write 2 Done Done New true
13232 4372 Acked 5371 Read 1 Done Done New true
13233 4373 Acked 5372 Flush 0 Done Done New true
13234 4374 Acked 5373 Read 8 Done Done New true
13235 4375 Acked 5374 Read 7 Done Done New true
13236 4376 Acked 5375 Read 3 Done Done New true
13237 4377 Acked 5376 Write 2 Done Done New true
13238 4378 Acked 5377 Read 6 Done Done New true
13239 4379 Acked 5378 Write 8 Done Done New true
13240 4380 Acked 5379 Read 7 Done Done New true
13241 4381 Acked 5380 Flush 0 Done Done New true
13242 4382 Acked 5381 Read 2 Done Done New true
13243 4383 Acked 5382 Read 2 Done Done New true
13244 4384 Acked 5383 Write 3 Done Done New true
13245 4385 Acked 5384 Read 1 Done Done New true
13246 4386 Acked 5385 Read 2 Done Done New true
13247 4387 Acked 5386 Write 4 Done Done New true
13248 4388 Acked 5387 Write 8 Done Done New true
13249 4389 Acked 5388 Read 3 Done Done New true
13250 4390 Acked 5389 Write 9 Done Done New true
13251 4391 Acked 5390 Read 9 Done Done New true
13252 4392 Acked 5391 Read 7 Done Done New true
13253 4393 Acked 5392 Write 3 Done Done New true
13254 4394 Acked 5393 Read 8 Done Done New true
13255 4395 Acked 5394 Write 2 Done Done New true
13256 4396 Acked 5395 Write 4 Done Done New true
13257 4397 Acked 5396 Read 3 Done Done New true
13258 4398 Acked 5397 Write 10 Done Done New true
13259 4399 Acked 5398 Write 9 Done Done New true
13260 4400 Acked 5399 Read 7 Done Done New true
13261 4401 Acked 5400 Write 7 Done Done New true
13262 4402 Acked 5401 Read 5 Done Done New true
13263 4403 Acked 5402 Read 8 Done Done New true
13264 4404 Acked 5403 Read 5 Done Done New true
13265 4405 Acked 5404 Read 5 Done Done New true
13266 4406 Acked 5405 Write 6 Done Done New true
13267 4407 Acked 5406 Read 7 Done Done New true
13268 4408 Acked 5407 Read 1 Done Done New true
13269 4409 Acked 5408 Read 10 Done Done New true
13270 4410 Acked 5409 Read 3 Done Done New true
13271 4411 Acked 5410 Write 9 Done Done New true
13272 4412 Acked 5411 Flush 0 Done Done New true
13273 4413 Acked 5412 Write 2 Done Done New true
13274 4414 Acked 5413 Read 2 Done Done New true
13275 4415 Acked 5414 Read 6 Done Done New true
13276 4416 Acked 5415 Read 2 Done Done New true
13277 4417 Acked 5416 Read 4 Done Done New true
13278 4418 Acked 5417 Flush 0 Done Done New true
13279 4419 Acked 5418 Read 5 Done Done New true
13280 4420 Acked 5419 Read 2 Done Done New true
13281 4421 Acked 5420 Read 4 Done Done New true
13282 4422 Acked 5421 Write 6 Done Done New true
13283 4423 Acked 5422 Write 3 Done Done New true
13284 4424 Acked 5423 Read 7 Done Done New true
13285 4425 Acked 5424 Write 7 Done Done New true
13286 4426 Acked 5425 Flush 0 Done Done New true
13287 4427 Acked 5426 Read 10 Done Done New true
13288 4428 Acked 5427 Read 8 Done Done New true
13289 4429 Acked 5428 Read 8 Done Done New true
13290 4430 Acked 5429 Read 7 Done Done New true
13291 4431 Acked 5430 Read 1 Done Done New true
13292 4432 Acked 5431 Read 5 Done Done New true
13293 4433 Acked 5432 Read 1 Done Done New true
13294 4434 Acked 5433 Write 4 Done Done New true
13295 4435 Acked 5434 Write 10 Done Done New true
13296 4436 Acked 5435 Write 6 Done Done New true
13297 4437 Acked 5436 Write 6 Done Done New true
13298 4438 Acked 5437 Read 2 Done Done New true
13299 4439 Acked 5438 Write 1 Done Done New true
13300 4440 Acked 5439 Write 10 Done Done New true
13301 4441 Acked 5440 Read 2 Done Done New true
13302 4442 Acked 5441 Write 6 Done Done New true
13303 4443 Acked 5442 Write 2 Done Done New true
13304 4444 Acked 5443 Read 10 Done Done New true
13305 4445 Acked 5444 Write 4 Done Done New true
13306 4446 Acked 5445 Read 9 Done Done New true
13307 4447 Acked 5446 Read 8 Done Done New true
13308 4448 Acked 5447 Read 7 Done Done New true
13309 4449 Acked 5448 Read 6 Done Done New true
13310 4450 Acked 5449 Flush 0 Done Done New true
13311 4451 Acked 5450 Read 4 Done Done New true
13312 4452 Acked 5451 Read 10 Done Done New true
13313 4453 Acked 5452 Flush 0 Done Done New true
13314 4454 Acked 5453 Read 8 Done Done New true
13315 4455 Acked 5454 Write 4 Done Done New true
13316 4456 Acked 5455 Read 4 Done Done New true
13317 4457 Acked 5456 Read 9 Done Done New true
13318 4458 Acked 5457 Write 6 Done Done New true
13319 4459 Acked 5458 Read 5 Done Done New true
13320 4460 Acked 5459 Read 10 Done Done New true
13321 4461 Acked 5460 Read 7 Done Done New true
13322 4462 Acked 5461 Read 5 Done Done New true
13323 4463 Acked 5462 Write 4 Done Done New true
13324 4464 Acked 5463 Read 1 Done Done New true
13325 4465 Acked 5464 Read 10 Done Done New true
13326 4466 Acked 5465 Read 6 Done Done New true
13327 4467 Acked 5466 Flush 0 Done Done New true
13328 4468 Acked 5467 Write 4 Done Done New true
13329 4469 Acked 5468 Read 5 Done Done New true
13330 4470 Acked 5469 Write 4 Done Done New true
13331 4471 Acked 5470 Read 10 Done Done New true
13332 4472 Acked 5471 Write 7 Done Done New true
13333 4473 Acked 5472 Read 8 Done Done New true
13334 4474 Acked 5473 Read 2 Done Done New true
13335 4475 Acked 5474 Read 7 Done Done New true
13336 4476 Acked 5475 Write 5 Done Done New true
13337 4477 Acked 5476 Write 2 Done Done New true
13338 4478 Acked 5477 Read 5 Done Done New true
13339 4479 Acked 5478 Read 9 Done Done New true
13340 4480 Acked 5479 Write 7 Done Done New true
13341 4481 Acked 5480 Write 9 Done Done New true
13342 4482 Acked 5481 Flush 0 Done Done New true
13343 4483 Acked 5482 Read 4 Done Done New true
13344 4484 Acked 5483 Read 4 Done Done New true
13345 4485 Acked 5484 Read 4 Done Done New true
13346 4486 Acked 5485 Write 7 Done Done New true
13347 4487 Acked 5486 Write 9 Done Done New true
13348 4488 Acked 5487 Write 7 Done Done New true
13349 4489 Acked 5488 Write 2 Done Done New true
13350 4490 Acked 5489 Write 7 Done Done New true
13351 4491 Acked 5490 Write 2 Done Done New true
13352 4492 Acked 5491 Write 1 Done Done New true
13353 4493 Acked 5492 Read 7 Done Done New true
13354 4494 Acked 5493 Write 6 Done Done New true
13355 4495 Acked 5494 Write 6 Done Done New true
13356 4496 Acked 5495 Flush 0 Done Done New true
13357 4497 Acked 5496 Write 3 Done Done New true
13358 4498 Acked 5497 Read 3 Done Done New true
13359 4499 Acked 5498 Read 5 Done Done New true
13360 4500 Acked 5499 Write 10 Done Done New true
13361 4501 Acked 5500 Flush 0 Done Done New true
13362 4502 Acked 5501 Write 6 Done Done New true
13363 4503 Acked 5502 Write 5 Done Done New true
13364 4504 Acked 5503 Read 3 Done Done New true
13365 4505 Acked 5504 Write 2 Done Done New true
13366 4506 Acked 5505 Read 6 Done Done New true
13367 4507 Acked 5506 Read 8 Done Done New true
13368 4508 Acked 5507 Write 1 Done Done New true
13369 4509 Acked 5508 Write 8 Done Done New true
13370 4510 Acked 5509 Write 4 Done Done New true
13371 4511 Acked 5510 Read 7 Done Done New true
13372 4512 Acked 5511 Flush 0 Done Done New true
13373 4513 Acked 5512 Flush 0 Done Done New true
13374 4514 Acked 5513 Read 6 Done Done New true
13375 4515 Acked 5514 Read 2 Done Done New true
13376 4516 Acked 5515 Read 7 Done Done New true
13377 4517 Acked 5516 Read 3 Done Done New true
13378 4518 Acked 5517 Read 3 Done Done New true
13379 4519 Acked 5518 Read 6 Done Done New true
13380 4520 Acked 5519 Read 9 Done Done New true
13381 4521 Acked 5520 Write 9 Done Done New true
13382 4522 Acked 5521 Read 1 Done Done New true
13383 4523 Acked 5522 Flush 0 Done Done New true
13384 STATES DS:0 DS:1 DS:2 TOTAL
13385 New 0 0 302 302
13386 Sent 0 0 0 0
13387 Done 302 302 0 604
13388 Skipped 0 0 0 0
13389 Error 0 0 0 0
13390 Last Flush: 5522 5522 5220
13391 Downstairs last five completed: 5220 5219 5218 5217 5216
13392 Upstairs last five completed: 4523 4522 4521 4520 4519
13393 CLIENT: Up:0 ds:302 act:2
133942024-03-02T01:08:15.763ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
133952024-03-02T01:08:15.764ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
133962024-03-02T01:08:15.765ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
133972024-03-02T01:08:15.765ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
133982024-03-02T01:08:15.765ZINFOcrucible: send last flush ID to this DS: 5220 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
133992024-03-02T01:08:15.765ZINFOcrucible: Replied this last flush ID: 5220 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134002024-03-02T01:08:15.765ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134012024-03-02T01:08:15.765ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134022024-03-02T01:08:15.765ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134032024-03-02T01:08:15.765ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13404 ----------------------------------------------------------------
13405 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13406 Downstairs last five completed: 5522 5521 5520 5519 5518
13407 Upstairs last five completed: 4523 4522 4521 4520 4519
13408 CLIENT: Up:0 ds:0 act:3
13409 Replay: All jobs finished, all DS active.
13410 Main task has work to do, go find it
13411 got dsc Stop(1)
13412 stop 1
13413 Mar 02 01:08:19.108 [8820] Got stop action so:false kr:false
13414 INFO request completed, latency_us: 347, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1c7bb224-48ef-400f-93a8-6476c5f8ae57, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
13415 Mar 02 01:08:19.109 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/1, method: GET, req_id: 65638bfe-92fa-4cc7-98f1-b1600fc5ea15, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
134162024-03-02T01:08:19.109ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134172024-03-02T01:08:19.109ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134182024-03-02T01:08:19.109ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134192024-03-02T01:08:19.109ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134202024-03-02T01:08:19.109ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134212024-03-02T01:08:19.109ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13422 [8820] Exited with: ExitStatus(unix_wait_status(9))
13423 [8820][1] reports Exit
134242024-03-02T01:08:20.110ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134252024-03-02T01:08:20.110ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134262024-03-02T01:08:20.110ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134272024-03-02T01:08:20.110ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134282024-03-02T01:08:20.110ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134292024-03-02T01:08:20.110ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134302024-03-02T01:08:20.110ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134312024-03-02T01:08:21.110ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134322024-03-02T01:08:21.111ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134332024-03-02T01:08:21.111ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134342024-03-02T01:08:21.111ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134352024-03-02T01:08:21.111ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134362024-03-02T01:08:21.111ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134372024-03-02T01:08:21.111ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134382024-03-02T01:08:22.112ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134392024-03-02T01:08:22.112ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134402024-03-02T01:08:22.112ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134412024-03-02T01:08:22.112ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134422024-03-02T01:08:22.112ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134432024-03-02T01:08:22.112ZINFOcrucible: [1] client re-new 0 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134442024-03-02T01:08:22.112ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13445 Mar 02 01:08:23.111 INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7cefef6f-6939-4165-bc88-d2e1f7c25d01, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
13446 001/300 Read block 1300 len 32768
134472024-03-02T01:08:23.113ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134482024-03-02T01:08:23.113ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134492024-03-02T01:08:23.113ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134502024-03-02T01:08:23.113ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134512024-03-02T01:08:23.113ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134522024-03-02T01:08:23.113ZINFOcrucible: [1] client re-new 1 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
134532024-03-02T01:08:23.114ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
13454 002/300 Write block 4490 len 32768 data: 3 3 3 3 2 2 3 3
13455 003/300 Flush
13456 004/300 Write block 1685 len 36864 data: 6 6 6 6 5 5 5 6 7
13457 005/300 Flush
13458 006/300 Flush
13459 007/300 Flush
13460 008/300 Read block 2528 len 4096
13461 009/300 Read block 4114 len 16384
13462 010/300 Read block 2116 len 28672
13463 011/300 Read block 1563 len 20480
13464 012/300 Flush
13465 013/300 Read block 919 len 40960
13466 014/300 Write block 1139 len 4096 data: 2
13467 015/300 Write block 3846 len 28672 data: 2 2 4 4 4 5 6
13468 016/300 Read block 407 len 12288
13469 017/300 Flush
13470 018/300 Read block 4931 len 36864
13471 019/300 Read block 3807 len 12288
13472 020/300 Read block 3128 len 4096
13473 021/300 Flush
13474 022/300 Write block 3483 len 20480 data: 5 5 6 4 5
13475 023/300 Read block 2041 len 40960
13476 024/300 Flush
13477 025/300 Write block 152 len 4096 data: 4
13478 026/300 Write block 830 len 40960 data: 5 6 6 5 4 4 4 3 4 4
13479 027/300 Flush
13480 028/300 Read block 3519 len 4096
13481 029/300 Write block 2965 len 16384 data: 3 2 2 2
13482 030/300 Read block 2381 len 40960
13483 031/300 Flush
13484 032/300 Write block 2094 len 32768 data: 5 5 5 3 3 4 4 4
13485 033/300 Read block 1108 len 20480
13486 034/300 Read block 695 len 24576
13487 035/300 Read block 1528 len 8192
13488 036/300 Write block 3515 len 36864 data: 8 7 4 3 4 4 4 6 7
13489 037/300 Read block 4577 len 8192
13490 038/300 Write block 2650 len 4096 data: 4
13491 039/300 Write block 4001 len 16384 data: 3 3 3 3
13492 040/300 Write block 1721 len 32768 data: 2 2 2 3 3 4 4 4
13493 041/300 Write block 1711 len 40960 data: 5 5 5 5 5 4 3 2 2 2
13494 042/300 Write block 4520 len 36864 data: 3 2 3 3 3 3 3 2 2
13495 043/300 Write block 4046 len 8192 data: 3 3
13496 044/300 Read block 3241 len 28672
13497 045/300 Write block 2885 len 24576 data: 4 5 4 4 3 3
13498 046/300 Write block 3698 len 20480 data: 3 5 6 5 5
13499 047/300 Write block 364 len 8192 data: 4 3
13500 048/300 Flush
13501 049/300 Write block 1533 len 4096 data: 7
13502 050/300 Read block 1949 len 36864
13503 051/300 Read block 823 len 12288
13504 052/300 Write block 388 len 20480 data: 4 4 6 6 4
13505 053/300 Write block 786 len 8192 data: 2 2
13506 054/300 Read block 3176 len 8192
13507 055/300 Read block 4968 len 28672
13508 056/300 Read block 4966 len 4096
13509 057/300 Flush
13510 058/300 Flush
13511 059/300 Write block 2672 len 20480 data: 3 3 3 5 4
13512 060/300 Read block 2988 len 32768
13513 061/300 Write block 1027 len 8192 data: 5 4
13514 062/300 Write block 4969 len 16384 data: 6 7 6 5
13515 063/300 Write block 365 len 8192 data: 4 2
13516 064/300 Flush
13517 065/300 Write block 1521 len 40960 data: 3 3 3 3 4 5 4 7 7 6
13518 066/300 Write block 3325 len 36864 data: 3 3 3 3 5 6 5 4 4
13519 067/300 Write block 3271 len 8192 data: 4 4
13520 068/300 Write block 3889 len 12288 data: 3 3 3
13521 069/300 Read block 1173 len 32768
13522 070/300 Flush
13523 071/300 Read block 2137 len 20480
13524 072/300 Write block 3454 len 36864 data: 5 5 3 3 4 3 3 3 4
13525 073/300 Read block 4358 len 16384
13526 074/300 Read block 4580 len 32768
13527 075/300 Write block 116 len 20480 data: 3 2 2 2 2
13528 076/300 Read block 2211 len 36864
13529 077/300 Read block 2780 len 8192
13530 078/300 Write block 3898 len 40960 data: 3 3 4 4 4 5 5 4 3 3
13531 079/300 Write block 3673 len 12288 data: 2 2 3
13532 080/300 Write block 342 len 12288 data: 7 6 4
13533 081/300 Read block 3905 len 32768
13534 082/300 Read block 2853 len 40960
13535 083/300 Write block 4888 len 4096 data: 4
13536 084/300 Write block 2248 len 36864 data: 4 6 6 7 8 5 5 4 4
13537 085/300 Read block 4560 len 36864
13538 086/300 Write block 4431 len 16384 data: 4 6 6 5
13539 087/300 Read block 767 len 36864
13540 088/300 Write block 142 len 4096 data: 2
13541 089/300 Flush
13542 090/300 Flush
13543 091/300 Flush
13544 092/300 Read block 1873 len 36864
13545 093/300 Read block 4577 len 24576
13546 094/300 Write block 4455 len 24576 data: 4 4 4 3 3 2
13547 095/300 Write block 1542 len 16384 data: 2 3 3 3
13548 096/300 Write block 2659 len 16384 data: 3 4 4 3
13549 097/300 Read block 1658 len 24576
13550 098/300 Write block 1088 len 40960 data: 5 3 3 2 2 2 2 5 5 6
13551 099/300 Write block 1209 len 20480 data: 5 6 5 5 5
13552 100/300 Flush
13553 101/300 Write block 2417 len 12288 data: 4 3 3
13554 102/300 Flush
13555 103/300 Flush
13556 104/300 Write block 933 len 24576 data: 4 3 5 7 7 5
13557 105/300 Read block 3721 len 20480
13558 106/300 Read block 2043 len 20480
13559 107/300 Write block 4438 len 36864 data: 4 4 4 4 3 3 4 5 5
13560 108/300 Write block 2140 len 40960 data: 3 2 2 3 3 4 4 4 3 4
13561 109/300 Read block 1829 len 8192
13562 110/300 Read block 4177 len 40960
13563 111/300 Write block 4877 len 20480 data: 3 4 3 3 3
13564 112/300 Flush
13565 113/300 Write block 169 len 24576 data: 3 2 3 4 4 4
13566 114/300 Flush
13567 115/300 Write block 545 len 20480 data: 3 3 3 3 3
13568 116/300 Flush
13569 117/300 Read block 76 len 32768
13570 118/300 Read block 2722 len 20480
13571 119/300 Read block 1431 len 28672
13572 120/300 Read block 4324 len 32768
13573 121/300 Read block 2036 len 36864
13574 122/300 Read block 1641 len 8192
13575 123/300 Read block 951 len 12288
13576 124/300 Read block 860 len 12288
13577 125/300 Write block 292 len 16384 data: 6 7 5 8
13578 126/300 Write block 2365 len 20480 data: 7 6 6 5 5
13579 127/300 Write block 3403 len 24576 data: 5 5 5 5 6 6
13580 128/300 Write block 4762 len 8192 data: 6 6
13581 129/300 Write block 1650 len 32768 data: 5 4 4 4 4 4 3 3
13582 130/300 Write block 2771 len 20480 data: 4 4 4 5 4
13583 131/300 Write block 2752 len 16384 data: 6 6 6 5
13584 132/300 Read block 984 len 20480
13585 133/300 Write block 641 len 8192 data: 7 6
13586 134/300 Read block 4308 len 40960
13587 135/300 Read block 4970 len 32768
13588 136/300 Read block 4597 len 20480
13589 137/300 Write block 4016 len 20480 data: 3 3 3 3 3
13590 138/300 Write block 263 len 16384 data: 4 4 4 6
13591 139/300 Write block 4992 len 32768 data: 5 3 3 4 3 3 3 3
13592 140/300 Read block 3127 len 4096
13593 141/300 Write block 745 len 32768 data: 5 6 6 4 5 6 6 5
13594 142/300 Flush
13595 143/300 Read block 1790 len 32768
13596 144/300 Flush
13597 145/300 Read block 1844 len 20480
13598 146/300 Write block 1899 len 4096 data: 5
13599 147/300 Read block 228 len 32768
13600 148/300 Read block 856 len 32768
13601 149/300 Read block 3793 len 32768
13602 150/300 Read block 1835 len 40960
13603 151/300 Write block 1190 len 12288 data: 5 5 5
13604 152/300 Read block 1099 len 20480
13605 153/300 Write block 1908 len 36864 data: 3 3 3 2 2 2 3 4 4
13606 154/300 Flush
13607 155/300 Write block 2218 len 16384 data: 5 3 3 3
13608 156/300 Write block 467 len 20480 data: 6 7 7 6 5
13609 157/300 Write block 2705 len 16384 data: 3 3 3 4
13610 158/300 Read block 3844 len 32768
13611 159/300 Write block 2603 len 20480 data: 3 4 3 2 3
13612 160/300 Flush
13613 161/300 Write block 857 len 32768 data: 4 4 5 5 4 3 4 4
13614 162/300 Read block 4369 len 40960
13615 163/300 Read block 2198 len 36864
13616 164/300 Read block 1429 len 12288
13617 165/300 Read block 842 len 32768
13618 166/300 Read block 823 len 28672
13619 167/300 Write block 4046 len 16384 data: 4 4 2 2
13620 168/300 Flush
13621 169/300 Read block 2384 len 36864
13622 170/300 Flush
13623 171/300 Read block 2764 len 32768
13624 172/300 Read block 3620 len 20480
13625 173/300 Write block 4937 len 12288 data: 3 4 4
13626 174/300 Flush
13627 175/300 Read block 3017 len 32768
13628 176/300 Read block 4271 len 8192
13629 177/300 Flush
13630 178/300 Write block 2238 len 12288 data: 3 4 4
13631 179/300 Write block 2168 len 20480 data: 4 5 5 6 6
13632 180/300 Write block 339 len 36864 data: 12 11 9 8 7 5 4 4 3
13633 181/300 Read block 1487 len 12288
13634 182/300 Read block 4334 len 4096
13635 183/300 Write block 3230 len 20480 data: 4 4 4 4 5
13636 184/300 Write block 2932 len 12288 data: 4 5 4
13637 185/300 Write block 1043 len 32768 data: 5 4 3 2 4 6 5 5
13638 186/300 Write block 3610 len 12288 data: 4 5 5
13639 187/300 Read block 4583 len 12288
13640 188/300 Read block 3089 len 40960
13641 189/300 Flush
13642 190/300 Read block 2422 len 12288
13643 191/300 Write block 4616 len 4096 data: 2
13644 192/300 Flush
13645 193/300 Read block 4773 len 32768
13646 194/300 Write block 2065 len 40960 data: 5 6 6 6 6 6 5 4 4 2
13647 195/300 Write block 2718 len 12288 data: 7 6 7
13648 196/300 Write block 3861 len 16384 data: 7 7 8 7
13649 197/300 Read block 3600 len 12288
13650 198/300 Write block 4350 len 4096 data: 4
13651 199/300 Read block 3351 len 36864
13652 200/300 Flush
13653 201/300 Write block 3445 len 36864 data: 5 6 6 6 6 6 6 6 5
13654 202/300 Write block 711 len 12288 data: 2 3 3
13655 203/300 Write block 3361 len 36864 data: 4 5 5 3 4 5 5 6 7
13656 204/300 Flush
13657 205/300 Flush
13658 206/300 Read block 1960 len 28672
13659 207/300 Read block 4265 len 40960
13660 208/300 Write block 2532 len 20480 data: 4 4 4 3 4
13661 209/300 Flush
13662 210/300 Write block 2820 len 24576 data: 6 4 3 2 3 5
13663 211/300 Write block 2624 len 40960 data: 2 2 3 3 4 3 3 3 3 3
13664 212/300 Read block 980 len 4096
13665 213/300 Write block 4494 len 12288 data: 3 3 4
13666 214/300 Write block 4911 len 32768 data: 3 3 5 5 5 5 3 4
13667 215/300 Read block 1141 len 36864
13668 216/300 Write block 1263 len 8192 data: 3 4
13669 217/300 Read block 3493 len 32768
13670 218/300 Read block 3027 len 16384
13671 219/300 Write block 1985 len 32768 data: 4 5 4 3 2 3 3 4
13672 220/300 Write block 2174 len 40960 data: 7 6 5 4 4 3 2 2 2 2
13673 221/300 Write block 2364 len 16384 data: 6 8 7 7
13674 222/300 Read block 4653 len 36864
13675 223/300 Read block 770 len 16384
13676 224/300 Read block 3579 len 8192
13677 225/300 Flush
13678 226/300 Flush
13679 227/300 Write block 4204 len 16384 data: 3 2 2 2
13680 228/300 Read block 3091 len 36864
13681 229/300 Read block 1196 len 24576
13682 230/300 Read block 4410 len 16384
13683 231/300 Write block 1818 len 12288 data: 3 4 4
13684 232/300 Read block 2639 len 32768
13685 233/300 Read block 996 len 16384
13686 234/300 Write block 2257 len 32768 data: 4 5 4 3 4 4 4 3
13687 235/300 Read block 1503 len 4096
13688 236/300 Write block 3145 len 4096 data: 4
13689 237/300 Read block 289 len 32768
13690 238/300 Read block 4865 len 4096
13691 239/300 Write block 4502 len 28672 data: 4 5 5 4 4 4 3
13692 240/300 Read block 492 len 12288
13693 241/300 Write block 387 len 16384 data: 3 5 5 7
13694 242/300 Write block 1728 len 24576 data: 5 3 3 4 4 5
13695 243/300 Read block 1867 len 28672
13696 244/300 Read block 4479 len 8192
13697 245/300 Read block 1479 len 28672
13698 246/300 Read block 2748 len 28672
13699 247/300 Read block 4665 len 24576
13700 248/300 Read block 4778 len 20480
13701 249/300 Read block 473 len 28672
13702 250/300 Write block 4438 len 24576 data: 5 5 5 5 4 4
13703 251/300 Read block 2806 len 32768
13704 252/300 Read block 3738 len 4096
13705 253/300 Read block 617 len 8192
13706 254/300 Read block 3430 len 36864
13707 255/300 Read block 870 len 32768
13708 256/300 Write block 2250 len 12288 data: 7 8 9
13709 257/300 Read block 1096 len 24576
13710 258/300 Write block 2873 len 4096 data: 3
13711 259/300 Flush
13712 260/300 Read block 4459 len 24576
13713 261/300 Write block 1545 len 8192 data: 4 3
13714 262/300 Read block 204 len 28672
13715 263/300 Write block 2448 len 20480 data: 5 5 4 3 2
13716 264/300 Read block 375 len 16384
13717 265/300 Read block 2035 len 40960
13718 266/300 Write block 3014 len 28672 data: 4 6 6 6 6 5 5
13719 267/300 Read block 35 len 20480
13720 268/300 Read block 3715 len 8192
13721 269/300 Write block 3468 len 36864 data: 5 4 4 3 2 2 2 2 2
13722 270/300 Read block 1667 len 20480
13723 271/300 Read block 1464 len 4096
13724 272/300 Write block 2778 len 16384 data: 6 6 6 5
13725 273/300 Read block 4737 len 28672
13726 274/300 Write block 2983 len 36864 data: 2 2 2 2 3 3 3 3 5
13727 275/300 Write block 1799 len 24576 data: 2 4 4 4 3 3
13728 276/300 Read block 3373 len 36864
13729 277/300 Write block 3982 len 36864 data: 5 3 5 5 6 5 5 5 6
13730 278/300 Write block 1568 len 36864 data: 4 7 7 7 6 6 6 5 6
13731 279/300 Read block 4122 len 16384
13732 280/300 Write block 3154 len 4096 data: 5
13733 281/300 Read block 2861 len 28672
13734 282/300 Read block 3103 len 12288
13735 283/300 Write block 1448 len 28672 data: 5 4 6 5 4 4 4
13736 284/300 Flush
13737 285/300 Read block 3301 len 40960
13738 286/300 Read block 57 len 40960
13739 287/300 Read block 57 len 20480
13740 288/300 Read block 2181 len 28672
13741 289/300 Write block 576 len 16384 data: 3 4 3 3
13742 290/300 Read block 2938 len 28672
13743 291/300 Write block 1113 len 40960 data: 4 4 4 4 4 4 4 2 4 2
13744 292/300 Read block 322 len 12288
13745 293/300 Read block 4403 len 16384
13746 294/300 Write block 1353 len 16384 data: 2 2 2 4
13747 295/300 Write block 3821 len 12288 data: 7 5 4
13748 296/300 Read block 908 len 12288
13749 297/300 Read block 1442 len 12288
13750 298/300 Write block 2578 len 16384 data: 4 3 5 5
13751 299/300 Write block 4665 len 4096 data: 3
13752 300/300 Read block 4866 len 8192
13753 Main task has work to do, go find it
13754 got dsc Start(1)
13755 start 1
13756 Mar 02 01:08:23.458 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: ba1e800a-74d2-4f49-a8b3-a6f4f69a2403, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
13757 Replay: started 1, returned:Ok(())
13758 ----------------------------------------------------------------
13759 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13760 States: Active Offline Active
13761 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13762 4524 Acked 5523 Read 8 Done New Done true
13763 4525 Acked 5524 Write 8 Done New Done false
13764 4526 Acked 5525 Flush 0 Done New Done false
13765 4527 Acked 5526 Write 9 Done New Done false
13766 4528 Acked 5527 Flush 0 Done New Done false
13767 4529 Acked 5528 Flush 0 Done New Done false
13768 4530 Acked 5529 Flush 0 Done New Done false
13769 4531 Acked 5530 Read 1 Done New Done false
13770 4532 Acked 5531 Read 4 Done New Done false
13771 4533 Acked 5532 Read 7 Done New Done false
13772 4534 Acked 5533 Read 5 Done New Done false
13773 4535 Acked 5534 Flush 0 Done New Done false
13774 4536 Acked 5535 Read 10 Done New Done false
13775 4537 Acked 5536 Write 1 Done New Done false
13776 4538 Acked 5537 Write 7 Done New Done false
13777 4539 Acked 5538 Read 3 Done New Done false
13778 4540 Acked 5539 Flush 0 Done New Done false
13779 4541 Acked 5540 Read 9 Done New Done false
13780 4542 Acked 5541 Read 3 Done New Done false
13781 4543 Acked 5542 Read 1 Done New Done false
13782 4544 Acked 5543 Flush 0 Done New Done false
13783 4545 Acked 5544 Write 5 Done New Done false
13784 4546 Acked 5545 Read 10 Done New Done false
13785 4547 Acked 5546 Flush 0 Done New Done false
13786 4548 Acked 5547 Write 1 Done New Done false
13787 4549 Acked 5548 Write 10 Done New Done false
13788 4550 Acked 5549 Flush 0 Done New Done false
13789 4551 Acked 5550 Read 1 Done New Done false
13790 4552 Acked 5551 Write 4 Done New Done false
13791 4553 Acked 5552 Read 10 Done New Done false
13792 4554 Acked 5553 Flush 0 Done New Done false
13793 4555 Acked 5554 Write 8 Done New Done false
13794 4556 Acked 5555 Read 5 Done New Done false
13795 4557 Acked 5556 Read 6 Done New Done false
13796 4558 Acked 5557 Read 2 Done New Done false
13797 4559 Acked 5558 Write 9 Done New Done false
13798 4560 Acked 5559 Read 2 Done New Done false
13799 4561 Acked 5560 Write 1 Done New Done false
13800 4562 Acked 5561 Write 4 Done New Done false
13801 4563 Acked 5562 Write 8 Done New Done false
13802 4564 Acked 5563 Write 10 Done New Done false
13803 4565 Acked 5564 Write 9 Done New Done false
13804 4566 Acked 5565 Write 2 Done New Done false
13805 4567 Acked 5566 Read 7 Done New Done false
13806 4568 Acked 5567 Write 6 Done New Done false
13807 4569 Acked 5568 Write 5 Done New Done false
13808 4570 Acked 5569 Write 2 Done New Done false
13809 4571 Acked 5570 Flush 0 Done New Done false
13810 4572 Acked 5571 Write 1 Done New Done false
13811 4573 Acked 5572 Read 9 Done New Done false
13812 4574 Acked 5573 Read 3 Done New Done false
13813 4575 Acked 5574 Write 5 Done New Done false
13814 4576 Acked 5575 Write 2 Done New Done false
13815 4577 Acked 5576 Read 2 Done New Done false
13816 4578 Acked 5577 Read 7 Done New Done false
13817 4579 Acked 5578 Read 1 Done New Done false
13818 4580 Acked 5579 Flush 0 Done New Done false
13819 4581 Acked 5580 Flush 0 Done New Done false
13820 4582 Acked 5581 Write 5 Done New Done false
13821 4583 Acked 5582 Read 8 Done New Done false
13822 4584 Acked 5583 Write 2 Done New Done false
13823 4585 Acked 5584 Write 4 Done New Done false
13824 4586 Acked 5585 Write 2 Done New Done false
13825 4587 Acked 5586 Flush 0 Done New Done false
13826 4588 Acked 5587 Write 10 Done New Done false
13827 4589 Acked 5588 Write 9 Done New Done false
13828 4590 Acked 5589 Write 2 Done New Done false
13829 4591 Acked 5590 Write 3 Done New Done false
13830 4592 Acked 5591 Read 8 Done New Done false
13831 4593 Acked 5592 Flush 0 Done New Done false
13832 4594 Acked 5593 Read 5 Done New Done false
13833 4595 Acked 5594 Write 9 Done New Done false
13834 4596 Acked 5595 Read 4 Done New Done false
13835 4597 Acked 5596 Read 8 Done New Done false
13836 4598 Acked 5597 Write 5 Done New Done false
13837 4599 Acked 5598 Read 9 Done New Done false
13838 4600 Acked 5599 Read 2 Done New Done false
13839 4601 Acked 5600 Write 10 Done New Done false
13840 4602 Acked 5601 Write 3 Done New Done false
13841 4603 Acked 5602 Write 3 Done New Done false
13842 4604 Acked 5603 Read 8 Done New Done false
13843 4605 Acked 5604 Read 10 Done New Done false
13844 4606 Acked 5605 Write 1 Done New Done false
13845 4607 Acked 5606 Write 9 Done New Done false
13846 4608 Acked 5607 Read 9 Done New Done false
13847 4609 Acked 5608 Write 4 Done New Done false
13848 4610 Acked 5609 Read 9 Done New Done false
13849 4611 Acked 5610 Write 1 Done New Done false
13850 4612 Acked 5611 Flush 0 Done New Done false
13851 4613 Acked 5612 Flush 0 Done New Done false
13852 4614 Acked 5613 Flush 0 Done New Done false
13853 4615 Acked 5614 Read 9 Done New Done false
13854 4616 Acked 5615 Read 6 Done New Done false
13855 4617 Acked 5616 Write 6 Done New Done false
13856 4618 Acked 5617 Write 4 Done New Done false
13857 4619 Acked 5618 Write 4 Done New Done false
13858 4620 Acked 5619 Read 6 Done New Done false
13859 4621 Acked 5620 Write 10 Done New Done false
13860 4622 Acked 5621 Write 5 Done New Done false
13861 4623 Acked 5622 Flush 0 Done New Done false
13862 4624 Acked 5623 Write 3 Done New Done false
13863 4625 Acked 5624 Flush 0 Done New Done false
13864 4626 Acked 5625 Flush 0 Done New Done false
13865 4627 Acked 5626 Write 6 Done New Done false
13866 4628 Acked 5627 Read 5 Done New Done false
13867 4629 Acked 5628 Read 5 Done New Done false
13868 4630 Acked 5629 Write 9 Done New Done false
13869 4631 Acked 5630 Write 10 Done New Done false
13870 4632 Acked 5631 Read 2 Done New Done false
13871 4633 Acked 5632 Read 10 Done New Done false
13872 4634 Acked 5633 Write 5 Done New Done false
13873 4635 Acked 5634 Flush 0 Done New Done false
13874 4636 Acked 5635 Write 6 Done New Done false
13875 4637 Acked 5636 Flush 0 Done New Done false
13876 4638 Acked 5637 Write 5 Done New Done false
13877 4639 Acked 5638 Flush 0 Done New Done false
13878 4640 Acked 5639 Read 8 Done New Done false
13879 4641 Acked 5640 Read 5 Done New Done false
13880 4642 Acked 5641 Read 7 Done New Done false
13881 4643 Acked 5642 Read 8 Done New Done false
13882 4644 Acked 5643 Read 9 Done New Done false
13883 4645 Acked 5644 Read 2 Done New Done false
13884 4646 Acked 5645 Read 3 Done New Done false
13885 4647 Acked 5646 Read 3 Done New Done false
13886 4648 Acked 5647 Write 4 Done New Done false
13887 4649 Acked 5648 Write 5 Done New Done false
13888 4650 Acked 5649 Write 6 Done New Done false
13889 4651 Acked 5650 Write 2 Done New Done false
13890 4652 Acked 5651 Write 8 Done New Done false
13891 4653 Acked 5652 Write 5 Done New Done false
13892 4654 Acked 5653 Write 4 Done New Done false
13893 4655 Acked 5654 Read 5 Done New Done false
13894 4656 Acked 5655 Write 2 Done New Done false
13895 4657 Acked 5656 Read 10 Done New Done false
13896 4658 Acked 5657 Read 8 Done New Done false
13897 4659 Acked 5658 Read 5 Done New Done false
13898 4660 Acked 5659 Write 5 Done New Done false
13899 4661 Acked 5660 Write 4 Done New Done false
13900 4662 Acked 5661 Write 8 Done New Done false
13901 4663 Acked 5662 Read 1 Done New Done false
13902 4664 Acked 5663 Write 8 Done New Done false
13903 4665 Acked 5664 Flush 0 Done New Done false
13904 4666 Acked 5665 Read 8 Done New Done false
13905 4667 Acked 5666 Flush 0 Done New Done false
13906 4668 Acked 5667 Read 5 Done New Done false
13907 4669 Acked 5668 Write 1 Done New Done false
13908 4670 Acked 5669 Read 8 Done New Done false
13909 4671 Acked 5670 Read 8 Done New Done false
13910 4672 Acked 5671 Read 8 Done New Done false
13911 4673 Acked 5672 Read 10 Done New Done false
13912 4674 Acked 5673 Write 3 Done New Done false
13913 4675 Acked 5674 Read 5 Done New Done false
13914 4676 Acked 5675 Write 9 Done New Done false
13915 4677 Acked 5676 Flush 0 Done New Done false
13916 4678 Acked 5677 Write 4 Done New Done false
13917 4679 Acked 5678 Write 5 Done New Done false
13918 4680 Acked 5679 Write 4 Done New Done false
13919 4681 Acked 5680 Read 8 Done New Done false
13920 4682 Acked 5681 Write 5 Done New Done false
13921 4683 Acked 5682 Flush 0 Done New Done false
13922 4684 Acked 5683 Write 8 Done New Done false
13923 4685 Acked 5684 Read 10 Done New Done false
13924 4686 Acked 5685 Read 9 Done New Done false
13925 4687 Acked 5686 Read 3 Done New Done false
13926 4688 Acked 5687 Read 8 Done New Done false
13927 4689 Acked 5688 Read 7 Done New Done false
13928 4690 Acked 5689 Write 4 Done New Done false
13929 4691 Acked 5690 Flush 0 Done New Done false
13930 4692 Acked 5691 Read 9 Done New Done false
13931 4693 Acked 5692 Flush 0 Done New Done false
13932 4694 Acked 5693 Read 8 Done New Done false
13933 4695 Acked 5694 Read 5 Done New Done false
13934 4696 Acked 5695 Write 3 Done New Done false
13935 4697 Acked 5696 Flush 0 Done New Done false
13936 4698 Acked 5697 Read 8 Done New Done false
13937 4699 Acked 5698 Read 2 Done New Done false
13938 4700 Acked 5699 Flush 0 Done New Done false
13939 4701 Acked 5700 Write 3 Done New Done false
13940 4702 Acked 5701 Write 5 Done New Done false
13941 4703 Acked 5702 Write 9 Done New Done false
13942 4704 Acked 5703 Read 3 Done New Done false
13943 4705 Acked 5704 Read 1 Done New Done false
13944 4706 Acked 5705 Write 5 Done New Done false
13945 4707 Acked 5706 Write 3 Done New Done false
13946 4708 Acked 5707 Write 8 Done New Done false
13947 4709 Acked 5708 Write 3 Done New Done false
13948 4710 Acked 5709 Read 3 Done New Done false
13949 4711 Acked 5710 Read 10 Done New Done false
13950 4712 Acked 5711 Flush 0 Done New Done false
13951 4713 Acked 5712 Read 3 Done New Done false
13952 4714 Acked 5713 Write 1 Done New Done false
13953 4715 Acked 5714 Flush 0 Done New Done false
13954 4716 Acked 5715 Read 8 Done New Done false
13955 4717 Acked 5716 Write 10 Done New Done false
13956 4718 Acked 5717 Write 3 Done New Done false
13957 4719 Acked 5718 Write 4 Done New Done false
13958 4720 Acked 5719 Read 3 Done New Done false
13959 4721 Acked 5720 Write 1 Done New Done false
13960 4722 Acked 5721 Read 9 Done New Done false
13961 4723 Acked 5722 Flush 0 Done New Done false
13962 4724 Acked 5723 Write 9 Done New Done false
13963 4725 Acked 5724 Write 3 Done New Done false
13964 4726 Acked 5725 Write 9 Done New Done false
13965 4727 Acked 5726 Flush 0 Done New Done false
13966 4728 Acked 5727 Flush 0 Done New Done false
13967 4729 Acked 5728 Read 7 Done New Done false
13968 4730 Acked 5729 Read 10 Done New Done false
13969 4731 Acked 5730 Write 5 Done New Done false
13970 4732 Acked 5731 Flush 0 Done New Done false
13971 4733 Acked 5732 Write 6 Done New Done false
13972 4734 Acked 5733 Write 10 Done New Done false
13973 4735 Acked 5734 Read 1 Done New Done false
13974 4736 Acked 5735 Write 3 Done New Done false
13975 4737 Acked 5736 Write 8 Done New Done false
13976 4738 Acked 5737 Read 9 Done New Done false
13977 4739 Acked 5738 Write 2 Done New Done false
13978 4740 Acked 5739 Read 8 Done New Done false
13979 4741 Acked 5740 Read 4 Done New Done false
13980 4742 Acked 5741 Write 8 Done New Done false
13981 4743 Acked 5742 Write 10 Done New Done false
13982 4744 Acked 5743 Write 4 Done New Done false
13983 4745 Acked 5744 Read 9 Done New Done false
13984 4746 Acked 5745 Read 4 Done New Done false
13985 4747 Acked 5746 Read 2 Done New Done false
13986 4748 Acked 5747 Flush 0 Done New Done false
13987 4749 Acked 5748 Flush 0 Done New Done false
13988 4750 Acked 5749 Write 4 Done New Done false
13989 4751 Acked 5750 Read 9 Done New Done false
13990 4752 Acked 5751 Read 6 Done New Done false
13991 4753 Acked 5752 Read 4 Done New Done false
13992 4754 Acked 5753 Write 3 Done New Done false
13993 4755 Acked 5754 Read 8 Done New Done false
13994 4756 Acked 5755 Read 4 Done New Done false
13995 4757 Acked 5756 Write 8 Done New Done false
13996 4758 Acked 5757 Read 1 Done New Done false
13997 4759 Acked 5758 Write 1 Done New Done false
13998 4760 Acked 5759 Read 8 Done New Done false
13999 4761 Acked 5760 Read 1 Done New Done false
14000 4762 Acked 5761 Write 7 Done New Done false
14001 4763 Acked 5762 Read 3 Done New Done false
14002 4764 Acked 5763 Write 4 Done New Done false
14003 4765 Acked 5764 Write 6 Done New Done false
14004 4766 Acked 5765 Read 7 Done New Done false
14005 4767 Acked 5766 Read 2 Done New Done false
14006 4768 Acked 5767 Read 7 Done New Done false
14007 4769 Acked 5768 Read 7 Done New Done false
14008 4770 Acked 5769 Read 6 Done New Done false
14009 4771 Acked 5770 Read 5 Done New Done false
14010 4772 Acked 5771 Read 7 Done New Done false
14011 4773 Acked 5772 Write 6 Done New Done false
14012 4774 Acked 5773 Read 8 Done New Done false
14013 4775 Acked 5774 Read 1 Done New Done false
14014 4776 Acked 5775 Read 2 Done New Done false
14015 4777 Acked 5776 Read 9 Done New Done false
14016 4778 Acked 5777 Read 8 Done New Done false
14017 4779 Acked 5778 Write 3 Done New Done false
14018 4780 Acked 5779 Read 6 Done New Done false
14019 4781 Acked 5780 Write 1 Done New Done false
14020 4782 Acked 5781 Flush 0 Done New Done false
14021 4783 Acked 5782 Read 6 Done New Done false
14022 4784 Acked 5783 Write 2 Done New Done false
14023 4785 Acked 5784 Read 7 Done New Done false
14024 4786 Acked 5785 Write 5 Done New Done false
14025 4787 Acked 5786 Read 4 Done New Done false
14026 4788 Acked 5787 Read 10 Done New Done false
14027 4789 Acked 5788 Write 7 Done New Done false
14028 4790 Acked 5789 Read 5 Done New Done false
14029 4791 Acked 5790 Flush 0 Done New Done false
14030 4792 Acked 5791 Read 2 Done New Done false
14031 4793 Acked 5792 Write 9 Done New Done false
14032 4794 Acked 5793 Read 5 Done New Done false
14033 4795 Acked 5794 Read 1 Done New Done false
14034 4796 Acked 5795 Write 4 Done New Done false
14035 4797 Acked 5796 Read 7 Done New Done false
14036 4798 Acked 5797 Write 9 Done New Done false
14037 4799 Acked 5798 Write 6 Done New Done false
14038 4800 Acked 5799 Read 9 Done New Done false
14039 4801 Acked 5800 Write 9 Done New Done false
14040 4802 Acked 5801 Write 9 Done New Done false
14041 4803 Acked 5802 Read 4 Done New Done false
14042 4804 Acked 5803 Write 1 Done New Done false
14043 4805 Acked 5804 Read 7 Done New Done false
14044 4806 Acked 5805 Read 3 Done New Done false
14045 4807 Acked 5806 Write 7 Done New Done false
14046 4808 Acked 5807 Flush 0 Done New Done false
14047 4809 Acked 5808 Read 10 Done New Done false
14048 4810 Acked 5809 Read 10 Done New Done false
14049 4811 Acked 5810 Read 5 Done New Done false
14050 4812 Acked 5811 Read 7 Done New Done false
14051 4813 Acked 5812 Write 4 Done New Done false
14052 4814 Acked 5813 Read 7 Done New Done false
14053 4815 Acked 5814 Write 10 Done New Done false
14054 4816 Acked 5815 Read 3 Done New Done false
14055 4817 Acked 5816 Read 4 Done New Done false
14056 4818 Acked 5817 Write 4 Done New Done false
14057 4819 Acked 5818 Write 3 Done New Done false
14058 4820 Acked 5819 Read 3 Done New Done false
14059 4821 Acked 5820 Read 3 Done New Done false
14060 4822 Acked 5821 Write 4 Done New Done false
14061 4823 Acked 5822 Write 1 Done New Done false
14062 4824 Acked 5823 Read 2 Done New Done false
14063 STATES DS:0 DS:1 DS:2 TOTAL
14064 New 0 301 0 301
14065 Sent 0 0 0 0
14066 Done 301 0 301 602
14067 Skipped 0 0 0 0
14068 Error 0 0 0 0
14069 Last Flush: 5807 5522 5807
14070 Downstairs last five completed: 5522 5521 5520 5519 5518
14071 Upstairs last five completed: 4824 4823 4822 4821 4820
14072 CLIENT: Up:0 ds:301 act:2
140732024-03-02T01:08:24.114ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140742024-03-02T01:08:24.114ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140752024-03-02T01:08:24.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140762024-03-02T01:08:24.115ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140772024-03-02T01:08:24.115ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140782024-03-02T01:08:24.115ZINFOcrucible: [1] client re-new 302 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140792024-03-02T01:08:24.115ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140802024-03-02T01:08:25.115ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140812024-03-02T01:08:25.115ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140822024-03-02T01:08:25.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140832024-03-02T01:08:25.116ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140842024-03-02T01:08:25.116ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140852024-03-02T01:08:25.116ZINFOcrucible: [1] client re-new 302 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140862024-03-02T01:08:25.116ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140872024-03-02T01:08:26.117ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140882024-03-02T01:08:26.117ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140892024-03-02T01:08:26.117ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140902024-03-02T01:08:26.117ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140912024-03-02T01:08:26.117ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140922024-03-02T01:08:26.117ZINFOcrucible: [1] client re-new 302 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
140932024-03-02T01:08:26.118ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14094 [8820] got start action
14095 [8820] I am going to restart
14096 Starting downstairs at port 8820
14097 Make output file at "/tmp/dsc/downstairs-8820.txt"
14098 [8820][1] reports Starting
140992024-03-02T01:08:27.119ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141002024-03-02T01:08:27.119ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141012024-03-02T01:08:27.119ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141022024-03-02T01:08:27.119ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141032024-03-02T01:08:27.119ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141042024-03-02T01:08:27.119ZINFOcrucible: [1] client re-new 302 jobs since flush 5522 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
141052024-03-02T01:08:27.119ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14106 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1178)
14107 [8820][1] reports Running
14108 ----------------------------------------------------------------
14109 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14110 States: Active Offline Active
14111 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14112 4524 Acked 5523 Read 8 Done New Done true
14113 4525 Acked 5524 Write 8 Done New Done true
14114 4526 Acked 5525 Flush 0 Done New Done true
14115 4527 Acked 5526 Write 9 Done New Done true
14116 4528 Acked 5527 Flush 0 Done New Done true
14117 4529 Acked 5528 Flush 0 Done New Done true
14118 4530 Acked 5529 Flush 0 Done New Done true
14119 4531 Acked 5530 Read 1 Done New Done true
14120 4532 Acked 5531 Read 4 Done New Done true
14121 4533 Acked 5532 Read 7 Done New Done true
14122 4534 Acked 5533 Read 5 Done New Done true
14123 4535 Acked 5534 Flush 0 Done New Done true
14124 4536 Acked 5535 Read 10 Done New Done true
14125 4537 Acked 5536 Write 1 Done New Done true
14126 4538 Acked 5537 Write 7 Done New Done true
14127 4539 Acked 5538 Read 3 Done New Done true
14128 4540 Acked 5539 Flush 0 Done New Done true
14129 4541 Acked 5540 Read 9 Done New Done true
14130 4542 Acked 5541 Read 3 Done New Done true
14131 4543 Acked 5542 Read 1 Done New Done true
14132 4544 Acked 5543 Flush 0 Done New Done true
14133 4545 Acked 5544 Write 5 Done New Done true
14134 4546 Acked 5545 Read 10 Done New Done true
14135 4547 Acked 5546 Flush 0 Done New Done true
14136 4548 Acked 5547 Write 1 Done New Done true
14137 4549 Acked 5548 Write 10 Done New Done true
14138 4550 Acked 5549 Flush 0 Done New Done true
14139 4551 Acked 5550 Read 1 Done New Done true
14140 4552 Acked 5551 Write 4 Done New Done true
14141 4553 Acked 5552 Read 10 Done New Done true
14142 4554 Acked 5553 Flush 0 Done New Done true
14143 4555 Acked 5554 Write 8 Done New Done true
14144 4556 Acked 5555 Read 5 Done New Done true
14145 4557 Acked 5556 Read 6 Done New Done true
14146 4558 Acked 5557 Read 2 Done New Done true
14147 4559 Acked 5558 Write 9 Done New Done true
14148 4560 Acked 5559 Read 2 Done New Done true
14149 4561 Acked 5560 Write 1 Done New Done true
14150 4562 Acked 5561 Write 4 Done New Done true
14151 4563 Acked 5562 Write 8 Done New Done true
14152 4564 Acked 5563 Write 10 Done New Done true
14153 4565 Acked 5564 Write 9 Done New Done true
14154 4566 Acked 5565 Write 2 Done New Done true
14155 4567 Acked 5566 Read 7 Done New Done true
14156 4568 Acked 5567 Write 6 Done New Done true
14157 4569 Acked 5568 Write 5 Done New Done true
14158 4570 Acked 5569 Write 2 Done New Done true
14159 4571 Acked 5570 Flush 0 Done New Done true
14160 4572 Acked 5571 Write 1 Done New Done true
14161 4573 Acked 5572 Read 9 Done New Done true
14162 4574 Acked 5573 Read 3 Done New Done true
14163 4575 Acked 5574 Write 5 Done New Done true
14164 4576 Acked 5575 Write 2 Done New Done true
14165 4577 Acked 5576 Read 2 Done New Done true
14166 4578 Acked 5577 Read 7 Done New Done true
14167 4579 Acked 5578 Read 1 Done New Done true
14168 4580 Acked 5579 Flush 0 Done New Done true
14169 4581 Acked 5580 Flush 0 Done New Done true
14170 4582 Acked 5581 Write 5 Done New Done true
14171 4583 Acked 5582 Read 8 Done New Done true
14172 4584 Acked 5583 Write 2 Done New Done true
14173 4585 Acked 5584 Write 4 Done New Done true
14174 4586 Acked 5585 Write 2 Done New Done true
14175 4587 Acked 5586 Flush 0 Done New Done true
14176 4588 Acked 5587 Write 10 Done New Done true
14177 4589 Acked 5588 Write 9 Done New Done true
14178 4590 Acked 5589 Write 2 Done New Done true
14179 4591 Acked 5590 Write 3 Done New Done true
14180 4592 Acked 5591 Read 8 Done New Done true
14181 4593 Acked 5592 Flush 0 Done New Done true
14182 4594 Acked 5593 Read 5 Done New Done true
14183 4595 Acked 5594 Write 9 Done New Done true
14184 4596 Acked 5595 Read 4 Done New Done true
14185 4597 Acked 5596 Read 8 Done New Done true
14186 4598 Acked 5597 Write 5 Done New Done true
14187 4599 Acked 5598 Read 9 Done New Done true
14188 4600 Acked 5599 Read 2 Done New Done true
14189 4601 Acked 5600 Write 10 Done New Done true
14190 4602 Acked 5601 Write 3 Done New Done true
14191 4603 Acked 5602 Write 3 Done New Done true
14192 4604 Acked 5603 Read 8 Done New Done true
14193 4605 Acked 5604 Read 10 Done New Done true
14194 4606 Acked 5605 Write 1 Done New Done true
14195 4607 Acked 5606 Write 9 Done New Done true
14196 4608 Acked 5607 Read 9 Done New Done true
14197 4609 Acked 5608 Write 4 Done New Done true
14198 4610 Acked 5609 Read 9 Done New Done true
14199 4611 Acked 5610 Write 1 Done New Done true
14200 4612 Acked 5611 Flush 0 Done New Done true
14201 4613 Acked 5612 Flush 0 Done New Done true
14202 4614 Acked 5613 Flush 0 Done New Done true
14203 4615 Acked 5614 Read 9 Done New Done true
14204 4616 Acked 5615 Read 6 Done New Done true
14205 4617 Acked 5616 Write 6 Done New Done true
14206 4618 Acked 5617 Write 4 Done New Done true
14207 4619 Acked 5618 Write 4 Done New Done true
14208 4620 Acked 5619 Read 6 Done New Done true
14209 4621 Acked 5620 Write 10 Done New Done true
14210 4622 Acked 5621 Write 5 Done New Done true
14211 4623 Acked 5622 Flush 0 Done New Done true
14212 4624 Acked 5623 Write 3 Done New Done true
14213 4625 Acked 5624 Flush 0 Done New Done true
14214 4626 Acked 5625 Flush 0 Done New Done true
14215 4627 Acked 5626 Write 6 Done New Done true
14216 4628 Acked 5627 Read 5 Done New Done true
14217 4629 Acked 5628 Read 5 Done New Done true
14218 4630 Acked 5629 Write 9 Done New Done true
14219 4631 Acked 5630 Write 10 Done New Done true
14220 4632 Acked 5631 Read 2 Done New Done true
14221 4633 Acked 5632 Read 10 Done New Done true
14222 4634 Acked 5633 Write 5 Done New Done true
14223 4635 Acked 5634 Flush 0 Done New Done true
14224 4636 Acked 5635 Write 6 Done New Done true
14225 4637 Acked 5636 Flush 0 Done New Done true
14226 4638 Acked 5637 Write 5 Done New Done true
14227 4639 Acked 5638 Flush 0 Done New Done true
14228 4640 Acked 5639 Read 8 Done New Done true
14229 4641 Acked 5640 Read 5 Done New Done true
14230 4642 Acked 5641 Read 7 Done New Done true
14231 4643 Acked 5642 Read 8 Done New Done true
14232 4644 Acked 5643 Read 9 Done New Done true
14233 4645 Acked 5644 Read 2 Done New Done true
14234 4646 Acked 5645 Read 3 Done New Done true
14235 4647 Acked 5646 Read 3 Done New Done true
14236 4648 Acked 5647 Write 4 Done New Done true
14237 4649 Acked 5648 Write 5 Done New Done true
14238 4650 Acked 5649 Write 6 Done New Done true
14239 4651 Acked 5650 Write 2 Done New Done true
14240 4652 Acked 5651 Write 8 Done New Done true
14241 4653 Acked 5652 Write 5 Done New Done true
14242 4654 Acked 5653 Write 4 Done New Done true
14243 4655 Acked 5654 Read 5 Done New Done true
14244 4656 Acked 5655 Write 2 Done New Done true
14245 4657 Acked 5656 Read 10 Done New Done true
14246 4658 Acked 5657 Read 8 Done New Done true
14247 4659 Acked 5658 Read 5 Done New Done true
14248 4660 Acked 5659 Write 5 Done New Done true
14249 4661 Acked 5660 Write 4 Done New Done true
14250 4662 Acked 5661 Write 8 Done New Done true
14251 4663 Acked 5662 Read 1 Done New Done true
14252 4664 Acked 5663 Write 8 Done New Done true
14253 4665 Acked 5664 Flush 0 Done New Done true
14254 4666 Acked 5665 Read 8 Done New Done true
14255 4667 Acked 5666 Flush 0 Done New Done true
14256 4668 Acked 5667 Read 5 Done New Done true
14257 4669 Acked 5668 Write 1 Done New Done true
14258 4670 Acked 5669 Read 8 Done New Done true
14259 4671 Acked 5670 Read 8 Done New Done true
14260 4672 Acked 5671 Read 8 Done New Done true
14261 4673 Acked 5672 Read 10 Done New Done true
14262 4674 Acked 5673 Write 3 Done New Done true
14263 4675 Acked 5674 Read 5 Done New Done true
14264 4676 Acked 5675 Write 9 Done New Done true
14265 4677 Acked 5676 Flush 0 Done New Done true
14266 4678 Acked 5677 Write 4 Done New Done true
14267 4679 Acked 5678 Write 5 Done New Done true
14268 4680 Acked 5679 Write 4 Done New Done true
14269 4681 Acked 5680 Read 8 Done New Done true
14270 4682 Acked 5681 Write 5 Done New Done true
14271 4683 Acked 5682 Flush 0 Done New Done true
14272 4684 Acked 5683 Write 8 Done New Done true
14273 4685 Acked 5684 Read 10 Done New Done true
14274 4686 Acked 5685 Read 9 Done New Done true
14275 4687 Acked 5686 Read 3 Done New Done true
14276 4688 Acked 5687 Read 8 Done New Done true
14277 4689 Acked 5688 Read 7 Done New Done true
14278 4690 Acked 5689 Write 4 Done New Done true
14279 4691 Acked 5690 Flush 0 Done New Done true
14280 4692 Acked 5691 Read 9 Done New Done true
14281 4693 Acked 5692 Flush 0 Done New Done true
14282 4694 Acked 5693 Read 8 Done New Done true
14283 4695 Acked 5694 Read 5 Done New Done true
14284 4696 Acked 5695 Write 3 Done New Done true
14285 4697 Acked 5696 Flush 0 Done New Done true
14286 4698 Acked 5697 Read 8 Done New Done true
14287 4699 Acked 5698 Read 2 Done New Done true
14288 4700 Acked 5699 Flush 0 Done New Done true
14289 4701 Acked 5700 Write 3 Done New Done true
14290 4702 Acked 5701 Write 5 Done New Done true
14291 4703 Acked 5702 Write 9 Done New Done true
14292 4704 Acked 5703 Read 3 Done New Done true
14293 4705 Acked 5704 Read 1 Done New Done true
14294 4706 Acked 5705 Write 5 Done New Done true
14295 4707 Acked 5706 Write 3 Done New Done true
14296 4708 Acked 5707 Write 8 Done New Done true
14297 4709 Acked 5708 Write 3 Done New Done true
14298 4710 Acked 5709 Read 3 Done New Done true
14299 4711 Acked 5710 Read 10 Done New Done true
14300 4712 Acked 5711 Flush 0 Done New Done true
14301 4713 Acked 5712 Read 3 Done New Done true
14302 4714 Acked 5713 Write 1 Done New Done true
14303 4715 Acked 5714 Flush 0 Done New Done true
14304 4716 Acked 5715 Read 8 Done New Done true
14305 4717 Acked 5716 Write 10 Done New Done true
14306 4718 Acked 5717 Write 3 Done New Done true
14307 4719 Acked 5718 Write 4 Done New Done true
14308 4720 Acked 5719 Read 3 Done New Done true
14309 4721 Acked 5720 Write 1 Done New Done true
14310 4722 Acked 5721 Read 9 Done New Done true
14311 4723 Acked 5722 Flush 0 Done New Done true
14312 4724 Acked 5723 Write 9 Done New Done true
14313 4725 Acked 5724 Write 3 Done New Done true
14314 4726 Acked 5725 Write 9 Done New Done true
14315 4727 Acked 5726 Flush 0 Done New Done true
14316 4728 Acked 5727 Flush 0 Done New Done true
14317 4729 Acked 5728 Read 7 Done New Done true
14318 4730 Acked 5729 Read 10 Done New Done true
14319 4731 Acked 5730 Write 5 Done New Done true
14320 4732 Acked 5731 Flush 0 Done New Done true
14321 4733 Acked 5732 Write 6 Done New Done true
14322 4734 Acked 5733 Write 10 Done New Done true
14323 4735 Acked 5734 Read 1 Done New Done true
14324 4736 Acked 5735 Write 3 Done New Done true
14325 4737 Acked 5736 Write 8 Done New Done true
14326 4738 Acked 5737 Read 9 Done New Done true
14327 4739 Acked 5738 Write 2 Done New Done true
14328 4740 Acked 5739 Read 8 Done New Done true
14329 4741 Acked 5740 Read 4 Done New Done true
14330 4742 Acked 5741 Write 8 Done New Done true
14331 4743 Acked 5742 Write 10 Done New Done true
14332 4744 Acked 5743 Write 4 Done New Done true
14333 4745 Acked 5744 Read 9 Done New Done true
14334 4746 Acked 5745 Read 4 Done New Done true
14335 4747 Acked 5746 Read 2 Done New Done true
14336 4748 Acked 5747 Flush 0 Done New Done true
14337 4749 Acked 5748 Flush 0 Done New Done true
14338 4750 Acked 5749 Write 4 Done New Done true
14339 4751 Acked 5750 Read 9 Done New Done true
14340 4752 Acked 5751 Read 6 Done New Done true
14341 4753 Acked 5752 Read 4 Done New Done true
14342 4754 Acked 5753 Write 3 Done New Done true
14343 4755 Acked 5754 Read 8 Done New Done true
14344 4756 Acked 5755 Read 4 Done New Done true
14345 4757 Acked 5756 Write 8 Done New Done true
14346 4758 Acked 5757 Read 1 Done New Done true
14347 4759 Acked 5758 Write 1 Done New Done true
14348 4760 Acked 5759 Read 8 Done New Done true
14349 4761 Acked 5760 Read 1 Done New Done true
14350 4762 Acked 5761 Write 7 Done New Done true
14351 4763 Acked 5762 Read 3 Done New Done true
14352 4764 Acked 5763 Write 4 Done New Done true
14353 4765 Acked 5764 Write 6 Done New Done true
14354 4766 Acked 5765 Read 7 Done New Done true
14355 4767 Acked 5766 Read 2 Done New Done true
14356 4768 Acked 5767 Read 7 Done New Done true
14357 4769 Acked 5768 Read 7 Done New Done true
14358 4770 Acked 5769 Read 6 Done New Done true
14359 4771 Acked 5770 Read 5 Done New Done true
14360 4772 Acked 5771 Read 7 Done New Done true
14361 4773 Acked 5772 Write 6 Done New Done true
14362 4774 Acked 5773 Read 8 Done New Done true
14363 4775 Acked 5774 Read 1 Done New Done true
14364 4776 Acked 5775 Read 2 Done New Done true
14365 4777 Acked 5776 Read 9 Done New Done true
14366 4778 Acked 5777 Read 8 Done New Done true
14367 4779 Acked 5778 Write 3 Done New Done true
14368 4780 Acked 5779 Read 6 Done New Done true
14369 4781 Acked 5780 Write 1 Done New Done true
14370 4782 Acked 5781 Flush 0 Done New Done true
14371 4783 Acked 5782 Read 6 Done New Done true
14372 4784 Acked 5783 Write 2 Done New Done true
14373 4785 Acked 5784 Read 7 Done New Done true
14374 4786 Acked 5785 Write 5 Done New Done true
14375 4787 Acked 5786 Read 4 Done New Done true
14376 4788 Acked 5787 Read 10 Done New Done true
14377 4789 Acked 5788 Write 7 Done New Done true
14378 4790 Acked 5789 Read 5 Done New Done true
14379 4791 Acked 5790 Flush 0 Done New Done true
14380 4792 Acked 5791 Read 2 Done New Done true
14381 4793 Acked 5792 Write 9 Done New Done true
14382 4794 Acked 5793 Read 5 Done New Done true
14383 4795 Acked 5794 Read 1 Done New Done true
14384 4796 Acked 5795 Write 4 Done New Done true
14385 4797 Acked 5796 Read 7 Done New Done true
14386 4798 Acked 5797 Write 9 Done New Done true
14387 4799 Acked 5798 Write 6 Done New Done true
14388 4800 Acked 5799 Read 9 Done New Done true
14389 4801 Acked 5800 Write 9 Done New Done true
14390 4802 Acked 5801 Write 9 Done New Done true
14391 4803 Acked 5802 Read 4 Done New Done true
14392 4804 Acked 5803 Write 1 Done New Done true
14393 4805 Acked 5804 Read 7 Done New Done true
14394 4806 Acked 5805 Read 3 Done New Done true
14395 4807 Acked 5806 Write 7 Done New Done true
14396 4808 Acked 5807 Flush 0 Done New Done true
14397 4809 Acked 5808 Read 10 Done New Done true
14398 4810 Acked 5809 Read 10 Done New Done true
14399 4811 Acked 5810 Read 5 Done New Done true
14400 4812 Acked 5811 Read 7 Done New Done true
14401 4813 Acked 5812 Write 4 Done New Done true
14402 4814 Acked 5813 Read 7 Done New Done true
14403 4815 Acked 5814 Write 10 Done New Done true
14404 4816 Acked 5815 Read 3 Done New Done true
14405 4817 Acked 5816 Read 4 Done New Done true
14406 4818 Acked 5817 Write 4 Done New Done true
14407 4819 Acked 5818 Write 3 Done New Done true
14408 4820 Acked 5819 Read 3 Done New Done true
14409 4821 Acked 5820 Read 3 Done New Done true
14410 4822 Acked 5821 Write 4 Done New Done true
14411 4823 Acked 5822 Write 1 Done New Done true
14412 4824 Acked 5823 Read 2 Done New Done true
14413 4825 Acked 5824 Flush 0 Done New Done true
14414 STATES DS:0 DS:1 DS:2 TOTAL
14415 New 0 302 0 302
14416 Sent 0 0 0 0
14417 Done 302 0 302 604
14418 Skipped 0 0 0 0
14419 Error 0 0 0 0
14420 Last Flush: 5824 5522 5824
14421 Downstairs last five completed: 5522 5521 5520 5519 5518
14422 Upstairs last five completed: 4825 4824 4823 4822 4821
14423 CLIENT: Up:0 ds:302 act:2
144242024-03-02T01:08:28.119ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144252024-03-02T01:08:28.120ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144262024-03-02T01:08:28.121ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144272024-03-02T01:08:28.121ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144282024-03-02T01:08:28.121ZINFOcrucible: send last flush ID to this DS: 5522 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144292024-03-02T01:08:28.121ZINFOcrucible: Replied this last flush ID: 5522 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144302024-03-02T01:08:28.121ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144312024-03-02T01:08:28.121ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144322024-03-02T01:08:28.121ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144332024-03-02T01:08:28.121ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14434 ----------------------------------------------------------------
14435 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14436 Downstairs last five completed: 5824 5823 5822 5821 5820
14437 Upstairs last five completed: 4825 4824 4823 4822 4821
14438 CLIENT: Up:0 ds:0 act:3
14439 Replay: All jobs finished, all DS active.
14440 Main task has work to do, go find it
14441 got dsc Stop(1)
14442 stop 1
14443 Mar 02 01:08:31.464 INFO request completed, latency_us: 353, response_code: 204, uri: /stop/cid/1, method: POST, req_id: [8820] Got stop action so:false kr:false
14444 9c25e439-9a12-46cd-8805-ca63db0bd4d7, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
144452024-03-02T01:08:31.464ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144462024-03-02T01:08:31.464ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144472024-03-02T01:08:31.464ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14448 Mar 02 01:08:31.464 INFO {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:08:31.464873546Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"1","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
14449 request completed, latency_us: 106, response_code: 200, uri: /state/cid/1, method: GET, req_id: c86086f7-cdf0-4deb-9b5c-4bd64efff1bc, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
144502024-03-02T01:08:31.464ZINFOcrucible: [1] client re-new 0 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144512024-03-02T01:08:31.465ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14452 [8820] Exited with: ExitStatus(unix_wait_status(9))
14453 [8820][1] reports Exit
144542024-03-02T01:08:32.466ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144552024-03-02T01:08:32.466ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144562024-03-02T01:08:32.466ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144572024-03-02T01:08:32.466ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144582024-03-02T01:08:32.466ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144592024-03-02T01:08:32.466ZINFOcrucible: [1] client re-new 0 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144602024-03-02T01:08:32.466ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144612024-03-02T01:08:33.467ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144622024-03-02T01:08:33.467ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144632024-03-02T01:08:33.467ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144642024-03-02T01:08:33.467ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144652024-03-02T01:08:33.467ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144662024-03-02T01:08:33.467ZINFOcrucible: [1] client re-new 0 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144672024-03-02T01:08:33.467ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144682024-03-02T01:08:34.468ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144692024-03-02T01:08:34.468ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144702024-03-02T01:08:34.468ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144712024-03-02T01:08:34.468ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144722024-03-02T01:08:34.468ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144732024-03-02T01:08:34.468ZINFOcrucible: [1] client re-new 0 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144742024-03-02T01:08:34.468ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14475 Mar 02 01:08:35.466 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8b6e5ebc-8825-43a7-8f0b-085bf134149a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
14476 001/300 Flush
14477 002/300 Read block 3065 len 24576
14478 003/300 Read block 1455 len 12288
144792024-03-02T01:08:35.469ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144802024-03-02T01:08:35.469ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144812024-03-02T01:08:35.469ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144822024-03-02T01:08:35.469ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144832024-03-02T01:08:35.469ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144842024-03-02T01:08:35.469ZINFOcrucible: [1] client re-new 3 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
144852024-03-02T01:08:35.469ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
14486 004/300 Read block 4776 len 28672
14487 005/300 Write block 2816 len 20480 data: 4 6 6 6 7
14488 006/300 Write block 296 len 24576 data: 7 7 5 3 3 3
14489 007/300 Write block 3157 len 16384 data: 5 5 5 5
14490 008/300 Write block 4764 len 40960 data: 7 6 6 7 6 4 5 4 4 5
14491 009/300 Write block 342 len 20480 data: 9 8 6 5 5
14492 010/300 Write block 1506 len 40960 data: 6 7 6 6 5 5 2 2 3 3
14493 011/300 Write block 2492 len 36864 data: 3 3 3 3 3 4 3 4 4
14494 012/300 Read block 2551 len 32768
14495 013/300 Read block 4177 len 28672
14496 014/300 Flush
14497 015/300 Read block 1280 len 36864
14498 016/300 Read block 1354 len 16384
14499 017/300 Write block 4639 len 28672 data: 4 4 4 5 8 8 7
14500 018/300 Read block 4893 len 24576
14501 019/300 Read block 1483 len 24576
14502 020/300 Read block 4337 len 24576
14503 021/300 Write block 2021 len 28672 data: 4 3 3 3 4 3 3
14504 022/300 Read block 577 len 24576
14505 023/300 Read block 3006 len 40960
14506 024/300 Read block 274 len 24576
14507 025/300 Write block 146 len 20480 data: 3 4 3 4 4
14508 026/300 Read block 3889 len 20480
14509 027/300 Write block 3926 len 20480 data: 7 7 4 4 5
14510 028/300 Read block 2151 len 40960
14511 029/300 Read block 3363 len 8192
14512 030/300 Write block 2661 len 16384 data: 5 4 4 4
14513 031/300 Write block 4978 len 4096 data: 5
14514 032/300 Write block 660 len 32768 data: 2 5 7 5 6 7 5 5
14515 033/300 Read block 2102 len 16384
14516 034/300 Read block 1800 len 24576
14517 035/300 Read block 620 len 24576
14518 036/300 Write block 4051 len 24576 data: 6 6 5 4 4 5
14519 037/300 Read block 1377 len 20480
14520 038/300 Read block 2558 len 8192
14521 039/300 Read block 1264 len 8192
14522 040/300 Read block 4718 len 4096
14523 041/300 Read block 4879 len 28672
14524 042/300 Read block 308 len 12288
14525 043/300 Write block 1998 len 40960 data: 3 2 2 3 3 3 3 3 3 4
14526 044/300 Read block 2148 len 4096
14527 045/300 Write block 3555 len 36864 data: 5 5 5 4 3 3 3 3 3
14528 046/300 Read block 3464 len 28672
14529 047/300 Write block 2859 len 4096 data: 3
14530 048/300 Read block 4814 len 32768
14531 049/300 Write block 2357 len 40960 data: 4 4 6 6 6 7 7 7 9 8
14532 050/300 Read block 2275 len 36864
14533 051/300 Read block 148 len 8192
14534 052/300 Write block 1776 len 32768 data: 2 3 3 3 3 4 4 3
14535 053/300 Read block 1768 len 4096
14536 054/300 Write block 1180 len 32768 data: 3 3 5 5 4 3 3 3
14537 055/300 Read block 4082 len 40960
14538 056/300 Write block 2190 len 8192 data: 2 3
14539 057/300 Read block 4020 len 32768
14540 058/300 Write block 132 len 28672 data: 3 2 2 2 2 3 3
14541 059/300 Read block 622 len 4096
14542 060/300 Write block 3996 len 40960 data: 5 5 5 4 4 4 4 4 4 3
14543 061/300 Read block 819 len 20480
14544 062/300 Read block 2994 len 4096
14545 063/300 Read block 1067 len 12288
14546 064/300 Write block 2948 len 16384 data: 4 4 5 4
14547 065/300 Read block 1476 len 40960
14548 066/300 Write block 1618 len 40960 data: 5 5 4 4 4 4 4 3 3 4
14549 067/300 Read block 1517 len 32768
14550 068/300 Write block 4473 len 4096 data: 2
14551 069/300 Read block 4359 len 40960
14552 070/300 Read block 3906 len 40960
14553 071/300 Flush
14554 072/300 Read block 2108 len 4096
14555 073/300 Read block 4001 len 16384
14556 074/300 Read block 171 len 28672
14557 075/300 Read block 4227 len 28672
14558 076/300 Read block 4011 len 4096
14559 077/300 Write block 4710 len 32768 data: 2 3 4 6 6 6 5 4
14560 078/300 Write block 1807 len 20480 data: 2 3 4 4 4
14561 079/300 Read block 1854 len 8192
14562 080/300 Read block 2055 len 4096
14563 081/300 Write block 4788 len 4096 data: 6
14564 082/300 Write block 3587 len 16384 data: 4 4 4 4
14565 083/300 Write block 4574 len 12288 data: 5 5 5
14566 084/300 Read block 694 len 4096
14567 085/300 Write block 1171 len 40960 data: 3 2 2 2 2 2 2 2 2 4
14568 086/300 Write block 778 len 12288 data: 3 3 2
14569 087/300 Flush
14570 088/300 Flush
14571 089/300 Write block 2553 len 4096 data: 4
14572 090/300 Read block 1152 len 28672
14573 091/300 Write block 3152 len 20480 data: 4 5 6 5 4
14574 092/300 Read block 3127 len 32768
14575 093/300 Write block 877 len 20480 data: 3 3 3 2 2
14576 094/300 Flush
14577 095/300 Read block 2125 len 12288
14578 096/300 Read block 1684 len 36864
14579 097/300 Read block 2672 len 24576
14580 098/300 Read block 603 len 8192
14581 099/300 Write block 41 len 40960 data: 4 4 6 6 6 2 2 3 5 5
14582 100/300 Read block 4248 len 28672
14583 101/300 Write block 3206 len 36864 data: 5 5 6 6 5 4 4 4 3
14584 102/300 Read block 229 len 28672
14585 103/300 Read block 2662 len 36864
14586 104/300 Read block 3998 len 20480
14587 105/300 Write block 2632 len 24576 data: 4 4 3 4 4 4
14588 106/300 Write block 3261 len 8192 data: 3 4
14589 107/300 Write block 2195 len 20480 data: 4 4 5 4 3
14590 108/300 Read block 1147 len 20480
14591 109/300 Read block 294 len 28672
14592 110/300 Flush
14593 111/300 Write block 329 len 36864 data: 5 5 4 6 6 10 10 11 11
14594 112/300 Read block 77 len 24576
14595 113/300 Write block 556 len 8192 data: 2 2
14596 114/300 Read block 776 len 20480
14597 115/300 Read block 1809 len 24576
14598 116/300 Read block 1948 len 36864
14599 117/300 Write block 588 len 16384 data: 5 4 4 4
14600 118/300 Read block 4574 len 16384
14601 119/300 Write block 2060 len 12288 data: 3 3 5
14602 120/300 Write block 4766 len 16384 data: 7 8 7 5
14603 121/300 Read block 3390 len 36864
14604 122/300 Write block 321 len 32768 data: 5 7 8 8 6 7 6 5
14605 123/300 Read block 4857 len 24576
14606 124/300 Read block 3713 len 32768
14607 125/300 Flush
14608 126/300 Write block 806 len 4096 data: 2
14609 127/300 Read block 2974 len 12288
14610 128/300 Read block 59 len 20480
14611 129/300 Flush
14612 130/300 Write block 623 len 16384 data: 6 6 6 5
14613 131/300 Write block 4144 len 36864 data: 3 3 2 2 2 2 2 2 3
14614 132/300 Read block 1302 len 12288
14615 133/300 Read block 4035 len 4096
14616 134/300 Read block 2783 len 28672
14617 135/300 Read block 4559 len 12288
14618 136/300 Flush
14619 137/300 Read block 2608 len 28672
14620 138/300 Write block 2086 len 40960 data: 3 4 5 6 7 7 5 5 6 6
14621 139/300 Flush
14622 140/300 Write block 4460 len 4096 data: 3
14623 141/300 Read block 3623 len 8192
14624 142/300 Read block 2719 len 32768
14625 143/300 Read block 4518 len 40960
14626 144/300 Flush
14627 145/300 Write block 2914 len 16384 data: 5 5 4 4
14628 146/300 Write block 4417 len 36864 data: 4 3 4 4 4 4 4 4 4
14629 147/300 Read block 1275 len 28672
14630 148/300 Read block 3144 len 8192
14631 149/300 Read block 1580 len 8192
14632 150/300 Flush
14633 151/300 Read block 3153 len 32768
14634 152/300 Read block 4361 len 8192
14635 153/300 Read block 483 len 40960
14636 154/300 Read block 2808 len 12288
14637 155/300 Write block 3120 len 8192 data: 3 3
14638 156/300 Write block 3615 len 24576 data: 5 3 4 4 4 3
14639 157/300 Read block 1073 len 32768
14640 158/300 Write block 3326 len 16384 data: 4 4 4 6
14641 159/300 Write block 528 len 32768 data: 5 6 7 7 9 7 7 8
14642 160/300 Read block 2865 len 16384
14643 161/300 Write block 2158 len 12288 data: 3 4 3
14644 162/300 Read block 3761 len 24576
14645 163/300 Write block 4131 len 40960 data: 2 2 4 4 4 5 5 5 5 4
14646 164/300 Read block 1131 len 28672
14647 165/300 Write block 3925 len 20480 data: 6 8 8 5 5
14648 166/300 Read block 1535 len 16384
14649 167/300 Write block 4425 len 8192 data: 5 3
14650 168/300 Write block 1991 len 24576 data: 4 5 4 4 4 3
14651 169/300 Write block 3789 len 16384 data: 5 4 2 2
14652 170/300 Read block 27 len 12288
14653 171/300 Read block 2686 len 20480
14654 172/300 Write block 3311 len 28672 data: 2 2 2 2 2 2 2
14655 173/300 Read block 1010 len 12288
14656 174/300 Read block 4877 len 20480
14657 175/300 Write block 3015 len 28672 data: 7 7 7 7 6 6 5
14658 176/300 Write block 1943 len 8192 data: 6 8
14659 177/300 Write block 1018 len 8192 data: 6 6
14660 178/300 Read block 2714 len 24576
14661 179/300 Write block 1725 len 40960 data: 4 5 5 6 4 4 5 5 6 6
14662 180/300 Flush
14663 181/300 Read block 2458 len 8192
14664 182/300 Flush
14665 183/300 Read block 1072 len 32768
14666 184/300 Read block 524 len 8192
14667 185/300 Write block 3334 len 24576 data: 5 4 5 5 6 4
14668 186/300 Read block 362 len 32768
14669 187/300 Read block 2977 len 24576
14670 188/300 Read block 4119 len 36864
14671 189/300 Write block 4682 len 16384 data: 7 8 7 6
14672 190/300 Write block 4033 len 16384 data: 6 3 3 4
14673 191/300 Write block 369 len 32768 data: 4 4 4 5 5 5 6 3
14674 192/300 Read block 493 len 32768
14675 193/300 Read block 4871 len 32768
14676 194/300 Write block 1937 len 12288 data: 6 5 6
14677 195/300 Write block 2460 len 24576 data: 5 4 3 3 2 3
14678 196/300 Read block 1408 len 8192
14679 197/300 Write block 1553 len 20480 data: 5 5 5 5 5
14680 198/300 Read block 2645 len 16384
14681 199/300 Read block 2043 len 20480
14682 200/300 Read block 4900 len 4096
14683 201/300 Write block 540 len 40960 data: 3 4 4 3 3 4 4 4 4 4
14684 202/300 Write block 1764 len 28672 data: 2 2 2 2 2 2 3
14685 203/300 Write block 3927 len 16384 data: 9 6 6 6
14686 204/300 Read block 1873 len 16384
14687 205/300 Write block 928 len 4096 data: 5
14688 206/300 Read block 3464 len 28672
14689 207/300 Read block 3278 len 8192
14690 208/300 Read block 2235 len 32768
14691 209/300 Read block 2789 len 32768
14692 210/300 Write block 1973 len 28672 data: 4 5 6 5 6 4 4
14693 211/300 Read block 2173 len 36864
14694 212/300 Read block 4494 len 16384
14695 213/300 Read block 514 len 32768
14696 214/300 Write block 3680 len 36864 data: 7 6 5 5 3 4 4 3 2
14697 215/300 Write block 3568 len 36864 data: 6 6 4 3 4 4 3 4 3
14698 216/300 Read block 1100 len 4096
14699 217/300 Write block 4332 len 16384 data: 5 5 6 5
14700 218/300 Read block 574 len 24576
14701 219/300 Read block 2990 len 4096
14702 220/300 Read block 3516 len 40960
14703 221/300 Read block 4941 len 36864
14704 222/300 Write block 4318 len 40960 data: 3 3 3 3 3 2 2 2 3 3
14705 223/300 Flush
14706 224/300 Read block 4861 len 40960
14707 225/300 Read block 852 len 12288
14708 226/300 Write block 4219 len 16384 data: 3 4 4 5
14709 227/300 Read block 3123 len 16384
14710 228/300 Flush
14711 229/300 Read block 393 len 36864
14712 230/300 Write block 4962 len 24576 data: 2 2 3 4 4 3
14713 231/300 Write block 4349 len 36864 data: 4 5 4 4 5 5 5 4 4
14714 232/300 Write block 4077 len 4096 data: 7
14715 233/300 Read block 761 len 8192
14716 234/300 Read block 1942 len 28672
14717 235/300 Read block 2621 len 40960
14718 236/300 Write block 2319 len 12288 data: 4 5 5
14719 237/300 Read block 2649 len 32768
14720 238/300 Read block 229 len 4096
14721 239/300 Read block 2278 len 8192
14722 240/300 Read block 3312 len 24576
14723 241/300 Write block 3225 len 12288 data: 6 3 5
14724 242/300 Read block 1575 len 40960
14725 243/300 Read block 4769 len 8192
14726 244/300 Read block 4849 len 28672
14727 245/300 Read block 4852 len 40960
14728 246/300 Read block 803 len 36864
14729 247/300 Read block 4123 len 12288
14730 248/300 Read block 3237 len 24576
14731 249/300 Write block 2563 len 16384 data: 6 6 4 4
14732 250/300 Read block 2034 len 24576
14733 251/300 Read block 1703 len 4096
14734 252/300 Write block 2035 len 16384 data: 3 2 3 4
14735 253/300 Read block 3661 len 28672
14736 254/300 Read block 658 len 8192
14737 255/300 Write block 3110 len 12288 data: 2 2 3
14738 256/300 Write block 3532 len 36864 data: 7 6 5 5 4 2 2 2 2
14739 257/300 Write block 703 len 8192 data: 7 6
14740 258/300 Read block 1071 len 32768
14741 259/300 Read block 2025 len 24576
14742 260/300 Read block 4306 len 16384
14743 261/300 Flush
14744 262/300 Read block 4372 len 40960
14745 263/300 Read block 4512 len 40960
14746 264/300 Write block 214 len 20480 data: 4 4 3 3 3
14747 265/300 Write block 740 len 40960 data: 4 4 4 6 5 6 7 7 5 6
14748 266/300 Read block 2785 len 32768
14749 267/300 Write block 858 len 8192 data: 5 6
14750 268/300 Read block 1159 len 4096
14751 269/300 Write block 4727 len 28672 data: 5 5 6 5 5 5 4
14752 270/300 Read block 2530 len 16384
14753 271/300 Flush
14754 272/300 Flush
14755 273/300 Read block 1102 len 20480
14756 274/300 Write block 792 len 28672 data: 4 3 3 3 3 3 3
14757 275/300 Read block 3764 len 20480
14758 276/300 Write block 230 len 40960 data: 5 4 4 3 4 4 4 3 3 5
14759 277/300 Read block 1532 len 40960
14760 278/300 Write block 2813 len 40960 data: 5 5 3 5 7 7 7 8 5 4
14761 279/300 Read block 233 len 36864
14762 280/300 Read block 3176 len 12288
14763 281/300 Flush
14764 282/300 Write block 2635 len 20480 data: 5 5 5 5 5
14765 283/300 Read block 3119 len 36864
14766 284/300 Read block 3979 len 40960
14767 285/300 Read block 2248 len 4096
14768 286/300 Write block 273 len 8192 data: 4 4
14769 287/300 Read block 755 len 8192
14770 288/300 Read block 1635 len 16384
14771 289/300 Write block 1578 len 12288 data: 5 5 7
14772 290/300 Read block 116 len 8192
14773 291/300 Read block 3135 len 32768
14774 292/300 Read block 2367 len 8192
14775 293/300 Read block 307 len 4096
14776 294/300 Read block 4261 len 16384
14777 295/300 Read block 2308 len 20480
14778 296/300 Write block 2118 len 8192 data: 5 6
14779 297/300 Read block 3147 len 32768
14780 298/300 Read block 3341 len 8192
14781 299/300 Write block 768 len 24576 data: 3 3 3 4 6 5
14782 300/300 Read block 4546 len 16384
14783 Main task has work to do, go find it
14784 got dsc Start(1)
14785 start 1
14786 Mar 02 01:08:35.816 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: 673f5240-c031-446e-83ef-f322ed369012, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
14787 Replay: started 1, returned:Ok(())
14788 ----------------------------------------------------------------
14789 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
14790 States: Active Offline Active
14791 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14792 4826 Acked 5825 Flush 0 Done New Done true
14793 4827 Acked 5826 Read 6 Done New Done true
14794 4828 Acked 5827 Read 3 Done New Done true
14795 4829 Acked 5828 Read 7 Done New Done false
14796 4830 Acked 5829 Write 5 Done New Done false
14797 4831 Acked 5830 Write 6 Done New Done false
14798 4832 Acked 5831 Write 4 Done New Done false
14799 4833 Acked 5832 Write 10 Done New Done false
14800 4834 Acked 5833 Write 5 Done New Done false
14801 4835 Acked 5834 Write 10 Done New Done false
14802 4836 Acked 5835 Write 9 Done New Done false
14803 4837 Acked 5836 Read 8 Done New Done false
14804 4838 Acked 5837 Read 7 Done New Done false
14805 4839 Acked 5838 Flush 0 Done New Done false
14806 4840 Acked 5839 Read 9 Done New Done false
14807 4841 Acked 5840 Read 4 Done New Done false
14808 4842 Acked 5841 Write 7 Done New Done false
14809 4843 Acked 5842 Read 6 Done New Done false
14810 4844 Acked 5843 Read 6 Done New Done false
14811 4845 Acked 5844 Read 6 Done New Done false
14812 4846 Acked 5845 Write 7 Done New Done false
14813 4847 Acked 5846 Read 6 Done New Done false
14814 4848 Acked 5847 Read 10 Done New Done false
14815 4849 Acked 5848 Read 6 Done New Done false
14816 4850 Acked 5849 Write 5 Done New Done false
14817 4851 Acked 5850 Read 5 Done New Done false
14818 4852 Acked 5851 Write 5 Done New Done false
14819 4853 Acked 5852 Read 10 Done New Done false
14820 4854 Acked 5853 Read 2 Done New Done false
14821 4855 Acked 5854 Write 4 Done New Done false
14822 4856 Acked 5855 Write 1 Done New Done false
14823 4857 Acked 5856 Write 8 Done New Done false
14824 4858 Acked 5857 Read 4 Done New Done false
14825 4859 Acked 5858 Read 6 Done New Done false
14826 4860 Acked 5859 Read 6 Done New Done false
14827 4861 Acked 5860 Write 6 Done New Done false
14828 4862 Acked 5861 Read 5 Done New Done false
14829 4863 Acked 5862 Read 2 Done New Done false
14830 4864 Acked 5863 Read 2 Done New Done false
14831 4865 Acked 5864 Read 1 Done New Done false
14832 4866 Acked 5865 Read 7 Done New Done false
14833 4867 Acked 5866 Read 3 Done New Done false
14834 4868 Acked 5867 Write 10 Done New Done false
14835 4869 Acked 5868 Read 1 Done New Done false
14836 4870 Acked 5869 Write 9 Done New Done false
14837 4871 Acked 5870 Read 7 Done New Done false
14838 4872 Acked 5871 Write 1 Done New Done false
14839 4873 Acked 5872 Read 8 Done New Done false
14840 4874 Acked 5873 Write 10 Done New Done false
14841 4875 Acked 5874 Read 9 Done New Done false
14842 4876 Acked 5875 Read 2 Done New Done false
14843 4877 Acked 5876 Write 8 Done New Done false
14844 4878 Acked 5877 Read 1 Done New Done false
14845 4879 Acked 5878 Write 8 Done New Done false
14846 4880 Acked 5879 Read 10 Done New Done false
14847 4881 Acked 5880 Write 2 Done New Done false
14848 4882 Acked 5881 Read 8 Done New Done false
14849 4883 Acked 5882 Write 7 Done New Done false
14850 4884 Acked 5883 Read 1 Done New Done false
14851 4885 Acked 5884 Write 10 Done New Done false
14852 4886 Acked 5885 Read 5 Done New Done false
14853 4887 Acked 5886 Read 1 Done New Done false
14854 4888 Acked 5887 Read 3 Done New Done false
14855 4889 Acked 5888 Write 4 Done New Done false
14856 4890 Acked 5889 Read 10 Done New Done false
14857 4891 Acked 5890 Write 10 Done New Done false
14858 4892 Acked 5891 Read 8 Done New Done false
14859 4893 Acked 5892 Write 1 Done New Done false
14860 4894 Acked 5893 Read 10 Done New Done false
14861 4895 Acked 5894 Read 10 Done New Done false
14862 4896 Acked 5895 Flush 0 Done New Done false
14863 4897 Acked 5896 Read 1 Done New Done false
14864 4898 Acked 5897 Read 4 Done New Done false
14865 4899 Acked 5898 Read 7 Done New Done false
14866 4900 Acked 5899 Read 7 Done New Done false
14867 4901 Acked 5900 Read 1 Done New Done false
14868 4902 Acked 5901 Write 8 Done New Done false
14869 4903 Acked 5902 Write 5 Done New Done false
14870 4904 Acked 5903 Read 2 Done New Done false
14871 4905 Acked 5904 Read 1 Done New Done false
14872 4906 Acked 5905 Write 1 Done New Done false
14873 4907 Acked 5906 Write 4 Done New Done false
14874 4908 Acked 5907 Write 3 Done New Done false
14875 4909 Acked 5908 Read 1 Done New Done false
14876 4910 Acked 5909 Write 10 Done New Done false
14877 4911 Acked 5910 Write 3 Done New Done false
14878 4912 Acked 5911 Flush 0 Done New Done false
14879 4913 Acked 5912 Flush 0 Done New Done false
14880 4914 Acked 5913 Write 1 Done New Done false
14881 4915 Acked 5914 Read 7 Done New Done false
14882 4916 Acked 5915 Write 5 Done New Done false
14883 4917 Acked 5916 Read 8 Done New Done false
14884 4918 Acked 5917 Write 5 Done New Done false
14885 4919 Acked 5918 Flush 0 Done New Done false
14886 4920 Acked 5919 Read 3 Done New Done false
14887 4921 Acked 5920 Read 9 Done New Done false
14888 4922 Acked 5921 Read 6 Done New Done false
14889 4923 Acked 5922 Read 2 Done New Done false
14890 4924 Acked 5923 Write 10 Done New Done false
14891 4925 Acked 5924 Read 7 Done New Done false
14892 4926 Acked 5925 Write 9 Done New Done false
14893 4927 Acked 5926 Read 7 Done New Done false
14894 4928 Acked 5927 Read 9 Done New Done false
14895 4929 Acked 5928 Read 5 Done New Done false
14896 4930 Acked 5929 Write 6 Done New Done false
14897 4931 Acked 5930 Write 2 Done New Done false
14898 4932 Acked 5931 Write 5 Done New Done false
14899 4933 Acked 5932 Read 5 Done New Done false
14900 4934 Acked 5933 Read 7 Done New Done false
14901 4935 Acked 5934 Flush 0 Done New Done false
14902 4936 Acked 5935 Write 9 Done New Done false
14903 4937 Acked 5936 Read 6 Done New Done false
14904 4938 Acked 5937 Write 2 Done New Done false
14905 4939 Acked 5938 Read 5 Done New Done false
14906 4940 Acked 5939 Read 6 Done New Done false
14907 4941 Acked 5940 Read 9 Done New Done false
14908 4942 Acked 5941 Write 4 Done New Done false
14909 4943 Acked 5942 Read 4 Done New Done false
14910 4944 Acked 5943 Write 3 Done New Done false
14911 4945 Acked 5944 Write 4 Done New Done false
14912 4946 Acked 5945 Read 9 Done New Done false
14913 4947 Acked 5946 Write 8 Done New Done false
14914 4948 Acked 5947 Read 6 Done New Done false
14915 4949 Acked 5948 Read 8 Done New Done false
14916 4950 Acked 5949 Flush 0 Done New Done false
14917 4951 Acked 5950 Write 1 Done New Done false
14918 4952 Acked 5951 Read 3 Done New Done false
14919 4953 Acked 5952 Read 5 Done New Done false
14920 4954 Acked 5953 Flush 0 Done New Done false
14921 4955 Acked 5954 Write 4 Done New Done false
14922 4956 Acked 5955 Write 9 Done New Done false
14923 4957 Acked 5956 Read 3 Done New Done false
14924 4958 Acked 5957 Read 1 Done New Done false
14925 4959 Acked 5958 Read 7 Done New Done false
14926 4960 Acked 5959 Read 3 Done New Done false
14927 4961 Acked 5960 Flush 0 Done New Done false
14928 4962 Acked 5961 Read 7 Done New Done false
14929 4963 Acked 5962 Write 10 Done New Done false
14930 4964 Acked 5963 Flush 0 Done New Done false
14931 4965 Acked 5964 Write 1 Done New Done false
14932 4966 Acked 5965 Read 2 Done New Done false
14933 4967 Acked 5966 Read 8 Done New Done false
14934 4968 Acked 5967 Read 10 Done New Done false
14935 4969 Acked 5968 Flush 0 Done New Done false
14936 4970 Acked 5969 Write 4 Done New Done false
14937 4971 Acked 5970 Write 9 Done New Done false
14938 4972 Acked 5971 Read 7 Done New Done false
14939 4973 Acked 5972 Read 2 Done New Done false
14940 4974 Acked 5973 Read 2 Done New Done false
14941 4975 Acked 5974 Flush 0 Done New Done false
14942 4976 Acked 5975 Read 8 Done New Done false
14943 4977 Acked 5976 Read 2 Done New Done false
14944 4978 Acked 5977 Read 10 Done New Done false
14945 4979 Acked 5978 Read 3 Done New Done false
14946 4980 Acked 5979 Write 2 Done New Done false
14947 4981 Acked 5980 Write 6 Done New Done false
14948 4982 Acked 5981 Read 8 Done New Done false
14949 4983 Acked 5982 Write 4 Done New Done false
14950 4984 Acked 5983 Write 8 Done New Done false
14951 4985 Acked 5984 Read 4 Done New Done false
14952 4986 Acked 5985 Write 3 Done New Done false
14953 4987 Acked 5986 Read 6 Done New Done false
14954 4988 Acked 5987 Write 10 Done New Done false
14955 4989 Acked 5988 Read 7 Done New Done false
14956 4990 Acked 5989 Write 5 Done New Done false
14957 4991 Acked 5990 Read 4 Done New Done false
14958 4992 Acked 5991 Write 2 Done New Done false
14959 4993 Acked 5992 Write 6 Done New Done false
14960 4994 Acked 5993 Write 4 Done New Done false
14961 4995 Acked 5994 Read 3 Done New Done false
14962 4996 Acked 5995 Read 5 Done New Done false
14963 4997 Acked 5996 Write 7 Done New Done false
14964 4998 Acked 5997 Read 3 Done New Done false
14965 4999 Acked 5998 Read 5 Done New Done false
14966 5000 Acked 5999 Write 7 Done New Done false
14967 5001 Acked 6000 Write 2 Done New Done false
14968 5002 Acked 6001 Write 2 Done New Done false
14969 5003 Acked 6002 Read 6 Done New Done false
14970 5004 Acked 6003 Write 10 Done New Done false
14971 5005 Acked 6004 Flush 0 Done New Done false
14972 5006 Acked 6005 Read 2 Done New Done false
14973 5007 Acked 6006 Flush 0 Done New Done false
14974 5008 Acked 6007 Read 8 Done New Done false
14975 5009 Acked 6008 Read 2 Done New Done false
14976 5010 Acked 6009 Write 6 Done New Done false
14977 5011 Acked 6010 Read 8 Done New Done false
14978 5012 Acked 6011 Read 6 Done New Done false
14979 5013 Acked 6012 Read 9 Done New Done false
14980 5014 Acked 6013 Write 4 Done New Done false
14981 5015 Acked 6014 Write 4 Done New Done false
14982 5016 Acked 6015 Write 8 Done New Done false
14983 5017 Acked 6016 Read 8 Done New Done false
14984 5018 Acked 6017 Read 8 Done New Done false
14985 5019 Acked 6018 Write 3 Done New Done false
14986 5020 Acked 6019 Write 6 Done New Done false
14987 5021 Acked 6020 Read 2 Done New Done false
14988 5022 Acked 6021 Write 5 Done New Done false
14989 5023 Acked 6022 Read 4 Done New Done false
14990 5024 Acked 6023 Read 5 Done New Done false
14991 5025 Acked 6024 Read 1 Done New Done false
14992 5026 Acked 6025 Write 10 Done New Done false
14993 5027 Acked 6026 Write 7 Done New Done false
14994 5028 Acked 6027 Write 4 Done New Done false
14995 5029 Acked 6028 Read 4 Done New Done false
14996 5030 Acked 6029 Write 1 Done New Done false
14997 5031 Acked 6030 Read 7 Done New Done false
14998 5032 Acked 6031 Read 2 Done New Done false
14999 5033 Acked 6032 Read 8 Done New Done false
15000 5034 Acked 6033 Read 8 Done New Done false
15001 5035 Acked 6034 Write 7 Done New Done false
15002 5036 Acked 6035 Read 9 Done New Done false
15003 5037 Acked 6036 Read 4 Done New Done false
15004 5038 Acked 6037 Read 8 Done New Done false
15005 5039 Acked 6038 Write 9 Done New Done false
15006 5040 Acked 6039 Write 9 Done New Done false
15007 5041 Acked 6040 Read 1 Done New Done false
15008 5042 Acked 6041 Write 4 Done New Done false
15009 5043 Acked 6042 Read 6 Done New Done false
15010 5044 Acked 6043 Read 1 Done New Done false
15011 5045 Acked 6044 Read 10 Done New Done false
15012 5046 Acked 6045 Read 9 Done New Done false
15013 5047 Acked 6046 Write 10 Done New Done false
15014 5048 Acked 6047 Flush 0 Done New Done false
15015 5049 Acked 6048 Read 10 Done New Done false
15016 5050 Acked 6049 Read 3 Done New Done false
15017 5051 Acked 6050 Write 4 Done New Done false
15018 5052 Acked 6051 Read 4 Done New Done false
15019 5053 Acked 6052 Flush 0 Done New Done false
15020 5054 Acked 6053 Read 9 Done New Done false
15021 5055 Acked 6054 Write 6 Done New Done false
15022 5056 Acked 6055 Write 9 Done New Done false
15023 5057 Acked 6056 Write 1 Done New Done false
15024 5058 Acked 6057 Read 2 Done New Done false
15025 5059 Acked 6058 Read 7 Done New Done false
15026 5060 Acked 6059 Read 10 Done New Done false
15027 5061 Acked 6060 Write 3 Done New Done false
15028 5062 Acked 6061 Read 8 Done New Done false
15029 5063 Acked 6062 Read 1 Done New Done false
15030 5064 Acked 6063 Read 2 Done New Done false
15031 5065 Acked 6064 Read 6 Done New Done false
15032 5066 Acked 6065 Write 3 Done New Done false
15033 5067 Acked 6066 Read 10 Done New Done false
15034 5068 Acked 6067 Read 2 Done New Done false
15035 5069 Acked 6068 Read 7 Done New Done false
15036 5070 Acked 6069 Read 10 Done New Done false
15037 5071 Acked 6070 Read 9 Done New Done false
15038 5072 Acked 6071 Read 3 Done New Done false
15039 5073 Acked 6072 Read 6 Done New Done false
15040 5074 Acked 6073 Write 4 Done New Done false
15041 5075 Acked 6074 Read 6 Done New Done false
15042 5076 Acked 6075 Read 1 Done New Done false
15043 5077 Acked 6076 Write 4 Done New Done false
15044 5078 Acked 6077 Read 7 Done New Done false
15045 5079 Acked 6078 Read 2 Done New Done false
15046 5080 Acked 6079 Write 3 Done New Done false
15047 5081 Acked 6080 Write 9 Done New Done false
15048 5082 Acked 6081 Write 2 Done New Done false
15049 5083 Acked 6082 Read 8 Done New Done false
15050 5084 Acked 6083 Read 6 Done New Done false
15051 5085 Acked 6084 Read 4 Done New Done false
15052 5086 Acked 6085 Flush 0 Done New Done false
15053 5087 Acked 6086 Read 10 Done New Done false
15054 5088 Acked 6087 Read 10 Done New Done false
15055 5089 Acked 6088 Write 5 Done New Done false
15056 5090 Acked 6089 Write 10 Done New Done false
15057 5091 Acked 6090 Read 8 Done New Done false
15058 5092 Acked 6091 Write 2 Done New Done false
15059 5093 Acked 6092 Read 1 Done New Done false
15060 5094 Acked 6093 Write 7 Done New Done false
15061 5095 Acked 6094 Read 4 Done New Done false
15062 5096 Acked 6095 Flush 0 Done New Done false
15063 5097 Acked 6096 Flush 0 Done New Done false
15064 5098 Acked 6097 Read 5 Done New Done false
15065 5099 Acked 6098 Write 7 Done New Done false
15066 5100 Acked 6099 Read 5 Done New Done false
15067 5101 Acked 6100 Write 10 Done New Done false
15068 5102 Acked 6101 Read 10 Done New Done false
15069 5103 Acked 6102 Write 10 Done New Done false
15070 5104 Acked 6103 Read 9 Done New Done false
15071 5105 Acked 6104 Read 3 Done New Done false
15072 5106 Acked 6105 Flush 0 Done New Done false
15073 5107 Acked 6106 Write 5 Done New Done false
15074 5108 Acked 6107 Read 9 Done New Done false
15075 5109 Acked 6108 Read 10 Done New Done false
15076 5110 Acked 6109 Read 1 Done New Done false
15077 5111 Acked 6110 Write 2 Done New Done false
15078 5112 Acked 6111 Read 2 Done New Done false
15079 5113 Acked 6112 Read 4 Done New Done false
15080 5114 Acked 6113 Write 3 Done New Done false
15081 5115 Acked 6114 Read 2 Done New Done false
15082 5116 Acked 6115 Read 8 Done New Done false
15083 5117 Acked 6116 Read 2 Done New Done false
15084 5118 Acked 6117 Read 1 Done New Done false
15085 5119 Acked 6118 Read 4 Done New Done false
15086 5120 Acked 6119 Read 5 Done New Done false
15087 5121 Acked 6120 Write 2 Done New Done false
15088 5122 Acked 6121 Read 8 Done New Done false
15089 5123 Acked 6122 Read 2 Done New Done false
15090 5124 Acked 6123 Write 6 Done New Done false
15091 5125 Acked 6124 Read 4 Done New Done false
15092 STATES DS:0 DS:1 DS:2 TOTAL
15093 New 0 300 0 300
15094 Sent 0 0 0 0
15095 Done 300 0 300 600
15096 Skipped 0 0 0 0
15097 Error 0 0 0 0
15098 Last Flush: 6105 5824 6105
15099 Downstairs last five completed: 5824 5823 5822 5821 5820
15100 Upstairs last five completed: 5125 5124 5123 5122 5121
15101 CLIENT: Up:0 ds:300 act:2
151022024-03-02T01:08:36.470ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151032024-03-02T01:08:36.470ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151042024-03-02T01:08:36.470ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151052024-03-02T01:08:36.470ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151062024-03-02T01:08:36.470ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151072024-03-02T01:08:36.471ZINFOcrucible: [1] client re-new 301 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151082024-03-02T01:08:36.471ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151092024-03-02T01:08:37.472ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151102024-03-02T01:08:37.472ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151112024-03-02T01:08:37.472ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151122024-03-02T01:08:37.472ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151132024-03-02T01:08:37.472ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151142024-03-02T01:08:37.472ZINFOcrucible: [1] client re-new 301 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151152024-03-02T01:08:37.472ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151162024-03-02T01:08:38.473ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151172024-03-02T01:08:38.473ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151182024-03-02T01:08:38.473ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151192024-03-02T01:08:38.473ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151202024-03-02T01:08:38.473ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151212024-03-02T01:08:38.473ZINFOcrucible: [1] client re-new 301 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151222024-03-02T01:08:38.473ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15123 [8820] got start action
15124 [8820] I am going to restart
15125 Starting downstairs at port 8820
15126 Make output file at "/tmp/dsc/downstairs-8820.txt"
15127 [8820][1] reports Starting
151282024-03-02T01:08:39.474ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151292024-03-02T01:08:39.474ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151302024-03-02T01:08:39.474ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151312024-03-02T01:08:39.474ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151322024-03-02T01:08:39.474ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151332024-03-02T01:08:39.474ZINFOcrucible: [1] client re-new 301 jobs since flush 5824 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
151342024-03-02T01:08:39.475ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15135 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1179)
15136 [8820][1] reports Running
15137 ----------------------------------------------------------------
15138 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15139 States: Active Offline Active
15140 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15141 4826 Acked 5825 Flush 0 Done New Done true
15142 4827 Acked 5826 Read 6 Done New Done true
15143 4828 Acked 5827 Read 3 Done New Done true
15144 4829 Acked 5828 Read 7 Done New Done true
15145 4830 Acked 5829 Write 5 Done New Done true
15146 4831 Acked 5830 Write 6 Done New Done true
15147 4832 Acked 5831 Write 4 Done New Done true
15148 4833 Acked 5832 Write 10 Done New Done true
15149 4834 Acked 5833 Write 5 Done New Done true
15150 4835 Acked 5834 Write 10 Done New Done true
15151 4836 Acked 5835 Write 9 Done New Done true
15152 4837 Acked 5836 Read 8 Done New Done true
15153 4838 Acked 5837 Read 7 Done New Done true
15154 4839 Acked 5838 Flush 0 Done New Done true
15155 4840 Acked 5839 Read 9 Done New Done true
15156 4841 Acked 5840 Read 4 Done New Done true
15157 4842 Acked 5841 Write 7 Done New Done true
15158 4843 Acked 5842 Read 6 Done New Done true
15159 4844 Acked 5843 Read 6 Done New Done true
15160 4845 Acked 5844 Read 6 Done New Done true
15161 4846 Acked 5845 Write 7 Done New Done true
15162 4847 Acked 5846 Read 6 Done New Done true
15163 4848 Acked 5847 Read 10 Done New Done true
15164 4849 Acked 5848 Read 6 Done New Done true
15165 4850 Acked 5849 Write 5 Done New Done true
15166 4851 Acked 5850 Read 5 Done New Done true
15167 4852 Acked 5851 Write 5 Done New Done true
15168 4853 Acked 5852 Read 10 Done New Done true
15169 4854 Acked 5853 Read 2 Done New Done true
15170 4855 Acked 5854 Write 4 Done New Done true
15171 4856 Acked 5855 Write 1 Done New Done true
15172 4857 Acked 5856 Write 8 Done New Done true
15173 4858 Acked 5857 Read 4 Done New Done true
15174 4859 Acked 5858 Read 6 Done New Done true
15175 4860 Acked 5859 Read 6 Done New Done true
15176 4861 Acked 5860 Write 6 Done New Done true
15177 4862 Acked 5861 Read 5 Done New Done true
15178 4863 Acked 5862 Read 2 Done New Done true
15179 4864 Acked 5863 Read 2 Done New Done true
15180 4865 Acked 5864 Read 1 Done New Done true
15181 4866 Acked 5865 Read 7 Done New Done true
15182 4867 Acked 5866 Read 3 Done New Done true
15183 4868 Acked 5867 Write 10 Done New Done true
15184 4869 Acked 5868 Read 1 Done New Done true
15185 4870 Acked 5869 Write 9 Done New Done true
15186 4871 Acked 5870 Read 7 Done New Done true
15187 4872 Acked 5871 Write 1 Done New Done true
15188 4873 Acked 5872 Read 8 Done New Done true
15189 4874 Acked 5873 Write 10 Done New Done true
15190 4875 Acked 5874 Read 9 Done New Done true
15191 4876 Acked 5875 Read 2 Done New Done true
15192 4877 Acked 5876 Write 8 Done New Done true
15193 4878 Acked 5877 Read 1 Done New Done true
15194 4879 Acked 5878 Write 8 Done New Done true
15195 4880 Acked 5879 Read 10 Done New Done true
15196 4881 Acked 5880 Write 2 Done New Done true
15197 4882 Acked 5881 Read 8 Done New Done true
15198 4883 Acked 5882 Write 7 Done New Done true
15199 4884 Acked 5883 Read 1 Done New Done true
15200 4885 Acked 5884 Write 10 Done New Done true
15201 4886 Acked 5885 Read 5 Done New Done true
15202 4887 Acked 5886 Read 1 Done New Done true
15203 4888 Acked 5887 Read 3 Done New Done true
15204 4889 Acked 5888 Write 4 Done New Done true
15205 4890 Acked 5889 Read 10 Done New Done true
15206 4891 Acked 5890 Write 10 Done New Done true
15207 4892 Acked 5891 Read 8 Done New Done true
15208 4893 Acked 5892 Write 1 Done New Done true
15209 4894 Acked 5893 Read 10 Done New Done true
15210 4895 Acked 5894 Read 10 Done New Done true
15211 4896 Acked 5895 Flush 0 Done New Done true
15212 4897 Acked 5896 Read 1 Done New Done true
15213 4898 Acked 5897 Read 4 Done New Done true
15214 4899 Acked 5898 Read 7 Done New Done true
15215 4900 Acked 5899 Read 7 Done New Done true
15216 4901 Acked 5900 Read 1 Done New Done true
15217 4902 Acked 5901 Write 8 Done New Done true
15218 4903 Acked 5902 Write 5 Done New Done true
15219 4904 Acked 5903 Read 2 Done New Done true
15220 4905 Acked 5904 Read 1 Done New Done true
15221 4906 Acked 5905 Write 1 Done New Done true
15222 4907 Acked 5906 Write 4 Done New Done true
15223 4908 Acked 5907 Write 3 Done New Done true
15224 4909 Acked 5908 Read 1 Done New Done true
15225 4910 Acked 5909 Write 10 Done New Done true
15226 4911 Acked 5910 Write 3 Done New Done true
15227 4912 Acked 5911 Flush 0 Done New Done true
15228 4913 Acked 5912 Flush 0 Done New Done true
15229 4914 Acked 5913 Write 1 Done New Done true
15230 4915 Acked 5914 Read 7 Done New Done true
15231 4916 Acked 5915 Write 5 Done New Done true
15232 4917 Acked 5916 Read 8 Done New Done true
15233 4918 Acked 5917 Write 5 Done New Done true
15234 4919 Acked 5918 Flush 0 Done New Done true
15235 4920 Acked 5919 Read 3 Done New Done true
15236 4921 Acked 5920 Read 9 Done New Done true
15237 4922 Acked 5921 Read 6 Done New Done true
15238 4923 Acked 5922 Read 2 Done New Done true
15239 4924 Acked 5923 Write 10 Done New Done true
15240 4925 Acked 5924 Read 7 Done New Done true
15241 4926 Acked 5925 Write 9 Done New Done true
15242 4927 Acked 5926 Read 7 Done New Done true
15243 4928 Acked 5927 Read 9 Done New Done true
15244 4929 Acked 5928 Read 5 Done New Done true
15245 4930 Acked 5929 Write 6 Done New Done true
15246 4931 Acked 5930 Write 2 Done New Done true
15247 4932 Acked 5931 Write 5 Done New Done true
15248 4933 Acked 5932 Read 5 Done New Done true
15249 4934 Acked 5933 Read 7 Done New Done true
15250 4935 Acked 5934 Flush 0 Done New Done true
15251 4936 Acked 5935 Write 9 Done New Done true
15252 4937 Acked 5936 Read 6 Done New Done true
15253 4938 Acked 5937 Write 2 Done New Done true
15254 4939 Acked 5938 Read 5 Done New Done true
15255 4940 Acked 5939 Read 6 Done New Done true
15256 4941 Acked 5940 Read 9 Done New Done true
15257 4942 Acked 5941 Write 4 Done New Done true
15258 4943 Acked 5942 Read 4 Done New Done true
15259 4944 Acked 5943 Write 3 Done New Done true
15260 4945 Acked 5944 Write 4 Done New Done true
15261 4946 Acked 5945 Read 9 Done New Done true
15262 4947 Acked 5946 Write 8 Done New Done true
15263 4948 Acked 5947 Read 6 Done New Done true
15264 4949 Acked 5948 Read 8 Done New Done true
15265 4950 Acked 5949 Flush 0 Done New Done true
15266 4951 Acked 5950 Write 1 Done New Done true
15267 4952 Acked 5951 Read 3 Done New Done true
15268 4953 Acked 5952 Read 5 Done New Done true
15269 4954 Acked 5953 Flush 0 Done New Done true
15270 4955 Acked 5954 Write 4 Done New Done true
15271 4956 Acked 5955 Write 9 Done New Done true
15272 4957 Acked 5956 Read 3 Done New Done true
15273 4958 Acked 5957 Read 1 Done New Done true
15274 4959 Acked 5958 Read 7 Done New Done true
15275 4960 Acked 5959 Read 3 Done New Done true
15276 4961 Acked 5960 Flush 0 Done New Done true
15277 4962 Acked 5961 Read 7 Done New Done true
15278 4963 Acked 5962 Write 10 Done New Done true
15279 4964 Acked 5963 Flush 0 Done New Done true
15280 4965 Acked 5964 Write 1 Done New Done true
15281 4966 Acked 5965 Read 2 Done New Done true
15282 4967 Acked 5966 Read 8 Done New Done true
15283 4968 Acked 5967 Read 10 Done New Done true
15284 4969 Acked 5968 Flush 0 Done New Done true
15285 4970 Acked 5969 Write 4 Done New Done true
15286 4971 Acked 5970 Write 9 Done New Done true
15287 4972 Acked 5971 Read 7 Done New Done true
15288 4973 Acked 5972 Read 2 Done New Done true
15289 4974 Acked 5973 Read 2 Done New Done true
15290 4975 Acked 5974 Flush 0 Done New Done true
15291 4976 Acked 5975 Read 8 Done New Done true
15292 4977 Acked 5976 Read 2 Done New Done true
15293 4978 Acked 5977 Read 10 Done New Done true
15294 4979 Acked 5978 Read 3 Done New Done true
15295 4980 Acked 5979 Write 2 Done New Done true
15296 4981 Acked 5980 Write 6 Done New Done true
15297 4982 Acked 5981 Read 8 Done New Done true
15298 4983 Acked 5982 Write 4 Done New Done true
15299 4984 Acked 5983 Write 8 Done New Done true
15300 4985 Acked 5984 Read 4 Done New Done true
15301 4986 Acked 5985 Write 3 Done New Done true
15302 4987 Acked 5986 Read 6 Done New Done true
15303 4988 Acked 5987 Write 10 Done New Done true
15304 4989 Acked 5988 Read 7 Done New Done true
15305 4990 Acked 5989 Write 5 Done New Done true
15306 4991 Acked 5990 Read 4 Done New Done true
15307 4992 Acked 5991 Write 2 Done New Done true
15308 4993 Acked 5992 Write 6 Done New Done true
15309 4994 Acked 5993 Write 4 Done New Done true
15310 4995 Acked 5994 Read 3 Done New Done true
15311 4996 Acked 5995 Read 5 Done New Done true
15312 4997 Acked 5996 Write 7 Done New Done true
15313 4998 Acked 5997 Read 3 Done New Done true
15314 4999 Acked 5998 Read 5 Done New Done true
15315 5000 Acked 5999 Write 7 Done New Done true
15316 5001 Acked 6000 Write 2 Done New Done true
15317 5002 Acked 6001 Write 2 Done New Done true
15318 5003 Acked 6002 Read 6 Done New Done true
15319 5004 Acked 6003 Write 10 Done New Done true
15320 5005 Acked 6004 Flush 0 Done New Done true
15321 5006 Acked 6005 Read 2 Done New Done true
15322 5007 Acked 6006 Flush 0 Done New Done true
15323 5008 Acked 6007 Read 8 Done New Done true
15324 5009 Acked 6008 Read 2 Done New Done true
15325 5010 Acked 6009 Write 6 Done New Done true
15326 5011 Acked 6010 Read 8 Done New Done true
15327 5012 Acked 6011 Read 6 Done New Done true
15328 5013 Acked 6012 Read 9 Done New Done true
15329 5014 Acked 6013 Write 4 Done New Done true
15330 5015 Acked 6014 Write 4 Done New Done true
15331 5016 Acked 6015 Write 8 Done New Done true
15332 5017 Acked 6016 Read 8 Done New Done true
15333 5018 Acked 6017 Read 8 Done New Done true
15334 5019 Acked 6018 Write 3 Done New Done true
15335 5020 Acked 6019 Write 6 Done New Done true
15336 5021 Acked 6020 Read 2 Done New Done true
15337 5022 Acked 6021 Write 5 Done New Done true
15338 5023 Acked 6022 Read 4 Done New Done true
15339 5024 Acked 6023 Read 5 Done New Done true
15340 5025 Acked 6024 Read 1 Done New Done true
15341 5026 Acked 6025 Write 10 Done New Done true
15342 5027 Acked 6026 Write 7 Done New Done true
15343 5028 Acked 6027 Write 4 Done New Done true
15344 5029 Acked 6028 Read 4 Done New Done true
15345 5030 Acked 6029 Write 1 Done New Done true
15346 5031 Acked 6030 Read 7 Done New Done true
15347 5032 Acked 6031 Read 2 Done New Done true
15348 5033 Acked 6032 Read 8 Done New Done true
15349 5034 Acked 6033 Read 8 Done New Done true
15350 5035 Acked 6034 Write 7 Done New Done true
15351 5036 Acked 6035 Read 9 Done New Done true
15352 5037 Acked 6036 Read 4 Done New Done true
15353 5038 Acked 6037 Read 8 Done New Done true
15354 5039 Acked 6038 Write 9 Done New Done true
15355 5040 Acked 6039 Write 9 Done New Done true
15356 5041 Acked 6040 Read 1 Done New Done true
15357 5042 Acked 6041 Write 4 Done New Done true
15358 5043 Acked 6042 Read 6 Done New Done true
15359 5044 Acked 6043 Read 1 Done New Done true
15360 5045 Acked 6044 Read 10 Done New Done true
15361 5046 Acked 6045 Read 9 Done New Done true
15362 5047 Acked 6046 Write 10 Done New Done true
15363 5048 Acked 6047 Flush 0 Done New Done true
15364 5049 Acked 6048 Read 10 Done New Done true
15365 5050 Acked 6049 Read 3 Done New Done true
15366 5051 Acked 6050 Write 4 Done New Done true
15367 5052 Acked 6051 Read 4 Done New Done true
15368 5053 Acked 6052 Flush 0 Done New Done true
15369 5054 Acked 6053 Read 9 Done New Done true
15370 5055 Acked 6054 Write 6 Done New Done true
15371 5056 Acked 6055 Write 9 Done New Done true
15372 5057 Acked 6056 Write 1 Done New Done true
15373 5058 Acked 6057 Read 2 Done New Done true
15374 5059 Acked 6058 Read 7 Done New Done true
15375 5060 Acked 6059 Read 10 Done New Done true
15376 5061 Acked 6060 Write 3 Done New Done true
15377 5062 Acked 6061 Read 8 Done New Done true
15378 5063 Acked 6062 Read 1 Done New Done true
15379 5064 Acked 6063 Read 2 Done New Done true
15380 5065 Acked 6064 Read 6 Done New Done true
15381 5066 Acked 6065 Write 3 Done New Done true
15382 5067 Acked 6066 Read 10 Done New Done true
15383 5068 Acked 6067 Read 2 Done New Done true
15384 5069 Acked 6068 Read 7 Done New Done true
15385 5070 Acked 6069 Read 10 Done New Done true
15386 5071 Acked 6070 Read 9 Done New Done true
15387 5072 Acked 6071 Read 3 Done New Done true
15388 5073 Acked 6072 Read 6 Done New Done true
15389 5074 Acked 6073 Write 4 Done New Done true
15390 5075 Acked 6074 Read 6 Done New Done true
15391 5076 Acked 6075 Read 1 Done New Done true
15392 5077 Acked 6076 Write 4 Done New Done true
15393 5078 Acked 6077 Read 7 Done New Done true
15394 5079 Acked 6078 Read 2 Done New Done true
15395 5080 Acked 6079 Write 3 Done New Done true
15396 5081 Acked 6080 Write 9 Done New Done true
15397 5082 Acked 6081 Write 2 Done New Done true
15398 5083 Acked 6082 Read 8 Done New Done true
15399 5084 Acked 6083 Read 6 Done New Done true
15400 5085 Acked 6084 Read 4 Done New Done true
15401 5086 Acked 6085 Flush 0 Done New Done true
15402 5087 Acked 6086 Read 10 Done New Done true
15403 5088 Acked 6087 Read 10 Done New Done true
15404 5089 Acked 6088 Write 5 Done New Done true
15405 5090 Acked 6089 Write 10 Done New Done true
15406 5091 Acked 6090 Read 8 Done New Done true
15407 5092 Acked 6091 Write 2 Done New Done true
15408 5093 Acked 6092 Read 1 Done New Done true
15409 5094 Acked 6093 Write 7 Done New Done true
15410 5095 Acked 6094 Read 4 Done New Done true
15411 5096 Acked 6095 Flush 0 Done New Done true
15412 5097 Acked 6096 Flush 0 Done New Done true
15413 5098 Acked 6097 Read 5 Done New Done true
15414 5099 Acked 6098 Write 7 Done New Done true
15415 5100 Acked 6099 Read 5 Done New Done true
15416 5101 Acked 6100 Write 10 Done New Done true
15417 5102 Acked 6101 Read 10 Done New Done true
15418 5103 Acked 6102 Write 10 Done New Done true
15419 5104 Acked 6103 Read 9 Done New Done true
15420 5105 Acked 6104 Read 3 Done New Done true
15421 5106 Acked 6105 Flush 0 Done New Done true
15422 5107 Acked 6106 Write 5 Done New Done true
15423 5108 Acked 6107 Read 9 Done New Done true
15424 5109 Acked 6108 Read 10 Done New Done true
15425 5110 Acked 6109 Read 1 Done New Done true
15426 5111 Acked 6110 Write 2 Done New Done true
15427 5112 Acked 6111 Read 2 Done New Done true
15428 5113 Acked 6112 Read 4 Done New Done true
15429 5114 Acked 6113 Write 3 Done New Done true
15430 5115 Acked 6114 Read 2 Done New Done true
15431 5116 Acked 6115 Read 8 Done New Done true
15432 5117 Acked 6116 Read 2 Done New Done true
15433 5118 Acked 6117 Read 1 Done New Done true
15434 5119 Acked 6118 Read 4 Done New Done true
15435 5120 Acked 6119 Read 5 Done New Done true
15436 5121 Acked 6120 Write 2 Done New Done true
15437 5122 Acked 6121 Read 8 Done New Done true
15438 5123 Acked 6122 Read 2 Done New Done true
15439 5124 Acked 6123 Write 6 Done New Done true
15440 5125 Acked 6124 Read 4 Done New Done true
15441 5126 Acked 6125 Flush 0 Done New Done true
15442 STATES DS:0 DS:1 DS:2 TOTAL
15443 New 0 301 0 301
15444 Sent 0 0 0 0
15445 Done 301 0 301 602
15446 Skipped 0 0 0 0
15447 Error 0 0 0 0
15448 Last Flush: 6125 5824 6125
15449 Downstairs last five completed: 5824 5823 5822 5821 5820
15450 Upstairs last five completed: 5126 5125 5124 5123 5122
15451 CLIENT: Up:0 ds:301 act:2
154522024-03-02T01:08:40.476ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154532024-03-02T01:08:40.476ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154542024-03-02T01:08:40.478ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154552024-03-02T01:08:40.478ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154562024-03-02T01:08:40.478ZINFOcrucible: send last flush ID to this DS: 5824 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154572024-03-02T01:08:40.478ZINFOcrucible: Replied this last flush ID: 5824 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154582024-03-02T01:08:40.478ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154592024-03-02T01:08:40.478ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154602024-03-02T01:08:40.478ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154612024-03-02T01:08:40.478ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15462 ----------------------------------------------------------------
15463 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15464 Downstairs last five completed: 6125 6124 6123 6122 6121
15465 Upstairs last five completed: 5126 5125 5124 5123 5122
15466 CLIENT: Up:0 ds:0 act:3
15467 Replay: All jobs finished, all DS active.
15468 Main task has work to do, go find it
15469 got dsc Stop(2)
15470 stop 2
15471 Mar 02 01:08:43.822 INFO request completed, latency_us: 324[8830] Got stop action so:false kr:false
15472 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: 39930144-364d-4188-8556-c35a412d6067, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
154732024-03-02T01:08:43.823ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154742024-03-02T01:08:43.823ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154752024-03-02T01:08:43.823ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154762024-03-02T01:08:43.823ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154772024-03-02T01:08:43.823ZINFOcrucible: [2] client re-new 0 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15478 Mar 02 01:08:43.823 INFO request completed, latency_us: 175, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5310204b-b0c4-4b9c-a995-d5913b8fe2d5, remote_addr: 127.0.0.1:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:08:43.82372002Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
15479 40735, local_addr: 127.0.0.1:9998
15480 [8830] Exited with: ExitStatus(unix_wait_status(9))
15481 [8830][2] reports Exit
154822024-03-02T01:08:44.825ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154832024-03-02T01:08:44.825ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154842024-03-02T01:08:44.825ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154852024-03-02T01:08:44.825ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154862024-03-02T01:08:44.825ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154872024-03-02T01:08:44.825ZINFOcrucible: [2] client re-new 0 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154882024-03-02T01:08:44.825ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154892024-03-02T01:08:45.826ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154902024-03-02T01:08:45.826ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154912024-03-02T01:08:45.826ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154922024-03-02T01:08:45.826ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154932024-03-02T01:08:45.826ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154942024-03-02T01:08:45.826ZINFOcrucible: [2] client re-new 0 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154952024-03-02T01:08:45.826ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154962024-03-02T01:08:46.827ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154972024-03-02T01:08:46.828ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154982024-03-02T01:08:46.828ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
154992024-03-02T01:08:46.828ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155002024-03-02T01:08:46.828ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155012024-03-02T01:08:46.828ZINFOcrucible: [2] client re-new 0 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155022024-03-02T01:08:46.828ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15503 Mar 02 01:08:47.825 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6f60bebb-7062-4de3-b4b2-e638753ac88a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
15504 001/300 Write block 3537 len 12288 data: 3 3 3
15505 002/300 Write block 3258 len 16384 data: 2 2 2 4
15506 003/300 Read block 1936 len 32768
155072024-03-02T01:08:47.828ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155082024-03-02T01:08:47.828ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155092024-03-02T01:08:47.828ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155102024-03-02T01:08:47.828ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155112024-03-02T01:08:47.828ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155122024-03-02T01:08:47.828ZINFOcrucible: [2] client re-new 3 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
155132024-03-02T01:08:47.828ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
15514 004/300 Read block 4988 len 24576
15515 005/300 Write block 965 len 4096 data: 4
15516 006/300 Write block 1900 len 24576 data: 4 3 5 4 3 3
15517 007/300 Write block 2286 len 12288 data: 5 4 4
15518 008/300 Read block 4783 len 24576
15519 009/300 Read block 1445 len 32768
15520 010/300 Read block 62 len 16384
15521 011/300 Read block 4579 len 32768
15522 012/300 Read block 2833 len 24576
15523 013/300 Read block 3470 len 40960
15524 014/300 Write block 3461 len 16384 data: 4 5 5 6
15525 015/300 Read block 457 len 20480
15526 016/300 Read block 126 len 32768
15527 017/300 Write block 271 len 20480 data: 8 6 5 5 4
15528 018/300 Read block 2299 len 36864
15529 019/300 Read block 1260 len 20480
15530 020/300 Write block 2036 len 4096 data: 3
15531 021/300 Read block 1557 len 32768
15532 022/300 Write block 2850 len 20480 data: 4 3 3 2 3
15533 023/300 Read block 3206 len 16384
15534 024/300 Read block 4805 len 8192
15535 025/300 Read block 2184 len 28672
15536 026/300 Flush
15537 027/300 Flush
15538 028/300 Read block 274 len 36864
15539 029/300 Flush
15540 030/300 Write block 2060 len 36864 data: 4 4 6 5 4 6 7 7 7
15541 031/300 Write block 1132 len 12288 data: 2 2 2
15542 032/300 Write block 786 len 20480 data: 3 3 2 2 3
15543 033/300 Read block 1582 len 20480
15544 034/300 Write block 1695 len 4096 data: 7
15545 035/300 Write block 2407 len 20480 data: 5 4 4 4 4
15546 036/300 Write block 4591 len 16384 data: 4 4 4 4
15547 037/300 Flush
15548 038/300 Write block 1564 len 4096 data: 3
15549 039/300 Flush
15550 040/300 Read block 965 len 4096
15551 041/300 Read block 2974 len 40960
15552 042/300 Read block 1206 len 36864
15553 043/300 Read block 1050 len 40960
15554 044/300 Flush
15555 045/300 Read block 4255 len 8192
15556 046/300 Read block 1763 len 36864
15557 047/300 Read block 3683 len 16384
15558 048/300 Write block 614 len 28672 data: 8 8 7 6 5 5 6
15559 049/300 Read block 2521 len 12288
15560 050/300 Read block 4014 len 32768
15561 051/300 Write block 944 len 20480 data: 8 8 5 5 5
15562 052/300 Write block 3778 len 12288 data: 6 8 10
15563 053/300 Flush
15564 054/300 Read block 714 len 20480
15565 055/300 Read block 1681 len 28672
15566 056/300 Flush
15567 057/300 Write block 728 len 24576 data: 3 4 4 3 4 6
15568 058/300 Write block 1072 len 20480 data: 3 3 3 3 3
15569 059/300 Read block 1423 len 28672
15570 060/300 Flush
15571 061/300 Write block 2726 len 40960 data: 4 4 5 3 2 2 3 3 5 5
15572 062/300 Read block 4436 len 24576
15573 063/300 Read block 1242 len 24576
15574 064/300 Write block 3619 len 36864 data: 5 4 3 3 4 4 2 4 4
15575 065/300 Read block 1626 len 20480
15576 066/300 Read block 3196 len 4096
15577 067/300 Write block 1886 len 36864 data: 3 2 2 3 4 4 4 5 5
15578 068/300 Write block 2359 len 12288 data: 7 7 7
15579 069/300 Read block 3904 len 40960
15580 070/300 Read block 1687 len 8192
15581 071/300 Write block 4330 len 40960 data: 4 4 6 6 7 6 5 6 5 4
15582 072/300 Write block 4124 len 8192 data: 5 2
15583 073/300 Flush
15584 074/300 Read block 1967 len 28672
15585 075/300 Write block 276 len 28672 data: 5 5 5 5 5 5 6
15586 076/300 Read block 1945 len 8192
15587 077/300 Read block 4716 len 40960
15588 078/300 Write block 3387 len 8192 data: 4 4
15589 079/300 Write block 757 len 12288 data: 5 5 4
15590 080/300 Write block 3044 len 4096 data: 2
15591 081/300 Write block 2807 len 32768 data: 3 4 5 6 6 5 6 6
15592 082/300 Write block 2431 len 8192 data: 7 6
15593 083/300 Read block 1474 len 24576
15594 084/300 Write block 4755 len 40960 data: 5 5 5 5 6 5 5 7 7 8
15595 085/300 Write block 1252 len 36864 data: 3 3 3 2 3 3 3 3 4
15596 086/300 Write block 3626 len 20480 data: 5 5 4 4 4
15597 087/300 Write block 4023 len 36864 data: 3 4 4 5 5 6 6 6 6
15598 088/300 Write block 2127 len 4096 data: 2
15599 089/300 Flush
15600 090/300 Read block 964 len 32768
15601 091/300 Read block 3455 len 32768
15602 092/300 Read block 885 len 24576
15603 093/300 Write block 2736 len 32768 data: 4 3 3 3 3 2 2 4
15604 094/300 Read block 2125 len 16384
15605 095/300 Read block 168 len 36864
15606 096/300 Read block 3904 len 28672
15607 097/300 Read block 249 len 20480
15608 098/300 Read block 4256 len 40960
15609 099/300 Read block 373 len 36864
15610 100/300 Read block 846 len 36864
15611 101/300 Write block 1713 len 20480 data: 6 6 6 5 4
15612 102/300 Write block 4781 len 4096 data: 3
15613 103/300 Read block 1178 len 4096
15614 104/300 Read block 3362 len 36864
15615 105/300 Read block 4719 len 12288
15616 106/300 Read block 539 len 24576
15617 107/300 Write block 1178 len 40960 data: 3 3 5 4 6 6 5 4 4 4
15618 108/300 Write block 2249 len 4096 data: 7
15619 109/300 Write block 2518 len 24576 data: 2 2 2 3 3 3
15620 110/300 Read block 2423 len 32768
15621 111/300 Write block 914 len 40960 data: 3 3 2 2 3 3 3 3 4 6
15622 112/300 Read block 2290 len 16384
15623 113/300 Read block 369 len 32768
15624 114/300 Read block 2301 len 8192
15625 115/300 Write block 3777 len 20480 data: 6 7 9 11 10
15626 116/300 Write block 1973 len 8192 data: 5 6
15627 117/300 Write block 1447 len 28672 data: 6 6 5 7 6 5 5
15628 118/300 Read block 4484 len 4096
15629 119/300 Flush
15630 120/300 Write block 4243 len 12288 data: 3 3 3
15631 121/300 Flush
15632 122/300 Write block 106 len 8192 data: 4 4
15633 123/300 Write block 4219 len 16384 data: 4 5 5 6
15634 124/300 Read block 1469 len 40960
15635 125/300 Read block 4364 len 8192
15636 126/300 Write block 4385 len 40960 data: 3 3 3 3 3 4 5 4 4 5
15637 127/300 Read block 3024 len 24576
15638 128/300 Flush
15639 129/300 Read block 4860 len 28672
15640 130/300 Write block 4986 len 20480 data: 7 5 6 6 7
15641 131/300 Write block 2754 len 12288 data: 7 6 3
15642 132/300 Flush
15643 133/300 Read block 4748 len 16384
15644 134/300 Read block 3910 len 28672
15645 135/300 Write block 985 len 28672 data: 5 4 3 4 3 3 4
15646 136/300 Read block 1611 len 40960
15647 137/300 Write block 3352 len 16384 data: 7 8 7 7
15648 138/300 Read block 4777 len 32768
15649 139/300 Flush
15650 140/300 Read block 4618 len 32768
15651 141/300 Write block 374 len 8192 data: 6 7
15652 142/300 Read block 2433 len 32768
15653 143/300 Write block 2253 len 4096 data: 6
15654 144/300 Read block 1347 len 24576
15655 145/300 Write block 3505 len 28672 data: 4 4 5 4 4 4 4
15656 146/300 Write block 3000 len 28672 data: 4 2 2 4 5 5 5
15657 147/300 Write block 1001 len 12288 data: 6 5 4
15658 148/300 Read block 1034 len 28672
15659 149/300 Read block 1192 len 32768
15660 150/300 Read block 4935 len 16384
15661 151/300 Flush
15662 152/300 Write block 1443 len 16384 data: 6 6 6 7
15663 153/300 Write block 2751 len 20480 data: 6 7 7 8 7
15664 154/300 Read block 2374 len 24576
15665 155/300 Read block 608 len 20480
15666 156/300 Read block 4126 len 24576
15667 157/300 Read block 3224 len 4096
15668 158/300 Read block 320 len 40960
15669 159/300 Write block 1228 len 16384 data: 2 2 4 5
15670 160/300 Read block 1245 len 40960
15671 161/300 Write block 2172 len 16384 data: 7 8 8 7
15672 162/300 Write block 4642 len 12288 data: 6 9 9
15673 163/300 Write block 155 len 28672 data: 3 4 3 4 5 4 5
15674 164/300 Read block 4867 len 12288
15675 165/300 Read block 1668 len 8192
15676 166/300 Read block 4151 len 12288
15677 167/300 Read block 278 len 24576
15678 168/300 Write block 293 len 20480 data: 8 6 9 8 8
15679 169/300 Flush
15680 170/300 Write block 1837 len 20480 data: 5 4 5 5 5
15681 171/300 Read block 1237 len 16384
15682 172/300 Read block 2761 len 32768
15683 173/300 Write block 3696 len 32768 data: 4 3 4 6 7 6 6 5
15684 174/300 Read block 4056 len 40960
15685 175/300 Read block 331 len 16384
15686 176/300 Flush
15687 177/300 Write block 2171 len 36864 data: 7 8 9 9 8 6 5 5 4
15688 178/300 Read block 3886 len 4096
15689 179/300 Write block 2399 len 36864 data: 3 4 4 4 4 5 5 5 6
15690 180/300 Write block 531 len 8192 data: 8 10
15691 181/300 Read block 3618 len 4096
15692 182/300 Write block 3784 len 36864 data: 8 6 5 5 5 6 5 3 3
15693 183/300 Read block 3476 len 32768
15694 184/300 Read block 1860 len 40960
15695 185/300 Write block 1242 len 24576 data: 3 3 4 4 3 3
15696 186/300 Write block 4527 len 8192 data: 3 3
15697 187/300 Read block 3208 len 16384
15698 188/300 Write block 585 len 28672 data: 4 3 4 6 5 5 5
15699 189/300 Read block 1153 len 32768
15700 190/300 Read block 3541 len 8192
15701 191/300 Flush
15702 192/300 Read block 1941 len 12288
15703 193/300 Write block 246 len 4096 data: 6
15704 194/300 Write block 2705 len 4096 data: 4
15705 195/300 Read block 3325 len 40960
15706 196/300 Write block 3995 len 24576 data: 5 6 6 6 5 5
15707 197/300 Read block 4282 len 12288
15708 198/300 Flush
15709 199/300 Read block 3833 len 4096
15710 200/300 Write block 779 len 8192 data: 4 3
15711 201/300 Read block 1740 len 24576
15712 202/300 Write block 4404 len 24576 data: 8 7 4 4 4 4
15713 203/300 Write block 2605 len 28672 data: 4 3 4 3 2 2 2
15714 204/300 Read block 3383 len 40960
15715 205/300 Write block 3023 len 40960 data: 5 5 4 3 3 3 3 3 3 3
15716 206/300 Flush
15717 207/300 Read block 4680 len 12288
15718 208/300 Read block 1681 len 12288
15719 209/300 Flush
15720 210/300 Write block 1337 len 24576 data: 3 3 3 3 3 2
15721 211/300 Write block 1530 len 8192 data: 7 7
15722 212/300 Write block 2841 len 36864 data: 6 7 7 7 6 5 5 6 5
15723 213/300 Read block 536 len 8192
15724 214/300 Write block 4841 len 32768 data: 6 5 4 5 5 5 3 4
15725 215/300 Write block 4989 len 40960 data: 7 8 5 6 4 4 5 4 4 4
15726 216/300 Read block 1098 len 36864
15727 217/300 Read block 4174 len 32768
15728 218/300 Write block 2590 len 12288 data: 5 3 4
15729 219/300 Write block 317 len 24576 data: 4 3 3 3 6 8
15730 220/300 Write block 280 len 36864 data: 6 6 7 6 7 5 5 7 7
15731 221/300 Write block 259 len 32768 data: 2 2 2 2 5 5 5 7
15732 222/300 Write block 758 len 40960 data: 6 5 4 4 3 3 4 5 3 3
15733 223/300 Write block 3385 len 16384 data: 3 3 5 5
15734 224/300 Write block 46 len 16384 data: 3 3 4 6
15735 225/300 Read block 2135 len 32768
15736 226/300 Read block 1136 len 16384
15737 227/300 Read block 3717 len 36864
15738 228/300 Read block 3445 len 8192
15739 229/300 Write block 397 len 4096 data: 2
15740 230/300 Write block 1407 len 28672 data: 4 2 2 3 4 4 4
15741 231/300 Write block 384 len 4096 data: 2
15742 232/300 Read block 4724 len 40960
15743 233/300 Read block 4900 len 40960
15744 234/300 Read block 615 len 12288
15745 235/300 Write block 345 len 12288 data: 6 6 4
15746 236/300 Write block 2932 len 16384 data: 5 6 5 4
15747 237/300 Read block 2528 len 16384
15748 238/300 Read block 2786 len 28672
15749 239/300 Read block 835 len 24576
15750 240/300 Flush
15751 241/300 Read block 1708 len 24576
15752 242/300 Read block 4934 len 20480
15753 243/300 Read block 185 len 4096
15754 244/300 Write block 997 len 20480 data: 4 5 6 6 7
15755 245/300 Write block 2018 len 32768 data: 7 6 3 5 4 4 4 5
15756 246/300 Flush
15757 247/300 Write block 591 len 40960 data: 6 4 3 4 4 3 3 3 3 3
15758 248/300 Write block 3688 len 24576 data: 3 2 2 3 4 4
15759 249/300 Read block 4343 len 20480
15760 250/300 Read block 961 len 20480
15761 251/300 Write block 2823 len 36864 data: 3 4 6 6 6 6 5 5 6
15762 252/300 Read block 4069 len 36864
15763 253/300 Read block 4469 len 32768
15764 254/300 Write block 2620 len 20480 data: 3 2 3 2 3
15765 255/300 Read block 2277 len 28672
15766 256/300 Write block 2268 len 32768 data: 5 5 6 7 6 5 4 5
15767 257/300 Read block 2473 len 20480
15768 258/300 Read block 2268 len 12288
15769 259/300 Read block 1174 len 36864
15770 260/300 Write block 1148 len 36864 data: 2 2 2 3 3 3 3 4 4
15771 261/300 Read block 1421 len 12288
15772 262/300 Read block 2569 len 40960
15773 263/300 Read block 3220 len 12288
15774 264/300 Write block 1193 len 4096 data: 6
15775 265/300 Read block 658 len 24576
15776 266/300 Flush
15777 267/300 Read block 927 len 16384
15778 268/300 Flush
15779 269/300 Read block 3258 len 36864
15780 270/300 Read block 361 len 16384
15781 271/300 Read block 4112 len 32768
15782 272/300 Read block 2792 len 20480
15783 273/300 Write block 1634 len 20480 data: 2 2 2 2 2
15784 274/300 Read block 4668 len 4096
15785 275/300 Read block 2245 len 28672
15786 276/300 Write block 1650 len 16384 data: 6 5 5 5
15787 277/300 Write block 2321 len 20480 data: 6 4 2 3 3
15788 278/300 Flush
15789 279/300 Read block 3488 len 4096
15790 280/300 Read block 2217 len 40960
15791 281/300 Read block 4517 len 36864
15792 282/300 Read block 4089 len 8192
15793 283/300 Write block 2769 len 12288 data: 5 5 5
15794 284/300 Write block 1361 len 40960 data: 5 5 5 5 2 3 3 3 2 2
15795 285/300 Read block 3875 len 32768
15796 286/300 Read block 1640 len 16384
15797 287/300 Write block 2794 len 8192 data: 5 4
15798 288/300 Read block 3459 len 36864
15799 289/300 Write block 786 len 4096 data: 4
15800 290/300 Read block 2321 len 8192
15801 291/300 Write block 4825 len 32768 data: 4 4 5 4 3 3 3 4
15802 292/300 Write block 4797 len 4096 data: 6
15803 293/300 Write block 1778 len 28672 data: 4 4 4 5 5 4 3
15804 294/300 Write block 1112 len 32768 data: 3 5 5 5 5 5 5 5
15805 295/300 Read block 2689 len 24576
15806 296/300 Flush
15807 297/300 Read block 2121 len 36864
15808 298/300 Write block 915 len 40960 data: 4 3 3 4 4 4 4 5 7 6
15809 299/300 Read block 2627 len 12288
15810 300/300 Write block 2783 len 40960 data: 3 3 3 2 2 2 2 3 4 4
15811 Main task has work to do, go find it
15812 got dsc Start(2)
15813 start 2
15814 Mar 02 01:08:48.176 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: eb4893ef-a23d-4574-9867-318e34999936, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
15815 Replay: started 2, returned:Ok(())
15816 ----------------------------------------------------------------
15817 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
15818 States: Active Active Offline
15819 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15820 5127 Acked 6126 Write 3 Done Done New true
15821 5128 Acked 6127 Write 4 Done Done New true
15822 5129 Acked 6128 Read 8 Done Done New true
15823 5130 Acked 6129 Read 6 Done Done New false
15824 5131 Acked 6130 Write 1 Done Done New false
15825 5132 Acked 6131 Write 6 Done Done New false
15826 5133 Acked 6132 Write 3 Done Done New false
15827 5134 Acked 6133 Read 6 Done Done New false
15828 5135 Acked 6134 Read 8 Done Done New false
15829 5136 Acked 6135 Read 4 Done Done New false
15830 5137 Acked 6136 Read 8 Done Done New false
15831 5138 Acked 6137 Read 6 Done Done New false
15832 5139 Acked 6138 Read 10 Done Done New false
15833 5140 Acked 6139 Write 4 Done Done New false
15834 5141 Acked 6140 Read 5 Done Done New false
15835 5142 Acked 6141 Read 8 Done Done New false
15836 5143 Acked 6142 Write 5 Done Done New false
15837 5144 Acked 6143 Read 9 Done Done New false
15838 5145 Acked 6144 Read 5 Done Done New false
15839 5146 Acked 6145 Write 1 Done Done New false
15840 5147 Acked 6146 Read 8 Done Done New false
15841 5148 Acked 6147 Write 5 Done Done New false
15842 5149 Acked 6148 Read 4 Done Done New false
15843 5150 Acked 6149 Read 2 Done Done New false
15844 5151 Acked 6150 Read 7 Done Done New false
15845 5152 Acked 6151 Flush 0 Done Done New false
15846 5153 Acked 6152 Flush 0 Done Done New false
15847 5154 Acked 6153 Read 9 Done Done New false
15848 5155 Acked 6154 Flush 0 Done Done New false
15849 5156 Acked 6155 Write 9 Done Done New false
15850 5157 Acked 6156 Write 3 Done Done New false
15851 5158 Acked 6157 Write 5 Done Done New false
15852 5159 Acked 6158 Read 5 Done Done New false
15853 5160 Acked 6159 Write 1 Done Done New false
15854 5161 Acked 6160 Write 5 Done Done New false
15855 5162 Acked 6161 Write 4 Done Done New false
15856 5163 Acked 6162 Flush 0 Done Done New false
15857 5164 Acked 6163 Write 1 Done Done New false
15858 5165 Acked 6164 Flush 0 Done Done New false
15859 5166 Acked 6165 Read 1 Done Done New false
15860 5167 Acked 6166 Read 10 Done Done New false
15861 5168 Acked 6167 Read 9 Done Done New false
15862 5169 Acked 6168 Read 10 Done Done New false
15863 5170 Acked 6169 Flush 0 Done Done New false
15864 5171 Acked 6170 Read 2 Done Done New false
15865 5172 Acked 6171 Read 9 Done Done New false
15866 5173 Acked 6172 Read 4 Done Done New false
15867 5174 Acked 6173 Write 7 Done Done New false
15868 5175 Acked 6174 Read 3 Done Done New false
15869 5176 Acked 6175 Read 8 Done Done New false
15870 5177 Acked 6176 Write 5 Done Done New false
15871 5178 Acked 6177 Write 3 Done Done New false
15872 5179 Acked 6178 Flush 0 Done Done New false
15873 5180 Acked 6179 Read 5 Done Done New false
15874 5181 Acked 6180 Read 7 Done Done New false
15875 5182 Acked 6181 Flush 0 Done Done New false
15876 5183 Acked 6182 Write 6 Done Done New false
15877 5184 Acked 6183 Write 5 Done Done New false
15878 5185 Acked 6184 Read 7 Done Done New false
15879 5186 Acked 6185 Flush 0 Done Done New false
15880 5187 Acked 6186 Write 10 Done Done New false
15881 5188 Acked 6187 Read 6 Done Done New false
15882 5189 Acked 6188 Read 6 Done Done New false
15883 5190 Acked 6189 Write 9 Done Done New false
15884 5191 Acked 6190 Read 5 Done Done New false
15885 5192 Acked 6191 Read 1 Done Done New false
15886 5193 Acked 6192 Write 9 Done Done New false
15887 5194 Acked 6193 Write 3 Done Done New false
15888 5195 Acked 6194 Read 10 Done Done New false
15889 5196 Acked 6195 Read 2 Done Done New false
15890 5197 Acked 6196 Write 10 Done Done New false
15891 5198 Acked 6197 Write 2 Done Done New false
15892 5199 Acked 6198 Flush 0 Done Done New false
15893 5200 Acked 6199 Read 7 Done Done New false
15894 5201 Acked 6200 Write 7 Done Done New false
15895 5202 Acked 6201 Read 2 Done Done New false
15896 5203 Acked 6202 Read 10 Done Done New false
15897 5204 Acked 6203 Write 2 Done Done New false
15898 5205 Acked 6204 Write 3 Done Done New false
15899 5206 Acked 6205 Write 1 Done Done New false
15900 5207 Acked 6206 Write 8 Done Done New false
15901 5208 Acked 6207 Write 2 Done Done New false
15902 5209 Acked 6208 Read 6 Done Done New false
15903 5210 Acked 6209 Write 10 Done Done New false
15904 5211 Acked 6210 Write 9 Done Done New false
15905 5212 Acked 6211 Write 5 Done Done New false
15906 5213 Acked 6212 Write 9 Done Done New false
15907 5214 Acked 6213 Write 1 Done Done New false
15908 5215 Acked 6214 Flush 0 Done Done New false
15909 5216 Acked 6215 Read 8 Done Done New false
15910 5217 Acked 6216 Read 8 Done Done New false
15911 5218 Acked 6217 Read 6 Done Done New false
15912 5219 Acked 6218 Write 8 Done Done New false
15913 5220 Acked 6219 Read 4 Done Done New false
15914 5221 Acked 6220 Read 9 Done Done New false
15915 5222 Acked 6221 Read 7 Done Done New false
15916 5223 Acked 6222 Read 5 Done Done New false
15917 5224 Acked 6223 Read 10 Done Done New false
15918 5225 Acked 6224 Read 9 Done Done New false
15919 5226 Acked 6225 Read 9 Done Done New false
15920 5227 Acked 6226 Write 5 Done Done New false
15921 5228 Acked 6227 Write 1 Done Done New false
15922 5229 Acked 6228 Read 1 Done Done New false
15923 5230 Acked 6229 Read 9 Done Done New false
15924 5231 Acked 6230 Read 3 Done Done New false
15925 5232 Acked 6231 Read 6 Done Done New false
15926 5233 Acked 6232 Write 10 Done Done New false
15927 5234 Acked 6233 Write 1 Done Done New false
15928 5235 Acked 6234 Write 6 Done Done New false
15929 5236 Acked 6235 Read 8 Done Done New false
15930 5237 Acked 6236 Write 10 Done Done New false
15931 5238 Acked 6237 Read 4 Done Done New false
15932 5239 Acked 6238 Read 8 Done Done New false
15933 5240 Acked 6239 Read 2 Done Done New false
15934 5241 Acked 6240 Write 5 Done Done New false
15935 5242 Acked 6241 Write 2 Done Done New false
15936 5243 Acked 6242 Write 7 Done Done New false
15937 5244 Acked 6243 Read 1 Done Done New false
15938 5245 Acked 6244 Flush 0 Done Done New false
15939 5246 Acked 6245 Write 3 Done Done New false
15940 5247 Acked 6246 Flush 0 Done Done New false
15941 5248 Acked 6247 Write 2 Done Done New false
15942 5249 Acked 6248 Write 4 Done Done New false
15943 5250 Acked 6249 Read 10 Done Done New false
15944 5251 Acked 6250 Read 2 Done Done New false
15945 5252 Acked 6251 Write 10 Done Done New false
15946 5253 Acked 6252 Read 6 Done Done New false
15947 5254 Acked 6253 Flush 0 Done Done New false
15948 5255 Acked 6254 Read 7 Done Done New false
15949 5256 Acked 6255 Write 5 Done Done New false
15950 5257 Acked 6256 Write 3 Done Done New false
15951 5258 Acked 6257 Flush 0 Done Done New false
15952 5259 Acked 6258 Read 4 Done Done New false
15953 5260 Acked 6259 Read 7 Done Done New false
15954 5261 Acked 6260 Write 7 Done Done New false
15955 5262 Acked 6261 Read 10 Done Done New false
15956 5263 Acked 6262 Write 4 Done Done New false
15957 5264 Acked 6263 Read 8 Done Done New false
15958 5265 Acked 6264 Flush 0 Done Done New false
15959 5266 Acked 6265 Read 8 Done Done New false
15960 5267 Acked 6266 Write 2 Done Done New false
15961 5268 Acked 6267 Read 8 Done Done New false
15962 5269 Acked 6268 Write 1 Done Done New false
15963 5270 Acked 6269 Read 6 Done Done New false
15964 5271 Acked 6270 Write 7 Done Done New false
15965 5272 Acked 6271 Write 7 Done Done New false
15966 5273 Acked 6272 Write 3 Done Done New false
15967 5274 Acked 6273 Read 7 Done Done New false
15968 5275 Acked 6274 Read 8 Done Done New false
15969 5276 Acked 6275 Read 4 Done Done New false
15970 5277 Acked 6276 Flush 0 Done Done New false
15971 5278 Acked 6277 Write 4 Done Done New false
15972 5279 Acked 6278 Write 5 Done Done New false
15973 5280 Acked 6279 Read 6 Done Done New false
15974 5281 Acked 6280 Read 5 Done Done New false
15975 5282 Acked 6281 Read 6 Done Done New false
15976 5283 Acked 6282 Read 1 Done Done New false
15977 5284 Acked 6283 Read 10 Done Done New false
15978 5285 Acked 6284 Write 4 Done Done New false
15979 5286 Acked 6285 Read 10 Done Done New false
15980 5287 Acked 6286 Write 4 Done Done New false
15981 5288 Acked 6287 Write 3 Done Done New false
15982 5289 Acked 6288 Write 7 Done Done New false
15983 5290 Acked 6289 Read 3 Done Done New false
15984 5291 Acked 6290 Read 2 Done Done New false
15985 5292 Acked 6291 Read 3 Done Done New false
15986 5293 Acked 6292 Read 6 Done Done New false
15987 5294 Acked 6293 Write 5 Done Done New false
15988 5295 Acked 6294 Flush 0 Done Done New false
15989 5296 Acked 6295 Write 5 Done Done New false
15990 5297 Acked 6296 Read 4 Done Done New false
15991 5298 Acked 6297 Read 8 Done Done New false
15992 5299 Acked 6298 Write 8 Done Done New false
15993 5300 Acked 6299 Read 10 Done Done New false
15994 5301 Acked 6300 Read 4 Done Done New false
15995 5302 Acked 6301 Flush 0 Done Done New false
15996 5303 Acked 6302 Write 9 Done Done New false
15997 5304 Acked 6303 Read 1 Done Done New false
15998 5305 Acked 6304 Write 9 Done Done New false
15999 5306 Acked 6305 Write 2 Done Done New false
16000 5307 Acked 6306 Read 1 Done Done New false
16001 5308 Acked 6307 Write 9 Done Done New false
16002 5309 Acked 6308 Read 8 Done Done New false
16003 5310 Acked 6309 Read 10 Done Done New false
16004 5311 Acked 6310 Write 6 Done Done New false
16005 5312 Acked 6311 Write 2 Done Done New false
16006 5313 Acked 6312 Read 4 Done Done New false
16007 5314 Acked 6313 Write 7 Done Done New false
16008 5315 Acked 6314 Read 8 Done Done New false
16009 5316 Acked 6315 Read 2 Done Done New false
16010 5317 Acked 6316 Flush 0 Done Done New false
16011 5318 Acked 6317 Read 3 Done Done New false
16012 5319 Acked 6318 Write 1 Done Done New false
16013 5320 Acked 6319 Write 1 Done Done New false
16014 5321 Acked 6320 Read 10 Done Done New false
16015 5322 Acked 6321 Write 6 Done Done New false
16016 5323 Acked 6322 Read 3 Done Done New false
16017 5324 Acked 6323 Flush 0 Done Done New false
16018 5325 Acked 6324 Read 1 Done Done New false
16019 5326 Acked 6325 Write 2 Done Done New false
16020 5327 Acked 6326 Read 6 Done Done New false
16021 5328 Acked 6327 Write 6 Done Done New false
16022 5329 Acked 6328 Write 7 Done Done New false
16023 5330 Acked 6329 Read 10 Done Done New false
16024 5331 Acked 6330 Write 10 Done Done New false
16025 5332 Acked 6331 Flush 0 Done Done New false
16026 5333 Acked 6332 Read 3 Done Done New false
16027 5334 Acked 6333 Read 3 Done Done New false
16028 5335 Acked 6334 Flush 0 Done Done New false
16029 5336 Acked 6335 Write 6 Done Done New false
16030 5337 Acked 6336 Write 2 Done Done New false
16031 5338 Acked 6337 Write 9 Done Done New false
16032 5339 Acked 6338 Read 2 Done Done New false
16033 5340 Acked 6339 Write 8 Done Done New false
16034 5341 Acked 6340 Write 10 Done Done New false
16035 5342 Acked 6341 Read 9 Done Done New false
16036 5343 Acked 6342 Read 8 Done Done New false
16037 5344 Acked 6343 Write 3 Done Done New false
16038 5345 Acked 6344 Write 6 Done Done New false
16039 5346 Acked 6345 Write 9 Done Done New false
16040 5347 Acked 6346 Write 8 Done Done New false
16041 5348 Acked 6347 Write 10 Done Done New false
16042 5349 Acked 6348 Write 4 Done Done New false
16043 5350 Acked 6349 Write 4 Done Done New false
16044 5351 Acked 6350 Read 8 Done Done New false
16045 5352 Acked 6351 Read 4 Done Done New false
16046 5353 Acked 6352 Read 9 Done Done New false
16047 5354 Acked 6353 Read 2 Done Done New false
16048 5355 Acked 6354 Write 1 Done Done New false
16049 5356 Acked 6355 Write 7 Done Done New false
16050 5357 Acked 6356 Write 1 Done Done New false
16051 5358 Acked 6357 Read 10 Done Done New false
16052 5359 Acked 6358 Read 10 Done Done New false
16053 5360 Acked 6359 Read 3 Done Done New false
16054 5361 Acked 6360 Write 3 Done Done New false
16055 5362 Acked 6361 Write 4 Done Done New false
16056 5363 Acked 6362 Read 4 Done Done New false
16057 5364 Acked 6363 Read 7 Done Done New false
16058 5365 Acked 6364 Read 6 Done Done New false
16059 5366 Acked 6365 Flush 0 Done Done New false
16060 5367 Acked 6366 Read 6 Done Done New false
16061 5368 Acked 6367 Read 5 Done Done New false
16062 5369 Acked 6368 Read 1 Done Done New false
16063 5370 Acked 6369 Write 5 Done Done New false
16064 5371 Acked 6370 Write 8 Done Done New false
16065 5372 Acked 6371 Flush 0 Done Done New false
16066 5373 Acked 6372 Write 10 Done Done New false
16067 5374 Acked 6373 Write 6 Done Done New false
16068 5375 Acked 6374 Read 5 Done Done New false
16069 5376 Acked 6375 Read 5 Done Done New false
16070 5377 Acked 6376 Write 9 Done Done New false
16071 5378 Acked 6377 Read 9 Done Done New false
16072 5379 Acked 6378 Read 8 Done Done New false
16073 5380 Acked 6379 Write 5 Done Done New false
16074 5381 Acked 6380 Read 7 Done Done New false
16075 5382 Acked 6381 Write 8 Done Done New false
16076 5383 Acked 6382 Read 5 Done Done New false
16077 5384 Acked 6383 Read 3 Done Done New false
16078 5385 Acked 6384 Read 9 Done Done New false
16079 5386 Acked 6385 Write 9 Done Done New false
16080 5387 Acked 6386 Read 3 Done Done New false
16081 5388 Acked 6387 Read 10 Done Done New false
16082 5389 Acked 6388 Read 3 Done Done New false
16083 5390 Acked 6389 Write 1 Done Done New false
16084 5391 Acked 6390 Read 6 Done Done New false
16085 5392 Acked 6391 Flush 0 Done Done New false
16086 5393 Acked 6392 Read 4 Done Done New false
16087 5394 Acked 6393 Flush 0 Done Done New false
16088 5395 Acked 6394 Read 9 Done Done New false
16089 5396 Acked 6395 Read 4 Done Done New false
16090 5397 Acked 6396 Read 8 Done Done New false
16091 5398 Acked 6397 Read 5 Done Done New false
16092 5399 Acked 6398 Write 5 Done Done New false
16093 5400 Acked 6399 Read 1 Done Done New false
16094 5401 Acked 6400 Read 7 Done Done New false
16095 5402 Acked 6401 Write 4 Done Done New false
16096 5403 Acked 6402 Write 5 Done Done New false
16097 5404 Acked 6403 Flush 0 Done Done New false
16098 5405 Acked 6404 Read 1 Done Done New false
16099 5406 Acked 6405 Read 10 Done Done New false
16100 5407 Acked 6406 Read 9 Done Done New false
16101 5408 Acked 6407 Read 2 Done Done New false
16102 5409 Acked 6408 Write 3 Done Done New false
16103 5410 Acked 6409 Write 10 Done Done New false
16104 5411 Acked 6410 Read 8 Done Done New false
16105 5412 Acked 6411 Read 4 Done Done New false
16106 5413 Acked 6412 Write 2 Done Done New false
16107 5414 Acked 6413 Read 9 Done Done New false
16108 5415 Acked 6414 Write 1 Done Done New false
16109 5416 Acked 6415 Read 2 Done Done New false
16110 5417 Acked 6416 Write 8 Done Done New false
16111 5418 Acked 6417 Write 1 Done Done New false
16112 5419 Acked 6418 Write 7 Done Done New false
16113 5420 Acked 6419 Write 8 Done Done New false
16114 5421 Acked 6420 Read 6 Done Done New false
16115 5422 Acked 6421 Flush 0 Done Done New false
16116 5423 Acked 6422 Read 9 Done Done New false
16117 5424 Acked 6423 Write 10 Done Done New false
16118 5425 Acked 6424 Read 3 Done Done New false
16119 5426 Acked 6425 Write 10 Done Done New false
16120 STATES DS:0 DS:1 DS:2 TOTAL
16121 New 0 0 300 300
16122 Sent 0 0 0 0
16123 Done 300 300 0 600
16124 Skipped 0 0 0 0
16125 Error 0 0 0 0
16126 Last Flush: 6421 6421 6125
16127 Downstairs last five completed: 6125 6124 6123 6122 6121
16128 Upstairs last five completed: 5426 5425 5424 5423 5422
16129 CLIENT: Up:0 ds:300 act:2
161302024-03-02T01:08:48.829ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161312024-03-02T01:08:48.829ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161322024-03-02T01:08:48.829ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161332024-03-02T01:08:48.829ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161342024-03-02T01:08:48.829ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161352024-03-02T01:08:48.829ZINFOcrucible: [2] client re-new 301 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161362024-03-02T01:08:48.830ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16137 [8830] got start action
16138 [8830] I am going to restart
16139 Starting downstairs at port 8830
16140 Make output file at "/tmp/dsc/downstairs-8830.txt"
16141 [8830][2] reports Starting
161422024-03-02T01:08:49.831ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161432024-03-02T01:08:49.831ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161442024-03-02T01:08:49.831ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161452024-03-02T01:08:49.831ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161462024-03-02T01:08:49.831ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161472024-03-02T01:08:49.831ZINFOcrucible: [2] client re-new 301 jobs since flush 6125 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161482024-03-02T01:08:49.831ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16149 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1180)
16150 [8830][2] reports Running
161512024-03-02T01:08:50.832ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161522024-03-02T01:08:50.832ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161532024-03-02T01:08:50.834ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161542024-03-02T01:08:50.834ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161552024-03-02T01:08:50.834ZINFOcrucible: send last flush ID to this DS: 6125 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161562024-03-02T01:08:50.834ZINFOcrucible: Replied this last flush ID: 6125 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161572024-03-02T01:08:50.834ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161582024-03-02T01:08:50.834ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161592024-03-02T01:08:50.834ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161602024-03-02T01:08:50.834ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16161 ----------------------------------------------------------------
16162 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16163 Downstairs last five completed: 6426 6425 6424 6423 6422
16164 Upstairs last five completed: 5427 5426 5425 5424 5423
16165 CLIENT: Up:0 ds:0 act:3
16166 Replay: All jobs finished, all DS active.
16167 Main task has work to do, go find it
16168 got dsc Stop(2)
16169 stop 2
16170 Mar 02 01:08:52.180 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false
16171 method: POST, req_id: b252e532-4b5d-45c1-90ae-0b54c78a94f7, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
161722024-03-02T01:08:52.181ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161732024-03-02T01:08:52.181ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161742024-03-02T01:08:52.181ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161752024-03-02T01:08:52.181ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161762024-03-02T01:08:52.181ZINFOcrucible: [2] client re-new 0 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161772024-03-02T01:08:52.181ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16178 Mar 02 01:08:52.181 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/2, method: GET, req_id: 6b5ebf16-2b09-45f1-90f8-d8c50279410a, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
16179 [8830] Exited with: ExitStatus(unix_wait_status(9))
16180 [8830][2] reports Exit
161812024-03-02T01:08:53.182ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161822024-03-02T01:08:53.183ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161832024-03-02T01:08:53.183ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161842024-03-02T01:08:53.183ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161852024-03-02T01:08:53.183ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161862024-03-02T01:08:53.183ZINFOcrucible: [2] client re-new 0 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161872024-03-02T01:08:53.183ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161882024-03-02T01:08:54.184ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161892024-03-02T01:08:54.184ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161902024-03-02T01:08:54.184ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161912024-03-02T01:08:54.184ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161922024-03-02T01:08:54.184ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161932024-03-02T01:08:54.184ZINFOcrucible: [2] client re-new 0 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161942024-03-02T01:08:54.184ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161952024-03-02T01:08:55.184ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161962024-03-02T01:08:55.185ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161972024-03-02T01:08:55.185ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161982024-03-02T01:08:55.185ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
161992024-03-02T01:08:55.185ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162002024-03-02T01:08:55.185ZINFOcrucible: [2] client re-new 0 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162012024-03-02T01:08:55.185ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16202 Mar 02 01:08:56.184 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 07851c53-33f1-4ca7-907c-0783d9bb4715, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
16203 001/300 Read block 930 len 28672
162042024-03-02T01:08:56.185ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162052024-03-02T01:08:56.185ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162062024-03-02T01:08:56.185ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162072024-03-02T01:08:56.185ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162082024-03-02T01:08:56.185ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162092024-03-02T01:08:56.185ZINFOcrucible: [2] client re-new 1 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
162102024-03-02T01:08:56.185ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16211 002/300 Read block 4845 len 20480
16212 003/300 Flush
16213 004/300 Read block 4141 len 8192
16214 005/300 Read block 2696 len 32768
16215 006/300 Write block 1432 len 8192 data: 4 3
16216 007/300 Write block 1849 len 20480 data: 4 3 2 3 3
16217 008/300 Read block 3393 len 28672
16218 009/300 Read block 1480 len 16384
16219 010/300 Read block 4862 len 8192
16220 011/300 Write block 138 len 4096 data: 4
16221 012/300 Write block 1939 len 24576 data: 7 6 6 6 7 9
16222 013/300 Flush
16223 014/300 Write block 469 len 32768 data: 8 7 6 5 4 4 3 3
16224 015/300 Flush
16225 016/300 Read block 2195 len 40960
16226 017/300 Read block 1342 len 28672
16227 018/300 Read block 537 len 28672
16228 019/300 Write block 4514 len 20480 data: 2 4 4 3 3
16229 020/300 Write block 2800 len 20480 data: 3 3 2 4 4
16230 021/300 Read block 479 len 4096
16231 022/300 Read block 3748 len 32768
16232 023/300 Write block 4256 len 8192 data: 2 2
16233 024/300 Flush
16234 025/300 Read block 2542 len 40960
16235 026/300 Read block 3770 len 4096
16236 027/300 Read block 274 len 32768
16237 028/300 Write block 4941 len 28672 data: 3 3 2 3 3 5 5
16238 029/300 Flush
16239 030/300 Read block 4655 len 16384
16240 031/300 Read block 3374 len 12288
16241 032/300 Write block 2305 len 4096 data: 4
16242 033/300 Read block 689 len 8192
16243 034/300 Flush
16244 035/300 Read block 3875 len 20480
16245 036/300 Write block 3956 len 36864 data: 4 4 3 4 4 5 5 4 4
16246 037/300 Write block 2065 len 36864 data: 7 8 8 8 7 7 6 5 5
16247 038/300 Read block 2333 len 40960
16248 039/300 Write block 1649 len 12288 data: 5 7 6
16249 040/300 Write block 3231 len 40960 data: 5 5 5 6 6 5 5 5 5 4
16250 041/300 Read block 4268 len 8192
16251 042/300 Read block 1168 len 28672
16252 043/300 Flush
16253 044/300 Write block 828 len 28672 data: 6 5 6 7 7 6 5
16254 045/300 Read block 1050 len 36864
16255 046/300 Read block 3735 len 4096
16256 047/300 Read block 259 len 16384
16257 048/300 Read block 4555 len 8192
16258 049/300 Flush
16259 050/300 Read block 3858 len 40960
16260 051/300 Write block 799 len 4096 data: 2
16261 052/300 Flush
16262 053/300 Write block 4599 len 20480 data: 4 5 6 7 7
16263 054/300 Read block 297 len 12288
16264 055/300 Write block 2876 len 28672 data: 5 4 5 3 3 2 2
16265 056/300 Write block 3457 len 8192 data: 4 5
16266 057/300 Read block 3743 len 28672
16267 058/300 Read block 820 len 4096
16268 059/300 Write block 4351 len 28672 data: 5 5 6 6 6 5 5
16269 060/300 Read block 2030 len 20480
16270 061/300 Read block 3923 len 40960
16271 062/300 Read block 871 len 16384
16272 063/300 Read block 1775 len 4096
16273 064/300 Flush
16274 065/300 Flush
16275 066/300 Read block 216 len 24576
16276 067/300 Read block 2223 len 12288
16277 068/300 Write block 2457 len 28672 data: 5 5 5 6 5 4 4
16278 069/300 Write block 2952 len 20480 data: 4 4 4 5 5
16279 070/300 Read block 3140 len 16384
16280 071/300 Read block 3450 len 40960
16281 072/300 Read block 2899 len 8192
16282 073/300 Read block 2454 len 28672
16283 074/300 Read block 2377 len 32768
16284 075/300 Read block 916 len 28672
16285 076/300 Read block 3972 len 24576
16286 077/300 Write block 2899 len 4096 data: 5
16287 078/300 Read block 3096 len 40960
16288 079/300 Read block 231 len 40960
16289 080/300 Read block 3727 len 24576
16290 081/300 Write block 3452 len 16384 data: 7 6 6 6
16291 082/300 Read block 4543 len 28672
16292 083/300 Read block 2128 len 32768
16293 084/300 Write block 4060 len 32768 data: 3 3 2 2 2 2 2 3
16294 085/300 Write block 1273 len 16384 data: 3 3 4 5
16295 086/300 Write block 1101 len 36864 data: 5 4 3 3 2 2 2 2 2
16296 087/300 Read block 4209 len 28672
16297 088/300 Write block 4794 len 4096 data: 7
16298 089/300 Write block 4315 len 16384 data: 3 3 4 4
16299 090/300 Read block 4827 len 24576
16300 091/300 Write block 340 len 40960 data: 12 10 10 9 7 7 7 5 3 3
16301 092/300 Read block 292 len 40960
16302 093/300 Write block 196 len 8192 data: 4 5
16303 094/300 Read block 4755 len 28672
16304 095/300 Flush
16305 096/300 Read block 1816 len 28672
16306 097/300 Read block 4658 len 28672
16307 098/300 Read block 3853 len 24576
16308 099/300 Write block 460 len 32768 data: 3 3 3 4 4 4 4 7
16309 100/300 Read block 3912 len 20480
16310 101/300 Write block 1526 len 20480 data: 6 5 8 8 8
16311 102/300 Read block 4370 len 40960
16312 103/300 Write block 755 len 40960 data: 4 4 6 7 6 5 5 4 4 5
16313 104/300 Read block 183 len 12288
16314 105/300 Write block 2494 len 8192 data: 4 4
16315 106/300 Read block 4420 len 28672
16316 107/300 Write block 4203 len 28672 data: 3 4 3 3 3 2 2
16317 108/300 Read block 712 len 8192
16318 109/300 Read block 566 len 8192
16319 110/300 Write block 3949 len 20480 data: 5 7 5 5 4
16320 111/300 Flush
16321 112/300 Read block 2296 len 12288
16322 113/300 Read block 2102 len 40960
16323 114/300 Read block 3228 len 32768
16324 115/300 Flush
16325 116/300 Read block 1125 len 4096
16326 117/300 Write block 456 len 24576 data: 2 2 2 2 4 4
16327 118/300 Read block 3990 len 28672
16328 119/300 Read block 3574 len 8192
16329 120/300 Read block 45 len 16384
16330 121/300 Read block 747 len 20480
16331 122/300 Write block 3836 len 40960 data: 3 4 4 4 4 3 3 2 2 3
16332 123/300 Read block 1114 len 4096
16333 124/300 Read block 54 len 36864
16334 125/300 Write block 315 len 40960 data: 3 3 5 4 4 4 7 9 9 9
16335 126/300 Read block 972 len 24576
16336 127/300 Read block 1285 len 8192
16337 128/300 Read block 1123 len 12288
16338 129/300 Read block 4011 len 24576
16339 130/300 Write block 1469 len 40960 data: 4 3 3 3 3 3 3 3 4 4
16340 131/300 Read block 828 len 20480
16341 132/300 Read block 3840 len 8192
16342 133/300 Read block 1860 len 16384
16343 134/300 Read block 4007 len 12288
16344 135/300 Flush
16345 136/300 Flush
16346 137/300 Write block 4425 len 40960 data: 6 4 2 3 3 4 5 7 7 6
16347 138/300 Read block 858 len 12288
16348 139/300 Read block 1649 len 24576
16349 140/300 Read block 4741 len 36864
16350 141/300 Read block 583 len 24576
16351 142/300 Write block 4725 len 8192 data: 5 5
16352 143/300 Write block 3905 len 40960 data: 5 4 4 3 5 6 6 6 6 6
16353 144/300 Write block 339 len 36864 data: 13 13 11 11 10 8 8 8 6
16354 145/300 Read block 3627 len 36864
16355 146/300 Write block 1200 len 8192 data: 4 6
16356 147/300 Flush
16357 148/300 Write block 1002 len 8192 data: 6 5
16358 149/300 Write block 1806 len 40960 data: 2 3 4 5 5 5 5 5 5 6
16359 150/300 Read block 3884 len 16384
16360 151/300 Write block 1848 len 40960 data: 4 5 4 3 4 4 3 5 5 4
16361 152/300 Write block 4183 len 28672 data: 4 4 6 7 6 5 5
16362 153/300 Read block 491 len 28672
16363 154/300 Write block 3814 len 16384 data: 3 2 3 4
16364 155/300 Write block 2441 len 12288 data: 5 5 5
16365 156/300 Read block 2035 len 12288
16366 157/300 Read block 2744 len 28672
16367 158/300 Write block 3391 len 32768 data: 3 3 3 3 3 3 3 4
16368 159/300 Write block 4590 len 32768 data: 5 5 5 5 5 3 4 3
16369 160/300 Read block 4932 len 36864
16370 161/300 Read block 4934 len 32768
16371 162/300 Write block 4191 len 24576 data: 6 5 5 6 6 7
16372 163/300 Write block 1630 len 40960 data: 4 4 3 3 3 3 3 3 3 3
16373 164/300 Write block 3274 len 16384 data: 4 4 5 4
16374 165/300 Read block 2430 len 40960
16375 166/300 Write block 183 len 40960 data: 4 4 4 5 5 5 4 3 3 5
16376 167/300 Read block 3773 len 40960
16377 168/300 Flush
16378 169/300 Write block 1177 len 32768 data: 3 4 4 6 5 7 7 6
16379 170/300 Read block 213 len 16384
16380 171/300 Write block 3565 len 8192 data: 6 6
16381 172/300 Read block 4335 len 4096
16382 173/300 Read block 515 len 36864
16383 174/300 Write block 51 len 20480 data: 5 5 4 3 2
16384 175/300 Read block 955 len 16384
16385 176/300 Flush
16386 177/300 Write block 1784 len 32768 data: 4 3 3 3 3 3 3 3
16387 178/300 Read block 3971 len 4096
16388 179/300 Flush
16389 180/300 Read block 4953 len 20480
16390 181/300 Read block 3513 len 32768
16391 182/300 Read block 4738 len 12288
16392 183/300 Flush
16393 184/300 Read block 4656 len 24576
16394 185/300 Write block 401 len 32768 data: 3 3 3 4 4 4 3 3
16395 186/300 Write block 832 len 4096 data: 8
16396 187/300 Read block 3869 len 40960
16397 188/300 Read block 1313 len 28672
16398 189/300 Write block 4253 len 16384 data: 3 2 2 3
16399 190/300 Flush
16400 191/300 Read block 3037 len 36864
16401 192/300 Write block 829 len 8192 data: 6 7
16402 193/300 Write block 1722 len 4096 data: 3
16403 194/300 Read block 1553 len 4096
16404 195/300 Write block 911 len 24576 data: 3 4 4 4 5 4
16405 196/300 Write block 2909 len 40960 data: 4 4 4 4 4 6 6 5 5 3
16406 197/300 Write block 832 len 24576 data: 9 7 6 5 5 4
16407 198/300 Read block 1063 len 40960
16408 199/300 Read block 1579 len 20480
16409 200/300 Read block 3167 len 8192
16410 201/300 Read block 4684 len 12288
16411 202/300 Write block 4953 len 8192 data: 5 5
16412 203/300 Read block 235 len 36864
16413 204/300 Write block 4594 len 12288 data: 6 4 5
16414 205/300 Write block 729 len 28672 data: 5 5 4 5 7 6 4
16415 206/300 Read block 951 len 28672
16416 207/300 Write block 2650 len 16384 data: 5 4 5 5
16417 208/300 Read block 738 len 40960
16418 209/300 Read block 331 len 12288
16419 210/300 Write block 1488 len 16384 data: 2 2 2 3
16420 211/300 Read block 4135 len 4096
16421 212/300 Read block 590 len 36864
16422 213/300 Read block 4704 len 20480
16423 214/300 Write block 1350 len 12288 data: 3 2 2
16424 215/300 Read block 24 len 36864
16425 216/300 Write block 4156 len 4096 data: 5
16426 217/300 Read block 2317 len 24576
16427 218/300 Read block 2622 len 36864
16428 219/300 Write block 4463 len 8192 data: 2 2
16429 220/300 Read block 4628 len 24576
16430 221/300 Read block 1753 len 28672
16431 222/300 Write block 4391 len 24576 data: 6 5 5 6 4 6
16432 223/300 Write block 667 len 4096 data: 6
16433 224/300 Read block 3993 len 4096
16434 225/300 Read block 2471 len 20480
16435 226/300 Write block 1702 len 4096 data: 2
16436 227/300 Read block 4444 len 8192
16437 228/300 Write block 566 len 8192 data: 3 3
16438 229/300 Write block 2389 len 40960 data: 3 4 4 4 4 4 4 3 2 3
16439 230/300 Write block 46 len 8192 data: 4 4
16440 231/300 Write block 4485 len 4096 data: 2
16441 232/300 Write block 4823 len 24576 data: 3 3 5 5 6 5
16442 233/300 Write block 2959 len 28672 data: 4 4 4 3 2 2 4
16443 234/300 Read block 4379 len 24576
16444 235/300 Read block 1893 len 40960
16445 236/300 Write block 1089 len 28672 data: 4 4 3 3 3 3 6
16446 237/300 Write block 3732 len 8192 data: 3 3
16447 238/300 Write block 4326 len 12288 data: 4 4 4
16448 239/300 Write block 551 len 32768 data: 3 3 3 3 2 3 3 3
16449 240/300 Flush
16450 241/300 Read block 1287 len 36864
16451 242/300 Write block 804 len 40960 data: 2 3 3 3 3 3 4 5 8 9
16452 243/300 Read block 3065 len 24576
16453 244/300 Write block 1007 len 40960 data: 3 3 3 3 3 3 3 4 5 4
16454 245/300 Write block 3181 len 12288 data: 5 5 4
16455 246/300 Write block 2557 len 24576 data: 5 6 6 8 7 6
16456 247/300 Read block 4071 len 16384
16457 248/300 Read block 1302 len 24576
16458 249/300 Write block 2942 len 12288 data: 3 3 3
16459 250/300 Flush
16460 251/300 Read block 2408 len 36864
16461 252/300 Write block 417 len 36864 data: 6 6 6 4 3 3 3 3 2
16462 253/300 Write block 2704 len 32768 data: 3 5 4 4 5 5 3 3
16463 254/300 Read block 2982 len 20480
16464 255/300 Write block 3184 len 8192 data: 3 3
16465 256/300 Read block 82 len 36864
16466 257/300 Flush
16467 258/300 Read block 2168 len 28672
16468 259/300 Flush
16469 260/300 Read block 4166 len 8192
16470 261/300 Write block 4179 len 4096 data: 7
16471 262/300 Read block 1051 len 4096
16472 263/300 Write block 4204 len 32768 data: 5 4 4 4 3 3 4 4
16473 264/300 Write block 1720 len 20480 data: 3 3 4 3 4
16474 265/300 Read block 2113 len 20480
16475 266/300 Write block 4073 len 12288 data: 8 8 8
16476 267/300 Write block 1922 len 8192 data: 7 5
16477 268/300 Write block 2626 len 16384 data: 4 4 5 4
16478 269/300 Read block 780 len 24576
16479 270/300 Write block 1524 len 36864 data: 4 5 7 6 9 9 9 8 7
16480 271/300 Read block 4663 len 24576
16481 272/300 Write block 4755 len 24576 data: 6 6 6 6 7 6
16482 273/300 Write block 1795 len 4096 data: 3
16483 274/300 Write block 2790 len 24576 data: 4 5 5 5 6 5
16484 275/300 Read block 828 len 32768
16485 276/300 Read block 2823 len 24576
16486 277/300 Flush
16487 278/300 Read block 584 len 32768
16488 279/300 Write block 2272 len 4096 data: 7
16489 280/300 Flush
16490 281/300 Read block 3876 len 24576
16491 282/300 Read block 1519 len 4096
16492 283/300 Flush
16493 284/300 Flush
16494 285/300 Write block 4319 len 28672 data: 4 4 4 4 3 3 3
16495 286/300 Read block 4638 len 12288
16496 287/300 Write block 1293 len 28672 data: 3 3 3 3 4 5 5
16497 288/300 Read block 2664 len 16384
16498 289/300 Write block 2210 len 20480 data: 4 4 5 3 3
16499 290/300 Read block 3790 len 28672
16500 291/300 Read block 4640 len 16384
16501 292/300 Read block 4405 len 36864
16502 293/300 Read block 2931 len 16384
16503 294/300 Write block 1531 len 4096 data: 9
16504 295/300 Read block 3405 len 28672
16505 296/300 Write block 3829 len 16384 data: 3 3 3 4
16506 297/300 Flush
16507 298/300 Read block 2711 len 12288
16508 299/300 Write block 3045 len 8192 data: 2 2
16509 300/300 Read block 4975 len 36864
16510 Main task has work to do, go find it
16511 got dsc Start(2)
16512 start 2
16513 Mar 02 01:08:56.537 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6f7d8ca1-4b42-4d21-923e-932ad6dab9e1, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
16514 Replay: started 2, returned:Ok(())
16515 ----------------------------------------------------------------
16516 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
16517 States: Active Active Offline
16518 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16519 5428 Acked 6427 Read 7 Done Done New true
16520 5429 Acked 6428 Read 5 Done Done New false
16521 5430 Acked 6429 Flush 0 Done Done New false
16522 5431 Acked 6430 Read 2 Done Done New false
16523 5432 Acked 6431 Read 8 Done Done New false
16524 5433 Acked 6432 Write 2 Done Done New false
16525 5434 Acked 6433 Write 5 Done Done New false
16526 5435 Acked 6434 Read 7 Done Done New false
16527 5436 Acked 6435 Read 4 Done Done New false
16528 5437 Acked 6436 Read 2 Done Done New false
16529 5438 Acked 6437 Write 1 Done Done New false
16530 5439 Acked 6438 Write 6 Done Done New false
16531 5440 Acked 6439 Flush 0 Done Done New false
16532 5441 Acked 6440 Write 8 Done Done New false
16533 5442 Acked 6441 Flush 0 Done Done New false
16534 5443 Acked 6442 Read 10 Done Done New false
16535 5444 Acked 6443 Read 7 Done Done New false
16536 5445 Acked 6444 Read 7 Done Done New false
16537 5446 Acked 6445 Write 5 Done Done New false
16538 5447 Acked 6446 Write 5 Done Done New false
16539 5448 Acked 6447 Read 1 Done Done New false
16540 5449 Acked 6448 Read 8 Done Done New false
16541 5450 Acked 6449 Write 2 Done Done New false
16542 5451 Acked 6450 Flush 0 Done Done New false
16543 5452 Acked 6451 Read 10 Done Done New false
16544 5453 Acked 6452 Read 1 Done Done New false
16545 5454 Acked 6453 Read 8 Done Done New false
16546 5455 Acked 6454 Write 7 Done Done New false
16547 5456 Acked 6455 Flush 0 Done Done New false
16548 5457 Acked 6456 Read 4 Done Done New false
16549 5458 Acked 6457 Read 3 Done Done New false
16550 5459 Acked 6458 Write 1 Done Done New false
16551 5460 Acked 6459 Read 2 Done Done New false
16552 5461 Acked 6460 Flush 0 Done Done New false
16553 5462 Acked 6461 Read 5 Done Done New false
16554 5463 Acked 6462 Write 9 Done Done New false
16555 5464 Acked 6463 Write 9 Done Done New false
16556 5465 Acked 6464 Read 10 Done Done New false
16557 5466 Acked 6465 Write 3 Done Done New false
16558 5467 Acked 6466 Write 10 Done Done New false
16559 5468 Acked 6467 Read 2 Done Done New false
16560 5469 Acked 6468 Read 7 Done Done New false
16561 5470 Acked 6469 Flush 0 Done Done New false
16562 5471 Acked 6470 Write 7 Done Done New false
16563 5472 Acked 6471 Read 9 Done Done New false
16564 5473 Acked 6472 Read 1 Done Done New false
16565 5474 Acked 6473 Read 4 Done Done New false
16566 5475 Acked 6474 Read 2 Done Done New false
16567 5476 Acked 6475 Flush 0 Done Done New false
16568 5477 Acked 6476 Read 10 Done Done New false
16569 5478 Acked 6477 Write 1 Done Done New false
16570 5479 Acked 6478 Flush 0 Done Done New false
16571 5480 Acked 6479 Write 5 Done Done New false
16572 5481 Acked 6480 Read 3 Done Done New false
16573 5482 Acked 6481 Write 7 Done Done New false
16574 5483 Acked 6482 Write 2 Done Done New false
16575 5484 Acked 6483 Read 7 Done Done New false
16576 5485 Acked 6484 Read 1 Done Done New false
16577 5486 Acked 6485 Write 7 Done Done New false
16578 5487 Acked 6486 Read 5 Done Done New false
16579 5488 Acked 6487 Read 10 Done Done New false
16580 5489 Acked 6488 Read 4 Done Done New false
16581 5490 Acked 6489 Read 1 Done Done New false
16582 5491 Acked 6490 Flush 0 Done Done New false
16583 5492 Acked 6491 Flush 0 Done Done New false
16584 5493 Acked 6492 Read 6 Done Done New false
16585 5494 Acked 6493 Read 3 Done Done New false
16586 5495 Acked 6494 Write 7 Done Done New false
16587 5496 Acked 6495 Write 5 Done Done New false
16588 5497 Acked 6496 Read 4 Done Done New false
16589 5498 Acked 6497 Read 10 Done Done New false
16590 5499 Acked 6498 Read 2 Done Done New false
16591 5500 Acked 6499 Read 7 Done Done New false
16592 5501 Acked 6500 Read 8 Done Done New false
16593 5502 Acked 6501 Read 7 Done Done New false
16594 5503 Acked 6502 Read 6 Done Done New false
16595 5504 Acked 6503 Write 1 Done Done New false
16596 5505 Acked 6504 Read 10 Done Done New false
16597 5506 Acked 6505 Read 10 Done Done New false
16598 5507 Acked 6506 Read 6 Done Done New false
16599 5508 Acked 6507 Write 4 Done Done New false
16600 5509 Acked 6508 Read 7 Done Done New false
16601 5510 Acked 6509 Read 8 Done Done New false
16602 5511 Acked 6510 Write 8 Done Done New false
16603 5512 Acked 6511 Write 4 Done Done New false
16604 5513 Acked 6512 Write 9 Done Done New false
16605 5514 Acked 6513 Read 7 Done Done New false
16606 5515 Acked 6514 Write 1 Done Done New false
16607 5516 Acked 6515 Write 4 Done Done New false
16608 5517 Acked 6516 Read 6 Done Done New false
16609 5518 Acked 6517 Write 10 Done Done New false
16610 5519 Acked 6518 Read 10 Done Done New false
16611 5520 Acked 6519 Write 2 Done Done New false
16612 5521 Acked 6520 Read 7 Done Done New false
16613 5522 Acked 6521 Flush 0 Done Done New false
16614 5523 Acked 6522 Read 7 Done Done New false
16615 5524 Acked 6523 Read 7 Done Done New false
16616 5525 Acked 6524 Read 6 Done Done New false
16617 5526 Acked 6525 Write 8 Done Done New false
16618 5527 Acked 6526 Read 5 Done Done New false
16619 5528 Acked 6527 Write 5 Done Done New false
16620 5529 Acked 6528 Read 10 Done Done New false
16621 5530 Acked 6529 Write 10 Done Done New false
16622 5531 Acked 6530 Read 3 Done Done New false
16623 5532 Acked 6531 Write 2 Done Done New false
16624 5533 Acked 6532 Read 7 Done Done New false
16625 5534 Acked 6533 Write 7 Done Done New false
16626 5535 Acked 6534 Read 2 Done Done New false
16627 5536 Acked 6535 Read 2 Done Done New false
16628 5537 Acked 6536 Write 5 Done Done New false
16629 5538 Acked 6537 Flush 0 Done Done New false
16630 5539 Acked 6538 Read 3 Done Done New false
16631 5540 Acked 6539 Read 10 Done Done New false
16632 5541 Acked 6540 Read 8 Done Done New false
16633 5542 Acked 6541 Flush 0 Done Done New false
16634 5543 Acked 6542 Read 1 Done Done New false
16635 5544 Acked 6543 Write 6 Done Done New false
16636 5545 Acked 6544 Read 7 Done Done New false
16637 5546 Acked 6545 Read 2 Done Done New false
16638 5547 Acked 6546 Read 4 Done Done New false
16639 5548 Acked 6547 Read 5 Done Done New false
16640 5549 Acked 6548 Write 10 Done Done New false
16641 5550 Acked 6549 Read 1 Done Done New false
16642 5551 Acked 6550 Read 9 Done Done New false
16643 5552 Acked 6551 Write 10 Done Done New false
16644 5553 Acked 6552 Read 6 Done Done New false
16645 5554 Acked 6553 Read 2 Done Done New false
16646 5555 Acked 6554 Read 3 Done Done New false
16647 5556 Acked 6555 Read 6 Done Done New false
16648 5557 Acked 6556 Write 10 Done Done New false
16649 5558 Acked 6557 Read 5 Done Done New false
16650 5559 Acked 6558 Read 2 Done Done New false
16651 5560 Acked 6559 Read 4 Done Done New false
16652 5561 Acked 6560 Read 3 Done Done New false
16653 5562 Acked 6561 Flush 0 Done Done New false
16654 5563 Acked 6562 Flush 0 Done Done New false
16655 5564 Acked 6563 Write 10 Done Done New false
16656 5565 Acked 6564 Read 3 Done Done New false
16657 5566 Acked 6565 Read 6 Done Done New false
16658 5567 Acked 6566 Read 9 Done Done New false
16659 5568 Acked 6567 Read 6 Done Done New false
16660 5569 Acked 6568 Write 2 Done Done New false
16661 5570 Acked 6569 Write 10 Done Done New false
16662 5571 Acked 6570 Write 9 Done Done New false
16663 5572 Acked 6571 Read 9 Done Done New false
16664 5573 Acked 6572 Write 2 Done Done New false
16665 5574 Acked 6573 Flush 0 Done Done New false
16666 5575 Acked 6574 Write 2 Done Done New false
16667 5576 Acked 6575 Write 10 Done Done New false
16668 5577 Acked 6576 Read 4 Done Done New false
16669 5578 Acked 6577 Write 10 Done Done New false
16670 5579 Acked 6578 Write 7 Done Done New false
16671 5580 Acked 6579 Read 7 Done Done New false
16672 5581 Acked 6580 Write 4 Done Done New false
16673 5582 Acked 6581 Write 3 Done Done New false
16674 5583 Acked 6582 Read 3 Done Done New false
16675 5584 Acked 6583 Read 7 Done Done New false
16676 5585 Acked 6584 Write 8 Done Done New false
16677 5586 Acked 6585 Write 8 Done Done New false
16678 5587 Acked 6586 Read 9 Done Done New false
16679 5588 Acked 6587 Read 8 Done Done New false
16680 5589 Acked 6588 Write 6 Done Done New false
16681 5590 Acked 6589 Write 10 Done Done New false
16682 5591 Acked 6590 Write 4 Done Done New false
16683 5592 Acked 6591 Read 10 Done Done New false
16684 5593 Acked 6592 Write 10 Done Done New false
16685 5594 Acked 6593 Read 10 Done Done New false
16686 5595 Acked 6594 Flush 0 Done Done New false
16687 5596 Acked 6595 Write 8 Done Done New false
16688 5597 Acked 6596 Read 4 Done Done New false
16689 5598 Acked 6597 Write 2 Done Done New false
16690 5599 Acked 6598 Read 1 Done Done New false
16691 5600 Acked 6599 Read 9 Done Done New false
16692 5601 Acked 6600 Write 5 Done Done New false
16693 5602 Acked 6601 Read 4 Done Done New false
16694 5603 Acked 6602 Flush 0 Done Done New false
16695 5604 Acked 6603 Write 8 Done Done New false
16696 5605 Acked 6604 Read 1 Done Done New false
16697 5606 Acked 6605 Flush 0 Done Done New false
16698 5607 Acked 6606 Read 5 Done Done New false
16699 5608 Acked 6607 Read 8 Done Done New false
16700 5609 Acked 6608 Read 3 Done Done New false
16701 5610 Acked 6609 Flush 0 Done Done New false
16702 5611 Acked 6610 Read 6 Done Done New false
16703 5612 Acked 6611 Write 8 Done Done New false
16704 5613 Acked 6612 Write 1 Done Done New false
16705 5614 Acked 6613 Read 10 Done Done New false
16706 5615 Acked 6614 Read 7 Done Done New false
16707 5616 Acked 6615 Write 4 Done Done New false
16708 5617 Acked 6616 Flush 0 Done Done New false
16709 5618 Acked 6617 Read 9 Done Done New false
16710 5619 Acked 6618 Write 2 Done Done New false
16711 5620 Acked 6619 Write 1 Done Done New false
16712 5621 Acked 6620 Read 1 Done Done New false
16713 5622 Acked 6621 Write 6 Done Done New false
16714 5623 Acked 6622 Write 10 Done Done New false
16715 5624 Acked 6623 Write 6 Done Done New false
16716 5625 Acked 6624 Read 10 Done Done New false
16717 5626 Acked 6625 Read 5 Done Done New false
16718 5627 Acked 6626 Read 2 Done Done New false
16719 5628 Acked 6627 Read 3 Done Done New false
16720 5629 Acked 6628 Write 2 Done Done New false
16721 5630 Acked 6629 Read 9 Done Done New false
16722 5631 Acked 6630 Write 3 Done Done New false
16723 5632 Acked 6631 Write 7 Done Done New false
16724 5633 Acked 6632 Read 7 Done Done New false
16725 5634 Acked 6633 Write 4 Done Done New false
16726 5635 Acked 6634 Read 10 Done Done New false
16727 5636 Acked 6635 Read 3 Done Done New false
16728 5637 Acked 6636 Write 4 Done Done New false
16729 5638 Acked 6637 Read 1 Done Done New false
16730 5639 Acked 6638 Read 9 Done Done New false
16731 5640 Acked 6639 Read 5 Done Done New false
16732 5641 Acked 6640 Write 3 Done Done New false
16733 5642 Acked 6641 Read 9 Done Done New false
16734 5643 Acked 6642 Write 1 Done Done New false
16735 5644 Acked 6643 Read 6 Done Done New false
16736 5645 Acked 6644 Read 9 Done Done New false
16737 5646 Acked 6645 Write 2 Done Done New false
16738 5647 Acked 6646 Read 6 Done Done New false
16739 5648 Acked 6647 Read 7 Done Done New false
16740 5649 Acked 6648 Write 6 Done Done New false
16741 5650 Acked 6649 Write 1 Done Done New false
16742 5651 Acked 6650 Read 1 Done Done New false
16743 5652 Acked 6651 Read 5 Done Done New false
16744 5653 Acked 6652 Write 1 Done Done New false
16745 5654 Acked 6653 Read 2 Done Done New false
16746 5655 Acked 6654 Write 2 Done Done New false
16747 5656 Acked 6655 Write 10 Done Done New false
16748 5657 Acked 6656 Write 2 Done Done New false
16749 5658 Acked 6657 Write 1 Done Done New false
16750 5659 Acked 6658 Write 6 Done Done New false
16751 5660 Acked 6659 Write 7 Done Done New false
16752 5661 Acked 6660 Read 6 Done Done New false
16753 5662 Acked 6661 Read 10 Done Done New false
16754 5663 Acked 6662 Write 7 Done Done New false
16755 5664 Acked 6663 Write 2 Done Done New false
16756 5665 Acked 6664 Write 3 Done Done New false
16757 5666 Acked 6665 Write 8 Done Done New false
16758 5667 Acked 6666 Flush 0 Done Done New false
16759 5668 Acked 6667 Read 9 Done Done New false
16760 5669 Acked 6668 Write 10 Done Done New false
16761 5670 Acked 6669 Read 6 Done Done New false
16762 5671 Acked 6670 Write 10 Done Done New false
16763 5672 Acked 6671 Write 3 Done Done New false
16764 5673 Acked 6672 Write 6 Done Done New false
16765 5674 Acked 6673 Read 4 Done Done New false
16766 5675 Acked 6674 Read 6 Done Done New false
16767 5676 Acked 6675 Write 3 Done Done New false
16768 5677 Acked 6676 Flush 0 Done Done New false
16769 5678 Acked 6677 Read 9 Done Done New false
16770 5679 Acked 6678 Write 9 Done Done New false
16771 5680 Acked 6679 Write 8 Done Done New false
16772 5681 Acked 6680 Read 5 Done Done New false
16773 5682 Acked 6681 Write 2 Done Done New false
16774 5683 Acked 6682 Read 9 Done Done New false
16775 5684 Acked 6683 Flush 0 Done Done New false
16776 5685 Acked 6684 Read 7 Done Done New false
16777 5686 Acked 6685 Flush 0 Done Done New false
16778 5687 Acked 6686 Read 2 Done Done New false
16779 5688 Acked 6687 Write 1 Done Done New false
16780 5689 Acked 6688 Read 1 Done Done New false
16781 5690 Acked 6689 Write 8 Done Done New false
16782 5691 Acked 6690 Write 5 Done Done New false
16783 5692 Acked 6691 Read 5 Done Done New false
16784 5693 Acked 6692 Write 3 Done Done New false
16785 5694 Acked 6693 Write 2 Done Done New false
16786 5695 Acked 6694 Write 4 Done Done New false
16787 5696 Acked 6695 Read 6 Done Done New false
16788 5697 Acked 6696 Write 9 Done Done New false
16789 5698 Acked 6697 Read 6 Done Done New false
16790 5699 Acked 6698 Write 6 Done Done New false
16791 5700 Acked 6699 Write 1 Done Done New false
16792 5701 Acked 6700 Write 6 Done Done New false
16793 5702 Acked 6701 Read 8 Done Done New false
16794 5703 Acked 6702 Read 6 Done Done New false
16795 5704 Acked 6703 Flush 0 Done Done New false
16796 5705 Acked 6704 Read 8 Done Done New false
16797 5706 Acked 6705 Write 1 Done Done New false
16798 5707 Acked 6706 Flush 0 Done Done New false
16799 5708 Acked 6707 Read 6 Done Done New false
16800 5709 Acked 6708 Read 1 Done Done New false
16801 5710 Acked 6709 Flush 0 Done Done New false
16802 5711 Acked 6710 Flush 0 Done Done New false
16803 5712 Acked 6711 Write 7 Done Done New false
16804 5713 Acked 6712 Read 3 Done Done New false
16805 5714 Acked 6713 Write 7 Done Done New false
16806 5715 Acked 6714 Read 4 Done Done New false
16807 5716 Acked 6715 Write 5 Done Done New false
16808 5717 Acked 6716 Read 7 Done Done New false
16809 5718 Acked 6717 Read 4 Done Done New false
16810 5719 Acked 6718 Read 9 Done Done New false
16811 5720 Acked 6719 Read 4 Done Done New false
16812 5721 Acked 6720 Write 1 Done Done New false
16813 5722 Acked 6721 Read 7 Done Done New false
16814 5723 Acked 6722 Write 4 Done Done New false
16815 5724 Acked 6723 Flush 0 Done Done New false
16816 5725 Acked 6724 Read 3 Done Done New false
16817 5726 Acked 6725 Write 2 Done Done New false
16818 5727 Acked 6726 Read 9 Done Done New false
16819 STATES DS:0 DS:1 DS:2 TOTAL
16820 New 0 0 300 300
16821 Sent 0 0 0 0
16822 Done 300 300 0 600
16823 Skipped 0 0 0 0
16824 Error 0 0 0 0
16825 Last Flush: 6723 6723 6426
16826 Downstairs last five completed: 6426 6425 6424 6423 6422
16827 Upstairs last five completed: 5727 5726 5725 5724 5723
16828 CLIENT: Up:0 ds:300 act:2
168292024-03-02T01:08:57.187ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168302024-03-02T01:08:57.187ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168312024-03-02T01:08:57.187ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168322024-03-02T01:08:57.187ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168332024-03-02T01:08:57.187ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168342024-03-02T01:08:57.187ZINFOcrucible: [2] client re-new 301 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168352024-03-02T01:08:57.187ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168362024-03-02T01:08:58.188ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168372024-03-02T01:08:58.188ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168382024-03-02T01:08:58.188ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168392024-03-02T01:08:58.188ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168402024-03-02T01:08:58.188ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168412024-03-02T01:08:58.188ZINFOcrucible: [2] client re-new 301 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168422024-03-02T01:08:58.188ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168432024-03-02T01:08:59.189ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168442024-03-02T01:08:59.189ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168452024-03-02T01:08:59.189ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168462024-03-02T01:08:59.189ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168472024-03-02T01:08:59.189ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168482024-03-02T01:08:59.189ZINFOcrucible: [2] client re-new 301 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168492024-03-02T01:08:59.190ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16850 [8830] got start action
16851 [8830] I am going to restart
16852 Starting downstairs at port 8830
16853 Make output file at "/tmp/dsc/downstairs-8830.txt"
16854 [8830][2] reports Starting
168552024-03-02T01:09:00.191ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168562024-03-02T01:09:00.191ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168572024-03-02T01:09:00.191ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168582024-03-02T01:09:00.191ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168592024-03-02T01:09:00.191ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168602024-03-02T01:09:00.191ZINFOcrucible: [2] client re-new 301 jobs since flush 6426 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
168612024-03-02T01:09:00.191ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
16862 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1181)
16863 [8830][2] reports Running
16864 ----------------------------------------------------------------
16865 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16866 States: Active Active Offline
16867 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16868 5428 Acked 6427 Read 7 Done Done New true
16869 5429 Acked 6428 Read 5 Done Done New true
16870 5430 Acked 6429 Flush 0 Done Done New true
16871 5431 Acked 6430 Read 2 Done Done New true
16872 5432 Acked 6431 Read 8 Done Done New true
16873 5433 Acked 6432 Write 2 Done Done New true
16874 5434 Acked 6433 Write 5 Done Done New true
16875 5435 Acked 6434 Read 7 Done Done New true
16876 5436 Acked 6435 Read 4 Done Done New true
16877 5437 Acked 6436 Read 2 Done Done New true
16878 5438 Acked 6437 Write 1 Done Done New true
16879 5439 Acked 6438 Write 6 Done Done New true
16880 5440 Acked 6439 Flush 0 Done Done New true
16881 5441 Acked 6440 Write 8 Done Done New true
16882 5442 Acked 6441 Flush 0 Done Done New true
16883 5443 Acked 6442 Read 10 Done Done New true
16884 5444 Acked 6443 Read 7 Done Done New true
16885 5445 Acked 6444 Read 7 Done Done New true
16886 5446 Acked 6445 Write 5 Done Done New true
16887 5447 Acked 6446 Write 5 Done Done New true
16888 5448 Acked 6447 Read 1 Done Done New true
16889 5449 Acked 6448 Read 8 Done Done New true
16890 5450 Acked 6449 Write 2 Done Done New true
16891 5451 Acked 6450 Flush 0 Done Done New true
16892 5452 Acked 6451 Read 10 Done Done New true
16893 5453 Acked 6452 Read 1 Done Done New true
16894 5454 Acked 6453 Read 8 Done Done New true
16895 5455 Acked 6454 Write 7 Done Done New true
16896 5456 Acked 6455 Flush 0 Done Done New true
16897 5457 Acked 6456 Read 4 Done Done New true
16898 5458 Acked 6457 Read 3 Done Done New true
16899 5459 Acked 6458 Write 1 Done Done New true
16900 5460 Acked 6459 Read 2 Done Done New true
16901 5461 Acked 6460 Flush 0 Done Done New true
16902 5462 Acked 6461 Read 5 Done Done New true
16903 5463 Acked 6462 Write 9 Done Done New true
16904 5464 Acked 6463 Write 9 Done Done New true
16905 5465 Acked 6464 Read 10 Done Done New true
16906 5466 Acked 6465 Write 3 Done Done New true
16907 5467 Acked 6466 Write 10 Done Done New true
16908 5468 Acked 6467 Read 2 Done Done New true
16909 5469 Acked 6468 Read 7 Done Done New true
16910 5470 Acked 6469 Flush 0 Done Done New true
16911 5471 Acked 6470 Write 7 Done Done New true
16912 5472 Acked 6471 Read 9 Done Done New true
16913 5473 Acked 6472 Read 1 Done Done New true
16914 5474 Acked 6473 Read 4 Done Done New true
16915 5475 Acked 6474 Read 2 Done Done New true
16916 5476 Acked 6475 Flush 0 Done Done New true
16917 5477 Acked 6476 Read 10 Done Done New true
16918 5478 Acked 6477 Write 1 Done Done New true
16919 5479 Acked 6478 Flush 0 Done Done New true
16920 5480 Acked 6479 Write 5 Done Done New true
16921 5481 Acked 6480 Read 3 Done Done New true
16922 5482 Acked 6481 Write 7 Done Done New true
16923 5483 Acked 6482 Write 2 Done Done New true
16924 5484 Acked 6483 Read 7 Done Done New true
16925 5485 Acked 6484 Read 1 Done Done New true
16926 5486 Acked 6485 Write 7 Done Done New true
16927 5487 Acked 6486 Read 5 Done Done New true
16928 5488 Acked 6487 Read 10 Done Done New true
16929 5489 Acked 6488 Read 4 Done Done New true
16930 5490 Acked 6489 Read 1 Done Done New true
16931 5491 Acked 6490 Flush 0 Done Done New true
16932 5492 Acked 6491 Flush 0 Done Done New true
16933 5493 Acked 6492 Read 6 Done Done New true
16934 5494 Acked 6493 Read 3 Done Done New true
16935 5495 Acked 6494 Write 7 Done Done New true
16936 5496 Acked 6495 Write 5 Done Done New true
16937 5497 Acked 6496 Read 4 Done Done New true
16938 5498 Acked 6497 Read 10 Done Done New true
16939 5499 Acked 6498 Read 2 Done Done New true
16940 5500 Acked 6499 Read 7 Done Done New true
16941 5501 Acked 6500 Read 8 Done Done New true
16942 5502 Acked 6501 Read 7 Done Done New true
16943 5503 Acked 6502 Read 6 Done Done New true
16944 5504 Acked 6503 Write 1 Done Done New true
16945 5505 Acked 6504 Read 10 Done Done New true
16946 5506 Acked 6505 Read 10 Done Done New true
16947 5507 Acked 6506 Read 6 Done Done New true
16948 5508 Acked 6507 Write 4 Done Done New true
16949 5509 Acked 6508 Read 7 Done Done New true
16950 5510 Acked 6509 Read 8 Done Done New true
16951 5511 Acked 6510 Write 8 Done Done New true
16952 5512 Acked 6511 Write 4 Done Done New true
16953 5513 Acked 6512 Write 9 Done Done New true
16954 5514 Acked 6513 Read 7 Done Done New true
16955 5515 Acked 6514 Write 1 Done Done New true
16956 5516 Acked 6515 Write 4 Done Done New true
16957 5517 Acked 6516 Read 6 Done Done New true
16958 5518 Acked 6517 Write 10 Done Done New true
16959 5519 Acked 6518 Read 10 Done Done New true
16960 5520 Acked 6519 Write 2 Done Done New true
16961 5521 Acked 6520 Read 7 Done Done New true
16962 5522 Acked 6521 Flush 0 Done Done New true
16963 5523 Acked 6522 Read 7 Done Done New true
16964 5524 Acked 6523 Read 7 Done Done New true
16965 5525 Acked 6524 Read 6 Done Done New true
16966 5526 Acked 6525 Write 8 Done Done New true
16967 5527 Acked 6526 Read 5 Done Done New true
16968 5528 Acked 6527 Write 5 Done Done New true
16969 5529 Acked 6528 Read 10 Done Done New true
16970 5530 Acked 6529 Write 10 Done Done New true
16971 5531 Acked 6530 Read 3 Done Done New true
16972 5532 Acked 6531 Write 2 Done Done New true
16973 5533 Acked 6532 Read 7 Done Done New true
16974 5534 Acked 6533 Write 7 Done Done New true
16975 5535 Acked 6534 Read 2 Done Done New true
16976 5536 Acked 6535 Read 2 Done Done New true
16977 5537 Acked 6536 Write 5 Done Done New true
16978 5538 Acked 6537 Flush 0 Done Done New true
16979 5539 Acked 6538 Read 3 Done Done New true
16980 5540 Acked 6539 Read 10 Done Done New true
16981 5541 Acked 6540 Read 8 Done Done New true
16982 5542 Acked 6541 Flush 0 Done Done New true
16983 5543 Acked 6542 Read 1 Done Done New true
16984 5544 Acked 6543 Write 6 Done Done New true
16985 5545 Acked 6544 Read 7 Done Done New true
16986 5546 Acked 6545 Read 2 Done Done New true
16987 5547 Acked 6546 Read 4 Done Done New true
16988 5548 Acked 6547 Read 5 Done Done New true
16989 5549 Acked 6548 Write 10 Done Done New true
16990 5550 Acked 6549 Read 1 Done Done New true
16991 5551 Acked 6550 Read 9 Done Done New true
16992 5552 Acked 6551 Write 10 Done Done New true
16993 5553 Acked 6552 Read 6 Done Done New true
16994 5554 Acked 6553 Read 2 Done Done New true
16995 5555 Acked 6554 Read 3 Done Done New true
16996 5556 Acked 6555 Read 6 Done Done New true
16997 5557 Acked 6556 Write 10 Done Done New true
16998 5558 Acked 6557 Read 5 Done Done New true
16999 5559 Acked 6558 Read 2 Done Done New true
17000 5560 Acked 6559 Read 4 Done Done New true
17001 5561 Acked 6560 Read 3 Done Done New true
17002 5562 Acked 6561 Flush 0 Done Done New true
17003 5563 Acked 6562 Flush 0 Done Done New true
17004 5564 Acked 6563 Write 10 Done Done New true
17005 5565 Acked 6564 Read 3 Done Done New true
17006 5566 Acked 6565 Read 6 Done Done New true
17007 5567 Acked 6566 Read 9 Done Done New true
17008 5568 Acked 6567 Read 6 Done Done New true
17009 5569 Acked 6568 Write 2 Done Done New true
17010 5570 Acked 6569 Write 10 Done Done New true
17011 5571 Acked 6570 Write 9 Done Done New true
17012 5572 Acked 6571 Read 9 Done Done New true
17013 5573 Acked 6572 Write 2 Done Done New true
17014 5574 Acked 6573 Flush 0 Done Done New true
17015 5575 Acked 6574 Write 2 Done Done New true
17016 5576 Acked 6575 Write 10 Done Done New true
17017 5577 Acked 6576 Read 4 Done Done New true
17018 5578 Acked 6577 Write 10 Done Done New true
17019 5579 Acked 6578 Write 7 Done Done New true
17020 5580 Acked 6579 Read 7 Done Done New true
17021 5581 Acked 6580 Write 4 Done Done New true
17022 5582 Acked 6581 Write 3 Done Done New true
17023 5583 Acked 6582 Read 3 Done Done New true
17024 5584 Acked 6583 Read 7 Done Done New true
17025 5585 Acked 6584 Write 8 Done Done New true
17026 5586 Acked 6585 Write 8 Done Done New true
17027 5587 Acked 6586 Read 9 Done Done New true
17028 5588 Acked 6587 Read 8 Done Done New true
17029 5589 Acked 6588 Write 6 Done Done New true
17030 5590 Acked 6589 Write 10 Done Done New true
17031 5591 Acked 6590 Write 4 Done Done New true
17032 5592 Acked 6591 Read 10 Done Done New true
17033 5593 Acked 6592 Write 10 Done Done New true
17034 5594 Acked 6593 Read 10 Done Done New true
17035 5595 Acked 6594 Flush 0 Done Done New true
17036 5596 Acked 6595 Write 8 Done Done New true
17037 5597 Acked 6596 Read 4 Done Done New true
17038 5598 Acked 6597 Write 2 Done Done New true
17039 5599 Acked 6598 Read 1 Done Done New true
17040 5600 Acked 6599 Read 9 Done Done New true
17041 5601 Acked 6600 Write 5 Done Done New true
17042 5602 Acked 6601 Read 4 Done Done New true
17043 5603 Acked 6602 Flush 0 Done Done New true
17044 5604 Acked 6603 Write 8 Done Done New true
17045 5605 Acked 6604 Read 1 Done Done New true
17046 5606 Acked 6605 Flush 0 Done Done New true
17047 5607 Acked 6606 Read 5 Done Done New true
17048 5608 Acked 6607 Read 8 Done Done New true
17049 5609 Acked 6608 Read 3 Done Done New true
17050 5610 Acked 6609 Flush 0 Done Done New true
17051 5611 Acked 6610 Read 6 Done Done New true
17052 5612 Acked 6611 Write 8 Done Done New true
17053 5613 Acked 6612 Write 1 Done Done New true
17054 5614 Acked 6613 Read 10 Done Done New true
17055 5615 Acked 6614 Read 7 Done Done New true
17056 5616 Acked 6615 Write 4 Done Done New true
17057 5617 Acked 6616 Flush 0 Done Done New true
17058 5618 Acked 6617 Read 9 Done Done New true
17059 5619 Acked 6618 Write 2 Done Done New true
17060 5620 Acked 6619 Write 1 Done Done New true
17061 5621 Acked 6620 Read 1 Done Done New true
17062 5622 Acked 6621 Write 6 Done Done New true
17063 5623 Acked 6622 Write 10 Done Done New true
17064 5624 Acked 6623 Write 6 Done Done New true
17065 5625 Acked 6624 Read 10 Done Done New true
17066 5626 Acked 6625 Read 5 Done Done New true
17067 5627 Acked 6626 Read 2 Done Done New true
17068 5628 Acked 6627 Read 3 Done Done New true
17069 5629 Acked 6628 Write 2 Done Done New true
17070 5630 Acked 6629 Read 9 Done Done New true
17071 5631 Acked 6630 Write 3 Done Done New true
17072 5632 Acked 6631 Write 7 Done Done New true
17073 5633 Acked 6632 Read 7 Done Done New true
17074 5634 Acked 6633 Write 4 Done Done New true
17075 5635 Acked 6634 Read 10 Done Done New true
17076 5636 Acked 6635 Read 3 Done Done New true
17077 5637 Acked 6636 Write 4 Done Done New true
17078 5638 Acked 6637 Read 1 Done Done New true
17079 5639 Acked 6638 Read 9 Done Done New true
17080 5640 Acked 6639 Read 5 Done Done New true
17081 5641 Acked 6640 Write 3 Done Done New true
17082 5642 Acked 6641 Read 9 Done Done New true
17083 5643 Acked 6642 Write 1 Done Done New true
17084 5644 Acked 6643 Read 6 Done Done New true
17085 5645 Acked 6644 Read 9 Done Done New true
17086 5646 Acked 6645 Write 2 Done Done New true
17087 5647 Acked 6646 Read 6 Done Done New true
17088 5648 Acked 6647 Read 7 Done Done New true
17089 5649 Acked 6648 Write 6 Done Done New true
17090 5650 Acked 6649 Write 1 Done Done New true
17091 5651 Acked 6650 Read 1 Done Done New true
17092 5652 Acked 6651 Read 5 Done Done New true
17093 5653 Acked 6652 Write 1 Done Done New true
17094 5654 Acked 6653 Read 2 Done Done New true
17095 5655 Acked 6654 Write 2 Done Done New true
17096 5656 Acked 6655 Write 10 Done Done New true
17097 5657 Acked 6656 Write 2 Done Done New true
17098 5658 Acked 6657 Write 1 Done Done New true
17099 5659 Acked 6658 Write 6 Done Done New true
17100 5660 Acked 6659 Write 7 Done Done New true
17101 5661 Acked 6660 Read 6 Done Done New true
17102 5662 Acked 6661 Read 10 Done Done New true
17103 5663 Acked 6662 Write 7 Done Done New true
17104 5664 Acked 6663 Write 2 Done Done New true
17105 5665 Acked 6664 Write 3 Done Done New true
17106 5666 Acked 6665 Write 8 Done Done New true
17107 5667 Acked 6666 Flush 0 Done Done New true
17108 5668 Acked 6667 Read 9 Done Done New true
17109 5669 Acked 6668 Write 10 Done Done New true
17110 5670 Acked 6669 Read 6 Done Done New true
17111 5671 Acked 6670 Write 10 Done Done New true
17112 5672 Acked 6671 Write 3 Done Done New true
17113 5673 Acked 6672 Write 6 Done Done New true
17114 5674 Acked 6673 Read 4 Done Done New true
17115 5675 Acked 6674 Read 6 Done Done New true
17116 5676 Acked 6675 Write 3 Done Done New true
17117 5677 Acked 6676 Flush 0 Done Done New true
17118 5678 Acked 6677 Read 9 Done Done New true
17119 5679 Acked 6678 Write 9 Done Done New true
17120 5680 Acked 6679 Write 8 Done Done New true
17121 5681 Acked 6680 Read 5 Done Done New true
17122 5682 Acked 6681 Write 2 Done Done New true
17123 5683 Acked 6682 Read 9 Done Done New true
17124 5684 Acked 6683 Flush 0 Done Done New true
17125 5685 Acked 6684 Read 7 Done Done New true
17126 5686 Acked 6685 Flush 0 Done Done New true
17127 5687 Acked 6686 Read 2 Done Done New true
17128 5688 Acked 6687 Write 1 Done Done New true
17129 5689 Acked 6688 Read 1 Done Done New true
17130 5690 Acked 6689 Write 8 Done Done New true
17131 5691 Acked 6690 Write 5 Done Done New true
17132 5692 Acked 6691 Read 5 Done Done New true
17133 5693 Acked 6692 Write 3 Done Done New true
17134 5694 Acked 6693 Write 2 Done Done New true
17135 5695 Acked 6694 Write 4 Done Done New true
17136 5696 Acked 6695 Read 6 Done Done New true
17137 5697 Acked 6696 Write 9 Done Done New true
17138 5698 Acked 6697 Read 6 Done Done New true
17139 5699 Acked 6698 Write 6 Done Done New true
17140 5700 Acked 6699 Write 1 Done Done New true
17141 5701 Acked 6700 Write 6 Done Done New true
17142 5702 Acked 6701 Read 8 Done Done New true
17143 5703 Acked 6702 Read 6 Done Done New true
17144 5704 Acked 6703 Flush 0 Done Done New true
17145 5705 Acked 6704 Read 8 Done Done New true
17146 5706 Acked 6705 Write 1 Done Done New true
17147 5707 Acked 6706 Flush 0 Done Done New true
17148 5708 Acked 6707 Read 6 Done Done New true
17149 5709 Acked 6708 Read 1 Done Done New true
17150 5710 Acked 6709 Flush 0 Done Done New true
17151 5711 Acked 6710 Flush 0 Done Done New true
17152 5712 Acked 6711 Write 7 Done Done New true
17153 5713 Acked 6712 Read 3 Done Done New true
17154 5714 Acked 6713 Write 7 Done Done New true
17155 5715 Acked 6714 Read 4 Done Done New true
17156 5716 Acked 6715 Write 5 Done Done New true
17157 5717 Acked 6716 Read 7 Done Done New true
17158 5718 Acked 6717 Read 4 Done Done New true
17159 5719 Acked 6718 Read 9 Done Done New true
17160 5720 Acked 6719 Read 4 Done Done New true
17161 5721 Acked 6720 Write 1 Done Done New true
17162 5722 Acked 6721 Read 7 Done Done New true
17163 5723 Acked 6722 Write 4 Done Done New true
17164 5724 Acked 6723 Flush 0 Done Done New true
17165 5725 Acked 6724 Read 3 Done Done New true
17166 5726 Acked 6725 Write 2 Done Done New true
17167 5727 Acked 6726 Read 9 Done Done New true
17168 5728 Acked 6727 Flush 0 Done Done New true
17169 STATES DS:0 DS:1 DS:2 TOTAL
17170 New 0 0 301 301
17171 Sent 0 0 0 0
17172 Done 301 301 0 602
17173 Skipped 0 0 0 0
17174 Error 0 0 0 0
17175 Last Flush: 6727 6727 6426
17176 Downstairs last five completed: 6426 6425 6424 6423 6422
17177 Upstairs last five completed: 5728 5727 5726 5725 5724
17178 CLIENT: Up:0 ds:301 act:2
171792024-03-02T01:09:01.193ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171802024-03-02T01:09:01.193ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171812024-03-02T01:09:01.194ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171822024-03-02T01:09:01.194ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171832024-03-02T01:09:01.194ZINFOcrucible: send last flush ID to this DS: 6426 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171842024-03-02T01:09:01.194ZINFOcrucible: Replied this last flush ID: 6426 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171852024-03-02T01:09:01.194ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171862024-03-02T01:09:01.194ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171872024-03-02T01:09:01.194ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
171882024-03-02T01:09:01.195ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17189 ----------------------------------------------------------------
17190 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17191 Downstairs last five completed: 6727 6726 6725 6724 6723
17192 Upstairs last five completed: 5728 5727 5726 5725 5724
17193 CLIENT: Up:0 ds:0 act:3
17194 Replay: All jobs finished, all DS active.
17195 Main task has work to do, go find it
17196 got dsc Stop(2)
17197 stop 2
17198 Mar 02 01:09:04.543 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/2, method: POST, req_id:[8830] Got stop action so:false kr:false
17199 d77b0098-c8ad-4cd3-8027-7e3d8d214aef, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
172002024-03-02T01:09:04.543ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172012024-03-02T01:09:04.543ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17202 Mar 02 01:09:04.543 INFO request completed, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:09:04.543929405Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
17203 latency_us: 106, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7fa57550-a205-4f7a-96ff-8363ef6245b4, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
172042024-03-02T01:09:04.544ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172052024-03-02T01:09:04.544ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172062024-03-02T01:09:04.544ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17207 [8830] Exited with: ExitStatus(unix_wait_status(9))
17208 [8830][2] reports Exit
172092024-03-02T01:09:05.545ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172102024-03-02T01:09:05.545ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172112024-03-02T01:09:05.545ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172122024-03-02T01:09:05.545ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172132024-03-02T01:09:05.545ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172142024-03-02T01:09:05.545ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172152024-03-02T01:09:05.545ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172162024-03-02T01:09:06.546ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172172024-03-02T01:09:06.546ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172182024-03-02T01:09:06.546ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172192024-03-02T01:09:06.546ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172202024-03-02T01:09:06.546ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172212024-03-02T01:09:06.546ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172222024-03-02T01:09:06.546ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172232024-03-02T01:09:07.547ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172242024-03-02T01:09:07.547ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172252024-03-02T01:09:07.547ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172262024-03-02T01:09:07.547ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172272024-03-02T01:09:07.547ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172282024-03-02T01:09:07.547ZINFOcrucible: [2] client re-new 0 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172292024-03-02T01:09:07.547ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17230 Mar 02 01:09:08.545 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/2, method: GET, req_id: d06fae19-fad8-4e74-aa57-d9b90de83d68, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
17231 001/300 Write block 2029 len 12288 data: 3 4 3
17232 002/300 Write block 263 len 28672 data: 6 6 6 8 6 5 6
17233 003/300 Write block 2392 len 20480 data: 5 5 5 5 4
17234 004/300 Write block 3572 len 36864 data: 5 5 4 5 4 3 3 3 2
172352024-03-02T01:09:08.548ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172362024-03-02T01:09:08.548ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172372024-03-02T01:09:08.548ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172382024-03-02T01:09:08.548ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172392024-03-02T01:09:08.548ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
172402024-03-02T01:09:08.548ZINFOcrucible: [2] client re-new 3 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17241 005/300 Read block 785 len 24576
172422024-03-02T01:09:08.548ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17243 006/300 Flush
17244 007/300 Read block 3593 len 16384
17245 008/300 Flush
17246 009/300 Flush
17247 010/300 Write block 3126 len 40960 data: 3 2 2 2 2 2 2 2 3 4
17248 011/300 Read block 1154 len 40960
17249 012/300 Read block 1229 len 36864
17250 013/300 Read block 4380 len 8192
17251 014/300 Read block 1247 len 16384
17252 015/300 Read block 1126 len 8192
17253 016/300 Write block 255 len 4096 data: 4
17254 017/300 Write block 2105 len 24576 data: 4 3 3 3 2 2
17255 018/300 Read block 476 len 8192
17256 019/300 Write block 4069 len 12288 data: 4 4 4
17257 020/300 Read block 3350 len 20480
17258 021/300 Write block 2854 len 32768 data: 4 4 4 4 3 4 2 2
17259 022/300 Read block 3496 len 16384
17260 023/300 Read block 1795 len 8192
17261 024/300 Read block 3332 len 24576
17262 025/300 Write block 3444 len 8192 data: 5 6
17263 026/300 Write block 347 len 28672 data: 7 4 4 4 3 3 3
17264 027/300 Write block 2950 len 24576 data: 6 5 5 5 5 6
17265 028/300 Read block 1607 len 32768
17266 029/300 Write block 2377 len 24576 data: 3 3 3 2 2 3
17267 030/300 Read block 3944 len 40960
17268 031/300 Read block 2167 len 16384
17269 032/300 Read block 2901 len 4096
17270 033/300 Write block 4590 len 16384 data: 6 6 6 6
17271 034/300 Flush
17272 035/300 Read block 2202 len 40960
17273 036/300 Read block 3879 len 12288
17274 037/300 Read block 1410 len 28672
17275 038/300 Write block 2423 len 12288 data: 4 4 3
17276 039/300 Flush
17277 040/300 Read block 3364 len 36864
17278 041/300 Read block 1676 len 32768
17279 042/300 Flush
17280 043/300 Write block 3332 len 4096 data: 5
17281 044/300 Read block 4805 len 16384
17282 045/300 Flush
17283 046/300 Read block 661 len 32768
17284 047/300 Read block 1952 len 4096
17285 048/300 Write block 2876 len 32768 data: 6 5 6 4 4 3 3 2
17286 049/300 Read block 2268 len 32768
17287 050/300 Read block 2442 len 24576
17288 051/300 Read block 3392 len 4096
17289 052/300 Write block 1792 len 36864 data: 3 3 3 4 3 4 3 3 5
17290 053/300 Read block 3230 len 28672
17291 054/300 Flush
17292 055/300 Read block 2129 len 4096
17293 056/300 Write block 842 len 4096 data: 4
17294 057/300 Write block 3565 len 16384 data: 7 7 6 7
17295 058/300 Write block 3095 len 24576 data: 6 6 6 5 4 4
17296 059/300 Write block 3290 len 12288 data: 4 3 3
17297 060/300 Read block 1564 len 20480
17298 061/300 Write block 956 len 24576 data: 3 3 3 3 3 4
17299 062/300 Read block 4832 len 28672
17300 063/300 Read block 343 len 36864
17301 064/300 Read block 1858 len 20480
17302 065/300 Read block 4431 len 28672
17303 066/300 Read block 3398 len 12288
17304 067/300 Write block 3925 len 24576 data: 7 9 10 7 7 7
17305 068/300 Write block 2526 len 40960 data: 4 3 3 3 4 4 5 5 5 4
17306 069/300 Write block 1642 len 28672 data: 4 4 3 5 4 4 4
17307 070/300 Read block 1133 len 16384
17308 071/300 Write block 638 len 20480 data: 8 8 8 8 7
17309 072/300 Read block 1681 len 28672
17310 073/300 Read block 45 len 12288
17311 074/300 Read block 4166 len 12288
17312 075/300 Read block 3351 len 36864
17313 076/300 Write block 4850 len 20480 data: 4 4 5 5 4
17314 077/300 Flush
17315 078/300 Write block 2848 len 8192 data: 7 6
17316 079/300 Write block 3730 len 32768 data: 2 3 4 4 3 3 3 3
17317 080/300 Read block 4241 len 4096
17318 081/300 Write block 2218 len 12288 data: 6 4 4
17319 082/300 Flush
17320 083/300 Write block 21 len 12288 data: 3 3 3
17321 084/300 Write block 4165 len 32768 data: 2 2 2 2 3 3 4 4
17322 085/300 Write block 1356 len 32768 data: 5 4 5 6 5 6 6 6
17323 086/300 Write block 3051 len 16384 data: 2 5 5 5
17324 087/300 Read block 2009 len 16384
17325 088/300 Write block 3914 len 24576 data: 7 6 4 3 4 5
17326 089/300 Read block 43 len 40960
17327 090/300 Write block 1439 len 40960 data: 4 4 4 5 7 7 7 8 7 7
17328 091/300 Write block 45 len 12288 data: 7 5 5
17329 092/300 Read block 3169 len 4096
17330 093/300 Read block 2012 len 40960
17331 094/300 Write block 728 len 12288 data: 4 6 6
17332 095/300 Write block 3649 len 4096 data: 5
17333 096/300 Read block 2279 len 4096
17334 097/300 Flush
17335 098/300 Write block 881 len 36864 data: 3 2 2 2 3 3 4 4 4
17336 099/300 Flush
17337 100/300 Read block 3769 len 24576
17338 101/300 Flush
17339 102/300 Read block 97 len 28672
17340 103/300 Read block 3048 len 28672
17341 104/300 Write block 3676 len 4096 data: 4
17342 105/300 Write block 2636 len 12288 data: 6 6 6
17343 106/300 Read block 3753 len 32768
17344 107/300 Read block 3662 len 12288
17345 108/300 Read block 3620 len 40960
17346 109/300 Read block 4436 len 24576
17347 110/300 Flush
17348 111/300 Read block 3848 len 20480
17349 112/300 Read block 2065 len 12288
17350 113/300 Read block 1499 len 16384
17351 114/300 Read block 4034 len 40960
17352 115/300 Read block 1105 len 8192
17353 116/300 Read block 3742 len 8192
17354 117/300 Read block 1517 len 32768
17355 118/300 Flush
17356 119/300 Read block 3311 len 16384
17357 120/300 Read block 1461 len 12288
17358 121/300 Read block 2696 len 40960
17359 122/300 Flush
17360 123/300 Read block 4036 len 32768
17361 124/300 Write block 980 len 20480 data: 3 4 5 5 5
17362 125/300 Flush
17363 126/300 Read block 1748 len 32768
17364 127/300 Write block 3144 len 32768 data: 3 5 4 4 4 5 5 4
17365 128/300 Write block 1752 len 4096 data: 3
17366 129/300 Read block 3888 len 20480
17367 130/300 Write block 1786 len 12288 data: 4 4 4
17368 131/300 Write block 3121 len 4096 data: 4
17369 132/300 Read block 663 len 40960
17370 133/300 Read block 3327 len 12288
17371 134/300 Read block 2273 len 28672
17372 135/300 Write block 750 len 20480 data: 7 7 6 4 4
17373 136/300 Read block 919 len 24576
17374 137/300 Write block 1861 len 8192 data: 5 5
17375 138/300 Read block 867 len 12288
17376 139/300 Write block 2538 len 28672 data: 3 3 3 3 3 3 2
17377 140/300 Write block 544 len 24576 data: 4 5 5 5 5 5
17378 141/300 Flush
17379 142/300 Write block 2575 len 20480 data: 3 3 4 5 4
17380 143/300 Read block 183 len 36864
17381 144/300 Read block 4069 len 40960
17382 145/300 Flush
17383 146/300 Read block 4553 len 40960
17384 147/300 Read block 4348 len 40960
17385 148/300 Read block 3876 len 12288
17386 149/300 Read block 2428 len 24576
17387 150/300 Read block 3195 len 12288
17388 151/300 Write block 964 len 28672 data: 3 5 4 3 2 2 2
17389 152/300 Read block 1686 len 32768
17390 153/300 Write block 1957 len 24576 data: 4 4 4 4 4 3
17391 154/300 Read block 2183 len 20480
17392 155/300 Read block 2814 len 4096
17393 156/300 Write block 1625 len 20480 data: 4 4 5 5 5
17394 157/300 Read block 3106 len 16384
17395 158/300 Read block 790 len 20480
17396 159/300 Write block 1211 len 32768 data: 6 6 6 5 4 3 3 2
17397 160/300 Read block 1601 len 16384
17398 161/300 Write block 1512 len 32768 data: 3 3 4 4 3 3 3 3
17399 162/300 Read block 4707 len 20480
17400 163/300 Write block 3757 len 20480 data: 4 4 2 5 4
17401 164/300 Read block 4052 len 16384
17402 165/300 Flush
17403 166/300 Write block 3227 len 20480 data: 6 4 4 5 6
17404 167/300 Write block 2649 len 36864 data: 4 6 5 6 6 4 3 3 4
17405 168/300 Read block 3770 len 16384
17406 169/300 Write block 4000 len 12288 data: 6 5 5
17407 170/300 Read block 3364 len 36864
17408 171/300 Write block 1331 len 32768 data: 4 4 4 3 2 2 4 4
17409 172/300 Write block 4562 len 32768 data: 4 4 5 4 4 3 3 4
17410 173/300 Flush
17411 174/300 Write block 4482 len 28672 data: 2 2 2 3 2 2 2
17412 175/300 Read block 1926 len 8192
17413 176/300 Read block 3164 len 8192
17414 177/300 Read block 2943 len 12288
17415 178/300 Read block 4949 len 36864
17416 179/300 Read block 1961 len 40960
17417 180/300 Read block 1272 len 16384
17418 181/300 Read block 4920 len 40960
17419 182/300 Write block 3716 len 12288 data: 6 6 6
17420 183/300 Read block 1090 len 24576
17421 184/300 Read block 12 len 24576
17422 185/300 Read block 2575 len 32768
17423 186/300 Read block 480 len 8192
17424 187/300 Flush
17425 188/300 Read block 2381 len 24576
17426 189/300 Read block 1366 len 32768
17427 190/300 Read block 3597 len 16384
17428 191/300 Flush
17429 192/300 Flush
17430 193/300 Flush
17431 194/300 Read block 4133 len 4096
17432 195/300 Read block 4489 len 40960
17433 196/300 Write block 1982 len 20480 data: 6 5 5 5 6
17434 197/300 Read block 2946 len 20480
17435 198/300 Write block 2868 len 20480 data: 4 3 3 3 3
17436 199/300 Read block 2881 len 20480
17437 200/300 Write block 3296 len 8192 data: 5 4
17438 201/300 Write block 1086 len 32768 data: 5 5 6 5 5 4 4 4
17439 202/300 Write block 6 len 16384 data: 5 5 4 4
17440 203/300 Write block 1981 len 36864 data: 6 7 6 6 6 7 5 4 3
17441 204/300 Read block 3602 len 32768
17442 205/300 Flush
17443 206/300 Read block 4639 len 40960
17444 207/300 Read block 2671 len 20480
17445 208/300 Read block 201 len 32768
17446 209/300 Read block 4673 len 16384
17447 210/300 Flush
17448 211/300 Write block 2911 len 12288 data: 5 5 5
17449 212/300 Flush
17450 213/300 Read block 4231 len 32768
17451 214/300 Write block 4038 len 32768 data: 4 5 5 5 4 4 3 3
17452 215/300 Read block 3211 len 24576
17453 216/300 Write block 194 len 36864 data: 5 5 5 6 5 5 5 5 6
17454 217/300 Read block 3403 len 20480
17455 218/300 Read block 976 len 32768
17456 219/300 Read block 2921 len 4096
17457 220/300 Write block 1242 len 40960 data: 4 4 5 5 4 4 3 2 3 4
17458 221/300 Write block 2405 len 24576 data: 6 6 7 5 5 5
17459 222/300 Read block 1779 len 8192
17460 223/300 Read block 107 len 16384
17461 224/300 Read block 691 len 4096
17462 225/300 Write block 3389 len 40960 data: 4 4 4 4 4 4 4 4 4 5
17463 226/300 Read block 3215 len 16384
17464 227/300 Write block 1346 len 36864 data: 4 3 3 3 4 3 3 3 3
17465 228/300 Write block 4485 len 28672 data: 4 3 3 3 2 4 4
17466 229/300 Flush
17467 230/300 Flush
17468 231/300 Read block 3941 len 32768
17469 232/300 Read block 3415 len 32768
17470 233/300 Write block 1355 len 16384 data: 3 6 5 6
17471 234/300 Read block 1859 len 32768
17472 235/300 Flush
17473 236/300 Read block 4286 len 32768
17474 237/300 Write block 1195 len 40960 data: 4 4 5 3 3 5 7 5 6 6
17475 238/300 Write block 3820 len 4096 data: 6
17476 239/300 Read block 1772 len 32768
17477 240/300 Write block 4216 len 20480 data: 4 5 4 5 6
17478 241/300 Read block 1144 len 36864
17479 242/300 Read block 3348 len 24576
17480 243/300 Write block 2922 len 32768 data: 5 6 5 4 4 6 6 4
17481 244/300 Write block 1988 len 28672 data: 5 4 4 5 6 5 5
17482 245/300 Read block 2208 len 4096
17483 246/300 Write block 2083 len 12288 data: 5 5 4
17484 247/300 Write block 257 len 16384 data: 3 2 3 3
17485 248/300 Read block 15 len 16384
17486 249/300 Read block 920 len 12288
17487 250/300 Read block 1890 len 36864
17488 251/300 Read block 4741 len 12288
17489 252/300 Read block 2226 len 20480
17490 253/300 Flush
17491 254/300 Write block 259 len 32768 data: 4 4 3 3 7 7 7 9
17492 255/300 Read block 887 len 24576
17493 256/300 Read block 4662 len 28672
17494 257/300 Read block 4580 len 4096
17495 258/300 Write block 33 len 12288 data: 4 5 4
17496 259/300 Read block 312 len 40960
17497 260/300 Read block 4196 len 20480
17498 261/300 Read block 4713 len 4096
17499 262/300 Write block 3701 len 28672 data: 7 7 6 5 6 6 8
17500 263/300 Write block 2548 len 8192 data: 3 2
17501 264/300 Read block 2973 len 8192
17502 265/300 Flush
17503 266/300 Read block 1194 len 28672
17504 267/300 Write block 970 len 24576 data: 3 3 2 3 3 3
17505 268/300 Read block 2434 len 20480
17506 269/300 Write block 102 len 4096 data: 2
17507 270/300 Read block 3136 len 24576
17508 271/300 Write block 3589 len 8192 data: 5 5
17509 272/300 Read block 3728 len 8192
17510 273/300 Read block 3461 len 28672
17511 274/300 Write block 1430 len 32768 data: 5 5 5 4 3 4 2 5
17512 275/300 Read block 2359 len 4096
17513 276/300 Write block 3663 len 16384 data: 2 2 2 2
17514 277/300 Write block 2511 len 20480 data: 4 4 4 3 3
17515 278/300 Write block 3980 len 36864 data: 5 5 6 4 6 6 7 6 6
17516 279/300 Read block 1970 len 8192
17517 280/300 Flush
17518 281/300 Read block 3382 len 8192
17519 282/300 Read block 4574 len 36864
17520 283/300 Read block 4519 len 12288
17521 284/300 Write block 1039 len 32768 data: 5 4 3 7 6 5 4 3
17522 285/300 Write block 3328 len 32768 data: 5 7 7 6 6 5 6 5
17523 286/300 Write block 4560 len 36864 data: 4 4 5 5 6 5 5 4 4
17524 287/300 Read block 3601 len 12288
17525 288/300 Read block 763 len 36864
17526 289/300 Write block 3753 len 40960 data: 4 4 6 4 5 5 3 6 5 4
17527 290/300 Write block 3359 len 32768 data: 7 6 5 6 6 4 5 6
17528 291/300 Flush
17529 292/300 Flush
17530 293/300 Read block 1649 len 4096
17531 294/300 Read block 3722 len 4096
17532 295/300 Read block 1718 len 40960
17533 296/300 Write block 1968 len 12288 data: 2 2 2
17534 297/300 Read block 3363 len 36864
17535 298/300 Flush
17536 299/300 Write block 1912 len 32768 data: 3 3 4 5 5 4 4 4
17537 300/300 Write block 824 len 28672 data: 4 5 5 6 7 7 8
17538 Main task has work to do, go find it
17539 got dsc Start(2)
17540 start 2
17541 Mar 02 01:09:08.894 INFO request completed, latency_us: 241, response_code: 204, uri: /start/cid/2, method: POST, req_id: f52fcc76-ba7f-409b-b022-64e8c88c26c3, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
17542 Replay: started 2, returned:Ok(())
17543 ----------------------------------------------------------------
17544 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
17545 States: Active Active Offline
17546 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17547 5729 Acked 6728 Write 3 Done Done New true
17548 5730 Acked 6729 Write 7 Done Done New true
17549 5731 Acked 6730 Write 5 Done Done New true
17550 5732 Acked 6731 Write 9 Done Done New false
17551 5733 Acked 6732 Read 6 Done Done New false
17552 5734 Acked 6733 Flush 0 Done Done New false
17553 5735 Acked 6734 Read 4 Done Done New false
17554 5736 Acked 6735 Flush 0 Done Done New false
17555 5737 Acked 6736 Flush 0 Done Done New false
17556 5738 Acked 6737 Write 10 Done Done New false
17557 5739 Acked 6738 Read 10 Done Done New false
17558 5740 Acked 6739 Read 9 Done Done New false
17559 5741 Acked 6740 Read 2 Done Done New false
17560 5742 Acked 6741 Read 4 Done Done New false
17561 5743 Acked 6742 Read 2 Done Done New false
17562 5744 Acked 6743 Write 1 Done Done New false
17563 5745 Acked 6744 Write 6 Done Done New false
17564 5746 Acked 6745 Read 2 Done Done New false
17565 5747 Acked 6746 Write 3 Done Done New false
17566 5748 Acked 6747 Read 5 Done Done New false
17567 5749 Acked 6748 Write 8 Done Done New false
17568 5750 Acked 6749 Read 4 Done Done New false
17569 5751 Acked 6750 Read 2 Done Done New false
17570 5752 Acked 6751 Read 6 Done Done New false
17571 5753 Acked 6752 Write 2 Done Done New false
17572 5754 Acked 6753 Write 7 Done Done New false
17573 5755 Acked 6754 Write 6 Done Done New false
17574 5756 Acked 6755 Read 8 Done Done New false
17575 5757 Acked 6756 Write 6 Done Done New false
17576 5758 Acked 6757 Read 10 Done Done New false
17577 5759 Acked 6758 Read 4 Done Done New false
17578 5760 Acked 6759 Read 1 Done Done New false
17579 5761 Acked 6760 Write 4 Done Done New false
17580 5762 Acked 6761 Flush 0 Done Done New false
17581 5763 Acked 6762 Read 10 Done Done New false
17582 5764 Acked 6763 Read 3 Done Done New false
17583 5765 Acked 6764 Read 7 Done Done New false
17584 5766 Acked 6765 Write 3 Done Done New false
17585 5767 Acked 6766 Flush 0 Done Done New false
17586 5768 Acked 6767 Read 9 Done Done New false
17587 5769 Acked 6768 Read 8 Done Done New false
17588 5770 Acked 6769 Flush 0 Done Done New false
17589 5771 Acked 6770 Write 1 Done Done New false
17590 5772 Acked 6771 Read 4 Done Done New false
17591 5773 Acked 6772 Flush 0 Done Done New false
17592 5774 Acked 6773 Read 8 Done Done New false
17593 5775 Acked 6774 Read 1 Done Done New false
17594 5776 Acked 6775 Write 8 Done Done New false
17595 5777 Acked 6776 Read 8 Done Done New false
17596 5778 Acked 6777 Read 6 Done Done New false
17597 5779 Acked 6778 Read 1 Done Done New false
17598 5780 Acked 6779 Write 9 Done Done New false
17599 5781 Acked 6780 Read 7 Done Done New false
17600 5782 Acked 6781 Flush 0 Done Done New false
17601 5783 Acked 6782 Read 1 Done Done New false
17602 5784 Acked 6783 Write 1 Done Done New false
17603 5785 Acked 6784 Write 4 Done Done New false
17604 5786 Acked 6785 Write 6 Done Done New false
17605 5787 Acked 6786 Write 3 Done Done New false
17606 5788 Acked 6787 Read 5 Done Done New false
17607 5789 Acked 6788 Write 6 Done Done New false
17608 5790 Acked 6789 Read 7 Done Done New false
17609 5791 Acked 6790 Read 9 Done Done New false
17610 5792 Acked 6791 Read 5 Done Done New false
17611 5793 Acked 6792 Read 7 Done Done New false
17612 5794 Acked 6793 Read 3 Done Done New false
17613 5795 Acked 6794 Write 6 Done Done New false
17614 5796 Acked 6795 Write 10 Done Done New false
17615 5797 Acked 6796 Write 7 Done Done New false
17616 5798 Acked 6797 Read 4 Done Done New false
17617 5799 Acked 6798 Write 5 Done Done New false
17618 5800 Acked 6799 Read 7 Done Done New false
17619 5801 Acked 6800 Read 3 Done Done New false
17620 5802 Acked 6801 Read 3 Done Done New false
17621 5803 Acked 6802 Read 9 Done Done New false
17622 5804 Acked 6803 Write 5 Done Done New false
17623 5805 Acked 6804 Flush 0 Done Done New false
17624 5806 Acked 6805 Write 2 Done Done New false
17625 5807 Acked 6806 Write 8 Done Done New false
17626 5808 Acked 6807 Read 1 Done Done New false
17627 5809 Acked 6808 Write 3 Done Done New false
17628 5810 Acked 6809 Flush 0 Done Done New false
17629 5811 Acked 6810 Write 3 Done Done New false
17630 5812 Acked 6811 Write 8 Done Done New false
17631 5813 Acked 6812 Write 8 Done Done New false
17632 5814 Acked 6813 Write 4 Done Done New false
17633 5815 Acked 6814 Read 4 Done Done New false
17634 5816 Acked 6815 Write 6 Done Done New false
17635 5817 Acked 6816 Read 10 Done Done New false
17636 5818 Acked 6817 Write 10 Done Done New false
17637 5819 Acked 6818 Write 3 Done Done New false
17638 5820 Acked 6819 Read 1 Done Done New false
17639 5821 Acked 6820 Read 10 Done Done New false
17640 5822 Acked 6821 Write 3 Done Done New false
17641 5823 Acked 6822 Write 1 Done Done New false
17642 5824 Acked 6823 Read 1 Done Done New false
17643 5825 Acked 6824 Flush 0 Done Done New false
17644 5826 Acked 6825 Write 9 Done Done New false
17645 5827 Acked 6826 Flush 0 Done Done New false
17646 5828 Acked 6827 Read 6 Done Done New false
17647 5829 Acked 6828 Flush 0 Done Done New false
17648 5830 Acked 6829 Read 7 Done Done New false
17649 5831 Acked 6830 Read 7 Done Done New false
17650 5832 Acked 6831 Write 1 Done Done New false
17651 5833 Acked 6832 Write 3 Done Done New false
17652 5834 Acked 6833 Read 8 Done Done New false
17653 5835 Acked 6834 Read 3 Done Done New false
17654 5836 Acked 6835 Read 10 Done Done New false
17655 5837 Acked 6836 Read 6 Done Done New false
17656 5838 Acked 6837 Flush 0 Done Done New false
17657 5839 Acked 6838 Read 5 Done Done New false
17658 5840 Acked 6839 Read 3 Done Done New false
17659 5841 Acked 6840 Read 4 Done Done New false
17660 5842 Acked 6841 Read 10 Done Done New false
17661 5843 Acked 6842 Read 2 Done Done New false
17662 5844 Acked 6843 Read 2 Done Done New false
17663 5845 Acked 6844 Read 8 Done Done New false
17664 5846 Acked 6845 Flush 0 Done Done New false
17665 5847 Acked 6846 Read 4 Done Done New false
17666 5848 Acked 6847 Read 3 Done Done New false
17667 5849 Acked 6848 Read 10 Done Done New false
17668 5850 Acked 6849 Flush 0 Done Done New false
17669 5851 Acked 6850 Read 8 Done Done New false
17670 5852 Acked 6851 Write 5 Done Done New false
17671 5853 Acked 6852 Flush 0 Done Done New false
17672 5854 Acked 6853 Read 8 Done Done New false
17673 5855 Acked 6854 Write 8 Done Done New false
17674 5856 Acked 6855 Write 1 Done Done New false
17675 5857 Acked 6856 Read 5 Done Done New false
17676 5858 Acked 6857 Write 3 Done Done New false
17677 5859 Acked 6858 Write 1 Done Done New false
17678 5860 Acked 6859 Read 10 Done Done New false
17679 5861 Acked 6860 Read 3 Done Done New false
17680 5862 Acked 6861 Read 7 Done Done New false
17681 5863 Acked 6862 Write 5 Done Done New false
17682 5864 Acked 6863 Read 6 Done Done New false
17683 5865 Acked 6864 Write 2 Done Done New false
17684 5866 Acked 6865 Read 3 Done Done New false
17685 5867 Acked 6866 Write 7 Done Done New false
17686 5868 Acked 6867 Write 6 Done Done New false
17687 5869 Acked 6868 Flush 0 Done Done New false
17688 5870 Acked 6869 Write 5 Done Done New false
17689 5871 Acked 6870 Read 9 Done Done New false
17690 5872 Acked 6871 Read 10 Done Done New false
17691 5873 Acked 6872 Flush 0 Done Done New false
17692 5874 Acked 6873 Read 10 Done Done New false
17693 5875 Acked 6874 Read 10 Done Done New false
17694 5876 Acked 6875 Read 3 Done Done New false
17695 5877 Acked 6876 Read 6 Done Done New false
17696 5878 Acked 6877 Read 3 Done Done New false
17697 5879 Acked 6878 Write 7 Done Done New false
17698 5880 Acked 6879 Read 8 Done Done New false
17699 5881 Acked 6880 Write 6 Done Done New false
17700 5882 Acked 6881 Read 5 Done Done New false
17701 5883 Acked 6882 Read 1 Done Done New false
17702 5884 Acked 6883 Write 5 Done Done New false
17703 5885 Acked 6884 Read 4 Done Done New false
17704 5886 Acked 6885 Read 5 Done Done New false
17705 5887 Acked 6886 Write 8 Done Done New false
17706 5888 Acked 6887 Read 4 Done Done New false
17707 5889 Acked 6888 Write 8 Done Done New false
17708 5890 Acked 6889 Read 5 Done Done New false
17709 5891 Acked 6890 Write 5 Done Done New false
17710 5892 Acked 6891 Read 4 Done Done New false
17711 5893 Acked 6892 Flush 0 Done Done New false
17712 5894 Acked 6893 Write 5 Done Done New false
17713 5895 Acked 6894 Write 9 Done Done New false
17714 5896 Acked 6895 Read 4 Done Done New false
17715 5897 Acked 6896 Write 3 Done Done New false
17716 5898 Acked 6897 Read 9 Done Done New false
17717 5899 Acked 6898 Write 8 Done Done New false
17718 5900 Acked 6899 Write 8 Done Done New false
17719 5901 Acked 6900 Flush 0 Done Done New false
17720 5902 Acked 6901 Write 7 Done Done New false
17721 5903 Acked 6902 Read 2 Done Done New false
17722 5904 Acked 6903 Read 2 Done Done New false
17723 5905 Acked 6904 Read 3 Done Done New false
17724 5906 Acked 6905 Read 9 Done Done New false
17725 5907 Acked 6906 Read 10 Done Done New false
17726 5908 Acked 6907 Read 4 Done Done New false
17727 5909 Acked 6908 Read 10 Done Done New false
17728 5910 Acked 6909 Write 3 Done Done New false
17729 5911 Acked 6910 Read 6 Done Done New false
17730 5912 Acked 6911 Read 6 Done Done New false
17731 5913 Acked 6912 Read 8 Done Done New false
17732 5914 Acked 6913 Read 2 Done Done New false
17733 5915 Acked 6914 Flush 0 Done Done New false
17734 5916 Acked 6915 Read 6 Done Done New false
17735 5917 Acked 6916 Read 8 Done Done New false
17736 5918 Acked 6917 Read 4 Done Done New false
17737 5919 Acked 6918 Flush 0 Done Done New false
17738 5920 Acked 6919 Flush 0 Done Done New false
17739 5921 Acked 6920 Flush 0 Done Done New false
17740 5922 Acked 6921 Read 1 Done Done New false
17741 5923 Acked 6922 Read 10 Done Done New false
17742 5924 Acked 6923 Write 5 Done Done New false
17743 5925 Acked 6924 Read 5 Done Done New false
17744 5926 Acked 6925 Write 5 Done Done New false
17745 5927 Acked 6926 Read 5 Done Done New false
17746 5928 Acked 6927 Write 2 Done Done New false
17747 5929 Acked 6928 Write 8 Done Done New false
17748 5930 Acked 6929 Write 4 Done Done New false
17749 5931 Acked 6930 Write 9 Done Done New false
17750 5932 Acked 6931 Read 8 Done Done New false
17751 5933 Acked 6932 Flush 0 Done Done New false
17752 5934 Acked 6933 Read 10 Done Done New false
17753 5935 Acked 6934 Read 5 Done Done New false
17754 5936 Acked 6935 Read 8 Done Done New false
17755 5937 Acked 6936 Read 4 Done Done New false
17756 5938 Acked 6937 Flush 0 Done Done New false
17757 5939 Acked 6938 Write 3 Done Done New false
17758 5940 Acked 6939 Flush 0 Done Done New false
17759 5941 Acked 6940 Read 8 Done Done New false
17760 5942 Acked 6941 Write 8 Done Done New false
17761 5943 Acked 6942 Read 6 Done Done New false
17762 5944 Acked 6943 Write 9 Done Done New false
17763 5945 Acked 6944 Read 5 Done Done New false
17764 5946 Acked 6945 Read 8 Done Done New false
17765 5947 Acked 6946 Read 1 Done Done New false
17766 5948 Acked 6947 Write 10 Done Done New false
17767 5949 Acked 6948 Write 6 Done Done New false
17768 5950 Acked 6949 Read 2 Done Done New false
17769 5951 Acked 6950 Read 4 Done Done New false
17770 5952 Acked 6951 Read 1 Done Done New false
17771 5953 Acked 6952 Write 10 Done Done New false
17772 5954 Acked 6953 Read 4 Done Done New false
17773 5955 Acked 6954 Write 9 Done Done New false
17774 5956 Acked 6955 Write 7 Done Done New false
17775 5957 Acked 6956 Flush 0 Done Done New false
17776 5958 Acked 6957 Flush 0 Done Done New false
17777 5959 Acked 6958 Read 8 Done Done New false
17778 5960 Acked 6959 Read 8 Done Done New false
17779 5961 Acked 6960 Write 4 Done Done New false
17780 5962 Acked 6961 Read 8 Done Done New false
17781 5963 Acked 6962 Flush 0 Done Done New false
17782 5964 Acked 6963 Read 8 Done Done New false
17783 5965 Acked 6964 Write 10 Done Done New false
17784 5966 Acked 6965 Write 1 Done Done New false
17785 5967 Acked 6966 Read 8 Done Done New false
17786 5968 Acked 6967 Write 5 Done Done New false
17787 5969 Acked 6968 Read 9 Done Done New false
17788 5970 Acked 6969 Read 6 Done Done New false
17789 5971 Acked 6970 Write 8 Done Done New false
17790 5972 Acked 6971 Write 7 Done Done New false
17791 5973 Acked 6972 Read 1 Done Done New false
17792 5974 Acked 6973 Write 3 Done Done New false
17793 5975 Acked 6974 Write 4 Done Done New false
17794 5976 Acked 6975 Read 4 Done Done New false
17795 5977 Acked 6976 Read 3 Done Done New false
17796 5978 Acked 6977 Read 9 Done Done New false
17797 5979 Acked 6978 Read 3 Done Done New false
17798 5980 Acked 6979 Read 5 Done Done New false
17799 5981 Acked 6980 Flush 0 Done Done New false
17800 5982 Acked 6981 Write 8 Done Done New false
17801 5983 Acked 6982 Read 6 Done Done New false
17802 5984 Acked 6983 Read 7 Done Done New false
17803 5985 Acked 6984 Read 1 Done Done New false
17804 5986 Acked 6985 Write 3 Done Done New false
17805 5987 Acked 6986 Read 10 Done Done New false
17806 5988 Acked 6987 Read 5 Done Done New false
17807 5989 Acked 6988 Read 1 Done Done New false
17808 5990 Acked 6989 Write 7 Done Done New false
17809 5991 Acked 6990 Write 2 Done Done New false
17810 5992 Acked 6991 Read 2 Done Done New false
17811 5993 Acked 6992 Flush 0 Done Done New false
17812 5994 Acked 6993 Read 7 Done Done New false
17813 5995 Acked 6994 Write 6 Done Done New false
17814 5996 Acked 6995 Read 5 Done Done New false
17815 5997 Acked 6996 Write 1 Done Done New false
17816 5998 Acked 6997 Read 6 Done Done New false
17817 5999 Acked 6998 Write 2 Done Done New false
17818 6000 Acked 6999 Read 2 Done Done New false
17819 6001 Acked 7000 Read 7 Done Done New false
17820 6002 Acked 7001 Write 8 Done Done New false
17821 6003 Acked 7002 Read 1 Done Done New false
17822 6004 Acked 7003 Write 4 Done Done New false
17823 6005 Acked 7004 Write 5 Done Done New false
17824 6006 Acked 7005 Write 9 Done Done New false
17825 6007 Acked 7006 Read 2 Done Done New false
17826 6008 Acked 7007 Flush 0 Done Done New false
17827 6009 Acked 7008 Read 2 Done Done New false
17828 6010 Acked 7009 Read 9 Done Done New false
17829 6011 Acked 7010 Read 3 Done Done New false
17830 6012 Acked 7011 Write 8 Done Done New false
17831 6013 Acked 7012 Write 8 Done Done New false
17832 6014 Acked 7013 Write 9 Done Done New false
17833 6015 Acked 7014 Read 3 Done Done New false
17834 6016 Acked 7015 Read 9 Done Done New false
17835 6017 Acked 7016 Write 10 Done Done New false
17836 6018 Acked 7017 Write 8 Done Done New false
17837 6019 Acked 7018 Flush 0 Done Done New false
17838 6020 Acked 7019 Flush 0 Done Done New false
17839 6021 Acked 7020 Read 1 Done Done New false
17840 6022 Acked 7021 Read 1 Done Done New false
17841 6023 Acked 7022 Read 10 Done Done New false
17842 6024 Acked 7023 Write 3 Done Done New false
17843 6025 Acked 7024 Read 9 Done Done New false
17844 6026 Acked 7025 Flush 0 Done Done New false
17845 6027 Acked 7026 Write 8 Done Done New false
17846 6028 Acked 7027 Write 7 Done Done New false
17847 STATES DS:0 DS:1 DS:2 TOTAL
17848 New 0 0 300 300
17849 Sent 0 0 0 0
17850 Done 300 300 0 600
17851 Skipped 0 0 0 0
17852 Error 0 0 0 0
17853 Last Flush: 7025 7025 6727
17854 Downstairs last five completed: 6727 6726 6725 6724 6723
17855 Upstairs last five completed: 6028 6027 6026 6025 6024
17856 CLIENT: Up:0 ds:300 act:2
178572024-03-02T01:09:09.550ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178582024-03-02T01:09:09.550ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178592024-03-02T01:09:09.550ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178602024-03-02T01:09:09.550ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178612024-03-02T01:09:09.550ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178622024-03-02T01:09:09.550ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178632024-03-02T01:09:09.551ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17864 [8830] got start action
17865 [8830] I am going to restart
17866 Starting downstairs at port 8830
17867 Make output file at "/tmp/dsc/downstairs-8830.txt"
17868 [8830][2] reports Starting
178692024-03-02T01:09:10.551ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178702024-03-02T01:09:10.551ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178712024-03-02T01:09:10.551ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178722024-03-02T01:09:10.551ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178732024-03-02T01:09:10.551ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178742024-03-02T01:09:10.552ZINFOcrucible: [2] client re-new 301 jobs since flush 6727 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178752024-03-02T01:09:10.552ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17876 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1182)
17877 [8830][2] reports Running
178782024-03-02T01:09:11.553ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178792024-03-02T01:09:11.553ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178802024-03-02T01:09:11.554ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178812024-03-02T01:09:11.555ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178822024-03-02T01:09:11.555ZINFOcrucible: send last flush ID to this DS: 6727 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178832024-03-02T01:09:11.555ZINFOcrucible: Replied this last flush ID: 6727 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178842024-03-02T01:09:11.555ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178852024-03-02T01:09:11.555ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178862024-03-02T01:09:11.555ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
178872024-03-02T01:09:11.555ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17888 ----------------------------------------------------------------
17889 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17890 Downstairs last five completed: 7028 7027 7026 7025 7024
17891 Upstairs last five completed: 6029 6028 6027 6026 6025
17892 CLIENT: Up:0 ds:0 act:3
17893 Replay: All jobs finished, all DS active.
17894 Main task has work to do, go find it
17895 got dsc Stop(2)
17896 stop 2
17897 Mar 02 01:09:12.897 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1fa3286f-6f05-46f4-b981-a14de9821133, remote_addr: 127.0.[8830] Got stop action so:false kr:false
17898 0.1:40735, local_addr: 127.0.0.1:9998
178992024-03-02T01:09:12.897ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179002024-03-02T01:09:12.897ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179012024-03-02T01:09:12.897ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179022024-03-02T01:09:12.897ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17903 Mar 02 01:09:12.897 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/2, method: GET, req_id: 686c1f29-6eee-4404-86d2-16c3c4a7fa6b, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
179042024-03-02T01:09:12.897ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179052024-03-02T01:09:12.898ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17906 [8830] Exited with: ExitStatus(unix_wait_status(9))
17907 [8830][2] reports Exit
179082024-03-02T01:09:13.899ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179092024-03-02T01:09:13.899ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179102024-03-02T01:09:13.899ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179112024-03-02T01:09:13.899ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179122024-03-02T01:09:13.899ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179132024-03-02T01:09:13.899ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179142024-03-02T01:09:13.899ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179152024-03-02T01:09:14.901ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179162024-03-02T01:09:14.901ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179172024-03-02T01:09:14.901ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179182024-03-02T01:09:14.901ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179192024-03-02T01:09:14.901ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179202024-03-02T01:09:14.901ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179212024-03-02T01:09:14.901ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179222024-03-02T01:09:15.903ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179232024-03-02T01:09:15.903ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179242024-03-02T01:09:15.903ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179252024-03-02T01:09:15.903ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179262024-03-02T01:09:15.903ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179272024-03-02T01:09:15.903ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179282024-03-02T01:09:15.903ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17929 Mar 02 01:09:16.900 INFO request completed, latency_us: 235, response_code: 200, uri: /state/cid/2, method: GET, req_id: e66f500b-bd3a-4279-bada-00951ff8546f, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
17930 001/300 Write block 43 len 28672 data: 7 7 8 6 6 5 7
17931 002/300 Flush
179322024-03-02T01:09:16.903ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179332024-03-02T01:09:16.903ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179342024-03-02T01:09:16.903ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179352024-03-02T01:09:16.903ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179362024-03-02T01:09:16.903ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179372024-03-02T01:09:16.903ZINFOcrucible: [2] client re-new 2 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
179382024-03-02T01:09:16.903ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
17939 003/300 Read block 1924 len 20480
17940 004/300 Flush
17941 005/300 Flush
17942 006/300 Read block 2914 len 12288
17943 007/300 Flush
17944 008/300 Write block 2437 len 32768 data: 5 4 5 5 6 6 6 6
17945 009/300 Read block 1834 len 4096
17946 010/300 Read block 2676 len 20480
17947 011/300 Write block 219 len 40960 data: 3 3 3 5 4 4 4 4 5 5
17948 012/300 Flush
17949 013/300 Read block 800 len 32768
17950 014/300 Flush
17951 015/300 Read block 2417 len 40960
17952 016/300 Read block 4796 len 20480
17953 017/300 Read block 1886 len 8192
17954 018/300 Read block 1392 len 40960
17955 019/300 Read block 4324 len 32768
17956 020/300 Read block 109 len 40960
17957 021/300 Write block 1941 len 4096 data: 7
17958 022/300 Write block 1770 len 4096 data: 4
17959 023/300 Read block 2408 len 12288
17960 024/300 Write block 4598 len 4096 data: 3
17961 025/300 Read block 3937 len 24576
17962 026/300 Write block 2786 len 36864 data: 3 3 3 3 5 6 6 6 7
17963 027/300 Write block 3163 len 12288 data: 2 2 2
17964 028/300 Read block 849 len 16384
17965 029/300 Read block 654 len 36864
17966 030/300 Write block 2216 len 32768 data: 3 3 7 5 5 4 3 3
17967 031/300 Write block 3535 len 16384 data: 6 5 4 4
17968 032/300 Read block 380 len 16384
17969 033/300 Write block 2671 len 36864 data: 3 4 4 4 6 5 5 5 5
17970 034/300 Read block 2977 len 28672
17971 035/300 Write block 4847 len 8192 data: 4 5
17972 036/300 Write block 3588 len 4096 data: 5
17973 037/300 Read block 956 len 28672
17974 038/300 Write block 4308 len 36864 data: 4 4 4 5 4 3 3 4 4
17975 039/300 Write block 3779 len 32768 data: 10 12 11 9 9 9 7 6
17976 040/300 Read block 4066 len 12288
17977 041/300 Write block 2679 len 24576 data: 6 4 4 5 3 4
17978 042/300 Read block 81 len 20480
17979 043/300 Read block 4066 len 16384
17980 044/300 Flush
17981 045/300 Write block 3766 len 20480 data: 4 4 4 4 3
17982 046/300 Write block 2713 len 40960 data: 4 4 4 6 6 8 7 8 6 6
17983 047/300 Flush
17984 048/300 Read block 2189 len 8192
17985 049/300 Write block 4974 len 20480 data: 4 3 3 4 6
17986 050/300 Write block 168 len 4096 data: 4
17987 051/300 Write block 3172 len 32768 data: 4 4 5 4 3 4 4 4
17988 052/300 Write block 4098 len 8192 data: 7 6
17989 053/300 Write block 2485 len 16384 data: 5 3 3 3
17990 054/300 Read block 4933 len 16384
17991 055/300 Read block 3327 len 24576
17992 056/300 Write block 1678 len 36864 data: 3 5 5 6 6 8 8 7 7
17993 057/300 Read block 853 len 40960
17994 058/300 Read block 111 len 24576
17995 059/300 Write block 4389 len 8192 data: 4 5
17996 060/300 Read block 4985 len 12288
17997 061/300 Read block 1112 len 16384
17998 062/300 Read block 3648 len 40960
17999 063/300 Read block 343 len 24576
18000 064/300 Read block 452 len 16384
18001 065/300 Write block 2638 len 36864 data: 7 6 3 3 3 3 2 2 2
18002 066/300 Read block 4485 len 28672
18003 067/300 Write block 4352 len 16384 data: 6 7 7 7
18004 068/300 Write block 962 len 20480 data: 4 4 4 6 5
18005 069/300 Read block 1549 len 40960
18006 070/300 Write block 4042 len 16384 data: 5 5 4 4
18007 071/300 Write block 139 len 20480 data: 3 3 2 3 2
18008 072/300 Write block 3830 len 12288 data: 4 4 5
18009 073/300 Read block 3041 len 28672
18010 074/300 Write block 4504 len 24576 data: 6 5 5 5 4 3
18011 075/300 Flush
18012 076/300 Write block 1771 len 36864 data: 4 3 3 3 2 3 4 5 5
18013 077/300 Write block 4881 len 12288 data: 4 3 2
18014 078/300 Flush
18015 079/300 Read block 4672 len 8192
18016 080/300 Write block 1931 len 36864 data: 6 5 6 5 6 6 7 6 8
18017 081/300 Write block 744 len 16384 data: 6 7 8 8
18018 082/300 Read block 4968 len 16384
18019 083/300 Write block 2069 len 12288 data: 8 8 7
18020 084/300 Read block 2113 len 12288
18021 085/300 Read block 445 len 32768
18022 086/300 Read block 2248 len 24576
18023 087/300 Read block 3429 len 36864
18024 088/300 Write block 3235 len 40960 data: 7 6 6 6 6 5 5 5 3 3
18025 089/300 Read block 208 len 36864
18026 090/300 Read block 799 len 28672
18027 091/300 Read block 3072 len 24576
18028 092/300 Read block 4256 len 12288
18029 093/300 Read block 200 len 8192
18030 094/300 Read block 2490 len 8192
18031 095/300 Write block 1961 len 12288 data: 5 4 2
18032 096/300 Read block 2434 len 20480
18033 097/300 Write block 4585 len 36864 data: 5 4 5 6 6 7 7 7 7
18034 098/300 Read block 3106 len 40960
18035 099/300 Read block 2446 len 12288
18036 100/300 Read block 257 len 24576
18037 101/300 Flush
18038 102/300 Read block 1620 len 20480
18039 103/300 Write block 2814 len 4096 data: 7
18040 104/300 Flush
18041 105/300 Flush
18042 106/300 Read block 4342 len 4096
18043 107/300 Read block 4431 len 32768
18044 108/300 Read block 1579 len 16384
18045 109/300 Read block 4487 len 4096
18046 110/300 Read block 680 len 40960
18047 111/300 Read block 2815 len 24576
18048 112/300 Write block 346 len 36864 data: 9 8 5 5 5 4 4 4 3
18049 113/300 Write block 4333 len 12288 data: 7 8 7
18050 114/300 Write block 2849 len 4096 data: 7
18051 115/300 Read block 849 len 12288
18052 116/300 Write block 986 len 24576 data: 5 4 5 4 4 5
18053 117/300 Flush
18054 118/300 Flush
18055 119/300 Read block 3450 len 40960
18056 120/300 Write block 2056 len 24576 data: 3 2 3 3 5 5
18057 121/300 Read block 605 len 24576
18058 122/300 Write block 4052 len 40960 data: 7 6 5 5 6 5 4 4 4 4
18059 123/300 Write block 1206 len 4096 data: 5
18060 124/300 Write block 4785 len 12288 data: 5 6 5
18061 125/300 Write block 2942 len 20480 data: 4 4 4 3 4
18062 126/300 Write block 4670 len 24576 data: 4 4 4 3 3 4
18063 127/300 Read block 1946 len 16384
18064 128/300 Read block 1671 len 16384
18065 129/300 Read block 1336 len 20480
18066 130/300 Read block 3705 len 28672
18067 131/300 Write block 2760 len 20480 data: 4 5 4 2 2
18068 132/300 Read block 3717 len 12288
18069 133/300 Read block 790 len 24576
18070 134/300 Write block 2458 len 24576 data: 6 6 7 6 5 5
18071 135/300 Flush
18072 136/300 Read block 1482 len 16384
18073 137/300 Read block 1380 len 28672
18074 138/300 Write block 982 len 40960 data: 6 6 6 6 6 5 6 5 5 6
18075 139/300 Read block 4849 len 8192
18076 140/300 Write block 4625 len 36864 data: 5 5 2 3 3 3 3 3 3
18077 141/300 Flush
18078 142/300 Write block 4645 len 12288 data: 8 8 8
18079 143/300 Write block 3669 len 32768 data: 2 2 2 2 3 3 4 5
18080 144/300 Read block 3981 len 24576
18081 145/300 Write block 375 len 32768 data: 8 4 3 5 5 3 3 3
18082 146/300 Read block 1299 len 28672
18083 147/300 Write block 2263 len 16384 data: 5 4 3 2
18084 148/300 Read block 4386 len 40960
18085 149/300 Flush
18086 150/300 Read block 4478 len 20480
18087 151/300 Write block 3874 len 28672 data: 4 4 4 2 2 4 4
18088 152/300 Write block 203 len 12288 data: 4 3 3
18089 153/300 Read block 3203 len 4096
18090 154/300 Write block 2398 len 32768 data: 4 4 5 5 5 5 6 7
18091 155/300 Write block 4313 len 16384 data: 4 4 5 5
18092 156/300 Read block 597 len 40960
18093 157/300 Write block 891 len 8192 data: 6 6
18094 158/300 Write block 2370 len 12288 data: 3 3 4
18095 159/300 Read block 4903 len 32768
18096 160/300 Read block 1322 len 24576
18097 161/300 Read block 3332 len 28672
18098 162/300 Read block 465 len 20480
18099 163/300 Read block 2063 len 12288
18100 164/300 Read block 1853 len 32768
18101 165/300 Read block 1862 len 16384
18102 166/300 Read block 3226 len 4096
18103 167/300 Read block 2774 len 24576
18104 168/300 Write block 166 len 36864 data: 3 4 5 4 3 4 5 5 5
18105 169/300 Read block 4674 len 16384
18106 170/300 Read block 86 len 8192
18107 171/300 Read block 4208 len 12288
18108 172/300 Read block 4122 len 16384
18109 173/300 Read block 1253 len 16384
18110 174/300 Read block 4803 len 8192
18111 175/300 Write block 3523 len 24576 data: 8 8 5 5 5 7
18112 176/300 Write block 642 len 24576 data: 8 6 6 7 7 6
18113 177/300 Read block 3287 len 32768
18114 178/300 Flush
18115 179/300 Write block 960 len 16384 data: 4 5 5 5
18116 180/300 Write block 797 len 40960 data: 4 4 3 2 2 2 2 3 4 4
18117 181/300 Read block 1990 len 40960
18118 182/300 Write block 2497 len 16384 data: 5 4 5 5
18119 183/300 Read block 3597 len 16384
18120 184/300 Write block 2906 len 24576 data: 3 3 4 5 5 6
18121 185/300 Write block 4906 len 24576 data: 4 3 3 3 4 4
18122 186/300 Read block 3662 len 28672
18123 187/300 Write block 2972 len 40960 data: 2 2 2 2 2 2 2 2 2 2
18124 188/300 Write block 3037 len 4096 data: 3
18125 189/300 Write block 2460 len 24576 data: 8 7 6 6 3 4
18126 190/300 Write block 1906 len 36864 data: 4 3 4 4 4 3 4 4 5
18127 191/300 Read block 3595 len 4096
18128 192/300 Write block 1256 len 12288 data: 4 4 4
18129 193/300 Write block 1144 len 20480 data: 3 2 2 3 3
18130 194/300 Write block 2523 len 24576 data: 4 4 5 5 4 4
18131 195/300 Read block 1584 len 12288
18132 196/300 Read block 3326 len 20480
18133 197/300 Read block 224 len 28672
18134 198/300 Write block 2809 len 36864 data: 6 7 7 6 7 8 4 6 8
18135 199/300 Write block 3755 len 20480 data: 7 5 6 6 4
18136 200/300 Write block 2108 len 8192 data: 4 3
18137 201/300 Write block 1305 len 40960 data: 4 3 2 2 3 3 3 3 3 3
18138 202/300 Read block 849 len 24576
18139 203/300 Read block 1754 len 28672
18140 204/300 Write block 477 len 40960 data: 3 3 4 4 4 3 3 3 3 3
18141 205/300 Read block 842 len 20480
18142 206/300 Read block 3488 len 20480
18143 207/300 Read block 3921 len 8192
18144 208/300 Read block 2047 len 8192
18145 209/300 Read block 1880 len 8192
18146 210/300 Flush
18147 211/300 Read block 1368 len 32768
18148 212/300 Read block 485 len 36864
18149 213/300 Write block 664 len 16384 data: 7 8 6 7
18150 214/300 Read block 332 len 20480
18151 215/300 Write block 1043 len 16384 data: 7 6 5 4
18152 216/300 Write block 515 len 36864 data: 2 2 2 2 3 3 3 3 5
18153 217/300 Flush
18154 218/300 Write block 4987 len 28672 data: 6 7 8 9 6 7 5
18155 219/300 Read block 4830 len 28672
18156 220/300 Write block 2735 len 36864 data: 6 5 4 4 4 4 3 3 5
18157 221/300 Flush
18158 222/300 Read block 3155 len 40960
18159 223/300 Read block 3130 len 8192
18160 224/300 Read block 1853 len 40960
18161 225/300 Read block 687 len 8192
18162 226/300 Read block 2631 len 8192
18163 227/300 Write block 4799 len 24576 data: 5 6 6 3 4 5
18164 228/300 Read block 2923 len 32768
18165 229/300 Write block 1737 len 20480 data: 5 5 5 4 5
18166 230/300 Flush
18167 231/300 Read block 2274 len 24576
18168 232/300 Flush
18169 233/300 Read block 2176 len 16384
18170 234/300 Flush
18171 235/300 Write block 4576 len 16384 data: 6 5 5 5
18172 236/300 Flush
18173 237/300 Write block 3024 len 4096 data: 6
18174 238/300 Write block 423 len 20480 data: 4 4 3 3 4
18175 239/300 Read block 1991 len 36864
18176 240/300 Write block 1405 len 24576 data: 5 5 5 3 3 4
18177 241/300 Write block 1249 len 16384 data: 3 4 5 4
18178 242/300 Write block 387 len 12288 data: 4 6 6
18179 243/300 Flush
18180 244/300 Read block 3153 len 40960
18181 245/300 Write block 2174 len 32768 data: 10 9 7 6 6 5 3 3
18182 246/300 Read block 3403 len 32768
18183 247/300 Read block 3661 len 28672
18184 248/300 Write block 2132 len 40960 data: 5 4 4 4 3 3 2 2 4 3
18185 249/300 Read block 3992 len 20480
18186 250/300 Read block 1255 len 28672
18187 251/300 Write block 2534 len 40960 data: 6 5 5 3 4 4 4 4 4 4
18188 252/300 Write block 701 len 4096 data: 5
18189 253/300 Read block 2874 len 16384
18190 254/300 Read block 3718 len 24576
18191 255/300 Read block 2470 len 40960
18192 256/300 Read block 4760 len 40960
18193 257/300 Read block 949 len 12288
18194 258/300 Read block 4913 len 40960
18195 259/300 Read block 2928 len 16384
18196 260/300 Write block 2320 len 40960 data: 6 7 5 3 4 4 3 3 2 2
18197 261/300 Read block 4175 len 8192
18198 262/300 Write block 3659 len 28672 data: 4 2 2 2 3 3 3
18199 263/300 Read block 1121 len 24576
18200 264/300 Read block 896 len 20480
18201 265/300 Read block 3366 len 36864
18202 266/300 Read block 4546 len 12288
18203 267/300 Read block 1314 len 36864
18204 268/300 Flush
18205 269/300 Read block 3594 len 4096
18206 270/300 Write block 3956 len 24576 data: 5 5 4 5 5 6
18207 271/300 Write block 1575 len 40960 data: 6 7 6 6 6 8 5 5 5 6
18208 272/300 Read block 1202 len 16384
18209 273/300 Write block 850 len 32768 data: 2 3 3 3 2 2 4 5
18210 274/300 Read block 587 len 32768
18211 275/300 Write block 3565 len 28672 data: 8 8 7 8 7 5 4
18212 276/300 Read block 3721 len 32768
18213 277/300 Read block 3299 len 36864
18214 278/300 Write block 4036 len 12288 data: 5 3 5
18215 279/300 Write block 2625 len 8192 data: 3 5
18216 280/300 Write block 17 len 40960 data: 3 2 3 3 4 4 4 2 2 3
18217 281/300 Flush
18218 282/300 Read block 4270 len 12288
18219 283/300 Read block 1531 len 12288
18220 284/300 Read block 4013 len 4096
18221 285/300 Read block 3128 len 28672
18222 286/300 Write block 2569 len 8192 data: 4 3
18223 287/300 Read block 3137 len 40960
18224 288/300 Flush
18225 289/300 Read block 4554 len 16384
18226 290/300 Read block 4293 len 36864
18227 291/300 Read block 1715 len 28672
18228 292/300 Write block 3358 len 12288 data: 7 8 7
18229 293/300 Write block 2473 len 40960 data: 4 5 5 5 5 5 3 4 4 5
18230 294/300 Flush
18231 295/300 Write block 1191 len 16384 data: 6 6 7 4
18232 296/300 Read block 283 len 16384
18233 297/300 Flush
18234 298/300 Read block 1817 len 12288
18235 299/300 Write block 2564 len 4096 data: 7
18236 300/300 Read block 4961 len 8192
18237 Main task has work to do, go find it
18238 got dsc Start(2)
18239 start 2
18240 Mar 02 01:09:17.250 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/2, method: POST, req_id: a2f193c6-e273-4857-87d1-582a95d9e0aa, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
18241 Replay: started 2, returned:Ok(())
18242 ----------------------------------------------------------------
18243 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18244 States: Active Active Offline
18245 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18246 6030 Acked 7029 Write 7 Done Done New true
18247 6031 Acked 7030 Flush 0 Done Done New true
18248 6032 Acked 7031 Read 5 Done Done New false
18249 6033 Acked 7032 Flush 0 Done Done New false
18250 6034 Acked 7033 Flush 0 Done Done New false
18251 6035 Acked 7034 Read 3 Done Done New false
18252 6036 Acked 7035 Flush 0 Done Done New false
18253 6037 Acked 7036 Write 8 Done Done New false
18254 6038 Acked 7037 Read 1 Done Done New false
18255 6039 Acked 7038 Read 5 Done Done New false
18256 6040 Acked 7039 Write 10 Done Done New false
18257 6041 Acked 7040 Flush 0 Done Done New false
18258 6042 Acked 7041 Read 8 Done Done New false
18259 6043 Acked 7042 Flush 0 Done Done New false
18260 6044 Acked 7043 Read 10 Done Done New false
18261 6045 Acked 7044 Read 5 Done Done New false
18262 6046 Acked 7045 Read 2 Done Done New false
18263 6047 Acked 7046 Read 10 Done Done New false
18264 6048 Acked 7047 Read 8 Done Done New false
18265 6049 Acked 7048 Read 10 Done Done New false
18266 6050 Acked 7049 Write 1 Done Done New false
18267 6051 Acked 7050 Write 1 Done Done New false
18268 6052 Acked 7051 Read 3 Done Done New false
18269 6053 Acked 7052 Write 1 Done Done New false
18270 6054 Acked 7053 Read 6 Done Done New false
18271 6055 Acked 7054 Write 9 Done Done New false
18272 6056 Acked 7055 Write 3 Done Done New false
18273 6057 Acked 7056 Read 4 Done Done New false
18274 6058 Acked 7057 Read 9 Done Done New false
18275 6059 Acked 7058 Write 8 Done Done New false
18276 6060 Acked 7059 Write 4 Done Done New false
18277 6061 Acked 7060 Read 4 Done Done New false
18278 6062 Acked 7061 Write 9 Done Done New false
18279 6063 Acked 7062 Read 7 Done Done New false
18280 6064 Acked 7063 Write 2 Done Done New false
18281 6065 Acked 7064 Write 1 Done Done New false
18282 6066 Acked 7065 Read 7 Done Done New false
18283 6067 Acked 7066 Write 9 Done Done New false
18284 6068 Acked 7067 Write 8 Done Done New false
18285 6069 Acked 7068 Read 3 Done Done New false
18286 6070 Acked 7069 Write 6 Done Done New false
18287 6071 Acked 7070 Read 5 Done Done New false
18288 6072 Acked 7071 Read 4 Done Done New false
18289 6073 Acked 7072 Flush 0 Done Done New false
18290 6074 Acked 7073 Write 5 Done Done New false
18291 6075 Acked 7074 Write 10 Done Done New false
18292 6076 Acked 7075 Flush 0 Done Done New false
18293 6077 Acked 7076 Read 2 Done Done New false
18294 6078 Acked 7077 Write 5 Done Done New false
18295 6079 Acked 7078 Write 1 Done Done New false
18296 6080 Acked 7079 Write 8 Done Done New false
18297 6081 Acked 7080 Write 2 Done Done New false
18298 6082 Acked 7081 Write 4 Done Done New false
18299 6083 Acked 7082 Read 4 Done Done New false
18300 6084 Acked 7083 Read 6 Done Done New false
18301 6085 Acked 7084 Write 9 Done Done New false
18302 6086 Acked 7085 Read 10 Done Done New false
18303 6087 Acked 7086 Read 6 Done Done New false
18304 6088 Acked 7087 Write 2 Done Done New false
18305 6089 Acked 7088 Read 3 Done Done New false
18306 6090 Acked 7089 Read 4 Done Done New false
18307 6091 Acked 7090 Read 10 Done Done New false
18308 6092 Acked 7091 Read 6 Done Done New false
18309 6093 Acked 7092 Read 4 Done Done New false
18310 6094 Acked 7093 Write 9 Done Done New false
18311 6095 Acked 7094 Read 7 Done Done New false
18312 6096 Acked 7095 Write 4 Done Done New false
18313 6097 Acked 7096 Write 5 Done Done New false
18314 6098 Acked 7097 Read 10 Done Done New false
18315 6099 Acked 7098 Write 4 Done Done New false
18316 6100 Acked 7099 Write 5 Done Done New false
18317 6101 Acked 7100 Write 3 Done Done New false
18318 6102 Acked 7101 Read 7 Done Done New false
18319 6103 Acked 7102 Write 6 Done Done New false
18320 6104 Acked 7103 Flush 0 Done Done New false
18321 6105 Acked 7104 Write 9 Done Done New false
18322 6106 Acked 7105 Write 3 Done Done New false
18323 6107 Acked 7106 Flush 0 Done Done New false
18324 6108 Acked 7107 Read 2 Done Done New false
18325 6109 Acked 7108 Write 9 Done Done New false
18326 6110 Acked 7109 Write 4 Done Done New false
18327 6111 Acked 7110 Read 4 Done Done New false
18328 6112 Acked 7111 Write 3 Done Done New false
18329 6113 Acked 7112 Read 3 Done Done New false
18330 6114 Acked 7113 Read 8 Done Done New false
18331 6115 Acked 7114 Read 6 Done Done New false
18332 6116 Acked 7115 Read 9 Done Done New false
18333 6117 Acked 7116 Write 10 Done Done New false
18334 6118 Acked 7117 Read 9 Done Done New false
18335 6119 Acked 7118 Read 7 Done Done New false
18336 6120 Acked 7119 Read 6 Done Done New false
18337 6121 Acked 7120 Read 3 Done Done New false
18338 6122 Acked 7121 Read 2 Done Done New false
18339 6123 Acked 7122 Read 2 Done Done New false
18340 6124 Acked 7123 Write 3 Done Done New false
18341 6125 Acked 7124 Read 5 Done Done New false
18342 6126 Acked 7125 Write 9 Done Done New false
18343 6127 Acked 7126 Read 10 Done Done New false
18344 6128 Acked 7127 Read 3 Done Done New false
18345 6129 Acked 7128 Read 6 Done Done New false
18346 6130 Acked 7129 Flush 0 Done Done New false
18347 6131 Acked 7130 Read 5 Done Done New false
18348 6132 Acked 7131 Write 1 Done Done New false
18349 6133 Acked 7132 Flush 0 Done Done New false
18350 6134 Acked 7133 Flush 0 Done Done New false
18351 6135 Acked 7134 Read 1 Done Done New false
18352 6136 Acked 7135 Read 8 Done Done New false
18353 6137 Acked 7136 Read 4 Done Done New false
18354 6138 Acked 7137 Read 1 Done Done New false
18355 6139 Acked 7138 Read 10 Done Done New false
18356 6140 Acked 7139 Read 6 Done Done New false
18357 6141 Acked 7140 Write 9 Done Done New false
18358 6142 Acked 7141 Write 3 Done Done New false
18359 6143 Acked 7142 Write 1 Done Done New false
18360 6144 Acked 7143 Read 3 Done Done New false
18361 6145 Acked 7144 Write 6 Done Done New false
18362 6146 Acked 7145 Flush 0 Done Done New false
18363 6147 Acked 7146 Flush 0 Done Done New false
18364 6148 Acked 7147 Read 10 Done Done New false
18365 6149 Acked 7148 Flush 0 Done Done New false
18366 6150 Acked 7149 Write 6 Done Done New false
18367 6151 Acked 7150 Read 6 Done Done New false
18368 6152 Acked 7151 Write 10 Done Done New false
18369 6153 Acked 7152 Write 1 Done Done New false
18370 6154 Acked 7153 Write 3 Done Done New false
18371 6155 Acked 7154 Write 5 Done Done New false
18372 6156 Acked 7155 Write 6 Done Done New false
18373 6157 Acked 7156 Read 4 Done Done New false
18374 6158 Acked 7157 Read 4 Done Done New false
18375 6159 Acked 7158 Read 5 Done Done New false
18376 6160 Acked 7159 Read 7 Done Done New false
18377 6161 Acked 7160 Write 5 Done Done New false
18378 6162 Acked 7161 Read 3 Done Done New false
18379 6163 Acked 7162 Read 6 Done Done New false
18380 6164 Acked 7163 Write 6 Done Done New false
18381 6165 Acked 7164 Flush 0 Done Done New false
18382 6166 Acked 7165 Read 4 Done Done New false
18383 6167 Acked 7166 Read 7 Done Done New false
18384 6168 Acked 7167 Write 10 Done Done New false
18385 6169 Acked 7168 Read 2 Done Done New false
18386 6170 Acked 7169 Write 9 Done Done New false
18387 6171 Acked 7170 Flush 0 Done Done New false
18388 6172 Acked 7171 Write 3 Done Done New false
18389 6173 Acked 7172 Write 8 Done Done New false
18390 6174 Acked 7173 Read 6 Done Done New false
18391 6175 Acked 7174 Write 8 Done Done New false
18392 6176 Acked 7175 Read 7 Done Done New false
18393 6177 Acked 7176 Write 4 Done Done New false
18394 6178 Acked 7177 Read 10 Done Done New false
18395 6179 Acked 7178 Flush 0 Done Done New false
18396 6180 Acked 7179 Read 5 Done Done New false
18397 6181 Acked 7180 Write 7 Done Done New false
18398 6182 Acked 7181 Write 3 Done Done New false
18399 6183 Acked 7182 Read 1 Done Done New false
18400 6184 Acked 7183 Write 8 Done Done New false
18401 6185 Acked 7184 Write 4 Done Done New false
18402 6186 Acked 7185 Read 10 Done Done New false
18403 6187 Acked 7186 Write 2 Done Done New false
18404 6188 Acked 7187 Write 3 Done Done New false
18405 6189 Acked 7188 Read 8 Done Done New false
18406 6190 Acked 7189 Read 6 Done Done New false
18407 6191 Acked 7190 Read 7 Done Done New false
18408 6192 Acked 7191 Read 5 Done Done New false
18409 6193 Acked 7192 Read 3 Done Done New false
18410 6194 Acked 7193 Read 8 Done Done New false
18411 6195 Acked 7194 Read 4 Done Done New false
18412 6196 Acked 7195 Read 1 Done Done New false
18413 6197 Acked 7196 Read 6 Done Done New false
18414 6198 Acked 7197 Write 9 Done Done New false
18415 6199 Acked 7198 Read 4 Done Done New false
18416 6200 Acked 7199 Read 2 Done Done New false
18417 6201 Acked 7200 Read 3 Done Done New false
18418 6202 Acked 7201 Read 4 Done Done New false
18419 6203 Acked 7202 Read 4 Done Done New false
18420 6204 Acked 7203 Read 2 Done Done New false
18421 6205 Acked 7204 Write 6 Done Done New false
18422 6206 Acked 7205 Write 6 Done Done New false
18423 6207 Acked 7206 Read 8 Done Done New false
18424 6208 Acked 7207 Flush 0 Done Done New false
18425 6209 Acked 7208 Write 4 Done Done New false
18426 6210 Acked 7209 Write 10 Done Done New false
18427 6211 Acked 7210 Read 10 Done Done New false
18428 6212 Acked 7211 Write 4 Done Done New false
18429 6213 Acked 7212 Read 4 Done Done New false
18430 6214 Acked 7213 Write 6 Done Done New false
18431 6215 Acked 7214 Write 6 Done Done New false
18432 6216 Acked 7215 Read 7 Done Done New false
18433 6217 Acked 7216 Write 10 Done Done New false
18434 6218 Acked 7217 Write 1 Done Done New false
18435 6219 Acked 7218 Write 6 Done Done New false
18436 6220 Acked 7219 Write 9 Done Done New false
18437 6221 Acked 7220 Read 1 Done Done New false
18438 6222 Acked 7221 Write 3 Done Done New false
18439 6223 Acked 7222 Write 5 Done Done New false
18440 6224 Acked 7223 Write 6 Done Done New false
18441 6225 Acked 7224 Read 3 Done Done New false
18442 6226 Acked 7225 Read 5 Done Done New false
18443 6227 Acked 7226 Read 7 Done Done New false
18444 6228 Acked 7227 Write 9 Done Done New false
18445 6229 Acked 7228 Write 5 Done Done New false
18446 6230 Acked 7229 Write 2 Done Done New false
18447 6231 Acked 7230 Write 10 Done Done New false
18448 6232 Acked 7231 Read 6 Done Done New false
18449 6233 Acked 7232 Read 7 Done Done New false
18450 6234 Acked 7233 Write 10 Done Done New false
18451 6235 Acked 7234 Read 5 Done Done New false
18452 6236 Acked 7235 Read 5 Done Done New false
18453 6237 Acked 7236 Read 2 Done Done New false
18454 6238 Acked 7237 Read 2 Done Done New false
18455 6239 Acked 7238 Read 2 Done Done New false
18456 6240 Acked 7239 Flush 0 Done Done New false
18457 6241 Acked 7240 Read 8 Done Done New false
18458 6242 Acked 7241 Read 9 Done Done New false
18459 6243 Acked 7242 Write 4 Done Done New false
18460 6244 Acked 7243 Read 5 Done Done New false
18461 6245 Acked 7244 Write 4 Done Done New false
18462 6246 Acked 7245 Write 9 Done Done New false
18463 6247 Acked 7246 Flush 0 Done Done New false
18464 6248 Acked 7247 Write 7 Done Done New false
18465 6249 Acked 7248 Read 7 Done Done New false
18466 6250 Acked 7249 Write 9 Done Done New false
18467 6251 Acked 7250 Flush 0 Done Done New false
18468 6252 Acked 7251 Read 10 Done Done New false
18469 6253 Acked 7252 Read 2 Done Done New false
18470 6254 Acked 7253 Read 10 Done Done New false
18471 6255 Acked 7254 Read 2 Done Done New false
18472 6256 Acked 7255 Read 2 Done Done New false
18473 6257 Acked 7256 Write 6 Done Done New false
18474 6258 Acked 7257 Read 8 Done Done New false
18475 6259 Acked 7258 Write 5 Done Done New false
18476 6260 Acked 7259 Flush 0 Done Done New false
18477 6261 Acked 7260 Read 6 Done Done New false
18478 6262 Acked 7261 Flush 0 Done Done New false
18479 6263 Acked 7262 Read 4 Done Done New false
18480 6264 Acked 7263 Flush 0 Done Done New false
18481 6265 Acked 7264 Write 4 Done Done New false
18482 6266 Acked 7265 Flush 0 Done Done New false
18483 6267 Acked 7266 Write 1 Done Done New false
18484 6268 Acked 7267 Write 5 Done Done New false
18485 6269 Acked 7268 Read 9 Done Done New false
18486 6270 Acked 7269 Write 6 Done Done New false
18487 6271 Acked 7270 Write 4 Done Done New false
18488 6272 Acked 7271 Write 3 Done Done New false
18489 6273 Acked 7272 Flush 0 Done Done New false
18490 6274 Acked 7273 Read 10 Done Done New false
18491 6275 Acked 7274 Write 8 Done Done New false
18492 6276 Acked 7275 Read 8 Done Done New false
18493 6277 Acked 7276 Read 7 Done Done New false
18494 6278 Acked 7277 Write 10 Done Done New false
18495 6279 Acked 7278 Read 5 Done Done New false
18496 6280 Acked 7279 Read 7 Done Done New false
18497 6281 Acked 7280 Write 10 Done Done New false
18498 6282 Acked 7281 Write 1 Done Done New false
18499 6283 Acked 7282 Read 4 Done Done New false
18500 6284 Acked 7283 Read 6 Done Done New false
18501 6285 Acked 7284 Read 10 Done Done New false
18502 6286 Acked 7285 Read 10 Done Done New false
18503 6287 Acked 7286 Read 3 Done Done New false
18504 6288 Acked 7287 Read 10 Done Done New false
18505 6289 Acked 7288 Read 4 Done Done New false
18506 6290 Acked 7289 Write 10 Done Done New false
18507 6291 Acked 7290 Read 2 Done Done New false
18508 6292 Acked 7291 Write 7 Done Done New false
18509 6293 Acked 7292 Read 6 Done Done New false
18510 6294 Acked 7293 Read 5 Done Done New false
18511 6295 Acked 7294 Read 9 Done Done New false
18512 6296 Acked 7295 Read 3 Done Done New false
18513 6297 Acked 7296 Read 9 Done Done New false
18514 6298 Acked 7297 Flush 0 Done Done New false
18515 6299 Acked 7298 Read 1 Done Done New false
18516 6300 Acked 7299 Write 6 Done Done New false
18517 6301 Acked 7300 Write 10 Done Done New false
18518 6302 Acked 7301 Read 4 Done Done New false
18519 6303 Acked 7302 Write 8 Done Done New false
18520 6304 Acked 7303 Read 8 Done Done New false
18521 6305 Acked 7304 Write 7 Done Done New false
18522 6306 Acked 7305 Read 8 Done Done New false
18523 6307 Acked 7306 Read 9 Done Done New false
18524 6308 Acked 7307 Write 3 Done Done New false
18525 6309 Acked 7308 Write 2 Done Done New false
18526 6310 Acked 7309 Write 10 Done Done New false
18527 6311 Acked 7310 Flush 0 Done Done New false
18528 6312 Acked 7311 Read 3 Done Done New false
18529 6313 Acked 7312 Read 3 Done Done New false
18530 6314 Acked 7313 Read 1 Done Done New false
18531 6315 Acked 7314 Read 7 Done Done New false
18532 6316 Acked 7315 Write 2 Done Done New false
18533 6317 Acked 7316 Read 10 Done Done New false
18534 6318 Acked 7317 Flush 0 Done Done New false
18535 6319 Acked 7318 Read 4 Done Done New false
18536 6320 Acked 7319 Read 9 Done Done New false
18537 6321 Acked 7320 Read 7 Done Done New false
18538 6322 Acked 7321 Write 3 Done Done New false
18539 6323 Acked 7322 Write 10 Done Done New false
18540 6324 Acked 7323 Flush 0 Done Done New false
18541 6325 Acked 7324 Write 4 Done Done New false
18542 6326 Acked 7325 Read 4 Done Done New false
18543 6327 Acked 7326 Flush 0 Done Done New false
18544 6328 Acked 7327 Read 3 Done Done New false
18545 6329 Acked 7328 Write 1 Done Done New false
18546 6330 Acked 7329 Read 2 Done Done New false
18547 STATES DS:0 DS:1 DS:2 TOTAL
18548 New 0 0 301 301
18549 Sent 0 0 0 0
18550 Done 301 301 0 602
18551 Skipped 0 0 0 0
18552 Error 0 0 0 0
18553 Last Flush: 7326 7326 7028
18554 Downstairs last five completed: 7028 7027 7026 7025 7024
18555 Upstairs last five completed: 6330 6329 6328 6327 6326
18556 CLIENT: Up:0 ds:301 act:2
185572024-03-02T01:09:17.905ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185582024-03-02T01:09:17.905ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185592024-03-02T01:09:17.905ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185602024-03-02T01:09:17.905ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185612024-03-02T01:09:17.905ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185622024-03-02T01:09:17.905ZINFOcrucible: [2] client re-new 302 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185632024-03-02T01:09:17.905ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18564 [8830] got start action
18565 [8830] I am going to restart
18566 Starting downstairs at port 8830
18567 Make output file at "/tmp/dsc/downstairs-8830.txt"
18568 [8830][2] reports Starting
185692024-03-02T01:09:18.906ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185702024-03-02T01:09:18.906ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185712024-03-02T01:09:18.906ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185722024-03-02T01:09:18.906ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185732024-03-02T01:09:18.906ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185742024-03-02T01:09:18.906ZINFOcrucible: [2] client re-new 302 jobs since flush 7028 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185752024-03-02T01:09:18.907ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18576 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1183)
18577 [8830][2] reports Running
185782024-03-02T01:09:19.908ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185792024-03-02T01:09:19.908ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185802024-03-02T01:09:19.909ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185812024-03-02T01:09:19.909ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185822024-03-02T01:09:19.909ZINFOcrucible: send last flush ID to this DS: 7028 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185832024-03-02T01:09:19.909ZINFOcrucible: Replied this last flush ID: 7028 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185842024-03-02T01:09:19.909ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185852024-03-02T01:09:19.909ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185862024-03-02T01:09:19.909ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
185872024-03-02T01:09:19.909ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18588 ----------------------------------------------------------------
18589 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18590 Downstairs last five completed: 7330 7329 7328 7327 7326
18591 Upstairs last five completed: 6331 6330 6329 6328 6327
18592 CLIENT: Up:0 ds:0 act:3
18593 Replay: All jobs finished, all DS active.
18594 Main task has work to do, go find it
18595 got dsc Stop(2)
18596 stop 2
18597 Mar 02 01:09:21.253 INFO request completed, latency_us: 358, response_code: 204, uri[8830] Got stop action so:false kr:false
18598 : /stop/cid/2, method: POST, req_id: f127b65c-8ba0-4663-93cd-4cbbdf87ea37, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
185992024-03-02T01:09:21.254ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186002024-03-02T01:09:21.254ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186012024-03-02T01:09:21.254ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186022024-03-02T01:09:21.254ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186032024-03-02T01:09:21.254ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186042024-03-02T01:09:21.254ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18605 Mar 02 01:09:21.254 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/2, method: GET, req_id: cdc81dc7-2e1b-451f-8baf-b180adf6c3be, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
18606 [8830] Exited with: ExitStatus(unix_wait_status(9))
18607 [8830][2] reports Exit
186082024-03-02T01:09:22.254ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186092024-03-02T01:09:22.254ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186102024-03-02T01:09:22.254ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186112024-03-02T01:09:22.254ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186122024-03-02T01:09:22.255ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186132024-03-02T01:09:22.255ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186142024-03-02T01:09:22.255ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186152024-03-02T01:09:23.256ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186162024-03-02T01:09:23.256ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186172024-03-02T01:09:23.256ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186182024-03-02T01:09:23.256ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186192024-03-02T01:09:23.256ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186202024-03-02T01:09:23.256ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186212024-03-02T01:09:23.256ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186222024-03-02T01:09:24.257ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186232024-03-02T01:09:24.257ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186242024-03-02T01:09:24.257ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186252024-03-02T01:09:24.257ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186262024-03-02T01:09:24.257ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186272024-03-02T01:09:24.257ZINFOcrucible: [2] client re-new 0 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186282024-03-02T01:09:24.257ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18629 Mar 02 01:09:25.256 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0945a8af-2c27-4bd3-a657-ecdb041be10d, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
18630 001/300 Write block 1758 len 12288 data: 4 6 5
18631 002/300 Read block 502 len 28672
186322024-03-02T01:09:25.258ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186332024-03-02T01:09:25.258ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186342024-03-02T01:09:25.258ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186352024-03-02T01:09:25.258ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186362024-03-02T01:09:25.258ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186372024-03-02T01:09:25.258ZINFOcrucible: [2] client re-new 2 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
186382024-03-02T01:09:25.258ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
18639 003/300 Read block 3930 len 12288
18640 004/300 Read block 692 len 32768
18641 005/300 Write block 1726 len 12288 data: 6 6 7
18642 006/300 Read block 19 len 24576
18643 007/300 Flush
18644 008/300 Write block 180 len 16384 data: 3 3 4 5
18645 009/300 Read block 1507 len 12288
18646 010/300 Write block 1289 len 28672 data: 3 3 3 3 4 4 4
18647 011/300 Write block 67 len 28672 data: 5 6 7 7 6 4 5
18648 012/300 Read block 4258 len 40960
18649 013/300 Read block 4426 len 16384
18650 014/300 Write block 3341 len 20480 data: 4 6 6 6 5
18651 015/300 Write block 1397 len 12288 data: 3 4 4
18652 016/300 Read block 3581 len 16384
18653 017/300 Write block 4992 len 16384 data: 8 6 5 6
18654 018/300 Read block 3591 len 12288
18655 019/300 Read block 1784 len 12288
18656 020/300 Write block 834 len 32768 data: 7 6 6 5 5 5 5 5
18657 021/300 Flush
18658 022/300 Write block 1690 len 40960 data: 6 6 7 8 7 8 6 6 8 5
18659 023/300 Read block 152 len 12288
18660 024/300 Read block 3332 len 24576
18661 025/300 Read block 3745 len 40960
18662 026/300 Flush
18663 027/300 Write block 3046 len 20480 data: 3 3 2 2 2
18664 028/300 Write block 295 len 8192 data: 10 9
18665 029/300 Read block 603 len 16384
18666 030/300 Read block 442 len 16384
18667 031/300 Read block 4052 len 40960
18668 032/300 Read block 664 len 40960
18669 033/300 Read block 3373 len 8192
18670 034/300 Read block 3081 len 8192
18671 035/300 Read block 1132 len 40960
18672 036/300 Read block 4205 len 24576
18673 037/300 Read block 2902 len 12288
18674 038/300 Write block 3425 len 8192 data: 4 4
18675 039/300 Flush
18676 040/300 Write block 1989 len 24576 data: 5 5 6 7 6 6
18677 041/300 Write block 2965 len 32768 data: 5 3 3 3 2 2 2 3
18678 042/300 Write block 290 len 8192 data: 7 7
18679 043/300 Read block 1589 len 24576
18680 044/300 Flush
18681 045/300 Read block 3806 len 12288
18682 046/300 Write block 1458 len 32768 data: 2 3 3 4 4 5 5 4
18683 047/300 Read block 1120 len 36864
18684 048/300 Write block 3771 len 24576 data: 4 4 4 3 4 5
18685 049/300 Read block 4631 len 36864
18686 050/300 Flush
18687 051/300 Read block 1575 len 40960
18688 052/300 Read block 4201 len 40960
18689 053/300 Write block 1207 len 28672 data: 4 4 6 7 7 7 7
18690 054/300 Flush
18691 055/300 Write block 692 len 36864 data: 6 6 5 5 6 6 6 4 4
18692 056/300 Read block 2195 len 28672
18693 057/300 Write block 4057 len 40960 data: 6 5 5 5 5 3 3 3 3 3
18694 058/300 Write block 2505 len 40960 data: 2 2 2 2 2 4 5 5 5 4
18695 059/300 Flush
18696 060/300 Write block 2957 len 36864 data: 5 4 5 5 5 4 3 3 6
18697 061/300 Write block 3036 len 28672 data: 4 4 4 4 3 3 3
18698 062/300 Write block 4309 len 4096 data: 5
18699 063/300 Write block 1369 len 32768 data: 3 3 2 2 3 3 3 2
18700 064/300 Write block 1230 len 12288 data: 5 6 5
18701 065/300 Flush
18702 066/300 Read block 776 len 20480
18703 067/300 Read block 856 len 32768
18704 068/300 Write block 3298 len 12288 data: 5 4 3
18705 069/300 Write block 1785 len 12288 data: 4 5 5
18706 070/300 Read block 402 len 12288
18707 071/300 Flush
18708 072/300 Flush
18709 073/300 Read block 80 len 24576
18710 074/300 Write block 2027 len 40960 data: 4 3 4 5 4 3 3 2 4 4
18711 075/300 Write block 1465 len 32768 data: 5 5 4 4 5 4 4 4
18712 076/300 Write block 4216 len 36864 data: 5 6 5 6 7 6 7 5 4
18713 077/300 Write block 4016 len 20480 data: 4 4 4 4 4
18714 078/300 Read block 4062 len 20480
18715 079/300 Write block 4224 len 28672 data: 5 3 3 4 4 4 5
18716 080/300 Read block 1513 len 32768
18717 081/300 Write block 3302 len 20480 data: 2 2 2 2 3
18718 082/300 Write block 756 len 32768 data: 5 7 8 7 6 6 5 5
18719 083/300 Flush
18720 084/300 Read block 1687 len 28672
18721 085/300 Read block 635 len 8192
18722 086/300 Write block 1604 len 36864 data: 4 4 4 4 6 7 5 4 4
18723 087/300 Write block 664 len 16384 data: 8 9 7 8
18724 088/300 Read block 936 len 28672
18725 089/300 Read block 2179 len 36864
18726 090/300 Write block 4034 len 32768 data: 4 4 6 4 6 6 6 6
18727 091/300 Read block 18 len 8192
18728 092/300 Read block 3640 len 28672
18729 093/300 Write block 924 len 24576 data: 7 5 6 5 6 3
18730 094/300 Read block 4607 len 36864
18731 095/300 Read block 1789 len 36864
18732 096/300 Write block 232 len 12288 data: 5 4 5
18733 097/300 Read block 4799 len 28672
18734 098/300 Write block 1788 len 8192 data: 5 4
18735 099/300 Write block 4283 len 28672 data: 2 2 2 2 3 3 3
18736 100/300 Write block 2482 len 24576 data: 6 5 5 6 4 4
18737 101/300 Read block 3642 len 36864
18738 102/300 Read block 4514 len 32768
18739 103/300 Read block 491 len 8192
18740 104/300 Write block 2384 len 28672 data: 3 3 2 2 3 4 5
18741 105/300 Flush
18742 106/300 Write block 2220 len 16384 data: 6 5 4 4
18743 107/300 Write block 874 len 40960 data: 3 2 3 4 4 4 3 4 3 3
18744 108/300 Read block 35 len 12288
18745 109/300 Read block 951 len 8192
18746 110/300 Read block 4701 len 16384
18747 111/300 Read block 4634 len 24576
18748 112/300 Read block 4595 len 12288
18749 113/300 Write block 50 len 40960 data: 6 6 6 5 4 3 2 5 4 4
18750 114/300 Write block 3892 len 40960 data: 4 4 4 4 4 3 4 4 5 5
18751 115/300 Write block 94 len 20480 data: 2 2 2 3 2
18752 116/300 Flush
18753 117/300 Flush
18754 118/300 Flush
18755 119/300 Read block 4130 len 12288
18756 120/300 Write block 4906 len 40960 data: 5 4 4 4 5 5 4 6 6 6
18757 121/300 Read block 3055 len 12288
18758 122/300 Write block 3782 len 36864 data: 10 10 10 8 7 6 6 7 6
18759 123/300 Read block 2997 len 40960
18760 124/300 Flush
18761 125/300 Write block 1741 len 20480 data: 6 3 2 3 4
18762 126/300 Read block 1907 len 40960
18763 127/300 Read block 1368 len 12288
18764 128/300 Read block 3551 len 28672
18765 129/300 Read block 4561 len 12288
18766 130/300 Write block 1828 len 40960 data: 2 3 4 4 4 6 6 6 4 6
18767 131/300 Write block 2759 len 12288 data: 3 5 6
18768 132/300 Write block 4003 len 8192 data: 5 5
18769 133/300 Write block 1433 len 12288 data: 5 4 5
18770 134/300 Write block 2241 len 24576 data: 4 4 3 4 4 3
18771 135/300 Read block 2768 len 32768
18772 136/300 Write block 1917 len 8192 data: 5 5
18773 137/300 Read block 4875 len 12288
18774 138/300 Read block 4227 len 36864
18775 139/300 Write block 3550 len 36864 data: 4 5 6 5 5 6 6 6 5
18776 140/300 Write block 3539 len 24576 data: 4 3 2 3 3 4
18777 141/300 Write block 4684 len 36864 data: 8 7 7 6 7 7 6 5 5
18778 142/300 Read block 2741 len 40960
18779 143/300 Write block 910 len 24576 data: 3 4 5 5 5 6
18780 144/300 Write block 199 len 36864 data: 6 6 6 7 5 4 4 3 3
18781 145/300 Read block 3213 len 28672
18782 146/300 Write block 3259 len 36864 data: 3 3 5 5 4 4 5 4 4
18783 147/300 Write block 1561 len 8192 data: 3 3
18784 148/300 Read block 913 len 12288
18785 149/300 Read block 2307 len 20480
18786 150/300 Write block 3531 len 4096 data: 7
18787 151/300 Read block 1895 len 24576
18788 152/300 Read block 2386 len 32768
18789 153/300 Flush
18790 154/300 Read block 1196 len 8192
18791 155/300 Write block 3826 len 16384 data: 4 4 3 4
18792 156/300 Read block 3911 len 36864
18793 157/300 Write block 2309 len 24576 data: 3 3 3 3 5 6
18794 158/300 Write block 1605 len 16384 data: 5 5 5 7
18795 159/300 Write block 1102 len 4096 data: 5
18796 160/300 Write block 1404 len 16384 data: 5 6 6 6
18797 161/300 Read block 2071 len 8192
18798 162/300 Read block 4271 len 36864
18799 163/300 Write block 761 len 4096 data: 7
18800 164/300 Read block 467 len 40960
18801 165/300 Read block 2653 len 16384
18802 166/300 Read block 4743 len 40960
18803 167/300 Read block 733 len 24576
18804 168/300 Flush
18805 169/300 Write block 4411 len 36864 data: 4 3 3 4 4 4 5 4 5
18806 170/300 Write block 877 len 16384 data: 5 5 5 4
18807 171/300 Read block 534 len 40960
18808 172/300 Write block 1514 len 24576 data: 5 5 4 4 4 4
18809 173/300 Read block 1450 len 40960
18810 174/300 Flush
18811 175/300 Write block 4632 len 4096 data: 4
18812 176/300 Write block 4207 len 24576 data: 5 4 4 5 5 4
18813 177/300 Read block 262 len 16384
18814 178/300 Write block 1098 len 16384 data: 5 5 5 6
18815 179/300 Read block 4943 len 32768
18816 180/300 Read block 2394 len 8192
18817 181/300 Read block 2864 len 36864
18818 182/300 Read block 4008 len 28672
18819 183/300 Read block 598 len 12288
18820 184/300 Write block 3971 len 28672 data: 4 4 2 3 2 2 4
18821 185/300 Read block 4774 len 8192
18822 186/300 Read block 4941 len 16384
18823 187/300 Write block 3656 len 8192 data: 3 4
18824 188/300 Read block 2297 len 24576
18825 189/300 Read block 2945 len 24576
18826 190/300 Write block 75 len 4096 data: 4
18827 191/300 Flush
18828 192/300 Flush
18829 193/300 Write block 2654 len 36864 data: 5 4 4 5 3 4 5 6 5
18830 194/300 Flush
18831 195/300 Read block 4443 len 40960
18832 196/300 Write block 4731 len 36864 data: 6 6 5 6 5 6 6 4 4
18833 197/300 Read block 1388 len 20480
18834 198/300 Read block 4689 len 32768
18835 199/300 Read block 2335 len 36864
18836 200/300 Read block 2254 len 36864
18837 201/300 Flush
18838 202/300 Write block 1057 len 16384 data: 4 4 5 5
18839 203/300 Read block 968 len 4096
18840 204/300 Write block 1924 len 16384 data: 6 6 6 6
18841 205/300 Flush
18842 206/300 Read block 906 len 12288
18843 207/300 Write block 4347 len 20480 data: 6 5 5 6 6
18844 208/300 Write block 4582 len 28672 data: 4 5 5 6 5 6 7
18845 209/300 Read block 2059 len 24576
18846 210/300 Read block 2383 len 16384
18847 211/300 Read block 1862 len 12288
18848 212/300 Read block 1089 len 28672
18849 213/300 Read block 4342 len 4096
18850 214/300 Write block 2171 len 36864 data: 8 9 10 11 10 8 7 7 6
18851 215/300 Write block 1101 len 36864 data: 7 6 4 4 3 3 3 3 3
18852 216/300 Read block 3677 len 24576
18853 217/300 Read block 4347 len 4096
18854 218/300 Read block 2369 len 16384
18855 219/300 Flush
18856 220/300 Read block 2395 len 32768
18857 221/300 Write block 3106 len 16384 data: 3 3 3 3
18858 222/300 Write block 3201 len 20480 data: 2 3 3 3 3
18859 223/300 Write block 3577 len 4096 data: 4
18860 224/300 Write block 3016 len 8192 data: 8 8
18861 225/300 Write block 3796 len 4096 data: 2
18862 226/300 Write block 1374 len 28672 data: 4 4 3 4 4 3 4
18863 227/300 Write block 3712 len 24576 data: 8 7 7 6 7 7
18864 228/300 Write block 2099 len 12288 data: 5 5 5
18865 229/300 Read block 3211 len 8192
18866 230/300 Read block 522 len 28672
18867 231/300 Write block 3580 len 40960 data: 3 3 3 3 3 3 3 5 6 6
18868 232/300 Read block 1601 len 4096
18869 233/300 Write block 4769 len 24576 data: 6 6 5 5 6 5
18870 234/300 Read block 731 len 4096
18871 235/300 Read block 3465 len 12288
18872 236/300 Write block 3256 len 28672 data: 2 2 3 4 4 6 6
18873 237/300 Read block 1969 len 24576
18874 238/300 Read block 4162 len 32768
18875 239/300 Read block 4950 len 40960
18876 240/300 Read block 1917 len 40960
18877 241/300 Read block 3862 len 36864
18878 242/300 Flush
18879 243/300 Write block 3151 len 8192 data: 5 5
18880 244/300 Write block 1985 len 8192 data: 7 8
18881 245/300 Read block 1278 len 8192
18882 246/300 Read block 4888 len 40960
18883 247/300 Write block 1940 len 8192 data: 7 8
18884 248/300 Read block 226 len 36864
18885 249/300 Read block 3546 len 40960
18886 250/300 Write block 2816 len 28672 data: 7 9 8 8 9 6 5
18887 251/300 Read block 2602 len 28672
18888 252/300 Read block 3614 len 32768
18889 253/300 Read block 1138 len 28672
18890 254/300 Flush
18891 255/300 Write block 4265 len 20480 data: 6 4 5 5 3
18892 256/300 Write block 2638 len 28672 data: 8 7 4 4 4 4 3
18893 257/300 Read block 4167 len 36864
18894 258/300 Read block 4928 len 20480
18895 259/300 Read block 2673 len 8192
18896 260/300 Read block 2559 len 8192
18897 261/300 Write block 1238 len 4096 data: 5
18898 262/300 Read block 4828 len 40960
18899 263/300 Write block 1618 len 28672 data: 6 6 5 5 5 5 5
18900 264/300 Flush
18901 265/300 Write block 1501 len 24576 data: 3 4 4 5 5 7
18902 266/300 Flush
18903 267/300 Read block 2790 len 4096
18904 268/300 Write block 2786 len 28672 data: 4 4 4 4 6 7 7
18905 269/300 Write block 4719 len 40960 data: 3 4 4 2 4 4 6 6 6 6
18906 270/300 Read block 471 len 32768
18907 271/300 Flush
18908 272/300 Read block 2995 len 36864
18909 273/300 Read block 2106 len 20480
18910 274/300 Flush
18911 275/300 Write block 2470 len 16384 data: 4 5 5 5
18912 276/300 Flush
18913 277/300 Flush
18914 278/300 Write block 4508 len 28672 data: 5 4 3 2 2 2 3
18915 279/300 Read block 4547 len 8192
18916 280/300 Write block 2565 len 16384 data: 5 5 4 5
18917 281/300 Flush
18918 282/300 Flush
18919 283/300 Write block 2702 len 36864 data: 3 3 4 6 5 5 6 6 4
18920 284/300 Read block 4261 len 16384
18921 285/300 Write block 104 len 4096 data: 4
18922 286/300 Read block 1581 len 8192
18923 287/300 Flush
18924 288/300 Write block 2167 len 16384 data: 4 5 6 6
18925 289/300 Read block 118 len 28672
18926 290/300 Write block 4658 len 36864 data: 6 8 8 8 6 3 3 4 3
18927 291/300 Write block 1636 len 4096 data: 4
18928 292/300 Write block 4200 len 8192 data: 3 3
18929 293/300 Read block 3395 len 8192
18930 294/300 Write block 4705 len 20480 data: 3 3 3 2 2
18931 295/300 Read block 3898 len 36864
18932 296/300 Flush
18933 297/300 Read block 306 len 20480
18934 298/300 Read block 3103 len 24576
18935 299/300 Read block 2283 len 24576
18936 300/300 Read block 1352 len 12288
18937 Main task has work to do, go find it
18938 got dsc Start(2)
18939 start 2
18940 Mar 02 01:09:25.622 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: a3f8da59-d94a-4daf-9831-4138cb9f38c0, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
18941 Replay: started 2, returned:Ok(())
18942 ----------------------------------------------------------------
18943 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18944 States: Active Active Offline
18945 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18946 6332 Acked 7331 Write 3 Done Done New true
18947 6333 Acked 7332 Read 7 Done Done New true
18948 6334 Acked 7333 Read 3 Done Done New false
18949 6335 Acked 7334 Read 8 Done Done New false
18950 6336 Acked 7335 Write 3 Done Done New false
18951 6337 Acked 7336 Read 6 Done Done New false
18952 6338 Acked 7337 Flush 0 Done Done New false
18953 6339 Acked 7338 Write 4 Done Done New false
18954 6340 Acked 7339 Read 3 Done Done New false
18955 6341 Acked 7340 Write 7 Done Done New false
18956 6342 Acked 7341 Write 7 Done Done New false
18957 6343 Acked 7342 Read 10 Done Done New false
18958 6344 Acked 7343 Read 4 Done Done New false
18959 6345 Acked 7344 Write 5 Done Done New false
18960 6346 Acked 7345 Write 3 Done Done New false
18961 6347 Acked 7346 Read 4 Done Done New false
18962 6348 Acked 7347 Write 4 Done Done New false
18963 6349 Acked 7348 Read 3 Done Done New false
18964 6350 Acked 7349 Read 3 Done Done New false
18965 6351 Acked 7350 Write 8 Done Done New false
18966 6352 Acked 7351 Flush 0 Done Done New false
18967 6353 Acked 7352 Write 10 Done Done New false
18968 6354 Acked 7353 Read 3 Done Done New false
18969 6355 Acked 7354 Read 6 Done Done New false
18970 6356 Acked 7355 Read 10 Done Done New false
18971 6357 Acked 7356 Flush 0 Done Done New false
18972 6358 Acked 7357 Write 5 Done Done New false
18973 6359 Acked 7358 Write 2 Done Done New false
18974 6360 Acked 7359 Read 4 Done Done New false
18975 6361 Acked 7360 Read 4 Done Done New false
18976 6362 Acked 7361 Read 10 Done Done New false
18977 6363 Acked 7362 Read 10 Done Done New false
18978 6364 Acked 7363 Read 2 Done Done New false
18979 6365 Acked 7364 Read 2 Done Done New false
18980 6366 Acked 7365 Read 10 Done Done New false
18981 6367 Acked 7366 Read 6 Done Done New false
18982 6368 Acked 7367 Read 3 Done Done New false
18983 6369 Acked 7368 Write 2 Done Done New false
18984 6370 Acked 7369 Flush 0 Done Done New false
18985 6371 Acked 7370 Write 6 Done Done New false
18986 6372 Acked 7371 Write 8 Done Done New false
18987 6373 Acked 7372 Write 2 Done Done New false
18988 6374 Acked 7373 Read 6 Done Done New false
18989 6375 Acked 7374 Flush 0 Done Done New false
18990 6376 Acked 7375 Read 3 Done Done New false
18991 6377 Acked 7376 Write 8 Done Done New false
18992 6378 Acked 7377 Read 9 Done Done New false
18993 6379 Acked 7378 Write 6 Done Done New false
18994 6380 Acked 7379 Read 9 Done Done New false
18995 6381 Acked 7380 Flush 0 Done Done New false
18996 6382 Acked 7381 Read 10 Done Done New false
18997 6383 Acked 7382 Read 10 Done Done New false
18998 6384 Acked 7383 Write 7 Done Done New false
18999 6385 Acked 7384 Flush 0 Done Done New false
19000 6386 Acked 7385 Write 9 Done Done New false
19001 6387 Acked 7386 Read 7 Done Done New false
19002 6388 Acked 7387 Write 10 Done Done New false
19003 6389 Acked 7388 Write 10 Done Done New false
19004 6390 Acked 7389 Flush 0 Done Done New false
19005 6391 Acked 7390 Write 9 Done Done New false
19006 6392 Acked 7391 Write 7 Done Done New false
19007 6393 Acked 7392 Write 1 Done Done New false
19008 6394 Acked 7393 Write 8 Done Done New false
19009 6395 Acked 7394 Write 3 Done Done New false
19010 6396 Acked 7395 Flush 0 Done Done New false
19011 6397 Acked 7396 Read 5 Done Done New false
19012 6398 Acked 7397 Read 8 Done Done New false
19013 6399 Acked 7398 Write 3 Done Done New false
19014 6400 Acked 7399 Write 3 Done Done New false
19015 6401 Acked 7400 Read 3 Done Done New false
19016 6402 Acked 7401 Flush 0 Done Done New false
19017 6403 Acked 7402 Flush 0 Done Done New false
19018 6404 Acked 7403 Read 6 Done Done New false
19019 6405 Acked 7404 Write 10 Done Done New false
19020 6406 Acked 7405 Write 8 Done Done New false
19021 6407 Acked 7406 Write 9 Done Done New false
19022 6408 Acked 7407 Write 5 Done Done New false
19023 6409 Acked 7408 Read 5 Done Done New false
19024 6410 Acked 7409 Write 7 Done Done New false
19025 6411 Acked 7410 Read 8 Done Done New false
19026 6412 Acked 7411 Write 5 Done Done New false
19027 6413 Acked 7412 Write 8 Done Done New false
19028 6414 Acked 7413 Flush 0 Done Done New false
19029 6415 Acked 7414 Read 7 Done Done New false
19030 6416 Acked 7415 Read 2 Done Done New false
19031 6417 Acked 7416 Write 9 Done Done New false
19032 6418 Acked 7417 Write 4 Done Done New false
19033 6419 Acked 7418 Read 7 Done Done New false
19034 6420 Acked 7419 Read 9 Done Done New false
19035 6421 Acked 7420 Write 8 Done Done New false
19036 6422 Acked 7421 Read 2 Done Done New false
19037 6423 Acked 7422 Read 7 Done Done New false
19038 6424 Acked 7423 Write 6 Done Done New false
19039 6425 Acked 7424 Read 9 Done Done New false
19040 6426 Acked 7425 Read 9 Done Done New false
19041 6427 Acked 7426 Write 3 Done Done New false
19042 6428 Acked 7427 Read 7 Done Done New false
19043 6429 Acked 7428 Write 2 Done Done New false
19044 6430 Acked 7429 Write 7 Done Done New false
19045 6431 Acked 7430 Write 6 Done Done New false
19046 6432 Acked 7431 Read 9 Done Done New false
19047 6433 Acked 7432 Read 8 Done Done New false
19048 6434 Acked 7433 Read 2 Done Done New false
19049 6435 Acked 7434 Write 7 Done Done New false
19050 6436 Acked 7435 Flush 0 Done Done New false
19051 6437 Acked 7436 Write 4 Done Done New false
19052 6438 Acked 7437 Write 10 Done Done New false
19053 6439 Acked 7438 Read 3 Done Done New false
19054 6440 Acked 7439 Read 2 Done Done New false
19055 6441 Acked 7440 Read 4 Done Done New false
19056 6442 Acked 7441 Read 6 Done Done New false
19057 6443 Acked 7442 Read 3 Done Done New false
19058 6444 Acked 7443 Write 10 Done Done New false
19059 6445 Acked 7444 Write 10 Done Done New false
19060 6446 Acked 7445 Write 5 Done Done New false
19061 6447 Acked 7446 Flush 0 Done Done New false
19062 6448 Acked 7447 Flush 0 Done Done New false
19063 6449 Acked 7448 Flush 0 Done Done New false
19064 6450 Acked 7449 Read 3 Done Done New false
19065 6451 Acked 7450 Write 10 Done Done New false
19066 6452 Acked 7451 Read 3 Done Done New false
19067 6453 Acked 7452 Write 9 Done Done New false
19068 6454 Acked 7453 Read 10 Done Done New false
19069 6455 Acked 7454 Flush 0 Done Done New false
19070 6456 Acked 7455 Write 5 Done Done New false
19071 6457 Acked 7456 Read 10 Done Done New false
19072 6458 Acked 7457 Read 3 Done Done New false
19073 6459 Acked 7458 Read 7 Done Done New false
19074 6460 Acked 7459 Read 3 Done Done New false
19075 6461 Acked 7460 Write 10 Done Done New false
19076 6462 Acked 7461 Write 3 Done Done New false
19077 6463 Acked 7462 Write 2 Done Done New false
19078 6464 Acked 7463 Write 3 Done Done New false
19079 6465 Acked 7464 Write 6 Done Done New false
19080 6466 Acked 7465 Read 8 Done Done New false
19081 6467 Acked 7466 Write 2 Done Done New false
19082 6468 Acked 7467 Read 3 Done Done New false
19083 6469 Acked 7468 Read 9 Done Done New false
19084 6470 Acked 7469 Write 9 Done Done New false
19085 6471 Acked 7470 Write 6 Done Done New false
19086 6472 Acked 7471 Write 9 Done Done New false
19087 6473 Acked 7472 Read 10 Done Done New false
19088 6474 Acked 7473 Write 6 Done Done New false
19089 6475 Acked 7474 Write 9 Done Done New false
19090 6476 Acked 7475 Read 7 Done Done New false
19091 6477 Acked 7476 Write 9 Done Done New false
19092 6478 Acked 7477 Write 2 Done Done New false
19093 6479 Acked 7478 Read 3 Done Done New false
19094 6480 Acked 7479 Read 5 Done Done New false
19095 6481 Acked 7480 Write 1 Done Done New false
19096 6482 Acked 7481 Read 6 Done Done New false
19097 6483 Acked 7482 Read 8 Done Done New false
19098 6484 Acked 7483 Flush 0 Done Done New false
19099 6485 Acked 7484 Read 2 Done Done New false
19100 6486 Acked 7485 Write 4 Done Done New false
19101 6487 Acked 7486 Read 9 Done Done New false
19102 6488 Acked 7487 Write 6 Done Done New false
19103 6489 Acked 7488 Write 4 Done Done New false
19104 6490 Acked 7489 Write 1 Done Done New false
19105 6491 Acked 7490 Write 4 Done Done New false
19106 6492 Acked 7491 Read 2 Done Done New false
19107 6493 Acked 7492 Read 9 Done Done New false
19108 6494 Acked 7493 Write 1 Done Done New false
19109 6495 Acked 7494 Read 10 Done Done New false
19110 6496 Acked 7495 Read 4 Done Done New false
19111 6497 Acked 7496 Read 10 Done Done New false
19112 6498 Acked 7497 Read 6 Done Done New false
19113 6499 Acked 7498 Flush 0 Done Done New false
19114 6500 Acked 7499 Write 9 Done Done New false
19115 6501 Acked 7500 Write 4 Done Done New false
19116 6502 Acked 7501 Read 10 Done Done New false
19117 6503 Acked 7502 Write 6 Done Done New false
19118 6504 Acked 7503 Read 10 Done Done New false
19119 6505 Acked 7504 Flush 0 Done Done New false
19120 6506 Acked 7505 Write 1 Done Done New false
19121 6507 Acked 7506 Write 6 Done Done New false
19122 6508 Acked 7507 Read 4 Done Done New false
19123 6509 Acked 7508 Write 4 Done Done New false
19124 6510 Acked 7509 Read 8 Done Done New false
19125 6511 Acked 7510 Read 2 Done Done New false
19126 6512 Acked 7511 Read 9 Done Done New false
19127 6513 Acked 7512 Read 7 Done Done New false
19128 6514 Acked 7513 Read 3 Done Done New false
19129 6515 Acked 7514 Write 7 Done Done New false
19130 6516 Acked 7515 Read 2 Done Done New false
19131 6517 Acked 7516 Read 4 Done Done New false
19132 6518 Acked 7517 Write 2 Done Done New false
19133 6519 Acked 7518 Read 6 Done Done New false
19134 6520 Acked 7519 Read 6 Done Done New false
19135 6521 Acked 7520 Write 1 Done Done New false
19136 6522 Acked 7521 Flush 0 Done Done New false
19137 6523 Acked 7522 Flush 0 Done Done New false
19138 6524 Acked 7523 Write 9 Done Done New false
19139 6525 Acked 7524 Flush 0 Done Done New false
19140 6526 Acked 7525 Read 10 Done Done New false
19141 6527 Acked 7526 Write 9 Done Done New false
19142 6528 Acked 7527 Read 5 Done Done New false
19143 6529 Acked 7528 Read 8 Done Done New false
19144 6530 Acked 7529 Read 9 Done Done New false
19145 6531 Acked 7530 Read 9 Done Done New false
19146 6532 Acked 7531 Flush 0 Done Done New false
19147 6533 Acked 7532 Write 4 Done Done New false
19148 6534 Acked 7533 Read 1 Done Done New false
19149 6535 Acked 7534 Write 4 Done Done New false
19150 6536 Acked 7535 Flush 0 Done Done New false
19151 6537 Acked 7536 Read 3 Done Done New false
19152 6538 Acked 7537 Write 5 Done Done New false
19153 6539 Acked 7538 Write 7 Done Done New false
19154 6540 Acked 7539 Read 6 Done Done New false
19155 6541 Acked 7540 Read 4 Done Done New false
19156 6542 Acked 7541 Read 3 Done Done New false
19157 6543 Acked 7542 Read 7 Done Done New false
19158 6544 Acked 7543 Read 1 Done Done New false
19159 6545 Acked 7544 Write 9 Done Done New false
19160 6546 Acked 7545 Write 9 Done Done New false
19161 6547 Acked 7546 Read 6 Done Done New false
19162 6548 Acked 7547 Read 1 Done Done New false
19163 6549 Acked 7548 Read 4 Done Done New false
19164 6550 Acked 7549 Flush 0 Done Done New false
19165 6551 Acked 7550 Read 8 Done Done New false
19166 6552 Acked 7551 Write 4 Done Done New false
19167 6553 Acked 7552 Write 5 Done Done New false
19168 6554 Acked 7553 Write 1 Done Done New false
19169 6555 Acked 7554 Write 2 Done Done New false
19170 6556 Acked 7555 Write 1 Done Done New false
19171 6557 Acked 7556 Write 7 Done Done New false
19172 6558 Acked 7557 Write 6 Done Done New false
19173 6559 Acked 7558 Write 3 Done Done New false
19174 6560 Acked 7559 Read 2 Done Done New false
19175 6561 Acked 7560 Read 7 Done Done New false
19176 6562 Acked 7561 Write 10 Done Done New false
19177 6563 Acked 7562 Read 1 Done Done New false
19178 6564 Acked 7563 Write 6 Done Done New false
19179 6565 Acked 7564 Read 1 Done Done New false
19180 6566 Acked 7565 Read 3 Done Done New false
19181 6567 Acked 7566 Write 7 Done Done New false
19182 6568 Acked 7567 Read 6 Done Done New false
19183 6569 Acked 7568 Read 8 Done Done New false
19184 6570 Acked 7569 Read 10 Done Done New false
19185 6571 Acked 7570 Read 10 Done Done New false
19186 6572 Acked 7571 Read 9 Done Done New false
19187 6573 Acked 7572 Flush 0 Done Done New false
19188 6574 Acked 7573 Write 2 Done Done New false
19189 6575 Acked 7574 Write 2 Done Done New false
19190 6576 Acked 7575 Read 2 Done Done New false
19191 6577 Acked 7576 Read 10 Done Done New false
19192 6578 Acked 7577 Write 2 Done Done New false
19193 6579 Acked 7578 Flush 0 Done Done New false
19194 6580 Acked 7579 Read 9 Done Done New false
19195 6581 Acked 7580 Read 10 Done Done New false
19196 6582 Acked 7581 Write 7 Done Done New false
19197 6583 Acked 7582 Read 7 Done Done New false
19198 6584 Acked 7583 Read 8 Done Done New false
19199 6585 Acked 7584 Read 7 Done Done New false
19200 6586 Acked 7585 Flush 0 Done Done New false
19201 6587 Acked 7586 Write 5 Done Done New false
19202 6588 Acked 7587 Write 7 Done Done New false
19203 6589 Acked 7588 Read 9 Done Done New false
19204 6590 Acked 7589 Read 5 Done Done New false
19205 6591 Acked 7590 Read 2 Done Done New false
19206 6592 Acked 7591 Read 2 Done Done New false
19207 6593 Acked 7592 Write 1 Done Done New false
19208 6594 Acked 7593 Read 10 Done Done New false
19209 6595 Acked 7594 Write 7 Done Done New false
19210 6596 Acked 7595 Flush 0 Done Done New false
19211 6597 Acked 7596 Write 6 Done Done New false
19212 6598 Acked 7597 Flush 0 Done Done New false
19213 6599 Acked 7598 Read 1 Done Done New false
19214 6600 Acked 7599 Write 7 Done Done New false
19215 6601 Acked 7600 Write 10 Done Done New false
19216 6602 Acked 7601 Read 8 Done Done New false
19217 6603 Acked 7602 Flush 0 Done Done New false
19218 6604 Acked 7603 Read 9 Done Done New false
19219 6605 Acked 7604 Read 5 Done Done New false
19220 6606 Acked 7605 Flush 0 Done Done New false
19221 6607 Acked 7606 Write 4 Done Done New false
19222 6608 Acked 7607 Flush 0 Done Done New false
19223 6609 Acked 7608 Flush 0 Done Done New false
19224 6610 Acked 7609 Write 7 Done Done New false
19225 6611 Acked 7610 Read 2 Done Done New false
19226 6612 Acked 7611 Write 4 Done Done New false
19227 6613 Acked 7612 Flush 0 Done Done New false
19228 6614 Acked 7613 Flush 0 Done Done New false
19229 6615 Acked 7614 Write 9 Done Done New false
19230 6616 Acked 7615 Read 4 Done Done New false
19231 6617 Acked 7616 Write 1 Done Done New false
19232 6618 Acked 7617 Read 2 Done Done New false
19233 6619 Acked 7618 Flush 0 Done Done New false
19234 6620 Acked 7619 Write 4 Done Done New false
19235 6621 Acked 7620 Read 7 Done Done New false
19236 6622 Acked 7621 Write 9 Done Done New false
19237 6623 Acked 7622 Write 1 Done Done New false
19238 6624 Acked 7623 Write 2 Done Done New false
19239 6625 Acked 7624 Read 2 Done Done New false
19240 6626 Acked 7625 Write 5 Done Done New false
19241 6627 Acked 7626 Read 9 Done Done New false
19242 6628 Acked 7627 Flush 0 Done Done New false
19243 6629 Acked 7628 Read 5 Done Done New false
19244 6630 Acked 7629 Read 6 Done Done New false
19245 6631 Acked 7630 Read 6 Done Done New false
19246 6632 Acked 7631 Read 3 Done Done New false
19247 STATES DS:0 DS:1 DS:2 TOTAL
19248 New 0 0 301 301
19249 Sent 0 0 0 0
19250 Done 301 301 0 602
19251 Skipped 0 0 0 0
19252 Error 0 0 0 0
19253 Last Flush: 7627 7627 7330
19254 Downstairs last five completed: 7330 7329 7328 7327 7326
19255 Upstairs last five completed: 6632 6631 6630 6629 6628
19256 CLIENT: Up:0 ds:301 act:2
192572024-03-02T01:09:26.260ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192582024-03-02T01:09:26.260ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192592024-03-02T01:09:26.260ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192602024-03-02T01:09:26.260ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192612024-03-02T01:09:26.260ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192622024-03-02T01:09:26.260ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192632024-03-02T01:09:26.260ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192642024-03-02T01:09:27.261ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192652024-03-02T01:09:27.261ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192662024-03-02T01:09:27.261ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192672024-03-02T01:09:27.261ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192682024-03-02T01:09:27.261ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192692024-03-02T01:09:27.262ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192702024-03-02T01:09:27.262ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192712024-03-02T01:09:28.263ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192722024-03-02T01:09:28.263ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192732024-03-02T01:09:28.263ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192742024-03-02T01:09:28.263ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192752024-03-02T01:09:28.263ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192762024-03-02T01:09:28.263ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192772024-03-02T01:09:28.263ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192782024-03-02T01:09:29.264ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192792024-03-02T01:09:29.264ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192802024-03-02T01:09:29.264ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192812024-03-02T01:09:29.264ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192822024-03-02T01:09:29.264ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192832024-03-02T01:09:29.264ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
192842024-03-02T01:09:29.264ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19285 ----------------------------------------------------------------
19286 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19287 States: Active Active Offline
19288 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19289 6332 Acked 7331 Write 3 Done Done New true
19290 6333 Acked 7332 Read 7 Done Done New true
19291 6334 Acked 7333 Read 3 Done Done New true
19292 6335 Acked 7334 Read 8 Done Done New true
19293 6336 Acked 7335 Write 3 Done Done New true
19294 6337 Acked 7336 Read 6 Done Done New true
19295 6338 Acked 7337 Flush 0 Done Done New true
19296 6339 Acked 7338 Write 4 Done Done New true
19297 6340 Acked 7339 Read 3 Done Done New true
19298 6341 Acked 7340 Write 7 Done Done New true
19299 6342 Acked 7341 Write 7 Done Done New true
19300 6343 Acked 7342 Read 10 Done Done New true
19301 6344 Acked 7343 Read 4 Done Done New true
19302 6345 Acked 7344 Write 5 Done Done New true
19303 6346 Acked 7345 Write 3 Done Done New true
19304 6347 Acked 7346 Read 4 Done Done New true
19305 6348 Acked 7347 Write 4 Done Done New true
19306 6349 Acked 7348 Read 3 Done Done New true
19307 6350 Acked 7349 Read 3 Done Done New true
19308 6351 Acked 7350 Write 8 Done Done New true
19309 6352 Acked 7351 Flush 0 Done Done New true
19310 6353 Acked 7352 Write 10 Done Done New true
19311 6354 Acked 7353 Read 3 Done Done New true
19312 6355 Acked 7354 Read 6 Done Done New true
19313 6356 Acked 7355 Read 10 Done Done New true
19314 6357 Acked 7356 Flush 0 Done Done New true
19315 6358 Acked 7357 Write 5 Done Done New true
19316 6359 Acked 7358 Write 2 Done Done New true
19317 6360 Acked 7359 Read 4 Done Done New true
19318 6361 Acked 7360 Read 4 Done Done New true
19319 6362 Acked 7361 Read 10 Done Done New true
19320 6363 Acked 7362 Read 10 Done Done New true
19321 6364 Acked 7363 Read 2 Done Done New true
19322 6365 Acked 7364 Read 2 Done Done New true
19323 6366 Acked 7365 Read 10 Done Done New true
19324 6367 Acked 7366 Read 6 Done Done New true
19325 6368 Acked 7367 Read 3 Done Done New true
19326 6369 Acked 7368 Write 2 Done Done New true
19327 6370 Acked 7369 Flush 0 Done Done New true
19328 6371 Acked 7370 Write 6 Done Done New true
19329 6372 Acked 7371 Write 8 Done Done New true
19330 6373 Acked 7372 Write 2 Done Done New true
19331 6374 Acked 7373 Read 6 Done Done New true
19332 6375 Acked 7374 Flush 0 Done Done New true
19333 6376 Acked 7375 Read 3 Done Done New true
19334 6377 Acked 7376 Write 8 Done Done New true
19335 6378 Acked 7377 Read 9 Done Done New true
19336 6379 Acked 7378 Write 6 Done Done New true
19337 6380 Acked 7379 Read 9 Done Done New true
19338 6381 Acked 7380 Flush 0 Done Done New true
19339 6382 Acked 7381 Read 10 Done Done New true
19340 6383 Acked 7382 Read 10 Done Done New true
19341 6384 Acked 7383 Write 7 Done Done New true
19342 6385 Acked 7384 Flush 0 Done Done New true
19343 6386 Acked 7385 Write 9 Done Done New true
19344 6387 Acked 7386 Read 7 Done Done New true
19345 6388 Acked 7387 Write 10 Done Done New true
19346 6389 Acked 7388 Write 10 Done Done New true
19347 6390 Acked 7389 Flush 0 Done Done New true
19348 6391 Acked 7390 Write 9 Done Done New true
19349 6392 Acked 7391 Write 7 Done Done New true
19350 6393 Acked 7392 Write 1 Done Done New true
19351 6394 Acked 7393 Write 8 Done Done New true
19352 6395 Acked 7394 Write 3 Done Done New true
19353 6396 Acked 7395 Flush 0 Done Done New true
19354 6397 Acked 7396 Read 5 Done Done New true
19355 6398 Acked 7397 Read 8 Done Done New true
19356 6399 Acked 7398 Write 3 Done Done New true
19357 6400 Acked 7399 Write 3 Done Done New true
19358 6401 Acked 7400 Read 3 Done Done New true
19359 6402 Acked 7401 Flush 0 Done Done New true
19360 6403 Acked 7402 Flush 0 Done Done New true
19361 6404 Acked 7403 Read 6 Done Done New true
19362 6405 Acked 7404 Write 10 Done Done New true
19363 6406 Acked 7405 Write 8 Done Done New true
19364 6407 Acked 7406 Write 9 Done Done New true
19365 6408 Acked 7407 Write 5 Done Done New true
19366 6409 Acked 7408 Read 5 Done Done New true
19367 6410 Acked 7409 Write 7 Done Done New true
19368 6411 Acked 7410 Read 8 Done Done New true
19369 6412 Acked 7411 Write 5 Done Done New true
19370 6413 Acked 7412 Write 8 Done Done New true
19371 6414 Acked 7413 Flush 0 Done Done New true
19372 6415 Acked 7414 Read 7 Done Done New true
19373 6416 Acked 7415 Read 2 Done Done New true
19374 6417 Acked 7416 Write 9 Done Done New true
19375 6418 Acked 7417 Write 4 Done Done New true
19376 6419 Acked 7418 Read 7 Done Done New true
19377 6420 Acked 7419 Read 9 Done Done New true
19378 6421 Acked 7420 Write 8 Done Done New true
19379 6422 Acked 7421 Read 2 Done Done New true
19380 6423 Acked 7422 Read 7 Done Done New true
19381 6424 Acked 7423 Write 6 Done Done New true
19382 6425 Acked 7424 Read 9 Done Done New true
19383 6426 Acked 7425 Read 9 Done Done New true
19384 6427 Acked 7426 Write 3 Done Done New true
19385 6428 Acked 7427 Read 7 Done Done New true
19386 6429 Acked 7428 Write 2 Done Done New true
19387 6430 Acked 7429 Write 7 Done Done New true
19388 6431 Acked 7430 Write 6 Done Done New true
19389 6432 Acked 7431 Read 9 Done Done New true
19390 6433 Acked 7432 Read 8 Done Done New true
19391 6434 Acked 7433 Read 2 Done Done New true
19392 6435 Acked 7434 Write 7 Done Done New true
19393 6436 Acked 7435 Flush 0 Done Done New true
19394 6437 Acked 7436 Write 4 Done Done New true
19395 6438 Acked 7437 Write 10 Done Done New true
19396 6439 Acked 7438 Read 3 Done Done New true
19397 6440 Acked 7439 Read 2 Done Done New true
19398 6441 Acked 7440 Read 4 Done Done New true
19399 6442 Acked 7441 Read 6 Done Done New true
19400 6443 Acked 7442 Read 3 Done Done New true
19401 6444 Acked 7443 Write 10 Done Done New true
19402 6445 Acked 7444 Write 10 Done Done New true
19403 6446 Acked 7445 Write 5 Done Done New true
19404 6447 Acked 7446 Flush 0 Done Done New true
19405 6448 Acked 7447 Flush 0 Done Done New true
19406 6449 Acked 7448 Flush 0 Done Done New true
19407 6450 Acked 7449 Read 3 Done Done New true
19408 6451 Acked 7450 Write 10 Done Done New true
19409 6452 Acked 7451 Read 3 Done Done New true
19410 6453 Acked 7452 Write 9 Done Done New true
19411 6454 Acked 7453 Read 10 Done Done New true
19412 6455 Acked 7454 Flush 0 Done Done New true
19413 6456 Acked 7455 Write 5 Done Done New true
19414 6457 Acked 7456 Read 10 Done Done New true
19415 6458 Acked 7457 Read 3 Done Done New true
19416 6459 Acked 7458 Read 7 Done Done New true
19417 6460 Acked 7459 Read 3 Done Done New true
19418 6461 Acked 7460 Write 10 Done Done New true
19419 6462 Acked 7461 Write 3 Done Done New true
19420 6463 Acked 7462 Write 2 Done Done New true
19421 6464 Acked 7463 Write 3 Done Done New true
19422 6465 Acked 7464 Write 6 Done Done New true
19423 6466 Acked 7465 Read 8 Done Done New true
19424 6467 Acked 7466 Write 2 Done Done New true
19425 6468 Acked 7467 Read 3 Done Done New true
19426 6469 Acked 7468 Read 9 Done Done New true
19427 6470 Acked 7469 Write 9 Done Done New true
19428 6471 Acked 7470 Write 6 Done Done New true
19429 6472 Acked 7471 Write 9 Done Done New true
19430 6473 Acked 7472 Read 10 Done Done New true
19431 6474 Acked 7473 Write 6 Done Done New true
19432 6475 Acked 7474 Write 9 Done Done New true
19433 6476 Acked 7475 Read 7 Done Done New true
19434 6477 Acked 7476 Write 9 Done Done New true
19435 6478 Acked 7477 Write 2 Done Done New true
19436 6479 Acked 7478 Read 3 Done Done New true
19437 6480 Acked 7479 Read 5 Done Done New true
19438 6481 Acked 7480 Write 1 Done Done New true
19439 6482 Acked 7481 Read 6 Done Done New true
19440 6483 Acked 7482 Read 8 Done Done New true
19441 6484 Acked 7483 Flush 0 Done Done New true
19442 6485 Acked 7484 Read 2 Done Done New true
19443 6486 Acked 7485 Write 4 Done Done New true
19444 6487 Acked 7486 Read 9 Done Done New true
19445 6488 Acked 7487 Write 6 Done Done New true
19446 6489 Acked 7488 Write 4 Done Done New true
19447 6490 Acked 7489 Write 1 Done Done New true
19448 6491 Acked 7490 Write 4 Done Done New true
19449 6492 Acked 7491 Read 2 Done Done New true
19450 6493 Acked 7492 Read 9 Done Done New true
19451 6494 Acked 7493 Write 1 Done Done New true
19452 6495 Acked 7494 Read 10 Done Done New true
19453 6496 Acked 7495 Read 4 Done Done New true
19454 6497 Acked 7496 Read 10 Done Done New true
19455 6498 Acked 7497 Read 6 Done Done New true
19456 6499 Acked 7498 Flush 0 Done Done New true
19457 6500 Acked 7499 Write 9 Done Done New true
19458 6501 Acked 7500 Write 4 Done Done New true
19459 6502 Acked 7501 Read 10 Done Done New true
19460 6503 Acked 7502 Write 6 Done Done New true
19461 6504 Acked 7503 Read 10 Done Done New true
19462 6505 Acked 7504 Flush 0 Done Done New true
19463 6506 Acked 7505 Write 1 Done Done New true
19464 6507 Acked 7506 Write 6 Done Done New true
19465 6508 Acked 7507 Read 4 Done Done New true
19466 6509 Acked 7508 Write 4 Done Done New true
19467 6510 Acked 7509 Read 8 Done Done New true
19468 6511 Acked 7510 Read 2 Done Done New true
19469 6512 Acked 7511 Read 9 Done Done New true
19470 6513 Acked 7512 Read 7 Done Done New true
19471 6514 Acked 7513 Read 3 Done Done New true
19472 6515 Acked 7514 Write 7 Done Done New true
19473 6516 Acked 7515 Read 2 Done Done New true
19474 6517 Acked 7516 Read 4 Done Done New true
19475 6518 Acked 7517 Write 2 Done Done New true
19476 6519 Acked 7518 Read 6 Done Done New true
19477 6520 Acked 7519 Read 6 Done Done New true
19478 6521 Acked 7520 Write 1 Done Done New true
19479 6522 Acked 7521 Flush 0 Done Done New true
19480 6523 Acked 7522 Flush 0 Done Done New true
19481 6524 Acked 7523 Write 9 Done Done New true
19482 6525 Acked 7524 Flush 0 Done Done New true
19483 6526 Acked 7525 Read 10 Done Done New true
19484 6527 Acked 7526 Write 9 Done Done New true
19485 6528 Acked 7527 Read 5 Done Done New true
19486 6529 Acked 7528 Read 8 Done Done New true
19487 6530 Acked 7529 Read 9 Done Done New true
19488 6531 Acked 7530 Read 9 Done Done New true
19489 6532 Acked 7531 Flush 0 Done Done New true
19490 6533 Acked 7532 Write 4 Done Done New true
19491 6534 Acked 7533 Read 1 Done Done New true
19492 6535 Acked 7534 Write 4 Done Done New true
19493 6536 Acked 7535 Flush 0 Done Done New true
19494 6537 Acked 7536 Read 3 Done Done New true
19495 6538 Acked 7537 Write 5 Done Done New true
19496 6539 Acked 7538 Write 7 Done Done New true
19497 6540 Acked 7539 Read 6 Done Done New true
19498 6541 Acked 7540 Read 4 Done Done New true
19499 6542 Acked 7541 Read 3 Done Done New true
19500 6543 Acked 7542 Read 7 Done Done New true
19501 6544 Acked 7543 Read 1 Done Done New true
19502 6545 Acked 7544 Write 9 Done Done New true
19503 6546 Acked 7545 Write 9 Done Done New true
19504 6547 Acked 7546 Read 6 Done Done New true
19505 6548 Acked 7547 Read 1 Done Done New true
19506 6549 Acked 7548 Read 4 Done Done New true
19507 6550 Acked 7549 Flush 0 Done Done New true
19508 6551 Acked 7550 Read 8 Done Done New true
19509 6552 Acked 7551 Write 4 Done Done New true
19510 6553 Acked 7552 Write 5 Done Done New true
19511 6554 Acked 7553 Write 1 Done Done New true
19512 6555 Acked 7554 Write 2 Done Done New true
19513 6556 Acked 7555 Write 1 Done Done New true
19514 6557 Acked 7556 Write 7 Done Done New true
19515 6558 Acked 7557 Write 6 Done Done New true
19516 6559 Acked 7558 Write 3 Done Done New true
19517 6560 Acked 7559 Read 2 Done Done New true
19518 6561 Acked 7560 Read 7 Done Done New true
19519 6562 Acked 7561 Write 10 Done Done New true
19520 6563 Acked 7562 Read 1 Done Done New true
19521 6564 Acked 7563 Write 6 Done Done New true
19522 6565 Acked 7564 Read 1 Done Done New true
19523 6566 Acked 7565 Read 3 Done Done New true
19524 6567 Acked 7566 Write 7 Done Done New true
19525 6568 Acked 7567 Read 6 Done Done New true
19526 6569 Acked 7568 Read 8 Done Done New true
19527 6570 Acked 7569 Read 10 Done Done New true
19528 6571 Acked 7570 Read 10 Done Done New true
19529 6572 Acked 7571 Read 9 Done Done New true
19530 6573 Acked 7572 Flush 0 Done Done New true
19531 6574 Acked 7573 Write 2 Done Done New true
19532 6575 Acked 7574 Write 2 Done Done New true
19533 6576 Acked 7575 Read 2 Done Done New true
19534 6577 Acked 7576 Read 10 Done Done New true
19535 6578 Acked 7577 Write 2 Done Done New true
19536 6579 Acked 7578 Flush 0 Done Done New true
19537 6580 Acked 7579 Read 9 Done Done New true
19538 6581 Acked 7580 Read 10 Done Done New true
19539 6582 Acked 7581 Write 7 Done Done New true
19540 6583 Acked 7582 Read 7 Done Done New true
19541 6584 Acked 7583 Read 8 Done Done New true
19542 6585 Acked 7584 Read 7 Done Done New true
19543 6586 Acked 7585 Flush 0 Done Done New true
19544 6587 Acked 7586 Write 5 Done Done New true
19545 6588 Acked 7587 Write 7 Done Done New true
19546 6589 Acked 7588 Read 9 Done Done New true
19547 6590 Acked 7589 Read 5 Done Done New true
19548 6591 Acked 7590 Read 2 Done Done New true
19549 6592 Acked 7591 Read 2 Done Done New true
19550 6593 Acked 7592 Write 1 Done Done New true
19551 6594 Acked 7593 Read 10 Done Done New true
19552 6595 Acked 7594 Write 7 Done Done New true
19553 6596 Acked 7595 Flush 0 Done Done New true
19554 6597 Acked 7596 Write 6 Done Done New true
19555 6598 Acked 7597 Flush 0 Done Done New true
19556 6599 Acked 7598 Read 1 Done Done New true
19557 6600 Acked 7599 Write 7 Done Done New true
19558 6601 Acked 7600 Write 10 Done Done New true
19559 6602 Acked 7601 Read 8 Done Done New true
19560 6603 Acked 7602 Flush 0 Done Done New true
19561 6604 Acked 7603 Read 9 Done Done New true
19562 6605 Acked 7604 Read 5 Done Done New true
19563 6606 Acked 7605 Flush 0 Done Done New true
19564 6607 Acked 7606 Write 4 Done Done New true
19565 6608 Acked 7607 Flush 0 Done Done New true
19566 6609 Acked 7608 Flush 0 Done Done New true
19567 6610 Acked 7609 Write 7 Done Done New true
19568 6611 Acked 7610 Read 2 Done Done New true
19569 6612 Acked 7611 Write 4 Done Done New true
19570 6613 Acked 7612 Flush 0 Done Done New true
19571 6614 Acked 7613 Flush 0 Done Done New true
19572 6615 Acked 7614 Write 9 Done Done New true
19573 6616 Acked 7615 Read 4 Done Done New true
19574 6617 Acked 7616 Write 1 Done Done New true
19575 6618 Acked 7617 Read 2 Done Done New true
19576 6619 Acked 7618 Flush 0 Done Done New true
19577 6620 Acked 7619 Write 4 Done Done New true
19578 6621 Acked 7620 Read 7 Done Done New true
19579 6622 Acked 7621 Write 9 Done Done New true
19580 6623 Acked 7622 Write 1 Done Done New true
19581 6624 Acked 7623 Write 2 Done Done New true
19582 6625 Acked 7624 Read 2 Done Done New true
19583 6626 Acked 7625 Write 5 Done Done New true
19584 6627 Acked 7626 Read 9 Done Done New true
19585 6628 Acked 7627 Flush 0 Done Done New true
19586 6629 Acked 7628 Read 5 Done Done New true
19587 6630 Acked 7629 Read 6 Done Done New true
19588 6631 Acked 7630 Read 6 Done Done New true
19589 6632 Acked 7631 Read 3 Done Done New true
19590 6633 Acked 7632 Flush 0 Done Done New true
19591 STATES DS:0 DS:1 DS:2 TOTAL
19592 New 0 0 302 302
19593 Sent 0 0 0 0
19594 Done 302 302 0 604
19595 Skipped 0 0 0 0
19596 Error 0 0 0 0
19597 Last Flush: 7632 7632 7330
19598 Downstairs last five completed: 7330 7329 7328 7327 7326
19599 Upstairs last five completed: 6633 6632 6631 6630 6629
19600 CLIENT: Up:0 ds:302 act:2
196012024-03-02T01:09:30.266ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196022024-03-02T01:09:30.266ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196032024-03-02T01:09:30.266ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196042024-03-02T01:09:30.266ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196052024-03-02T01:09:30.266ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196062024-03-02T01:09:30.266ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196072024-03-02T01:09:30.266ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19608 [8830] got start action
19609 [8830] I am going to restart
19610 Starting downstairs at port 8830
19611 Make output file at "/tmp/dsc/downstairs-8830.txt"
19612 [8830][2] reports Starting
19613 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1184)
19614 [8830][2] reports Running
196152024-03-02T01:09:31.267ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196162024-03-02T01:09:31.267ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196172024-03-02T01:09:31.267ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196182024-03-02T01:09:31.267ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196192024-03-02T01:09:31.267ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196202024-03-02T01:09:31.267ZINFOcrucible: [2] client re-new 302 jobs since flush 7330 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196212024-03-02T01:09:31.268ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196222024-03-02T01:09:32.269ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196232024-03-02T01:09:32.269ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196242024-03-02T01:09:32.271ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196252024-03-02T01:09:32.271ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196262024-03-02T01:09:32.271ZINFOcrucible: send last flush ID to this DS: 7330 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196272024-03-02T01:09:32.271ZINFOcrucible: Replied this last flush ID: 7330 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196282024-03-02T01:09:32.271ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196292024-03-02T01:09:32.271ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196302024-03-02T01:09:32.271ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196312024-03-02T01:09:32.271ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19632 ----------------------------------------------------------------
19633 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19634 Downstairs last five completed: 7632 7631 7630 7629 7628
19635 Upstairs last five completed: 6633 6632 6631 6630 6629
19636 CLIENT: Up:0 ds:0 act:3
19637 Replay: All jobs finished, all DS active.
19638 Main task has work to do, go find it
19639 got dsc Stop(1)
19640 stop 1
19641 Mar 02 01:09:33.628 INFO request completed, latency_us[8820] Got stop action so:false kr:false
19642 : 360, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8ff3d9ec-d42b-4938-aebe-6186e15caca5, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
196432024-03-02T01:09:33.628ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196442024-03-02T01:09:33.628ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196452024-03-02T01:09:33.628ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196462024-03-02T01:09:33.629ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19647 Mar 02 01:09:33.629{"msg":"[1] client re-new 0 jobs since flush 7632","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:09:33.629037959Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
19648 INFO request completed, latency_us: 136, response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:09:33.629101205Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"1","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
19649 , uri: /state/cid/1, method: GET, req_id: 53448e43-78b9-41d9-a9f0-4d9667a1f10e, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
19650 [8820] Exited with: ExitStatus(unix_wait_status(9))
19651 [8820][1] reports Exit
196522024-03-02T01:09:34.630ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196532024-03-02T01:09:34.630ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196542024-03-02T01:09:34.630ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196552024-03-02T01:09:34.630ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196562024-03-02T01:09:34.630ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196572024-03-02T01:09:34.630ZINFOcrucible: [1] client re-new 0 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196582024-03-02T01:09:34.630ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196592024-03-02T01:09:35.632ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196602024-03-02T01:09:35.632ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196612024-03-02T01:09:35.632ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196622024-03-02T01:09:35.632ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196632024-03-02T01:09:35.632ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196642024-03-02T01:09:35.632ZINFOcrucible: [1] client re-new 0 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196652024-03-02T01:09:35.633ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196662024-03-02T01:09:36.633ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196672024-03-02T01:09:36.633ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196682024-03-02T01:09:36.634ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196692024-03-02T01:09:36.634ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196702024-03-02T01:09:36.634ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196712024-03-02T01:09:36.634ZINFOcrucible: [1] client re-new 0 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196722024-03-02T01:09:36.634ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19673 Mar 02 01:09:37.631 INFO request completed, latency_us: 318, response_code: 200, uri: /state/cid/1, method: GET, req_id: abcd537b-c8af-4029-8436-94eb4742e565, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
19674 001/300 Read block 341 len 12288
19675 002/300 Read block 4795 len 40960
196762024-03-02T01:09:37.634ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196772024-03-02T01:09:37.634ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196782024-03-02T01:09:37.634ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196792024-03-02T01:09:37.634ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196802024-03-02T01:09:37.634ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196812024-03-02T01:09:37.634ZINFOcrucible: [1] client re-new 2 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
196822024-03-02T01:09:37.634ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
19683 003/300 Flush
19684 004/300 Read block 2452 len 16384
19685 005/300 Write block 2050 len 32768 data: 5 5 5 4 4 3 4 3
19686 006/300 Write block 4165 len 24576 data: 3 3 3 3 4 4
19687 007/300 Write block 1440 len 40960 data: 5 5 6 8 8 8 9 8 8 6
19688 008/300 Read block 4460 len 8192
19689 009/300 Write block 1173 len 12288 data: 3 3 3
19690 010/300 Read block 2609 len 40960
19691 011/300 Read block 2349 len 12288
19692 012/300 Write block 2235 len 36864 data: 2 2 2 4 5 5 5 5 4
19693 013/300 Write block 2645 len 36864 data: 3 3 2 3 5 7 6 7 7
19694 014/300 Read block 775 len 12288
19695 015/300 Read block 3104 len 40960
19696 016/300 Write block 24 len 28672 data: 3 3 4 4 4 4 5
19697 017/300 Read block 3491 len 8192
19698 018/300 Write block 3443 len 32768 data: 5 6 7 7 7 7 7 7
19699 019/300 Read block 3692 len 40960
19700 020/300 Write block 2397 len 8192 data: 3 5
19701 021/300 Read block 294 len 20480
19702 022/300 Write block 2965 len 32768 data: 7 4 4 4 3 3 3 4
19703 023/300 Read block 1542 len 8192
19704 024/300 Read block 2632 len 32768
19705 025/300 Read block 1091 len 12288
19706 026/300 Read block 2479 len 32768
19707 027/300 Read block 3478 len 12288
19708 028/300 Read block 1119 len 40960
19709 029/300 Read block 4200 len 36864
19710 030/300 Write block 3387 len 12288 data: 6 6 5
19711 031/300 Flush
19712 032/300 Write block 1621 len 16384 data: 6 6 6 6
19713 033/300 Read block 1706 len 12288
19714 034/300 Read block 1579 len 20480
19715 035/300 Flush
19716 036/300 Read block 2342 len 8192
19717 037/300 Write block 483 len 4096 data: 4
19718 038/300 Read block 254 len 40960
19719 039/300 Write block 3951 len 40960 data: 6 6 5 3 3 6 6 5 6 6
19720 040/300 Flush
19721 041/300 Write block 4405 len 40960 data: 8 5 5 5 5 3 5 4 4 5
19722 042/300 Write block 1627 len 8192 data: 6 6
19723 043/300 Read block 3570 len 32768
19724 044/300 Read block 1857 len 16384
19725 045/300 Read block 701 len 16384
19726 046/300 Read block 4026 len 24576
19727 047/300 Read block 1680 len 4096
19728 048/300 Flush
19729 049/300 Read block 3670 len 32768
19730 050/300 Write block 562 len 8192 data: 3 3
19731 051/300 Read block 4789 len 28672
19732 052/300 Read block 3979 len 20480
19733 053/300 Write block 1127 len 40960 data: 2 2 2 2 2 3 3 3 2 2
19734 054/300 Read block 371 len 32768
19735 055/300 Write block 4580 len 32768 data: 5 4 5 6 6 7 6 7
19736 056/300 Flush
19737 057/300 Read block 3528 len 24576
19738 058/300 Read block 444 len 36864
19739 059/300 Flush
19740 060/300 Read block 4931 len 28672
19741 061/300 Write block 3743 len 20480 data: 3 3 3 5 5
19742 062/300 Read block 2915 len 4096
19743 063/300 Read block 1799 len 40960
19744 064/300 Read block 3917 len 32768
19745 065/300 Write block 2495 len 8192 data: 5 4
19746 066/300 Read block 2131 len 36864
19747 067/300 Flush
19748 068/300 Read block 1340 len 36864
19749 069/300 Write block 834 len 8192 data: 8 7
19750 070/300 Write block 386 len 8192 data: 3 5
19751 071/300 Flush
19752 072/300 Read block 128 len 16384
19753 073/300 Read block 3696 len 32768
19754 074/300 Read block 33 len 8192
19755 075/300 Write block 1802 len 12288 data: 5 4 4
19756 076/300 Write block 984 len 24576 data: 7 7 7 6 7 6
19757 077/300 Write block 1790 len 4096 data: 4
19758 078/300 Write block 3327 len 20480 data: 5 6 8 8 7
19759 079/300 Write block 348 len 20480 data: 6 6 6 5 5
19760 080/300 Write block 4594 len 40960 data: 7 5 6 4 4 5 6 7 8 8
19761 081/300 Read block 86 len 20480
19762 082/300 Flush
19763 083/300 Read block 2623 len 28672
19764 084/300 Read block 4159 len 8192
19765 085/300 Write block 858 len 24576 data: 6 7 6 5 4 5
19766 086/300 Write block 4888 len 32768 data: 5 3 2 2 2 2 2 2
19767 087/300 Flush
19768 088/300 Read block 1337 len 40960
19769 089/300 Write block 424 len 8192 data: 5 4
19770 090/300 Read block 2440 len 4096
19771 091/300 Flush
19772 092/300 Write block 3076 len 40960 data: 3 3 3 3 2 2 2 2 2 2
19773 093/300 Read block 4983 len 20480
19774 094/300 Read block 4797 len 16384
19775 095/300 Write block 641 len 32768 data: 9 9 7 7 8 8 7 6
19776 096/300 Read block 4968 len 20480
19777 097/300 Flush
19778 098/300 Read block 68 len 36864
19779 099/300 Read block 1208 len 20480
19780 100/300 Read block 4591 len 36864
19781 101/300 Read block 4588 len 36864
19782 102/300 Flush
19783 103/300 Flush
19784 104/300 Read block 4377 len 8192
19785 105/300 Read block 1876 len 32768
19786 106/300 Read block 1292 len 32768
19787 107/300 Write block 3702 len 4096 data: 8
19788 108/300 Write block 4693 len 36864 data: 4 4 4 4 4 3 2 2 3
19789 109/300 Read block 4454 len 20480
19790 110/300 Write block 305 len 28672 data: 5 4 4 4 4 4 4
19791 111/300 Write block 4947 len 8192 data: 6 5
19792 112/300 Read block 2687 len 8192
19793 113/300 Write block 132 len 16384 data: 4 3 3 3
19794 114/300 Flush
19795 115/300 Write block 4913 len 40960 data: 7 7 7 6 4 5 3 3 3 4
19796 116/300 Write block 1970 len 12288 data: 3 4 4
19797 117/300 Read block 2572 len 32768
19798 118/300 Write block 3445 len 16384 data: 8 8 8 8
19799 119/300 Read block 1965 len 40960
19800 120/300 Write block 680 len 40960 data: 4 5 5 6 6 6 6 4 4 5
19801 121/300 Read block 2284 len 32768
19802 122/300 Write block 3432 len 24576 data: 5 5 5 3 3 4
19803 123/300 Read block 4955 len 20480
19804 124/300 Read block 3592 len 28672
19805 125/300 Read block 2151 len 4096
19806 126/300 Write block 167 len 12288 data: 5 6 5
19807 127/300 Write block 1516 len 20480 data: 5 5 5 5 3
19808 128/300 Read block 2885 len 24576
19809 129/300 Read block 2622 len 4096
19810 130/300 Read block 3724 len 32768
19811 131/300 Write block 728 len 28672 data: 5 7 7 5 6 8 7
19812 132/300 Flush
19813 133/300 Read block 490 len 28672
19814 134/300 Write block 1016 len 16384 data: 5 5 7 7
19815 135/300 Write block 4541 len 28672 data: 6 7 6 4 4 4 3
19816 136/300 Write block 1824 len 24576 data: 4 4 3 2 3 4
19817 137/300 Read block 4577 len 16384
19818 138/300 Read block 756 len 4096
19819 139/300 Write block 634 len 4096 data: 2
19820 140/300 Read block 3925 len 4096
19821 141/300 Read block 948 len 24576
19822 142/300 Write block 3551 len 40960 data: 6 7 6 6 7 7 7 6 4 4
19823 143/300 Write block 2376 len 40960 data: 3 4 4 4 3 3 4 3 4 4
19824 144/300 Flush
19825 145/300 Write block 4987 len 4096 data: 7
19826 146/300 Read block 1204 len 32768
19827 147/300 Read block 4048 len 36864
19828 148/300 Read block 1310 len 24576
19829 149/300 Write block 619 len 16384 data: 6 7 5 6
19830 150/300 Write block 3662 len 28672 data: 3 4 4 4 3 2 2
19831 151/300 Read block 3092 len 24576
19832 152/300 Read block 3438 len 20480
19833 153/300 Write block 1905 len 28672 data: 4 5 4 5 5 5 4
19834 154/300 Read block 3822 len 12288
19835 155/300 Read block 2256 len 16384
19836 156/300 Flush
19837 157/300 Read block 192 len 32768
19838 158/300 Write block 2886 len 32768 data: 6 5 5 4 4 3 3 4
19839 159/300 Write block 4602 len 20480 data: 9 9 6 4 3
19840 160/300 Read block 1390 len 28672
19841 161/300 Write block 2188 len 4096 data: 2
19842 162/300 Read block 2023 len 16384
19843 163/300 Write block 4961 len 36864 data: 3 3 3 4 5 5 4 4 7
19844 164/300 Write block 4885 len 12288 data: 3 3 3
19845 165/300 Read block 4922 len 12288
19846 166/300 Flush
19847 167/300 Read block 3966 len 12288
19848 168/300 Write block 2402 len 8192 data: 6 6
19849 169/300 Read block 1510 len 12288
19850 170/300 Write block 2732 len 4096 data: 4
19851 171/300 Write block 3705 len 32768 data: 7 7 9 8 8 7 7 9
19852 172/300 Write block 4145 len 4096 data: 4
19853 173/300 Read block 308 len 4096
19854 174/300 Read block 466 len 12288
19855 175/300 Read block 2495 len 16384
19856 176/300 Write block 2123 len 28672 data: 4 4 3 3 3 3 4
19857 177/300 Read block 1777 len 24576
19858 178/300 Read block 2121 len 16384
19859 179/300 Read block 2331 len 4096
19860 180/300 Read block 3320 len 20480
19861 181/300 Write block 1472 len 28672 data: 5 4 4 4 4 5 5
19862 182/300 Write block 2478 len 4096 data: 6
19863 183/300 Flush
19864 184/300 Flush
19865 185/300 Flush
19866 186/300 Read block 740 len 16384
19867 187/300 Read block 3596 len 32768
19868 188/300 Flush
19869 189/300 Read block 741 len 16384
19870 190/300 Flush
19871 191/300 Read block 735 len 20480
19872 192/300 Flush
19873 193/300 Read block 2876 len 12288
19874 194/300 Read block 4674 len 40960
19875 195/300 Read block 2051 len 16384
19876 196/300 Read block 2409 len 40960
19877 197/300 Read block 4864 len 24576
19878 198/300 Write block 554 len 16384 data: 4 3 4 4
19879 199/300 Write block 4810 len 28672 data: 4 5 6 5 4 4 4
19880 200/300 Write block 4541 len 4096 data: 7
19881 201/300 Read block 4853 len 28672
19882 202/300 Read block 16 len 8192
19883 203/300 Write block 4054 len 40960 data: 6 6 7 7 6 6 6 6 4 4
19884 204/300 Write block 565 len 16384 data: 4 4 4 2
19885 205/300 Read block 270 len 40960
19886 206/300 Read block 4448 len 8192
19887 207/300 Write block 2443 len 24576 data: 7 7 6 6 5 6
19888 208/300 Write block 366 len 32768 data: 3 2 4 5 5 5 6 6
19889 209/300 Flush
19890 210/300 Write block 2896 len 24576 data: 3 4 5 6 5 5
19891 211/300 Write block 4213 len 16384 data: 4 4 4 6
19892 212/300 Write block 832 len 24576 data: 10 8 9 8 7 6
19893 213/300 Read block 3427 len 28672
19894 214/300 Write block 1633 len 36864 data: 4 4 4 5 4 4 4 4 4
19895 215/300 Read block 389 len 36864
19896 216/300 Write block 1108 len 32768 data: 4 4 2 2 4 6 6 6
19897 217/300 Write block 4488 len 40960 data: 4 3 5 5 4 4 4 4 5 4
19898 218/300 Read block 2493 len 28672
19899 219/300 Read block 4972 len 32768
19900 220/300 Write block 2477 len 16384 data: 6 7 4 5
19901 221/300 Read block 1723 len 4096
19902 222/300 Read block 4837 len 8192
19903 223/300 Flush
19904 224/300 Write block 3650 len 4096 data: 7
19905 225/300 Read block 3832 len 28672
19906 226/300 Read block 4709 len 16384
19907 227/300 Read block 3292 len 16384
19908 228/300 Write block 682 len 4096 data: 6
19909 229/300 Write block 382 len 40960 data: 4 3 3 3 4 6 7 7 8 7
19910 230/300 Read block 4824 len 24576
19911 231/300 Write block 218 len 24576 data: 4 4 4 4 6 5
19912 232/300 Read block 2485 len 4096
19913 233/300 Flush
19914 234/300 Read block 605 len 16384
19915 235/300 Flush
19916 236/300 Read block 3742 len 36864
19917 237/300 Read block 1545 len 20480
19918 238/300 Write block 1619 len 4096 data: 7
19919 239/300 Read block 3551 len 12288
19920 240/300 Write block 1028 len 32768 data: 5 5 5 6 6 6 6 5
19921 241/300 Write block 2825 len 40960 data: 7 7 7 7 6 6 7 5 4 5
19922 242/300 Read block 4627 len 20480
19923 243/300 Write block 772 len 32768 data: 7 6 5 4 4 4 4 5
19924 244/300 Read block 2060 len 16384
19925 245/300 Read block 2204 len 12288
19926 246/300 Write block 2608 len 20480 data: 4 3 3 3 3
19927 247/300 Read block 4376 len 12288
19928 248/300 Write block 4540 len 4096 data: 5
19929 249/300 Write block 1433 len 40960 data: 6 5 6 3 6 5 5 6 6 7
19930 250/300 Read block 534 len 40960
19931 251/300 Write block 328 len 12288 data: 6 6 6
19932 252/300 Read block 3963 len 8192
19933 253/300 Write block 1176 len 32768 data: 3 4 5 5 7 6 8 8
19934 254/300 Read block 4505 len 24576
19935 255/300 Read block 3258 len 20480
19936 256/300 Read block 1054 len 12288
19937 257/300 Read block 698 len 4096
19938 258/300 Read block 907 len 28672
19939 259/300 Read block 1680 len 16384
19940 260/300 Write block 3176 len 24576 data: 4 5 5 5 5 6
19941 261/300 Flush
19942 262/300 Write block 916 len 40960 data: 5 4 5 5 5 5 6 8 8 6
19943 263/300 Write block 4324 len 24576 data: 4 4 5 5 5 4
19944 264/300 Read block 2341 len 40960
19945 265/300 Write block 75 len 8192 data: 5 4
19946 266/300 Read block 2110 len 36864
19947 267/300 Write block 574 len 40960 data: 4 3 4 5 4 4 2 2 3 3
19948 268/300 Flush
19949 269/300 Write block 211 len 20480 data: 4 3 3 5 5
19950 270/300 Read block 4301 len 8192
19951 271/300 Write block 1121 len 32768 data: 5 3 2 2 2 2 3 3
19952 272/300 Read block 2869 len 28672
19953 273/300 Write block 3634 len 4096 data: 3
19954 274/300 Read block 1484 len 8192
19955 275/300 Read block 2775 len 12288
19956 276/300 Write block 916 len 12288 data: 6 5 6
19957 277/300 Write block 4699 len 24576 data: 3 3 4 4 4 3
19958 278/300 Flush
19959 279/300 Write block 1355 len 40960 data: 4 7 6 7 7 6 7 7 7 6
19960 280/300 Write block 15 len 40960 data: 3 3 4 3 4 4 5 5 5 4
19961 281/300 Read block 1662 len 4096
19962 282/300 Write block 4526 len 32768 data: 4 4 4 2 2 3 3 3
19963 283/300 Write block 2314 len 8192 data: 7 6
19964 284/300 Read block 3591 len 4096
19965 285/300 Read block 3012 len 32768
19966 286/300 Write block 3765 len 12288 data: 5 5 5
19967 287/300 Read block 1084 len 20480
19968 288/300 Read block 2356 len 8192
19969 289/300 Write block 3279 len 8192 data: 6 5
19970 290/300 Flush
19971 291/300 Read block 2533 len 28672
19972 292/300 Write block 975 len 32768 data: 4 3 3 3 3 4 5 7
19973 293/300 Write block 2072 len 16384 data: 6 6 3 2
19974 294/300 Read block 2819 len 36864
19975 295/300 Read block 1758 len 16384
19976 296/300 Read block 478 len 4096
19977 297/300 Read block 2810 len 32768
19978 298/300 Write block 2425 len 32768 data: 4 3 2 3 5 6 8 7
19979 299/300 Write block 386 len 36864 data: 5 7 8 8 9 8 5 4 4
19980 300/300 Flush
19981 Main task has work to do, go find it
19982 got dsc Start(1)
19983 start 1
19984 Mar 02 01:09:37.973 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/1, method: POST, req_id: 0aece18b-0816-4a30-aa16-963937a58997, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
19985 Replay: started 1, returned:Ok(())
19986 ----------------------------------------------------------------
19987 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19988 States: Active Offline Active
19989 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19990 6634 Acked 7633 Read 3 Done New Done true
19991 6635 Acked 7634 Read 10 Done New Done true
19992 6636 Acked 7635 Flush 0 Done New Done false
19993 6637 Acked 7636 Read 4 Done New Done false
19994 6638 Acked 7637 Write 8 Done New Done false
19995 6639 Acked 7638 Write 6 Done New Done false
19996 6640 Acked 7639 Write 10 Done New Done false
19997 6641 Acked 7640 Read 2 Done New Done false
19998 6642 Acked 7641 Write 3 Done New Done false
19999 6643 Acked 7642 Read 10 Done New Done false
20000 6644 Acked 7643 Read 3 Done New Done false
20001 6645 Acked 7644 Write 9 Done New Done false
20002 6646 Acked 7645 Write 9 Done New Done false
20003 6647 Acked 7646 Read 3 Done New Done false
20004 6648 Acked 7647 Read 10 Done New Done false
20005 6649 Acked 7648 Write 7 Done New Done false
20006 6650 Acked 7649 Read 2 Done New Done false
20007 6651 Acked 7650 Write 8 Done New Done false
20008 6652 Acked 7651 Read 10 Done New Done false
20009 6653 Acked 7652 Write 2 Done New Done false
20010 6654 Acked 7653 Read 5 Done New Done false
20011 6655 Acked 7654 Write 8 Done New Done false
20012 6656 Acked 7655 Read 2 Done New Done false
20013 6657 Acked 7656 Read 8 Done New Done false
20014 6658 Acked 7657 Read 3 Done New Done false
20015 6659 Acked 7658 Read 8 Done New Done false
20016 6660 Acked 7659 Read 3 Done New Done false
20017 6661 Acked 7660 Read 10 Done New Done false
20018 6662 Acked 7661 Read 9 Done New Done false
20019 6663 Acked 7662 Write 3 Done New Done false
20020 6664 Acked 7663 Flush 0 Done New Done false
20021 6665 Acked 7664 Write 4 Done New Done false
20022 6666 Acked 7665 Read 3 Done New Done false
20023 6667 Acked 7666 Read 5 Done New Done false
20024 6668 Acked 7667 Flush 0 Done New Done false
20025 6669 Acked 7668 Read 2 Done New Done false
20026 6670 Acked 7669 Write 1 Done New Done false
20027 6671 Acked 7670 Read 10 Done New Done false
20028 6672 Acked 7671 Write 10 Done New Done false
20029 6673 Acked 7672 Flush 0 Done New Done false
20030 6674 Acked 7673 Write 10 Done New Done false
20031 6675 Acked 7674 Write 2 Done New Done false
20032 6676 Acked 7675 Read 8 Done New Done false
20033 6677 Acked 7676 Read 4 Done New Done false
20034 6678 Acked 7677 Read 4 Done New Done false
20035 6679 Acked 7678 Read 6 Done New Done false
20036 6680 Acked 7679 Read 1 Done New Done false
20037 6681 Acked 7680 Flush 0 Done New Done false
20038 6682 Acked 7681 Read 8 Done New Done false
20039 6683 Acked 7682 Write 2 Done New Done false
20040 6684 Acked 7683 Read 7 Done New Done false
20041 6685 Acked 7684 Read 5 Done New Done false
20042 6686 Acked 7685 Write 10 Done New Done false
20043 6687 Acked 7686 Read 8 Done New Done false
20044 6688 Acked 7687 Write 8 Done New Done false
20045 6689 Acked 7688 Flush 0 Done New Done false
20046 6690 Acked 7689 Read 6 Done New Done false
20047 6691 Acked 7690 Read 9 Done New Done false
20048 6692 Acked 7691 Flush 0 Done New Done false
20049 6693 Acked 7692 Read 7 Done New Done false
20050 6694 Acked 7693 Write 5 Done New Done false
20051 6695 Acked 7694 Read 1 Done New Done false
20052 6696 Acked 7695 Read 10 Done New Done false
20053 6697 Acked 7696 Read 8 Done New Done false
20054 6698 Acked 7697 Write 2 Done New Done false
20055 6699 Acked 7698 Read 9 Done New Done false
20056 6700 Acked 7699 Flush 0 Done New Done false
20057 6701 Acked 7700 Read 9 Done New Done false
20058 6702 Acked 7701 Write 2 Done New Done false
20059 6703 Acked 7702 Write 2 Done New Done false
20060 6704 Acked 7703 Flush 0 Done New Done false
20061 6705 Acked 7704 Read 4 Done New Done false
20062 6706 Acked 7705 Read 8 Done New Done false
20063 6707 Acked 7706 Read 2 Done New Done false
20064 6708 Acked 7707 Write 3 Done New Done false
20065 6709 Acked 7708 Write 6 Done New Done false
20066 6710 Acked 7709 Write 1 Done New Done false
20067 6711 Acked 7710 Write 5 Done New Done false
20068 6712 Acked 7711 Write 5 Done New Done false
20069 6713 Acked 7712 Write 10 Done New Done false
20070 6714 Acked 7713 Read 5 Done New Done false
20071 6715 Acked 7714 Flush 0 Done New Done false
20072 6716 Acked 7715 Read 7 Done New Done false
20073 6717 Acked 7716 Read 2 Done New Done false
20074 6718 Acked 7717 Write 6 Done New Done false
20075 6719 Acked 7718 Write 8 Done New Done false
20076 6720 Acked 7719 Flush 0 Done New Done false
20077 6721 Acked 7720 Read 10 Done New Done false
20078 6722 Acked 7721 Write 2 Done New Done false
20079 6723 Acked 7722 Read 1 Done New Done false
20080 6724 Acked 7723 Flush 0 Done New Done false
20081 6725 Acked 7724 Write 10 Done New Done false
20082 6726 Acked 7725 Read 5 Done New Done false
20083 6727 Acked 7726 Read 4 Done New Done false
20084 6728 Acked 7727 Write 8 Done New Done false
20085 6729 Acked 7728 Read 5 Done New Done false
20086 6730 Acked 7729 Flush 0 Done New Done false
20087 6731 Acked 7730 Read 9 Done New Done false
20088 6732 Acked 7731 Read 5 Done New Done false
20089 6733 Acked 7732 Read 9 Done New Done false
20090 6734 Acked 7733 Read 9 Done New Done false
20091 6735 Acked 7734 Flush 0 Done New Done false
20092 6736 Acked 7735 Flush 0 Done New Done false
20093 6737 Acked 7736 Read 2 Done New Done false
20094 6738 Acked 7737 Read 8 Done New Done false
20095 6739 Acked 7738 Read 8 Done New Done false
20096 6740 Acked 7739 Write 1 Done New Done false
20097 6741 Acked 7740 Write 9 Done New Done false
20098 6742 Acked 7741 Read 5 Done New Done false
20099 6743 Acked 7742 Write 7 Done New Done false
20100 6744 Acked 7743 Write 2 Done New Done false
20101 6745 Acked 7744 Read 2 Done New Done false
20102 6746 Acked 7745 Write 4 Done New Done false
20103 6747 Acked 7746 Flush 0 Done New Done false
20104 6748 Acked 7747 Write 10 Done New Done false
20105 6749 Acked 7748 Write 3 Done New Done false
20106 6750 Acked 7749 Read 8 Done New Done false
20107 6751 Acked 7750 Write 4 Done New Done false
20108 6752 Acked 7751 Read 10 Done New Done false
20109 6753 Acked 7752 Write 10 Done New Done false
20110 6754 Acked 7753 Read 8 Done New Done false
20111 6755 Acked 7754 Write 6 Done New Done false
20112 6756 Acked 7755 Read 5 Done New Done false
20113 6757 Acked 7756 Read 7 Done New Done false
20114 6758 Acked 7757 Read 1 Done New Done false
20115 6759 Acked 7758 Write 3 Done New Done false
20116 6760 Acked 7759 Write 5 Done New Done false
20117 6761 Acked 7760 Read 6 Done New Done false
20118 6762 Acked 7761 Read 1 Done New Done false
20119 6763 Acked 7762 Read 8 Done New Done false
20120 6764 Acked 7763 Write 7 Done New Done false
20121 6765 Acked 7764 Flush 0 Done New Done false
20122 6766 Acked 7765 Read 7 Done New Done false
20123 6767 Acked 7766 Write 4 Done New Done false
20124 6768 Acked 7767 Write 7 Done New Done false
20125 6769 Acked 7768 Write 6 Done New Done false
20126 6770 Acked 7769 Read 4 Done New Done false
20127 6771 Acked 7770 Read 1 Done New Done false
20128 6772 Acked 7771 Write 1 Done New Done false
20129 6773 Acked 7772 Read 1 Done New Done false
20130 6774 Acked 7773 Read 6 Done New Done false
20131 6775 Acked 7774 Write 10 Done New Done false
20132 6776 Acked 7775 Write 10 Done New Done false
20133 6777 Acked 7776 Flush 0 Done New Done false
20134 6778 Acked 7777 Write 1 Done New Done false
20135 6779 Acked 7778 Read 8 Done New Done false
20136 6780 Acked 7779 Read 9 Done New Done false
20137 6781 Acked 7780 Read 6 Done New Done false
20138 6782 Acked 7781 Write 4 Done New Done false
20139 6783 Acked 7782 Write 7 Done New Done false
20140 6784 Acked 7783 Read 6 Done New Done false
20141 6785 Acked 7784 Read 5 Done New Done false
20142 6786 Acked 7785 Write 7 Done New Done false
20143 6787 Acked 7786 Read 3 Done New Done false
20144 6788 Acked 7787 Read 4 Done New Done false
20145 6789 Acked 7788 Flush 0 Done New Done false
20146 6790 Acked 7789 Read 8 Done New Done false
20147 6791 Acked 7790 Write 8 Done New Done false
20148 6792 Acked 7791 Write 5 Done New Done false
20149 6793 Acked 7792 Read 7 Done New Done false
20150 6794 Acked 7793 Write 1 Done New Done false
20151 6795 Acked 7794 Read 4 Done New Done false
20152 6796 Acked 7795 Write 9 Done New Done false
20153 6797 Acked 7796 Write 3 Done New Done false
20154 6798 Acked 7797 Read 3 Done New Done false
20155 6799 Acked 7798 Flush 0 Done New Done false
20156 6800 Acked 7799 Read 3 Done New Done false
20157 6801 Acked 7800 Write 2 Done New Done false
20158 6802 Acked 7801 Read 3 Done New Done false
20159 6803 Acked 7802 Write 1 Done New Done false
20160 6804 Acked 7803 Write 8 Done New Done false
20161 6805 Acked 7804 Write 1 Done New Done false
20162 6806 Acked 7805 Read 1 Done New Done false
20163 6807 Acked 7806 Read 3 Done New Done false
20164 6808 Acked 7807 Read 4 Done New Done false
20165 6809 Acked 7808 Write 7 Done New Done false
20166 6810 Acked 7809 Read 6 Done New Done false
20167 6811 Acked 7810 Read 4 Done New Done false
20168 6812 Acked 7811 Read 1 Done New Done false
20169 6813 Acked 7812 Read 5 Done New Done false
20170 6814 Acked 7813 Write 7 Done New Done false
20171 6815 Acked 7814 Write 1 Done New Done false
20172 6816 Acked 7815 Flush 0 Done New Done false
20173 6817 Acked 7816 Flush 0 Done New Done false
20174 6818 Acked 7817 Flush 0 Done New Done false
20175 6819 Acked 7818 Read 4 Done New Done false
20176 6820 Acked 7819 Read 8 Done New Done false
20177 6821 Acked 7820 Flush 0 Done New Done false
20178 6822 Acked 7821 Read 4 Done New Done false
20179 6823 Acked 7822 Flush 0 Done New Done false
20180 6824 Acked 7823 Read 5 Done New Done false
20181 6825 Acked 7824 Flush 0 Done New Done false
20182 6826 Acked 7825 Read 3 Done New Done false
20183 6827 Acked 7826 Read 10 Done New Done false
20184 6828 Acked 7827 Read 4 Done New Done false
20185 6829 Acked 7828 Read 10 Done New Done false
20186 6830 Acked 7829 Read 6 Done New Done false
20187 6831 Acked 7830 Write 4 Done New Done false
20188 6832 Acked 7831 Write 7 Done New Done false
20189 6833 Acked 7832 Write 1 Done New Done false
20190 6834 Acked 7833 Read 7 Done New Done false
20191 6835 Acked 7834 Read 2 Done New Done false
20192 6836 Acked 7835 Write 10 Done New Done false
20193 6837 Acked 7836 Write 4 Done New Done false
20194 6838 Acked 7837 Read 10 Done New Done false
20195 6839 Acked 7838 Read 2 Done New Done false
20196 6840 Acked 7839 Write 6 Done New Done false
20197 6841 Acked 7840 Write 8 Done New Done false
20198 6842 Acked 7841 Flush 0 Done New Done false
20199 6843 Acked 7842 Write 6 Done New Done false
20200 6844 Acked 7843 Write 4 Done New Done false
20201 6845 Acked 7844 Write 6 Done New Done false
20202 6846 Acked 7845 Read 7 Done New Done false
20203 6847 Acked 7846 Write 9 Done New Done false
20204 6848 Acked 7847 Read 9 Done New Done false
20205 6849 Acked 7848 Write 8 Done New Done false
20206 6850 Acked 7849 Write 10 Done New Done false
20207 6851 Acked 7850 Read 7 Done New Done false
20208 6852 Acked 7851 Read 8 Done New Done false
20209 6853 Acked 7852 Write 4 Done New Done false
20210 6854 Acked 7853 Read 1 Done New Done false
20211 6855 Acked 7854 Read 2 Done New Done false
20212 6856 Acked 7855 Flush 0 Done New Done false
20213 6857 Acked 7856 Write 1 Done New Done false
20214 6858 Acked 7857 Read 7 Done New Done false
20215 6859 Acked 7858 Read 4 Done New Done false
20216 6860 Acked 7859 Read 4 Done New Done false
20217 6861 Acked 7860 Write 1 Done New Done false
20218 6862 Acked 7861 Write 10 Done New Done false
20219 6863 Acked 7862 Read 6 Done New Done false
20220 6864 Acked 7863 Write 6 Done New Done false
20221 6865 Acked 7864 Read 1 Done New Done false
20222 6866 Acked 7865 Flush 0 Done New Done false
20223 6867 Acked 7866 Read 4 Done New Done false
20224 6868 Acked 7867 Flush 0 Done New Done false
20225 6869 Acked 7868 Read 9 Done New Done false
20226 6870 Acked 7869 Read 5 Done New Done false
20227 6871 Acked 7870 Write 1 Done New Done false
20228 6872 Acked 7871 Read 3 Done New Done false
20229 6873 Acked 7872 Write 8 Done New Done false
20230 6874 Acked 7873 Write 10 Done New Done false
20231 6875 Acked 7874 Read 5 Done New Done false
20232 6876 Acked 7875 Write 8 Done New Done false
20233 6877 Acked 7876 Read 4 Done New Done false
20234 6878 Acked 7877 Read 3 Done New Done false
20235 6879 Acked 7878 Write 5 Done New Done false
20236 6880 Acked 7879 Read 3 Done New Done false
20237 6881 Acked 7880 Write 1 Done New Done false
20238 6882 Acked 7881 Write 10 Done New Done false
20239 6883 Acked 7882 Read 10 Done New Done false
20240 6884 Acked 7883 Write 3 Done New Done false
20241 6885 Acked 7884 Read 2 Done New Done false
20242 6886 Acked 7885 Write 8 Done New Done false
20243 6887 Acked 7886 Read 6 Done New Done false
20244 6888 Acked 7887 Read 5 Done New Done false
20245 6889 Acked 7888 Read 3 Done New Done false
20246 6890 Acked 7889 Read 1 Done New Done false
20247 6891 Acked 7890 Read 7 Done New Done false
20248 6892 Acked 7891 Read 4 Done New Done false
20249 6893 Acked 7892 Write 6 Done New Done false
20250 6894 Acked 7893 Flush 0 Done New Done false
20251 6895 Acked 7894 Write 10 Done New Done false
20252 6896 Acked 7895 Write 6 Done New Done false
20253 6897 Acked 7896 Read 10 Done New Done false
20254 6898 Acked 7897 Write 2 Done New Done false
20255 6899 Acked 7898 Read 9 Done New Done false
20256 6900 Acked 7899 Write 10 Done New Done false
20257 6901 Acked 7900 Flush 0 Done New Done false
20258 6902 Acked 7901 Write 5 Done New Done false
20259 6903 Acked 7902 Read 2 Done New Done false
20260 6904 Acked 7903 Write 8 Done New Done false
20261 6905 Acked 7904 Read 7 Done New Done false
20262 6906 Acked 7905 Write 1 Done New Done false
20263 6907 Acked 7906 Read 2 Done New Done false
20264 6908 Acked 7907 Read 3 Done New Done false
20265 6909 Acked 7908 Write 3 Done New Done false
20266 6910 Acked 7909 Write 6 Done New Done false
20267 6911 Acked 7910 Flush 0 Done New Done false
20268 6912 Acked 7911 Write 10 Done New Done false
20269 6913 Acked 7912 Write 10 Done New Done false
20270 6914 Acked 7913 Read 1 Done New Done false
20271 6915 Acked 7914 Write 8 Done New Done false
20272 6916 Acked 7915 Write 2 Done New Done false
20273 6917 Acked 7916 Read 1 Done New Done false
20274 6918 Acked 7917 Read 8 Done New Done false
20275 6919 Acked 7918 Write 3 Done New Done false
20276 6920 Acked 7919 Read 5 Done New Done false
20277 6921 Acked 7920 Read 2 Done New Done false
20278 6922 Acked 7921 Write 2 Done New Done false
20279 6923 Acked 7922 Flush 0 Done New Done false
20280 6924 Acked 7923 Read 7 Done New Done false
20281 6925 Acked 7924 Write 8 Done New Done false
20282 6926 Acked 7925 Write 4 Done New Done false
20283 6927 Acked 7926 Read 9 Done New Done false
20284 6928 Acked 7927 Read 4 Done New Done false
20285 6929 Acked 7928 Read 1 Done New Done false
20286 6930 Acked 7929 Read 8 Done New Done false
20287 6931 Acked 7930 Write 8 Done New Done false
20288 6932 Acked 7931 Write 9 Done New Done false
20289 6933 Acked 7932 Flush 0 Done New Done false
20290 STATES DS:0 DS:1 DS:2 TOTAL
20291 New 0 300 0 300
20292 Sent 0 0 0 0
20293 Done 300 0 300 600
20294 Skipped 0 0 0 0
20295 Error 0 0 0 0
20296 Last Flush: 7932 7632 7932
20297 Downstairs last five completed: 7632 7631 7630 7629 7628
20298 Upstairs last five completed: 6933 6932 6931 6930 6929
20299 CLIENT: Up:0 ds:300 act:2
203002024-03-02T01:09:38.635ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203012024-03-02T01:09:38.635ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203022024-03-02T01:09:38.635ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203032024-03-02T01:09:38.635ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203042024-03-02T01:09:38.635ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203052024-03-02T01:09:38.635ZINFOcrucible: [1] client re-new 300 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203062024-03-02T01:09:38.635ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20307 [8820] got start action
20308 [8820] I am going to restart
20309 Starting downstairs at port 8820
20310 Make output file at "/tmp/dsc/downstairs-8820.txt"
20311 [8820][1] reports Starting
203122024-03-02T01:09:39.636ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203132024-03-02T01:09:39.636ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203142024-03-02T01:09:39.636ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203152024-03-02T01:09:39.636ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203162024-03-02T01:09:39.636ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203172024-03-02T01:09:39.636ZINFOcrucible: [1] client re-new 300 jobs since flush 7632 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203182024-03-02T01:09:39.637ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20319 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1185)
20320 [8820][1] reports Running
203212024-03-02T01:09:40.638ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203222024-03-02T01:09:40.638ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203232024-03-02T01:09:40.640ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203242024-03-02T01:09:40.640ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203252024-03-02T01:09:40.640ZINFOcrucible: send last flush ID to this DS: 7632 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203262024-03-02T01:09:40.640ZINFOcrucible: Replied this last flush ID: 7632 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203272024-03-02T01:09:40.640ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203282024-03-02T01:09:40.640ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203292024-03-02T01:09:40.640ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203302024-03-02T01:09:40.640ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20331 ----------------------------------------------------------------
20332 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20333 Downstairs last five completed: 7932 7931 7930 7929 7928
20334 Upstairs last five completed: 6933 6932 6931 6930 6929
20335 CLIENT: Up:0 ds:0 act:3
20336 Replay: All jobs finished, all DS active.
20337 Main task has work to do, go find it
20338 got dsc Stop(2)
20339 stop 2
20340 Mar 02 01:09:41.976 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2d9bec9c-34dc-4b48-869c-8e2d3d8df95c, remote_addr: 127[8830] Got stop action so:false kr:false
20341 .0.0.1:40735, local_addr: 127.0.0.1:9998
203422024-03-02T01:09:41.977ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203432024-03-02T01:09:41.977ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203442024-03-02T01:09:41.977ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20345 Mar 02 01:09:41.977 INFO request completed, latency_us: 194, response_code: 200, uri:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:09:41.977513685Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
20346 /state/cid/2, method: GET, req_id{"msg":"[2] client re-new 0 jobs since flush 7932","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:09:41.977557653Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
20347 : ff97834b-a431-4805-86cb-1623c38c4116, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
203482024-03-02T01:09:41.977ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20349 [8830] Exited with: ExitStatus(unix_wait_status(9))
20350 [8830][2] reports Exit
203512024-03-02T01:09:42.979ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203522024-03-02T01:09:42.979ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203532024-03-02T01:09:42.979ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203542024-03-02T01:09:42.979ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203552024-03-02T01:09:42.979ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203562024-03-02T01:09:42.979ZINFOcrucible: [2] client re-new 0 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203572024-03-02T01:09:42.979ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203582024-03-02T01:09:43.980ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203592024-03-02T01:09:43.980ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203602024-03-02T01:09:43.980ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203612024-03-02T01:09:43.980ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203622024-03-02T01:09:43.980ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203632024-03-02T01:09:43.981ZINFOcrucible: [2] client re-new 0 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203642024-03-02T01:09:43.981ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203652024-03-02T01:09:44.982ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203662024-03-02T01:09:44.982ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203672024-03-02T01:09:44.982ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203682024-03-02T01:09:44.982ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203692024-03-02T01:09:44.982ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203702024-03-02T01:09:44.982ZINFOcrucible: [2] client re-new 0 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203712024-03-02T01:09:44.982ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20372 Mar 02 01:09:45.979 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: e9f8c886-8bce-4c90-839a-1b0cd01409d4, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
20373 001/300 Read block 2782 len 36864
20374 002/300 Read block 3730 len 40960
203752024-03-02T01:09:45.982ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203762024-03-02T01:09:45.983ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203772024-03-02T01:09:45.983ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203782024-03-02T01:09:45.983ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203792024-03-02T01:09:45.983ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203802024-03-02T01:09:45.983ZINFOcrucible: [2] client re-new 2 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
203812024-03-02T01:09:45.983ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
20382 003/300 Read block 2073 len 32768
20383 004/300 Read block 2214 len 20480
20384 005/300 Read block 2612 len 40960
20385 006/300 Read block 2493 len 16384
20386 007/300 Read block 1325 len 20480
20387 008/300 Write block 4268 len 8192 data: 6 4
20388 009/300 Read block 1106 len 8192
20389 010/300 Read block 4629 len 32768
20390 011/300 Write block 597 len 40960 data: 4 4 4 4 4 4 3 5 6 6
20391 012/300 Read block 1130 len 12288
20392 013/300 Flush
20393 014/300 Write block 912 len 4096 data: 6
20394 015/300 Read block 2150 len 32768
20395 016/300 Read block 1204 len 8192
20396 017/300 Write block 1037 len 4096 data: 3
20397 018/300 Write block 1453 len 4096 data: 6
20398 019/300 Read block 1361 len 28672
20399 020/300 Write block 3438 len 36864 data: 4 3 4 5 4 6 7 9 9
20400 021/300 Flush
20401 022/300 Read block 3732 len 36864
20402 023/300 Read block 3341 len 28672
20403 024/300 Flush
20404 025/300 Read block 4301 len 24576
20405 026/300 Read block 3371 len 28672
20406 027/300 Write block 4308 len 20480 data: 5 6 5 6 5
20407 028/300 Read block 3370 len 36864
20408 029/300 Read block 2674 len 8192
20409 030/300 Read block 2792 len 12288
20410 031/300 Read block 4555 len 16384
20411 032/300 Write block 2888 len 8192 data: 6 5
20412 033/300 Read block 290 len 24576
20413 034/300 Write block 4672 len 36864 data: 5 4 4 5 4 3 4 6 6
20414 035/300 Read block 1862 len 24576
20415 036/300 Write block 4191 len 36864 data: 7 6 6 7 7 8 7 7 5
20416 037/300 Read block 484 len 4096
20417 038/300 Write block 2998 len 8192 data: 4 3
20418 039/300 Read block 2549 len 20480
20419 040/300 Read block 3692 len 32768
20420 041/300 Write block 3585 len 12288 data: 4 4 6
20421 042/300 Read block 3731 len 28672
20422 043/300 Read block 2182 len 40960
20423 044/300 Read block 2624 len 32768
20424 045/300 Write block 1636 len 32768 data: 6 5 5 5 5 5 5 5
20425 046/300 Write block 4468 len 24576 data: 2 2 2 2 2 3
20426 047/300 Write block 952 len 16384 data: 4 5 5 5
20427 048/300 Read block 4662 len 16384
20428 049/300 Read block 4468 len 28672
20429 050/300 Read block 4512 len 40960
20430 051/300 Write block 4852 len 4096 data: 6
20431 052/300 Write block 3236 len 12288 data: 7 7 7
20432 053/300 Read block 2023 len 24576
20433 054/300 Read block 2525 len 40960
20434 055/300 Read block 2746 len 12288
20435 056/300 Write block 4906 len 16384 data: 6 5 5 5
20436 057/300 Write block 4948 len 28672 data: 6 4 5 5 5 6 6
20437 058/300 Read block 1381 len 4096
20438 059/300 Write block 3487 len 8192 data: 6 6
20439 060/300 Read block 1787 len 24576
20440 061/300 Read block 1879 len 4096
20441 062/300 Read block 4001 len 4096
20442 063/300 Write block 2220 len 40960 data: 7 6 5 5 3 3 4 3 4 3
20443 064/300 Read block 90 len 28672
20444 065/300 Read block 367 len 8192
20445 066/300 Read block 510 len 4096
20446 067/300 Write block 1974 len 28672 data: 7 7 6 7 5 5 6
20447 068/300 Read block 2553 len 32768
20448 069/300 Read block 4185 len 32768
20449 070/300 Read block 4039 len 8192
20450 071/300 Read block 1151 len 4096
20451 072/300 Read block 4055 len 36864
20452 073/300 Read block 4850 len 28672
20453 074/300 Read block 1366 len 8192
20454 075/300 Write block 2696 len 4096 data: 4
20455 076/300 Write block 3613 len 24576 data: 5 5 6 4 5 5
20456 077/300 Write block 3668 len 36864 data: 3 3 3 3 3 4 4 5 6
20457 078/300 Flush
20458 079/300 Read block 2795 len 28672
20459 080/300 Write block 178 len 20480 data: 2 2 4 4 5
20460 081/300 Read block 389 len 24576
20461 082/300 Read block 916 len 24576
20462 083/300 Read block 753 len 12288
20463 084/300 Write block 2745 len 8192 data: 5 6
20464 085/300 Flush
20465 086/300 Write block 1699 len 28672 data: 6 3 3 3 2 2 3
20466 087/300 Read block 2008 len 12288
20467 088/300 Read block 1060 len 20480
20468 089/300 Write block 1363 len 12288 data: 8 7 3
20469 090/300 Read block 4197 len 36864
20470 091/300 Write block 1212 len 12288 data: 8 8 6
20471 092/300 Read block 3878 len 24576
20472 093/300 Read block 3176 len 4096
20473 094/300 Read block 4319 len 36864
20474 095/300 Write block 4677 len 4096 data: 4
20475 096/300 Write block 1578 len 40960 data: 7 7 9 6 6 6 7 5 5 4
20476 097/300 Read block 3586 len 28672
20477 098/300 Read block 1013 len 16384
20478 099/300 Write block 3944 len 28672 data: 2 2 3 3 3 6 8
20479 100/300 Read block 88 len 32768
20480 101/300 Read block 2503 len 16384
20481 102/300 Read block 3560 len 20480
20482 103/300 Read block 3601 len 20480
20483 104/300 Read block 2281 len 24576
20484 105/300 Read block 762 len 16384
20485 106/300 Write block 707 len 8192 data: 4 3
20486 107/300 Read block 18 len 4096
20487 108/300 Write block 1573 len 8192 data: 7 7
20488 109/300 Flush
20489 110/300 Write block 4462 len 8192 data: 2 3
20490 111/300 Write block 4162 len 20480 data: 6 4 4 4 4
20491 112/300 Read block 2706 len 20480
20492 113/300 Read block 119 len 8192
20493 114/300 Write block 599 len 40960 data: 5 5 5 5 4 6 7 7 4 4
20494 115/300 Write block 165 len 8192 data: 4 4
20495 116/300 Write block 276 len 28672 data: 6 6 6 6 7 7 8
20496 117/300 Read block 1800 len 8192
20497 118/300 Read block 2540 len 24576
20498 119/300 Read block 2599 len 16384
20499 120/300 Write block 2842 len 36864 data: 8 8 8 7 6 6 8 8 5
20500 121/300 Write block 3270 len 16384 data: 4 5 5 4
20501 122/300 Write block 3841 len 24576 data: 4 4 3 3 4 3
20502 123/300 Read block 4040 len 4096
20503 124/300 Write block 2329 len 16384 data: 3 3 3 3
20504 125/300 Flush
20505 126/300 Read block 1145 len 16384
20506 127/300 Read block 3158 len 40960
20507 128/300 Write block 3276 len 8192 data: 6 5
20508 129/300 Write block 2448 len 32768 data: 7 6 5 4 3 3 4 4
20509 130/300 Flush
20510 131/300 Read block 1597 len 12288
20511 132/300 Read block 3290 len 4096
20512 133/300 Write block 1050 len 24576 data: 6 6 6 7 6 5
20513 134/300 Write block 3920 len 12288 data: 4 4 5
20514 135/300 Read block 4020 len 20480
20515 136/300 Write block 3563 len 32768 data: 4 6 9 9 8 9 8 6
20516 137/300 Flush
20517 138/300 Flush
20518 139/300 Flush
20519 140/300 Read block 2779 len 16384
20520 141/300 Read block 3172 len 24576
20521 142/300 Read block 3597 len 4096
20522 143/300 Write block 1203 len 28672 data: 7 7 5 6 5 5 7
20523 144/300 Read block 677 len 8192
20524 145/300 Read block 3198 len 12288
20525 146/300 Write block 4532 len 16384 data: 4 4 3 3
20526 147/300 Write block 3038 len 20480 data: 5 5 4 4 4
20527 148/300 Read block 4403 len 36864
20528 149/300 Write block 4361 len 12288 data: 4 4 5
20529 150/300 Read block 4347 len 16384
20530 151/300 Read block 1825 len 8192
20531 152/300 Read block 1076 len 40960
20532 153/300 Write block 1655 len 16384 data: 5 4 4 3
20533 154/300 Read block 2876 len 28672
20534 155/300 Write block 1149 len 40960 data: 3 3 4 4 4 4 5 5 4 5
20535 156/300 Read block 4861 len 20480
20536 157/300 Write block 3933 len 8192 data: 3 4
20537 158/300 Read block 2544 len 24576
20538 159/300 Read block 3419 len 28672
20539 160/300 Write block 4913 len 28672 data: 8 8 8 7 5 6 4
20540 161/300 Write block 2741 len 4096 data: 4
20541 162/300 Flush
20542 163/300 Write block 3953 len 36864 data: 6 4 4 7 7 6 7 7 7
20543 164/300 Write block 1976 len 12288 data: 7 8 6
20544 165/300 Read block 2694 len 20480
20545 166/300 Write block 510 len 24576 data: 2 2 2 2 2 3
20546 167/300 Write block 1222 len 4096 data: 3
20547 168/300 Read block 2994 len 28672
20548 169/300 Write block 2312 len 12288 data: 4 6 8
20549 170/300 Read block 4207 len 32768
20550 171/300 Write block 1029 len 36864 data: 6 6 7 7 7 7 6 4 4
20551 172/300 Write block 4863 len 28672 data: 3 3 3 3 3 3 3
20552 173/300 Read block 770 len 24576
20553 174/300 Read block 3541 len 8192
20554 175/300 Write block 573 len 32768 data: 4 5 4 5 6 5 5 3
20555 176/300 Write block 2114 len 16384 data: 5 5 6 6
20556 177/300 Read block 2671 len 28672
20557 178/300 Read block 3705 len 20480
20558 179/300 Read block 2208 len 28672
20559 180/300 Read block 1162 len 16384
20560 181/300 Write block 1380 len 20480 data: 5 4 3 5 5
20561 182/300 Read block 459 len 28672
20562 183/300 Read block 2434 len 16384
20563 184/300 Read block 1477 len 4096
20564 185/300 Write block 3035 len 28672 data: 4 5 5 6 6 5 5
20565 186/300 Flush
20566 187/300 Write block 2738 len 28672 data: 5 5 5 5 4 6 5
20567 188/300 Write block 3202 len 40960 data: 4 4 4 4 6 6 7 7 6 5
20568 189/300 Read block 2754 len 36864
20569 190/300 Write block 3190 len 4096 data: 4
20570 191/300 Write block 1494 len 8192 data: 4 4
20571 192/300 Read block 2785 len 28672
20572 193/300 Read block 30 len 40960
20573 194/300 Read block 308 len 20480
20574 195/300 Write block 450 len 40960 data: 8 6 7 6 6 3 3 3 3 3
20575 196/300 Write block 2568 len 12288 data: 6 5 4
20576 197/300 Write block 3233 len 28672 data: 6 7 8 8 8 8 7
20577 198/300 Read block 4157 len 32768
20578 199/300 Read block 4532 len 28672
20579 200/300 Read block 345 len 20480
20580 201/300 Read block 4011 len 20480
20581 202/300 Flush
20582 203/300 Read block 4652 len 32768
20583 204/300 Read block 4560 len 8192
20584 205/300 Read block 3382 len 32768
20585 206/300 Read block 2136 len 16384
20586 207/300 Read block 3496 len 16384
20587 208/300 Read block 855 len 12288
20588 209/300 Read block 2779 len 40960
20589 210/300 Read block 1426 len 16384
20590 211/300 Read block 2733 len 40960
20591 212/300 Read block 2469 len 36864
20592 213/300 Write block 2607 len 4096 data: 5
20593 214/300 Write block 839 len 16384 data: 6 6 6 5
20594 215/300 Write block 1197 len 16384 data: 6 4 4 6
20595 216/300 Write block 2477 len 4096 data: 7
20596 217/300 Flush
20597 218/300 Read block 2762 len 4096
20598 219/300 Read block 4683 len 20480
20599 220/300 Read block 4246 len 40960
20600 221/300 Read block 2184 len 8192
20601 222/300 Read block 468 len 40960
20602 223/300 Read block 831 len 8192
20603 224/300 Read block 2374 len 20480
20604 225/300 Read block 2641 len 24576
20605 226/300 Read block 1408 len 12288
20606 227/300 Read block 4228 len 20480
20607 228/300 Read block 1101 len 24576
20608 229/300 Read block 3718 len 16384
20609 230/300 Write block 3606 len 40960 data: 2 2 4 4 5 6 6 6 6 7
20610 231/300 Read block 3998 len 40960
20611 232/300 Read block 4827 len 4096
20612 233/300 Flush
20613 234/300 Write block 645 len 12288 data: 9 9 8
20614 235/300 Flush
20615 236/300 Write block 2541 len 20480 data: 5 5 5 3 2
20616 237/300 Read block 4147 len 8192
20617 238/300 Write block 3533 len 40960 data: 7 6 7 6 5 5 5 4 3 4
20618 239/300 Write block 786 len 4096 data: 5
20619 240/300 Write block 221 len 8192 data: 5 7
20620 241/300 Write block 95 len 40960 data: 3 3 4 3 2 2 3 3 4 5
20621 242/300 Read block 3392 len 36864
20622 243/300 Read block 3772 len 28672
20623 244/300 Write block 3372 len 8192 data: 7 7
20624 245/300 Read block 1179 len 20480
20625 246/300 Flush
20626 247/300 Read block 167 len 32768
20627 248/300 Read block 259 len 32768
20628 249/300 Read block 4845 len 12288
20629 250/300 Read block 4690 len 20480
20630 251/300 Write block 4531 len 20480 data: 4 5 5 4 4
20631 252/300 Write block 2380 len 28672 data: 4 4 5 4 5 5 3
20632 253/300 Read block 1152 len 8192
20633 254/300 Read block 2096 len 40960
20634 255/300 Read block 2978 len 12288
20635 256/300 Read block 2574 len 20480
20636 257/300 Read block 3470 len 32768
20637 258/300 Read block 4348 len 28672
20638 259/300 Write block 4617 len 28672 data: 3 3 4 5 6 5 5
20639 260/300 Write block 473 len 12288 data: 5 5 4
20640 261/300 Read block 615 len 4096
20641 262/300 Flush
20642 263/300 Write block 2142 len 32768 data: 3 4 4 5 5 5 4 5
20643 264/300 Read block 1295 len 36864
20644 265/300 Read block 4578 len 16384
20645 266/300 Read block 670 len 24576
20646 267/300 Read block 3876 len 16384
20647 268/300 Read block 2539 len 12288
20648 269/300 Write block 2094 len 32768 data: 7 7 6 4 4 6 6 6
20649 270/300 Read block 2324 len 24576
20650 271/300 Read block 3476 len 4096
20651 272/300 Write block 2466 len 4096 data: 2
20652 273/300 Write block 1091 len 36864 data: 5 5 5 4 7 6 7 6 6
20653 274/300 Write block 2739 len 8192 data: 6 6
20654 275/300 Write block 3726 len 8192 data: 2 2
20655 276/300 Flush
20656 277/300 Flush
20657 278/300 Read block 1772 len 20480
20658 279/300 Read block 4215 len 12288
20659 280/300 Write block 2950 len 40960 data: 7 6 6 6 6 7 6 6 5 6
20660 281/300 Read block 648 len 40960
20661 282/300 Read block 4053 len 12288
20662 283/300 Write block 3733 len 20480 data: 5 4 4 4 4
20663 284/300 Read block 3016 len 4096
20664 285/300 Write block 2780 len 20480 data: 7 6 4 4 4
20665 286/300 Write block 1650 len 8192 data: 8 7
20666 287/300 Read block 1689 len 4096
20667 288/300 Write block 693 len 16384 data: 7 6 6 7
20668 289/300 Read block 741 len 40960
20669 290/300 Read block 429 len 40960
20670 291/300 Write block 4181 len 40960 data: 4 4 5 5 7 8 7 6 6 6
20671 292/300 Read block 2421 len 28672
20672 293/300 Read block 4387 len 8192
20673 294/300 Read block 2931 len 32768
20674 295/300 Read block 1384 len 36864
20675 296/300 Read block 903 len 32768
20676 297/300 Write block 3737 len 8192 data: 5 2
20677 298/300 Write block 3275 len 20480 data: 5 7 6 6 7
20678 299/300 Read block 4261 len 8192
20679 300/300 Write block 3412 len 36864 data: 6 5 7 6 4 2 4 4 4
20680 Main task has work to do, go find it
20681 got dsc Start(2)
20682 start 2
20683 Mar 02 01:09:46.325 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: e7d0361f-506e-4d30-873f-07d45ba35db5, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
20684 Replay: started 2, returned:Ok(())
20685 ----------------------------------------------------------------
20686 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20687 States: Active Active Offline
20688 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20689 6934 Acked 7933 Read 9 Done Done New true
20690 6935 Acked 7934 Read 10 Done Done New true
20691 6936 Acked 7935 Read 8 Done Done New false
20692 6937 Acked 7936 Read 5 Done Done New false
20693 6938 Acked 7937 Read 10 Done Done New false
20694 6939 Acked 7938 Read 4 Done Done New false
20695 6940 Acked 7939 Read 5 Done Done New false
20696 6941 Acked 7940 Write 2 Done Done New false
20697 6942 Acked 7941 Read 2 Done Done New false
20698 6943 Acked 7942 Read 8 Done Done New false
20699 6944 Acked 7943 Write 10 Done Done New false
20700 6945 Acked 7944 Read 3 Done Done New false
20701 6946 Acked 7945 Flush 0 Done Done New false
20702 6947 Acked 7946 Write 1 Done Done New false
20703 6948 Acked 7947 Read 8 Done Done New false
20704 6949 Acked 7948 Read 2 Done Done New false
20705 6950 Acked 7949 Write 1 Done Done New false
20706 6951 Acked 7950 Write 1 Done Done New false
20707 6952 Acked 7951 Read 7 Done Done New false
20708 6953 Acked 7952 Write 9 Done Done New false
20709 6954 Acked 7953 Flush 0 Done Done New false
20710 6955 Acked 7954 Read 9 Done Done New false
20711 6956 Acked 7955 Read 7 Done Done New false
20712 6957 Acked 7956 Flush 0 Done Done New false
20713 6958 Acked 7957 Read 6 Done Done New false
20714 6959 Acked 7958 Read 7 Done Done New false
20715 6960 Acked 7959 Write 5 Done Done New false
20716 6961 Acked 7960 Read 9 Done Done New false
20717 6962 Acked 7961 Read 2 Done Done New false
20718 6963 Acked 7962 Read 3 Done Done New false
20719 6964 Acked 7963 Read 4 Done Done New false
20720 6965 Acked 7964 Write 2 Done Done New false
20721 6966 Acked 7965 Read 6 Done Done New false
20722 6967 Acked 7966 Write 9 Done Done New false
20723 6968 Acked 7967 Read 6 Done Done New false
20724 6969 Acked 7968 Write 9 Done Done New false
20725 6970 Acked 7969 Read 1 Done Done New false
20726 6971 Acked 7970 Write 2 Done Done New false
20727 6972 Acked 7971 Read 5 Done Done New false
20728 6973 Acked 7972 Read 8 Done Done New false
20729 6974 Acked 7973 Write 3 Done Done New false
20730 6975 Acked 7974 Read 7 Done Done New false
20731 6976 Acked 7975 Read 10 Done Done New false
20732 6977 Acked 7976 Read 8 Done Done New false
20733 6978 Acked 7977 Write 8 Done Done New false
20734 6979 Acked 7978 Write 6 Done Done New false
20735 6980 Acked 7979 Write 4 Done Done New false
20736 6981 Acked 7980 Read 4 Done Done New false
20737 6982 Acked 7981 Read 7 Done Done New false
20738 6983 Acked 7982 Read 10 Done Done New false
20739 6984 Acked 7983 Write 1 Done Done New false
20740 6985 Acked 7984 Write 3 Done Done New false
20741 6986 Acked 7985 Read 6 Done Done New false
20742 6987 Acked 7986 Read 10 Done Done New false
20743 6988 Acked 7987 Read 3 Done Done New false
20744 6989 Acked 7988 Write 4 Done Done New false
20745 6990 Acked 7989 Write 7 Done Done New false
20746 6991 Acked 7990 Read 1 Done Done New false
20747 6992 Acked 7991 Write 2 Done Done New false
20748 6993 Acked 7992 Read 6 Done Done New false
20749 6994 Acked 7993 Read 1 Done Done New false
20750 6995 Acked 7994 Read 1 Done Done New false
20751 6996 Acked 7995 Write 10 Done Done New false
20752 6997 Acked 7996 Read 7 Done Done New false
20753 6998 Acked 7997 Read 2 Done Done New false
20754 6999 Acked 7998 Read 1 Done Done New false
20755 7000 Acked 7999 Write 7 Done Done New false
20756 7001 Acked 8000 Read 8 Done Done New false
20757 7002 Acked 8001 Read 8 Done Done New false
20758 7003 Acked 8002 Read 2 Done Done New false
20759 7004 Acked 8003 Read 1 Done Done New false
20760 7005 Acked 8004 Read 9 Done Done New false
20761 7006 Acked 8005 Read 7 Done Done New false
20762 7007 Acked 8006 Read 2 Done Done New false
20763 7008 Acked 8007 Write 1 Done Done New false
20764 7009 Acked 8008 Write 6 Done Done New false
20765 7010 Acked 8009 Write 9 Done Done New false
20766 7011 Acked 8010 Flush 0 Done Done New false
20767 7012 Acked 8011 Read 7 Done Done New false
20768 7013 Acked 8012 Write 5 Done Done New false
20769 7014 Acked 8013 Read 6 Done Done New false
20770 7015 Acked 8014 Read 6 Done Done New false
20771 7016 Acked 8015 Read 3 Done Done New false
20772 7017 Acked 8016 Write 2 Done Done New false
20773 7018 Acked 8017 Flush 0 Done Done New false
20774 7019 Acked 8018 Write 7 Done Done New false
20775 7020 Acked 8019 Read 3 Done Done New false
20776 7021 Acked 8020 Read 5 Done Done New false
20777 7022 Acked 8021 Write 3 Done Done New false
20778 7023 Acked 8022 Read 9 Done Done New false
20779 7024 Acked 8023 Write 3 Done Done New false
20780 7025 Acked 8024 Read 6 Done Done New false
20781 7026 Acked 8025 Read 1 Done Done New false
20782 7027 Acked 8026 Read 9 Done Done New false
20783 7028 Acked 8027 Write 1 Done Done New false
20784 7029 Acked 8028 Write 10 Done Done New false
20785 7030 Acked 8029 Read 7 Done Done New false
20786 7031 Acked 8030 Read 4 Done Done New false
20787 7032 Acked 8031 Write 7 Done Done New false
20788 7033 Acked 8032 Read 8 Done Done New false
20789 7034 Acked 8033 Read 4 Done Done New false
20790 7035 Acked 8034 Read 5 Done Done New false
20791 7036 Acked 8035 Read 5 Done Done New false
20792 7037 Acked 8036 Read 6 Done Done New false
20793 7038 Acked 8037 Read 4 Done Done New false
20794 7039 Acked 8038 Write 2 Done Done New false
20795 7040 Acked 8039 Read 1 Done Done New false
20796 7041 Acked 8040 Flush 0 Done Done New false
20797 7042 Acked 8041 Write 2 Done Done New false
20798 7043 Acked 8042 Flush 0 Done Done New false
20799 7044 Acked 8043 Write 2 Done Done New false
20800 7045 Acked 8044 Write 5 Done Done New false
20801 7046 Acked 8045 Read 5 Done Done New false
20802 7047 Acked 8046 Read 2 Done Done New false
20803 7048 Acked 8047 Write 10 Done Done New false
20804 7049 Acked 8048 Write 2 Done Done New false
20805 7050 Acked 8049 Write 7 Done Done New false
20806 7051 Acked 8050 Read 2 Done Done New false
20807 7052 Acked 8051 Read 6 Done Done New false
20808 7053 Acked 8052 Read 4 Done Done New false
20809 7054 Acked 8053 Write 9 Done Done New false
20810 7055 Acked 8054 Write 4 Done Done New false
20811 7056 Acked 8055 Write 6 Done Done New false
20812 7057 Acked 8056 Read 1 Done Done New false
20813 7058 Acked 8057 Write 4 Done Done New false
20814 7059 Acked 8058 Flush 0 Done Done New false
20815 7060 Acked 8059 Read 4 Done Done New false
20816 7061 Acked 8060 Read 10 Done Done New false
20817 7062 Acked 8061 Write 2 Done Done New false
20818 7063 Acked 8062 Write 8 Done Done New false
20819 7064 Acked 8063 Flush 0 Done Done New false
20820 7065 Acked 8064 Read 3 Done Done New false
20821 7066 Acked 8065 Read 1 Done Done New false
20822 7067 Acked 8066 Write 6 Done Done New false
20823 7068 Acked 8067 Write 3 Done Done New false
20824 7069 Acked 8068 Read 5 Done Done New false
20825 7070 Acked 8069 Write 8 Done Done New false
20826 7071 Acked 8070 Flush 0 Done Done New false
20827 7072 Acked 8071 Flush 0 Done Done New false
20828 7073 Acked 8072 Flush 0 Done Done New false
20829 7074 Acked 8073 Read 4 Done Done New false
20830 7075 Acked 8074 Read 6 Done Done New false
20831 7076 Acked 8075 Read 1 Done Done New false
20832 7077 Acked 8076 Write 7 Done Done New false
20833 7078 Acked 8077 Read 2 Done Done New false
20834 7079 Acked 8078 Read 3 Done Done New false
20835 7080 Acked 8079 Write 4 Done Done New false
20836 7081 Acked 8080 Write 5 Done Done New false
20837 7082 Acked 8081 Read 9 Done Done New false
20838 7083 Acked 8082 Write 3 Done Done New false
20839 7084 Acked 8083 Read 4 Done Done New false
20840 7085 Acked 8084 Read 2 Done Done New false
20841 7086 Acked 8085 Read 10 Done Done New false
20842 7087 Acked 8086 Write 4 Done Done New false
20843 7088 Acked 8087 Read 7 Done Done New false
20844 7089 Acked 8088 Write 10 Done Done New false
20845 7090 Acked 8089 Read 5 Done Done New false
20846 7091 Acked 8090 Write 2 Done Done New false
20847 7092 Acked 8091 Read 6 Done Done New false
20848 7093 Acked 8092 Read 7 Done Done New false
20849 7094 Acked 8093 Write 7 Done Done New false
20850 7095 Acked 8094 Write 1 Done Done New false
20851 7096 Acked 8095 Flush 0 Done Done New false
20852 7097 Acked 8096 Write 9 Done Done New false
20853 7098 Acked 8097 Write 3 Done Done New false
20854 7099 Acked 8098 Read 5 Done Done New false
20855 7100 Acked 8099 Write 6 Done Done New false
20856 7101 Acked 8100 Write 1 Done Done New false
20857 7102 Acked 8101 Read 7 Done Done New false
20858 7103 Acked 8102 Write 3 Done Done New false
20859 7104 Acked 8103 Read 8 Done Done New false
20860 7105 Acked 8104 Write 9 Done Done New false
20861 7106 Acked 8105 Write 7 Done Done New false
20862 7107 Acked 8106 Read 6 Done Done New false
20863 7108 Acked 8107 Read 2 Done Done New false
20864 7109 Acked 8108 Write 8 Done Done New false
20865 7110 Acked 8109 Write 4 Done Done New false
20866 7111 Acked 8110 Read 7 Done Done New false
20867 7112 Acked 8111 Read 5 Done Done New false
20868 7113 Acked 8112 Read 7 Done Done New false
20869 7114 Acked 8113 Read 4 Done Done New false
20870 7115 Acked 8114 Write 5 Done Done New false
20871 7116 Acked 8115 Read 7 Done Done New false
20872 7117 Acked 8116 Read 4 Done Done New false
20873 7118 Acked 8117 Read 1 Done Done New false
20874 7119 Acked 8118 Write 7 Done Done New false
20875 7120 Acked 8119 Flush 0 Done Done New false
20876 7121 Acked 8120 Write 7 Done Done New false
20877 7122 Acked 8121 Write 10 Done Done New false
20878 7123 Acked 8122 Read 9 Done Done New false
20879 7124 Acked 8123 Write 1 Done Done New false
20880 7125 Acked 8124 Write 2 Done Done New false
20881 7126 Acked 8125 Read 7 Done Done New false
20882 7127 Acked 8126 Read 10 Done Done New false
20883 7128 Acked 8127 Read 5 Done Done New false
20884 7129 Acked 8128 Write 10 Done Done New false
20885 7130 Acked 8129 Write 3 Done Done New false
20886 7131 Acked 8130 Write 7 Done Done New false
20887 7132 Acked 8131 Read 8 Done Done New false
20888 7133 Acked 8132 Read 7 Done Done New false
20889 7134 Acked 8133 Read 5 Done Done New false
20890 7135 Acked 8134 Read 5 Done Done New false
20891 7136 Acked 8135 Flush 0 Done Done New false
20892 7137 Acked 8136 Read 8 Done Done New false
20893 7138 Acked 8137 Read 2 Done Done New false
20894 7139 Acked 8138 Read 8 Done Done New false
20895 7140 Acked 8139 Read 4 Done Done New false
20896 7141 Acked 8140 Read 4 Done Done New false
20897 7142 Acked 8141 Read 3 Done Done New false
20898 7143 Acked 8142 Read 10 Done Done New false
20899 7144 Acked 8143 Read 4 Done Done New false
20900 7145 Acked 8144 Read 10 Done Done New false
20901 7146 Acked 8145 Read 9 Done Done New false
20902 7147 Acked 8146 Write 1 Done Done New false
20903 7148 Acked 8147 Write 4 Done Done New false
20904 7149 Acked 8148 Write 4 Done Done New false
20905 7150 Acked 8149 Write 1 Done Done New false
20906 7151 Acked 8150 Flush 0 Done Done New false
20907 7152 Acked 8151 Read 1 Done Done New false
20908 7153 Acked 8152 Read 5 Done Done New false
20909 7154 Acked 8153 Read 10 Done Done New false
20910 7155 Acked 8154 Read 2 Done Done New false
20911 7156 Acked 8155 Read 10 Done Done New false
20912 7157 Acked 8156 Read 2 Done Done New false
20913 7158 Acked 8157 Read 5 Done Done New false
20914 7159 Acked 8158 Read 6 Done Done New false
20915 7160 Acked 8159 Read 3 Done Done New false
20916 7161 Acked 8160 Read 5 Done Done New false
20917 7162 Acked 8161 Read 6 Done Done New false
20918 7163 Acked 8162 Read 4 Done Done New false
20919 7164 Acked 8163 Write 10 Done Done New false
20920 7165 Acked 8164 Read 10 Done Done New false
20921 7166 Acked 8165 Read 1 Done Done New false
20922 7167 Acked 8166 Flush 0 Done Done New false
20923 7168 Acked 8167 Write 3 Done Done New false
20924 7169 Acked 8168 Flush 0 Done Done New false
20925 7170 Acked 8169 Write 5 Done Done New false
20926 7171 Acked 8170 Read 2 Done Done New false
20927 7172 Acked 8171 Write 10 Done Done New false
20928 7173 Acked 8172 Write 1 Done Done New false
20929 7174 Acked 8173 Write 2 Done Done New false
20930 7175 Acked 8174 Write 10 Done Done New false
20931 7176 Acked 8175 Read 9 Done Done New false
20932 7177 Acked 8176 Read 7 Done Done New false
20933 7178 Acked 8177 Write 2 Done Done New false
20934 7179 Acked 8178 Read 5 Done Done New false
20935 7180 Acked 8179 Flush 0 Done Done New false
20936 7181 Acked 8180 Read 8 Done Done New false
20937 7182 Acked 8181 Read 8 Done Done New false
20938 7183 Acked 8182 Read 3 Done Done New false
20939 7184 Acked 8183 Read 5 Done Done New false
20940 7185 Acked 8184 Write 5 Done Done New false
20941 7186 Acked 8185 Write 7 Done Done New false
20942 7187 Acked 8186 Read 2 Done Done New false
20943 7188 Acked 8187 Read 10 Done Done New false
20944 7189 Acked 8188 Read 3 Done Done New false
20945 7190 Acked 8189 Read 5 Done Done New false
20946 7191 Acked 8190 Read 8 Done Done New false
20947 7192 Acked 8191 Read 7 Done Done New false
20948 7193 Acked 8192 Write 7 Done Done New false
20949 7194 Acked 8193 Write 3 Done Done New false
20950 7195 Acked 8194 Read 1 Done Done New false
20951 7196 Acked 8195 Flush 0 Done Done New false
20952 7197 Acked 8196 Write 8 Done Done New false
20953 7198 Acked 8197 Read 9 Done Done New false
20954 7199 Acked 8198 Read 4 Done Done New false
20955 7200 Acked 8199 Read 6 Done Done New false
20956 7201 Acked 8200 Read 4 Done Done New false
20957 7202 Acked 8201 Read 3 Done Done New false
20958 7203 Acked 8202 Write 8 Done Done New false
20959 7204 Acked 8203 Read 6 Done Done New false
20960 7205 Acked 8204 Read 1 Done Done New false
20961 7206 Acked 8205 Write 1 Done Done New false
20962 7207 Acked 8206 Write 9 Done Done New false
20963 7208 Acked 8207 Write 2 Done Done New false
20964 7209 Acked 8208 Write 2 Done Done New false
20965 7210 Acked 8209 Flush 0 Done Done New false
20966 7211 Acked 8210 Flush 0 Done Done New false
20967 7212 Acked 8211 Read 5 Done Done New false
20968 7213 Acked 8212 Read 3 Done Done New false
20969 7214 Acked 8213 Write 10 Done Done New false
20970 7215 Acked 8214 Read 10 Done Done New false
20971 7216 Acked 8215 Read 3 Done Done New false
20972 7217 Acked 8216 Write 5 Done Done New false
20973 7218 Acked 8217 Read 1 Done Done New false
20974 7219 Acked 8218 Write 5 Done Done New false
20975 7220 Acked 8219 Write 2 Done Done New false
20976 7221 Acked 8220 Read 1 Done Done New false
20977 7222 Acked 8221 Write 4 Done Done New false
20978 7223 Acked 8222 Read 10 Done Done New false
20979 7224 Acked 8223 Read 10 Done Done New false
20980 7225 Acked 8224 Write 10 Done Done New false
20981 7226 Acked 8225 Read 7 Done Done New false
20982 7227 Acked 8226 Read 2 Done Done New false
20983 7228 Acked 8227 Read 8 Done Done New false
20984 7229 Acked 8228 Read 9 Done Done New false
20985 7230 Acked 8229 Read 8 Done Done New false
20986 7231 Acked 8230 Write 2 Done Done New false
20987 7232 Acked 8231 Write 5 Done Done New false
20988 7233 Acked 8232 Read 2 Done Done New false
20989 7234 Acked 8233 Write 9 Done Done New false
20990 STATES DS:0 DS:1 DS:2 TOTAL
20991 New 0 0 301 301
20992 Sent 0 0 0 0
20993 Done 301 301 0 602
20994 Skipped 0 0 0 0
20995 Error 0 0 0 0
20996 Last Flush: 8210 8210 7932
20997 Downstairs last five completed: 7932 7931 7930 7929 7928
20998 Upstairs last five completed: 7234 7233 7232 7231 7230
20999 CLIENT: Up:0 ds:301 act:2
210002024-03-02T01:09:46.984ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210012024-03-02T01:09:46.984ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210022024-03-02T01:09:46.984ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210032024-03-02T01:09:46.984ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210042024-03-02T01:09:46.984ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210052024-03-02T01:09:46.984ZINFOcrucible: [2] client re-new 302 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210062024-03-02T01:09:46.984ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210072024-03-02T01:09:47.985ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210082024-03-02T01:09:47.985ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210092024-03-02T01:09:47.985ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210102024-03-02T01:09:47.985ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210112024-03-02T01:09:47.985ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210122024-03-02T01:09:47.985ZINFOcrucible: [2] client re-new 302 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210132024-03-02T01:09:47.986ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210142024-03-02T01:09:48.987ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210152024-03-02T01:09:48.987ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210162024-03-02T01:09:48.987ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210172024-03-02T01:09:48.987ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210182024-03-02T01:09:48.987ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210192024-03-02T01:09:48.987ZINFOcrucible: [2] client re-new 302 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210202024-03-02T01:09:48.987ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21021 [8830] got start action
21022 [8830] I am going to restart
21023 Starting downstairs at port 8830
21024 Make output file at "/tmp/dsc/downstairs-8830.txt"
21025 [8830][2] reports Starting
210262024-03-02T01:09:49.988ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210272024-03-02T01:09:49.988ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210282024-03-02T01:09:49.988ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21029 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
210302024-03-02T01:09:49.988ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210312024-03-02T01:09:49.988ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
210322024-03-02T01:09:49.988ZINFOcrucible: [2] client re-new 302 jobs since flush 7932 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21033 [8830][2] reports Running
210342024-03-02T01:09:49.988ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21035 ----------------------------------------------------------------
21036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21037 States: Active Active Offline
21038 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21039 6934 Acked 7933 Read 9 Done Done New true
21040 6935 Acked 7934 Read 10 Done Done New true
21041 6936 Acked 7935 Read 8 Done Done New true
21042 6937 Acked 7936 Read 5 Done Done New true
21043 6938 Acked 7937 Read 10 Done Done New true
21044 6939 Acked 7938 Read 4 Done Done New true
21045 6940 Acked 7939 Read 5 Done Done New true
21046 6941 Acked 7940 Write 2 Done Done New true
21047 6942 Acked 7941 Read 2 Done Done New true
21048 6943 Acked 7942 Read 8 Done Done New true
21049 6944 Acked 7943 Write 10 Done Done New true
21050 6945 Acked 7944 Read 3 Done Done New true
21051 6946 Acked 7945 Flush 0 Done Done New true
21052 6947 Acked 7946 Write 1 Done Done New true
21053 6948 Acked 7947 Read 8 Done Done New true
21054 6949 Acked 7948 Read 2 Done Done New true
21055 6950 Acked 7949 Write 1 Done Done New true
21056 6951 Acked 7950 Write 1 Done Done New true
21057 6952 Acked 7951 Read 7 Done Done New true
21058 6953 Acked 7952 Write 9 Done Done New true
21059 6954 Acked 7953 Flush 0 Done Done New true
21060 6955 Acked 7954 Read 9 Done Done New true
21061 6956 Acked 7955 Read 7 Done Done New true
21062 6957 Acked 7956 Flush 0 Done Done New true
21063 6958 Acked 7957 Read 6 Done Done New true
21064 6959 Acked 7958 Read 7 Done Done New true
21065 6960 Acked 7959 Write 5 Done Done New true
21066 6961 Acked 7960 Read 9 Done Done New true
21067 6962 Acked 7961 Read 2 Done Done New true
21068 6963 Acked 7962 Read 3 Done Done New true
21069 6964 Acked 7963 Read 4 Done Done New true
21070 6965 Acked 7964 Write 2 Done Done New true
21071 6966 Acked 7965 Read 6 Done Done New true
21072 6967 Acked 7966 Write 9 Done Done New true
21073 6968 Acked 7967 Read 6 Done Done New true
21074 6969 Acked 7968 Write 9 Done Done New true
21075 6970 Acked 7969 Read 1 Done Done New true
21076 6971 Acked 7970 Write 2 Done Done New true
21077 6972 Acked 7971 Read 5 Done Done New true
21078 6973 Acked 7972 Read 8 Done Done New true
21079 6974 Acked 7973 Write 3 Done Done New true
21080 6975 Acked 7974 Read 7 Done Done New true
21081 6976 Acked 7975 Read 10 Done Done New true
21082 6977 Acked 7976 Read 8 Done Done New true
21083 6978 Acked 7977 Write 8 Done Done New true
21084 6979 Acked 7978 Write 6 Done Done New true
21085 6980 Acked 7979 Write 4 Done Done New true
21086 6981 Acked 7980 Read 4 Done Done New true
21087 6982 Acked 7981 Read 7 Done Done New true
21088 6983 Acked 7982 Read 10 Done Done New true
21089 6984 Acked 7983 Write 1 Done Done New true
21090 6985 Acked 7984 Write 3 Done Done New true
21091 6986 Acked 7985 Read 6 Done Done New true
21092 6987 Acked 7986 Read 10 Done Done New true
21093 6988 Acked 7987 Read 3 Done Done New true
21094 6989 Acked 7988 Write 4 Done Done New true
21095 6990 Acked 7989 Write 7 Done Done New true
21096 6991 Acked 7990 Read 1 Done Done New true
21097 6992 Acked 7991 Write 2 Done Done New true
21098 6993 Acked 7992 Read 6 Done Done New true
21099 6994 Acked 7993 Read 1 Done Done New true
21100 6995 Acked 7994 Read 1 Done Done New true
21101 6996 Acked 7995 Write 10 Done Done New true
21102 6997 Acked 7996 Read 7 Done Done New true
21103 6998 Acked 7997 Read 2 Done Done New true
21104 6999 Acked 7998 Read 1 Done Done New true
21105 7000 Acked 7999 Write 7 Done Done New true
21106 7001 Acked 8000 Read 8 Done Done New true
21107 7002 Acked 8001 Read 8 Done Done New true
21108 7003 Acked 8002 Read 2 Done Done New true
21109 7004 Acked 8003 Read 1 Done Done New true
21110 7005 Acked 8004 Read 9 Done Done New true
21111 7006 Acked 8005 Read 7 Done Done New true
21112 7007 Acked 8006 Read 2 Done Done New true
21113 7008 Acked 8007 Write 1 Done Done New true
21114 7009 Acked 8008 Write 6 Done Done New true
21115 7010 Acked 8009 Write 9 Done Done New true
21116 7011 Acked 8010 Flush 0 Done Done New true
21117 7012 Acked 8011 Read 7 Done Done New true
21118 7013 Acked 8012 Write 5 Done Done New true
21119 7014 Acked 8013 Read 6 Done Done New true
21120 7015 Acked 8014 Read 6 Done Done New true
21121 7016 Acked 8015 Read 3 Done Done New true
21122 7017 Acked 8016 Write 2 Done Done New true
21123 7018 Acked 8017 Flush 0 Done Done New true
21124 7019 Acked 8018 Write 7 Done Done New true
21125 7020 Acked 8019 Read 3 Done Done New true
21126 7021 Acked 8020 Read 5 Done Done New true
21127 7022 Acked 8021 Write 3 Done Done New true
21128 7023 Acked 8022 Read 9 Done Done New true
21129 7024 Acked 8023 Write 3 Done Done New true
21130 7025 Acked 8024 Read 6 Done Done New true
21131 7026 Acked 8025 Read 1 Done Done New true
21132 7027 Acked 8026 Read 9 Done Done New true
21133 7028 Acked 8027 Write 1 Done Done New true
21134 7029 Acked 8028 Write 10 Done Done New true
21135 7030 Acked 8029 Read 7 Done Done New true
21136 7031 Acked 8030 Read 4 Done Done New true
21137 7032 Acked 8031 Write 7 Done Done New true
21138 7033 Acked 8032 Read 8 Done Done New true
21139 7034 Acked 8033 Read 4 Done Done New true
21140 7035 Acked 8034 Read 5 Done Done New true
21141 7036 Acked 8035 Read 5 Done Done New true
21142 7037 Acked 8036 Read 6 Done Done New true
21143 7038 Acked 8037 Read 4 Done Done New true
21144 7039 Acked 8038 Write 2 Done Done New true
21145 7040 Acked 8039 Read 1 Done Done New true
21146 7041 Acked 8040 Flush 0 Done Done New true
21147 7042 Acked 8041 Write 2 Done Done New true
21148 7043 Acked 8042 Flush 0 Done Done New true
21149 7044 Acked 8043 Write 2 Done Done New true
21150 7045 Acked 8044 Write 5 Done Done New true
21151 7046 Acked 8045 Read 5 Done Done New true
21152 7047 Acked 8046 Read 2 Done Done New true
21153 7048 Acked 8047 Write 10 Done Done New true
21154 7049 Acked 8048 Write 2 Done Done New true
21155 7050 Acked 8049 Write 7 Done Done New true
21156 7051 Acked 8050 Read 2 Done Done New true
21157 7052 Acked 8051 Read 6 Done Done New true
21158 7053 Acked 8052 Read 4 Done Done New true
21159 7054 Acked 8053 Write 9 Done Done New true
21160 7055 Acked 8054 Write 4 Done Done New true
21161 7056 Acked 8055 Write 6 Done Done New true
21162 7057 Acked 8056 Read 1 Done Done New true
21163 7058 Acked 8057 Write 4 Done Done New true
21164 7059 Acked 8058 Flush 0 Done Done New true
21165 7060 Acked 8059 Read 4 Done Done New true
21166 7061 Acked 8060 Read 10 Done Done New true
21167 7062 Acked 8061 Write 2 Done Done New true
21168 7063 Acked 8062 Write 8 Done Done New true
21169 7064 Acked 8063 Flush 0 Done Done New true
21170 7065 Acked 8064 Read 3 Done Done New true
21171 7066 Acked 8065 Read 1 Done Done New true
21172 7067 Acked 8066 Write 6 Done Done New true
21173 7068 Acked 8067 Write 3 Done Done New true
21174 7069 Acked 8068 Read 5 Done Done New true
21175 7070 Acked 8069 Write 8 Done Done New true
21176 7071 Acked 8070 Flush 0 Done Done New true
21177 7072 Acked 8071 Flush 0 Done Done New true
21178 7073 Acked 8072 Flush 0 Done Done New true
21179 7074 Acked 8073 Read 4 Done Done New true
21180 7075 Acked 8074 Read 6 Done Done New true
21181 7076 Acked 8075 Read 1 Done Done New true
21182 7077 Acked 8076 Write 7 Done Done New true
21183 7078 Acked 8077 Read 2 Done Done New true
21184 7079 Acked 8078 Read 3 Done Done New true
21185 7080 Acked 8079 Write 4 Done Done New true
21186 7081 Acked 8080 Write 5 Done Done New true
21187 7082 Acked 8081 Read 9 Done Done New true
21188 7083 Acked 8082 Write 3 Done Done New true
21189 7084 Acked 8083 Read 4 Done Done New true
21190 7085 Acked 8084 Read 2 Done Done New true
21191 7086 Acked 8085 Read 10 Done Done New true
21192 7087 Acked 8086 Write 4 Done Done New true
21193 7088 Acked 8087 Read 7 Done Done New true
21194 7089 Acked 8088 Write 10 Done Done New true
21195 7090 Acked 8089 Read 5 Done Done New true
21196 7091 Acked 8090 Write 2 Done Done New true
21197 7092 Acked 8091 Read 6 Done Done New true
21198 7093 Acked 8092 Read 7 Done Done New true
21199 7094 Acked 8093 Write 7 Done Done New true
21200 7095 Acked 8094 Write 1 Done Done New true
21201 7096 Acked 8095 Flush 0 Done Done New true
21202 7097 Acked 8096 Write 9 Done Done New true
21203 7098 Acked 8097 Write 3 Done Done New true
21204 7099 Acked 8098 Read 5 Done Done New true
21205 7100 Acked 8099 Write 6 Done Done New true
21206 7101 Acked 8100 Write 1 Done Done New true
21207 7102 Acked 8101 Read 7 Done Done New true
21208 7103 Acked 8102 Write 3 Done Done New true
21209 7104 Acked 8103 Read 8 Done Done New true
21210 7105 Acked 8104 Write 9 Done Done New true
21211 7106 Acked 8105 Write 7 Done Done New true
21212 7107 Acked 8106 Read 6 Done Done New true
21213 7108 Acked 8107 Read 2 Done Done New true
21214 7109 Acked 8108 Write 8 Done Done New true
21215 7110 Acked 8109 Write 4 Done Done New true
21216 7111 Acked 8110 Read 7 Done Done New true
21217 7112 Acked 8111 Read 5 Done Done New true
21218 7113 Acked 8112 Read 7 Done Done New true
21219 7114 Acked 8113 Read 4 Done Done New true
21220 7115 Acked 8114 Write 5 Done Done New true
21221 7116 Acked 8115 Read 7 Done Done New true
21222 7117 Acked 8116 Read 4 Done Done New true
21223 7118 Acked 8117 Read 1 Done Done New true
21224 7119 Acked 8118 Write 7 Done Done New true
21225 7120 Acked 8119 Flush 0 Done Done New true
21226 7121 Acked 8120 Write 7 Done Done New true
21227 7122 Acked 8121 Write 10 Done Done New true
21228 7123 Acked 8122 Read 9 Done Done New true
21229 7124 Acked 8123 Write 1 Done Done New true
21230 7125 Acked 8124 Write 2 Done Done New true
21231 7126 Acked 8125 Read 7 Done Done New true
21232 7127 Acked 8126 Read 10 Done Done New true
21233 7128 Acked 8127 Read 5 Done Done New true
21234 7129 Acked 8128 Write 10 Done Done New true
21235 7130 Acked 8129 Write 3 Done Done New true
21236 7131 Acked 8130 Write 7 Done Done New true
21237 7132 Acked 8131 Read 8 Done Done New true
21238 7133 Acked 8132 Read 7 Done Done New true
21239 7134 Acked 8133 Read 5 Done Done New true
21240 7135 Acked 8134 Read 5 Done Done New true
21241 7136 Acked 8135 Flush 0 Done Done New true
21242 7137 Acked 8136 Read 8 Done Done New true
21243 7138 Acked 8137 Read 2 Done Done New true
21244 7139 Acked 8138 Read 8 Done Done New true
21245 7140 Acked 8139 Read 4 Done Done New true
21246 7141 Acked 8140 Read 4 Done Done New true
21247 7142 Acked 8141 Read 3 Done Done New true
21248 7143 Acked 8142 Read 10 Done Done New true
21249 7144 Acked 8143 Read 4 Done Done New true
21250 7145 Acked 8144 Read 10 Done Done New true
21251 7146 Acked 8145 Read 9 Done Done New true
21252 7147 Acked 8146 Write 1 Done Done New true
21253 7148 Acked 8147 Write 4 Done Done New true
21254 7149 Acked 8148 Write 4 Done Done New true
21255 7150 Acked 8149 Write 1 Done Done New true
21256 7151 Acked 8150 Flush 0 Done Done New true
21257 7152 Acked 8151 Read 1 Done Done New true
21258 7153 Acked 8152 Read 5 Done Done New true
21259 7154 Acked 8153 Read 10 Done Done New true
21260 7155 Acked 8154 Read 2 Done Done New true
21261 7156 Acked 8155 Read 10 Done Done New true
21262 7157 Acked 8156 Read 2 Done Done New true
21263 7158 Acked 8157 Read 5 Done Done New true
21264 7159 Acked 8158 Read 6 Done Done New true
21265 7160 Acked 8159 Read 3 Done Done New true
21266 7161 Acked 8160 Read 5 Done Done New true
21267 7162 Acked 8161 Read 6 Done Done New true
21268 7163 Acked 8162 Read 4 Done Done New true
21269 7164 Acked 8163 Write 10 Done Done New true
21270 7165 Acked 8164 Read 10 Done Done New true
21271 7166 Acked 8165 Read 1 Done Done New true
21272 7167 Acked 8166 Flush 0 Done Done New true
21273 7168 Acked 8167 Write 3 Done Done New true
21274 7169 Acked 8168 Flush 0 Done Done New true
21275 7170 Acked 8169 Write 5 Done Done New true
21276 7171 Acked 8170 Read 2 Done Done New true
21277 7172 Acked 8171 Write 10 Done Done New true
21278 7173 Acked 8172 Write 1 Done Done New true
21279 7174 Acked 8173 Write 2 Done Done New true
21280 7175 Acked 8174 Write 10 Done Done New true
21281 7176 Acked 8175 Read 9 Done Done New true
21282 7177 Acked 8176 Read 7 Done Done New true
21283 7178 Acked 8177 Write 2 Done Done New true
21284 7179 Acked 8178 Read 5 Done Done New true
21285 7180 Acked 8179 Flush 0 Done Done New true
21286 7181 Acked 8180 Read 8 Done Done New true
21287 7182 Acked 8181 Read 8 Done Done New true
21288 7183 Acked 8182 Read 3 Done Done New true
21289 7184 Acked 8183 Read 5 Done Done New true
21290 7185 Acked 8184 Write 5 Done Done New true
21291 7186 Acked 8185 Write 7 Done Done New true
21292 7187 Acked 8186 Read 2 Done Done New true
21293 7188 Acked 8187 Read 10 Done Done New true
21294 7189 Acked 8188 Read 3 Done Done New true
21295 7190 Acked 8189 Read 5 Done Done New true
21296 7191 Acked 8190 Read 8 Done Done New true
21297 7192 Acked 8191 Read 7 Done Done New true
21298 7193 Acked 8192 Write 7 Done Done New true
21299 7194 Acked 8193 Write 3 Done Done New true
21300 7195 Acked 8194 Read 1 Done Done New true
21301 7196 Acked 8195 Flush 0 Done Done New true
21302 7197 Acked 8196 Write 8 Done Done New true
21303 7198 Acked 8197 Read 9 Done Done New true
21304 7199 Acked 8198 Read 4 Done Done New true
21305 7200 Acked 8199 Read 6 Done Done New true
21306 7201 Acked 8200 Read 4 Done Done New true
21307 7202 Acked 8201 Read 3 Done Done New true
21308 7203 Acked 8202 Write 8 Done Done New true
21309 7204 Acked 8203 Read 6 Done Done New true
21310 7205 Acked 8204 Read 1 Done Done New true
21311 7206 Acked 8205 Write 1 Done Done New true
21312 7207 Acked 8206 Write 9 Done Done New true
21313 7208 Acked 8207 Write 2 Done Done New true
21314 7209 Acked 8208 Write 2 Done Done New true
21315 7210 Acked 8209 Flush 0 Done Done New true
21316 7211 Acked 8210 Flush 0 Done Done New true
21317 7212 Acked 8211 Read 5 Done Done New true
21318 7213 Acked 8212 Read 3 Done Done New true
21319 7214 Acked 8213 Write 10 Done Done New true
21320 7215 Acked 8214 Read 10 Done Done New true
21321 7216 Acked 8215 Read 3 Done Done New true
21322 7217 Acked 8216 Write 5 Done Done New true
21323 7218 Acked 8217 Read 1 Done Done New true
21324 7219 Acked 8218 Write 5 Done Done New true
21325 7220 Acked 8219 Write 2 Done Done New true
21326 7221 Acked 8220 Read 1 Done Done New true
21327 7222 Acked 8221 Write 4 Done Done New true
21328 7223 Acked 8222 Read 10 Done Done New true
21329 7224 Acked 8223 Read 10 Done Done New true
21330 7225 Acked 8224 Write 10 Done Done New true
21331 7226 Acked 8225 Read 7 Done Done New true
21332 7227 Acked 8226 Read 2 Done Done New true
21333 7228 Acked 8227 Read 8 Done Done New true
21334 7229 Acked 8228 Read 9 Done Done New true
21335 7230 Acked 8229 Read 8 Done Done New true
21336 7231 Acked 8230 Write 2 Done Done New true
21337 7232 Acked 8231 Write 5 Done Done New true
21338 7233 Acked 8232 Read 2 Done Done New true
21339 7234 Acked 8233 Write 9 Done Done New true
21340 7235 Acked 8234 Flush 0 Done Done New true
21341 STATES DS:0 DS:1 DS:2 TOTAL
21342 New 0 0 302 302
21343 Sent 0 0 0 0
21344 Done 302 302 0 604
21345 Skipped 0 0 0 0
21346 Error 0 0 0 0
21347 Last Flush: 8234 8234 7932
21348 Downstairs last five completed: 7932 7931 7930 7929 7928
21349 Upstairs last five completed: 7235 7234 7233 7232 7231
21350 CLIENT: Up:0 ds:302 act:2
213512024-03-02T01:09:50.990ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213522024-03-02T01:09:50.990ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213532024-03-02T01:09:50.991ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213542024-03-02T01:09:50.991ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213552024-03-02T01:09:50.991ZINFOcrucible: send last flush ID to this DS: 7932 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213562024-03-02T01:09:50.991ZINFOcrucible: Replied this last flush ID: 7932 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213572024-03-02T01:09:50.991ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213582024-03-02T01:09:50.991ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213592024-03-02T01:09:50.991ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213602024-03-02T01:09:50.991ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21361 ----------------------------------------------------------------
21362 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21363 Downstairs last five completed: 8234 8233 8232 8231 8230
21364 Upstairs last five completed: 7235 7234 7233 7232 7231
21365 CLIENT: Up:0 ds:0 act:3
21366 Replay: All jobs finished, all DS active.
21367 Main task has work to do, go find it
21368 got dsc Stop(1)
21369 stop 1
21370 Mar 02 01:09:54.330 INFO request completed, latency_us: 354, response_code: [8820] Got stop action so:false kr:false
21371 204, uri: /stop/cid/1, method: POST, req_id: edd6eeb6-0852-4116-8ed9-3a042caa76f2, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
21372 Mar 02 01:09:54.331 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/1, method: GET, req_id: 39e728f9-f51f-43d2-8089-a48ab0759c6c, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
213732024-03-02T01:09:54.331ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213742024-03-02T01:09:54.331ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213752024-03-02T01:09:54.331ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213762024-03-02T01:09:54.331ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213772024-03-02T01:09:54.331ZINFOcrucible: [1] client re-new 0 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213782024-03-02T01:09:54.331ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21379 [8820] Exited with: ExitStatus(unix_wait_status(9))
21380 [8820][1] reports Exit
213812024-03-02T01:09:55.333ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213822024-03-02T01:09:55.333ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213832024-03-02T01:09:55.333ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213842024-03-02T01:09:55.333ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213852024-03-02T01:09:55.333ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213862024-03-02T01:09:55.333ZINFOcrucible: [1] client re-new 0 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213872024-03-02T01:09:55.333ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213882024-03-02T01:09:56.335ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213892024-03-02T01:09:56.335ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213902024-03-02T01:09:56.335ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213912024-03-02T01:09:56.335ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213922024-03-02T01:09:56.335ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213932024-03-02T01:09:56.335ZINFOcrucible: [1] client re-new 0 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213942024-03-02T01:09:56.335ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213952024-03-02T01:09:57.336ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213962024-03-02T01:09:57.336ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213972024-03-02T01:09:57.336ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213982024-03-02T01:09:57.336ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
213992024-03-02T01:09:57.336ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214002024-03-02T01:09:57.336ZINFOcrucible: [1] client re-new 0 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214012024-03-02T01:09:57.336ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21402 Mar 02 01:09:58.333 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5a6da558-8684-4698-88f4-77785b648db7, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
21403 001/300 Read block 4174 len 36864
21404 002/300 Read block 4713 len 16384
21405 003/300 Read block 2890 len 24576
21406 004/300 Read block 601 len 28672
214072024-03-02T01:09:58.338ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214082024-03-02T01:09:58.338ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214092024-03-02T01:09:58.338ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214102024-03-02T01:09:58.338ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214112024-03-02T01:09:58.338ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214122024-03-02T01:09:58.338ZINFOcrucible: [1] client re-new 4 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
214132024-03-02T01:09:58.338ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
21414 005/300 Read block 4211 len 28672
21415 006/300 Write block 3587 len 24576 data: 7 7 7 6 4 5
21416 007/300 Read block 4315 len 32768
21417 008/300 Read block 2162 len 4096
21418 009/300 Read block 1864 len 20480
21419 010/300 Write block 1769 len 8192 data: 3 5
21420 011/300 Write block 4786 len 16384 data: 7 6 7 5
21421 012/300 Read block 4837 len 28672
21422 013/300 Write block 1485 len 4096 data: 2
21423 014/300 Read block 4914 len 28672
21424 015/300 Write block 4964 len 24576 data: 5 6 6 5 5 8
21425 016/300 Flush
21426 017/300 Write block 1599 len 16384 data: 3 3 3 4
21427 018/300 Read block 3814 len 8192
21428 019/300 Read block 3595 len 20480
21429 020/300 Flush
21430 021/300 Write block 1607 len 4096 data: 6
21431 022/300 Read block 4688 len 12288
21432 023/300 Write block 4515 len 40960 data: 5 5 4 4 3 4 3 4 4 4
21433 024/300 Write block 3337 len 12288 data: 6 7 5
21434 025/300 Read block 1359 len 12288
21435 026/300 Read block 2463 len 20480
21436 027/300 Write block 3614 len 40960 data: 7 8 5 6 6 6 5 4 4 5
21437 028/300 Write block 2478 len 24576 data: 8 5 6 5 7 6
21438 029/300 Read block 2688 len 32768
21439 030/300 Read block 3202 len 20480
21440 031/300 Read block 1138 len 20480
21441 032/300 Flush
21442 033/300 Flush
21443 034/300 Read block 856 len 32768
21444 035/300 Write block 4333 len 36864 data: 8 9 8 6 7 6 5 3 4
21445 036/300 Flush
21446 037/300 Read block 856 len 40960
21447 038/300 Flush
21448 039/300 Write block 4879 len 20480 data: 4 4 5 4 3
21449 040/300 Read block 723 len 32768
21450 041/300 Write block 4061 len 40960 data: 7 5 5 4 4 4 4 3 5 5
21451 042/300 Flush
21452 043/300 Read block 2688 len 16384
21453 044/300 Read block 4465 len 40960
21454 045/300 Write block 2888 len 32768 data: 7 6 5 4 4 5 4 4
21455 046/300 Write block 4558 len 8192 data: 3 4
21456 047/300 Read block 2172 len 8192
21457 048/300 Write block 4007 len 20480 data: 3 3 4 3 3
21458 049/300 Write block 4900 len 16384 data: 3 5 5 6
21459 050/300 Read block 2906 len 20480
21460 051/300 Flush
21461 052/300 Read block 3355 len 12288
21462 053/300 Write block 4463 len 12288 data: 4 3 2
21463 054/300 Write block 2970 len 24576 data: 4 4 5 3 3 3
21464 055/300 Write block 1213 len 32768 data: 9 7 5 4 4 3 3 4
21465 056/300 Read block 72 len 28672
21466 057/300 Write block 1862 len 28672 data: 6 3 4 3 4 4 4
21467 058/300 Read block 720 len 16384
21468 059/300 Write block 3509 len 28672 data: 5 5 5 5 7 8 9
21469 060/300 Read block 177 len 40960
21470 061/300 Write block 3629 len 40960 data: 5 5 5 4 3 4 3 3 2 2
21471 062/300 Flush
21472 063/300 Read block 126 len 32768
21473 064/300 Flush
21474 065/300 Read block 3444 len 24576
21475 066/300 Write block 588 len 20480 data: 7 6 6 7 5
21476 067/300 Write block 2953 len 4096 data: 7
21477 068/300 Read block 100 len 28672
21478 069/300 Flush
21479 070/300 Read block 3626 len 40960
21480 071/300 Write block 2782 len 8192 data: 5 5
21481 072/300 Read block 931 len 20480
21482 073/300 Read block 3840 len 32768
21483 074/300 Flush
21484 075/300 Read block 400 len 24576
21485 076/300 Write block 3829 len 32768 data: 5 5 5 6 4 3 3 4
21486 077/300 Read block 4640 len 28672
21487 078/300 Read block 1105 len 36864
21488 079/300 Write block 1300 len 32768 data: 7 6 4 4 4 5 4 3
21489 080/300 Read block 2782 len 8192
21490 081/300 Read block 987 len 40960
21491 082/300 Write block 2375 len 28672 data: 3 4 5 5 5 5 5
21492 083/300 Read block 1830 len 8192
21493 084/300 Flush
21494 085/300 Write block 1672 len 4096 data: 3
21495 086/300 Read block 4459 len 20480
21496 087/300 Read block 4367 len 20480
21497 088/300 Write block 2433 len 32768 data: 6 7 7 6 6 5 6 6
21498 089/300 Flush
21499 090/300 Write block 4373 len 24576 data: 3 3 3 2 3 3
21500 091/300 Read block 1318 len 4096
21501 092/300 Write block 4640 len 36864 data: 5 5 7 10 10 9 9 9 6
21502 093/300 Read block 4396 len 40960
21503 094/300 Write block 1156 len 24576 data: 6 5 6 4 5 6
21504 095/300 Read block 4967 len 32768
21505 096/300 Read block 3992 len 24576
21506 097/300 Flush
21507 098/300 Read block 4372 len 8192
21508 099/300 Read block 140 len 12288
21509 100/300 Read block 4856 len 20480
21510 101/300 Flush
21511 102/300 Write block 3763 len 20480 data: 3 4 6 6 6
21512 103/300 Write block 2665 len 36864 data: 4 4 4 3 3 3 4 5 5
21513 104/300 Write block 2919 len 20480 data: 2 2 2 6 7
21514 105/300 Write block 4312 len 20480 data: 6 5 5 6 6
21515 106/300 Read block 4144 len 36864
21516 107/300 Flush
21517 108/300 Write block 2543 len 28672 data: 6 4 3 3 3 4 3
21518 109/300 Write block 3344 len 28672 data: 7 6 4 4 6 6 5
21519 110/300 Flush
21520 111/300 Write block 2476 len 8192 data: 6 8
21521 112/300 Read block 4972 len 40960
21522 113/300 Read block 1515 len 20480
21523 114/300 Write block 3976 len 8192 data: 3 5
21524 115/300 Read block 1139 len 28672
21525 116/300 Read block 939 len 8192
21526 117/300 Read block 3107 len 24576
21527 118/300 Write block 1419 len 8192 data: 7 6
21528 119/300 Read block 1244 len 36864
21529 120/300 Write block 3160 len 20480 data: 6 6 3 3 3
21530 121/300 Write block 538 len 28672 data: 5 4 4 5 5 4 5
21531 122/300 Flush
21532 123/300 Write block 1751 len 16384 data: 3 4 3 4
21533 124/300 Flush
21534 125/300 Write block 3975 len 24576 data: 3 4 6 5 6 6
21535 126/300 Write block 3583 len 40960 data: 4 4 5 5 8 8 8 7 5 6
21536 127/300 Flush
21537 128/300 Write block 3394 len 24576 data: 5 5 5 5 6 3
21538 129/300 Read block 2195 len 24576
21539 130/300 Read block 219 len 20480
21540 131/300 Read block 366 len 24576
21541 132/300 Write block 625 len 16384 data: 7 6 5 5
21542 133/300 Read block 4404 len 12288
21543 134/300 Read block 4761 len 28672
21544 135/300 Read block 1584 len 40960
21545 136/300 Flush
21546 137/300 Read block 1959 len 4096
21547 138/300 Write block 497 len 40960 data: 3 3 3 3 2 2 2 2 2 3
21548 139/300 Flush
21549 140/300 Read block 3443 len 16384
21550 141/300 Read block 654 len 36864
21551 142/300 Read block 3297 len 40960
21552 143/300 Write block 704 len 4096 data: 7
21553 144/300 Read block 3880 len 16384
21554 145/300 Read block 1332 len 16384
21555 146/300 Read block 711 len 12288
21556 147/300 Write block 544 len 4096 data: 6
21557 148/300 Write block 1068 len 16384 data: 4 4 4 3
21558 149/300 Read block 764 len 36864
21559 150/300 Read block 3286 len 4096
21560 151/300 Write block 708 len 8192 data: 4 2
21561 152/300 Read block 459 len 28672
21562 153/300 Read block 2884 len 16384
21563 154/300 Read block 2601 len 12288
21564 155/300 Read block 244 len 8192
21565 156/300 Flush
21566 157/300 Read block 3827 len 40960
21567 158/300 Write block 2803 len 40960 data: 5 5 3 4 4 5 7 8 8 7
21568 159/300 Write block 1948 len 40960 data: 6 6 6 6 6 5 6 5 5 5
21569 160/300 Write block 3162 len 24576 data: 4 4 4 3 2 3
21570 161/300 Write block 2700 len 8192 data: 3 3
21571 162/300 Read block 4810 len 16384
21572 163/300 Read block 2370 len 28672
21573 164/300 Read block 4939 len 4096
21574 165/300 Read block 2259 len 16384
21575 166/300 Write block 4725 len 16384 data: 7 7 7 7
21576 167/300 Write block 1423 len 16384 data: 5 3 3 4
21577 168/300 Write block 3206 len 32768 data: 7 7 8 8 7 6 5 5
21578 169/300 Read block 1510 len 12288
21579 170/300 Write block 332 len 8192 data: 7 7
21580 171/300 Write block 2957 len 20480 data: 7 6 7 6 6
21581 172/300 Read block 2798 len 8192
21582 173/300 Read block 3596 len 4096
21583 174/300 Write block 2974 len 40960 data: 4 4 3 3 3 3 3 3 2 3
21584 175/300 Read block 4439 len 16384
21585 176/300 Read block 1891 len 20480
21586 177/300 Write block 3524 len 12288 data: 9 6 6
21587 178/300 Read block 2075 len 36864
21588 179/300 Write block 630 len 20480 data: 5 3 3 3 3
21589 180/300 Read block 3584 len 4096
21590 181/300 Write block 525 len 36864 data: 4 4 5 6 7 8 9 11 8
21591 182/300 Write block 161 len 12288 data: 6 5 5
21592 183/300 Write block 1911 len 4096 data: 5
21593 184/300 Read block 3299 len 40960
21594 185/300 Write block 4395 len 28672 data: 5 7 5 4 4 7 7
21595 186/300 Read block 1982 len 36864
21596 187/300 Read block 3863 len 12288
21597 188/300 Read block 1491 len 12288
21598 189/300 Read block 2391 len 16384
21599 190/300 Write block 2849 len 12288 data: 9 6 4
21600 191/300 Read block 1366 len 16384
21601 192/300 Flush
21602 193/300 Write block 2672 len 20480 data: 6 6 5 7 6
21603 194/300 Read block 1637 len 40960
21604 195/300 Write block 4921 len 8192 data: 4 5
21605 196/300 Write block 291 len 28672 data: 8 7 9 7 11 10 9
21606 197/300 Write block 1256 len 4096 data: 5
21607 198/300 Read block 4812 len 32768
21608 199/300 Write block 4953 len 16384 data: 7 7 5 3
21609 200/300 Write block 2524 len 24576 data: 5 6 6 5 5 4
21610 201/300 Read block 3487 len 32768
21611 202/300 Flush
21612 203/300 Write block 540 len 20480 data: 5 6 6 5 7
21613 204/300 Read block 1892 len 4096
21614 205/300 Read block 1643 len 8192
21615 206/300 Write block 2978 len 32768 data: 4 4 4 4 3 4 3 3
21616 207/300 Read block 2691 len 16384
21617 208/300 Read block 2217 len 36864
21618 209/300 Read block 1116 len 8192
21619 210/300 Flush
21620 211/300 Flush
21621 212/300 Write block 3213 len 8192 data: 6 4
21622 213/300 Write block 3550 len 16384 data: 5 7 8 7
21623 214/300 Read block 1462 len 4096
21624 215/300 Write block 4881 len 8192 data: 6 5
21625 216/300 Read block 4994 len 20480
21626 217/300 Read block 3814 len 24576
21627 218/300 Flush
21628 219/300 Read block 3985 len 28672
21629 220/300 Write block 3089 len 40960 data: 3 3 3 3 3 4 7 7 7 6
21630 221/300 Read block 3992 len 8192
21631 222/300 Read block 4268 len 12288
21632 223/300 Read block 1969 len 40960
21633 224/300 Write block 3134 len 36864 data: 4 5 5 5 6 5 4 4 4
21634 225/300 Flush
21635 226/300 Read block 1153 len 8192
21636 227/300 Read block 3600 len 12288
21637 228/300 Write block 116 len 24576 data: 4 3 3 3 3 2
21638 229/300 Write block 4153 len 4096 data: 3
21639 230/300 Write block 1952 len 32768 data: 7 6 7 6 6 6 5 5
21640 231/300 Write block 596 len 36864 data: 4 5 5 6 6 6 6 5 7
21641 232/300 Flush
21642 233/300 Flush
21643 234/300 Write block 4172 len 12288 data: 5 4 5
21644 235/300 Read block 4295 len 32768
21645 236/300 Read block 1220 len 40960
21646 237/300 Write block 427 len 16384 data: 5 4 4 4
21647 238/300 Read block 740 len 8192
21648 239/300 Write block 3454 len 36864 data: 7 7 4 5 6 4 4 5 6
21649 240/300 Flush
21650 241/300 Read block 346 len 28672
21651 242/300 Flush
21652 243/300 Flush
21653 244/300 Write block 1435 len 16384 data: 7 4 7 6
21654 245/300 Flush
21655 246/300 Read block 4218 len 4096
21656 247/300 Write block 2984 len 32768 data: 4 4 3 4 4 4 4 6
21657 248/300 Read block 4267 len 8192
21658 249/300 Read block 1880 len 28672
21659 250/300 Write block 4202 len 40960 data: 4 4 6 5 5 6 5 5 6 6
21660 251/300 Write block 1906 len 40960 data: 6 5 6 6 6 6 5 5 6 6
21661 252/300 Write block 1783 len 8192 data: 5 5
21662 253/300 Write block 124 len 8192 data: 5 5
21663 254/300 Flush
21664 255/300 Read block 897 len 8192
21665 256/300 Write block 2718 len 20480 data: 9 8 9 7 7
21666 257/300 Read block 199 len 16384
21667 258/300 Read block 3318 len 24576
21668 259/300 Write block 3015 len 16384 data: 8 9 9 8
21669 260/300 Flush
21670 261/300 Read block 4869 len 12288
21671 262/300 Write block 3926 len 4096 data: 10
21672 263/300 Write block 3066 len 8192 data: 2 2
21673 264/300 Write block 1909 len 24576 data: 7 7 7 6 6 7
21674 265/300 Read block 2381 len 28672
21675 266/300 Read block 4720 len 12288
21676 267/300 Read block 3091 len 16384
21677 268/300 Read block 2924 len 28672
21678 269/300 Read block 4961 len 12288
21679 270/300 Write block 561 len 8192 data: 3 4
21680 271/300 Write block 326 len 28672 data: 8 7 7 7 7 5 8
21681 272/300 Flush
21682 273/300 Read block 813 len 36864
21683 274/300 Write block 1547 len 4096 data: 2
21684 275/300 Write block 3176 len 8192 data: 5 6
21685 276/300 Read block 164 len 8192
21686 277/300 Write block 82 len 32768 data: 5 5 4 4 4 4 4 5
21687 278/300 Read block 3435 len 40960
21688 279/300 Write block 340 len 16384 data: 14 12 12 11
21689 280/300 Write block 2824 len 16384 data: 5 8 8 8
21690 281/300 Write block 1044 len 32768 data: 7 6 5 5 7 6 7 7
21691 282/300 Read block 45 len 24576
21692 283/300 Write block 1863 len 36864 data: 4 5 4 5 5 5 5 5 5
21693 284/300 Read block 3438 len 36864
21694 285/300 Write block 4233 len 8192 data: 6 7
21695 286/300 Read block 2951 len 8192
21696 287/300 Flush
21697 288/300 Write block 2623 len 8192 data: 3 4
21698 289/300 Read block 3552 len 28672
21699 290/300 Read block 1298 len 16384
21700 291/300 Write block 3679 len 8192 data: 7 8
21701 292/300 Read block 2828 len 8192
21702 293/300 Read block 3856 len 4096
21703 294/300 Read block 4624 len 12288
21704 295/300 Read block 4726 len 16384
21705 296/300 Write block 3357 len 32768 data: 8 8 9 8 6 7 7 5
21706 297/300 Read block 4011 len 20480
21707 298/300 Write block 2962 len 16384 data: 5 4 4 8
21708 299/300 Write block 4106 len 4096 data: 2
21709 300/300 Flush
21710 Main task has work to do, go find it
21711 got dsc Start(1)
21712 start 1
21713 Mar 02 01:09:58.685 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: 97a001b7-c216-4d15-844c-2722bc933b4e, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
21714 Replay: started 1, returned:Ok(())
21715 ----------------------------------------------------------------
21716 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21717 States: Active Offline Active
21718 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21719 7236 Acked 8235 Read 9 Done New Done true
21720 7237 Acked 8236 Read 4 Done New Done true
21721 7238 Acked 8237 Read 6 Done New Done true
21722 7239 Acked 8238 Read 7 Done New Done true
21723 7240 Acked 8239 Read 7 Done New Done false
21724 7241 Acked 8240 Write 6 Done New Done false
21725 7242 Acked 8241 Read 8 Done New Done false
21726 7243 Acked 8242 Read 1 Done New Done false
21727 7244 Acked 8243 Read 5 Done New Done false
21728 7245 Acked 8244 Write 2 Done New Done false
21729 7246 Acked 8245 Write 4 Done New Done false
21730 7247 Acked 8246 Read 7 Done New Done false
21731 7248 Acked 8247 Write 1 Done New Done false
21732 7249 Acked 8248 Read 7 Done New Done false
21733 7250 Acked 8249 Write 6 Done New Done false
21734 7251 Acked 8250 Flush 0 Done New Done false
21735 7252 Acked 8251 Write 4 Done New Done false
21736 7253 Acked 8252 Read 2 Done New Done false
21737 7254 Acked 8253 Read 5 Done New Done false
21738 7255 Acked 8254 Flush 0 Done New Done false
21739 7256 Acked 8255 Write 1 Done New Done false
21740 7257 Acked 8256 Read 3 Done New Done false
21741 7258 Acked 8257 Write 10 Done New Done false
21742 7259 Acked 8258 Write 3 Done New Done false
21743 7260 Acked 8259 Read 3 Done New Done false
21744 7261 Acked 8260 Read 5 Done New Done false
21745 7262 Acked 8261 Write 10 Done New Done false
21746 7263 Acked 8262 Write 6 Done New Done false
21747 7264 Acked 8263 Read 8 Done New Done false
21748 7265 Acked 8264 Read 5 Done New Done false
21749 7266 Acked 8265 Read 5 Done New Done false
21750 7267 Acked 8266 Flush 0 Done New Done false
21751 7268 Acked 8267 Flush 0 Done New Done false
21752 7269 Acked 8268 Read 8 Done New Done false
21753 7270 Acked 8269 Write 9 Done New Done false
21754 7271 Acked 8270 Flush 0 Done New Done false
21755 7272 Acked 8271 Read 10 Done New Done false
21756 7273 Acked 8272 Flush 0 Done New Done false
21757 7274 Acked 8273 Write 5 Done New Done false
21758 7275 Acked 8274 Read 8 Done New Done false
21759 7276 Acked 8275 Write 10 Done New Done false
21760 7277 Acked 8276 Flush 0 Done New Done false
21761 7278 Acked 8277 Read 4 Done New Done false
21762 7279 Acked 8278 Read 10 Done New Done false
21763 7280 Acked 8279 Write 8 Done New Done false
21764 7281 Acked 8280 Write 2 Done New Done false
21765 7282 Acked 8281 Read 2 Done New Done false
21766 7283 Acked 8282 Write 5 Done New Done false
21767 7284 Acked 8283 Write 4 Done New Done false
21768 7285 Acked 8284 Read 5 Done New Done false
21769 7286 Acked 8285 Flush 0 Done New Done false
21770 7287 Acked 8286 Read 3 Done New Done false
21771 7288 Acked 8287 Write 3 Done New Done false
21772 7289 Acked 8288 Write 6 Done New Done false
21773 7290 Acked 8289 Write 8 Done New Done false
21774 7291 Acked 8290 Read 7 Done New Done false
21775 7292 Acked 8291 Write 7 Done New Done false
21776 7293 Acked 8292 Read 4 Done New Done false
21777 7294 Acked 8293 Write 7 Done New Done false
21778 7295 Acked 8294 Read 10 Done New Done false
21779 7296 Acked 8295 Write 10 Done New Done false
21780 7297 Acked 8296 Flush 0 Done New Done false
21781 7298 Acked 8297 Read 8 Done New Done false
21782 7299 Acked 8298 Flush 0 Done New Done false
21783 7300 Acked 8299 Read 6 Done New Done false
21784 7301 Acked 8300 Write 5 Done New Done false
21785 7302 Acked 8301 Write 1 Done New Done false
21786 7303 Acked 8302 Read 7 Done New Done false
21787 7304 Acked 8303 Flush 0 Done New Done false
21788 7305 Acked 8304 Read 10 Done New Done false
21789 7306 Acked 8305 Write 2 Done New Done false
21790 7307 Acked 8306 Read 5 Done New Done false
21791 7308 Acked 8307 Read 8 Done New Done false
21792 7309 Acked 8308 Flush 0 Done New Done false
21793 7310 Acked 8309 Read 6 Done New Done false
21794 7311 Acked 8310 Write 8 Done New Done false
21795 7312 Acked 8311 Read 7 Done New Done false
21796 7313 Acked 8312 Read 9 Done New Done false
21797 7314 Acked 8313 Write 8 Done New Done false
21798 7315 Acked 8314 Read 2 Done New Done false
21799 7316 Acked 8315 Read 10 Done New Done false
21800 7317 Acked 8316 Write 7 Done New Done false
21801 7318 Acked 8317 Read 2 Done New Done false
21802 7319 Acked 8318 Flush 0 Done New Done false
21803 7320 Acked 8319 Write 1 Done New Done false
21804 7321 Acked 8320 Read 5 Done New Done false
21805 7322 Acked 8321 Read 5 Done New Done false
21806 7323 Acked 8322 Write 8 Done New Done false
21807 7324 Acked 8323 Flush 0 Done New Done false
21808 7325 Acked 8324 Write 6 Done New Done false
21809 7326 Acked 8325 Read 1 Done New Done false
21810 7327 Acked 8326 Write 9 Done New Done false
21811 7328 Acked 8327 Read 10 Done New Done false
21812 7329 Acked 8328 Write 6 Done New Done false
21813 7330 Acked 8329 Read 8 Done New Done false
21814 7331 Acked 8330 Read 6 Done New Done false
21815 7332 Acked 8331 Flush 0 Done New Done false
21816 7333 Acked 8332 Read 2 Done New Done false
21817 7334 Acked 8333 Read 3 Done New Done false
21818 7335 Acked 8334 Read 5 Done New Done false
21819 7336 Acked 8335 Flush 0 Done New Done false
21820 7337 Acked 8336 Write 5 Done New Done false
21821 7338 Acked 8337 Write 9 Done New Done false
21822 7339 Acked 8338 Write 5 Done New Done false
21823 7340 Acked 8339 Write 5 Done New Done false
21824 7341 Acked 8340 Read 9 Done New Done false
21825 7342 Acked 8341 Flush 0 Done New Done false
21826 7343 Acked 8342 Write 7 Done New Done false
21827 7344 Acked 8343 Write 7 Done New Done false
21828 7345 Acked 8344 Flush 0 Done New Done false
21829 7346 Acked 8345 Write 2 Done New Done false
21830 7347 Acked 8346 Read 10 Done New Done false
21831 7348 Acked 8347 Read 5 Done New Done false
21832 7349 Acked 8348 Write 2 Done New Done false
21833 7350 Acked 8349 Read 7 Done New Done false
21834 7351 Acked 8350 Read 2 Done New Done false
21835 7352 Acked 8351 Read 6 Done New Done false
21836 7353 Acked 8352 Write 2 Done New Done false
21837 7354 Acked 8353 Read 9 Done New Done false
21838 7355 Acked 8354 Write 5 Done New Done false
21839 7356 Acked 8355 Write 7 Done New Done false
21840 7357 Acked 8356 Flush 0 Done New Done false
21841 7358 Acked 8357 Write 4 Done New Done false
21842 7359 Acked 8358 Flush 0 Done New Done false
21843 7360 Acked 8359 Write 6 Done New Done false
21844 7361 Acked 8360 Write 10 Done New Done false
21845 7362 Acked 8361 Flush 0 Done New Done false
21846 7363 Acked 8362 Write 6 Done New Done false
21847 7364 Acked 8363 Read 6 Done New Done false
21848 7365 Acked 8364 Read 5 Done New Done false
21849 7366 Acked 8365 Read 6 Done New Done false
21850 7367 Acked 8366 Write 4 Done New Done false
21851 7368 Acked 8367 Read 3 Done New Done false
21852 7369 Acked 8368 Read 7 Done New Done false
21853 7370 Acked 8369 Read 10 Done New Done false
21854 7371 Acked 8370 Flush 0 Done New Done false
21855 7372 Acked 8371 Read 1 Done New Done false
21856 7373 Acked 8372 Write 10 Done New Done false
21857 7374 Acked 8373 Flush 0 Done New Done false
21858 7375 Acked 8374 Read 4 Done New Done false
21859 7376 Acked 8375 Read 9 Done New Done false
21860 7377 Acked 8376 Read 10 Done New Done false
21861 7378 Acked 8377 Write 1 Done New Done false
21862 7379 Acked 8378 Read 4 Done New Done false
21863 7380 Acked 8379 Read 4 Done New Done false
21864 7381 Acked 8380 Read 3 Done New Done false
21865 7382 Acked 8381 Write 1 Done New Done false
21866 7383 Acked 8382 Write 4 Done New Done false
21867 7384 Acked 8383 Read 9 Done New Done false
21868 7385 Acked 8384 Read 1 Done New Done false
21869 7386 Acked 8385 Write 2 Done New Done false
21870 7387 Acked 8386 Read 7 Done New Done false
21871 7388 Acked 8387 Read 4 Done New Done false
21872 7389 Acked 8388 Read 3 Done New Done false
21873 7390 Acked 8389 Read 2 Done New Done false
21874 7391 Acked 8390 Flush 0 Done New Done false
21875 7392 Acked 8391 Read 10 Done New Done false
21876 7393 Acked 8392 Write 10 Done New Done false
21877 7394 Acked 8393 Write 10 Done New Done false
21878 7395 Acked 8394 Write 6 Done New Done false
21879 7396 Acked 8395 Write 2 Done New Done false
21880 7397 Acked 8396 Read 4 Done New Done false
21881 7398 Acked 8397 Read 7 Done New Done false
21882 7399 Acked 8398 Read 1 Done New Done false
21883 7400 Acked 8399 Read 4 Done New Done false
21884 7401 Acked 8400 Write 4 Done New Done false
21885 7402 Acked 8401 Write 4 Done New Done false
21886 7403 Acked 8402 Write 8 Done New Done false
21887 7404 Acked 8403 Read 3 Done New Done false
21888 7405 Acked 8404 Write 2 Done New Done false
21889 7406 Acked 8405 Write 5 Done New Done false
21890 7407 Acked 8406 Read 2 Done New Done false
21891 7408 Acked 8407 Read 1 Done New Done false
21892 7409 Acked 8408 Write 10 Done New Done false
21893 7410 Acked 8409 Read 4 Done New Done false
21894 7411 Acked 8410 Read 5 Done New Done false
21895 7412 Acked 8411 Write 3 Done New Done false
21896 7413 Acked 8412 Read 9 Done New Done false
21897 7414 Acked 8413 Write 5 Done New Done false
21898 7415 Acked 8414 Read 1 Done New Done false
21899 7416 Acked 8415 Write 9 Done New Done false
21900 7417 Acked 8416 Write 3 Done New Done false
21901 7418 Acked 8417 Write 1 Done New Done false
21902 7419 Acked 8418 Read 10 Done New Done false
21903 7420 Acked 8419 Write 7 Done New Done false
21904 7421 Acked 8420 Read 9 Done New Done false
21905 7422 Acked 8421 Read 3 Done New Done false
21906 7423 Acked 8422 Read 3 Done New Done false
21907 7424 Acked 8423 Read 4 Done New Done false
21908 7425 Acked 8424 Write 3 Done New Done false
21909 7426 Acked 8425 Read 4 Done New Done false
21910 7427 Acked 8426 Flush 0 Done New Done false
21911 7428 Acked 8427 Write 5 Done New Done false
21912 7429 Acked 8428 Read 10 Done New Done false
21913 7430 Acked 8429 Write 2 Done New Done false
21914 7431 Acked 8430 Write 7 Done New Done false
21915 7432 Acked 8431 Write 1 Done New Done false
21916 7433 Acked 8432 Read 8 Done New Done false
21917 7434 Acked 8433 Write 4 Done New Done false
21918 7435 Acked 8434 Write 6 Done New Done false
21919 7436 Acked 8435 Read 8 Done New Done false
21920 7437 Acked 8436 Flush 0 Done New Done false
21921 7438 Acked 8437 Write 5 Done New Done false
21922 7439 Acked 8438 Read 1 Done New Done false
21923 7440 Acked 8439 Read 2 Done New Done false
21924 7441 Acked 8440 Write 8 Done New Done false
21925 7442 Acked 8441 Read 4 Done New Done false
21926 7443 Acked 8442 Read 9 Done New Done false
21927 7444 Acked 8443 Read 2 Done New Done false
21928 7445 Acked 8444 Flush 0 Done New Done false
21929 7446 Acked 8445 Flush 0 Done New Done false
21930 7447 Acked 8446 Write 2 Done New Done false
21931 7448 Acked 8447 Write 4 Done New Done false
21932 7449 Acked 8448 Read 1 Done New Done false
21933 7450 Acked 8449 Write 2 Done New Done false
21934 7451 Acked 8450 Read 5 Done New Done false
21935 7452 Acked 8451 Read 6 Done New Done false
21936 7453 Acked 8452 Flush 0 Done New Done false
21937 7454 Acked 8453 Read 7 Done New Done false
21938 7455 Acked 8454 Write 10 Done New Done false
21939 7456 Acked 8455 Read 2 Done New Done false
21940 7457 Acked 8456 Read 3 Done New Done false
21941 7458 Acked 8457 Read 10 Done New Done false
21942 7459 Acked 8458 Write 9 Done New Done false
21943 7460 Acked 8459 Flush 0 Done New Done false
21944 7461 Acked 8460 Read 2 Done New Done false
21945 7462 Acked 8461 Read 3 Done New Done false
21946 7463 Acked 8462 Write 6 Done New Done false
21947 7464 Acked 8463 Write 1 Done New Done false
21948 7465 Acked 8464 Write 8 Done New Done false
21949 7466 Acked 8465 Write 9 Done New Done false
21950 7467 Acked 8466 Flush 0 Done New Done false
21951 7468 Acked 8467 Flush 0 Done New Done false
21952 7469 Acked 8468 Write 3 Done New Done false
21953 7470 Acked 8469 Read 8 Done New Done false
21954 7471 Acked 8470 Read 10 Done New Done false
21955 7472 Acked 8471 Write 4 Done New Done false
21956 7473 Acked 8472 Read 2 Done New Done false
21957 7474 Acked 8473 Write 9 Done New Done false
21958 7475 Acked 8474 Flush 0 Done New Done false
21959 7476 Acked 8475 Read 7 Done New Done false
21960 7477 Acked 8476 Flush 0 Done New Done false
21961 7478 Acked 8477 Flush 0 Done New Done false
21962 7479 Acked 8478 Write 4 Done New Done false
21963 7480 Acked 8479 Flush 0 Done New Done false
21964 7481 Acked 8480 Read 1 Done New Done false
21965 7482 Acked 8481 Write 8 Done New Done false
21966 7483 Acked 8482 Read 2 Done New Done false
21967 7484 Acked 8483 Read 7 Done New Done false
21968 7485 Acked 8484 Flush 0 Done New Done false
21969 7486 Acked 8485 Write 10 Done New Done false
21970 7487 Acked 8486 Write 10 Done New Done false
21971 7488 Acked 8487 Write 2 Done New Done false
21972 7489 Acked 8488 Write 2 Done New Done false
21973 7490 Acked 8489 Flush 0 Done New Done false
21974 7491 Acked 8490 Read 2 Done New Done false
21975 7492 Acked 8491 Write 5 Done New Done false
21976 7493 Acked 8492 Read 4 Done New Done false
21977 7494 Acked 8493 Read 6 Done New Done false
21978 7495 Acked 8494 Write 4 Done New Done false
21979 7496 Acked 8495 Flush 0 Done New Done false
21980 7497 Acked 8496 Read 3 Done New Done false
21981 7498 Acked 8497 Write 1 Done New Done false
21982 7499 Acked 8498 Write 2 Done New Done false
21983 7500 Acked 8499 Write 6 Done New Done false
21984 7501 Acked 8500 Read 7 Done New Done false
21985 7502 Acked 8501 Read 3 Done New Done false
21986 7503 Acked 8502 Read 4 Done New Done false
21987 7504 Acked 8503 Read 7 Done New Done false
21988 7505 Acked 8504 Read 3 Done New Done false
21989 7506 Acked 8505 Write 2 Done New Done false
21990 7507 Acked 8506 Write 7 Done New Done false
21991 7508 Acked 8507 Flush 0 Done New Done false
21992 7509 Acked 8508 Read 9 Done New Done false
21993 7510 Acked 8509 Write 1 Done New Done false
21994 7511 Acked 8510 Write 2 Done New Done false
21995 7512 Acked 8511 Read 2 Done New Done false
21996 7513 Acked 8512 Write 8 Done New Done false
21997 7514 Acked 8513 Read 10 Done New Done false
21998 7515 Acked 8514 Write 4 Done New Done false
21999 7516 Acked 8515 Write 4 Done New Done false
22000 7517 Acked 8516 Write 8 Done New Done false
22001 7518 Acked 8517 Read 6 Done New Done false
22002 7519 Acked 8518 Write 9 Done New Done false
22003 7520 Acked 8519 Read 9 Done New Done false
22004 7521 Acked 8520 Write 2 Done New Done false
22005 7522 Acked 8521 Read 2 Done New Done false
22006 7523 Acked 8522 Flush 0 Done New Done false
22007 7524 Acked 8523 Write 2 Done New Done false
22008 7525 Acked 8524 Read 7 Done New Done false
22009 7526 Acked 8525 Read 4 Done New Done false
22010 7527 Acked 8526 Write 2 Done New Done false
22011 7528 Acked 8527 Read 2 Done New Done false
22012 7529 Acked 8528 Read 1 Done New Done false
22013 7530 Acked 8529 Read 3 Done New Done false
22014 7531 Acked 8530 Read 4 Done New Done false
22015 7532 Acked 8531 Write 8 Done New Done false
22016 7533 Acked 8532 Read 5 Done New Done false
22017 7534 Acked 8533 Write 4 Done New Done false
22018 7535 Acked 8534 Write 1 Done New Done false
22019 7536 Acked 8535 Flush 0 Done New Done false
22020 STATES DS:0 DS:1 DS:2 TOTAL
22021 New 0 301 0 301
22022 Sent 0 0 0 0
22023 Done 301 0 301 602
22024 Skipped 0 0 0 0
22025 Error 0 0 0 0
22026 Last Flush: 8535 8234 8535
22027 Downstairs last five completed: 8234 8233 8232 8231 8230
22028 Upstairs last five completed: 7536 7535 7534 7533 7532
22029 CLIENT: Up:0 ds:301 act:2
220302024-03-02T01:09:59.338ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220312024-03-02T01:09:59.339ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220322024-03-02T01:09:59.339ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220332024-03-02T01:09:59.339ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220342024-03-02T01:09:59.339ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220352024-03-02T01:09:59.339ZINFOcrucible: [1] client re-new 301 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220362024-03-02T01:09:59.339ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22037 [8820] got start action
22038 [8820] I am going to restart
22039 Starting downstairs at port 8820
22040 Make output file at "/tmp/dsc/downstairs-8820.txt"
22041 [8820][1] reports Starting
220422024-03-02T01:10:00.340ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220432024-03-02T01:10:00.340ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220442024-03-02T01:10:00.340ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220452024-03-02T01:10:00.340ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220462024-03-02T01:10:00.340ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220472024-03-02T01:10:00.340ZINFOcrucible: [1] client re-new 301 jobs since flush 8234 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220482024-03-02T01:10:00.341ZINFOcrucible: client task is exiting = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22049 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1187)
22050 [8820][1] reports Running
220512024-03-02T01:10:01.342ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220522024-03-02T01:10:01.342ZINFOcrucible: ds_connection connected = io task client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220532024-03-02T01:10:01.344ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220542024-03-02T01:10:01.344ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220552024-03-02T01:10:01.344ZINFOcrucible: send last flush ID to this DS: 8234 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220562024-03-02T01:10:01.344ZINFOcrucible: Replied this last flush ID: 8234 = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220572024-03-02T01:10:01.344ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220582024-03-02T01:10:01.344ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220592024-03-02T01:10:01.344ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220602024-03-02T01:10:01.344ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22061 ----------------------------------------------------------------
22062 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22063 Downstairs last five completed: 8535 8534 8533 8532 8531
22064 Upstairs last five completed: 7536 7535 7534 7533 7532
22065 CLIENT: Up:0 ds:0 act:3
22066 Replay: All jobs finished, all DS active.
22067 Main task has work to do, go find it
22068 got dsc Stop(0)
22069 stop 0
22070 Mar 02 01:10:02.689 INFO request completed, latency_us: 382, response_code: 204, uri: /stop/cid/0, method: POST, req_id: be5d376d-fd6c-4109-911f-dd734c3beb61, remote_addr: 127.0.0.1[8810] Got stop action so:false kr:false
22071 :40735, local_addr: 127.0.0.1:9998
220722024-03-02T01:10:02.689ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220732024-03-02T01:10:02.690ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220742024-03-02T01:10:02.690ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220752024-03-02T01:10:02.690ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22076 Mar 02 01:10:02.690 INFO request completed, latency_us: 138, response_code: 200, uri{"msg":"[0] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:10:02.690100177Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
22077 : /state/cid/0, method: GET, req_id: 0a2fdb3c-6a65-470c-9086-409faf77e895, remote_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:10:02.690169093Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
22078 .0.1:40735, local_addr: 127.0.0.1:9998
22079 [8810] Exited with: ExitStatus(unix_wait_status(9))
22080 [8810][0] reports Exit
220812024-03-02T01:10:03.691ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220822024-03-02T01:10:03.691ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220832024-03-02T01:10:03.691ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220842024-03-02T01:10:03.691ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220852024-03-02T01:10:03.691ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220862024-03-02T01:10:03.691ZINFOcrucible: [0] client re-new 0 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220872024-03-02T01:10:03.691ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220882024-03-02T01:10:04.692ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220892024-03-02T01:10:04.692ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220902024-03-02T01:10:04.692ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220912024-03-02T01:10:04.692ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220922024-03-02T01:10:04.692ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220932024-03-02T01:10:04.692ZINFOcrucible: [0] client re-new 0 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220942024-03-02T01:10:04.692ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220952024-03-02T01:10:05.694ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220962024-03-02T01:10:05.694ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220972024-03-02T01:10:05.694ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220982024-03-02T01:10:05.694ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
220992024-03-02T01:10:05.694ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221002024-03-02T01:10:05.694ZINFOcrucible: [0] client re-new 0 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221012024-03-02T01:10:05.694ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22102 Mar 02 01:10:06.691 INFO request completed, latency_us: 184, response_code: 200, uri: /state/cid/0, method: GET, req_id: a4158654-4657-414e-8371-1312e3509ab6, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
22103 001/300 Write block 1791 len 8192 data: 4 4
22104 002/300 Read block 719 len 20480
22105 003/300 Read block 1386 len 24576
22106 004/300 Write block 1318 len 32768 data: 4 4 5 5 6 4 5 4
221072024-03-02T01:10:06.695ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221082024-03-02T01:10:06.695ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221092024-03-02T01:10:06.695ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221102024-03-02T01:10:06.695ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221112024-03-02T01:10:06.695ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221122024-03-02T01:10:06.695ZINFOcrucible: [0] client re-new 3 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
221132024-03-02T01:10:06.696ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22114 005/300 Read block 3957 len 12288
22115 006/300 Write block 4775 len 4096 data: 5
22116 007/300 Flush
22117 008/300 Read block 1326 len 20480
22118 009/300 Read block 1459 len 40960
22119 010/300 Flush
22120 011/300 Read block 1554 len 12288
22121 012/300 Read block 3935 len 32768
22122 013/300 Read block 489 len 28672
22123 014/300 Read block 4809 len 36864
22124 015/300 Write block 855 len 40960 data: 3 5 6 7 8 7 6 5 6 5
22125 016/300 Write block 2635 len 32768 data: 6 7 7 9 8 5 5 5
22126 017/300 Flush
22127 018/300 Read block 3405 len 36864
22128 019/300 Flush
22129 020/300 Write block 493 len 12288 data: 5 4 5
22130 021/300 Read block 2484 len 24576
22131 022/300 Read block 988 len 4096
22132 023/300 Read block 848 len 4096
22133 024/300 Read block 4443 len 36864
22134 025/300 Read block 1245 len 4096
22135 026/300 Write block 2140 len 28672 data: 5 4 4 5 5 6 6
22136 027/300 Read block 4688 len 40960
22137 028/300 Write block 1591 len 32768 data: 5 5 2 2 2 2 2 3
22138 029/300 Write block 4419 len 8192 data: 6 5
22139 030/300 Read block 3965 len 12288
22140 031/300 Read block 2662 len 40960
22141 032/300 Write block 3737 len 28672 data: 6 3 2 2 2 2 4
22142 033/300 Write block 4954 len 40960 data: 8 6 4 4 4 3 3 4 4 4
22143 034/300 Read block 236 len 32768
22144 035/300 Read block 1531 len 4096
22145 036/300 Read block 963 len 28672
22146 037/300 Write block 1279 len 20480 data: 5 5 5 3 3
22147 038/300 Write block 4779 len 28672 data: 2 2 4 3 5 5 6
22148 039/300 Read block 2372 len 4096
22149 040/300 Read block 3552 len 12288
22150 041/300 Read block 4889 len 40960
22151 042/300 Read block 1928 len 20480
22152 043/300 Write block 4569 len 40960 data: 5 4 5 4 4 6 6 7 6 6
22153 044/300 Write block 525 len 4096 data: 5
22154 045/300 Read block 1786 len 24576
22155 046/300 Read block 1545 len 24576
22156 047/300 Write block 1946 len 8192 data: 6 6
22157 048/300 Read block 3918 len 28672
22158 049/300 Read block 4376 len 32768
22159 050/300 Write block 4018 len 28672 data: 5 5 5 3 4 4 5
22160 051/300 Flush
22161 052/300 Read block 1627 len 32768
22162 053/300 Read block 453 len 32768
22163 054/300 Write block 1655 len 4096 data: 6
22164 055/300 Flush
22165 056/300 Write block 471 len 20480 data: 7 6 6 6 5
22166 057/300 Read block 1820 len 12288
22167 058/300 Write block 752 len 20480 data: 7 5 5 5 6
22168 059/300 Read block 4019 len 40960
22169 060/300 Flush
22170 061/300 Write block 579 len 4096 data: 6
22171 062/300 Write block 1372 len 12288 data: 3 4 5
22172 063/300 Write block 2966 len 32768 data: 5 5 5 4 5 5 6 4
22173 064/300 Write block 1017 len 12288 data: 6 8 8
22174 065/300 Read block 2119 len 8192
22175 066/300 Flush
22176 067/300 Write block 164 len 28672 data: 5 5 5 6 7 6 4
22177 068/300 Write block 4062 len 40960 data: 6 6 5 5 5 5 4 6 6 5
22178 069/300 Read block 4313 len 4096
22179 070/300 Read block 1121 len 4096
22180 071/300 Read block 2699 len 32768
22181 072/300 Write block 4371 len 36864 data: 4 5 4 4 4 3 4 4 4
22182 073/300 Read block 47 len 8192
22183 074/300 Write block 4319 len 12288 data: 5 5 5
22184 075/300 Read block 232 len 4096
22185 076/300 Read block 3297 len 32768
22186 077/300 Read block 3196 len 28672
22187 078/300 Write block 4080 len 4096 data: 5
22188 079/300 Read block 600 len 36864
22189 080/300 Flush
22190 081/300 Write block 2950 len 40960 data: 8 7 7 8 7 8 7 8 7 8
22191 082/300 Write block 4601 len 36864 data: 8 10 10 7 5 4 3 3 2
22192 083/300 Read block 1172 len 20480
22193 084/300 Write block 969 len 4096 data: 3
22194 085/300 Write block 3538 len 36864 data: 6 6 5 4 5 4 5 4 4
22195 086/300 Read block 737 len 28672
22196 087/300 Read block 4460 len 24576
22197 088/300 Write block 4518 len 8192 data: 5 4
22198 089/300 Write block 1538 len 8192 data: 3 2
22199 090/300 Read block 2214 len 12288
22200 091/300 Flush
22201 092/300 Read block 4189 len 28672
22202 093/300 Write block 1842 len 36864 data: 5 4 4 4 5 5 5 6 5
22203 094/300 Read block 1495 len 12288
22204 095/300 Read block 3654 len 40960
22205 096/300 Read block 4537 len 40960
22206 097/300 Read block 2497 len 32768
22207 098/300 Write block 1232 len 36864 data: 6 5 6 6 6 6 6 4 3
22208 099/300 Read block 3921 len 36864
22209 100/300 Read block 244 len 40960
22210 101/300 Read block 2912 len 28672
22211 102/300 Write block 3552 len 4096 data: 9
22212 103/300 Write block 4660 len 28672 data: 9 9 7 4 4 5 4
22213 104/300 Read block 83 len 32768
22214 105/300 Read block 370 len 4096
22215 106/300 Read block 4286 len 36864
22216 107/300 Write block 3231 len 32768 data: 7 6 7 8 9 9 9 9
22217 108/300 Read block 2059 len 28672
22218 109/300 Write block 3900 len 4096 data: 6
22219 110/300 Read block 1626 len 28672
22220 111/300 Write block 1832 len 24576 data: 5 7 7 7 5 7
22221 112/300 Read block 1459 len 24576
22222 113/300 Read block 2797 len 12288
22223 114/300 Write block 36 len 8192 data: 4 4
22224 115/300 Flush
22225 116/300 Read block 3433 len 28672
22226 117/300 Flush
22227 118/300 Write block 2296 len 28672 data: 3 3 4 4 4 4 3
22228 119/300 Read block 447 len 20480
22229 120/300 Write block 3577 len 12288 data: 5 4 4
22230 121/300 Read block 615 len 20480
22231 122/300 Read block 2942 len 12288
22232 123/300 Write block 2007 len 12288 data: 5 4 4
22233 124/300 Read block 1722 len 8192
22234 125/300 Flush
22235 126/300 Read block 4623 len 20480
22236 127/300 Write block 4513 len 16384 data: 3 4 6 6
22237 128/300 Read block 4304 len 36864
22238 129/300 Write block 1025 len 20480 data: 4 4 6 6 7
22239 130/300 Write block 4366 len 4096 data: 6
22240 131/300 Write block 3511 len 36864 data: 6 6 8 9 10 8 5 4 5
22241 132/300 Read block 4404 len 24576
22242 133/300 Read block 2404 len 20480
22243 134/300 Read block 3673 len 32768
22244 135/300 Flush
22245 136/300 Read block 4714 len 40960
22246 137/300 Read block 1777 len 32768
22247 138/300 Read block 3484 len 4096
22248 139/300 Write block 1945 len 12288 data: 7 7 7
22249 140/300 Write block 544 len 16384 data: 8 6 6 6
22250 141/300 Read block 3419 len 32768
22251 142/300 Read block 3820 len 16384
22252 143/300 Read block 1285 len 32768
22253 144/300 Read block 1625 len 8192
22254 145/300 Write block 1001 len 16384 data: 8 7 6 4
22255 146/300 Read block 4186 len 8192
22256 147/300 Read block 809 len 8192
22257 148/300 Read block 2072 len 12288
22258 149/300 Write block 4411 len 8192 data: 6 5
22259 150/300 Write block 3844 len 24576 data: 4 5 4 3 5 5
22260 151/300 Flush
22261 152/300 Write block 3103 len 16384 data: 2 3 3 4
22262 153/300 Read block 2873 len 24576
22263 154/300 Flush
22264 155/300 Read block 3342 len 4096
22265 156/300 Write block 694 len 16384 data: 7 7 8 7
22266 157/300 Write block 2129 len 20480 data: 5 4 5 6 5
22267 158/300 Read block 3394 len 36864
22268 159/300 Read block 3926 len 28672
22269 160/300 Read block 2839 len 8192
22270 161/300 Write block 4818 len 20480 data: 3 3 4 3 3
22271 162/300 Read block 4986 len 8192
22272 163/300 Flush
22273 164/300 Flush
22274 165/300 Write block 3151 len 36864 data: 6 6 6 7 6 5 6 6 6
22275 166/300 Read block 1813 len 12288
22276 167/300 Write block 3234 len 16384 data: 9 10 10 10
22277 168/300 Write block 1480 len 32768 data: 4 3 3 2 2 3 2 2
22278 169/300 Read block 1557 len 4096
22279 170/300 Write block 1995 len 24576 data: 5 4 3 4 3 3
22280 171/300 Flush
22281 172/300 Write block 2166 len 8192 data: 5 5
22282 173/300 Read block 90 len 8192
22283 174/300 Write block 347 len 36864 data: 9 7 7 7 6 6 5 4 3
22284 175/300 Read block 3974 len 24576
22285 176/300 Read block 4328 len 40960
22286 177/300 Flush
22287 178/300 Read block 105 len 24576
22288 179/300 Read block 3186 len 40960
22289 180/300 Write block 1115 len 12288 data: 7 6 6
22290 181/300 Write block 2640 len 28672 data: 6 6 6 5 4 4 4
22291 182/300 Read block 3658 len 8192
22292 183/300 Write block 769 len 36864 data: 4 4 5 8 7 6 5 5 5
22293 184/300 Write block 2419 len 4096 data: 4
22294 185/300 Read block 1761 len 4096
22295 186/300 Write block 4326 len 12288 data: 6 6 6
22296 187/300 Write block 2252 len 28672 data: 10 7 6 5 5 5 6
22297 188/300 Read block 1606 len 8192
22298 189/300 Flush
22299 190/300 Write block 1673 len 36864 data: 3 5 4 3 4 4 6 6 7
22300 191/300 Write block 576 len 4096 data: 6
22301 192/300 Read block 2765 len 8192
22302 193/300 Read block 1000 len 32768
22303 194/300 Write block 4557 len 16384 data: 4 4 5 5
22304 195/300 Flush
22305 196/300 Write block 939 len 4096 data: 7
22306 197/300 Read block 151 len 32768
22307 198/300 Write block 849 len 24576 data: 2 3 4 4 4 3
22308 199/300 Read block 3556 len 36864
22309 200/300 Read block 258 len 32768
22310 201/300 Write block 946 len 8192 data: 6 6
22311 202/300 Read block 1089 len 4096
22312 203/300 Write block 1764 len 28672 data: 3 3 3 3 3 4 6
22313 204/300 Read block 2719 len 40960
22314 205/300 Write block 583 len 16384 data: 4 4 5 4
22315 206/300 Read block 1842 len 4096
22316 207/300 Read block 3775 len 16384
22317 208/300 Write block 1043 len 40960 data: 8 8 7 6 6 8 7 8 8 7
22318 209/300 Read block 860 len 12288
22319 210/300 Write block 677 len 12288 data: 5 5 4
22320 211/300 Read block 2345 len 36864
22321 212/300 Read block 4211 len 40960
22322 213/300 Write block 1603 len 24576 data: 5 5 6 6 7 8
22323 214/300 Write block 1997 len 28672 data: 4 5 4 4 4 4 4
22324 215/300 Read block 2166 len 28672
22325 216/300 Flush
22326 217/300 Read block 4743 len 36864
22327 218/300 Read block 260 len 36864
22328 219/300 Read block 2878 len 24576
22329 220/300 Read block 989 len 32768
22330 221/300 Read block 840 len 12288
22331 222/300 Write block 377 len 8192 data: 4 6
22332 223/300 Write block 4992 len 12288 data: 9 7 6
22333 224/300 Write block 1498 len 12288 data: 5 3 3
22334 225/300 Read block 3581 len 12288
22335 226/300 Read block 3257 len 16384
22336 227/300 Write block 2723 len 16384 data: 7 6 5 5
22337 228/300 Flush
22338 229/300 Read block 2725 len 40960
22339 230/300 Flush
22340 231/300 Write block 3310 len 8192 data: 3 3
22341 232/300 Read block 589 len 4096
22342 233/300 Read block 3358 len 24576
22343 234/300 Read block 3279 len 20480
22344 235/300 Read block 1165 len 28672
22345 236/300 Read block 4501 len 8192
22346 237/300 Read block 2419 len 24576
22347 238/300 Read block 527 len 24576
22348 239/300 Write block 2068 len 28672 data: 9 9 9 8 7 7 4
22349 240/300 Read block 3679 len 40960
22350 241/300 Write block 4253 len 32768 data: 4 3 3 4 3 2 3 3
22351 242/300 Write block 2759 len 36864 data: 4 6 7 5 3 3 4 5 6
22352 243/300 Write block 1513 len 16384 data: 4 6 6 6
22353 244/300 Write block 3342 len 40960 data: 7 7 8 7 5 5 7 7 6 6
22354 245/300 Read block 3510 len 36864
22355 246/300 Read block 2798 len 4096
22356 247/300 Flush
22357 248/300 Write block 581 len 28672 data: 3 4 5 5 6 5 5
22358 249/300 Read block 4268 len 12288
22359 250/300 Read block 3503 len 40960
22360 251/300 Write block 1752 len 20480 data: 5 4 5 5 5
22361 252/300 Write block 2157 len 12288 data: 3 4 5
22362 253/300 Read block 3974 len 24576
22363 254/300 Write block 2996 len 36864 data: 6 5 5 4 5 3 3 5 6
22364 255/300 Write block 3449 len 8192 data: 8 8
22365 256/300 Write block 766 len 16384 data: 4 4 4 5
22366 257/300 Write block 4102 len 20480 data: 5 4 4 3 3
22367 258/300 Flush
22368 259/300 Write block 3592 len 20480 data: 7 5 6 6 6
22369 260/300 Read block 1481 len 16384
22370 261/300 Read block 2337 len 36864
22371 262/300 Write block 3642 len 8192 data: 5 6
22372 263/300 Write block 72 len 8192 data: 5 6
22373 264/300 Write block 959 len 36864 data: 4 5 6 6 6 5 7 6 4
22374 265/300 Write block 4983 len 20480 data: 6 6 6 8 8
22375 266/300 Read block 1562 len 40960
22376 267/300 Write block 3631 len 36864 data: 6 5 4 5 4 4 3 3 4
22377 268/300 Write block 3330 len 12288 data: 9 8 7
22378 269/300 Read block 4478 len 32768
22379 270/300 Flush
22380 271/300 Write block 3849 len 16384 data: 6 5 6 7
22381 272/300 Write block 344 len 36864 data: 9 9 10 10 8 8 8 7 7
22382 273/300 Read block 4462 len 8192
22383 274/300 Flush
22384 275/300 Write block 74 len 8192 data: 6 6
22385 276/300 Flush
22386 277/300 Read block 4346 len 20480
22387 278/300 Read block 1240 len 16384
22388 279/300 Write block 4455 len 36864 data: 5 5 5 4 4 4 2 3 5
22389 280/300 Write block 4627 len 36864 data: 3 4 4 4 4 5 4 3 2
22390 281/300 Read block 605 len 20480
22391 282/300 Write block 2149 len 24576 data: 6 3 3 3 4 4
22392 283/300 Read block 4517 len 24576
22393 284/300 Write block 1020 len 4096 data: 7
22394 285/300 Write block 2581 len 8192 data: 6 5
22395 286/300 Write block 3756 len 32768 data: 6 7 7 5 7 6 5 4
22396 287/300 Read block 2261 len 24576
22397 288/300 Read block 4709 len 28672
22398 289/300 Write block 1594 len 28672 data: 3 3 3 3 4 4 4
22399 290/300 Read block 249 len 4096
22400 291/300 Read block 2510 len 12288
22401 292/300 Flush
22402 293/300 Write block 4417 len 16384 data: 6 5 7 6
22403 294/300 Write block 3446 len 32768 data: 10 9 9 9 9 7 8 7
22404 295/300 Write block 1669 len 16384 data: 3 3 3 4
22405 296/300 Read block 253 len 36864
22406 297/300 Read block 2768 len 40960
22407 298/300 Read block 1830 len 32768
22408 299/300 Write block 4736 len 8192 data: 7 7
22409 300/300 Write block 2368 len 8192 data: 6 6
22410 Main task has work to do, go find it
22411 got dsc Start(0)
22412 start 0
22413 Mar 02 01:10:07.041 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/0, method: POST, req_id: 33831850-1680-4e16-812e-bc8a1c9e8d7e, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
22414 Replay: started 0, returned:Ok(())
22415 ----------------------------------------------------------------
22416 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
22417 States: Offline Active Active
22418 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22419 7537 Acked 8536 Write 2 New Done Done true
22420 7538 Acked 8537 Read 5 New Done Done true
22421 7539 Acked 8538 Read 6 New Done Done true
22422 7540 Acked 8539 Write 8 New Done Done false
22423 7541 Acked 8540 Read 3 New Done Done false
22424 7542 Acked 8541 Write 1 New Done Done false
22425 7543 Acked 8542 Flush 0 New Done Done false
22426 7544 Acked 8543 Read 5 New Done Done false
22427 7545 Acked 8544 Read 10 New Done Done false
22428 7546 Acked 8545 Flush 0 New Done Done false
22429 7547 Acked 8546 Read 3 New Done Done false
22430 7548 Acked 8547 Read 8 New Done Done false
22431 7549 Acked 8548 Read 7 New Done Done false
22432 7550 Acked 8549 Read 9 New Done Done false
22433 7551 Acked 8550 Write 10 New Done Done false
22434 7552 Acked 8551 Write 8 New Done Done false
22435 7553 Acked 8552 Flush 0 New Done Done false
22436 7554 Acked 8553 Read 9 New Done Done false
22437 7555 Acked 8554 Flush 0 New Done Done false
22438 7556 Acked 8555 Write 3 New Done Done false
22439 7557 Acked 8556 Read 6 New Done Done false
22440 7558 Acked 8557 Read 1 New Done Done false
22441 7559 Acked 8558 Read 1 New Done Done false
22442 7560 Acked 8559 Read 9 New Done Done false
22443 7561 Acked 8560 Read 1 New Done Done false
22444 7562 Acked 8561 Write 7 New Done Done false
22445 7563 Acked 8562 Read 10 New Done Done false
22446 7564 Acked 8563 Write 8 New Done Done false
22447 7565 Acked 8564 Write 2 New Done Done false
22448 7566 Acked 8565 Read 3 New Done Done false
22449 7567 Acked 8566 Read 10 New Done Done false
22450 7568 Acked 8567 Write 7 New Done Done false
22451 7569 Acked 8568 Write 10 New Done Done false
22452 7570 Acked 8569 Read 8 New Done Done false
22453 7571 Acked 8570 Read 1 New Done Done false
22454 7572 Acked 8571 Read 7 New Done Done false
22455 7573 Acked 8572 Write 5 New Done Done false
22456 7574 Acked 8573 Write 7 New Done Done false
22457 7575 Acked 8574 Read 1 New Done Done false
22458 7576 Acked 8575 Read 3 New Done Done false
22459 7577 Acked 8576 Read 10 New Done Done false
22460 7578 Acked 8577 Read 5 New Done Done false
22461 7579 Acked 8578 Write 10 New Done Done false
22462 7580 Acked 8579 Write 1 New Done Done false
22463 7581 Acked 8580 Read 6 New Done Done false
22464 7582 Acked 8581 Read 6 New Done Done false
22465 7583 Acked 8582 Write 2 New Done Done false
22466 7584 Acked 8583 Read 7 New Done Done false
22467 7585 Acked 8584 Read 8 New Done Done false
22468 7586 Acked 8585 Write 7 New Done Done false
22469 7587 Acked 8586 Flush 0 New Done Done false
22470 7588 Acked 8587 Read 8 New Done Done false
22471 7589 Acked 8588 Read 8 New Done Done false
22472 7590 Acked 8589 Write 1 New Done Done false
22473 7591 Acked 8590 Flush 0 New Done Done false
22474 7592 Acked 8591 Write 5 New Done Done false
22475 7593 Acked 8592 Read 3 New Done Done false
22476 7594 Acked 8593 Write 5 New Done Done false
22477 7595 Acked 8594 Read 10 New Done Done false
22478 7596 Acked 8595 Flush 0 New Done Done false
22479 7597 Acked 8596 Write 1 New Done Done false
22480 7598 Acked 8597 Write 3 New Done Done false
22481 7599 Acked 8598 Write 8 New Done Done false
22482 7600 Acked 8599 Write 3 New Done Done false
22483 7601 Acked 8600 Read 2 New Done Done false
22484 7602 Acked 8601 Flush 0 New Done Done false
22485 7603 Acked 8602 Write 7 New Done Done false
22486 7604 Acked 8603 Write 10 New Done Done false
22487 7605 Acked 8604 Read 1 New Done Done false
22488 7606 Acked 8605 Read 1 New Done Done false
22489 7607 Acked 8606 Read 8 New Done Done false
22490 7608 Acked 8607 Write 9 New Done Done false
22491 7609 Acked 8608 Read 2 New Done Done false
22492 7610 Acked 8609 Write 3 New Done Done false
22493 7611 Acked 8610 Read 1 New Done Done false
22494 7612 Acked 8611 Read 8 New Done Done false
22495 7613 Acked 8612 Read 7 New Done Done false
22496 7614 Acked 8613 Write 1 New Done Done false
22497 7615 Acked 8614 Read 9 New Done Done false
22498 7616 Acked 8615 Flush 0 New Done Done false
22499 7617 Acked 8616 Write 10 New Done Done false
22500 7618 Acked 8617 Write 9 New Done Done false
22501 7619 Acked 8618 Read 5 New Done Done false
22502 7620 Acked 8619 Write 1 New Done Done false
22503 7621 Acked 8620 Write 9 New Done Done false
22504 7622 Acked 8621 Read 7 New Done Done false
22505 7623 Acked 8622 Read 6 New Done Done false
22506 7624 Acked 8623 Write 2 New Done Done false
22507 7625 Acked 8624 Write 2 New Done Done false
22508 7626 Acked 8625 Read 3 New Done Done false
22509 7627 Acked 8626 Flush 0 New Done Done false
22510 7628 Acked 8627 Read 7 New Done Done false
22511 7629 Acked 8628 Write 9 New Done Done false
22512 7630 Acked 8629 Read 3 New Done Done false
22513 7631 Acked 8630 Read 10 New Done Done false
22514 7632 Acked 8631 Read 10 New Done Done false
22515 7633 Acked 8632 Read 8 New Done Done false
22516 7634 Acked 8633 Write 9 New Done Done false
22517 7635 Acked 8634 Read 9 New Done Done false
22518 7636 Acked 8635 Read 10 New Done Done false
22519 7637 Acked 8636 Read 7 New Done Done false
22520 7638 Acked 8637 Write 1 New Done Done false
22521 7639 Acked 8638 Write 7 New Done Done false
22522 7640 Acked 8639 Read 8 New Done Done false
22523 7641 Acked 8640 Read 1 New Done Done false
22524 7642 Acked 8641 Read 9 New Done Done false
22525 7643 Acked 8642 Write 8 New Done Done false
22526 7644 Acked 8643 Read 7 New Done Done false
22527 7645 Acked 8644 Write 1 New Done Done false
22528 7646 Acked 8645 Read 7 New Done Done false
22529 7647 Acked 8646 Write 6 New Done Done false
22530 7648 Acked 8647 Read 6 New Done Done false
22531 7649 Acked 8648 Read 3 New Done Done false
22532 7650 Acked 8649 Write 2 New Done Done false
22533 7651 Acked 8650 Flush 0 New Done Done false
22534 7652 Acked 8651 Read 7 New Done Done false
22535 7653 Acked 8652 Flush 0 New Done Done false
22536 7654 Acked 8653 Write 7 New Done Done false
22537 7655 Acked 8654 Read 5 New Done Done false
22538 7656 Acked 8655 Write 3 New Done Done false
22539 7657 Acked 8656 Read 5 New Done Done false
22540 7658 Acked 8657 Read 3 New Done Done false
22541 7659 Acked 8658 Write 3 New Done Done false
22542 7660 Acked 8659 Read 2 New Done Done false
22543 7661 Acked 8660 Flush 0 New Done Done false
22544 7662 Acked 8661 Read 5 New Done Done false
22545 7663 Acked 8662 Write 4 New Done Done false
22546 7664 Acked 8663 Read 9 New Done Done false
22547 7665 Acked 8664 Write 5 New Done Done false
22548 7666 Acked 8665 Write 1 New Done Done false
22549 7667 Acked 8666 Write 9 New Done Done false
22550 7668 Acked 8667 Read 6 New Done Done false
22551 7669 Acked 8668 Read 5 New Done Done false
22552 7670 Acked 8669 Read 8 New Done Done false
22553 7671 Acked 8670 Flush 0 New Done Done false
22554 7672 Acked 8671 Read 10 New Done Done false
22555 7673 Acked 8672 Read 8 New Done Done false
22556 7674 Acked 8673 Read 1 New Done Done false
22557 7675 Acked 8674 Write 3 New Done Done false
22558 7676 Acked 8675 Write 4 New Done Done false
22559 7677 Acked 8676 Read 8 New Done Done false
22560 7678 Acked 8677 Read 4 New Done Done false
22561 7679 Acked 8678 Read 8 New Done Done false
22562 7680 Acked 8679 Read 2 New Done Done false
22563 7681 Acked 8680 Write 4 New Done Done false
22564 7682 Acked 8681 Read 2 New Done Done false
22565 7683 Acked 8682 Read 2 New Done Done false
22566 7684 Acked 8683 Read 3 New Done Done false
22567 7685 Acked 8684 Write 2 New Done Done false
22568 7686 Acked 8685 Write 6 New Done Done false
22569 7687 Acked 8686 Flush 0 New Done Done false
22570 7688 Acked 8687 Write 4 New Done Done false
22571 7689 Acked 8688 Read 6 New Done Done false
22572 7690 Acked 8689 Flush 0 New Done Done false
22573 7691 Acked 8690 Read 1 New Done Done false
22574 7692 Acked 8691 Write 4 New Done Done false
22575 7693 Acked 8692 Write 5 New Done Done false
22576 7694 Acked 8693 Read 9 New Done Done false
22577 7695 Acked 8694 Read 7 New Done Done false
22578 7696 Acked 8695 Read 2 New Done Done false
22579 7697 Acked 8696 Write 5 New Done Done false
22580 7698 Acked 8697 Read 2 New Done Done false
22581 7699 Acked 8698 Flush 0 New Done Done false
22582 7700 Acked 8699 Flush 0 New Done Done false
22583 7701 Acked 8700 Write 9 New Done Done false
22584 7702 Acked 8701 Read 3 New Done Done false
22585 7703 Acked 8702 Write 4 New Done Done false
22586 7704 Acked 8703 Write 8 New Done Done false
22587 7705 Acked 8704 Read 1 New Done Done false
22588 7706 Acked 8705 Write 6 New Done Done false
22589 7707 Acked 8706 Flush 0 New Done Done false
22590 7708 Acked 8707 Write 2 New Done Done false
22591 7709 Acked 8708 Read 2 New Done Done false
22592 7710 Acked 8709 Write 9 New Done Done false
22593 7711 Acked 8710 Read 6 New Done Done false
22594 7712 Acked 8711 Read 10 New Done Done false
22595 7713 Acked 8712 Flush 0 New Done Done false
22596 7714 Acked 8713 Read 6 New Done Done false
22597 7715 Acked 8714 Read 10 New Done Done false
22598 7716 Acked 8715 Write 3 New Done Done false
22599 7717 Acked 8716 Write 7 New Done Done false
22600 7718 Acked 8717 Read 2 New Done Done false
22601 7719 Acked 8718 Write 9 New Done Done false
22602 7720 Acked 8719 Write 1 New Done Done false
22603 7721 Acked 8720 Read 1 New Done Done false
22604 7722 Acked 8721 Write 3 New Done Done false
22605 7723 Acked 8722 Write 7 New Done Done false
22606 7724 Acked 8723 Read 2 New Done Done false
22607 7725 Acked 8724 Flush 0 New Done Done false
22608 7726 Acked 8725 Write 9 New Done Done false
22609 7727 Acked 8726 Write 1 New Done Done false
22610 7728 Acked 8727 Read 2 New Done Done false
22611 7729 Acked 8728 Read 8 New Done Done false
22612 7730 Acked 8729 Write 4 New Done Done false
22613 7731 Acked 8730 Flush 0 New Done Done false
22614 7732 Acked 8731 Write 1 New Done Done false
22615 7733 Acked 8732 Read 8 New Done Done false
22616 7734 Acked 8733 Write 6 New Done Done false
22617 7735 Acked 8734 Read 9 New Done Done false
22618 7736 Acked 8735 Read 8 New Done Done false
22619 7737 Acked 8736 Write 2 New Done Done false
22620 7738 Acked 8737 Read 1 New Done Done false
22621 7739 Acked 8738 Write 7 New Done Done false
22622 7740 Acked 8739 Read 10 New Done Done false
22623 7741 Acked 8740 Write 4 New Done Done false
22624 7742 Acked 8741 Read 1 New Done Done false
22625 7743 Acked 8742 Read 4 New Done Done false
22626 7744 Acked 8743 Write 10 New Done Done false
22627 7745 Acked 8744 Read 3 New Done Done false
22628 7746 Acked 8745 Write 3 New Done Done false
22629 7747 Acked 8746 Read 9 New Done Done false
22630 7748 Acked 8747 Read 10 New Done Done false
22631 7749 Acked 8748 Write 6 New Done Done false
22632 7750 Acked 8749 Write 7 New Done Done false
22633 7751 Acked 8750 Read 7 New Done Done false
22634 7752 Acked 8751 Flush 0 New Done Done false
22635 7753 Acked 8752 Read 9 New Done Done false
22636 7754 Acked 8753 Read 9 New Done Done false
22637 7755 Acked 8754 Read 6 New Done Done false
22638 7756 Acked 8755 Read 8 New Done Done false
22639 7757 Acked 8756 Read 3 New Done Done false
22640 7758 Acked 8757 Write 2 New Done Done false
22641 7759 Acked 8758 Write 3 New Done Done false
22642 7760 Acked 8759 Write 3 New Done Done false
22643 7761 Acked 8760 Read 3 New Done Done false
22644 7762 Acked 8761 Read 4 New Done Done false
22645 7763 Acked 8762 Write 4 New Done Done false
22646 7764 Acked 8763 Flush 0 New Done Done false
22647 7765 Acked 8764 Read 10 New Done Done false
22648 7766 Acked 8765 Flush 0 New Done Done false
22649 7767 Acked 8766 Write 2 New Done Done false
22650 7768 Acked 8767 Read 1 New Done Done false
22651 7769 Acked 8768 Read 6 New Done Done false
22652 7770 Acked 8769 Read 5 New Done Done false
22653 7771 Acked 8770 Read 7 New Done Done false
22654 7772 Acked 8771 Read 2 New Done Done false
22655 7773 Acked 8772 Read 6 New Done Done false
22656 7774 Acked 8773 Read 6 New Done Done false
22657 7775 Acked 8774 Write 7 New Done Done false
22658 7776 Acked 8775 Read 10 New Done Done false
22659 7777 Acked 8776 Write 8 New Done Done false
22660 7778 Acked 8777 Write 9 New Done Done false
22661 7779 Acked 8778 Write 4 New Done Done false
22662 7780 Acked 8779 Write 10 New Done Done false
22663 7781 Acked 8780 Read 9 New Done Done false
22664 7782 Acked 8781 Read 1 New Done Done false
22665 7783 Acked 8782 Flush 0 New Done Done false
22666 7784 Acked 8783 Write 7 New Done Done false
22667 7785 Acked 8784 Read 3 New Done Done false
22668 7786 Acked 8785 Read 10 New Done Done false
22669 7787 Acked 8786 Write 5 New Done Done false
22670 7788 Acked 8787 Write 3 New Done Done false
22671 7789 Acked 8788 Read 6 New Done Done false
22672 7790 Acked 8789 Write 9 New Done Done false
22673 7791 Acked 8790 Write 2 New Done Done false
22674 7792 Acked 8791 Write 4 New Done Done false
22675 7793 Acked 8792 Write 5 New Done Done false
22676 7794 Acked 8793 Flush 0 New Done Done false
22677 7795 Acked 8794 Write 5 New Done Done false
22678 7796 Acked 8795 Read 4 New Done Done false
22679 7797 Acked 8796 Read 9 New Done Done false
22680 7798 Acked 8797 Write 2 New Done Done false
22681 7799 Acked 8798 Write 2 New Done Done false
22682 7800 Acked 8799 Write 9 New Done Done false
22683 7801 Acked 8800 Write 5 New Done Done false
22684 7802 Acked 8801 Read 10 New Done Done false
22685 7803 Acked 8802 Write 9 New Done Done false
22686 7804 Acked 8803 Write 3 New Done Done false
22687 7805 Acked 8804 Read 8 New Done Done false
22688 7806 Acked 8805 Flush 0 New Done Done false
22689 7807 Acked 8806 Write 4 New Done Done false
22690 7808 Acked 8807 Write 9 New Done Done false
22691 7809 Acked 8808 Read 2 New Done Done false
22692 7810 Acked 8809 Flush 0 New Done Done false
22693 7811 Acked 8810 Write 2 New Done Done false
22694 7812 Acked 8811 Flush 0 New Done Done false
22695 7813 Acked 8812 Read 5 New Done Done false
22696 7814 Acked 8813 Read 4 New Done Done false
22697 7815 Acked 8814 Write 9 New Done Done false
22698 7816 Acked 8815 Write 9 New Done Done false
22699 7817 Acked 8816 Read 5 New Done Done false
22700 7818 Acked 8817 Write 6 New Done Done false
22701 7819 Acked 8818 Read 6 New Done Done false
22702 7820 Acked 8819 Write 1 New Done Done false
22703 7821 Acked 8820 Write 2 New Done Done false
22704 7822 Acked 8821 Write 8 New Done Done false
22705 7823 Acked 8822 Read 6 New Done Done false
22706 7824 Acked 8823 Read 7 New Done Done false
22707 7825 Acked 8824 Write 7 New Done Done false
22708 7826 Acked 8825 Read 1 New Done Done false
22709 7827 Acked 8826 Read 3 New Done Done false
22710 7828 Acked 8827 Flush 0 New Done Done false
22711 7829 Acked 8828 Write 4 New Done Done false
22712 7830 Acked 8829 Write 8 New Done Done false
22713 7831 Acked 8830 Write 4 New Done Done false
22714 7832 Acked 8831 Read 9 New Done Done false
22715 7833 Acked 8832 Read 10 New Done Done false
22716 7834 Acked 8833 Read 8 New Done Done false
22717 7835 Acked 8834 Write 2 New Done Done false
22718 7836 Acked 8835 Write 2 New Done Done false
22719 STATES DS:0 DS:1 DS:2 TOTAL
22720 New 300 0 0 300
22721 Sent 0 0 0 0
22722 Done 0 300 300 600
22723 Skipped 0 0 0 0
22724 Error 0 0 0 0
22725 Last Flush: 8535 8827 8827
22726 Downstairs last five completed: 8535 8534 8533 8532 8531
22727 Upstairs last five completed: 7836 7835 7834 7833 7832
22728 CLIENT: Up:0 ds:300 act:2
227292024-03-02T01:10:07.696ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227302024-03-02T01:10:07.696ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227312024-03-02T01:10:07.696ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227322024-03-02T01:10:07.696ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227332024-03-02T01:10:07.696ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227342024-03-02T01:10:07.696ZINFOcrucible: [0] client re-new 301 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227352024-03-02T01:10:07.697ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22736 [8810] got start action
22737 [8810] I am going to restart
22738 Starting downstairs at port 8810
22739 Make output file at "/tmp/dsc/downstairs-8810.txt"
22740 [8810][0] reports Starting
227412024-03-02T01:10:08.698ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227422024-03-02T01:10:08.698ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227432024-03-02T01:10:08.698ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227442024-03-02T01:10:08.698ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227452024-03-02T01:10:08.698ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227462024-03-02T01:10:08.698ZINFOcrucible: [0] client re-new 301 jobs since flush 8535 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227472024-03-02T01:10:08.698ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22748 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188)
22749 [8810][0] reports Running
227502024-03-02T01:10:09.699ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227512024-03-02T01:10:09.699ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227522024-03-02T01:10:09.700ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227532024-03-02T01:10:09.700ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227542024-03-02T01:10:09.700ZINFOcrucible: send last flush ID to this DS: 8535 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227552024-03-02T01:10:09.701ZINFOcrucible: Replied this last flush ID: 8535 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227562024-03-02T01:10:09.701ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227572024-03-02T01:10:09.701ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227582024-03-02T01:10:09.701ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227592024-03-02T01:10:09.701ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22760 ----------------------------------------------------------------
22761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22762 Downstairs last five completed: 8836 8835 8834 8833 8832
22763 Upstairs last five completed: 7837 7836 7835 7834 7833
22764 CLIENT: Up:0 ds:0 act:3
22765 Replay: All jobs finished, all DS active.
22766 Main task has work to do, go find it
22767 got dsc Stop(2)
22768 stop 2
22769 Mar 02 01:10:11.044 INFO request completed, [8830] Got stop action so:false kr:false
22770 latency_us: 325, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 34fcee72-3508-4f27-8fe6-b9971c027483, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
22771 Mar 02 01:10:11.045 INFO request completed, latency_us: 104, response_code:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:10:11.045256132Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
22772 200, uri: /state/cid/2, method: GET, req_id: 61b58cd5-eb67-4fe3-a1a4-d8cb03b830c9, remote_addr: 127{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-03-02T01:10:11.045330318Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"io task","client":"2","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
22773 .0.0.1:40735, local_addr: 127.0.0.1:9998
227742024-03-02T01:10:11.045ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227752024-03-02T01:10:11.045ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227762024-03-02T01:10:11.045ZINFOcrucible: [2] client re-new 0 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227772024-03-02T01:10:11.045ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22778 [8830] Exited with: ExitStatus(unix_wait_status(9))
22779 [8830][2] reports Exit
227802024-03-02T01:10:12.046ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227812024-03-02T01:10:12.046ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227822024-03-02T01:10:12.046ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227832024-03-02T01:10:12.046ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227842024-03-02T01:10:12.046ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227852024-03-02T01:10:12.046ZINFOcrucible: [2] client re-new 0 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227862024-03-02T01:10:12.046ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227872024-03-02T01:10:13.047ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227882024-03-02T01:10:13.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227892024-03-02T01:10:13.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227902024-03-02T01:10:13.047ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227912024-03-02T01:10:13.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227922024-03-02T01:10:13.047ZINFOcrucible: [2] client re-new 0 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227932024-03-02T01:10:13.047ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227942024-03-02T01:10:14.049ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227952024-03-02T01:10:14.049ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227962024-03-02T01:10:14.049ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227972024-03-02T01:10:14.049ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227982024-03-02T01:10:14.049ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
227992024-03-02T01:10:14.049ZINFOcrucible: [2] client re-new 0 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228002024-03-02T01:10:14.049ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22801 Mar 02 01:10:15.047 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 580391a6-0952-4bde-8e67-613cd1120406, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
22802 001/300 Read block 3604 len 8192
22803 002/300 Read block 3306 len 36864
22804 003/300 Read block 57 len 4096
228052024-03-02T01:10:15.050ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228062024-03-02T01:10:15.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228072024-03-02T01:10:15.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228082024-03-02T01:10:15.051ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228092024-03-02T01:10:15.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
228102024-03-02T01:10:15.051ZINFOcrucible: [2] client re-new 3 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22811 004/300 Read block 3472 len 12288
228122024-03-02T01:10:15.051ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
22813 005/300 Write block 4908 len 20480 data: 6 6 6 6 5
22814 006/300 Write block 4606 len 12288 data: 5 4 4
22815 007/300 Read block 1556 len 4096
22816 008/300 Flush
22817 009/300 Read block 1609 len 40960
22818 010/300 Read block 2895 len 32768
22819 011/300 Flush
22820 012/300 Read block 4578 len 20480
22821 013/300 Write block 3131 len 12288 data: 3 3 3
22822 014/300 Flush
22823 015/300 Write block 3356 len 8192 data: 9 9
22824 016/300 Read block 3550 len 32768
22825 017/300 Read block 3387 len 20480
22826 018/300 Read block 3624 len 36864
22827 019/300 Write block 785 len 28672 data: 2 6 4 3 3 4 3
22828 020/300 Read block 1925 len 20480
22829 021/300 Write block 1632 len 8192 data: 4 5
22830 022/300 Write block 3025 len 16384 data: 5 4 4 4
22831 023/300 Flush
22832 024/300 Read block 2118 len 36864
22833 025/300 Read block 1825 len 24576
22834 026/300 Flush
22835 027/300 Read block 1687 len 12288
22836 028/300 Write block 1765 len 4096 data: 4
22837 029/300 Read block 1155 len 16384
22838 030/300 Flush
22839 031/300 Read block 1442 len 20480
22840 032/300 Read block 4339 len 32768
22841 033/300 Write block 4694 len 12288 data: 5 5 5
22842 034/300 Read block 1778 len 32768
22843 035/300 Read block 597 len 12288
22844 036/300 Write block 2777 len 8192 data: 4 7
22845 037/300 Read block 951 len 36864
22846 038/300 Read block 600 len 28672
22847 039/300 Write block 2897 len 12288 data: 5 6 7
22848 040/300 Flush
22849 041/300 Read block 1009 len 32768
22850 042/300 Flush
22851 043/300 Read block 2935 len 36864
22852 044/300 Write block 2767 len 16384 data: 7 6 6 6
22853 045/300 Write block 1837 len 16384 data: 8 5 6 6
22854 046/300 Write block 59 len 12288 data: 5 4 5
22855 047/300 Read block 1743 len 8192
22856 048/300 Flush
22857 049/300 Read block 3501 len 40960
22858 050/300 Read block 1610 len 12288
22859 051/300 Write block 2911 len 8192 data: 7 6
22860 052/300 Read block 4068 len 20480
22861 053/300 Write block 2300 len 32768 data: 5 5 4 3 4 5 4 3
22862 054/300 Read block 1563 len 8192
22863 055/300 Read block 2716 len 32768
22864 056/300 Write block 831 len 4096 data: 8
22865 057/300 Read block 2665 len 4096
22866 058/300 Write block 4024 len 20480 data: 6 5 6 6 7
22867 059/300 Write block 3669 len 16384 data: 4 4 4 4
22868 060/300 Read block 347 len 4096
22869 061/300 Read block 4472 len 24576
22870 062/300 Read block 609 len 32768
22871 063/300 Read block 2488 len 28672
22872 064/300 Read block 3146 len 36864
22873 065/300 Read block 1190 len 32768
22874 066/300 Read block 2618 len 20480
22875 067/300 Read block 406 len 40960
22876 068/300 Read block 890 len 4096
22877 069/300 Write block 2677 len 12288 data: 6 6 7
22878 070/300 Read block 1032 len 20480
22879 071/300 Read block 337 len 20480
22880 072/300 Write block 2671 len 8192 data: 5 7
22881 073/300 Write block 2744 len 16384 data: 6 6 7 6
22882 074/300 Read block 2365 len 12288
22883 075/300 Read block 86 len 12288
22884 076/300 Read block 4742 len 4096
22885 077/300 Read block 1838 len 4096
22886 078/300 Read block 1230 len 28672
22887 079/300 Read block 3468 len 12288
22888 080/300 Read block 1594 len 4096
22889 081/300 Write block 1703 len 4096 data: 3
22890 082/300 Read block 3430 len 40960
22891 083/300 Flush
22892 084/300 Flush
22893 085/300 Read block 3379 len 24576
22894 086/300 Flush
22895 087/300 Read block 1434 len 12288
22896 088/300 Read block 1969 len 8192
22897 089/300 Write block 1276 len 20480 data: 6 5 5 6 6
22898 090/300 Write block 2152 len 20480 data: 4 5 5 4 4
22899 091/300 Read block 479 len 40960
22900 092/300 Write block 3490 len 16384 data: 4 4 5 4
22901 093/300 Read block 3525 len 28672
22902 094/300 Write block 3318 len 40960 data: 2 3 3 3 4 4 3 4 5 6
22903 095/300 Read block 2962 len 16384
22904 096/300 Write block 2916 len 12288 data: 6 6 4
22905 097/300 Read block 1951 len 12288
22906 098/300 Read block 1093 len 8192
22907 099/300 Write block 608 len 12288 data: 5 6 7
22908 100/300 Read block 2604 len 36864
22909 101/300 Write block 2714 len 12288 data: 5 5 7
22910 102/300 Flush
22911 103/300 Read block 268 len 24576
22912 104/300 Read block 3936 len 28672
22913 105/300 Write block 1990 len 16384 data: 6 7 8 7
22914 106/300 Write block 2543 len 32768 data: 7 5 4 4 4 5 4 2
22915 107/300 Write block 2555 len 40960 data: 5 6 6 7 7 9 8 7 7 8
22916 108/300 Read block 2992 len 16384
22917 109/300 Write block 1856 len 36864 data: 6 5 5 5 5 6 7 5 6
22918 110/300 Write block 82 len 12288 data: 6 6 5
22919 111/300 Read block 315 len 32768
22920 112/300 Write block 2016 len 32768 data: 7 6 8 7 4 6 5 5
22921 113/300 Read block 3351 len 40960
22922 114/300 Write block 1558 len 28672 data: 4 2 4 4 4 3 4
22923 115/300 Read block 346 len 16384
22924 116/300 Write block 3331 len 28672 data: 9 8 6 7 6 6 7
22925 117/300 Write block 3826 len 28672 data: 5 5 4 6 6 6 7
22926 118/300 Write block 925 len 8192 data: 7 7
22927 119/300 Write block 3578 len 24576 data: 5 5 4 4 4 5
22928 120/300 Write block 2211 len 40960 data: 5 6 4 4 2 4 4 8 6 8
22929 121/300 Write block 2167 len 4096 data: 6
22930 122/300 Flush
22931 123/300 Read block 736 len 36864
22932 124/300 Read block 2184 len 16384
22933 125/300 Write block 969 len 28672 data: 4 4 4 3 4 4 5
22934 126/300 Write block 2376 len 28672 data: 5 6 6 6 6 6 6
22935 127/300 Write block 3081 len 28672 data: 3 3 3 3 3 2 2
22936 128/300 Write block 3138 len 4096 data: 7
22937 129/300 Read block 3295 len 28672
22938 130/300 Read block 3224 len 24576
22939 131/300 Write block 1422 len 20480 data: 5 6 4 4 5
22940 132/300 Write block 228 len 20480 data: 6 5 6 5 6
22941 133/300 Write block 2566 len 36864 data: 6 5 7 6 5 3 2 2 2
22942 134/300 Flush
22943 135/300 Write block 4380 len 8192 data: 4 4
22944 136/300 Write block 106 len 8192 data: 5 5
22945 137/300 Read block 2105 len 16384
22946 138/300 Write block 3228 len 8192 data: 5 5
22947 139/300 Read block 1956 len 28672
22948 140/300 Write block 2937 len 8192 data: 4 5
22949 141/300 Flush
22950 142/300 Read block 1514 len 28672
22951 143/300 Read block 1353 len 40960
22952 144/300 Read block 639 len 16384
22953 145/300 Read block 4667 len 36864
22954 146/300 Read block 1723 len 20480
22955 147/300 Write block 978 len 20480 data: 4 4 5 6 8
22956 148/300 Read block 2423 len 40960
22957 149/300 Read block 1501 len 20480
22958 150/300 Read block 4429 len 28672
22959 151/300 Write block 2827 len 4096 data: 9
22960 152/300 Write block 4517 len 40960 data: 5 6 5 5 4 5 5 5 4 5
22961 153/300 Write block 3728 len 20480 data: 2 2 3 4 5
22962 154/300 Write block 472 len 40960 data: 7 7 7 6 4 4 4 5 5 5
22963 155/300 Read block 2046 len 4096
22964 156/300 Read block 374 len 20480
22965 157/300 Write block 1912 len 8192 data: 7 7
22966 158/300 Write block 2246 len 4096 data: 4
22967 159/300 Write block 1442 len 32768 data: 8 9 9 9 10 9 9 7
22968 160/300 Write block 3006 len 20480 data: 6 5 5 5 4
22969 161/300 Write block 1991 len 12288 data: 8 9 8
22970 162/300 Read block 2321 len 20480
22971 163/300 Read block 3716 len 16384
22972 164/300 Write block 62 len 36864 data: 4 3 3 4 4 6 7 8 8
22973 165/300 Write block 4211 len 36864 data: 7 5 5 5 5 7 7 6 7
22974 166/300 Read block 3956 len 36864
22975 167/300 Read block 54 len 24576
22976 168/300 Read block 3903 len 4096
22977 169/300 Write block 550 len 24576 data: 3 4 4 4 5 4
22978 170/300 Read block 180 len 24576
22979 171/300 Write block 437 len 28672 data: 3 3 2 3 3 3 4
22980 172/300 Read block 3282 len 16384
22981 173/300 Read block 4613 len 16384
22982 174/300 Write block 412 len 12288 data: 4 5 7
22983 175/300 Read block 2189 len 12288
22984 176/300 Flush
22985 177/300 Write block 340 len 36864 data: 15 13 13 12 10 10 11 11 9
22986 178/300 Read block 735 len 32768
22987 179/300 Write block 1292 len 4096 data: 4
22988 180/300 Read block 227 len 28672
22989 181/300 Read block 1485 len 28672
22990 182/300 Write block 2836 len 28672 data: 4 5 6 7 7 7 9
22991 183/300 Flush
22992 184/300 Read block 2504 len 24576
22993 185/300 Read block 4853 len 20480
22994 186/300 Read block 1928 len 4096
22995 187/300 Read block 3849 len 4096
22996 188/300 Read block 812 len 4096
22997 189/300 Read block 3150 len 32768
22998 190/300 Write block 4604 len 16384 data: 8 6 6 5
22999 191/300 Write block 3493 len 24576 data: 5 4 5 5 5 5
23000 192/300 Write block 3964 len 24576 data: 5 4 4 4 4 4
23001 193/300 Read block 3650 len 12288
23002 194/300 Read block 2502 len 40960
23003 195/300 Write block 2007 len 4096 data: 6
23004 196/300 Read block 2362 len 36864
23005 197/300 Write block 3990 len 8192 data: 7 6
23006 198/300 Read block 3122 len 40960
23007 199/300 Write block 1716 len 4096 data: 6
23008 200/300 Write block 4090 len 28672 data: 4 5 7 7 5 5 6
23009 201/300 Flush
23010 202/300 Write block 3881 len 28672 data: 3 3 3 2 3 3 3
23011 203/300 Write block 1211 len 28672 data: 8 9 10 8 6 5 5
23012 204/300 Flush
23013 205/300 Read block 3211 len 8192
23014 206/300 Write block 4953 len 40960 data: 8 9 7 5 5 5 4 4 5 5
23015 207/300 Write block 1697 len 16384 data: 7 9 7 4
23016 208/300 Write block 3904 len 4096 data: 6
23017 209/300 Write block 2421 len 20480 data: 4 4 5 5 5
23018 210/300 Read block 144 len 36864
23019 211/300 Read block 3414 len 12288
23020 212/300 Read block 1959 len 28672
23021 213/300 Read block 4615 len 40960
23022 214/300 Read block 4095 len 16384
23023 215/300 Flush
23024 216/300 Flush
23025 217/300 Read block 4827 len 24576
23026 218/300 Read block 3270 len 16384
23027 219/300 Flush
23028 220/300 Write block 2494 len 16384 data: 5 6 5 6
23029 221/300 Write block 2935 len 8192 data: 5 4
23030 222/300 Read block 3215 len 4096
23031 223/300 Read block 2229 len 12288
23032 224/300 Write block 1488 len 40960 data: 3 3 3 4 3 4 5 5 4 4
23033 225/300 Read block 1170 len 40960
23034 226/300 Read block 517 len 36864
23035 227/300 Read block 456 len 16384
23036 228/300 Flush
23037 229/300 Write block 1683 len 12288 data: 9 9 8
23038 230/300 Write block 3207 len 28672 data: 8 9 9 8 7 6 7
23039 231/300 Write block 2929 len 16384 data: 5 5 5 6
23040 232/300 Read block 3130 len 24576
23041 233/300 Write block 3819 len 32768 data: 6 7 8 6 5 4 4 6
23042 234/300 Read block 268 len 36864
23043 235/300 Write block 2212 len 36864 data: 7 5 5 3 5 5 9 7 9
23044 236/300 Read block 3713 len 40960
23045 237/300 Flush
23046 238/300 Write block 2656 len 24576 data: 5 6 4 5 6 7
23047 239/300 Read block 1981 len 28672
23048 240/300 Read block 4616 len 16384
23049 241/300 Read block 427 len 24576
23050 242/300 Write block 3732 len 4096 data: 6
23051 243/300 Read block 2224 len 8192
23052 244/300 Read block 3040 len 32768
23053 245/300 Write block 571 len 28672 data: 4 4 5 6 5 7 7
23054 246/300 Read block 1292 len 32768
23055 247/300 Read block 3359 len 20480
23056 248/300 Write block 265 len 40960 data: 8 10 7 6 7 6 9 7 6 6
23057 249/300 Flush
23058 250/300 Write block 1375 len 24576 data: 5 4 5 5 4 6
23059 251/300 Read block 4818 len 12288
23060 252/300 Read block 1607 len 28672
23061 253/300 Write block 4532 len 20480 data: 6 6 5 5 4
23062 254/300 Read block 1630 len 12288
23063 255/300 Write block 215 len 24576 data: 6 4 4 5 5 5
23064 256/300 Read block 812 len 20480
23065 257/300 Write block 4762 len 28672 data: 8 8 9 7 8 9 8
23066 258/300 Write block 3713 len 24576 data: 8 8 7 8 8 7
23067 259/300 Write block 2424 len 24576 data: 6 6 4 3 4 6
23068 260/300 Write block 173 len 28672 data: 6 6 3 2 2 3 3
23069 261/300 Write block 1166 len 40960 data: 3 3 3 4 4 4 3 4 4 4
23070 262/300 Read block 3721 len 4096
23071 263/300 Read block 2765 len 12288
23072 264/300 Write block 391 len 40960 data: 9 6 5 5 3 2 3 2 3 3
23073 265/300 Read block 4793 len 36864
23074 266/300 Flush
23075 267/300 Write block 3296 len 4096 data: 6
23076 268/300 Read block 1596 len 20480
23077 269/300 Read block 134 len 32768
23078 270/300 Write block 4629 len 36864 data: 5 5 5 6 5 4 3 3 3
23079 271/300 Write block 313 len 8192 data: 3 3
23080 272/300 Flush
23081 273/300 Read block 1340 len 4096
23082 274/300 Write block 4186 len 40960 data: 9 8 7 7 7 8 7 7 8 8
23083 275/300 Read block 1116 len 20480
23084 276/300 Write block 2353 len 20480 data: 3 4 3 3 5
23085 277/300 Write block 301 len 36864 data: 4 3 4 6 6 5 5 5 5
23086 278/300 Flush
23087 279/300 Write block 2857 len 24576 data: 5 4 5 3 3 4
23088 280/300 Write block 3206 len 28672 data: 8 9 10 10 9 8 7
23089 281/300 Write block 2489 len 8192 data: 2 3
23090 282/300 Flush
23091 283/300 Write block 3644 len 12288 data: 5 5 5
23092 284/300 Read block 31 len 16384
23093 285/300 Read block 4052 len 24576
23094 286/300 Read block 2105 len 12288
23095 287/300 Read block 2028 len 36864
23096 288/300 Read block 3613 len 40960
23097 289/300 Read block 3597 len 16384
23098 290/300 Write block 156 len 28672 data: 5 4 5 6 5 7 6
23099 291/300 Read block 1926 len 36864
23100 292/300 Read block 3551 len 28672
23101 293/300 Write block 21 len 28672 data: 6 6 6 5 4 5 5
23102 294/300 Write block 2329 len 24576 data: 4 4 4 4 3 3
23103 295/300 Read block 4193 len 36864
23104 296/300 Write block 4448 len 4096 data: 5
23105 297/300 Read block 558 len 16384
23106 298/300 Read block 4064 len 12288
23107 299/300 Read block 2701 len 16384
23108 300/300 Read block 2151 len 32768
23109 Main task has work to do, go find it
23110 got dsc Start(2)
23111 start 2
23112 Mar 02 01:10:15.389 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/2, method: POST, req_id: b5fb1e54-00ab-48d1-9091-687551068873, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
23113 Replay: started 2, returned:Ok(())
23114 ----------------------------------------------------------------
23115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23116 States: Active Active Offline
23117 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23118 7838 Acked 8837 Read 2 Done Done New true
23119 7839 Acked 8838 Read 9 Done Done New true
23120 7840 Acked 8839 Read 1 Done Done New true
23121 7841 Acked 8840 Read 3 Done Done New false
23122 7842 Acked 8841 Write 5 Done Done New false
23123 7843 Acked 8842 Write 3 Done Done New false
23124 7844 Acked 8843 Read 1 Done Done New false
23125 7845 Acked 8844 Flush 0 Done Done New false
23126 7846 Acked 8845 Read 10 Done Done New false
23127 7847 Acked 8846 Read 8 Done Done New false
23128 7848 Acked 8847 Flush 0 Done Done New false
23129 7849 Acked 8848 Read 5 Done Done New false
23130 7850 Acked 8849 Write 3 Done Done New false
23131 7851 Acked 8850 Flush 0 Done Done New false
23132 7852 Acked 8851 Write 2 Done Done New false
23133 7853 Acked 8852 Read 8 Done Done New false
23134 7854 Acked 8853 Read 5 Done Done New false
23135 7855 Acked 8854 Read 9 Done Done New false
23136 7856 Acked 8855 Write 7 Done Done New false
23137 7857 Acked 8856 Read 5 Done Done New false
23138 7858 Acked 8857 Write 2 Done Done New false
23139 7859 Acked 8858 Write 4 Done Done New false
23140 7860 Acked 8859 Flush 0 Done Done New false
23141 7861 Acked 8860 Read 9 Done Done New false
23142 7862 Acked 8861 Read 6 Done Done New false
23143 7863 Acked 8862 Flush 0 Done Done New false
23144 7864 Acked 8863 Read 3 Done Done New false
23145 7865 Acked 8864 Write 1 Done Done New false
23146 7866 Acked 8865 Read 4 Done Done New false
23147 7867 Acked 8866 Flush 0 Done Done New false
23148 7868 Acked 8867 Read 5 Done Done New false
23149 7869 Acked 8868 Read 8 Done Done New false
23150 7870 Acked 8869 Write 3 Done Done New false
23151 7871 Acked 8870 Read 8 Done Done New false
23152 7872 Acked 8871 Read 3 Done Done New false
23153 7873 Acked 8872 Write 2 Done Done New false
23154 7874 Acked 8873 Read 9 Done Done New false
23155 7875 Acked 8874 Read 7 Done Done New false
23156 7876 Acked 8875 Write 3 Done Done New false
23157 7877 Acked 8876 Flush 0 Done Done New false
23158 7878 Acked 8877 Read 8 Done Done New false
23159 7879 Acked 8878 Flush 0 Done Done New false
23160 7880 Acked 8879 Read 9 Done Done New false
23161 7881 Acked 8880 Write 4 Done Done New false
23162 7882 Acked 8881 Write 4 Done Done New false
23163 7883 Acked 8882 Write 3 Done Done New false
23164 7884 Acked 8883 Read 2 Done Done New false
23165 7885 Acked 8884 Flush 0 Done Done New false
23166 7886 Acked 8885 Read 10 Done Done New false
23167 7887 Acked 8886 Read 3 Done Done New false
23168 7888 Acked 8887 Write 2 Done Done New false
23169 7889 Acked 8888 Read 5 Done Done New false
23170 7890 Acked 8889 Write 8 Done Done New false
23171 7891 Acked 8890 Read 2 Done Done New false
23172 7892 Acked 8891 Read 8 Done Done New false
23173 7893 Acked 8892 Write 1 Done Done New false
23174 7894 Acked 8893 Read 1 Done Done New false
23175 7895 Acked 8894 Write 5 Done Done New false
23176 7896 Acked 8895 Write 4 Done Done New false
23177 7897 Acked 8896 Read 1 Done Done New false
23178 7898 Acked 8897 Read 6 Done Done New false
23179 7899 Acked 8898 Read 8 Done Done New false
23180 7900 Acked 8899 Read 7 Done Done New false
23181 7901 Acked 8900 Read 9 Done Done New false
23182 7902 Acked 8901 Read 8 Done Done New false
23183 7903 Acked 8902 Read 5 Done Done New false
23184 7904 Acked 8903 Read 10 Done Done New false
23185 7905 Acked 8904 Read 1 Done Done New false
23186 7906 Acked 8905 Write 3 Done Done New false
23187 7907 Acked 8906 Read 5 Done Done New false
23188 7908 Acked 8907 Read 5 Done Done New false
23189 7909 Acked 8908 Write 2 Done Done New false
23190 7910 Acked 8909 Write 4 Done Done New false
23191 7911 Acked 8910 Read 3 Done Done New false
23192 7912 Acked 8911 Read 3 Done Done New false
23193 7913 Acked 8912 Read 1 Done Done New false
23194 7914 Acked 8913 Read 1 Done Done New false
23195 7915 Acked 8914 Read 7 Done Done New false
23196 7916 Acked 8915 Read 3 Done Done New false
23197 7917 Acked 8916 Read 1 Done Done New false
23198 7918 Acked 8917 Write 1 Done Done New false
23199 7919 Acked 8918 Read 10 Done Done New false
23200 7920 Acked 8919 Flush 0 Done Done New false
23201 7921 Acked 8920 Flush 0 Done Done New false
23202 7922 Acked 8921 Read 6 Done Done New false
23203 7923 Acked 8922 Flush 0 Done Done New false
23204 7924 Acked 8923 Read 3 Done Done New false
23205 7925 Acked 8924 Read 2 Done Done New false
23206 7926 Acked 8925 Write 5 Done Done New false
23207 7927 Acked 8926 Write 5 Done Done New false
23208 7928 Acked 8927 Read 10 Done Done New false
23209 7929 Acked 8928 Write 4 Done Done New false
23210 7930 Acked 8929 Read 7 Done Done New false
23211 7931 Acked 8930 Write 10 Done Done New false
23212 7932 Acked 8931 Read 4 Done Done New false
23213 7933 Acked 8932 Write 3 Done Done New false
23214 7934 Acked 8933 Read 3 Done Done New false
23215 7935 Acked 8934 Read 2 Done Done New false
23216 7936 Acked 8935 Write 3 Done Done New false
23217 7937 Acked 8936 Read 9 Done Done New false
23218 7938 Acked 8937 Write 3 Done Done New false
23219 7939 Acked 8938 Flush 0 Done Done New false
23220 7940 Acked 8939 Read 6 Done Done New false
23221 7941 Acked 8940 Read 7 Done Done New false
23222 7942 Acked 8941 Write 4 Done Done New false
23223 7943 Acked 8942 Write 8 Done Done New false
23224 7944 Acked 8943 Write 10 Done Done New false
23225 7945 Acked 8944 Read 4 Done Done New false
23226 7946 Acked 8945 Write 9 Done Done New false
23227 7947 Acked 8946 Write 3 Done Done New false
23228 7948 Acked 8947 Read 8 Done Done New false
23229 7949 Acked 8948 Flush 0 Done Done New false
23230 7950 Acked 8949 Write 8 Done Done New false
23231 7951 Acked 8950 Read 10 Done Done New false
23232 7952 Acked 8951 Write 7 Done Done New false
23233 7953 Acked 8952 Read 4 Done Done New false
23234 7954 Acked 8953 Write 7 Done Done New false
23235 7955 Acked 8954 Write 7 Done Done New false
23236 7956 Acked 8955 Write 2 Done Done New false
23237 7957 Acked 8956 Write 6 Done Done New false
23238 7958 Acked 8957 Write 10 Done Done New false
23239 7959 Acked 8958 Write 1 Done Done New false
23240 7960 Acked 8959 Flush 0 Done Done New false
23241 7961 Acked 8960 Read 9 Done Done New false
23242 7962 Acked 8961 Read 4 Done Done New false
23243 7963 Acked 8962 Write 7 Done Done New false
23244 7964 Acked 8963 Write 7 Done Done New false
23245 7965 Acked 8964 Write 7 Done Done New false
23246 7966 Acked 8965 Write 1 Done Done New false
23247 7967 Acked 8966 Read 7 Done Done New false
23248 7968 Acked 8967 Read 6 Done Done New false
23249 7969 Acked 8968 Write 5 Done Done New false
23250 7970 Acked 8969 Write 5 Done Done New false
23251 7971 Acked 8970 Write 9 Done Done New false
23252 7972 Acked 8971 Flush 0 Done Done New false
23253 7973 Acked 8972 Write 2 Done Done New false
23254 7974 Acked 8973 Write 2 Done Done New false
23255 7975 Acked 8974 Read 4 Done Done New false
23256 7976 Acked 8975 Write 2 Done Done New false
23257 7977 Acked 8976 Read 7 Done Done New false
23258 7978 Acked 8977 Write 2 Done Done New false
23259 7979 Acked 8978 Flush 0 Done Done New false
23260 7980 Acked 8979 Read 7 Done Done New false
23261 7981 Acked 8980 Read 10 Done Done New false
23262 7982 Acked 8981 Read 4 Done Done New false
23263 7983 Acked 8982 Read 9 Done Done New false
23264 7984 Acked 8983 Read 5 Done Done New false
23265 7985 Acked 8984 Write 5 Done Done New false
23266 7986 Acked 8985 Read 10 Done Done New false
23267 7987 Acked 8986 Read 5 Done Done New false
23268 7988 Acked 8987 Read 7 Done Done New false
23269 7989 Acked 8988 Write 1 Done Done New false
23270 7990 Acked 8989 Write 10 Done Done New false
23271 7991 Acked 8990 Write 5 Done Done New false
23272 7992 Acked 8991 Write 10 Done Done New false
23273 7993 Acked 8992 Read 1 Done Done New false
23274 7994 Acked 8993 Read 5 Done Done New false
23275 7995 Acked 8994 Write 2 Done Done New false
23276 7996 Acked 8995 Write 1 Done Done New false
23277 7997 Acked 8996 Write 8 Done Done New false
23278 7998 Acked 8997 Write 5 Done Done New false
23279 7999 Acked 8998 Write 3 Done Done New false
23280 8000 Acked 8999 Read 5 Done Done New false
23281 8001 Acked 9000 Read 4 Done Done New false
23282 8002 Acked 9001 Write 9 Done Done New false
23283 8003 Acked 9002 Write 9 Done Done New false
23284 8004 Acked 9003 Read 9 Done Done New false
23285 8005 Acked 9004 Read 6 Done Done New false
23286 8006 Acked 9005 Read 1 Done Done New false
23287 8007 Acked 9006 Write 6 Done Done New false
23288 8008 Acked 9007 Read 6 Done Done New false
23289 8009 Acked 9008 Write 7 Done Done New false
23290 8010 Acked 9009 Read 4 Done Done New false
23291 8011 Acked 9010 Read 4 Done Done New false
23292 8012 Acked 9011 Write 3 Done Done New false
23293 8013 Acked 9012 Read 3 Done Done New false
23294 8014 Acked 9013 Flush 0 Done Done New false
23295 8015 Acked 9014 Write 9 Done Done New false
23296 8016 Acked 9015 Read 8 Done Done New false
23297 8017 Acked 9016 Write 1 Done Done New false
23298 8018 Acked 9017 Read 7 Done Done New false
23299 8019 Acked 9018 Read 7 Done Done New false
23300 8020 Acked 9019 Write 7 Done Done New false
23301 8021 Acked 9020 Flush 0 Done Done New false
23302 8022 Acked 9021 Read 6 Done Done New false
23303 8023 Acked 9022 Read 5 Done Done New false
23304 8024 Acked 9023 Read 1 Done Done New false
23305 8025 Acked 9024 Read 1 Done Done New false
23306 8026 Acked 9025 Read 1 Done Done New false
23307 8027 Acked 9026 Read 8 Done Done New false
23308 8028 Acked 9027 Write 4 Done Done New false
23309 8029 Acked 9028 Write 6 Done Done New false
23310 8030 Acked 9029 Write 6 Done Done New false
23311 8031 Acked 9030 Read 3 Done Done New false
23312 8032 Acked 9031 Read 10 Done Done New false
23313 8033 Acked 9032 Write 1 Done Done New false
23314 8034 Acked 9033 Read 9 Done Done New false
23315 8035 Acked 9034 Write 2 Done Done New false
23316 8036 Acked 9035 Read 10 Done Done New false
23317 8037 Acked 9036 Write 1 Done Done New false
23318 8038 Acked 9037 Write 7 Done Done New false
23319 8039 Acked 9038 Flush 0 Done Done New false
23320 8040 Acked 9039 Write 7 Done Done New false
23321 8041 Acked 9040 Write 7 Done Done New false
23322 8042 Acked 9041 Flush 0 Done Done New false
23323 8043 Acked 9042 Read 2 Done Done New false
23324 8044 Acked 9043 Write 10 Done Done New false
23325 8045 Acked 9044 Write 4 Done Done New false
23326 8046 Acked 9045 Write 1 Done Done New false
23327 8047 Acked 9046 Write 5 Done Done New false
23328 8048 Acked 9047 Read 9 Done Done New false
23329 8049 Acked 9048 Read 3 Done Done New false
23330 8050 Acked 9049 Read 7 Done Done New false
23331 8051 Acked 9050 Read 10 Done Done New false
23332 8052 Acked 9051 Read 4 Done Done New false
23333 8053 Acked 9052 Flush 0 Done Done New false
23334 8054 Acked 9053 Flush 0 Done Done New false
23335 8055 Acked 9054 Read 6 Done Done New false
23336 8056 Acked 9055 Read 4 Done Done New false
23337 8057 Acked 9056 Flush 0 Done Done New false
23338 8058 Acked 9057 Write 4 Done Done New false
23339 8059 Acked 9058 Write 2 Done Done New false
23340 8060 Acked 9059 Read 1 Done Done New false
23341 8061 Acked 9060 Read 3 Done Done New false
23342 8062 Acked 9061 Write 10 Done Done New false
23343 8063 Acked 9062 Read 10 Done Done New false
23344 8064 Acked 9063 Read 9 Done Done New false
23345 8065 Acked 9064 Read 4 Done Done New false
23346 8066 Acked 9065 Flush 0 Done Done New false
23347 8067 Acked 9066 Write 3 Done Done New false
23348 8068 Acked 9067 Write 7 Done Done New false
23349 8069 Acked 9068 Write 4 Done Done New false
23350 8070 Acked 9069 Read 6 Done Done New false
23351 8071 Acked 9070 Write 8 Done Done New false
23352 8072 Acked 9071 Read 9 Done Done New false
23353 8073 Acked 9072 Write 9 Done Done New false
23354 8074 Acked 9073 Read 10 Done Done New false
23355 8075 Acked 9074 Flush 0 Done Done New false
23356 8076 Acked 9075 Write 6 Done Done New false
23357 8077 Acked 9076 Read 7 Done Done New false
23358 8078 Acked 9077 Read 4 Done Done New false
23359 8079 Acked 9078 Read 6 Done Done New false
23360 8080 Acked 9079 Write 1 Done Done New false
23361 8081 Acked 9080 Read 2 Done Done New false
23362 8082 Acked 9081 Read 8 Done Done New false
23363 8083 Acked 9082 Write 7 Done Done New false
23364 8084 Acked 9083 Read 8 Done Done New false
23365 8085 Acked 9084 Read 5 Done Done New false
23366 8086 Acked 9085 Write 10 Done Done New false
23367 8087 Acked 9086 Flush 0 Done Done New false
23368 8088 Acked 9087 Write 6 Done Done New false
23369 8089 Acked 9088 Read 3 Done Done New false
23370 8090 Acked 9089 Read 7 Done Done New false
23371 8091 Acked 9090 Write 5 Done Done New false
23372 8092 Acked 9091 Read 3 Done Done New false
23373 8093 Acked 9092 Write 6 Done Done New false
23374 8094 Acked 9093 Read 5 Done Done New false
23375 8095 Acked 9094 Write 7 Done Done New false
23376 8096 Acked 9095 Write 6 Done Done New false
23377 8097 Acked 9096 Write 6 Done Done New false
23378 8098 Acked 9097 Write 7 Done Done New false
23379 8099 Acked 9098 Write 10 Done Done New false
23380 8100 Acked 9099 Read 1 Done Done New false
23381 8101 Acked 9100 Read 3 Done Done New false
23382 8102 Acked 9101 Write 10 Done Done New false
23383 8103 Acked 9102 Read 9 Done Done New false
23384 8104 Acked 9103 Flush 0 Done Done New false
23385 8105 Acked 9104 Write 1 Done Done New false
23386 8106 Acked 9105 Read 5 Done Done New false
23387 8107 Acked 9106 Read 8 Done Done New false
23388 8108 Acked 9107 Write 9 Done Done New false
23389 8109 Acked 9108 Write 2 Done Done New false
23390 8110 Acked 9109 Flush 0 Done Done New false
23391 8111 Acked 9110 Read 1 Done Done New false
23392 8112 Acked 9111 Write 10 Done Done New false
23393 8113 Acked 9112 Read 5 Done Done New false
23394 8114 Acked 9113 Write 5 Done Done New false
23395 8115 Acked 9114 Write 9 Done Done New false
23396 8116 Acked 9115 Flush 0 Done Done New false
23397 8117 Acked 9116 Write 6 Done Done New false
23398 8118 Acked 9117 Write 7 Done Done New false
23399 8119 Acked 9118 Write 2 Done Done New false
23400 8120 Acked 9119 Flush 0 Done Done New false
23401 8121 Acked 9120 Write 3 Done Done New false
23402 8122 Acked 9121 Read 4 Done Done New false
23403 8123 Acked 9122 Read 6 Done Done New false
23404 8124 Acked 9123 Read 3 Done Done New false
23405 8125 Acked 9124 Read 9 Done Done New false
23406 8126 Acked 9125 Read 10 Done Done New false
23407 8127 Acked 9126 Read 4 Done Done New false
23408 8128 Acked 9127 Write 7 Done Done New false
23409 8129 Acked 9128 Read 9 Done Done New false
23410 8130 Acked 9129 Read 7 Done Done New false
23411 8131 Acked 9130 Write 7 Done Done New false
23412 8132 Acked 9131 Write 6 Done Done New false
23413 8133 Acked 9132 Read 9 Done Done New false
23414 8134 Acked 9133 Write 1 Done Done New false
23415 8135 Acked 9134 Read 4 Done Done New false
23416 8136 Acked 9135 Read 3 Done Done New false
23417 8137 Acked 9136 Read 4 Done Done New false
23418 8138 Acked 9137 Read 8 Done Done New false
23419 STATES DS:0 DS:1 DS:2 TOTAL
23420 New 0 0 301 301
23421 Sent 0 0 0 0
23422 Done 301 301 0 602
23423 Skipped 0 0 0 0
23424 Error 0 0 0 0
23425 Last Flush: 9119 9119 8836
23426 Downstairs last five completed: 8836 8835 8834 8833 8832
23427 Upstairs last five completed: 8138 8137 8136 8135 8134
23428 CLIENT: Up:0 ds:301 act:2
234292024-03-02T01:10:16.051ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234302024-03-02T01:10:16.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234312024-03-02T01:10:16.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234322024-03-02T01:10:16.051ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234332024-03-02T01:10:16.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234342024-03-02T01:10:16.051ZINFOcrucible: [2] client re-new 302 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234352024-03-02T01:10:16.051ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234362024-03-02T01:10:17.053ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234372024-03-02T01:10:17.053ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234382024-03-02T01:10:17.053ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234392024-03-02T01:10:17.053ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234402024-03-02T01:10:17.053ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234412024-03-02T01:10:17.053ZINFOcrucible: [2] client re-new 302 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234422024-03-02T01:10:17.054ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234432024-03-02T01:10:18.054ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234442024-03-02T01:10:18.054ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234452024-03-02T01:10:18.054ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234462024-03-02T01:10:18.054ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234472024-03-02T01:10:18.054ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234482024-03-02T01:10:18.054ZINFOcrucible: [2] client re-new 302 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234492024-03-02T01:10:18.055ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23450 [8830] got start action
23451 [8830] I am going to restart
23452 Starting downstairs at port 8830
23453 Make output file at "/tmp/dsc/downstairs-8830.txt"
23454 [8830][2] reports Starting
234552024-03-02T01:10:19.056ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234562024-03-02T01:10:19.056ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234572024-03-02T01:10:19.056ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234582024-03-02T01:10:19.056ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234592024-03-02T01:10:19.056ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234602024-03-02T01:10:19.056ZINFOcrucible: [2] client re-new 302 jobs since flush 8836 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
234612024-03-02T01:10:19.056ZINFOcrucible: client task is exiting = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23462 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1189)
23463 [8830][2] reports Running
23464 ----------------------------------------------------------------
23465 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23466 States: Active Active Offline
23467 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23468 7838 Acked 8837 Read 2 Done Done New true
23469 7839 Acked 8838 Read 9 Done Done New true
23470 7840 Acked 8839 Read 1 Done Done New true
23471 7841 Acked 8840 Read 3 Done Done New true
23472 7842 Acked 8841 Write 5 Done Done New true
23473 7843 Acked 8842 Write 3 Done Done New true
23474 7844 Acked 8843 Read 1 Done Done New true
23475 7845 Acked 8844 Flush 0 Done Done New true
23476 7846 Acked 8845 Read 10 Done Done New true
23477 7847 Acked 8846 Read 8 Done Done New true
23478 7848 Acked 8847 Flush 0 Done Done New true
23479 7849 Acked 8848 Read 5 Done Done New true
23480 7850 Acked 8849 Write 3 Done Done New true
23481 7851 Acked 8850 Flush 0 Done Done New true
23482 7852 Acked 8851 Write 2 Done Done New true
23483 7853 Acked 8852 Read 8 Done Done New true
23484 7854 Acked 8853 Read 5 Done Done New true
23485 7855 Acked 8854 Read 9 Done Done New true
23486 7856 Acked 8855 Write 7 Done Done New true
23487 7857 Acked 8856 Read 5 Done Done New true
23488 7858 Acked 8857 Write 2 Done Done New true
23489 7859 Acked 8858 Write 4 Done Done New true
23490 7860 Acked 8859 Flush 0 Done Done New true
23491 7861 Acked 8860 Read 9 Done Done New true
23492 7862 Acked 8861 Read 6 Done Done New true
23493 7863 Acked 8862 Flush 0 Done Done New true
23494 7864 Acked 8863 Read 3 Done Done New true
23495 7865 Acked 8864 Write 1 Done Done New true
23496 7866 Acked 8865 Read 4 Done Done New true
23497 7867 Acked 8866 Flush 0 Done Done New true
23498 7868 Acked 8867 Read 5 Done Done New true
23499 7869 Acked 8868 Read 8 Done Done New true
23500 7870 Acked 8869 Write 3 Done Done New true
23501 7871 Acked 8870 Read 8 Done Done New true
23502 7872 Acked 8871 Read 3 Done Done New true
23503 7873 Acked 8872 Write 2 Done Done New true
23504 7874 Acked 8873 Read 9 Done Done New true
23505 7875 Acked 8874 Read 7 Done Done New true
23506 7876 Acked 8875 Write 3 Done Done New true
23507 7877 Acked 8876 Flush 0 Done Done New true
23508 7878 Acked 8877 Read 8 Done Done New true
23509 7879 Acked 8878 Flush 0 Done Done New true
23510 7880 Acked 8879 Read 9 Done Done New true
23511 7881 Acked 8880 Write 4 Done Done New true
23512 7882 Acked 8881 Write 4 Done Done New true
23513 7883 Acked 8882 Write 3 Done Done New true
23514 7884 Acked 8883 Read 2 Done Done New true
23515 7885 Acked 8884 Flush 0 Done Done New true
23516 7886 Acked 8885 Read 10 Done Done New true
23517 7887 Acked 8886 Read 3 Done Done New true
23518 7888 Acked 8887 Write 2 Done Done New true
23519 7889 Acked 8888 Read 5 Done Done New true
23520 7890 Acked 8889 Write 8 Done Done New true
23521 7891 Acked 8890 Read 2 Done Done New true
23522 7892 Acked 8891 Read 8 Done Done New true
23523 7893 Acked 8892 Write 1 Done Done New true
23524 7894 Acked 8893 Read 1 Done Done New true
23525 7895 Acked 8894 Write 5 Done Done New true
23526 7896 Acked 8895 Write 4 Done Done New true
23527 7897 Acked 8896 Read 1 Done Done New true
23528 7898 Acked 8897 Read 6 Done Done New true
23529 7899 Acked 8898 Read 8 Done Done New true
23530 7900 Acked 8899 Read 7 Done Done New true
23531 7901 Acked 8900 Read 9 Done Done New true
23532 7902 Acked 8901 Read 8 Done Done New true
23533 7903 Acked 8902 Read 5 Done Done New true
23534 7904 Acked 8903 Read 10 Done Done New true
23535 7905 Acked 8904 Read 1 Done Done New true
23536 7906 Acked 8905 Write 3 Done Done New true
23537 7907 Acked 8906 Read 5 Done Done New true
23538 7908 Acked 8907 Read 5 Done Done New true
23539 7909 Acked 8908 Write 2 Done Done New true
23540 7910 Acked 8909 Write 4 Done Done New true
23541 7911 Acked 8910 Read 3 Done Done New true
23542 7912 Acked 8911 Read 3 Done Done New true
23543 7913 Acked 8912 Read 1 Done Done New true
23544 7914 Acked 8913 Read 1 Done Done New true
23545 7915 Acked 8914 Read 7 Done Done New true
23546 7916 Acked 8915 Read 3 Done Done New true
23547 7917 Acked 8916 Read 1 Done Done New true
23548 7918 Acked 8917 Write 1 Done Done New true
23549 7919 Acked 8918 Read 10 Done Done New true
23550 7920 Acked 8919 Flush 0 Done Done New true
23551 7921 Acked 8920 Flush 0 Done Done New true
23552 7922 Acked 8921 Read 6 Done Done New true
23553 7923 Acked 8922 Flush 0 Done Done New true
23554 7924 Acked 8923 Read 3 Done Done New true
23555 7925 Acked 8924 Read 2 Done Done New true
23556 7926 Acked 8925 Write 5 Done Done New true
23557 7927 Acked 8926 Write 5 Done Done New true
23558 7928 Acked 8927 Read 10 Done Done New true
23559 7929 Acked 8928 Write 4 Done Done New true
23560 7930 Acked 8929 Read 7 Done Done New true
23561 7931 Acked 8930 Write 10 Done Done New true
23562 7932 Acked 8931 Read 4 Done Done New true
23563 7933 Acked 8932 Write 3 Done Done New true
23564 7934 Acked 8933 Read 3 Done Done New true
23565 7935 Acked 8934 Read 2 Done Done New true
23566 7936 Acked 8935 Write 3 Done Done New true
23567 7937 Acked 8936 Read 9 Done Done New true
23568 7938 Acked 8937 Write 3 Done Done New true
23569 7939 Acked 8938 Flush 0 Done Done New true
23570 7940 Acked 8939 Read 6 Done Done New true
23571 7941 Acked 8940 Read 7 Done Done New true
23572 7942 Acked 8941 Write 4 Done Done New true
23573 7943 Acked 8942 Write 8 Done Done New true
23574 7944 Acked 8943 Write 10 Done Done New true
23575 7945 Acked 8944 Read 4 Done Done New true
23576 7946 Acked 8945 Write 9 Done Done New true
23577 7947 Acked 8946 Write 3 Done Done New true
23578 7948 Acked 8947 Read 8 Done Done New true
23579 7949 Acked 8948 Flush 0 Done Done New true
23580 7950 Acked 8949 Write 8 Done Done New true
23581 7951 Acked 8950 Read 10 Done Done New true
23582 7952 Acked 8951 Write 7 Done Done New true
23583 7953 Acked 8952 Read 4 Done Done New true
23584 7954 Acked 8953 Write 7 Done Done New true
23585 7955 Acked 8954 Write 7 Done Done New true
23586 7956 Acked 8955 Write 2 Done Done New true
23587 7957 Acked 8956 Write 6 Done Done New true
23588 7958 Acked 8957 Write 10 Done Done New true
23589 7959 Acked 8958 Write 1 Done Done New true
23590 7960 Acked 8959 Flush 0 Done Done New true
23591 7961 Acked 8960 Read 9 Done Done New true
23592 7962 Acked 8961 Read 4 Done Done New true
23593 7963 Acked 8962 Write 7 Done Done New true
23594 7964 Acked 8963 Write 7 Done Done New true
23595 7965 Acked 8964 Write 7 Done Done New true
23596 7966 Acked 8965 Write 1 Done Done New true
23597 7967 Acked 8966 Read 7 Done Done New true
23598 7968 Acked 8967 Read 6 Done Done New true
23599 7969 Acked 8968 Write 5 Done Done New true
23600 7970 Acked 8969 Write 5 Done Done New true
23601 7971 Acked 8970 Write 9 Done Done New true
23602 7972 Acked 8971 Flush 0 Done Done New true
23603 7973 Acked 8972 Write 2 Done Done New true
23604 7974 Acked 8973 Write 2 Done Done New true
23605 7975 Acked 8974 Read 4 Done Done New true
23606 7976 Acked 8975 Write 2 Done Done New true
23607 7977 Acked 8976 Read 7 Done Done New true
23608 7978 Acked 8977 Write 2 Done Done New true
23609 7979 Acked 8978 Flush 0 Done Done New true
23610 7980 Acked 8979 Read 7 Done Done New true
23611 7981 Acked 8980 Read 10 Done Done New true
23612 7982 Acked 8981 Read 4 Done Done New true
23613 7983 Acked 8982 Read 9 Done Done New true
23614 7984 Acked 8983 Read 5 Done Done New true
23615 7985 Acked 8984 Write 5 Done Done New true
23616 7986 Acked 8985 Read 10 Done Done New true
23617 7987 Acked 8986 Read 5 Done Done New true
23618 7988 Acked 8987 Read 7 Done Done New true
23619 7989 Acked 8988 Write 1 Done Done New true
23620 7990 Acked 8989 Write 10 Done Done New true
23621 7991 Acked 8990 Write 5 Done Done New true
23622 7992 Acked 8991 Write 10 Done Done New true
23623 7993 Acked 8992 Read 1 Done Done New true
23624 7994 Acked 8993 Read 5 Done Done New true
23625 7995 Acked 8994 Write 2 Done Done New true
23626 7996 Acked 8995 Write 1 Done Done New true
23627 7997 Acked 8996 Write 8 Done Done New true
23628 7998 Acked 8997 Write 5 Done Done New true
23629 7999 Acked 8998 Write 3 Done Done New true
23630 8000 Acked 8999 Read 5 Done Done New true
23631 8001 Acked 9000 Read 4 Done Done New true
23632 8002 Acked 9001 Write 9 Done Done New true
23633 8003 Acked 9002 Write 9 Done Done New true
23634 8004 Acked 9003 Read 9 Done Done New true
23635 8005 Acked 9004 Read 6 Done Done New true
23636 8006 Acked 9005 Read 1 Done Done New true
23637 8007 Acked 9006 Write 6 Done Done New true
23638 8008 Acked 9007 Read 6 Done Done New true
23639 8009 Acked 9008 Write 7 Done Done New true
23640 8010 Acked 9009 Read 4 Done Done New true
23641 8011 Acked 9010 Read 4 Done Done New true
23642 8012 Acked 9011 Write 3 Done Done New true
23643 8013 Acked 9012 Read 3 Done Done New true
23644 8014 Acked 9013 Flush 0 Done Done New true
23645 8015 Acked 9014 Write 9 Done Done New true
23646 8016 Acked 9015 Read 8 Done Done New true
23647 8017 Acked 9016 Write 1 Done Done New true
23648 8018 Acked 9017 Read 7 Done Done New true
23649 8019 Acked 9018 Read 7 Done Done New true
23650 8020 Acked 9019 Write 7 Done Done New true
23651 8021 Acked 9020 Flush 0 Done Done New true
23652 8022 Acked 9021 Read 6 Done Done New true
23653 8023 Acked 9022 Read 5 Done Done New true
23654 8024 Acked 9023 Read 1 Done Done New true
23655 8025 Acked 9024 Read 1 Done Done New true
23656 8026 Acked 9025 Read 1 Done Done New true
23657 8027 Acked 9026 Read 8 Done Done New true
23658 8028 Acked 9027 Write 4 Done Done New true
23659 8029 Acked 9028 Write 6 Done Done New true
23660 8030 Acked 9029 Write 6 Done Done New true
23661 8031 Acked 9030 Read 3 Done Done New true
23662 8032 Acked 9031 Read 10 Done Done New true
23663 8033 Acked 9032 Write 1 Done Done New true
23664 8034 Acked 9033 Read 9 Done Done New true
23665 8035 Acked 9034 Write 2 Done Done New true
23666 8036 Acked 9035 Read 10 Done Done New true
23667 8037 Acked 9036 Write 1 Done Done New true
23668 8038 Acked 9037 Write 7 Done Done New true
23669 8039 Acked 9038 Flush 0 Done Done New true
23670 8040 Acked 9039 Write 7 Done Done New true
23671 8041 Acked 9040 Write 7 Done Done New true
23672 8042 Acked 9041 Flush 0 Done Done New true
23673 8043 Acked 9042 Read 2 Done Done New true
23674 8044 Acked 9043 Write 10 Done Done New true
23675 8045 Acked 9044 Write 4 Done Done New true
23676 8046 Acked 9045 Write 1 Done Done New true
23677 8047 Acked 9046 Write 5 Done Done New true
23678 8048 Acked 9047 Read 9 Done Done New true
23679 8049 Acked 9048 Read 3 Done Done New true
23680 8050 Acked 9049 Read 7 Done Done New true
23681 8051 Acked 9050 Read 10 Done Done New true
23682 8052 Acked 9051 Read 4 Done Done New true
23683 8053 Acked 9052 Flush 0 Done Done New true
23684 8054 Acked 9053 Flush 0 Done Done New true
23685 8055 Acked 9054 Read 6 Done Done New true
23686 8056 Acked 9055 Read 4 Done Done New true
23687 8057 Acked 9056 Flush 0 Done Done New true
23688 8058 Acked 9057 Write 4 Done Done New true
23689 8059 Acked 9058 Write 2 Done Done New true
23690 8060 Acked 9059 Read 1 Done Done New true
23691 8061 Acked 9060 Read 3 Done Done New true
23692 8062 Acked 9061 Write 10 Done Done New true
23693 8063 Acked 9062 Read 10 Done Done New true
23694 8064 Acked 9063 Read 9 Done Done New true
23695 8065 Acked 9064 Read 4 Done Done New true
23696 8066 Acked 9065 Flush 0 Done Done New true
23697 8067 Acked 9066 Write 3 Done Done New true
23698 8068 Acked 9067 Write 7 Done Done New true
23699 8069 Acked 9068 Write 4 Done Done New true
23700 8070 Acked 9069 Read 6 Done Done New true
23701 8071 Acked 9070 Write 8 Done Done New true
23702 8072 Acked 9071 Read 9 Done Done New true
23703 8073 Acked 9072 Write 9 Done Done New true
23704 8074 Acked 9073 Read 10 Done Done New true
23705 8075 Acked 9074 Flush 0 Done Done New true
23706 8076 Acked 9075 Write 6 Done Done New true
23707 8077 Acked 9076 Read 7 Done Done New true
23708 8078 Acked 9077 Read 4 Done Done New true
23709 8079 Acked 9078 Read 6 Done Done New true
23710 8080 Acked 9079 Write 1 Done Done New true
23711 8081 Acked 9080 Read 2 Done Done New true
23712 8082 Acked 9081 Read 8 Done Done New true
23713 8083 Acked 9082 Write 7 Done Done New true
23714 8084 Acked 9083 Read 8 Done Done New true
23715 8085 Acked 9084 Read 5 Done Done New true
23716 8086 Acked 9085 Write 10 Done Done New true
23717 8087 Acked 9086 Flush 0 Done Done New true
23718 8088 Acked 9087 Write 6 Done Done New true
23719 8089 Acked 9088 Read 3 Done Done New true
23720 8090 Acked 9089 Read 7 Done Done New true
23721 8091 Acked 9090 Write 5 Done Done New true
23722 8092 Acked 9091 Read 3 Done Done New true
23723 8093 Acked 9092 Write 6 Done Done New true
23724 8094 Acked 9093 Read 5 Done Done New true
23725 8095 Acked 9094 Write 7 Done Done New true
23726 8096 Acked 9095 Write 6 Done Done New true
23727 8097 Acked 9096 Write 6 Done Done New true
23728 8098 Acked 9097 Write 7 Done Done New true
23729 8099 Acked 9098 Write 10 Done Done New true
23730 8100 Acked 9099 Read 1 Done Done New true
23731 8101 Acked 9100 Read 3 Done Done New true
23732 8102 Acked 9101 Write 10 Done Done New true
23733 8103 Acked 9102 Read 9 Done Done New true
23734 8104 Acked 9103 Flush 0 Done Done New true
23735 8105 Acked 9104 Write 1 Done Done New true
23736 8106 Acked 9105 Read 5 Done Done New true
23737 8107 Acked 9106 Read 8 Done Done New true
23738 8108 Acked 9107 Write 9 Done Done New true
23739 8109 Acked 9108 Write 2 Done Done New true
23740 8110 Acked 9109 Flush 0 Done Done New true
23741 8111 Acked 9110 Read 1 Done Done New true
23742 8112 Acked 9111 Write 10 Done Done New true
23743 8113 Acked 9112 Read 5 Done Done New true
23744 8114 Acked 9113 Write 5 Done Done New true
23745 8115 Acked 9114 Write 9 Done Done New true
23746 8116 Acked 9115 Flush 0 Done Done New true
23747 8117 Acked 9116 Write 6 Done Done New true
23748 8118 Acked 9117 Write 7 Done Done New true
23749 8119 Acked 9118 Write 2 Done Done New true
23750 8120 Acked 9119 Flush 0 Done Done New true
23751 8121 Acked 9120 Write 3 Done Done New true
23752 8122 Acked 9121 Read 4 Done Done New true
23753 8123 Acked 9122 Read 6 Done Done New true
23754 8124 Acked 9123 Read 3 Done Done New true
23755 8125 Acked 9124 Read 9 Done Done New true
23756 8126 Acked 9125 Read 10 Done Done New true
23757 8127 Acked 9126 Read 4 Done Done New true
23758 8128 Acked 9127 Write 7 Done Done New true
23759 8129 Acked 9128 Read 9 Done Done New true
23760 8130 Acked 9129 Read 7 Done Done New true
23761 8131 Acked 9130 Write 7 Done Done New true
23762 8132 Acked 9131 Write 6 Done Done New true
23763 8133 Acked 9132 Read 9 Done Done New true
23764 8134 Acked 9133 Write 1 Done Done New true
23765 8135 Acked 9134 Read 4 Done Done New true
23766 8136 Acked 9135 Read 3 Done Done New true
23767 8137 Acked 9136 Read 4 Done Done New true
23768 8138 Acked 9137 Read 8 Done Done New true
23769 8139 Acked 9138 Flush 0 Done Done New true
23770 STATES DS:0 DS:1 DS:2 TOTAL
23771 New 0 0 302 302
23772 Sent 0 0 0 0
23773 Done 302 302 0 604
23774 Skipped 0 0 0 0
23775 Error 0 0 0 0
23776 Last Flush: 9138 9138 8836
23777 Downstairs last five completed: 8836 8835 8834 8833 8832
23778 Upstairs last five completed: 8139 8138 8137 8136 8135
23779 CLIENT: Up:0 ds:302 act:2
237802024-03-02T01:10:20.057ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237812024-03-02T01:10:20.057ZINFOcrucible: ds_connection connected = io task client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237822024-03-02T01:10:20.058ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237832024-03-02T01:10:20.058ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237842024-03-02T01:10:20.058ZINFOcrucible: send last flush ID to this DS: 8836 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237852024-03-02T01:10:20.059ZINFOcrucible: Replied this last flush ID: 8836 = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237862024-03-02T01:10:20.059ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237872024-03-02T01:10:20.059ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237882024-03-02T01:10:20.059ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
237892024-03-02T01:10:20.059ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23790 ----------------------------------------------------------------
23791 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23792 Downstairs last five completed: 9138 9137 9136 9135 9134
23793 Upstairs last five completed: 8139 8138 8137 8136 8135
23794 CLIENT: Up:0 ds:0 act:3
23795 Replay: All jobs finished, all DS active.
23796 Main task has work to do, go find it
23797 got dsc Stop(0)
23798 stop 0
23799 Mar 02 01:10:23.395 INFO request completed, latency_us: 337, response_code: 204[8810] Got stop action so:false kr:false
23800 , uri: /stop/cid/0, method: POST, req_id: 9cff9424-80da-448f-8c9c-42a703e68cdc, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
23801 Mar 02 01:10:23.396 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0be6d8d5-6e79-4668-bef1-b576e11a595d, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
238022024-03-02T01:10:23.396ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238032024-03-02T01:10:23.396ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238042024-03-02T01:10:23.396ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238052024-03-02T01:10:23.396ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238062024-03-02T01:10:23.396ZINFOcrucible: [0] client re-new 0 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238072024-03-02T01:10:23.396ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23808 [8810] Exited with: ExitStatus(unix_wait_status(9))
23809 [8810][0] reports Exit
238102024-03-02T01:10:24.397ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238112024-03-02T01:10:24.397ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238122024-03-02T01:10:24.397ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238132024-03-02T01:10:24.397ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238142024-03-02T01:10:24.397ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238152024-03-02T01:10:24.398ZINFOcrucible: [0] client re-new 0 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238162024-03-02T01:10:24.398ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238172024-03-02T01:10:25.398ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238182024-03-02T01:10:25.398ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238192024-03-02T01:10:25.398ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238202024-03-02T01:10:25.398ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238212024-03-02T01:10:25.398ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238222024-03-02T01:10:25.399ZINFOcrucible: [0] client re-new 0 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238232024-03-02T01:10:25.399ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238242024-03-02T01:10:26.399ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238252024-03-02T01:10:26.399ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238262024-03-02T01:10:26.399ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238272024-03-02T01:10:26.399ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238282024-03-02T01:10:26.399ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238292024-03-02T01:10:26.399ZINFOcrucible: [0] client re-new 0 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238302024-03-02T01:10:26.399ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23831 Mar 02 01:10:27.398 INFO request completed, latency_us: 185, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0730cfe1-99bc-45fc-9fd8-cc8795a2dbcd, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
23832 001/300 Read block 3811 len 8192
23833 002/300 Write block 1535 len 12288 data: 5 4 3
23834 003/300 Flush
238352024-03-02T01:10:27.400ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238362024-03-02T01:10:27.400ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238372024-03-02T01:10:27.400ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238382024-03-02T01:10:27.400ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238392024-03-02T01:10:27.400ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238402024-03-02T01:10:27.400ZINFOcrucible: [0] client re-new 3 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
238412024-03-02T01:10:27.400ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
23842 004/300 Write block 3107 len 40960 data: 4 4 4 3 3 4 3 3 5 6
23843 005/300 Flush
23844 006/300 Write block 4184 len 16384 data: 6 8 10 9
23845 007/300 Read block 3310 len 40960
23846 008/300 Write block 2009 len 8192 data: 5 3
23847 009/300 Read block 528 len 4096
23848 010/300 Read block 1705 len 32768
23849 011/300 Read block 2471 len 8192
23850 012/300 Write block 609 len 8192 data: 7 8
23851 013/300 Write block 1646 len 8192 data: 5 5
23852 014/300 Write block 4677 len 36864 data: 5 5 7 7 6 8 9 9 8
23853 015/300 Read block 765 len 28672
23854 016/300 Read block 3702 len 40960
23855 017/300 Read block 28 len 28672
23856 018/300 Read block 1081 len 16384
23857 019/300 Flush
23858 020/300 Write block 2654 len 32768 data: 6 5 6 7 5 6 7 8
23859 021/300 Read block 201 len 28672
23860 022/300 Flush
23861 023/300 Write block 4741 len 32768 data: 5 5 5 4 4 3 2 2
23862 024/300 Read block 1230 len 24576
23863 025/300 Read block 1691 len 32768
23864 026/300 Read block 458 len 4096
23865 027/300 Read block 2451 len 16384
23866 028/300 Read block 1905 len 16384
23867 029/300 Flush
23868 030/300 Read block 1799 len 40960
23869 031/300 Write block 3451 len 12288 data: 8 9 8
23870 032/300 Read block 4187 len 28672
23871 033/300 Write block 1666 len 28672 data: 6 6 5 4 4 4 5
23872 034/300 Write block 782 len 36864 data: 2 2 2 3 7 5 4 4 5
23873 035/300 Read block 2488 len 28672
23874 036/300 Write block 2312 len 20480 data: 5 7 9 7 6
23875 037/300 Flush
23876 038/300 Read block 3746 len 20480
23877 039/300 Read block 2300 len 8192
23878 040/300 Read block 3944 len 24576
23879 041/300 Read block 3359 len 16384
23880 042/300 Read block 3498 len 24576
23881 043/300 Write block 1777 len 24576 data: 5 6 6 5 6 6
23882 044/300 Read block 3156 len 20480
23883 045/300 Read block 1500 len 40960
23884 046/300 Write block 3546 len 32768 data: 5 4 3 3 6 8 10 8
23885 047/300 Read block 2622 len 12288
23886 048/300 Write block 3792 len 40960 data: 4 2 2 2 3 2 2 2 2 3
23887 049/300 Read block 1179 len 20480
23888 050/300 Read block 1098 len 4096
23889 051/300 Read block 1054 len 16384
23890 052/300 Write block 2344 len 32768 data: 3 3 2 2 2 2 3 3
23891 053/300 Read block 1436 len 16384
23892 054/300 Write block 1284 len 32768 data: 3 2 2 3 3 4 4 4
23893 055/300 Read block 2542 len 12288
23894 056/300 Read block 2289 len 12288
23895 057/300 Read block 3805 len 16384
23896 058/300 Flush
23897 059/300 Write block 2084 len 20480 data: 6 5 4 5 6
23898 060/300 Flush
23899 061/300 Read block 281 len 20480
23900 062/300 Write block 1346 len 40960 data: 5 4 4 4 5 4 4 4 4 5
23901 063/300 Read block 1064 len 28672
23902 064/300 Read block 4500 len 36864
23903 065/300 Flush
23904 066/300 Flush
23905 067/300 Write block 229 len 16384 data: 6 7 6 7
23906 068/300 Read block 716 len 20480
23907 069/300 Write block 3389 len 16384 data: 6 5 5 5
23908 070/300 Read block 3902 len 4096
23909 071/300 Read block 2459 len 40960
23910 072/300 Read block 633 len 12288
23911 073/300 Read block 1685 len 12288
23912 074/300 Flush
23913 075/300 Read block 4914 len 16384
23914 076/300 Write block 520 len 36864 data: 4 4 4 6 4 6 5 6 7
23915 077/300 Write block 1307 len 32768 data: 4 3 4 4 4 4 4 4
23916 078/300 Flush
23917 079/300 Read block 1839 len 28672
23918 080/300 Write block 1161 len 12288 data: 7 4 3
23919 081/300 Read block 3831 len 8192
23920 082/300 Read block 2841 len 36864
23921 083/300 Read block 4806 len 32768
23922 084/300 Write block 1654 len 12288 data: 5 7 5
23923 085/300 Read block 1793 len 8192
23924 086/300 Write block 4079 len 8192 data: 6 6
23925 087/300 Read block 2556 len 8192
23926 088/300 Read block 933 len 32768
23927 089/300 Write block 1387 len 40960 data: 5 5 4 4 4 4 2 2 3 3
23928 090/300 Write block 272 len 20480 data: 8 7 7 5 7
23929 091/300 Write block 1106 len 16384 data: 4 4 5 5
23930 092/300 Read block 4832 len 24576
23931 093/300 Write block 4751 len 36864 data: 2 3 3 4 7 7 7 7 8
23932 094/300 Read block 2973 len 4096
23933 095/300 Read block 4484 len 16384
23934 096/300 Read block 140 len 28672
23935 097/300 Read block 1415 len 12288
23936 098/300 Read block 2781 len 28672
23937 099/300 Read block 4738 len 16384
23938 100/300 Read block 960 len 40960
23939 101/300 Write block 3627 len 20480 data: 6 5 6 6 7
23940 102/300 Flush
23941 103/300 Flush
23942 104/300 Write block 3291 len 24576 data: 4 4 5 5 5 7
23943 105/300 Read block 367 len 40960
23944 106/300 Read block 2437 len 40960
23945 107/300 Flush
23946 108/300 Write block 2429 len 20480 data: 7 7 9 8 7
23947 109/300 Read block 4993 len 20480
23948 110/300 Read block 2211 len 16384
23949 111/300 Read block 1632 len 12288
23950 112/300 Read block 359 len 28672
23951 113/300 Read block 4543 len 36864
23952 114/300 Write block 3197 len 32768 data: 3 2 2 2 3 5 5 5
23953 115/300 Write block 3084 len 16384 data: 4 4 3 3
23954 116/300 Write block 816 len 36864 data: 7 8 8 7 6 7 5 3 5
23955 117/300 Read block 2315 len 28672
23956 118/300 Read block 1172 len 28672
23957 119/300 Read block 3515 len 40960
23958 120/300 Read block 1952 len 32768
23959 121/300 Read block 1730 len 4096
23960 122/300 Flush
23961 123/300 Read block 3583 len 28672
23962 124/300 Read block 1520 len 36864
23963 125/300 Write block 4954 len 16384 data: 10 8 6 6
23964 126/300 Read block 3610 len 28672
23965 127/300 Read block 2689 len 12288
23966 128/300 Write block 788 len 16384 data: 5 5 6 4
23967 129/300 Read block 443 len 24576
23968 130/300 Read block 2043 len 28672
23969 131/300 Read block 4622 len 20480
23970 132/300 Write block 277 len 4096 data: 7
23971 133/300 Write block 4231 len 20480 data: 6 6 7 8 6
23972 134/300 Read block 2060 len 28672
23973 135/300 Write block 4610 len 4096 data: 2
23974 136/300 Read block 1417 len 20480
23975 137/300 Read block 1798 len 16384
23976 138/300 Read block 176 len 16384
23977 139/300 Write block 1242 len 24576 data: 5 5 6 6 5 5
23978 140/300 Flush
23979 141/300 Read block 3527 len 8192
23980 142/300 Write block 2561 len 12288 data: 9 8 8
23981 143/300 Flush
23982 144/300 Write block 4343 len 36864 data: 4 5 7 7 7 6 6 7 7
23983 145/300 Read block 4228 len 8192
23984 146/300 Write block 3717 len 32768 data: 9 8 6 6 6 7 5 5
23985 147/300 Read block 985 len 8192
23986 148/300 Write block 2217 len 12288 data: 6 10 8
23987 149/300 Read block 2637 len 32768
23988 150/300 Read block 248 len 36864
23989 151/300 Read block 3639 len 20480
23990 152/300 Read block 603 len 20480
23991 153/300 Write block 1934 len 32768 data: 6 7 7 8 7 9 8 9
23992 154/300 Read block 1703 len 20480
23993 155/300 Write block 3439 len 8192 data: 4 5
23994 156/300 Flush
23995 157/300 Write block 383 len 32768 data: 4 4 4 6 8 9 9 10
23996 158/300 Read block 2019 len 20480
23997 159/300 Write block 2527 len 28672 data: 6 6 5 5 5 6 6
23998 160/300 Write block 2110 len 32768 data: 3 2 2 2 6 6 7 7
23999 161/300 Read block 2712 len 36864
24000 162/300 Read block 4135 len 4096
24001 163/300 Write block 1658 len 8192 data: 4 3
24002 164/300 Write block 3539 len 12288 data: 7 6 5
24003 165/300 Write block 1761 len 4096 data: 4
24004 166/300 Write block 99 len 36864 data: 3 3 4 4 5 6 4 6 6
24005 167/300 Read block 1713 len 28672
24006 168/300 Read block 2332 len 32768
24007 169/300 Write block 1194 len 24576 data: 5 5 5 7 5 5
24008 170/300 Write block 1150 len 8192 data: 4 5
24009 171/300 Read block 905 len 24576
24010 172/300 Write block 2991 len 8192 data: 7 5
24011 173/300 Read block 3918 len 28672
24012 174/300 Read block 3669 len 24576
24013 175/300 Write block 2966 len 12288 data: 6 6 6
24014 176/300 Write block 955 len 40960 data: 6 4 4 4 5 6 7 7 7 6
24015 177/300 Write block 562 len 8192 data: 5 4
24016 178/300 Flush
24017 179/300 Read block 2392 len 20480
24018 180/300 Read block 2827 len 12288
24019 181/300 Write block 465 len 24576 data: 5 5 8 8 9 8
24020 182/300 Write block 4111 len 32768 data: 3 3 3 3 3 3 3 3
24021 183/300 Read block 3682 len 24576
24022 184/300 Write block 3927 len 4096 data: 11
24023 185/300 Read block 1207 len 28672
24024 186/300 Flush
24025 187/300 Read block 1085 len 16384
24026 188/300 Write block 79 len 32768 data: 5 4 4 7 7 6 5 5
24027 189/300 Read block 2128 len 24576
24028 190/300 Write block 2454 len 28672 data: 5 5 7 6 7 7 9
24029 191/300 Read block 2240 len 32768
24030 192/300 Read block 4454 len 32768
24031 193/300 Write block 1947 len 8192 data: 8 7
24032 194/300 Flush
24033 195/300 Read block 4308 len 28672
24034 196/300 Write block 4357 len 8192 data: 6 5
24035 197/300 Write block 4665 len 12288 data: 6 5 3
24036 198/300 Write block 616 len 32768 data: 8 7 6 7 8 6 7 7
24037 199/300 Write block 3708 len 4096 data: 9
24038 200/300 Flush
24039 201/300 Write block 1831 len 24576 data: 5 6 8 8 8 6
24040 202/300 Write block 1632 len 32768 data: 5 6 5 5 7 6 6 6
24041 203/300 Write block 1203 len 8192 data: 8 8
24042 204/300 Write block 1 len 36864 data: 4 4 4 4 4 6 6 5 5
24043 205/300 Flush
24044 206/300 Write block 4369 len 32768 data: 4 4 5 6 5 5 5 4
24045 207/300 Write block 2431 len 36864 data: 10 9 8 8 8 7 7 6 7
24046 208/300 Write block 410 len 4096 data: 4
24047 209/300 Read block 1113 len 36864
24048 210/300 Write block 491 len 40960 data: 4 3 6 5 6 3 4 4 4 4
24049 211/300 Read block 3662 len 16384
24050 212/300 Flush
24051 213/300 Write block 2958 len 24576 data: 8 9 7 7 6 5
24052 214/300 Read block 3277 len 28672
24053 215/300 Read block 752 len 36864
24054 216/300 Read block 2394 len 36864
24055 217/300 Read block 1936 len 40960
24056 218/300 Read block 2268 len 16384
24057 219/300 Read block 1124 len 24576
24058 220/300 Flush
24059 221/300 Read block 4253 len 4096
24060 222/300 Write block 3976 len 8192 data: 5 7
24061 223/300 Write block 4352 len 32768 data: 7 8 8 8 6 7 6 5
24062 224/300 Write block 1091 len 12288 data: 6 6 6
24063 225/300 Read block 3389 len 24576
24064 226/300 Write block 4878 len 20480 data: 5 5 5 7 6
24065 227/300 Read block 3857 len 4096
24066 228/300 Read block 3350 len 24576
24067 229/300 Write block 381 len 8192 data: 4 5
24068 230/300 Write block 3084 len 32768 data: 5 5 4 4 2 4 4 4
24069 231/300 Read block 4611 len 20480
24070 232/300 Write block 4952 len 36864 data: 6 9 11 9 7 7 6 5 5
24071 233/300 Write block 1452 len 28672 data: 6 7 5 3 2 2 3
24072 234/300 Write block 422 len 28672 data: 4 5 6 5 4 6 5
24073 235/300 Read block 2053 len 32768
24074 236/300 Read block 4660 len 36864
24075 237/300 Flush
24076 238/300 Read block 2374 len 32768
24077 239/300 Write block 572 len 40960 data: 5 6 7 6 8 8 6 7 4 4
24078 240/300 Read block 3005 len 20480
24079 241/300 Write block 4907 len 24576 data: 6 7 7 7 7 6
24080 242/300 Write block 2249 len 24576 data: 8 8 9 11 8 7
24081 243/300 Read block 4857 len 36864
24082 244/300 Write block 1177 len 12288 data: 5 6 6
24083 245/300 Write block 3542 len 24576 data: 6 5 6 5 6 5
24084 246/300 Flush
24085 247/300 Read block 2873 len 16384
24086 248/300 Write block 2356 len 24576 data: 4 6 5 8 8 8
24087 249/300 Flush
24088 250/300 Write block 2726 len 40960 data: 6 5 6 4 3 3 5 4 6 7
24089 251/300 Read block 4689 len 20480
24090 252/300 Read block 933 len 36864
24091 253/300 Read block 216 len 8192
24092 254/300 Read block 1974 len 12288
24093 255/300 Read block 1576 len 8192
24094 256/300 Read block 4571 len 24576
24095 257/300 Write block 1609 len 8192 data: 8 6
24096 258/300 Write block 4549 len 16384 data: 2 2 2 2
24097 259/300 Flush
24098 260/300 Flush
24099 261/300 Read block 3320 len 4096
24100 262/300 Read block 3053 len 12288
24101 263/300 Flush
24102 264/300 Write block 2144 len 20480 data: 6 7 7 6 5
24103 265/300 Write block 142 len 32768 data: 4 3 3 4 4 5 4 5
24104 266/300 Read block 430 len 20480
24105 267/300 Read block 4415 len 24576
24106 268/300 Read block 1212 len 8192
24107 269/300 Write block 2163 len 32768 data: 3 3 4 6 7 6 7 7
24108 270/300 Write block 3395 len 20480 data: 6 6 6 7 4
24109 271/300 Write block 458 len 8192 data: 4 4
24110 272/300 Write block 4521 len 4096 data: 5
24111 273/300 Read block 752 len 28672
24112 274/300 Write block 1874 len 36864 data: 4 4 4 4 3 2 2 2 2
24113 275/300 Write block 3521 len 16384 data: 5 7 9 10
24114 276/300 Read block 3958 len 12288
24115 277/300 Read block 2315 len 12288
24116 278/300 Read block 2023 len 28672
24117 279/300 Read block 2369 len 12288
24118 280/300 Write block 2058 len 24576 data: 4 4 6 6 7 6
24119 281/300 Read block 1124 len 4096
24120 282/300 Read block 3050 len 36864
24121 283/300 Read block 4259 len 12288
24122 284/300 Read block 3740 len 8192
24123 285/300 Write block 3172 len 4096 data: 5
24124 286/300 Write block 2606 len 20480 data: 4 6 5 4 4
24125 287/300 Write block 1349 len 28672 data: 5 6 5 5 5 5 6
24126 288/300 Read block 613 len 24576
24127 289/300 Read block 4237 len 32768
24128 290/300 Write block 681 len 16384 data: 6 7 7 7
24129 291/300 Flush
24130 292/300 Read block 2405 len 40960
24131 293/300 Write block 3991 len 28672 data: 7 6 4 5 6 7 7
24132 294/300 Read block 3052 len 36864
24133 295/300 Read block 2865 len 4096
24134 296/300 Flush
24135 297/300 Write block 1188 len 8192 data: 3 4
24136 298/300 Read block 3773 len 4096
24137 299/300 Write block 658 len 20480 data: 3 3 3 6 8
24138 300/300 Read block 1527 len 24576
24139 Main task has work to do, go find it
24140 got dsc Start(0)
24141 start 0
24142 Mar 02 01:10:27.751 INFO request completed, latency_us: 260, response_code: 204, uri: /start/cid/0, method: POST, req_id: ae3c5d62-22cf-4370-b396-315e4ffe3691, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
24143 Replay: started 0, returned:Ok(())
24144 ----------------------------------------------------------------
24145 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24146 States: Offline Active Active
24147 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24148 8140 Acked 9139 Read 2 New Done Done true
24149 8141 Acked 9140 Write 3 New Done Done true
24150 8142 Acked 9141 Flush 0 New Done Done true
24151 8143 Acked 9142 Write 10 New Done Done false
24152 8144 Acked 9143 Flush 0 New Done Done false
24153 8145 Acked 9144 Write 4 New Done Done false
24154 8146 Acked 9145 Read 10 New Done Done false
24155 8147 Acked 9146 Write 2 New Done Done false
24156 8148 Acked 9147 Read 1 New Done Done false
24157 8149 Acked 9148 Read 8 New Done Done false
24158 8150 Acked 9149 Read 2 New Done Done false
24159 8151 Acked 9150 Write 2 New Done Done false
24160 8152 Acked 9151 Write 2 New Done Done false
24161 8153 Acked 9152 Write 9 New Done Done false
24162 8154 Acked 9153 Read 7 New Done Done false
24163 8155 Acked 9154 Read 10 New Done Done false
24164 8156 Acked 9155 Read 7 New Done Done false
24165 8157 Acked 9156 Read 4 New Done Done false
24166 8158 Acked 9157 Flush 0 New Done Done false
24167 8159 Acked 9158 Write 8 New Done Done false
24168 8160 Acked 9159 Read 7 New Done Done false
24169 8161 Acked 9160 Flush 0 New Done Done false
24170 8162 Acked 9161 Write 8 New Done Done false
24171 8163 Acked 9162 Read 6 New Done Done false
24172 8164 Acked 9163 Read 8 New Done Done false
24173 8165 Acked 9164 Read 1 New Done Done false
24174 8166 Acked 9165 Read 4 New Done Done false
24175 8167 Acked 9166 Read 4 New Done Done false
24176 8168 Acked 9167 Flush 0 New Done Done false
24177 8169 Acked 9168 Read 10 New Done Done false
24178 8170 Acked 9169 Write 3 New Done Done false
24179 8171 Acked 9170 Read 7 New Done Done false
24180 8172 Acked 9171 Write 7 New Done Done false
24181 8173 Acked 9172 Write 9 New Done Done false
24182 8174 Acked 9173 Read 7 New Done Done false
24183 8175 Acked 9174 Write 5 New Done Done false
24184 8176 Acked 9175 Flush 0 New Done Done false
24185 8177 Acked 9176 Read 5 New Done Done false
24186 8178 Acked 9177 Read 2 New Done Done false
24187 8179 Acked 9178 Read 6 New Done Done false
24188 8180 Acked 9179 Read 4 New Done Done false
24189 8181 Acked 9180 Read 6 New Done Done false
24190 8182 Acked 9181 Write 6 New Done Done false
24191 8183 Acked 9182 Read 5 New Done Done false
24192 8184 Acked 9183 Read 10 New Done Done false
24193 8185 Acked 9184 Write 8 New Done Done false
24194 8186 Acked 9185 Read 3 New Done Done false
24195 8187 Acked 9186 Write 10 New Done Done false
24196 8188 Acked 9187 Read 5 New Done Done false
24197 8189 Acked 9188 Read 1 New Done Done false
24198 8190 Acked 9189 Read 4 New Done Done false
24199 8191 Acked 9190 Write 8 New Done Done false
24200 8192 Acked 9191 Read 4 New Done Done false
24201 8193 Acked 9192 Write 8 New Done Done false
24202 8194 Acked 9193 Read 3 New Done Done false
24203 8195 Acked 9194 Read 3 New Done Done false
24204 8196 Acked 9195 Read 4 New Done Done false
24205 8197 Acked 9196 Flush 0 New Done Done false
24206 8198 Acked 9197 Write 5 New Done Done false
24207 8199 Acked 9198 Flush 0 New Done Done false
24208 8200 Acked 9199 Read 5 New Done Done false
24209 8201 Acked 9200 Write 10 New Done Done false
24210 8202 Acked 9201 Read 7 New Done Done false
24211 8203 Acked 9202 Read 9 New Done Done false
24212 8204 Acked 9203 Flush 0 New Done Done false
24213 8205 Acked 9204 Flush 0 New Done Done false
24214 8206 Acked 9205 Write 4 New Done Done false
24215 8207 Acked 9206 Read 5 New Done Done false
24216 8208 Acked 9207 Write 4 New Done Done false
24217 8209 Acked 9208 Read 1 New Done Done false
24218 8210 Acked 9209 Read 10 New Done Done false
24219 8211 Acked 9210 Read 3 New Done Done false
24220 8212 Acked 9211 Read 3 New Done Done false
24221 8213 Acked 9212 Flush 0 New Done Done false
24222 8214 Acked 9213 Read 4 New Done Done false
24223 8215 Acked 9214 Write 9 New Done Done false
24224 8216 Acked 9215 Write 8 New Done Done false
24225 8217 Acked 9216 Flush 0 New Done Done false
24226 8218 Acked 9217 Read 7 New Done Done false
24227 8219 Acked 9218 Write 3 New Done Done false
24228 8220 Acked 9219 Read 2 New Done Done false
24229 8221 Acked 9220 Read 9 New Done Done false
24230 8222 Acked 9221 Read 8 New Done Done false
24231 8223 Acked 9222 Write 3 New Done Done false
24232 8224 Acked 9223 Read 2 New Done Done false
24233 8225 Acked 9224 Write 2 New Done Done false
24234 8226 Acked 9225 Read 2 New Done Done false
24235 8227 Acked 9226 Read 8 New Done Done false
24236 8228 Acked 9227 Write 10 New Done Done false
24237 8229 Acked 9228 Write 5 New Done Done false
24238 8230 Acked 9229 Write 4 New Done Done false
24239 8231 Acked 9230 Read 6 New Done Done false
24240 8232 Acked 9231 Write 9 New Done Done false
24241 8233 Acked 9232 Read 1 New Done Done false
24242 8234 Acked 9233 Read 4 New Done Done false
24243 8235 Acked 9234 Read 7 New Done Done false
24244 8236 Acked 9235 Read 3 New Done Done false
24245 8237 Acked 9236 Read 7 New Done Done false
24246 8238 Acked 9237 Read 4 New Done Done false
24247 8239 Acked 9238 Read 10 New Done Done false
24248 8240 Acked 9239 Write 5 New Done Done false
24249 8241 Acked 9240 Flush 0 New Done Done false
24250 8242 Acked 9241 Flush 0 New Done Done false
24251 8243 Acked 9242 Write 6 New Done Done false
24252 8244 Acked 9243 Read 10 New Done Done false
24253 8245 Acked 9244 Read 10 New Done Done false
24254 8246 Acked 9245 Flush 0 New Done Done false
24255 8247 Acked 9246 Write 5 New Done Done false
24256 8248 Acked 9247 Read 5 New Done Done false
24257 8249 Acked 9248 Read 4 New Done Done false
24258 8250 Acked 9249 Read 3 New Done Done false
24259 8251 Acked 9250 Read 7 New Done Done false
24260 8252 Acked 9251 Read 9 New Done Done false
24261 8253 Acked 9252 Write 8 New Done Done false
24262 8254 Acked 9253 Write 4 New Done Done false
24263 8255 Acked 9254 Write 9 New Done Done false
24264 8256 Acked 9255 Read 7 New Done Done false
24265 8257 Acked 9256 Read 7 New Done Done false
24266 8258 Acked 9257 Read 10 New Done Done false
24267 8259 Acked 9258 Read 8 New Done Done false
24268 8260 Acked 9259 Read 1 New Done Done false
24269 8261 Acked 9260 Flush 0 New Done Done false
24270 8262 Acked 9261 Read 7 New Done Done false
24271 8263 Acked 9262 Read 9 New Done Done false
24272 8264 Acked 9263 Write 4 New Done Done false
24273 8265 Acked 9264 Read 7 New Done Done false
24274 8266 Acked 9265 Read 3 New Done Done false
24275 8267 Acked 9266 Write 4 New Done Done false
24276 8268 Acked 9267 Read 6 New Done Done false
24277 8269 Acked 9268 Read 7 New Done Done false
24278 8270 Acked 9269 Read 5 New Done Done false
24279 8271 Acked 9270 Write 1 New Done Done false
24280 8272 Acked 9271 Write 5 New Done Done false
24281 8273 Acked 9272 Read 7 New Done Done false
24282 8274 Acked 9273 Write 1 New Done Done false
24283 8275 Acked 9274 Read 5 New Done Done false
24284 8276 Acked 9275 Read 4 New Done Done false
24285 8277 Acked 9276 Read 4 New Done Done false
24286 8278 Acked 9277 Write 6 New Done Done false
24287 8279 Acked 9278 Flush 0 New Done Done false
24288 8280 Acked 9279 Read 2 New Done Done false
24289 8281 Acked 9280 Write 3 New Done Done false
24290 8282 Acked 9281 Flush 0 New Done Done false
24291 8283 Acked 9282 Write 9 New Done Done false
24292 8284 Acked 9283 Read 2 New Done Done false
24293 8285 Acked 9284 Write 8 New Done Done false
24294 8286 Acked 9285 Read 2 New Done Done false
24295 8287 Acked 9286 Write 3 New Done Done false
24296 8288 Acked 9287 Read 8 New Done Done false
24297 8289 Acked 9288 Read 9 New Done Done false
24298 8290 Acked 9289 Read 5 New Done Done false
24299 8291 Acked 9290 Read 5 New Done Done false
24300 8292 Acked 9291 Write 8 New Done Done false
24301 8293 Acked 9292 Read 5 New Done Done false
24302 8294 Acked 9293 Write 2 New Done Done false
24303 8295 Acked 9294 Flush 0 New Done Done false
24304 8296 Acked 9295 Write 8 New Done Done false
24305 8297 Acked 9296 Read 5 New Done Done false
24306 8298 Acked 9297 Write 7 New Done Done false
24307 8299 Acked 9298 Write 8 New Done Done false
24308 8300 Acked 9299 Read 9 New Done Done false
24309 8301 Acked 9300 Read 1 New Done Done false
24310 8302 Acked 9301 Write 2 New Done Done false
24311 8303 Acked 9302 Write 3 New Done Done false
24312 8304 Acked 9303 Write 1 New Done Done false
24313 8305 Acked 9304 Write 9 New Done Done false
24314 8306 Acked 9305 Read 7 New Done Done false
24315 8307 Acked 9306 Read 8 New Done Done false
24316 8308 Acked 9307 Write 6 New Done Done false
24317 8309 Acked 9308 Write 2 New Done Done false
24318 8310 Acked 9309 Read 6 New Done Done false
24319 8311 Acked 9310 Write 2 New Done Done false
24320 8312 Acked 9311 Read 7 New Done Done false
24321 8313 Acked 9312 Read 6 New Done Done false
24322 8314 Acked 9313 Write 3 New Done Done false
24323 8315 Acked 9314 Write 10 New Done Done false
24324 8316 Acked 9315 Write 2 New Done Done false
24325 8317 Acked 9316 Flush 0 New Done Done false
24326 8318 Acked 9317 Read 5 New Done Done false
24327 8319 Acked 9318 Read 3 New Done Done false
24328 8320 Acked 9319 Write 6 New Done Done false
24329 8321 Acked 9320 Write 8 New Done Done false
24330 8322 Acked 9321 Read 6 New Done Done false
24331 8323 Acked 9322 Write 1 New Done Done false
24332 8324 Acked 9323 Read 7 New Done Done false
24333 8325 Acked 9324 Flush 0 New Done Done false
24334 8326 Acked 9325 Read 4 New Done Done false
24335 8327 Acked 9326 Write 8 New Done Done false
24336 8328 Acked 9327 Read 6 New Done Done false
24337 8329 Acked 9328 Write 7 New Done Done false
24338 8330 Acked 9329 Read 8 New Done Done false
24339 8331 Acked 9330 Read 8 New Done Done false
24340 8332 Acked 9331 Write 2 New Done Done false
24341 8333 Acked 9332 Flush 0 New Done Done false
24342 8334 Acked 9333 Read 7 New Done Done false
24343 8335 Acked 9334 Write 2 New Done Done false
24344 8336 Acked 9335 Write 3 New Done Done false
24345 8337 Acked 9336 Write 8 New Done Done false
24346 8338 Acked 9337 Write 1 New Done Done false
24347 8339 Acked 9338 Flush 0 New Done Done false
24348 8340 Acked 9339 Write 6 New Done Done false
24349 8341 Acked 9340 Write 8 New Done Done false
24350 8342 Acked 9341 Write 2 New Done Done false
24351 8343 Acked 9342 Write 9 New Done Done false
24352 8344 Acked 9343 Flush 0 New Done Done false
24353 8345 Acked 9344 Write 8 New Done Done false
24354 8346 Acked 9345 Write 9 New Done Done false
24355 8347 Acked 9346 Write 1 New Done Done false
24356 8348 Acked 9347 Read 9 New Done Done false
24357 8349 Acked 9348 Write 10 New Done Done false
24358 8350 Acked 9349 Read 4 New Done Done false
24359 8351 Acked 9350 Flush 0 New Done Done false
24360 8352 Acked 9351 Write 6 New Done Done false
24361 8353 Acked 9352 Read 7 New Done Done false
24362 8354 Acked 9353 Read 9 New Done Done false
24363 8355 Acked 9354 Read 9 New Done Done false
24364 8356 Acked 9355 Read 10 New Done Done false
24365 8357 Acked 9356 Read 4 New Done Done false
24366 8358 Acked 9357 Read 6 New Done Done false
24367 8359 Acked 9358 Flush 0 New Done Done false
24368 8360 Acked 9359 Read 1 New Done Done false
24369 8361 Acked 9360 Write 2 New Done Done false
24370 8362 Acked 9361 Write 8 New Done Done false
24371 8363 Acked 9362 Write 3 New Done Done false
24372 8364 Acked 9363 Read 6 New Done Done false
24373 8365 Acked 9364 Write 5 New Done Done false
24374 8366 Acked 9365 Read 1 New Done Done false
24375 8367 Acked 9366 Read 6 New Done Done false
24376 8368 Acked 9367 Write 2 New Done Done false
24377 8369 Acked 9368 Write 8 New Done Done false
24378 8370 Acked 9369 Read 5 New Done Done false
24379 8371 Acked 9370 Write 9 New Done Done false
24380 8372 Acked 9371 Write 7 New Done Done false
24381 8373 Acked 9372 Write 7 New Done Done false
24382 8374 Acked 9373 Read 8 New Done Done false
24383 8375 Acked 9374 Read 9 New Done Done false
24384 8376 Acked 9375 Flush 0 New Done Done false
24385 8377 Acked 9376 Read 8 New Done Done false
24386 8378 Acked 9377 Write 10 New Done Done false
24387 8379 Acked 9378 Read 5 New Done Done false
24388 8380 Acked 9379 Write 6 New Done Done false
24389 8381 Acked 9380 Write 6 New Done Done false
24390 8382 Acked 9381 Read 9 New Done Done false
24391 8383 Acked 9382 Write 3 New Done Done false
24392 8384 Acked 9383 Write 6 New Done Done false
24393 8385 Acked 9384 Flush 0 New Done Done false
24394 8386 Acked 9385 Read 4 New Done Done false
24395 8387 Acked 9386 Flush 0 New Done Done false
24396 8388 Acked 9387 Write 6 New Done Done false
24397 8389 Acked 9388 Flush 0 New Done Done false
24398 8390 Acked 9389 Write 10 New Done Done false
24399 8391 Acked 9390 Read 5 New Done Done false
24400 8392 Acked 9391 Read 9 New Done Done false
24401 8393 Acked 9392 Read 2 New Done Done false
24402 8394 Acked 9393 Read 3 New Done Done false
24403 8395 Acked 9394 Read 2 New Done Done false
24404 8396 Acked 9395 Read 6 New Done Done false
24405 8397 Acked 9396 Write 2 New Done Done false
24406 8398 Acked 9397 Write 4 New Done Done false
24407 8399 Acked 9398 Flush 0 New Done Done false
24408 8400 Acked 9399 Flush 0 New Done Done false
24409 8401 Acked 9400 Read 1 New Done Done false
24410 8402 Acked 9401 Read 3 New Done Done false
24411 8403 Acked 9402 Flush 0 New Done Done false
24412 8404 Acked 9403 Write 5 New Done Done false
24413 8405 Acked 9404 Write 8 New Done Done false
24414 8406 Acked 9405 Read 5 New Done Done false
24415 8407 Acked 9406 Read 6 New Done Done false
24416 8408 Acked 9407 Read 2 New Done Done false
24417 8409 Acked 9408 Write 8 New Done Done false
24418 8410 Acked 9409 Write 5 New Done Done false
24419 8411 Acked 9410 Write 2 New Done Done false
24420 8412 Acked 9411 Write 1 New Done Done false
24421 8413 Acked 9412 Read 7 New Done Done false
24422 8414 Acked 9413 Write 9 New Done Done false
24423 8415 Acked 9414 Write 4 New Done Done false
24424 8416 Acked 9415 Read 3 New Done Done false
24425 8417 Acked 9416 Read 3 New Done Done false
24426 8418 Acked 9417 Read 7 New Done Done false
24427 8419 Acked 9418 Read 3 New Done Done false
24428 8420 Acked 9419 Write 6 New Done Done false
24429 8421 Acked 9420 Read 1 New Done Done false
24430 8422 Acked 9421 Read 9 New Done Done false
24431 8423 Acked 9422 Read 3 New Done Done false
24432 8424 Acked 9423 Read 2 New Done Done false
24433 8425 Acked 9424 Write 1 New Done Done false
24434 8426 Acked 9425 Write 5 New Done Done false
24435 8427 Acked 9426 Write 7 New Done Done false
24436 8428 Acked 9427 Read 6 New Done Done false
24437 8429 Acked 9428 Read 8 New Done Done false
24438 8430 Acked 9429 Write 4 New Done Done false
24439 8431 Acked 9430 Flush 0 New Done Done false
24440 8432 Acked 9431 Read 10 New Done Done false
24441 8433 Acked 9432 Write 7 New Done Done false
24442 8434 Acked 9433 Read 9 New Done Done false
24443 8435 Acked 9434 Read 1 New Done Done false
24444 8436 Acked 9435 Flush 0 New Done Done false
24445 8437 Acked 9436 Write 2 New Done Done false
24446 8438 Acked 9437 Read 1 New Done Done false
24447 8439 Acked 9438 Write 5 New Done Done false
24448 8440 Acked 9439 Read 6 New Done Done false
24449 STATES DS:0 DS:1 DS:2 TOTAL
24450 New 301 0 0 301
24451 Sent 0 0 0 0
24452 Done 0 301 301 602
24453 Skipped 0 0 0 0
24454 Error 0 0 0 0
24455 Last Flush: 9138 9435 9435
24456 Downstairs last five completed: 9138 9137 9136 9135 9134
24457 Upstairs last five completed: 8440 8439 8438 8437 8436
24458 CLIENT: Up:0 ds:301 act:2
244592024-03-02T01:10:28.401ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244602024-03-02T01:10:28.402ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244612024-03-02T01:10:28.402ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244622024-03-02T01:10:28.402ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244632024-03-02T01:10:28.402ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244642024-03-02T01:10:28.402ZINFOcrucible: [0] client re-new 302 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244652024-03-02T01:10:28.402ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244662024-03-02T01:10:29.403ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244672024-03-02T01:10:29.403ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244682024-03-02T01:10:29.403ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244692024-03-02T01:10:29.403ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244702024-03-02T01:10:29.403ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244712024-03-02T01:10:29.403ZINFOcrucible: [0] client re-new 302 jobs since flush 9138 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244722024-03-02T01:10:29.403ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
24473 [8810] got start action
24474 [8810] I am going to restart
24475 Starting downstairs at port 8810
24476 Make output file at "/tmp/dsc/downstairs-8810.txt"
24477 [8810][0] reports Starting
24478 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1190)
24479 [8810][0] reports Running
244802024-03-02T01:10:30.405ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244812024-03-02T01:10:30.405ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244822024-03-02T01:10:30.406ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244832024-03-02T01:10:30.406ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244842024-03-02T01:10:30.407ZINFOcrucible: send last flush ID to this DS: 9138 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244852024-03-02T01:10:30.407ZINFOcrucible: Replied this last flush ID: 9138 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244862024-03-02T01:10:30.407ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244872024-03-02T01:10:30.407ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244882024-03-02T01:10:30.407ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
244892024-03-02T01:10:30.407ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
24490 ----------------------------------------------------------------
24491 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24492 Downstairs last five completed: 9440 9439 9438 9437 9436
24493 Upstairs last five completed: 8441 8440 8439 8438 8437
24494 CLIENT: Up:0 ds:0 act:3
24495 Replay: All jobs finished, all DS active.
24496 Main task has work to do, go find it
24497 got dsc Stop(0)
24498 stop 0
24499 Mar 02 01:10:31.755 INFO request completed, [8810] Got stop action so:false kr:false
24500 latency_us: 369, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 97189616-9c7b-45a8-ba25-fbd3988722a6, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
24501 Mar 02 01:10:31.756 INFO accepted connection, remote_addr: 127.0.0.1:58992, local_addr: 127.0.0.1:9998
245022024-03-02T01:10:31.756ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245032024-03-02T01:10:31.756ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245042024-03-02T01:10:31.756ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245052024-03-02T01:10:31.756ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245062024-03-02T01:10:31.756ZINFOcrucible: [0] client re-new 0 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245072024-03-02T01:10:31.756ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
24508 Mar 02 01:10:31.756 INFO request completed, latency_us: 111, response_code: 200, uri: /state/cid/0, method: GET, req_id: a1643fc3-8d6d-4269-98fa-9717c776f4e4, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
24509 [8810] Exited with: ExitStatus(unix_wait_status(9))
24510 [8810][0] reports Exit
245112024-03-02T01:10:32.758ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245122024-03-02T01:10:32.758ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245132024-03-02T01:10:32.758ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245142024-03-02T01:10:32.758ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245152024-03-02T01:10:32.758ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245162024-03-02T01:10:32.758ZINFOcrucible: [0] client re-new 0 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245172024-03-02T01:10:32.758ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245182024-03-02T01:10:33.759ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245192024-03-02T01:10:33.759ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245202024-03-02T01:10:33.759ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245212024-03-02T01:10:33.759ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245222024-03-02T01:10:33.759ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245232024-03-02T01:10:33.759ZINFOcrucible: [0] client re-new 0 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245242024-03-02T01:10:33.759ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245252024-03-02T01:10:34.760ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245262024-03-02T01:10:34.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245272024-03-02T01:10:34.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245282024-03-02T01:10:34.760ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245292024-03-02T01:10:34.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245302024-03-02T01:10:34.761ZINFOcrucible: [0] client re-new 0 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245312024-03-02T01:10:34.761ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
24532 Mar 02 01:10:35.758 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6e6e64ac-02a2-43af-bb8d-30c49531c775, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
24533 001/300 Read block 4330 len 20480
24534 002/300 Flush
24535 003/300 Write block 3394 len 32768 data: 6 7 7 7 8 5 4 4
245362024-03-02T01:10:35.762ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245372024-03-02T01:10:35.762ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245382024-03-02T01:10:35.762ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245392024-03-02T01:10:35.762ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245402024-03-02T01:10:35.762ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245412024-03-02T01:10:35.762ZINFOcrucible: [0] client re-new 3 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
245422024-03-02T01:10:35.762ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
24543 004/300 Write block 731 len 36864 data: 6 7 9 8 5 3 3 3 4
24544 005/300 Flush
24545 006/300 Read block 3967 len 8192
24546 007/300 Read block 4724 len 24576
24547 008/300 Write block 1242 len 16384 data: 6 6 7 7
24548 009/300 Write block 3337 len 28672 data: 8 8 6 5 5 8 8
24549 010/300 Write block 1638 len 36864 data: 7 7 6 6 6 6 4 6 6
24550 011/300 Write block 3849 len 40960 data: 7 6 7 8 5 4 3 5 6 6
24551 012/300 Read block 1066 len 28672
24552 013/300 Read block 3623 len 20480
24553 014/300 Write block 2397 len 4096 data: 4
24554 015/300 Read block 1787 len 4096
24555 016/300 Read block 4963 len 24576
24556 017/300 Write block 217 len 32768 data: 5 6 6 6 6 8 6 5
24557 018/300 Read block 2582 len 24576
24558 019/300 Write block 3779 len 28672 data: 11 13 12 11 11 11 9
24559 020/300 Write block 129 len 28672 data: 5 4 4 5 4 4 4
24560 021/300 Read block 1887 len 12288
24561 022/300 Write block 1743 len 40960 data: 3 4 5 4 5 5 4 4 4 6
24562 023/300 Read block 1054 len 40960
24563 024/300 Write block 1208 len 28672 data: 6 8 8 9 10 11 9
24564 025/300 Read block 4039 len 24576
24565 026/300 Write block 4954 len 16384 data: 12 10 8 8
24566 027/300 Write block 2378 len 4096 data: 7
24567 028/300 Read block 2929 len 20480
24568 029/300 Read block 2742 len 36864
24569 030/300 Read block 1400 len 40960
24570 031/300 Write block 4726 len 12288 data: 8 8 8
24571 032/300 Read block 3314 len 28672
24572 033/300 Read block 4786 len 20480
24573 034/300 Read block 4037 len 24576
24574 035/300 Read block 2965 len 40960
24575 036/300 Read block 3280 len 4096
24576 037/300 Read block 3741 len 28672
24577 038/300 Read block 249 len 16384
24578 039/300 Write block 4854 len 16384 data: 5 4 4 2
24579 040/300 Write block 4704 len 12288 data: 4 4 4
24580 041/300 Read block 1796 len 36864
24581 042/300 Read block 4609 len 40960
24582 043/300 Write block 2040 len 12288 data: 7 7 7
24583 044/300 Write block 2226 len 20480 data: 5 4 5 4 3
24584 045/300 Read block 1172 len 28672
24585 046/300 Read block 304 len 4096
24586 047/300 Write block 2117 len 8192 data: 8 6
24587 048/300 Read block 1596 len 4096
24588 049/300 Write block 502 len 12288 data: 3 3 3
24589 050/300 Write block 197 len 28672 data: 7 6 7 7 7 8 6
24590 051/300 Write block 170 len 4096 data: 5
24591 052/300 Read block 434 len 8192
24592 053/300 Read block 4836 len 8192
24593 054/300 Read block 1111 len 32768
24594 055/300 Read block 2875 len 20480
24595 056/300 Read block 3696 len 32768
24596 057/300 Write block 3791 len 12288 data: 4 5 3
24597 058/300 Read block 4132 len 40960
24598 059/300 Write block 2790 len 8192 data: 7 8
24599 060/300 Read block 2670 len 32768
24600 061/300 Read block 1587 len 20480
24601 062/300 Read block 1835 len 32768
24602 063/300 Read block 3876 len 8192
24603 064/300 Read block 3874 len 40960
24604 065/300 Write block 3291 len 32768 data: 5 5 6 6 6 8 5 6
24605 066/300 Read block 3449 len 8192
24606 067/300 Read block 577 len 4096
24607 068/300 Flush
24608 069/300 Read block 1531 len 8192
24609 070/300 Read block 1097 len 12288
24610 071/300 Flush
24611 072/300 Write block 4011 len 36864 data: 4 2 3 3 3 5 5 6 6
24612 073/300 Write block 2394 len 36864 data: 6 6 5 5 6 5 6 6 7
24613 074/300 Read block 4794 len 28672
24614 075/300 Flush
24615 076/300 Read block 258 len 32768
24616 077/300 Read block 303 len 28672
24617 078/300 Read block 3554 len 32768
24618 079/300 Write block 1283 len 36864 data: 4 4 3 3 4 4 5 5 5
24619 080/300 Write block 2783 len 8192 data: 6 5
24620 081/300 Write block 4373 len 24576 data: 6 6 6 5 5 5
24621 082/300 Write block 96 len 12288 data: 4 5 4
24622 083/300 Read block 1870 len 28672
24623 084/300 Read block 1412 len 40960
24624 085/300 Write block 2314 len 16384 data: 10 8 7 6
24625 086/300 Write block 221 len 40960 data: 7 9 7 6 5 5 6 7 7 8
24626 087/300 Write block 4516 len 28672 data: 7 6 7 6 6 6 6
24627 088/300 Read block 4247 len 12288
24628 089/300 Write block 1462 len 40960 data: 5 6 6 6 6 5 5 6 5 5
24629 090/300 Write block 1246 len 24576 data: 6 6 4 4 5 6
24630 091/300 Write block 964 len 12288 data: 7 8 7
24631 092/300 Read block 27 len 12288
24632 093/300 Write block 4263 len 28672 data: 5 5 7 5 6 7 5
24633 094/300 Read block 3131 len 8192
24634 095/300 Read block 1132 len 8192
24635 096/300 Read block 1213 len 36864
24636 097/300 Write block 1573 len 8192 data: 8 8
24637 098/300 Read block 78 len 24576
24638 099/300 Read block 1583 len 4096
24639 100/300 Read block 2570 len 4096
24640 101/300 Read block 596 len 28672
24641 102/300 Write block 3405 len 32768 data: 6 6 7 7 5 5 5 7
24642 103/300 Read block 251 len 40960
24643 104/300 Flush
24644 105/300 Read block 953 len 32768
24645 106/300 Read block 4130 len 16384
24646 107/300 Read block 2232 len 8192
24647 108/300 Write block 4857 len 16384 data: 3 2 3 2
24648 109/300 Flush
24649 110/300 Write block 2752 len 24576 data: 8 8 9 8 4 3
24650 111/300 Read block 2456 len 36864
24651 112/300 Flush
24652 113/300 Read block 4763 len 24576
24653 114/300 Read block 2326 len 20480
24654 115/300 Flush
24655 116/300 Read block 2647 len 20480
24656 117/300 Read block 4358 len 4096
24657 118/300 Write block 1659 len 40960 data: 4 3 4 4 4 5 6 7 7 6
24658 119/300 Read block 4327 len 36864
24659 120/300 Write block 3750 len 16384 data: 6 5 6 5
24660 121/300 Read block 213 len 12288
24661 122/300 Write block 4246 len 4096 data: 3
24662 123/300 Write block 3136 len 36864 data: 6 6 8 6 5 5 5 4 4
24663 124/300 Flush
24664 125/300 Write block 584 len 16384 data: 6 7 6 6
24665 126/300 Write block 4428 len 8192 data: 4 4
24666 127/300 Write block 38 len 24576 data: 4 3 3 5 5 8
24667 128/300 Write block 4452 len 40960 data: 6 4 4 6 6 6 5 5 5 3
24668 129/300 Write block 2906 len 40960 data: 4 4 5 6 6 8 7 6 7 7
24669 130/300 Write block 396 len 20480 data: 3 4 3 4 4
24670 131/300 Write block 4629 len 4096 data: 6
24671 132/300 Flush
24672 133/300 Read block 2363 len 4096
24673 134/300 Read block 1391 len 32768
24674 135/300 Write block 3986 len 40960 data: 8 7 7 6 8 8 7 5 6 7
24675 136/300 Write block 2805 len 24576 data: 4 5 5 6 8 9
24676 137/300 Read block 312 len 28672
24677 138/300 Read block 4219 len 36864
24678 139/300 Flush
24679 140/300 Write block 1275 len 28672 data: 5 7 6 6 7 7 6
24680 141/300 Read block 823 len 40960
24681 142/300 Read block 1207 len 8192
24682 143/300 Write block 3098 len 4096 data: 7
24683 144/300 Read block 2226 len 8192
24684 145/300 Write block 1986 len 4096 data: 9
24685 146/300 Read block 161 len 8192
24686 147/300 Write block 3725 len 12288 data: 6 3 3
24687 148/300 Read block 4140 len 16384
24688 149/300 Read block 2370 len 36864
24689 150/300 Read block 3040 len 28672
24690 151/300 Flush
24691 152/300 Write block 3640 len 20480 data: 4 4 6 7 6
24692 153/300 Write block 1359 len 8192 data: 8 7
24693 154/300 Read block 4465 len 36864
24694 155/300 Read block 3168 len 40960
24695 156/300 Flush
24696 157/300 Read block 3130 len 40960
24697 158/300 Read block 2838 len 12288
24698 159/300 Read block 1369 len 32768
24699 160/300 Read block 4929 len 28672
24700 161/300 Read block 449 len 20480
24701 162/300 Read block 4961 len 12288
24702 163/300 Write block 928 len 32768 data: 7 4 3 3 4 5 4 6
24703 164/300 Read block 4592 len 36864
24704 165/300 Read block 163 len 8192
24705 166/300 Read block 718 len 8192
24706 167/300 Write block 3736 len 20480 data: 5 7 4 3 3
24707 168/300 Write block 4505 len 12288 data: 6 6 6
24708 169/300 Write block 4770 len 36864 data: 7 6 6 7 6 6 4 4 3
24709 170/300 Write block 1161 len 32768 data: 8 5 4 3 2 4 4 4
24710 171/300 Flush
24711 172/300 Write block 2780 len 32768 data: 8 7 6 7 6 4 5 5
24712 173/300 Write block 356 len 8192 data: 3 4
24713 174/300 Write block 609 len 36864 data: 8 9 8 8 9 9 9 9 8
24714 175/300 Read block 4248 len 4096
24715 176/300 Read block 1889 len 8192
24716 177/300 Read block 4565 len 16384
24717 178/300 Read block 4726 len 28672
24718 179/300 Read block 2295 len 8192
24719 180/300 Read block 543 len 28672
24720 181/300 Write block 2034 len 28672 data: 3 5 5 4 5 5 8
24721 182/300 Write block 4655 len 40960 data: 5 5 6 7 9 10 10 8 5 5
24722 183/300 Read block 3552 len 24576
24723 184/300 Read block 4156 len 28672
24724 185/300 Read block 3180 len 16384
24725 186/300 Read block 2749 len 36864
24726 187/300 Write block 2888 len 36864 data: 8 7 6 5 5 6 5 5 4
24727 188/300 Read block 275 len 24576
24728 189/300 Read block 3133 len 40960
24729 190/300 Write block 502 len 40960 data: 4 4 4 3 4 5 5 4 3 3
24730 191/300 Write block 2795 len 28672 data: 6 5 5 3 3 4 4
24731 192/300 Flush
24732 193/300 Flush
24733 194/300 Read block 4590 len 4096
24734 195/300 Read block 4511 len 24576
24735 196/300 Read block 3526 len 28672
24736 197/300 Write block 101 len 40960 data: 5 5 6 7 5 7 7 4 4 5
24737 198/300 Write block 3550 len 40960 data: 7 9 11 9 7 8 8 8 7 5
24738 199/300 Read block 3648 len 36864
24739 200/300 Read block 3668 len 16384
24740 201/300 Write block 1233 len 40960 data: 6 7 7 7 7 7 5 4 3 7
24741 202/300 Write block 1110 len 4096 data: 3
24742 203/300 Read block 3501 len 40960
24743 204/300 Write block 3137 len 32768 data: 7 9 7 6 6 6 5 5
24744 205/300 Write block 1873 len 28672 data: 4 5 5 5 5 4 3
24745 206/300 Write block 1816 len 40960 data: 6 4 4 5 5 5 4 4 5 5
24746 207/300 Read block 4679 len 20480
24747 208/300 Write block 2924 len 40960 data: 6 5 5 7 7 6 6 6 7 7
24748 209/300 Read block 4769 len 28672
24749 210/300 Flush
24750 211/300 Flush
24751 212/300 Read block 1879 len 40960
24752 213/300 Flush
24753 214/300 Read block 3689 len 20480
24754 215/300 Write block 3789 len 16384 data: 8 7 5 6
24755 216/300 Read block 4613 len 20480
24756 217/300 Read block 4215 len 12288
24757 218/300 Write block 4420 len 28672 data: 7 5 5 5 5 7 5
24758 219/300 Read block 1236 len 28672
24759 220/300 Write block 2761 len 20480 data: 8 6 4 4 5
24760 221/300 Write block 4804 len 32768 data: 6 6 6 7 7 7 5 6
24761 222/300 Write block 1184 len 4096 data: 7
24762 223/300 Write block 1184 len 16384 data: 8 5 5 5
24763 224/300 Write block 2880 len 8192 data: 5 4
24764 225/300 Read block 2874 len 36864
24765 226/300 Write block 1925 len 24576 data: 7 7 7 7 6 6
24766 227/300 Read block 3162 len 16384
24767 228/300 Read block 2112 len 20480
24768 229/300 Read block 4812 len 16384
24769 230/300 Read block 4105 len 28672
24770 231/300 Write block 1569 len 40960 data: 8 8 8 7 9 9 7 8 7 8
24771 232/300 Read block 2856 len 12288
24772 233/300 Flush
24773 234/300 Read block 1808 len 4096
24774 235/300 Read block 3958 len 8192
24775 236/300 Read block 4600 len 4096
24776 237/300 Read block 3872 len 24576
24777 238/300 Read block 1493 len 20480
24778 239/300 Write block 883 len 4096 data: 4
24779 240/300 Read block 2505 len 16384
24780 241/300 Read block 168 len 32768
24781 242/300 Read block 3532 len 4096
24782 243/300 Write block 4392 len 16384 data: 6 6 7 6
24783 244/300 Write block 4300 len 12288 data: 5 5 5
24784 245/300 Read block 1154 len 4096
24785 246/300 Read block 405 len 36864
24786 247/300 Read block 3431 len 28672
24787 248/300 Read block 4217 len 32768
24788 249/300 Flush
24789 250/300 Read block 2472 len 16384
24790 251/300 Write block 3737 len 4096 data: 8
24791 252/300 Write block 2976 len 28672 data: 4 4 5 5 5 5 4
24792 253/300 Write block 2921 len 32768 data: 3 7 8 7 6 6 8 8
24793 254/300 Flush
24794 255/300 Write block 4267 len 20480 data: 7 8 6 3 3
24795 256/300 Write block 2561 len 36864 data: 10 9 9 9 6 7 6 8 7
24796 257/300 Read block 1060 len 16384
24797 258/300 Read block 924 len 4096
24798 259/300 Read block 3653 len 4096
24799 260/300 Write block 4299 len 32768 data: 7 6 6 6 4 5 4 4
24800 261/300 Read block 2315 len 28672
24801 262/300 Flush
24802 263/300 Read block 4307 len 20480
24803 264/300 Read block 1311 len 20480
24804 265/300 Read block 4137 len 4096
24805 266/300 Read block 1836 len 12288
24806 267/300 Flush
24807 268/300 Write block 2070 len 40960 data: 10 9 8 8 5 3 3 3 3 4
24808 269/300 Read block 4114 len 28672
24809 270/300 Write block 903 len 4096 data: 3
24810 271/300 Write block 3291 len 24576 data: 6 6 7 7 7 9
24811 272/300 Flush
24812 273/300 Flush
24813 274/300 Write block 4029 len 8192 data: 7 7
24814 275/300 Read block 25 len 24576
24815 276/300 Flush
24816 277/300 Write block 3687 len 32768 data: 4 4 3 3 4 5 5 4
24817 278/300 Write block 223 len 32768 data: 8 7 6 6 7 8 8 9
24818 279/300 Write block 432 len 8192 data: 4 3
24819 280/300 Read block 2587 len 16384
24820 281/300 Flush
24821 282/300 Write block 688 len 40960 data: 5 6 3 5 7 8 8 8 9 8
24822 283/300 Flush
24823 284/300 Write block 4692 len 28672 data: 6 5 6 6 6 5 4
24824 285/300 Read block 2714 len 28672
24825 286/300 Read block 3153 len 12288
24826 287/300 Read block 4423 len 16384
24827 288/300 Write block 4053 len 12288 data: 7 7 7
24828 289/300 Write block 2890 len 32768 data: 7 6 6 7 6 6 5 6
24829 290/300 Read block 3745 len 16384
24830 291/300 Read block 4690 len 40960
24831 292/300 Write block 753 len 8192 data: 6 6
24832 293/300 Read block 2026 len 4096
24833 294/300 Write block 3338 len 16384 data: 9 7 6 6
24834 295/300 Write block 1150 len 40960 data: 5 6 5 5 5 6 7 6 7 5
24835 296/300 Write block 4748 len 8192 data: 3 2
24836 297/300 Read block 3341 len 12288
24837 298/300 Read block 2402 len 28672
24838 299/300 Read block 3765 len 12288
24839 300/300 Write block 81 len 20480 data: 5 8 8 7 6
24840 Main task has work to do, go find it
24841 got dsc Start(0)
24842 start 0
24843 Mar 02 01:10:36.104 INFO request completed, latency_us: 258, response_code: 204, uri: /start/cid/0, method: POST, req_id: c6855fa6-9c48-4665-90cf-b666e00506d7, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
24844 Replay: started 0, returned:Ok(())
24845 ----------------------------------------------------------------
24846 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
24847 States: Offline Active Active
24848 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24849 8442 Acked 9441 Read 5 New Done Done true
24850 8443 Acked 9442 Flush 0 New Done Done true
24851 8444 Acked 9443 Write 8 New Done Done true
24852 8445 Acked 9444 Write 9 New Done Done false
24853 8446 Acked 9445 Flush 0 New Done Done false
24854 8447 Acked 9446 Read 2 New Done Done false
24855 8448 Acked 9447 Read 6 New Done Done false
24856 8449 Acked 9448 Write 4 New Done Done false
24857 8450 Acked 9449 Write 7 New Done Done false
24858 8451 Acked 9450 Write 9 New Done Done false
24859 8452 Acked 9451 Write 10 New Done Done false
24860 8453 Acked 9452 Read 7 New Done Done false
24861 8454 Acked 9453 Read 5 New Done Done false
24862 8455 Acked 9454 Write 1 New Done Done false
24863 8456 Acked 9455 Read 1 New Done Done false
24864 8457 Acked 9456 Read 6 New Done Done false
24865 8458 Acked 9457 Write 8 New Done Done false
24866 8459 Acked 9458 Read 6 New Done Done false
24867 8460 Acked 9459 Write 7 New Done Done false
24868 8461 Acked 9460 Write 7 New Done Done false
24869 8462 Acked 9461 Read 3 New Done Done false
24870 8463 Acked 9462 Write 10 New Done Done false
24871 8464 Acked 9463 Read 10 New Done Done false
24872 8465 Acked 9464 Write 7 New Done Done false
24873 8466 Acked 9465 Read 6 New Done Done false
24874 8467 Acked 9466 Write 4 New Done Done false
24875 8468 Acked 9467 Write 1 New Done Done false
24876 8469 Acked 9468 Read 5 New Done Done false
24877 8470 Acked 9469 Read 9 New Done Done false
24878 8471 Acked 9470 Read 10 New Done Done false
24879 8472 Acked 9471 Write 3 New Done Done false
24880 8473 Acked 9472 Read 7 New Done Done false
24881 8474 Acked 9473 Read 5 New Done Done false
24882 8475 Acked 9474 Read 6 New Done Done false
24883 8476 Acked 9475 Read 10 New Done Done false
24884 8477 Acked 9476 Read 1 New Done Done false
24885 8478 Acked 9477 Read 7 New Done Done false
24886 8479 Acked 9478 Read 4 New Done Done false
24887 8480 Acked 9479 Write 4 New Done Done false
24888 8481 Acked 9480 Write 3 New Done Done false
24889 8482 Acked 9481 Read 9 New Done Done false
24890 8483 Acked 9482 Read 10 New Done Done false
24891 8484 Acked 9483 Write 3 New Done Done false
24892 8485 Acked 9484 Write 5 New Done Done false
24893 8486 Acked 9485 Read 7 New Done Done false
24894 8487 Acked 9486 Read 1 New Done Done false
24895 8488 Acked 9487 Write 2 New Done Done false
24896 8489 Acked 9488 Read 1 New Done Done false
24897 8490 Acked 9489 Write 3 New Done Done false
24898 8491 Acked 9490 Write 7 New Done Done false
24899 8492 Acked 9491 Write 1 New Done Done false
24900 8493 Acked 9492 Read 2 New Done Done false
24901 8494 Acked 9493 Read 2 New Done Done false
24902 8495 Acked 9494 Read 8 New Done Done false
24903 8496 Acked 9495 Read 5 New Done Done false
24904 8497 Acked 9496 Read 8 New Done Done false
24905 8498 Acked 9497 Write 3 New Done Done false
24906 8499 Acked 9498 Read 10 New Done Done false
24907 8500 Acked 9499 Write 2 New Done Done false
24908 8501 Acked 9500 Read 8 New Done Done false
24909 8502 Acked 9501 Read 5 New Done Done false
24910 8503 Acked 9502 Read 8 New Done Done false
24911 8504 Acked 9503 Read 2 New Done Done false
24912 8505 Acked 9504 Read 10 New Done Done false
24913 8506 Acked 9505 Write 8 New Done Done false
24914 8507 Acked 9506 Read 2 New Done Done false
24915 8508 Acked 9507 Read 1 New Done Done false
24916 8509 Acked 9508 Flush 0 New Done Done false
24917 8510 Acked 9509 Read 2 New Done Done false
24918 8511 Acked 9510 Read 3 New Done Done false
24919 8512 Acked 9511 Flush 0 New Done Done false
24920 8513 Acked 9512 Write 9 New Done Done false
24921 8514 Acked 9513 Write 9 New Done Done false
24922 8515 Acked 9514 Read 7 New Done Done false
24923 8516 Acked 9515 Flush 0 New Done Done false
24924 8517 Acked 9516 Read 8 New Done Done false
24925 8518 Acked 9517 Read 7 New Done Done false
24926 8519 Acked 9518 Read 8 New Done Done false
24927 8520 Acked 9519 Write 9 New Done Done false
24928 8521 Acked 9520 Write 2 New Done Done false
24929 8522 Acked 9521 Write 6 New Done Done false
24930 8523 Acked 9522 Write 3 New Done Done false
24931 8524 Acked 9523 Read 7 New Done Done false
24932 8525 Acked 9524 Read 10 New Done Done false
24933 8526 Acked 9525 Write 4 New Done Done false
24934 8527 Acked 9526 Write 10 New Done Done false
24935 8528 Acked 9527 Write 7 New Done Done false
24936 8529 Acked 9528 Read 3 New Done Done false
24937 8530 Acked 9529 Write 10 New Done Done false
24938 8531 Acked 9530 Write 6 New Done Done false
24939 8532 Acked 9531 Write 3 New Done Done false
24940 8533 Acked 9532 Read 3 New Done Done false
24941 8534 Acked 9533 Write 7 New Done Done false
24942 8535 Acked 9534 Read 2 New Done Done false
24943 8536 Acked 9535 Read 2 New Done Done false
24944 8537 Acked 9536 Read 9 New Done Done false
24945 8538 Acked 9537 Write 2 New Done Done false
24946 8539 Acked 9538 Read 6 New Done Done false
24947 8540 Acked 9539 Read 1 New Done Done false
24948 8541 Acked 9540 Read 1 New Done Done false
24949 8542 Acked 9541 Read 7 New Done Done false
24950 8543 Acked 9542 Write 8 New Done Done false
24951 8544 Acked 9543 Read 10 New Done Done false
24952 8545 Acked 9544 Flush 0 New Done Done false
24953 8546 Acked 9545 Read 8 New Done Done false
24954 8547 Acked 9546 Read 4 New Done Done false
24955 8548 Acked 9547 Read 2 New Done Done false
24956 8549 Acked 9548 Write 4 New Done Done false
24957 8550 Acked 9549 Flush 0 New Done Done false
24958 8551 Acked 9550 Write 6 New Done Done false
24959 8552 Acked 9551 Read 9 New Done Done false
24960 8553 Acked 9552 Flush 0 New Done Done false
24961 8554 Acked 9553 Read 6 New Done Done false
24962 8555 Acked 9554 Read 5 New Done Done false
24963 8556 Acked 9555 Flush 0 New Done Done false
24964 8557 Acked 9556 Read 5 New Done Done false
24965 8558 Acked 9557 Read 1 New Done Done false
24966 8559 Acked 9558 Write 10 New Done Done false
24967 8560 Acked 9559 Read 9 New Done Done false
24968 8561 Acked 9560 Write 4 New Done Done false
24969 8562 Acked 9561 Read 3 New Done Done false
24970 8563 Acked 9562 Write 1 New Done Done false
24971 8564 Acked 9563 Write 9 New Done Done false
24972 8565 Acked 9564 Flush 0 New Done Done false
24973 8566 Acked 9565 Write 4 New Done Done false
24974 8567 Acked 9566 Write 2 New Done Done false
24975 8568 Acked 9567 Write 6 New Done Done false
24976 8569 Acked 9568 Write 10 New Done Done false
24977 8570 Acked 9569 Write 10 New Done Done false
24978 8571 Acked 9570 Write 5 New Done Done false
24979 8572 Acked 9571 Write 1 New Done Done false
24980 8573 Acked 9572 Flush 0 New Done Done false
24981 8574 Acked 9573 Read 1 New Done Done false
24982 8575 Acked 9574 Read 8 New Done Done false
24983 8576 Acked 9575 Write 10 New Done Done false
24984 8577 Acked 9576 Write 6 New Done Done false
24985 8578 Acked 9577 Read 7 New Done Done false
24986 8579 Acked 9578 Read 9 New Done Done false
24987 8580 Acked 9579 Flush 0 New Done Done false
24988 8581 Acked 9580 Write 7 New Done Done false
24989 8582 Acked 9581 Read 10 New Done Done false
24990 8583 Acked 9582 Read 2 New Done Done false
24991 8584 Acked 9583 Write 1 New Done Done false
24992 8585 Acked 9584 Read 2 New Done Done false
24993 8586 Acked 9585 Write 1 New Done Done false
24994 8587 Acked 9586 Read 2 New Done Done false
24995 8588 Acked 9587 Write 3 New Done Done false
24996 8589 Acked 9588 Read 4 New Done Done false
24997 8590 Acked 9589 Read 9 New Done Done false
24998 8591 Acked 9590 Read 7 New Done Done false
24999 8592 Acked 9591 Flush 0 New Done Done false
25000 8593 Acked 9592 Write 5 New Done Done false
25001 8594 Acked 9593 Write 2 New Done Done false
25002 8595 Acked 9594 Read 9 New Done Done false
25003 8596 Acked 9595 Read 10 New Done Done false
25004 8597 Acked 9596 Flush 0 New Done Done false
25005 8598 Acked 9597 Read 10 New Done Done false
25006 8599 Acked 9598 Read 3 New Done Done false
25007 8600 Acked 9599 Read 8 New Done Done false
25008 8601 Acked 9600 Read 7 New Done Done false
25009 8602 Acked 9601 Read 5 New Done Done false
25010 8603 Acked 9602 Read 3 New Done Done false
25011 8604 Acked 9603 Write 8 New Done Done false
25012 8605 Acked 9604 Read 9 New Done Done false
25013 8606 Acked 9605 Read 2 New Done Done false
25014 8607 Acked 9606 Read 2 New Done Done false
25015 8608 Acked 9607 Write 5 New Done Done false
25016 8609 Acked 9608 Write 3 New Done Done false
25017 8610 Acked 9609 Write 9 New Done Done false
25018 8611 Acked 9610 Write 8 New Done Done false
25019 8612 Acked 9611 Flush 0 New Done Done false
25020 8613 Acked 9612 Write 8 New Done Done false
25021 8614 Acked 9613 Write 2 New Done Done false
25022 8615 Acked 9614 Write 9 New Done Done false
25023 8616 Acked 9615 Read 1 New Done Done false
25024 8617 Acked 9616 Read 2 New Done Done false
25025 8618 Acked 9617 Read 4 New Done Done false
25026 8619 Acked 9618 Read 7 New Done Done false
25027 8620 Acked 9619 Read 2 New Done Done false
25028 8621 Acked 9620 Read 7 New Done Done false
25029 8622 Acked 9621 Write 7 New Done Done false
25030 8623 Acked 9622 Write 10 New Done Done false
25031 8624 Acked 9623 Read 6 New Done Done false
25032 8625 Acked 9624 Read 7 New Done Done false
25033 8626 Acked 9625 Read 4 New Done Done false
25034 8627 Acked 9626 Read 9 New Done Done false
25035 8628 Acked 9627 Write 9 New Done Done false
25036 8629 Acked 9628 Read 6 New Done Done false
25037 8630 Acked 9629 Read 10 New Done Done false
25038 8631 Acked 9630 Write 10 New Done Done false
25039 8632 Acked 9631 Write 7 New Done Done false
25040 8633 Acked 9632 Flush 0 New Done Done false
25041 8634 Acked 9633 Flush 0 New Done Done false
25042 8635 Acked 9634 Read 1 New Done Done false
25043 8636 Acked 9635 Read 6 New Done Done false
25044 8637 Acked 9636 Read 7 New Done Done false
25045 8638 Acked 9637 Write 10 New Done Done false
25046 8639 Acked 9638 Write 10 New Done Done false
25047 8640 Acked 9639 Read 9 New Done Done false
25048 8641 Acked 9640 Read 4 New Done Done false
25049 8642 Acked 9641 Write 10 New Done Done false
25050 8643 Acked 9642 Write 1 New Done Done false
25051 8644 Acked 9643 Read 10 New Done Done false
25052 8645 Acked 9644 Write 8 New Done Done false
25053 8646 Acked 9645 Write 7 New Done Done false
25054 8647 Acked 9646 Write 10 New Done Done false
25055 8648 Acked 9647 Read 5 New Done Done false
25056 8649 Acked 9648 Write 10 New Done Done false
25057 8650 Acked 9649 Read 7 New Done Done false
25058 8651 Acked 9650 Flush 0 New Done Done false
25059 8652 Acked 9651 Flush 0 New Done Done false
25060 8653 Acked 9652 Read 10 New Done Done false
25061 8654 Acked 9653 Flush 0 New Done Done false
25062 8655 Acked 9654 Read 5 New Done Done false
25063 8656 Acked 9655 Write 4 New Done Done false
25064 8657 Acked 9656 Read 5 New Done Done false
25065 8658 Acked 9657 Read 3 New Done Done false
25066 8659 Acked 9658 Write 7 New Done Done false
25067 8660 Acked 9659 Read 7 New Done Done false
25068 8661 Acked 9660 Write 5 New Done Done false
25069 8662 Acked 9661 Write 8 New Done Done false
25070 8663 Acked 9662 Write 1 New Done Done false
25071 8664 Acked 9663 Write 4 New Done Done false
25072 8665 Acked 9664 Write 2 New Done Done false
25073 8666 Acked 9665 Read 9 New Done Done false
25074 8667 Acked 9666 Write 6 New Done Done false
25075 8668 Acked 9667 Read 4 New Done Done false
25076 8669 Acked 9668 Read 5 New Done Done false
25077 8670 Acked 9669 Read 4 New Done Done false
25078 8671 Acked 9670 Read 7 New Done Done false
25079 8672 Acked 9671 Write 10 New Done Done false
25080 8673 Acked 9672 Read 3 New Done Done false
25081 8674 Acked 9673 Flush 0 New Done Done false
25082 8675 Acked 9674 Read 1 New Done Done false
25083 8676 Acked 9675 Read 2 New Done Done false
25084 8677 Acked 9676 Read 1 New Done Done false
25085 8678 Acked 9677 Read 6 New Done Done false
25086 8679 Acked 9678 Read 5 New Done Done false
25087 8680 Acked 9679 Write 1 New Done Done false
25088 8681 Acked 9680 Read 4 New Done Done false
25089 8682 Acked 9681 Read 8 New Done Done false
25090 8683 Acked 9682 Read 1 New Done Done false
25091 8684 Acked 9683 Write 4 New Done Done false
25092 8685 Acked 9684 Write 3 New Done Done false
25093 8686 Acked 9685 Read 1 New Done Done false
25094 8687 Acked 9686 Read 9 New Done Done false
25095 8688 Acked 9687 Read 7 New Done Done false
25096 8689 Acked 9688 Read 8 New Done Done false
25097 8690 Acked 9689 Flush 0 New Done Done false
25098 8691 Acked 9690 Read 4 New Done Done false
25099 8692 Acked 9691 Write 1 New Done Done false
25100 8693 Acked 9692 Write 7 New Done Done false
25101 8694 Acked 9693 Write 8 New Done Done false
25102 8695 Acked 9694 Flush 0 New Done Done false
25103 8696 Acked 9695 Write 5 New Done Done false
25104 8697 Acked 9696 Write 9 New Done Done false
25105 8698 Acked 9697 Read 4 New Done Done false
25106 8699 Acked 9698 Read 1 New Done Done false
25107 8700 Acked 9699 Read 1 New Done Done false
25108 8701 Acked 9700 Write 8 New Done Done false
25109 8702 Acked 9701 Read 7 New Done Done false
25110 8703 Acked 9702 Flush 0 New Done Done false
25111 8704 Acked 9703 Read 5 New Done Done false
25112 8705 Acked 9704 Read 5 New Done Done false
25113 8706 Acked 9705 Read 1 New Done Done false
25114 8707 Acked 9706 Read 3 New Done Done false
25115 8708 Acked 9707 Flush 0 New Done Done false
25116 8709 Acked 9708 Write 10 New Done Done false
25117 8710 Acked 9709 Read 7 New Done Done false
25118 8711 Acked 9710 Write 1 New Done Done false
25119 8712 Acked 9711 Write 6 New Done Done false
25120 8713 Acked 9712 Flush 0 New Done Done false
25121 8714 Acked 9713 Flush 0 New Done Done false
25122 8715 Acked 9714 Write 2 New Done Done false
25123 8716 Acked 9715 Read 6 New Done Done false
25124 8717 Acked 9716 Flush 0 New Done Done false
25125 8718 Acked 9717 Write 8 New Done Done false
25126 8719 Acked 9718 Write 8 New Done Done false
25127 8720 Acked 9719 Write 2 New Done Done false
25128 8721 Acked 9720 Read 4 New Done Done false
25129 8722 Acked 9721 Flush 0 New Done Done false
25130 8723 Acked 9722 Write 10 New Done Done false
25131 8724 Acked 9723 Flush 0 New Done Done false
25132 8725 Acked 9724 Write 7 New Done Done false
25133 8726 Acked 9725 Read 7 New Done Done false
25134 8727 Acked 9726 Read 3 New Done Done false
25135 8728 Acked 9727 Read 4 New Done Done false
25136 8729 Acked 9728 Write 3 New Done Done false
25137 8730 Acked 9729 Write 8 New Done Done false
25138 8731 Acked 9730 Read 4 New Done Done false
25139 8732 Acked 9731 Read 10 New Done Done false
25140 8733 Acked 9732 Write 2 New Done Done false
25141 8734 Acked 9733 Read 1 New Done Done false
25142 8735 Acked 9734 Write 4 New Done Done false
25143 8736 Acked 9735 Write 10 New Done Done false
25144 8737 Acked 9736 Write 2 New Done Done false
25145 8738 Acked 9737 Read 3 New Done Done false
25146 8739 Acked 9738 Read 7 New Done Done false
25147 8740 Acked 9739 Read 3 New Done Done false
25148 8741 Acked 9740 Write 5 New Done Done false
25149 STATES DS:0 DS:1 DS:2 TOTAL
25150 New 300 0 0 300
25151 Sent 0 0 0 0
25152 Done 0 300 300 600
25153 Skipped 0 0 0 0
25154 Error 0 0 0 0
25155 Last Flush: 9440 9723 9723
25156 Downstairs last five completed: 9440 9439 9438 9437 9436
25157 Upstairs last five completed: 8741 8740 8739 8738 8737
25158 CLIENT: Up:0 ds:300 act:2
251592024-03-02T01:10:36.763ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251602024-03-02T01:10:36.763ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251612024-03-02T01:10:36.763ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251622024-03-02T01:10:36.763ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251632024-03-02T01:10:36.763ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251642024-03-02T01:10:36.763ZINFOcrucible: [0] client re-new 301 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251652024-03-02T01:10:36.763ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25166 [8810] got start action
25167 [8810] I am going to restart
25168 Starting downstairs at port 8810
25169 Make output file at "/tmp/dsc/downstairs-8810.txt"
25170 [8810][0] reports Starting
251712024-03-02T01:10:37.765ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251722024-03-02T01:10:37.765ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251732024-03-02T01:10:37.765ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251742024-03-02T01:10:37.765ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251752024-03-02T01:10:37.765ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251762024-03-02T01:10:37.765ZINFOcrucible: [0] client re-new 301 jobs since flush 9440 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251772024-03-02T01:10:37.766ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25178 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
25179 [8810][0] reports Running
251802024-03-02T01:10:38.767ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251812024-03-02T01:10:38.767ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251822024-03-02T01:10:38.768ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251832024-03-02T01:10:38.768ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251842024-03-02T01:10:38.768ZINFOcrucible: send last flush ID to this DS: 9440 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251852024-03-02T01:10:38.768ZINFOcrucible: Replied this last flush ID: 9440 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251862024-03-02T01:10:38.768ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251872024-03-02T01:10:38.769ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251882024-03-02T01:10:38.769ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
251892024-03-02T01:10:38.769ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25190 ----------------------------------------------------------------
25191 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25192 Downstairs last five completed: 9741 9740 9739 9738 9737
25193 Upstairs last five completed: 8742 8741 8740 8739 8738
25194 CLIENT: Up:0 ds:0 act:3
25195 Replay: All jobs finished, all DS active.
25196 Main task has work to do, go find it
25197 got dsc Stop(0)
25198 stop 0
25199 Mar 02 01:10:40.107 INFO request completed, latency_us: 383[8810] Got stop action so:false kr:false
25200 , response_code: 204, uri: /stop/cid/0, method: POST, req_id: 915722a8-bb9c-417e-b752-fff11e494479, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
252012024-03-02T01:10:40.108ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252022024-03-02T01:10:40.108ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252032024-03-02T01:10:40.108ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25204 Mar 02 01:10:40.108 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:10:40.10840585Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"client":"0","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
25205 GET, req_id: 19907ea3-9d57-4dbc-a2b5-33b5e5fe2fa3, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.{"msg":"[0] client re-new 0 jobs since flush 9741","v":0,"name":"crucible","level":30,"time":"2024-03-02T01:10:40.108436598Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1162,"":"downstairs","":"downstairs","session_id":"b12e3267-1b4e-4723-a1fa-a377ace8dffb"}
25206 1:9998
252072024-03-02T01:10:40.108ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25208 [8810] Exited with: ExitStatus(unix_wait_status(9))
25209 [8810][0] reports Exit
252102024-03-02T01:10:41.108ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252112024-03-02T01:10:41.108ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252122024-03-02T01:10:41.108ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252132024-03-02T01:10:41.108ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252142024-03-02T01:10:41.108ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252152024-03-02T01:10:41.108ZINFOcrucible: [0] client re-new 0 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252162024-03-02T01:10:41.108ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252172024-03-02T01:10:42.110ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252182024-03-02T01:10:42.110ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252192024-03-02T01:10:42.110ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252202024-03-02T01:10:42.110ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252212024-03-02T01:10:42.110ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252222024-03-02T01:10:42.110ZINFOcrucible: [0] client re-new 0 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252232024-03-02T01:10:42.110ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252242024-03-02T01:10:43.111ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252252024-03-02T01:10:43.111ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252262024-03-02T01:10:43.111ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252272024-03-02T01:10:43.111ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252282024-03-02T01:10:43.111ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252292024-03-02T01:10:43.111ZINFOcrucible: [0] client re-new 0 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252302024-03-02T01:10:43.111ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25231 Mar 02 01:10:44.110 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/0, method: GET, req_id: 64ee110e-2c0a-4baa-a7b6-b4f26e82948c, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
25232 001/300 Flush
25233 002/300 Read block 3706 len 20480
252342024-03-02T01:10:44.112ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252352024-03-02T01:10:44.112ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252362024-03-02T01:10:44.112ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252372024-03-02T01:10:44.112ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252382024-03-02T01:10:44.112ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252392024-03-02T01:10:44.112ZINFOcrucible: [0] client re-new 2 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
252402024-03-02T01:10:44.112ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25241 003/300 Write block 988 len 16384 data: 8 7 6 7
25242 004/300 Flush
25243 005/300 Write block 3423 len 24576 data: 5 4 5 5 3 2
25244 006/300 Read block 3970 len 4096
25245 007/300 Write block 1017 len 32768 data: 7 9 9 8 6 6 6 5
25246 008/300 Read block 3352 len 40960
25247 009/300 Read block 4760 len 4096
25248 010/300 Write block 2719 len 32768 data: 9 10 8 8 8 7 6 7
25249 011/300 Write block 2451 len 36864 data: 5 4 4 6 6 8 7 8 8
25250 012/300 Read block 1614 len 24576
25251 013/300 Read block 4073 len 36864
25252 014/300 Write block 1655 len 40960 data: 8 6 5 5 5 4 5 5 5 6
25253 015/300 Write block 1096 len 12288 data: 7 8 7
25254 016/300 Write block 3776 len 36864 data: 6 7 8 12 14 13 12 12 12
25255 017/300 Flush
25256 018/300 Read block 4206 len 4096
25257 019/300 Write block 4291 len 16384 data: 5 6 4 5
25258 020/300 Write block 1955 len 36864 data: 7 7 7 6 6 5 6 5 3
25259 021/300 Flush
25260 022/300 Read block 809 len 40960
25261 023/300 Read block 1246 len 16384
25262 024/300 Read block 2173 len 40960
25263 025/300 Write block 2043 len 20480 data: 8 5 5 4 5
25264 026/300 Read block 957 len 8192
25265 027/300 Read block 4761 len 16384
25266 028/300 Read block 658 len 28672
25267 029/300 Read block 1239 len 20480
25268 030/300 Flush
25269 031/300 Read block 3306 len 8192
25270 032/300 Write block 2704 len 20480 data: 5 7 6 6 7
25271 033/300 Write block 691 len 32768 data: 6 8 9 9 9 10 9 7
25272 034/300 Read block 2098 len 4096
25273 035/300 Read block 2780 len 12288
25274 036/300 Read block 3822 len 28672
25275 037/300 Read block 591 len 4096
25276 038/300 Flush
25277 039/300 Read block 4204 len 8192
25278 040/300 Read block 100 len 20480
25279 041/300 Read block 2226 len 4096
25280 042/300 Write block 3183 len 40960 data: 5 4 4 5 6 6 5 5 4 3
25281 043/300 Write block 3134 len 8192 data: 5 6
25282 044/300 Read block 4369 len 28672
25283 045/300 Read block 701 len 20480
25284 046/300 Write block 730 len 16384 data: 8 7 8 10
25285 047/300 Write block 1696 len 4096 data: 7
25286 048/300 Read block 1510 len 36864
25287 049/300 Read block 4155 len 40960
25288 050/300 Read block 2219 len 28672
25289 051/300 Flush
25290 052/300 Write block 284 len 28672 data: 8 6 6 8 8 7 8
25291 053/300 Read block 3327 len 36864
25292 054/300 Write block 3602 len 4096 data: 2
25293 055/300 Write block 2537 len 36864 data: 4 5 5 5 6 6 8 6 5
25294 056/300 Write block 4068 len 4096 data: 5
25295 057/300 Read block 1593 len 8192
25296 058/300 Read block 2747 len 8192
25297 059/300 Read block 2420 len 8192
25298 060/300 Read block 2718 len 4096
25299 061/300 Write block 4370 len 36864 data: 5 6 7 7 7 7 6 6 6
25300 062/300 Flush
25301 063/300 Flush
25302 064/300 Read block 2265 len 20480
25303 065/300 Flush
25304 066/300 Write block 4956 len 36864 data: 9 9 7 6 6 6 6 5 6
25305 067/300 Read block 4551 len 12288
25306 068/300 Read block 4833 len 36864
25307 069/300 Read block 1865 len 8192
25308 070/300 Write block 2848 len 20480 data: 9 10 7 5 4
25309 071/300 Write block 4040 len 12288 data: 7 7 6
25310 072/300 Read block 4159 len 12288
25311 073/300 Write block 4206 len 12288 data: 6 7 6
25312 074/300 Write block 841 len 40960 data: 7 6 5 4 3 3 3 3 3 4
25313 075/300 Write block 1571 len 40960 data: 9 8 10 10 8 9 8 9 8 10
25314 076/300 Read block 2229 len 20480
25315 077/300 Write block 4738 len 12288 data: 5 5 5
25316 078/300 Read block 3870 len 40960
25317 079/300 Read block 4030 len 8192
25318 080/300 Write block 2527 len 28672 data: 7 7 6 6 6 7 7
25319 081/300 Read block 2301 len 16384
25320 082/300 Read block 2551 len 40960
25321 083/300 Write block 2594 len 40960 data: 5 5 4 3 4 3 3 2 3 4
25322 084/300 Write block 3257 len 40960 data: 3 4 5 5 7 7 5 5 6 5
25323 085/300 Write block 1307 len 4096 data: 5
25324 086/300 Write block 2191 len 20480 data: 4 3 3 4 5
25325 087/300 Write block 3771 len 20480 data: 5 5 5 4 5
25326 088/300 Write block 1755 len 36864 data: 6 6 4 5 7 6 5 3 3
25327 089/300 Read block 2680 len 24576
25328 090/300 Write block 1867 len 32768 data: 6 6 6 6 6 4 5 6
25329 091/300 Write block 4170 len 32768 data: 5 5 6 5 6 6 8 7
25330 092/300 Write block 1508 len 40960 data: 7 7 6 6 4 5 7 7 7 6
25331 093/300 Read block 243 len 36864
25332 094/300 Flush
25333 095/300 Write block 2177 len 36864 data: 8 8 7 4 4 3 3 3 3
25334 096/300 Write block 2716 len 8192 data: 8 7
25335 097/300 Read block 2542 len 4096
25336 098/300 Read block 553 len 8192
25337 099/300 Flush
25338 100/300 Read block 3649 len 36864
25339 101/300 Read block 2991 len 16384
25340 102/300 Write block 1308 len 28672 data: 4 5 5 5 5 5 5
25341 103/300 Write block 3985 len 28672 data: 7 9 8 8 7 9 9
25342 104/300 Write block 1435 len 24576 data: 8 5 8 7 6 7
25343 105/300 Write block 3806 len 36864 data: 6 7 8 7 7 4 2 3 4
25344 106/300 Read block 2022 len 4096
25345 107/300 Write block 4464 len 28672 data: 4 3 2 2 3 3 3
25346 108/300 Write block 2334 len 16384 data: 4 4 4 4
25347 109/300 Flush
25348 110/300 Write block 3136 len 36864 data: 7 8 10 8 7 7 7 6 6
25349 111/300 Flush
25350 112/300 Read block 1401 len 20480
25351 113/300 Read block 2120 len 36864
25352 114/300 Write block 143 len 32768 data: 4 4 5 5 6 5 6 5
25353 115/300 Read block 4587 len 36864
25354 116/300 Read block 2307 len 40960
25355 117/300 Read block 430 len 24576
25356 118/300 Read block 4858 len 40960
25357 119/300 Read block 4575 len 20480
25358 120/300 Flush
25359 121/300 Write block 115 len 8192 data: 3 5
25360 122/300 Write block 3140 len 24576 data: 8 8 8 7 7 6
25361 123/300 Read block 550 len 40960
25362 124/300 Write block 653 len 8192 data: 3 3
25363 125/300 Write block 4220 len 8192 data: 8 7
25364 126/300 Write block 467 len 24576 data: 9 9 10 9 8 8
25365 127/300 Write block 4243 len 24576 data: 4 4 4 4 3 3
25366 128/300 Read block 1289 len 32768
25367 129/300 Read block 4293 len 36864
25368 130/300 Read block 2035 len 12288
25369 131/300 Read block 1856 len 4096
25370 132/300 Write block 2157 len 40960 data: 4 5 6 4 4 3 4 4 5 7
25371 133/300 Read block 1503 len 36864
25372 134/300 Read block 1363 len 24576
25373 135/300 Write block 1633 len 20480 data: 7 6 6 8 7
25374 136/300 Write block 3832 len 20480 data: 8 5 4 4 5
25375 137/300 Write block 894 len 28672 data: 6 7 7 5 4 4 2
25376 138/300 Write block 4514 len 24576 data: 5 7 8 7 8 7
25377 139/300 Write block 4895 len 36864 data: 3 2 3 3 3 4 6 6 7
25378 140/300 Write block 4005 len 32768 data: 4 4 4 4 5 4 5 3
25379 141/300 Read block 555 len 24576
25380 142/300 Write block 2614 len 32768 data: 4 4 3 4 4 4 4 3
25381 143/300 Write block 4693 len 4096 data: 6
25382 144/300 Read block 2609 len 8192
25383 145/300 Write block 228 len 28672 data: 9 9 10 7 8 5 6
25384 146/300 Read block 4421 len 24576
25385 147/300 Flush
25386 148/300 Read block 3754 len 28672
25387 149/300 Read block 1014 len 28672
25388 150/300 Read block 924 len 32768
25389 151/300 Read block 3718 len 32768
25390 152/300 Write block 902 len 20480 data: 2 4 3 5 5
25391 153/300 Write block 4011 len 28672 data: 6 4 4 4 4 6 6
25392 154/300 Read block 3492 len 24576
25393 155/300 Read block 2095 len 32768
25394 156/300 Read block 351 len 24576
25395 157/300 Write block 304 len 40960 data: 7 7 6 6 6 6 5 5 3 4
25396 158/300 Write block 3824 len 20480 data: 5 5 7 6 5
25397 159/300 Read block 4207 len 4096
25398 160/300 Read block 500 len 36864
25399 161/300 Read block 3137 len 4096
25400 162/300 Write block 3092 len 4096 data: 4
25401 163/300 Read block 748 len 20480
25402 164/300 Read block 892 len 20480
25403 165/300 Flush
25404 166/300 Write block 814 len 16384 data: 10 8 8 9
25405 167/300 Read block 320 len 32768
25406 168/300 Write block 4441 len 20480 data: 6 5 5 5 6
25407 169/300 Read block 1400 len 24576
25408 170/300 Read block 1910 len 4096
25409 171/300 Read block 2358 len 20480
25410 172/300 Write block 2785 len 12288 data: 5 6 6
25411 173/300 Write block 4175 len 32768 data: 7 9 8 8 8 3 5 5
25412 174/300 Read block 2208 len 4096
25413 175/300 Read block 4753 len 40960
25414 176/300 Write block 4821 len 20480 data: 4 4 4 4 6
25415 177/300 Read block 113 len 8192
25416 178/300 Write block 1554 len 12288 data: 6 6 6
25417 179/300 Write block 580 len 36864 data: 5 5 5 6 7 8 7 7 8
25418 180/300 Write block 2532 len 16384 data: 8 8 7 6
25419 181/300 Read block 3753 len 8192
25420 182/300 Write block 3894 len 8192 data: 5 5
25421 183/300 Write block 3938 len 16384 data: 3 3 5 3
25422 184/300 Write block 423 len 20480 data: 6 7 6 5 7
25423 185/300 Read block 3744 len 16384
25424 186/300 Read block 1972 len 40960
25425 187/300 Flush
25426 188/300 Read block 233 len 32768
25427 189/300 Read block 3148 len 36864
25428 190/300 Flush
25429 191/300 Write block 2385 len 32768 data: 6 4 3 4 5 6 5 6
25430 192/300 Flush
25431 193/300 Read block 819 len 36864
25432 194/300 Flush
25433 195/300 Read block 1213 len 40960
25434 196/300 Write block 3204 len 20480 data: 6 5 9 10 11
25435 197/300 Write block 759 len 12288 data: 8 7 8
25436 198/300 Write block 4659 len 40960 data: 10 11 11 9 6 6 7 6 4 3
25437 199/300 Read block 3141 len 40960
25438 200/300 Write block 3096 len 32768 data: 8 8 8 5 5 3 3 3
25439 201/300 Read block 809 len 40960
25440 202/300 Flush
25441 203/300 Read block 3745 len 24576
25442 204/300 Write block 2661 len 4096 data: 9
25443 205/300 Write block 3708 len 20480 data: 10 9 8 8 10
25444 206/300 Read block 4891 len 36864
25445 207/300 Read block 1804 len 32768
25446 208/300 Write block 488 len 20480 data: 3 4 5 5 4
25447 209/300 Write block 2281 len 36864 data: 3 3 3 4 4 6 5 5 3
25448 210/300 Read block 900 len 36864
25449 211/300 Read block 2857 len 8192
25450 212/300 Write block 425 len 4096 data: 7
25451 213/300 Write block 3085 len 36864 data: 6 5 5 3 5 5 5 5 4
25452 214/300 Flush
25453 215/300 Read block 424 len 12288
25454 216/300 Read block 1416 len 40960
25455 217/300 Read block 4504 len 40960
25456 218/300 Write block 2970 len 32768 data: 6 6 7 5 5 5 5 5
25457 219/300 Read block 1181 len 40960
25458 220/300 Read block 3088 len 16384
25459 221/300 Read block 4801 len 40960
25460 222/300 Read block 281 len 8192
25461 223/300 Read block 1372 len 28672
25462 224/300 Write block 1406 len 36864 data: 7 7 4 4 5 5 5 5 4
25463 225/300 Read block 1952 len 32768
25464 226/300 Read block 3694 len 20480
25465 227/300 Flush
25466 228/300 Read block 1328 len 36864
25467 229/300 Write block 1647 len 32768 data: 6 5 6 9 8 6 6 6
25468 230/300 Write block 2620 len 12288 data: 5 4 4
25469 231/300 Read block 4686 len 40960
25470 232/300 Read block 4940 len 16384
25471 233/300 Write block 3049 len 28672 data: 3 3 3 6 6 6 5
25472 234/300 Read block 4391 len 20480
25473 235/300 Write block 1939 len 36864 data: 10 9 10 7 8 10 8 8 9
25474 236/300 Read block 2188 len 24576
25475 237/300 Read block 3417 len 4096
25476 238/300 Read block 992 len 12288
25477 239/300 Write block 3246 len 16384 data: 6 6 5 5
25478 240/300 Flush
25479 241/300 Read block 4328 len 8192
25480 242/300 Read block 489 len 8192
25481 243/300 Flush
25482 244/300 Write block 4465 len 28672 data: 4 3 3 4 4 4 3
25483 245/300 Write block 78 len 36864 data: 5 6 5 6 9 9 8 7 6
25484 246/300 Flush
25485 247/300 Read block 653 len 32768
25486 248/300 Read block 2995 len 12288
25487 249/300 Read block 3026 len 8192
25488 250/300 Read block 473 len 20480
25489 251/300 Write block 2146 len 16384 data: 8 7 6 7
25490 252/300 Read block 3768 len 16384
25491 253/300 Write block 2144 len 4096 data: 7
25492 254/300 Flush
25493 255/300 Read block 1196 len 36864
25494 256/300 Write block 1654 len 36864 data: 7 9 7 6 6 6 5 6 6
25495 257/300 Read block 4379 len 4096
25496 258/300 Read block 3100 len 36864
25497 259/300 Read block 2927 len 16384
25498 260/300 Write block 129 len 32768 data: 6 5 5 6 5 5 5 3
25499 261/300 Read block 412 len 24576
25500 262/300 Read block 694 len 36864
25501 263/300 Read block 1255 len 12288
25502 264/300 Read block 1875 len 24576
25503 265/300 Write block 66 len 40960 data: 5 7 8 9 9 7 6 7 7 7
25504 266/300 Read block 4387 len 36864
25505 267/300 Write block 2426 len 20480 data: 5 4 5 8 8
25506 268/300 Read block 1357 len 8192
25507 269/300 Read block 726 len 28672
25508 270/300 Flush
25509 271/300 Read block 2101 len 20480
25510 272/300 Write block 1525 len 40960 data: 6 8 7 10 10 10 10 8 8 6
25511 273/300 Write block 1347 len 16384 data: 5 5 6 7
25512 274/300 Write block 3302 len 24576 data: 3 3 3 3 4 3
25513 275/300 Read block 3386 len 28672
25514 276/300 Read block 3115 len 16384
25515 277/300 Read block 3233 len 40960
25516 278/300 Write block 1688 len 32768 data: 7 6 7 7 8 9 8 9
25517 279/300 Read block 523 len 8192
25518 280/300 Read block 2050 len 12288
25519 281/300 Read block 1550 len 12288
25520 282/300 Write block 806 len 24576 data: 5 4 4 4 5 6
25521 283/300 Flush
25522 284/300 Flush
25523 285/300 Write block 2845 len 40960 data: 8 7 7 10 11 8 6 5 3 5
25524 286/300 Read block 1284 len 12288
25525 287/300 Read block 4104 len 20480
25526 288/300 Write block 4057 len 8192 data: 8 7
25527 289/300 Read block 4281 len 16384
25528 290/300 Write block 1083 len 24576 data: 4 5 6 6 6 7
25529 291/300 Read block 625 len 4096
25530 292/300 Write block 485 len 12288 data: 4 4 2
25531 293/300 Write block 2133 len 8192 data: 6 5
25532 294/300 Write block 4594 len 12288 data: 8 6 7
25533 295/300 Write block 3631 len 24576 data: 8 6 5 6 5 5
25534 296/300 Read block 511 len 4096
25535 297/300 Read block 3203 len 8192
25536 298/300 Read block 1465 len 28672
25537 299/300 Read block 3282 len 4096
25538 300/300 Flush
25539 Main task has work to do, go find it
25540 got dsc Start(0)
25541 start 0
25542 Mar 02 01:10:44.460 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 2fdaa590-0747-4a3e-8ed3-c1b84b07bca4, remote_addr: 127.0.0.1:40735, local_addr: 127.0.0.1:9998
25543 Replay: started 0, returned:Ok(())
25544 ----------------------------------------------------------------
25545 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25546 States: Offline Active Active
25547 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25548 8743 Acked 9742 Flush 0 New Done Done true
25549 8744 Acked 9743 Read 5 New Done Done true
25550 8745 Acked 9744 Write 4 New Done Done false
25551 8746 Acked 9745 Flush 0 New Done Done false
25552 8747 Acked 9746 Write 6 New Done Done false
25553 8748 Acked 9747 Read 1 New Done Done false
25554 8749 Acked 9748 Write 8 New Done Done false
25555 8750 Acked 9749 Read 10 New Done Done false
25556 8751 Acked 9750 Read 1 New Done Done false
25557 8752 Acked 9751 Write 8 New Done Done false
25558 8753 Acked 9752 Write 9 New Done Done false
25559 8754 Acked 9753 Read 6 New Done Done false
25560 8755 Acked 9754 Read 9 New Done Done false
25561 8756 Acked 9755 Write 10 New Done Done false
25562 8757 Acked 9756 Write 3 New Done Done false
25563 8758 Acked 9757 Write 9 New Done Done false
25564 8759 Acked 9758 Flush 0 New Done Done false
25565 8760 Acked 9759 Read 1 New Done Done false
25566 8761 Acked 9760 Write 4 New Done Done false
25567 8762 Acked 9761 Write 9 New Done Done false
25568 8763 Acked 9762 Flush 0 New Done Done false
25569 8764 Acked 9763 Read 10 New Done Done false
25570 8765 Acked 9764 Read 4 New Done Done false
25571 8766 Acked 9765 Read 10 New Done Done false
25572 8767 Acked 9766 Write 5 New Done Done false
25573 8768 Acked 9767 Read 2 New Done Done false
25574 8769 Acked 9768 Read 4 New Done Done false
25575 8770 Acked 9769 Read 7 New Done Done false
25576 8771 Acked 9770 Read 5 New Done Done false
25577 8772 Acked 9771 Flush 0 New Done Done false
25578 8773 Acked 9772 Read 2 New Done Done false
25579 8774 Acked 9773 Write 5 New Done Done false
25580 8775 Acked 9774 Write 8 New Done Done false
25581 8776 Acked 9775 Read 1 New Done Done false
25582 8777 Acked 9776 Read 3 New Done Done false
25583 8778 Acked 9777 Read 7 New Done Done false
25584 8779 Acked 9778 Read 1 New Done Done false
25585 8780 Acked 9779 Flush 0 New Done Done false
25586 8781 Acked 9780 Read 2 New Done Done false
25587 8782 Acked 9781 Read 5 New Done Done false
25588 8783 Acked 9782 Read 1 New Done Done false
25589 8784 Acked 9783 Write 10 New Done Done false
25590 8785 Acked 9784 Write 2 New Done Done false
25591 8786 Acked 9785 Read 7 New Done Done false
25592 8787 Acked 9786 Read 5 New Done Done false
25593 8788 Acked 9787 Write 4 New Done Done false
25594 8789 Acked 9788 Write 1 New Done Done false
25595 8790 Acked 9789 Read 9 New Done Done false
25596 8791 Acked 9790 Read 10 New Done Done false
25597 8792 Acked 9791 Read 7 New Done Done false
25598 8793 Acked 9792 Flush 0 New Done Done false
25599 8794 Acked 9793 Write 7 New Done Done false
25600 8795 Acked 9794 Read 9 New Done Done false
25601 8796 Acked 9795 Write 1 New Done Done false
25602 8797 Acked 9796 Write 9 New Done Done false
25603 8798 Acked 9797 Write 1 New Done Done false
25604 8799 Acked 9798 Read 2 New Done Done false
25605 8800 Acked 9799 Read 2 New Done Done false
25606 8801 Acked 9800 Read 2 New Done Done false
25607 8802 Acked 9801 Read 1 New Done Done false
25608 8803 Acked 9802 Write 9 New Done Done false
25609 8804 Acked 9803 Flush 0 New Done Done false
25610 8805 Acked 9804 Flush 0 New Done Done false
25611 8806 Acked 9805 Read 5 New Done Done false
25612 8807 Acked 9806 Flush 0 New Done Done false
25613 8808 Acked 9807 Write 9 New Done Done false
25614 8809 Acked 9808 Read 3 New Done Done false
25615 8810 Acked 9809 Read 9 New Done Done false
25616 8811 Acked 9810 Read 2 New Done Done false
25617 8812 Acked 9811 Write 5 New Done Done false
25618 8813 Acked 9812 Write 3 New Done Done false
25619 8814 Acked 9813 Read 3 New Done Done false
25620 8815 Acked 9814 Write 3 New Done Done false
25621 8816 Acked 9815 Write 10 New Done Done false
25622 8817 Acked 9816 Write 10 New Done Done false
25623 8818 Acked 9817 Read 5 New Done Done false
25624 8819 Acked 9818 Write 3 New Done Done false
25625 8820 Acked 9819 Read 10 New Done Done false
25626 8821 Acked 9820 Read 2 New Done Done false
25627 8822 Acked 9821 Write 7 New Done Done false
25628 8823 Acked 9822 Read 4 New Done Done false
25629 8824 Acked 9823 Read 10 New Done Done false
25630 8825 Acked 9824 Write 10 New Done Done false
25631 8826 Acked 9825 Write 10 New Done Done false
25632 8827 Acked 9826 Write 1 New Done Done false
25633 8828 Acked 9827 Write 5 New Done Done false
25634 8829 Acked 9828 Write 5 New Done Done false
25635 8830 Acked 9829 Write 9 New Done Done false
25636 8831 Acked 9830 Read 6 New Done Done false
25637 8832 Acked 9831 Write 8 New Done Done false
25638 8833 Acked 9832 Write 8 New Done Done false
25639 8834 Acked 9833 Write 10 New Done Done false
25640 8835 Acked 9834 Read 9 New Done Done false
25641 8836 Acked 9835 Flush 0 New Done Done false
25642 8837 Acked 9836 Write 9 New Done Done false
25643 8838 Acked 9837 Write 2 New Done Done false
25644 8839 Acked 9838 Read 1 New Done Done false
25645 8840 Acked 9839 Read 2 New Done Done false
25646 8841 Acked 9840 Flush 0 New Done Done false
25647 8842 Acked 9841 Read 9 New Done Done false
25648 8843 Acked 9842 Read 4 New Done Done false
25649 8844 Acked 9843 Write 7 New Done Done false
25650 8845 Acked 9844 Write 7 New Done Done false
25651 8846 Acked 9845 Write 6 New Done Done false
25652 8847 Acked 9846 Write 9 New Done Done false
25653 8848 Acked 9847 Read 1 New Done Done false
25654 8849 Acked 9848 Flush 0 New Done Done false
25655 8850 Acked 9849 Write 7 New Done Done false
25656 8851 Acked 9850 Write 4 New Done Done false
25657 8852 Acked 9851 Flush 0 New Done Done false
25658 8853 Acked 9852 Write 9 New Done Done false
25659 8854 Acked 9853 Flush 0 New Done Done false
25660 8855 Acked 9854 Read 5 New Done Done false
25661 8856 Acked 9855 Read 9 New Done Done false
25662 8857 Acked 9856 Write 8 New Done Done false
25663 8858 Acked 9857 Read 9 New Done Done false
25664 8859 Acked 9858 Read 10 New Done Done false
25665 8860 Acked 9859 Read 6 New Done Done false
25666 8861 Acked 9860 Read 10 New Done Done false
25667 8862 Acked 9861 Read 5 New Done Done false
25668 8863 Acked 9862 Flush 0 New Done Done false
25669 8864 Acked 9863 Write 2 New Done Done false
25670 8865 Acked 9864 Write 6 New Done Done false
25671 8866 Acked 9865 Read 10 New Done Done false
25672 8867 Acked 9866 Write 2 New Done Done false
25673 8868 Acked 9867 Write 2 New Done Done false
25674 8869 Acked 9868 Write 6 New Done Done false
25675 8870 Acked 9869 Write 6 New Done Done false
25676 8871 Acked 9870 Read 8 New Done Done false
25677 8872 Acked 9871 Read 9 New Done Done false
25678 8873 Acked 9872 Read 3 New Done Done false
25679 8874 Acked 9873 Read 1 New Done Done false
25680 8875 Acked 9874 Write 10 New Done Done false
25681 8876 Acked 9875 Read 9 New Done Done false
25682 8877 Acked 9876 Read 6 New Done Done false
25683 8878 Acked 9877 Write 5 New Done Done false
25684 8879 Acked 9878 Write 5 New Done Done false
25685 8880 Acked 9879 Write 7 New Done Done false
25686 8881 Acked 9880 Write 6 New Done Done false
25687 8882 Acked 9881 Write 9 New Done Done false
25688 8883 Acked 9882 Write 8 New Done Done false
25689 8884 Acked 9883 Read 6 New Done Done false
25690 8885 Acked 9884 Write 8 New Done Done false
25691 8886 Acked 9885 Write 1 New Done Done false
25692 8887 Acked 9886 Read 2 New Done Done false
25693 8888 Acked 9887 Write 7 New Done Done false
25694 8889 Acked 9888 Read 6 New Done Done false
25695 8890 Acked 9889 Flush 0 New Done Done false
25696 8891 Acked 9890 Read 7 New Done Done false
25697 8892 Acked 9891 Read 7 New Done Done false
25698 8893 Acked 9892 Read 8 New Done Done false
25699 8894 Acked 9893 Read 8 New Done Done false
25700 8895 Acked 9894 Write 5 New Done Done false
25701 8896 Acked 9895 Write 7 New Done Done false
25702 8897 Acked 9896 Read 6 New Done Done false
25703 8898 Acked 9897 Read 8 New Done Done false
25704 8899 Acked 9898 Read 6 New Done Done false
25705 8900 Acked 9899 Write 10 New Done Done false
25706 8901 Acked 9900 Write 5 New Done Done false
25707 8902 Acked 9901 Read 1 New Done Done false
25708 8903 Acked 9902 Read 9 New Done Done false
25709 8904 Acked 9903 Read 1 New Done Done false
25710 8905 Acked 9904 Write 1 New Done Done false
25711 8906 Acked 9905 Read 5 New Done Done false
25712 8907 Acked 9906 Read 5 New Done Done false
25713 8908 Acked 9907 Flush 0 New Done Done false
25714 8909 Acked 9908 Write 4 New Done Done false
25715 8910 Acked 9909 Read 8 New Done Done false
25716 8911 Acked 9910 Write 5 New Done Done false
25717 8912 Acked 9911 Read 6 New Done Done false
25718 8913 Acked 9912 Read 1 New Done Done false
25719 8914 Acked 9913 Read 5 New Done Done false
25720 8915 Acked 9914 Write 3 New Done Done false
25721 8916 Acked 9915 Write 8 New Done Done false
25722 8917 Acked 9916 Read 1 New Done Done false
25723 8918 Acked 9917 Read 10 New Done Done false
25724 8919 Acked 9918 Write 5 New Done Done false
25725 8920 Acked 9919 Read 2 New Done Done false
25726 8921 Acked 9920 Write 3 New Done Done false
25727 8922 Acked 9921 Write 9 New Done Done false
25728 8923 Acked 9922 Write 4 New Done Done false
25729 8924 Acked 9923 Read 2 New Done Done false
25730 8925 Acked 9924 Write 2 New Done Done false
25731 8926 Acked 9925 Write 4 New Done Done false
25732 8927 Acked 9926 Write 5 New Done Done false
25733 8928 Acked 9927 Read 4 New Done Done false
25734 8929 Acked 9928 Read 10 New Done Done false
25735 8930 Acked 9929 Flush 0 New Done Done false
25736 8931 Acked 9930 Read 8 New Done Done false
25737 8932 Acked 9931 Read 9 New Done Done false
25738 8933 Acked 9932 Flush 0 New Done Done false
25739 8934 Acked 9933 Write 8 New Done Done false
25740 8935 Acked 9934 Flush 0 New Done Done false
25741 8936 Acked 9935 Read 9 New Done Done false
25742 8937 Acked 9936 Flush 0 New Done Done false
25743 8938 Acked 9937 Read 10 New Done Done false
25744 8939 Acked 9938 Write 5 New Done Done false
25745 8940 Acked 9939 Write 3 New Done Done false
25746 8941 Acked 9940 Write 10 New Done Done false
25747 8942 Acked 9941 Read 10 New Done Done false
25748 8943 Acked 9942 Write 8 New Done Done false
25749 8944 Acked 9943 Read 10 New Done Done false
25750 8945 Acked 9944 Flush 0 New Done Done false
25751 8946 Acked 9945 Read 6 New Done Done false
25752 8947 Acked 9946 Write 1 New Done Done false
25753 8948 Acked 9947 Write 5 New Done Done false
25754 8949 Acked 9948 Read 9 New Done Done false
25755 8950 Acked 9949 Read 8 New Done Done false
25756 8951 Acked 9950 Write 5 New Done Done false
25757 8952 Acked 9951 Write 9 New Done Done false
25758 8953 Acked 9952 Read 9 New Done Done false
25759 8954 Acked 9953 Read 2 New Done Done false
25760 8955 Acked 9954 Write 1 New Done Done false
25761 8956 Acked 9955 Write 9 New Done Done false
25762 8957 Acked 9956 Flush 0 New Done Done false
25763 8958 Acked 9957 Read 3 New Done Done false
25764 8959 Acked 9958 Read 10 New Done Done false
25765 8960 Acked 9959 Read 10 New Done Done false
25766 8961 Acked 9960 Write 8 New Done Done false
25767 8962 Acked 9961 Read 10 New Done Done false
25768 8963 Acked 9962 Read 4 New Done Done false
25769 8964 Acked 9963 Read 10 New Done Done false
25770 8965 Acked 9964 Read 2 New Done Done false
25771 8966 Acked 9965 Read 7 New Done Done false
25772 8967 Acked 9966 Write 9 New Done Done false
25773 8968 Acked 9967 Read 8 New Done Done false
25774 8969 Acked 9968 Read 5 New Done Done false
25775 8970 Acked 9969 Flush 0 New Done Done false
25776 8971 Acked 9970 Read 9 New Done Done false
25777 8972 Acked 9971 Write 8 New Done Done false
25778 8973 Acked 9972 Write 3 New Done Done false
25779 8974 Acked 9973 Read 10 New Done Done false
25780 8975 Acked 9974 Read 4 New Done Done false
25781 8976 Acked 9975 Write 7 New Done Done false
25782 8977 Acked 9976 Read 5 New Done Done false
25783 8978 Acked 9977 Write 9 New Done Done false
25784 8979 Acked 9978 Read 6 New Done Done false
25785 8980 Acked 9979 Read 1 New Done Done false
25786 8981 Acked 9980 Read 3 New Done Done false
25787 8982 Acked 9981 Write 4 New Done Done false
25788 8983 Acked 9982 Flush 0 New Done Done false
25789 8984 Acked 9983 Read 2 New Done Done false
25790 8985 Acked 9984 Read 2 New Done Done false
25791 8986 Acked 9985 Flush 0 New Done Done false
25792 8987 Acked 9986 Write 7 New Done Done false
25793 8988 Acked 9987 Write 9 New Done Done false
25794 8989 Acked 9988 Flush 0 New Done Done false
25795 8990 Acked 9989 Read 8 New Done Done false
25796 8991 Acked 9990 Read 3 New Done Done false
25797 8992 Acked 9991 Read 2 New Done Done false
25798 8993 Acked 9992 Read 5 New Done Done false
25799 8994 Acked 9993 Write 4 New Done Done false
25800 8995 Acked 9994 Read 4 New Done Done false
25801 8996 Acked 9995 Write 1 New Done Done false
25802 8997 Acked 9996 Flush 0 New Done Done false
25803 8998 Acked 9997 Read 9 New Done Done false
25804 8999 Acked 9998 Write 9 New Done Done false
25805 9000 Acked 9999 Read 1 New Done Done false
25806 9001 Acked 10000 Read 9 New Done Done false
25807 9002 Acked 10001 Read 4 New Done Done false
25808 9003 Acked 10002 Write 8 New Done Done false
25809 9004 Acked 10003 Read 6 New Done Done false
25810 9005 Acked 10004 Read 9 New Done Done false
25811 9006 Acked 10005 Read 3 New Done Done false
25812 9007 Acked 10006 Read 6 New Done Done false
25813 9008 Acked 10007 Write 10 New Done Done false
25814 9009 Acked 10008 Read 9 New Done Done false
25815 9010 Acked 10009 Write 5 New Done Done false
25816 9011 Acked 10010 Read 2 New Done Done false
25817 9012 Acked 10011 Read 7 New Done Done false
25818 9013 Acked 10012 Flush 0 New Done Done false
25819 9014 Acked 10013 Read 5 New Done Done false
25820 9015 Acked 10014 Write 10 New Done Done false
25821 9016 Acked 10015 Write 4 New Done Done false
25822 9017 Acked 10016 Write 6 New Done Done false
25823 9018 Acked 10017 Read 7 New Done Done false
25824 9019 Acked 10018 Read 4 New Done Done false
25825 9020 Acked 10019 Read 10 New Done Done false
25826 9021 Acked 10020 Write 8 New Done Done false
25827 9022 Acked 10021 Read 2 New Done Done false
25828 9023 Acked 10022 Read 3 New Done Done false
25829 9024 Acked 10023 Read 3 New Done Done false
25830 9025 Acked 10024 Write 6 New Done Done false
25831 9026 Acked 10025 Flush 0 New Done Done false
25832 9027 Acked 10026 Flush 0 New Done Done false
25833 9028 Acked 10027 Write 10 New Done Done false
25834 9029 Acked 10028 Read 3 New Done Done false
25835 9030 Acked 10029 Read 5 New Done Done false
25836 9031 Acked 10030 Write 2 New Done Done false
25837 9032 Acked 10031 Read 4 New Done Done false
25838 9033 Acked 10032 Write 6 New Done Done false
25839 9034 Acked 10033 Read 1 New Done Done false
25840 9035 Acked 10034 Write 3 New Done Done false
25841 9036 Acked 10035 Write 2 New Done Done false
25842 9037 Acked 10036 Write 3 New Done Done false
25843 9038 Acked 10037 Write 6 New Done Done false
25844 9039 Acked 10038 Read 1 New Done Done false
25845 9040 Acked 10039 Read 2 New Done Done false
25846 9041 Acked 10040 Read 7 New Done Done false
25847 9042 Acked 10041 Read 1 New Done Done false
25848 9043 Acked 10042 Flush 0 New Done Done false
25849 STATES DS:0 DS:1 DS:2 TOTAL
25850 New 301 0 0 301
25851 Sent 0 0 0 0
25852 Done 0 301 301 602
25853 Skipped 0 0 0 0
25854 Error 0 0 0 0
25855 Last Flush: 9741 10042 10042
25856 Downstairs last five completed: 9741 9740 9739 9738 9737
25857 Upstairs last five completed: 9043 9042 9041 9040 9039
25858 CLIENT: Up:0 ds:301 act:2
258592024-03-02T01:10:45.113ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258602024-03-02T01:10:45.113ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258612024-03-02T01:10:45.113ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258622024-03-02T01:10:45.113ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258632024-03-02T01:10:45.113ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258642024-03-02T01:10:45.114ZINFOcrucible: [0] client re-new 301 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258652024-03-02T01:10:45.114ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25866 [8810] got start action
25867 [8810] I am going to restart
25868 Starting downstairs at port 8810
25869 Make output file at "/tmp/dsc/downstairs-8810.txt"
258702024-03-02T01:10:46.115ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25871 [8810][0] reports Starting
258722024-03-02T01:10:46.115ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258732024-03-02T01:10:46.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258742024-03-02T01:10:46.115ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258752024-03-02T01:10:46.115ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258762024-03-02T01:10:46.115ZINFOcrucible: [0] client re-new 301 jobs since flush 9741 = downstairs session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258772024-03-02T01:10:46.115ZINFOcrucible: client task is exiting = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25878 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1192)
25879 [8810][0] reports Running
258802024-03-02T01:10:47.117ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258812024-03-02T01:10:47.117ZINFOcrucible: ds_connection connected = io task client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258822024-03-02T01:10:47.118ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258832024-03-02T01:10:47.118ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258842024-03-02T01:10:47.118ZINFOcrucible: send last flush ID to this DS: 9741 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258852024-03-02T01:10:47.118ZINFOcrucible: Replied this last flush ID: 9741 = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258862024-03-02T01:10:47.118ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258872024-03-02T01:10:47.118ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258882024-03-02T01:10:47.118ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
258892024-03-02T01:10:47.118ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = b12e3267-1b4e-4723-a1fa-a377ace8dffb
25890 ----------------------------------------------------------------
25891 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25892 Downstairs last five completed: 10042 10041 10040 10039 10038
25893 Upstairs last five completed: 9043 9042 9041 9040 9039
25894 CLIENT: Up:0 ds:0 act:3
25895 Replay: All jobs finished, all DS active.
25896 Test replay has completed
25897 Wrote out file "alan"
25898 CLIENT: Tests done. All submitted work has been ACK'd
25899 ----------------------------------------------------------------
25900 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25901 Downstairs last five completed: 10042 10041 10040 10039 10038
25902 Upstairs last five completed: 9043 9042 9041 9040 9039
25903 CLIENT: Up:0 ds:0 act:3
25904 CLIENT: All jobs finished, all DS active.
259052024-03-02T01:10:48.466ZWARNcrucible: Guest handle has been dropped
25906 Replays:30 time: 5:10
25907 Do final verify
259082024-03-02T01:10:48.497ZINFOcrucible: Upstairs starts
259092024-03-02T01:10:48.497ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "1c1574fb721f98f2df1b23e3fd27d83be018882e", git_commit_timestamp: "2024-03-02T00:46:46.000000000Z", git_branch: "main", rustc_semver: "1.76.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "07dca489ac2d933c78d3c5158e3f43beefeb02ce", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
259102024-03-02T01:10:48.497ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
25911 Crucible runtime is spawned
259122024-03-02T01:10:48.497ZINFOcrucible: Crucible 550d7fb4-ead9-4012-ba04-b391575814aa has session id: a143ea5e-dbd7-4757-8448-ae6c24c466eb session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259132024-03-02T01:10:48.497ZINFOcrucible: Upstairs opts: Upstairs UUID: 550d7fb4-ead9-4012-ba04-b391575814aa, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259142024-03-02T01:10:48.497ZINFOcrucible: Crucible stats registered with UUID: 550d7fb4-ead9-4012-ba04-b391575814aa session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259152024-03-02T01:10:48.497ZINFOcrucible: The guest has requested activation with gen:3
259162024-03-02T01:10:48.497ZINFOcrucible: 550d7fb4-ead9-4012-ba04-b391575814aa active request set session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259172024-03-02T01:10:48.497ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259182024-03-02T01:10:48.497ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259192024-03-02T01:10:48.497ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259202024-03-02T01:10:48.497ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259212024-03-02T01:10:48.497ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259222024-03-02T01:10:48.497ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259232024-03-02T01:10:48.497ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259242024-03-02T01:10:48.497ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259252024-03-02T01:10:48.497ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259262024-03-02T01:10:48.498ZINFOcrucible: ds_connection connected = io task client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259272024-03-02T01:10:48.498ZINFOcrucible: ds_connection connected = io task client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259282024-03-02T01:10:48.498ZINFOcrucible: ds_connection connected = io task client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259292024-03-02T01:10:48.498ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259302024-03-02T01:10:48.498ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259312024-03-02T01:10:48.499ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259322024-03-02T01:10:48.499ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259332024-03-02T01:10:48.499ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259342024-03-02T01:10:48.499ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259352024-03-02T01:10:48.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259362024-03-02T01:10:48.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259372024-03-02T01:10:48.499ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259382024-03-02T01:10:48.500ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259392024-03-02T01:10:48.500ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259402024-03-02T01:10:48.500ZINFOcrucible: Waiting for true more clients to be ready session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259412024-03-02T01:10:48.500ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259422024-03-02T01:10:48.500ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259432024-03-02T01:10:48.500ZINFOcrucible: Waiting for true more clients to be ready session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259442024-03-02T01:10:48.500ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259452024-03-02T01:10:48.500ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259462024-03-02T01:10:48.500ZINFOcrucible: [0]R flush_numbers[0..12]: [953, 953, 940, 941, 956, 942, 940, 946, 954, 941, 956, 923] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259472024-03-02T01:10:48.500ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259482024-03-02T01:10:48.500ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259492024-03-02T01:10:48.500ZINFOcrucible: [1]R flush_numbers[0..12]: [953, 953, 940, 941, 956, 942, 940, 946, 954, 941, 956, 923] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259502024-03-02T01:10:48.500ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259512024-03-02T01:10:48.500ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259522024-03-02T01:10:48.500ZINFOcrucible: [2]R flush_numbers[0..12]: [953, 953, 940, 941, 956, 942, 940, 946, 954, 941, 956, 923] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259532024-03-02T01:10:48.500ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
25954 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
25955 ,"time":"2024-03-02T01:10:48.500674766Z","hostname":"ip-10-150-1-64.us-west-2.compute.internal","pid":1197,"":"downstairs","":"downstairs","session_id":"a143ea5e-dbd7-4757-8448-ae6c24c466eb"}
259562024-03-02T01:10:48.500ZINFOcrucible: Max found gen is 3 = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259572024-03-02T01:10:48.500ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259582024-03-02T01:10:48.500ZINFOcrucible: Next flush: 957 = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259592024-03-02T01:10:48.500ZINFOcrucible: All extents match = downstairs session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259602024-03-02T01:10:48.500ZINFOcrucible: No downstairs reconciliation required session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259612024-03-02T01:10:48.500ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259622024-03-02T01:10:48.500ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259632024-03-02T01:10:48.500ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259642024-03-02T01:10:48.500ZINFOcrucible: All required reconciliation work is completed session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259652024-03-02T01:10:48.500ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259662024-03-02T01:10:48.500ZINFOcrucible: 550d7fb4-ead9-4012-ba04-b391575814aa is now active with session: a143ea5e-dbd7-4757-8448-ae6c24c466eb session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259672024-03-02T01:10:48.500ZINFOcrucible: Set Active after no reconciliation session_id = a143ea5e-dbd7-4757-8448-ae6c24c466eb
259682024-03-02T01:10:48.500ZINFOcrucible: The guest has finished waiting for activation with:3
25969 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
25970 Loading write count information from file "alan"
25971 Read and Verify all blocks (0..5000 range:false)
25972 Wrote out file "alan"
25973 Verify test completed
25974 Wrote out file "alan"
25975 CLIENT: Tests done. All submitted work has been ACK'd
25976 ----------------------------------------------------------------
25977 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:6
25978 States: Active Active Active
25979 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25980 46 Acked 1045 Read 100 Done Done Done false
25981 47 Acked 1046 Read 100 Done Done Done false
25982 48 Acked 1047 Read 100 Done Done Done false
25983 49 Acked 1048 Read 100 Done Done Done false
25984 50 Acked 1049 Read 100 Done Done Done false
25985 51 Acked 1050 Read 100 Done Done Done false
25986 STATES DS:0 DS:1 DS:2 TOTAL
25987 New 0 0 0 0
25988 Sent 0 0 0 0
25989 Done 6 6 6 18
25990 Skipped 0 0 0 0
25991 Error 0 0 0 0
25992 Last Flush: 1044 1044 1044
25993 Downstairs last five completed: 1044 1043 1042 1041 1040
25994 Upstairs last five completed: 51 50 49 48 47
25995 CLIENT: Up:0 ds:6 act:3
25996 ----------------------------------------------------------------
25997 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
25998 Downstairs last five completed: 1051 1050 1049 1048 1047
25999 Upstairs last five completed: 52 51 50 49 48
26000 CLIENT: Up:0 ds:0 act:3
26001 CLIENT: All crucible jobs finished, exiting program
260022024-03-02T01:10:53.072ZWARNcrucible: Guest handle has been dropped
26003 Mar 02 01:10:53.131 INFO accepted connection, remote_addr: 127.0.0.1:43231, local_addr: 127.0.0.1:9998
26004 Main task has work to do, go find it
26005 got dsc Shutdown
26006 Shutdown
26007 Shut it down
26008 [8810] Disable keep_running
26009 Mar 02 01:10:53.131[8810] Got stop action so:false kr:false
26010 [8830] Disable keep_running
26011 INFO request completed, latency_us: 185, response_code: 204, uri: /shutdown, method: POST, req_id: 7712c97f-cd0f-44ac-82df-c5e1f9c17622, remote_addr: 127.0.[8830] Got stop action so:false kr:false
26012 0.1:43231, local_addr: 127.0.0.1:9998
26013 [8820] Disable keep_running
26014 [8820] Got stop action so:false kr:false
26015 [8830] Exited with: ExitStatus(unix_wait_status(9))
26016 [8820] Exited with: ExitStatus(unix_wait_status(9))
26017 [8830][2] reports Exit
26018 [8820][1] reports Exit
26019 [8810] Exited with: ExitStatus(unix_wait_status(9))
26020 [8810][0] reports Exit
26021 March 2, 2024 at 01:10:58 AM UTC Test ends with 0