Max level shown:
1 starting January 29, 2025 at 05:10:33 AM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2597485
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.0213452
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.021717742
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start access at:127.0.0.1:9998
14 start ds: 8820
15 start ds: 8830
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Make output file at "/tmp/dsc/downstairs-8820.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1192)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1193)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1194)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Jan 29 05:10:33.679 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Jan 29 05:10:38.741 INFO accepted connection, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
36 Jan 29 05:10:38.743 INFO request completed, latency_us: 627, response_code: 200, uri: /regioninfo, method: GET, req_id: 34056824-35d1-47fa-a788-4b1957ac395a, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
37 Jan 29 05:10:38.745 INFO request completed, latency_us: 94, response_code: 200, uri: /regioncount, method: GET, req_id: 28fc23f5-12ec-40a2-b8fb-9eb693cb18a5, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
382025-01-29T05:10:38.745ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-01-29T05:10:38.745ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Jan 29 05:10:38.746 INFO request completed, latency_us: 135, response_code: 200, uri: /port/cid/0, method: GET, req_id: a64b0c4c-2f57-4be5-9cea-b331328e7048, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
41 Jan 29 05:10:38.746 INFO request completed, latency_us: 89, response_code: 200, uri: /port/cid/1, method: GET, req_id: e2217494-25dc-47f8-84ca-951e3ef47a8a, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
42 Jan 29 05:10:38.747 INFO request completed, latency_us: 95, response_code: 200, uri: /port/cid/2, method: GET, req_id: 98e2fde9-27a5-4c69-86bb-757242349d48, remote_addr: 127.0.0.1:58285, local_addr: 127.0.0.1:9998
432025-01-29T05:10:38.747ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-01-29T05:10:38.748ZINFOcrucible: Upstairs starts
452025-01-29T05:10:38.748ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "442be2395eb3bf8ebe11d34acf0678a1aae4fa71", git_commit_timestamp: "2025-01-29T04:48:13.000000000Z", git_branch: "main", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-01-29T05:10:38.748ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-01-29T05:10:38.748ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-01-29T05:10:38.748ZINFOcrucible: Crucible b921dc5a-f7fb-4eaa-acd7-90404b90203c has session id: d5150fed-e7a4-484e-9875-a11b4461e9c9 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
492025-01-29T05:10:38.749ZINFOcrucible: Upstairs opts: Upstairs UUID: b921dc5a-f7fb-4eaa-acd7-90404b90203c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
502025-01-29T05:10:38.749ZINFOcrucible: Crucible stats registered with UUID: b921dc5a-f7fb-4eaa-acd7-90404b90203c session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
512025-01-29T05:10:38.749ZWARNcrucible: could not find Downstairs address for Nexus session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
522025-01-29T05:10:38.750ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-01-29T05:10:38.750ZINFOcrucible: The guest has requested activation with gen:1
542025-01-29T05:10:38.750ZINFOcrucible: b921dc5a-f7fb-4eaa-acd7-90404b90203c active request set session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
552025-01-29T05:10:38.750ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
562025-01-29T05:10:38.750ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
572025-01-29T05:10:38.750ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
582025-01-29T05:10:38.750ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
592025-01-29T05:10:38.750ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
602025-01-29T05:10:38.750ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
612025-01-29T05:10:38.750ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
622025-01-29T05:10:38.750ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
632025-01-29T05:10:38.750ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
642025-01-29T05:10:38.750ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64383) = io task client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
652025-01-29T05:10:38.750ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43956) = io task client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
662025-01-29T05:10:38.750ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46423) = io task client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
672025-01-29T05:10:38.753ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
682025-01-29T05:10:38.753ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
692025-01-29T05:10:38.753ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
702025-01-29T05:10:38.754ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
712025-01-29T05:10:38.754ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
722025-01-29T05:10:38.754ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
732025-01-29T05:10:38.755ZINFOcrucible: Waiting for more clients to be ready session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
742025-01-29T05:10:38.755ZINFOcrucible: Waiting for more clients to be ready session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
752025-01-29T05:10:38.755ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
762025-01-29T05:10:38.755ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
772025-01-29T05:10:38.755ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
782025-01-29T05:10:38.755ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
792025-01-29T05:10:38.755ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
802025-01-29T05:10:38.755ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
812025-01-29T05:10:38.755ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
822025-01-29T05:10:38.755ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
832025-01-29T05:10:38.755ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
842025-01-29T05:10:38.755ZINFOcrucible: Max found gen is 1 = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
852025-01-29T05:10:38.755ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
862025-01-29T05:10:38.755ZINFOcrucible: Next flush: 1 = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
872025-01-29T05:10:38.755ZINFOcrucible: All extents match = downstairs session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
882025-01-29T05:10:38.755ZINFOcrucible: No downstairs reconciliation required session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
892025-01-29T05:10:38.755ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
902025-01-29T05:10:38.755ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
91 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
92 Wait for a query_work_queue command to finish before sending IO
93 ,"time":"2025-01-29T05:10:38.75590062Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1198,"client":"2","":"downstairs","session_id":"d5150fed-e7a4-484e-9875-a11b4461e9c9"}
942025-01-29T05:10:38.755ZINFOcrucible: All required reconciliation work is completed session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
952025-01-29T05:10:38.755ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
962025-01-29T05:10:38.755ZINFOcrucible: b921dc5a-f7fb-4eaa-acd7-90404b90203c is now active with session: d5150fed-e7a4-484e-9875-a11b4461e9c9 session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
972025-01-29T05:10:38.755ZINFOcrucible: Set Active after no reconciliation session_id = d5150fed-e7a4-484e-9875-a11b4461e9c9
982025-01-29T05:10:38.755ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 136 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 256 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
143 Replay loop starts now January 29, 2025 at 05:10:43 AM UTC
144 Jan 29 05:10:43.397 INFO accepted connection, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
145 Jan 29 05:10:43.398 INFO request completed, latency_us: 146, response_code: 200, uri: /regioninfo, method: GET, req_id: 5106f03e-f65d-413c-abbd-82fb27583398, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
1462025-01-29T05:10:43.399ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
147 Jan 29 05:10:43.399 INFO request completed, latency_us: 110, response_code: 200, uri: /regioncount, method: GET, req_id: 409c7606-3ed1-4a8d-8b8b-154a7b6386dc, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
1482025-01-29T05:10:43.399ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
149 Jan 29 05:10:43.400 INFO request completed, latency_us: 131, response_code: 200, uri: /port/cid/0, method: GET, req_id: 4311006e-1112-46eb-8571-f2acdfb1b2bb, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
150 Jan 29 05:10:43.400 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/1, method: GET, req_id: 66b81df9-b4dc-456b-979b-c7d86d637ba7, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
151 Jan 29 05:10:43.401 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/2, method: GET, req_id: 0fb42685-0780-40a4-9a07-3169233e6475, remote_addr: 127.0.0.1:54391, local_addr: 127.0.0.1:9998
1522025-01-29T05:10:43.401ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1532025-01-29T05:10:43.401ZINFOcrucible: Upstairs starts
1542025-01-29T05:10:43.401ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "442be2395eb3bf8ebe11d34acf0678a1aae4fa71", git_commit_timestamp: "2025-01-29T04:48:13.000000000Z", git_branch: "main", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1552025-01-29T05:10:43.401ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1562025-01-29T05:10:43.401ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1572025-01-29T05:10:43.402ZINFOcrucible: Crucible b1bc6540-1566-4764-a26e-7290fdeb71eb has session id: 3b9f53a2-1179-49c9-b40f-5354d75015be session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1582025-01-29T05:10:43.402ZINFOcrucible: Upstairs opts: Upstairs UUID: b1bc6540-1566-4764-a26e-7290fdeb71eb, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1592025-01-29T05:10:43.402ZINFOcrucible: Crucible stats registered with UUID: b1bc6540-1566-4764-a26e-7290fdeb71eb session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1602025-01-29T05:10:43.402ZWARNcrucible: could not find Downstairs address for Nexus session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1612025-01-29T05:10:43.403ZINFOcrucible: Downstairs in volume = 3 task = crutest
1622025-01-29T05:10:43.403ZINFOcrucible: The guest has requested activation with gen:2
1632025-01-29T05:10:43.403ZINFOcrucible: b1bc6540-1566-4764-a26e-7290fdeb71eb active request set session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1642025-01-29T05:10:43.403ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1652025-01-29T05:10:43.403ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1662025-01-29T05:10:43.403ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1672025-01-29T05:10:43.403ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1682025-01-29T05:10:43.403ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1692025-01-29T05:10:43.403ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1702025-01-29T05:10:43.403ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1712025-01-29T05:10:43.403ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1722025-01-29T05:10:43.403ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1732025-01-29T05:10:43.403ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57671) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1742025-01-29T05:10:43.403ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38982) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1752025-01-29T05:10:43.403ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43521) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1762025-01-29T05:10:43.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1772025-01-29T05:10:43.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1782025-01-29T05:10:43.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1792025-01-29T05:10:43.404ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1802025-01-29T05:10:43.405ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1812025-01-29T05:10:43.405ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1822025-01-29T05:10:43.405ZINFOcrucible: Waiting for more clients to be ready session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1832025-01-29T05:10:43.405ZINFOcrucible: Waiting for more clients to be ready session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1842025-01-29T05:10:43.405ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1852025-01-29T05:10:43.405ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1862025-01-29T05:10:43.405ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1872025-01-29T05:10:43.405ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1882025-01-29T05:10:43.405ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
189 {"msg":"Activate successful
190 Wait for a query_work_queue command to finish before sending IO
191 [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:10:43.406019961Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"downstairs","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
1922025-01-29T05:10:43.406ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1932025-01-29T05:10:43.406ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1942025-01-29T05:10:43.406ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1952025-01-29T05:10:43.406ZINFOcrucible: Max found gen is 2 = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1962025-01-29T05:10:43.406ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1972025-01-29T05:10:43.406ZINFOcrucible: Next flush: 2 = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
198 {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:10:43.406120173Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
199 1202,"":"downstairs","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
2002025-01-29T05:10:43.406ZINFOcrucible: No downstairs reconciliation required session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2012025-01-29T05:10:43.406ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2022025-01-29T05:10:43.406ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2032025-01-29T05:10:43.406ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2042025-01-29T05:10:43.406ZINFOcrucible: All required reconciliation work is completed session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2052025-01-29T05:10:43.406ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2062025-01-29T05:10:43.406ZINFOcrucible: b1bc6540-1566-4764-a26e-7290fdeb71eb is now active with session: 3b9f53a2-1179-49c9-b40f-5354d75015be session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2072025-01-29T05:10:43.406ZINFOcrucible: Set Active after no reconciliation session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2082025-01-29T05:10:43.406ZINFOcrucible: The guest has finished waiting for activation with:2
209 Loading write count information from file "/tmp/test_replay_verify.log"
210 Jan 29 05:10:43.439 INFO accepted connection, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
211 Main task has work to do, go find it
212 got dsc Stop(2)
213 stop 2
214 Jan 29 05:10:43.439 INFO request completed, latency_us: 161, response_code: 204, uri: /stop/cid/2[8830] Got stop action so:false kr:false
215 , method: POST, req_id: 3e58fbf3-9444-452a-8b3b-037ef0d26773, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
2162025-01-29T05:10:43.440ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2172025-01-29T05:10:43.440ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2182025-01-29T05:10:43.440ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2192025-01-29T05:10:43.440ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220 Jan 29 05:10:43.440 INFO request completed, latency_us: 325, response_code: 200, uri: /state/cid/2, method: GET, req_id: fb6308cb-d603-4928-99e9-6985e827dc46, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
221 [8830] Exited with: ExitStatus(unix_wait_status(9))
222 [8830][2] reports Exit
223 Jan 29 05:10:47.443 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2b8c5851-5fdc-4963-a964-1cf4644a6e45, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
224 001/300 Write block 4004 len 36864 data: 2 2 2 2 2 2 2 2 2
225 002/300 Read block 4298 len 16384
226 003/300 Write block 1041 len 40960 data: 2 2 2 2 2 2 2 2 2 2
227 004/300 Write block 2008 len 32768 data: 2 2 2 2 2 2 2 2
228 005/300 Write block 192 len 12288 data: 2 2 2
229 006/300 Write block 216 len 28672 data: 2 2 2 2 2 2 2
230 007/300 Read block 974 len 20480
231 008/300 Write block 913 len 24576 data: 2 2 2 2 2 2
232 009/300 Read block 939 len 4096
233 010/300 Read block 1440 len 28672
234 011/300 Read block 2990 len 32768
235 012/300 Flush
236 013/300 Read block 2492 len 20480
237 014/300 Read block 1488 len 8192
238 015/300 Read block 93 len 36864
239 016/300 Write block 2259 len 32768 data: 2 2 2 2 2 2 2 2
240 017/300 Write block 3641 len 20480 data: 2 2 2 2 2
241 018/300 Write block 4042 len 36864 data: 2 2 2 2 2 2 2 2 2
242 019/300 Read block 4297 len 24576
243 020/300 Write block 2944 len 28672 data: 2 2 2 2 2 2 2
244 021/300 Read block 2417 len 16384
245 022/300 Read block 4220 len 12288
246 023/300 Read block 4586 len 24576
247 024/300 Write block 3216 len 40960 data: 2 2 2 2 2 2 2 2 2 2
248 025/300 Write block 2322 len 8192 data: 2 2
249 026/300 Flush
250 027/300 Read block 1579 len 24576
251 028/300 Read block 210 len 32768
252 029/300 Flush
253 030/300 Read block 987 len 32768
254 031/300 Read block 3088 len 24576
255 032/300 Write block 4200 len 24576 data: 2 2 2 2 2 2
256 033/300 Flush
257 034/300 Read block 1834 len 36864
258 035/300 Write block 3439 len 20480 data: 2 2 2 2 2
259 036/300 Read block 2314 len 32768
260 037/300 Read block 3069 len 28672
261 038/300 Write block 3002 len 24576 data: 2 2 2 2 2 2
262 039/300 Read block 4570 len 40960
263 040/300 Read block 3318 len 36864
264 041/300 Write block 3283 len 32768 data: 2 2 2 2 2 2 2 2
265 042/300 Write block 877 len 36864 data: 2 2 2 2 2 2 2 2 2
266 043/300 Write block 4292 len 20480 data: 2 2 2 2 2
267 044/300 Write block 2314 len 28672 data: 2 2 2 2 2 2 2
268 045/300 Write block 439 len 4096 data: 2
269 046/300 Read block 2379 len 8192
270 047/300 Write block 4773 len 32768 data: 2 2 2 2 2 2 2 2
271 048/300 Read block 4449 len 36864
272 049/300 Flush
273 050/300 Write block 1690 len 8192 data: 2 2
274 051/300 Write block 1617 len 12288 data: 2 2 2
275 052/300 Write block 44 len 32768 data: 2 2 2 2 2 2 2 2
276 053/300 Write block 4392 len 16384 data: 2 2 2 2
277 054/300 Read block 3085 len 36864
278 055/300 Read block 1125 len 16384
279 056/300 Read block 2422 len 20480
280 057/300 Read block 82 len 8192
281 058/300 Read block 1971 len 40960
282 059/300 Read block 678 len 36864
283 060/300 Write block 736 len 40960 data: 2 2 2 2 2 2 2 2 2 2
284 061/300 Write block 865 len 40960 data: 2 2 2 2 2 2 2 2 2 2
285 062/300 Read block 314 len 40960
286 063/300 Write block 589 len 40960 data: 2 2 2 2 2 2 2 2 2 2
287 064/300 Read block 2001 len 40960
288 065/300 Flush
289 066/300 Read block 3544 len 8192
290 067/300 Read block 2434 len 4096
291 068/300 Read block 3740 len 32768
292 069/300 Read block 3585 len 4096
293 070/300 Write block 3414 len 40960 data: 2 2 2 2 2 2 2 2 2 2
294 071/300 Write block 3757 len 8192 data: 2 2
295 072/300 Write block 3382 len 4096 data: 2
296 073/300 Read block 582 len 4096
297 074/300 Read block 1203 len 20480
298 075/300 Read block 3243 len 24576
299 076/300 Read block 3000 len 40960
300 077/300 Write block 2341 len 28672 data: 2 2 2 2 2 2 2
301 078/300 Read block 2058 len 16384
302 079/300 Read block 979 len 20480
303 080/300 Read block 3488 len 8192
304 081/300 Flush
305 082/300 Read block 4565 len 36864
306 083/300 Read block 583 len 4096
307 084/300 Read block 4862 len 32768
308 085/300 Read block 1132 len 16384
309 086/300 Read block 862 len 36864
310 087/300 Read block 1097 len 20480
311 088/300 Write block 1931 len 4096 data: 2
312 089/300 Write block 1745 len 40960 data: 2 2 2 2 2 2 2 2 2 2
313 090/300 Write block 2789 len 8192 data: 2 2
314 091/300 Write block 3711 len 40960 data: 2 2 2 2 2 2 2 2 2 2
315 092/300 Read block 4720 len 36864
316 093/300 Write block 1213 len 20480 data: 2 2 2 2 2
317 094/300 Read block 4348 len 16384
318 095/300 Write block 3989 len 28672 data: 2 2 2 2 2 2 2
319 096/300 Write block 3364 len 4096 data: 2
320 097/300 Write block 527 len 40960 data: 2 2 2 2 2 2 2 2 2 2
321 098/300 Read block 3883 len 36864
322 099/300 Write block 1690 len 40960 data: 3 3 2 2 2 2 2 2 2 2
323 100/300 Write block 216 len 28672 data: 3 3 3 3 3 3 3
324 101/300 Write block 2932 len 4096 data: 2
325 102/300 Read block 4289 len 12288
326 103/300 Write block 2779 len 36864 data: 2 2 2 2 2 2 2 2 2
327 104/300 Flush
328 105/300 Write block 1949 len 16384 data: 2 2 2 2
329 106/300 Write block 4953 len 28672 data: 2 2 2 2 2 2 2
330 107/300 Flush
331 108/300 Write block 2748 len 36864 data: 2 2 2 2 2 2 2 2 2
332 109/300 Flush
333 110/300 Read block 2411 len 20480
334 111/300 Write block 689 len 16384 data: 2 2 2 2
335 112/300 Write block 2286 len 24576 data: 2 2 2 2 2 2
336 113/300 Write block 2686 len 16384 data: 2 2 2 2
337 114/300 Write block 4502 len 16384 data: 2 2 2 2
338 115/300 Read block 3774 len 28672
339 116/300 Read block 2912 len 20480
340 117/300 Write block 2659 len 20480 data: 2 2 2 2 2
341 118/300 Write block 1369 len 28672 data: 2 2 2 2 2 2 2
342 119/300 Write block 4528 len 8192 data: 2 2
343 120/300 Write block 2960 len 32768 data: 2 2 2 2 2 2 2 2
344 121/300 Flush
345 122/300 Read block 910 len 20480
346 123/300 Write block 4675 len 40960 data: 2 2 2 2 2 2 2 2 2 2
347 124/300 Read block 2178 len 16384
348 125/300 Read block 2185 len 16384
349 126/300 Read block 29 len 12288
350 127/300 Write block 3018 len 16384 data: 2 2 2 2
351 128/300 Write block 2470 len 16384 data: 2 2 2 2
352 129/300 Write block 2627 len 28672 data: 2 2 2 2 2 2 2
353 130/300 Write block 4110 len 4096 data: 2
354 131/300 Read block 3991 len 12288
355 132/300 Read block 1077 len 28672
356 133/300 Flush
357 134/300 Read block 4692 len 40960
358 135/300 Read block 3054 len 36864
359 136/300 Read block 2772 len 36864
360 137/300 Write block 952 len 16384 data: 2 2 2 2
361 138/300 Flush
362 139/300 Read block 2319 len 12288
363 140/300 Read block 4433 len 4096
364 141/300 Read block 439 len 4096
365 142/300 Read block 1582 len 24576
366 143/300 Read block 2119 len 20480
367 144/300 Write block 2532 len 4096 data: 2
368 145/300 Read block 4023 len 20480
369 146/300 Read block 3608 len 36864
370 147/300 Read block 4880 len 32768
371 148/300 Read block 1529 len 24576
372 149/300 Read block 299 len 8192
373 150/300 Write block 1272 len 8192 data: 2 2
374 151/300 Write block 3115 len 24576 data: 2 2 2 2 2 2
375 152/300 Write block 2489 len 20480 data: 2 2 2 2 2
376 153/300 Flush
377 154/300 Read block 1877 len 8192
378 155/300 Read block 3358 len 4096
379 156/300 Write block 3438 len 8192 data: 2 3
380 157/300 Read block 344 len 16384
381 158/300 Write block 308 len 36864 data: 2 2 2 2 2 2 2 2 2
382 159/300 Read block 3148 len 40960
383 160/300 Write block 4859 len 24576 data: 2 2 2 2 2 2
384 161/300 Write block 608 len 40960 data: 2 2 2 2 2 2 2 2 2 2
385 162/300 Write block 3450 len 8192 data: 2 2
386 163/300 Write block 2402 len 32768 data: 2 2 2 2 2 2 2 2
387 164/300 Read block 690 len 36864
388 165/300 Read block 3887 len 12288
389 166/300 Read block 4294 len 32768
390 167/300 Write block 4280 len 20480 data: 2 2 2 2 2
391 168/300 Flush
392 169/300 Read block 3784 len 12288
393 170/300 Write block 2281 len 16384 data: 2 2 2 2
394 171/300 Read block 2316 len 16384
395 172/300 Write block 3513 len 32768 data: 2 2 2 2 2 2 2 2
396 173/300 Read block 2620 len 20480
397 174/300 Read block 4777 len 24576
398 175/300 Write block 3954 len 20480 data: 2 2 2 2 2
399 176/300 Read block 4988 len 8192
400 177/300 Read block 2673 len 40960
401 178/300 Write block 2609 len 32768 data: 2 2 2 2 2 2 2 2
402 179/300 Flush
403 180/300 Write block 3416 len 32768 data: 3 3 3 3 3 3 3 3
404 181/300 Read block 1255 len 40960
405 182/300 Read block 2013 len 20480
406 183/300 Write block 4058 len 40960 data: 2 2 2 2 2 2 2 2 2 2
407 184/300 Read block 2509 len 16384
408 185/300 Write block 3074 len 40960 data: 2 2 2 2 2 2 2 2 2 2
409 186/300 Read block 3481 len 32768
410 187/300 Read block 313 len 36864
411 188/300 Read block 3290 len 32768
412 189/300 Write block 2635 len 40960 data: 2 2 2 2 2 2 2 2 2 2
413 190/300 Read block 4674 len 36864
414 191/300 Read block 2491 len 36864
415 192/300 Read block 610 len 8192
416 193/300 Write block 4449 len 36864 data: 2 2 2 2 2 2 2 2 2
417 194/300 Read block 1535 len 20480
418 195/300 Write block 3558 len 24576 data: 2 2 2 2 2 2
419 196/300 Write block 4638 len 12288 data: 2 2 2
420 197/300 Read block 3245 len 20480
421 198/300 Read block 1175 len 28672
422 199/300 Flush
423 200/300 Write block 4 len 8192 data: 2 2
424 201/300 Read block 4648 len 4096
425 202/300 Read block 834 len 40960
426 203/300 Read block 2033 len 28672
427 204/300 Write block 4678 len 36864 data: 3 3 3 3 3 3 3 2 2
428 205/300 Flush
429 206/300 Write block 2425 len 12288 data: 2 2 2
430 207/300 Flush
431 208/300 Flush
432 209/300 Read block 3551 len 12288
433 210/300 Write block 1443 len 36864 data: 2 2 2 2 2 2 2 2 2
434 211/300 Read block 4176 len 8192
435 212/300 Read block 4244 len 36864
436 213/300 Write block 1309 len 36864 data: 2 2 2 2 2 2 2 2 2
437 214/300 Write block 1483 len 12288 data: 2 2 2
438 215/300 Read block 0 len 28672
439 216/300 Write block 731 len 20480 data: 2 2 2 2 2
440 217/300 Flush
441 218/300 Flush
442 219/300 Read block 4110 len 36864
443 220/300 Write block 1628 len 28672 data: 2 2 2 2 2 2 2
444 221/300 Read block 4388 len 28672
445 222/300 Read block 3824 len 36864
446 223/300 Flush
447 224/300 Read block 2681 len 28672
448 225/300 Write block 2111 len 8192 data: 2 2
449 226/300 Read block 716 len 28672
450 227/300 Read block 2648 len 12288
451 228/300 Flush
452 229/300 Read block 2973 len 4096
453 230/300 Write block 967 len 28672 data: 2 2 2 2 2 2 2
454 231/300 Write block 2553 len 28672 data: 2 2 2 2 2 2 2
455 232/300 Read block 1496 len 28672
456 233/300 Read block 3926 len 24576
457 234/300 Write block 3771 len 40960 data: 2 2 2 2 2 2 2 2 2 2
458 235/300 Write block 891 len 8192 data: 2 2
459 236/300 Write block 3942 len 36864 data: 2 2 2 2 2 2 2 2 2
460 237/300 Read block 1716 len 16384
461 238/300 Write block 2531 len 20480 data: 2 3 2 2 2
462 239/300 Write block 4370 len 20480 data: 2 2 2 2 2
463 240/300 Read block 466 len 28672
464 241/300 Write block 2416 len 36864 data: 2 2 2 2 2 2 2 2 2
465 242/300 Write block 1704 len 40960 data: 2 2 2 2 2 2 2 2 2 2
466 243/300 Read block 243 len 36864
467 244/300 Read block 3210 len 20480
468 245/300 Write block 4610 len 16384 data: 2 2 2 2
469 246/300 Read block 3872 len 20480
470 247/300 Write block 4641 len 32768 data: 2 2 2 2 2 2 2 2
471 248/300 Write block 2573 len 24576 data: 2 2 2 2 2 2
472 249/300 Write block 4277 len 8192 data: 2 2
473 250/300 Flush
474 251/300 Flush
475 252/300 Write block 4838 len 32768 data: 2 2 2 2 2 2 2 2
476 253/300 Read block 2786 len 28672
477 254/300 Write block 2293 len 20480 data: 2 2 2 2 2
478 255/300 Read block 4474 len 32768
479 256/300 Read block 4151 len 4096
480 257/300 Read block 2494 len 16384
481 258/300 Read block 3620 len 32768
482 259/300 Read block 11 len 40960
483 260/300 Write block 961 len 12288 data: 2 2 2
484 261/300 Read block 1514 len 28672
485 262/300 Flush
486 263/300 Read block 3348 len 40960
487 264/300 Write block 144 len 4096 data: 2
488 265/300 Write block 1028 len 8192 data: 2 2
489 266/300 Read block 210 len 16384
490 267/300 Write block 2111 len 16384 data: 3 3 2 2
491 268/300 Read block 1424 len 28672
492 269/300 Read block 345 len 4096
493 270/300 Write block 478 len 40960 data: 2 2 2 2 2 2 2 2 2 2
494 271/300 Read block 3738 len 24576
495 272/300 Write block 48 len 36864 data: 3 3 3 3 2 2 2 2 2
496 273/300 Write block 2310 len 32768 data: 2 2 2 2 3 3 3 3
497 274/300 Write block 241 len 28672 data: 2 2 2 2 2 2 2
498 275/300 Read block 4289 len 12288
499 276/300 Read block 2195 len 12288
500 277/300 Read block 673 len 28672
501 278/300 Read block 2031 len 16384
502 279/300 Write block 3444 len 28672 data: 2 2 2 2 2 2 3
503 280/300 Flush
504 281/300 Read block 1049 len 4096
505 282/300 Write block 4458 len 36864 data: 2 2 2 2 2 2 2 2 2
506 283/300 Read block 3674 len 8192
507 284/300 Read block 4626 len 36864
508 285/300 Flush
509 286/300 Write block 1672 len 20480 data: 2 2 2 2 2
510 287/300 Write block 2337 len 24576 data: 2 2 2 2 3 3
511 288/300 Read block 3693 len 16384
512 289/300 Flush
513 290/300 Flush
514 291/300 Flush
515 292/300 Write block 2881 len 8192 data: 2 2
516 293/300 Read block 4491 len 4096
517 294/300 Read block 1302 len 24576
518 295/300 Read block 2484 len 36864
519 296/300 Read block 3760 len 16384
520 297/300 Read block 2230 len 36864
521 298/300 Read block 1282 len 32768
522 299/300 Write block 3816 len 8192 data: 2 2
523 300/300 Read block 132 len 40960
524 301/300 Read block 2872 len 4096
525 Main task has work to do, go find it
526 got dsc Start(2)
527 start 2
528 Jan 29 05:10:47.691 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: b4386bbb-bfe8-43df-bbaf-c3e640d27c95, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
529 Replay: started 2, returned:Ok(())
530 ----------------------------------------------------------------
531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
532 States: Active Active Offline
533 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
534 1000 Acked Write 9 Done Done Sent false
535 1001 Acked Read 4 Done Done Sent false
536 1002 Acked Write 10 Done Done Sent false
537 1003 Acked Write 8 Done Done Sent false
538 1004 Acked Write 3 Done Done Sent false
539 1005 Acked Write 7 Done Done Sent false
540 1006 Acked Read 5 Done Done Sent false
541 1007 Acked Write 6 Done Done Sent false
542 1008 Acked Read 1 Done Done Sent false
543 1009 Acked Read 7 Done Done Sent false
544 1010 Acked Read 8 Done Done Sent false
545 1011 Acked Flush 0 Done Done Sent false
546 1012 Acked Read 5 Done Done Sent false
547 1013 Acked Read 2 Done Done Sent false
548 1014 Acked Read 9 Done Done Sent false
549 1015 Acked Write 8 Done Done Sent false
550 1016 Acked Write 5 Done Done Sent false
551 1017 Acked Write 9 Done Done Sent false
552 1018 Acked Read 6 Done Done Sent false
553 1019 Acked Write 7 Done Done Sent false
554 1020 Acked Read 4 Done Done Sent false
555 1021 Acked Read 3 Done Done Sent false
556 1022 Acked Read 6 Done Done Sent false
557 1023 Acked Write 10 Done Done Sent false
558 1024 Acked Write 2 Done Done Sent false
559 1025 Acked Flush 0 Done Done Sent false
560 1026 Acked Read 6 Done Done Sent false
561 1027 Acked Read 8 Done Done Sent false
562 1028 Acked Flush 0 Done Done Sent false
563 1029 Acked Read 8 Done Done Sent false
564 1030 Acked Read 6 Done Done Sent false
565 1031 Acked Write 6 Done Done Sent false
566 1032 Acked Flush 0 Done Done Sent false
567 1033 Acked Read 9 Done Done Sent false
568 1034 Acked Write 5 Done Done Sent false
569 1035 Acked Read 8 Done Done Sent false
570 1036 Acked Read 7 Done Done Sent false
571 1037 Acked Write 6 Done Done Sent false
572 1038 Acked Read 10 Done Done Sent false
573 1039 Acked Read 9 Done Done Sent false
574 1040 Acked Write 8 Done Done Sent false
575 1041 Acked Write 9 Done Done Sent false
576 1042 Acked Write 5 Done Done Sent false
577 1043 Acked Write 7 Done Done Sent false
578 1044 Acked Write 1 Done Done Sent false
579 1045 Acked Read 2 Done Done Sent false
580 1046 Acked Write 8 Done Done Sent false
581 1047 Acked Read 9 Done Done Sent false
582 1048 Acked Flush 0 Done Done Sent false
583 1049 Acked Write 2 Done Done Sent false
584 1050 Acked Write 3 Done Done Sent false
585 1051 Acked Write 8 Done Done Sent false
586 1052 Acked Write 4 Done Done Sent false
587 1053 Acked Read 9 Done Done Sent false
588 1054 Acked Read 4 Done Done Sent false
589 1055 Acked Read 5 Done Done Sent false
590 1056 Acked Read 2 Done Done Sent false
591 1057 Acked Read 10 Done Done Sent false
592 1058 Acked Read 9 Done Done Sent false
593 1059 Acked Write 10 Done Done Sent false
594 1060 Acked Write 10 Done Done Sent false
595 1061 Acked Read 10 Done Done Sent false
596 1062 Acked Write 10 Done Done Sent false
597 1063 Acked Read 10 Done Done Sent false
598 1064 Acked Flush 0 Done Done Sent false
599 1065 Acked Read 2 Done Done Sent false
600 1066 Acked Read 1 Done Done Sent false
601 1067 Acked Read 8 Done Done Sent false
602 1068 Acked Read 1 Done Done Sent false
603 1069 Acked Write 10 Done Done Sent false
604 1070 Acked Write 2 Done Done Sent false
605 1071 Acked Write 1 Done Done Sent false
606 1072 Acked Read 1 Done Done Sent false
607 1073 Acked Read 5 Done Done Sent false
608 1074 Acked Read 6 Done Done Sent false
609 1075 Acked Read 10 Done Done Sent false
610 1076 Acked Write 7 Done Done Sent false
611 1077 Acked Read 4 Done Done Sent false
612 1078 Acked Read 5 Done Done Sent false
613 1079 Acked Read 2 Done Done Sent false
614 1080 Acked Flush 0 Done Done Sent false
615 1081 Acked Read 9 Done Done Sent false
616 1082 Acked Read 1 Done Done Sent false
617 1083 Acked Read 8 Done Done Sent false
618 1084 Acked Read 4 Done Done Sent false
619 1085 Acked Read 9 Done Done Sent false
620 1086 Acked Read 5 Done Done Sent false
621 1087 Acked Write 1 Done Done Sent false
622 1088 Acked Write 10 Done Done Sent false
623 1089 Acked Write 2 Done Done Sent false
624 1090 Acked Write 10 Done Done Sent false
625 1091 Acked Read 9 Done Done Sent false
626 1092 Acked Write 5 Done Done Sent false
627 1093 Acked Read 4 Done Done Sent false
628 1094 Acked Write 7 Done Done Sent false
629 1095 Acked Write 1 Done Done Sent false
630 1096 Acked Write 10 Done Done Sent false
631 1097 Acked Read 9 Done Done Sent false
632 1098 Acked Write 10 Done Done Sent false
633 1099 Acked Write 7 Done Done Sent false
634 1100 Acked Write 1 Done Done Sent false
635 1101 Acked Read 3 Done Done Sent false
636 1102 Acked Write 9 Done Done Sent false
637 1103 Acked Flush 0 Done Done Sent false
638 1104 Acked Write 4 Done Done Sent false
639 1105 Acked Write 7 Done Done Sent false
640 1106 Acked Flush 0 Done Done Sent false
641 1107 Acked Write 9 Done Done Sent false
642 1108 Acked Flush 0 Done Done Sent false
643 1109 Acked Read 5 Done Done Sent false
644 1110 Acked Write 4 Done Done Sent false
645 1111 Acked Write 6 Done Done Sent false
646 1112 Acked Write 4 Done Done Sent false
647 1113 Acked Write 4 Done Done Sent false
648 1114 Acked Read 7 Done Done Sent false
649 1115 Acked Read 5 Done Done Sent false
650 1116 Acked Write 5 Done Done Sent false
651 1117 Acked Write 7 Done Done Sent false
652 1118 Acked Write 2 Done Done Sent false
653 1119 Acked Write 8 Done Done Sent false
654 1120 Acked Flush 0 Done Done Sent false
655 1121 Acked Read 5 Done Done Sent false
656 1122 Acked Write 10 Done Done Sent false
657 1123 Acked Read 4 Done Done Sent false
658 1124 Acked Read 4 Done Done Sent false
659 1125 Acked Read 3 Done Done Sent false
660 1126 Acked Write 4 Done Done Sent false
661 1127 Acked Write 4 Done Done Sent false
662 1128 Acked Write 7 Done Done Sent false
663 1129 Acked Write 1 Done Done Sent false
664 1130 Acked Read 3 Done Done Sent false
665 1131 Acked Read 7 Done Done Sent false
666 1132 Acked Flush 0 Done Done Sent false
667 1133 Acked Read 10 Done Done Sent false
668 1134 Acked Read 9 Done Done Sent false
669 1135 Acked Read 9 Done Done Sent false
670 1136 Acked Write 4 Done Done Sent false
671 1137 Acked Flush 0 Done Done Sent false
672 1138 Acked Read 3 Done Done Sent false
673 1139 Acked Read 1 Done Done Sent false
674 1140 Acked Read 1 Done Done Sent false
675 1141 Acked Read 6 Done Done Sent false
676 1142 Acked Read 5 Done Done Sent false
677 1143 Acked Write 1 Done Done Sent false
678 1144 Acked Read 5 Done Done Sent false
679 1145 Acked Read 9 Done Done Sent false
680 1146 Acked Read 8 Done Done Sent false
681 1147 Acked Read 6 Done Done Sent false
682 1148 Acked Read 2 Done Done Sent false
683 1149 Acked Write 2 Done Done Sent false
684 1150 Acked Write 6 Done Done Sent false
685 1151 Acked Write 5 Done Done Sent false
686 1152 Acked Flush 0 Done Done Sent false
687 1153 Acked Read 2 Done Done Sent false
688 1154 Acked Read 1 Done Done Sent false
689 1155 Acked Write 2 Done Done Sent false
690 1156 Acked Read 4 Done Done Sent false
691 1157 Acked Write 9 Done Done Sent false
692 1158 Acked Read 10 Done Done Sent false
693 1159 Acked Write 6 Done Done Sent false
694 1160 Acked Write 10 Done Done Sent false
695 1161 Acked Write 2 Done Done Sent false
696 1162 Acked Write 8 Done Done Sent false
697 1163 Acked Read 9 Done Done Sent false
698 1164 Acked Read 3 Done Done Sent false
699 1165 Acked Read 8 Done Done Sent false
700 1166 Acked Write 5 Done Done Sent false
701 1167 Acked Flush 0 Done Done Sent false
702 1168 Acked Read 3 Done Done Sent false
703 1169 Acked Write 4 Done Done Sent false
704 1170 Acked Read 4 Done Done Sent false
705 1171 Acked Write 8 Done Done Sent false
706 1172 Acked Read 5 Done Done Sent false
707 1173 Acked Read 6 Done Done Sent false
708 1174 Acked Write 5 Done Done Sent false
709 1175 Acked Read 2 Done Done Sent false
710 1176 Acked Read 10 Done Done Sent false
711 1177 Acked Write 8 Done Done Sent false
712 1178 Acked Flush 0 Done Done Sent false
713 1179 Acked Write 8 Done Done Sent false
714 1180 Acked Read 10 Done Done Sent false
715 1181 Acked Read 5 Done Done Sent false
716 1182 Acked Write 10 Done Done Sent false
717 1183 Acked Read 4 Done Done Sent false
718 1184 Acked Write 10 Done Done Sent false
719 1185 Acked Read 8 Done Done Sent false
720 1186 Acked Read 9 Done Done Sent false
721 1187 Acked Read 8 Done Done Sent false
722 1188 Acked Write 10 Done Done Sent false
723 1189 Acked Read 9 Done Done Sent false
724 1190 Acked Read 9 Done Done Sent false
725 1191 Acked Read 2 Done Done Sent false
726 1192 Acked Write 9 Done Done Sent false
727 1193 Acked Read 5 Done Done Sent false
728 1194 Acked Write 6 Done Done Sent false
729 1195 Acked Write 3 Done Done Sent false
730 1196 Acked Read 5 Done Done Sent false
731 1197 Acked Read 7 Done Done Sent false
732 1198 Acked Flush 0 Done Done Sent false
733 1199 Acked Write 2 Done Done Sent false
734 1200 Acked Read 1 Done Done Sent false
735 1201 Acked Read 10 Done Done Sent false
736 1202 Acked Read 7 Done Done Sent false
737 1203 Acked Write 9 Done Done Sent false
738 1204 Acked Flush 0 Done Done Sent false
739 1205 Acked Write 3 Done Done Sent false
740 1206 Acked Flush 0 Done Done Sent false
741 1207 Acked Flush 0 Done Done Sent false
742 1208 Acked Read 3 Done Done Sent false
743 1209 Acked Write 9 Done Done Sent false
744 1210 Acked Read 2 Done Done Sent false
745 1211 Acked Read 9 Done Done Sent false
746 1212 Acked Write 9 Done Done Sent false
747 1213 Acked Write 3 Done Done Sent false
748 1214 Acked Read 7 Done Done Sent false
749 1215 Acked Write 5 Done Done Sent false
750 1216 Acked Flush 0 Done Done Sent false
751 1217 Acked Flush 0 Done Done Sent false
752 1218 Acked Read 9 Done Done Sent false
753 1219 Acked Write 7 Done Done Sent false
754 1220 Acked Read 7 Done Done Sent false
755 1221 Acked Read 9 Done Done Sent false
756 1222 Acked Flush 0 Done Done Sent false
757 1223 Acked Read 7 Done Done Sent false
758 1224 Acked Write 2 Done Done Sent false
759 1225 Acked Read 7 Done Done Sent false
760 1226 Acked Read 3 Done Done Sent false
761 1227 Acked Flush 0 Done Done Sent false
762 1228 Acked Read 1 Done Done Sent false
763 1229 Acked Write 7 Done Done Sent false
764 1230 Acked Write 7 Done Done Sent false
765 1231 Acked Read 7 Done Done Sent false
766 1232 Acked Read 6 Done Done Sent false
767 1233 Acked Write 10 Done Done Sent false
768 1234 Acked Write 2 Done Done Sent false
769 1235 Acked Write 9 Done Done Sent false
770 1236 Acked Read 4 Done Done Sent false
771 1237 Acked Write 5 Done Done Sent false
772 1238 Acked Write 5 Done Done Sent false
773 1239 Acked Read 7 Done Done Sent false
774 1240 Acked Write 9 Done Done Sent false
775 1241 Acked Write 10 Done Done Sent false
776 1242 Acked Read 9 Done Done Sent false
777 1243 Acked Read 5 Done Done Sent false
778 1244 Acked Write 4 Done Done Sent false
779 1245 Acked Read 5 Done Done Sent false
780 1246 Acked Write 8 Done Done Sent false
781 1247 Acked Write 6 Done Done Sent false
782 1248 Acked Write 2 Done Done Sent false
783 1249 Acked Flush 0 Done Done Sent false
784 1250 Acked Flush 0 Done Done Sent false
785 1251 Acked Write 8 Done Done Sent false
786 1252 Acked Read 7 Done Done Sent false
787 1253 Acked Write 5 Done Done Sent false
788 1254 Acked Read 8 Done Done Sent false
789 1255 Acked Read 1 Done Done Sent false
790 1256 Acked Read 4 Done Done Sent false
791 1257 Acked Read 8 Done Done Sent false
792 1258 Acked Read 10 Done Done Sent false
793 1259 Acked Write 3 Done Done Sent false
794 1260 Acked Read 7 Done Done Sent false
795 1261 Acked Flush 0 Done Done Sent false
796 1262 Acked Read 10 Done Done Sent false
797 1263 Acked Write 1 Done Done Sent false
798 1264 Acked Write 2 Done Done Sent false
799 1265 Acked Read 4 Done Done Sent false
800 1266 Acked Write 4 Done Done Sent false
801 1267 Acked Read 7 Done Done Sent false
802 1268 Acked Read 1 Done Done Sent false
803 1269 Acked Write 10 Done Done Sent false
804 1270 Acked Read 6 Done Done Sent false
805 1271 Acked Write 9 Done Done Sent false
806 1272 Acked Write 8 Done Done Sent false
807 1273 Acked Write 7 Done Done Sent false
808 1274 Acked Read 3 Done Done Sent false
809 1275 Acked Read 3 Done Done Sent false
810 1276 Acked Read 7 Done Done Sent false
811 1277 Acked Read 4 Done Done Sent false
812 1278 Acked Write 7 Done Done Sent false
813 1279 Acked Flush 0 Done Done Sent false
814 1280 Acked Read 1 Done Done Sent false
815 1281 Acked Write 9 Done Done Sent false
816 1282 Acked Read 2 Done Done Sent false
817 1283 Acked Read 9 Done Done Sent false
818 1284 Acked Flush 0 Done Done Sent false
819 1285 Acked Write 5 Done Done Sent false
820 1286 Acked Write 6 Done Done Sent false
821 1287 Acked Read 4 Done Done Sent false
822 1288 Acked Flush 0 Done Done Sent false
823 1289 Acked Flush 0 Done Done Sent false
824 1290 Acked Flush 0 Done Done Sent false
825 1291 Acked Write 2 Done Done Sent false
826 1292 Acked Read 1 Done Done Sent false
827 1293 Acked Read 6 Done Done Sent false
828 1294 Acked Read 9 Done Done Sent false
829 1295 Acked Read 4 Done Done Sent false
830 1296 Acked Read 9 Done Done Sent false
831 1297 Acked Read 8 Done Done Sent false
832 1298 Acked Write 2 Done Done Sent false
833 1299 Acked Read 10 Done Done Sent false
834 1300 Acked Read 1 Done Done Sent false
835 STATES DS:0 DS:1 DS:2 TOTAL
836 Sent 0 0 301 301
837 Done 301 301 0 602
838 Skipped 0 0 0 0
839 Error 0 0 0 0
840 Last Flush: Some(JobId(1290)) Some(JobId(1290)) None
841 Downstairs last five completed:
842 Upstairs last five completed: 1300 1299 1298 1297 1296
843 CLIENT: Up:0 ds:301 act:2
8442025-01-29T05:10:48.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
8452025-01-29T05:10:48.913ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
846 [8830] got start action
847 [8830] I am going to restart
848 Starting downstairs at port 8830
849 Make output file at "/tmp/dsc/downstairs-8830.txt"
850 [8830][2] reports Starting
851 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1203)
852 [8830][2] reports Running
8532025-01-29T05:10:49.914ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
8542025-01-29T05:10:50.916ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
855 ----------------------------------------------------------------
856 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
857 States: Active Active Offline
858 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
859 1000 Acked Write 9 Done Done Sent false
860 1001 Acked Read 4 Done Done Sent false
861 1002 Acked Write 10 Done Done Sent false
862 1003 Acked Write 8 Done Done Sent false
863 1004 Acked Write 3 Done Done Sent false
864 1005 Acked Write 7 Done Done Sent false
865 1006 Acked Read 5 Done Done Sent false
866 1007 Acked Write 6 Done Done Sent false
867 1008 Acked Read 1 Done Done Sent false
868 1009 Acked Read 7 Done Done Sent false
869 1010 Acked Read 8 Done Done Sent false
870 1011 Acked Flush 0 Done Done Sent false
871 1012 Acked Read 5 Done Done Sent false
872 1013 Acked Read 2 Done Done Sent false
873 1014 Acked Read 9 Done Done Sent false
874 1015 Acked Write 8 Done Done Sent false
875 1016 Acked Write 5 Done Done Sent false
876 1017 Acked Write 9 Done Done Sent false
877 1018 Acked Read 6 Done Done Sent false
878 1019 Acked Write 7 Done Done Sent false
879 1020 Acked Read 4 Done Done Sent false
880 1021 Acked Read 3 Done Done Sent false
881 1022 Acked Read 6 Done Done Sent false
882 1023 Acked Write 10 Done Done Sent false
883 1024 Acked Write 2 Done Done Sent false
884 1025 Acked Flush 0 Done Done Sent false
885 1026 Acked Read 6 Done Done Sent false
886 1027 Acked Read 8 Done Done Sent false
887 1028 Acked Flush 0 Done Done Sent false
888 1029 Acked Read 8 Done Done Sent false
889 1030 Acked Read 6 Done Done Sent false
890 1031 Acked Write 6 Done Done Sent false
891 1032 Acked Flush 0 Done Done Sent false
892 1033 Acked Read 9 Done Done Sent false
893 1034 Acked Write 5 Done Done Sent false
894 1035 Acked Read 8 Done Done Sent false
895 1036 Acked Read 7 Done Done Sent false
896 1037 Acked Write 6 Done Done Sent false
897 1038 Acked Read 10 Done Done Sent false
898 1039 Acked Read 9 Done Done Sent false
899 1040 Acked Write 8 Done Done Sent false
900 1041 Acked Write 9 Done Done Sent false
901 1042 Acked Write 5 Done Done Sent false
902 1043 Acked Write 7 Done Done Sent false
903 1044 Acked Write 1 Done Done Sent false
904 1045 Acked Read 2 Done Done Sent false
905 1046 Acked Write 8 Done Done Sent false
906 1047 Acked Read 9 Done Done Sent false
907 1048 Acked Flush 0 Done Done Sent false
908 1049 Acked Write 2 Done Done Sent false
909 1050 Acked Write 3 Done Done Sent false
910 1051 Acked Write 8 Done Done Sent false
911 1052 Acked Write 4 Done Done Sent false
912 1053 Acked Read 9 Done Done Sent false
913 1054 Acked Read 4 Done Done Sent false
914 1055 Acked Read 5 Done Done Sent false
915 1056 Acked Read 2 Done Done Sent false
916 1057 Acked Read 10 Done Done Sent false
917 1058 Acked Read 9 Done Done Sent false
918 1059 Acked Write 10 Done Done Sent false
919 1060 Acked Write 10 Done Done Sent false
920 1061 Acked Read 10 Done Done Sent false
921 1062 Acked Write 10 Done Done Sent false
922 1063 Acked Read 10 Done Done Sent false
923 1064 Acked Flush 0 Done Done Sent false
924 1065 Acked Read 2 Done Done Sent false
925 1066 Acked Read 1 Done Done Sent false
926 1067 Acked Read 8 Done Done Sent false
927 1068 Acked Read 1 Done Done Sent false
928 1069 Acked Write 10 Done Done Sent false
929 1070 Acked Write 2 Done Done Sent false
930 1071 Acked Write 1 Done Done Sent false
931 1072 Acked Read 1 Done Done Sent false
932 1073 Acked Read 5 Done Done Sent false
933 1074 Acked Read 6 Done Done Sent false
934 1075 Acked Read 10 Done Done Sent false
935 1076 Acked Write 7 Done Done Sent false
936 1077 Acked Read 4 Done Done Sent false
937 1078 Acked Read 5 Done Done Sent false
938 1079 Acked Read 2 Done Done Sent false
939 1080 Acked Flush 0 Done Done Sent false
940 1081 Acked Read 9 Done Done Sent false
941 1082 Acked Read 1 Done Done Sent false
942 1083 Acked Read 8 Done Done Sent false
943 1084 Acked Read 4 Done Done Sent false
944 1085 Acked Read 9 Done Done Sent false
945 1086 Acked Read 5 Done Done Sent false
946 1087 Acked Write 1 Done Done Sent false
947 1088 Acked Write 10 Done Done Sent false
948 1089 Acked Write 2 Done Done Sent false
949 1090 Acked Write 10 Done Done Sent false
950 1091 Acked Read 9 Done Done Sent false
951 1092 Acked Write 5 Done Done Sent false
952 1093 Acked Read 4 Done Done Sent false
953 1094 Acked Write 7 Done Done Sent false
954 1095 Acked Write 1 Done Done Sent false
955 1096 Acked Write 10 Done Done Sent false
956 1097 Acked Read 9 Done Done Sent false
957 1098 Acked Write 10 Done Done Sent false
958 1099 Acked Write 7 Done Done Sent false
959 1100 Acked Write 1 Done Done Sent false
960 1101 Acked Read 3 Done Done Sent false
961 1102 Acked Write 9 Done Done Sent false
962 1103 Acked Flush 0 Done Done Sent false
963 1104 Acked Write 4 Done Done Sent false
964 1105 Acked Write 7 Done Done Sent false
965 1106 Acked Flush 0 Done Done Sent false
966 1107 Acked Write 9 Done Done Sent false
967 1108 Acked Flush 0 Done Done Sent false
968 1109 Acked Read 5 Done Done Sent false
969 1110 Acked Write 4 Done Done Sent false
970 1111 Acked Write 6 Done Done Sent false
971 1112 Acked Write 4 Done Done Sent false
972 1113 Acked Write 4 Done Done Sent false
973 1114 Acked Read 7 Done Done Sent false
974 1115 Acked Read 5 Done Done Sent false
975 1116 Acked Write 5 Done Done Sent false
976 1117 Acked Write 7 Done Done Sent false
977 1118 Acked Write 2 Done Done Sent false
978 1119 Acked Write 8 Done Done Sent false
979 1120 Acked Flush 0 Done Done Sent false
980 1121 Acked Read 5 Done Done Sent false
981 1122 Acked Write 10 Done Done Sent false
982 1123 Acked Read 4 Done Done Sent false
983 1124 Acked Read 4 Done Done Sent false
984 1125 Acked Read 3 Done Done Sent false
985 1126 Acked Write 4 Done Done Sent false
986 1127 Acked Write 4 Done Done Sent false
987 1128 Acked Write 7 Done Done Sent false
988 1129 Acked Write 1 Done Done Sent false
989 1130 Acked Read 3 Done Done Sent false
990 1131 Acked Read 7 Done Done Sent false
991 1132 Acked Flush 0 Done Done Sent false
992 1133 Acked Read 10 Done Done Sent false
993 1134 Acked Read 9 Done Done Sent false
994 1135 Acked Read 9 Done Done Sent false
995 1136 Acked Write 4 Done Done Sent false
996 1137 Acked Flush 0 Done Done Sent false
997 1138 Acked Read 3 Done Done Sent false
998 1139 Acked Read 1 Done Done Sent false
999 1140 Acked Read 1 Done Done Sent false
1000 1141 Acked Read 6 Done Done Sent false
1001 1142 Acked Read 5 Done Done Sent false
1002 1143 Acked Write 1 Done Done Sent false
1003 1144 Acked Read 5 Done Done Sent false
1004 1145 Acked Read 9 Done Done Sent false
1005 1146 Acked Read 8 Done Done Sent false
1006 1147 Acked Read 6 Done Done Sent false
1007 1148 Acked Read 2 Done Done Sent false
1008 1149 Acked Write 2 Done Done Sent false
1009 1150 Acked Write 6 Done Done Sent false
1010 1151 Acked Write 5 Done Done Sent false
1011 1152 Acked Flush 0 Done Done Sent false
1012 1153 Acked Read 2 Done Done Sent false
1013 1154 Acked Read 1 Done Done Sent false
1014 1155 Acked Write 2 Done Done Sent false
1015 1156 Acked Read 4 Done Done Sent false
1016 1157 Acked Write 9 Done Done Sent false
1017 1158 Acked Read 10 Done Done Sent false
1018 1159 Acked Write 6 Done Done Sent false
1019 1160 Acked Write 10 Done Done Sent false
1020 1161 Acked Write 2 Done Done Sent false
1021 1162 Acked Write 8 Done Done Sent false
1022 1163 Acked Read 9 Done Done Sent false
1023 1164 Acked Read 3 Done Done Sent false
1024 1165 Acked Read 8 Done Done Sent false
1025 1166 Acked Write 5 Done Done Sent false
1026 1167 Acked Flush 0 Done Done Sent false
1027 1168 Acked Read 3 Done Done Sent false
1028 1169 Acked Write 4 Done Done Sent false
1029 1170 Acked Read 4 Done Done Sent false
1030 1171 Acked Write 8 Done Done Sent false
1031 1172 Acked Read 5 Done Done Sent false
1032 1173 Acked Read 6 Done Done Sent false
1033 1174 Acked Write 5 Done Done Sent false
1034 1175 Acked Read 2 Done Done Sent false
1035 1176 Acked Read 10 Done Done Sent false
1036 1177 Acked Write 8 Done Done Sent false
1037 1178 Acked Flush 0 Done Done Sent false
1038 1179 Acked Write 8 Done Done Sent false
1039 1180 Acked Read 10 Done Done Sent false
1040 1181 Acked Read 5 Done Done Sent false
1041 1182 Acked Write 10 Done Done Sent false
1042 1183 Acked Read 4 Done Done Sent false
1043 1184 Acked Write 10 Done Done Sent false
1044 1185 Acked Read 8 Done Done Sent false
1045 1186 Acked Read 9 Done Done Sent false
1046 1187 Acked Read 8 Done Done Sent false
1047 1188 Acked Write 10 Done Done Sent false
1048 1189 Acked Read 9 Done Done Sent false
1049 1190 Acked Read 9 Done Done Sent false
1050 1191 Acked Read 2 Done Done Sent false
1051 1192 Acked Write 9 Done Done Sent false
1052 1193 Acked Read 5 Done Done Sent false
1053 1194 Acked Write 6 Done Done Sent false
1054 1195 Acked Write 3 Done Done Sent false
1055 1196 Acked Read 5 Done Done Sent false
1056 1197 Acked Read 7 Done Done Sent false
1057 1198 Acked Flush 0 Done Done Sent false
1058 1199 Acked Write 2 Done Done Sent false
1059 1200 Acked Read 1 Done Done Sent false
1060 1201 Acked Read 10 Done Done Sent false
1061 1202 Acked Read 7 Done Done Sent false
1062 1203 Acked Write 9 Done Done Sent false
1063 1204 Acked Flush 0 Done Done Sent false
1064 1205 Acked Write 3 Done Done Sent false
1065 1206 Acked Flush 0 Done Done Sent false
1066 1207 Acked Flush 0 Done Done Sent false
1067 1208 Acked Read 3 Done Done Sent false
1068 1209 Acked Write 9 Done Done Sent false
1069 1210 Acked Read 2 Done Done Sent false
1070 1211 Acked Read 9 Done Done Sent false
1071 1212 Acked Write 9 Done Done Sent false
1072 1213 Acked Write 3 Done Done Sent false
1073 1214 Acked Read 7 Done Done Sent false
1074 1215 Acked Write 5 Done Done Sent false
1075 1216 Acked Flush 0 Done Done Sent false
1076 1217 Acked Flush 0 Done Done Sent false
1077 1218 Acked Read 9 Done Done Sent false
1078 1219 Acked Write 7 Done Done Sent false
1079 1220 Acked Read 7 Done Done Sent false
1080 1221 Acked Read 9 Done Done Sent false
1081 1222 Acked Flush 0 Done Done Sent false
1082 1223 Acked Read 7 Done Done Sent false
1083 1224 Acked Write 2 Done Done Sent false
1084 1225 Acked Read 7 Done Done Sent false
1085 1226 Acked Read 3 Done Done Sent false
1086 1227 Acked Flush 0 Done Done Sent false
1087 1228 Acked Read 1 Done Done Sent false
1088 1229 Acked Write 7 Done Done Sent false
1089 1230 Acked Write 7 Done Done Sent false
1090 1231 Acked Read 7 Done Done Sent false
1091 1232 Acked Read 6 Done Done Sent false
1092 1233 Acked Write 10 Done Done Sent false
1093 1234 Acked Write 2 Done Done Sent false
1094 1235 Acked Write 9 Done Done Sent false
1095 1236 Acked Read 4 Done Done Sent false
1096 1237 Acked Write 5 Done Done Sent false
1097 1238 Acked Write 5 Done Done Sent false
1098 1239 Acked Read 7 Done Done Sent false
1099 1240 Acked Write 9 Done Done Sent false
1100 1241 Acked Write 10 Done Done Sent false
1101 1242 Acked Read 9 Done Done Sent false
1102 1243 Acked Read 5 Done Done Sent false
1103 1244 Acked Write 4 Done Done Sent false
1104 1245 Acked Read 5 Done Done Sent false
1105 1246 Acked Write 8 Done Done Sent false
1106 1247 Acked Write 6 Done Done Sent false
1107 1248 Acked Write 2 Done Done Sent false
1108 1249 Acked Flush 0 Done Done Sent false
1109 1250 Acked Flush 0 Done Done Sent false
1110 1251 Acked Write 8 Done Done Sent false
1111 1252 Acked Read 7 Done Done Sent false
1112 1253 Acked Write 5 Done Done Sent false
1113 1254 Acked Read 8 Done Done Sent false
1114 1255 Acked Read 1 Done Done Sent false
1115 1256 Acked Read 4 Done Done Sent false
1116 1257 Acked Read 8 Done Done Sent false
1117 1258 Acked Read 10 Done Done Sent false
1118 1259 Acked Write 3 Done Done Sent false
1119 1260 Acked Read 7 Done Done Sent false
1120 1261 Acked Flush 0 Done Done Sent false
1121 1262 Acked Read 10 Done Done Sent false
1122 1263 Acked Write 1 Done Done Sent false
1123 1264 Acked Write 2 Done Done Sent false
1124 1265 Acked Read 4 Done Done Sent false
1125 1266 Acked Write 4 Done Done Sent false
1126 1267 Acked Read 7 Done Done Sent false
1127 1268 Acked Read 1 Done Done Sent false
1128 1269 Acked Write 10 Done Done Sent false
1129 1270 Acked Read 6 Done Done Sent false
1130 1271 Acked Write 9 Done Done Sent false
1131 1272 Acked Write 8 Done Done Sent false
1132 1273 Acked Write 7 Done Done Sent false
1133 1274 Acked Read 3 Done Done Sent false
1134 1275 Acked Read 3 Done Done Sent false
1135 1276 Acked Read 7 Done Done Sent false
1136 1277 Acked Read 4 Done Done Sent false
1137 1278 Acked Write 7 Done Done Sent false
1138 1279 Acked Flush 0 Done Done Sent false
1139 1280 Acked Read 1 Done Done Sent false
1140 1281 Acked Write 9 Done Done Sent false
1141 1282 Acked Read 2 Done Done Sent false
1142 1283 Acked Read 9 Done Done Sent false
1143 1284 Acked Flush 0 Done Done Sent false
1144 1285 Acked Write 5 Done Done Sent false
1145 1286 Acked Write 6 Done Done Sent false
1146 1287 Acked Read 4 Done Done Sent false
1147 1288 Acked Flush 0 Done Done Sent false
1148 1289 Acked Flush 0 Done Done Sent false
1149 1290 Acked Flush 0 Done Done Sent false
1150 1291 Acked Write 2 Done Done Sent false
1151 1292 Acked Read 1 Done Done Sent false
1152 1293 Acked Read 6 Done Done Sent false
1153 1294 Acked Read 9 Done Done Sent false
1154 1295 Acked Read 4 Done Done Sent false
1155 1296 Acked Read 9 Done Done Sent false
1156 1297 Acked Read 8 Done Done Sent false
1157 1298 Acked Write 2 Done Done Sent false
1158 1299 Acked Read 10 Done Done Sent false
1159 1300 Acked Read 1 Done Done Sent false
1160 STATES DS:0 DS:1 DS:2 TOTAL
1161 Sent 0 0 301 301
1162 Done 301 301 0 602
1163 Skipped 0 0 0 0
1164 Error 0 0 0 0
1165 Last Flush: Some(JobId(1290)) Some(JobId(1290)) None
1166 Downstairs last five completed:
1167 Upstairs last five completed: 1300 1299 1298 1297 1296
1168 CLIENT: Up:0 ds:301 act:2
11692025-01-29T05:10:52.196ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11702025-01-29T05:10:52.920ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11712025-01-29T05:10:53.441ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11722025-01-29T05:10:53.441ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43701) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11732025-01-29T05:10:53.442ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11742025-01-29T05:10:53.443ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11752025-01-29T05:10:53.443ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11762025-01-29T05:10:53.443ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11772025-01-29T05:10:53.443ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11782025-01-29T05:10:53.443ZINFOcrucible: [2] client re-new 301 jobs since flush None = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11792025-01-29T05:10:53.443ZINFOcrucible: [2] Replayed 301 jobs since flush: None = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1180 ----------------------------------------------------------------
1181 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1182 Downstairs last five completed: 1301 1300 1299 1298 1297
1183 Upstairs last five completed: 1301 1300 1299 1298 1297
1184 CLIENT: Up:0 ds:0 act:3
1185 Replay: All jobs finished
1186 Main task has work to do, go find it
1187 got dsc Stop(1)
1188 stop 1
1189 Jan 29 05:10:55.697 INFO request completed[8820] Got stop action so:false kr:false
1190 , latency_us: 359, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 71fef729-4a11-4aa7-b0f0-9036411e4c75, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
11912025-01-29T05:10:55.697ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11922025-01-29T05:10:55.697ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11932025-01-29T05:10:55.697ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11942025-01-29T05:10:55.697ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1195 Jan 29 05:10:55.697 INFO request completed, latency_us: 165, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1f9df034-7a0e-49c0-90d1-3238bb2b89dc, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
1196 [8820] Exited with: ExitStatus(unix_wait_status(9))
1197 [8820][1] reports Exit
1198 Jan 29 05:10:59.700 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3f2de732-1e37-47d6-954c-1f6808dfcfaf, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
1199 001/300 Read block 2902 len 28672
1200 002/300 Read block 1696 len 28672
1201 003/300 Read block 2196 len 24576
1202 004/300 Read block 1642 len 28672
1203 005/300 Flush
1204 006/300 Write block 2779 len 24576 data: 3 3 3 3 3 3
1205 007/300 Flush
1206 008/300 Write block 491 len 16384 data: 2 2 2 2
1207 009/300 Write block 141 len 8192 data: 2 2
1208 010/300 Write block 2219 len 16384 data: 2 2 2 2
1209 011/300 Read block 4610 len 4096
1210 012/300 Read block 366 len 28672
1211 013/300 Write block 3267 len 4096 data: 2
1212 014/300 Write block 2673 len 16384 data: 2 2 2 2
1213 015/300 Write block 2829 len 32768 data: 2 2 2 2 2 2 2 2
1214 016/300 Flush
1215 017/300 Write block 1609 len 12288 data: 2 2 2
1216 018/300 Write block 4916 len 36864 data: 2 2 2 2 2 2 2 2 2
1217 019/300 Write block 1673 len 40960 data: 3 3 3 3 2 2 2 2 2 2
1218 020/300 Read block 4389 len 12288
1219 021/300 Flush
1220 022/300 Flush
1221 023/300 Write block 4776 len 32768 data: 3 3 3 3 3 2 2 2
1222 024/300 Write block 3401 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1223 025/300 Read block 3738 len 36864
1224 026/300 Write block 4802 len 24576 data: 2 2 2 2 2 2
1225 027/300 Read block 4169 len 24576
1226 028/300 Write block 4053 len 4096 data: 2
1227 029/300 Write block 1891 len 36864 data: 2 2 2 2 2 2 2 2 2
1228 030/300 Read block 2189 len 24576
1229 031/300 Read block 4550 len 20480
1230 032/300 Read block 467 len 12288
1231 033/300 Read block 3472 len 8192
1232 034/300 Write block 3038 len 16384 data: 2 2 2 2
1233 035/300 Read block 1091 len 8192
1234 036/300 Read block 3392 len 40960
1235 037/300 Write block 1397 len 32768 data: 2 2 2 2 2 2 2 2
1236 038/300 Flush
1237 039/300 Write block 1510 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1238 040/300 Read block 3046 len 40960
1239 041/300 Read block 4766 len 20480
1240 042/300 Read block 1194 len 32768
1241 043/300 Write block 2486 len 4096 data: 2
1242 044/300 Read block 2329 len 16384
1243 045/300 Read block 1045 len 20480
1244 046/300 Flush
1245 047/300 Read block 3489 len 8192
1246 048/300 Flush
1247 049/300 Flush
1248 050/300 Write block 1516 len 24576 data: 3 3 3 3 2 2
1249 051/300 Write block 4844 len 12288 data: 3 3 2
1250 052/300 Read block 1662 len 16384
1251 053/300 Read block 498 len 32768
1252 054/300 Write block 4227 len 12288 data: 2 2 2
1253 055/300 Write block 812 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1254 056/300 Read block 3925 len 36864
1255 057/300 Read block 930 len 24576
1256 058/300 Read block 1059 len 20480
1257 059/300 Read block 1200 len 16384
1258 060/300 Read block 3647 len 28672
1259 061/300 Read block 4266 len 8192
1260 062/300 Read block 3227 len 16384
1261 063/300 Flush
1262 064/300 Read block 1072 len 4096
1263 065/300 Flush
1264 066/300 Read block 4895 len 32768
1265 067/300 Read block 3461 len 36864
1266 068/300 Read block 3248 len 12288
1267 069/300 Read block 4893 len 36864
1268 070/300 Read block 3323 len 4096
1269 071/300 Read block 253 len 40960
1270 072/300 Write block 1467 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1271 073/300 Write block 1838 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1272 074/300 Write block 4697 len 36864 data: 2 2 2 2 2 2 2 2 2
1273 075/300 Read block 1136 len 32768
1274 076/300 Flush
1275 077/300 Flush
1276 078/300 Read block 1872 len 4096
1277 079/300 Write block 1557 len 16384 data: 2 2 2 2
1278 080/300 Write block 3754 len 20480 data: 2 2 2 3 3
1279 081/300 Write block 2946 len 12288 data: 3 3 3
1280 082/300 Write block 638 len 28672 data: 2 2 2 2 2 2 2
1281 083/300 Read block 892 len 8192
1282 084/300 Read block 3478 len 16384
1283 085/300 Write block 3499 len 32768 data: 2 2 2 2 2 2 2 2
1284 086/300 Read block 1996 len 4096
1285 087/300 Write block 3352 len 20480 data: 2 2 2 2 2
1286 088/300 Read block 621 len 36864
1287 089/300 Read block 3379 len 8192
1288 090/300 Read block 4415 len 12288
1289 091/300 Write block 4600 len 20480 data: 2 2 2 2 2
1290 092/300 Read block 3296 len 24576
1291 093/300 Write block 71 len 36864 data: 2 2 2 2 2 2 2 2 2
1292 094/300 Read block 3071 len 16384
1293 095/300 Flush
1294 096/300 Read block 1516 len 12288
1295 097/300 Write block 1999 len 32768 data: 2 2 2 2 2 2 2 2
1296 098/300 Flush
1297 099/300 Flush
1298 100/300 Write block 1622 len 24576 data: 2 2 2 2 2 2
1299 101/300 Read block 2630 len 36864
1300 102/300 Write block 378 len 20480 data: 2 2 2 2 2
1301 103/300 Write block 3080 len 4096 data: 3
1302 104/300 Read block 4237 len 40960
1303 105/300 Read block 2231 len 28672
1304 106/300 Read block 3581 len 32768
1305 107/300 Read block 1215 len 28672
1306 108/300 Write block 2601 len 36864 data: 2 2 2 2 2 2 2 2 3
1307 109/300 Read block 111 len 20480
1308 110/300 Flush
1309 111/300 Write block 1216 len 20480 data: 3 3 2 2 2
1310 112/300 Read block 2653 len 20480
1311 113/300 Write block 4252 len 28672 data: 2 2 2 2 2 2 2
1312 114/300 Read block 1224 len 32768
1313 115/300 Write block 501 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1314 116/300 Read block 2840 len 20480
1315 117/300 Write block 965 len 40960 data: 2 2 3 3 3 3 3 3 3 2
1316 118/300 Write block 4023 len 32768 data: 2 2 2 2 2 2 2 2
1317 119/300 Read block 3352 len 28672
1318 120/300 Read block 878 len 32768
1319 121/300 Read block 3154 len 24576
1320 122/300 Read block 2734 len 32768
1321 123/300 Read block 1818 len 40960
1322 124/300 Read block 2010 len 36864
1323 125/300 Read block 1318 len 36864
1324 126/300 Read block 4629 len 28672
1325 127/300 Read block 1837 len 24576
1326 128/300 Read block 3694 len 40960
1327 129/300 Read block 2109 len 36864
1328 130/300 Read block 182 len 12288
1329 131/300 Read block 2212 len 4096
1330 132/300 Read block 1517 len 16384
1331 133/300 Write block 4362 len 24576 data: 2 2 2 2 2 2
1332 134/300 Read block 1914 len 4096
1333 135/300 Read block 2750 len 8192
1334 136/300 Read block 3081 len 24576
1335 137/300 Read block 534 len 28672
1336 138/300 Write block 2014 len 12288 data: 3 3 2
1337 139/300 Read block 1876 len 12288
1338 140/300 Flush
1339 141/300 Flush
1340 142/300 Write block 236 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1341 143/300 Read block 1953 len 24576
1342 144/300 Read block 4955 len 12288
1343 145/300 Read block 1437 len 16384
1344 146/300 Write block 2435 len 16384 data: 2 2 2 2
1345 147/300 Read block 2885 len 24576
1346 148/300 Read block 1299 len 32768
1347 149/300 Read block 3232 len 28672
1348 150/300 Write block 3544 len 32768 data: 2 2 2 2 2 2 2 2
1349 151/300 Read block 4229 len 20480
1350 152/300 Write block 1906 len 8192 data: 2 2
1351 153/300 Write block 3749 len 36864 data: 2 2 2 2 2 3 3 3 4
1352 154/300 Read block 1050 len 24576
1353 155/300 Write block 4473 len 36864 data: 2 2 2 2 2 2 2 2 2
1354 156/300 Read block 2845 len 36864
1355 157/300 Read block 2905 len 8192
1356 158/300 Read block 1870 len 20480
1357 159/300 Flush
1358 160/300 Write block 4830 len 32768 data: 2 2 2 2 2 2 2 2
1359 161/300 Read block 1477 len 36864
1360 162/300 Flush
1361 163/300 Read block 4547 len 36864
1362 164/300 Write block 115 len 4096 data: 2
1363 165/300 Write block 3970 len 8192 data: 2 2
1364 166/300 Write block 3547 len 24576 data: 3 3 3 3 3 2
1365 167/300 Read block 4282 len 8192
1366 168/300 Flush
1367 169/300 Read block 2120 len 16384
1368 170/300 Read block 1964 len 12288
1369 171/300 Read block 1486 len 36864
1370 172/300 Read block 4203 len 4096
1371 173/300 Read block 3060 len 40960
1372 174/300 Read block 2763 len 28672
1373 175/300 Write block 2643 len 20480 data: 3 3 2 2 2
1374 176/300 Write block 2953 len 28672 data: 2 2 2 2 2 2 2
1375 177/300 Read block 3363 len 36864
1376 178/300 Read block 2686 len 20480
1377 179/300 Read block 2808 len 24576
1378 180/300 Flush
1379 181/300 Flush
1380 182/300 Write block 397 len 20480 data: 2 2 2 2 2
1381 183/300 Flush
1382 184/300 Read block 3239 len 4096
1383 185/300 Write block 4138 len 16384 data: 2 2 2 2
1384 186/300 Write block 4503 len 24576 data: 3 3 3 2 2 2
1385 187/300 Read block 4350 len 16384
1386 188/300 Flush
1387 189/300 Read block 4834 len 12288
1388 190/300 Flush
1389 191/300 Read block 1313 len 12288
1390 192/300 Flush
1391 193/300 Read block 3793 len 8192
1392 194/300 Read block 507 len 4096
1393 195/300 Read block 3923 len 24576
1394 196/300 Read block 4954 len 4096
1395 197/300 Read block 3000 len 4096
1396 198/300 Write block 2378 len 36864 data: 2 2 2 2 2 2 2 2 2
1397 199/300 Flush
1398 200/300 Read block 644 len 36864
1399 201/300 Write block 467 len 8192 data: 2 2
1400 202/300 Write block 171 len 4096 data: 2
1401 203/300 Write block 4674 len 40960 data: 2 3 3 3 4 4 4 4 4 4
1402 204/300 Write block 2976 len 16384 data: 2 2 2 2
1403 205/300 Read block 1241 len 24576
1404 206/300 Read block 2082 len 4096
1405 207/300 Write block 2331 len 32768 data: 2 2 2 2 2 2 3 3
1406 208/300 Read block 3803 len 36864
1407 209/300 Read block 1090 len 16384
1408 210/300 Write block 1432 len 20480 data: 2 2 2 2 2
1409 211/300 Write block 1859 len 4096 data: 2
1410 212/300 Read block 1308 len 32768
1411 213/300 Read block 1050 len 20480
1412 214/300 Flush
1413 215/300 Write block 4219 len 8192 data: 2 2
1414 216/300 Read block 910 len 28672
1415 217/300 Read block 363 len 36864
1416 218/300 Read block 1021 len 4096
1417 219/300 Flush
1418 220/300 Write block 3677 len 24576 data: 2 2 2 2 2 2
1419 221/300 Write block 286 len 36864 data: 2 2 2 2 2 2 2 2 2
1420 222/300 Write block 4294 len 8192 data: 3 3
1421 223/300 Write block 3788 len 24576 data: 2 2 2 2 2 2
1422 224/300 Read block 2008 len 32768
1423 225/300 Read block 699 len 40960
1424 226/300 Read block 237 len 28672
1425 227/300 Write block 980 len 16384 data: 2 2 2 2
1426 228/300 Write block 14 len 16384 data: 2 2 2 2
1427 229/300 Read block 3466 len 4096
1428 230/300 Read block 1576 len 16384
1429 231/300 Write block 2557 len 28672 data: 3 3 3 2 2 2 2
1430 232/300 Read block 2338 len 20480
1431 233/300 Read block 1020 len 16384
1432 234/300 Write block 1502 len 24576 data: 2 2 2 2 2 2
1433 235/300 Read block 3495 len 20480
1434 236/300 Read block 2798 len 20480
1435 237/300 Write block 1448 len 8192 data: 3 3
1436 238/300 Flush
1437 239/300 Read block 3035 len 36864
1438 240/300 Read block 422 len 40960
1439 241/300 Write block 3166 len 32768 data: 2 2 2 2 2 2 2 2
1440 242/300 Read block 1056 len 28672
1441 243/300 Read block 3318 len 12288
1442 244/300 Read block 57 len 4096
1443 245/300 Read block 2676 len 40960
1444 246/300 Write block 2135 len 28672 data: 2 2 2 2 2 2 2
1445 247/300 Write block 4210 len 8192 data: 2 2
1446 248/300 Write block 515 len 32768 data: 2 2 2 2 2 2 2 2
1447 249/300 Write block 279 len 36864 data: 2 2 2 2 2 2 2 3 3
1448 250/300 Write block 587 len 16384 data: 2 2 3 3
1449 251/300 Read block 3502 len 8192
1450 252/300 Write block 4220 len 24576 data: 3 2 2 2 2 2
1451 253/300 Read block 4192 len 4096
1452 254/300 Read block 623 len 32768
1453 255/300 Write block 1998 len 20480 data: 2 3 3 3 3
1454 256/300 Write block 4119 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1455 257/300 Read block 2187 len 32768
1456 258/300 Read block 3724 len 24576
1457 259/300 Write block 169 len 40960 data: 2 2 3 2 2 2 2 2 2 2
1458 260/300 Read block 863 len 32768
1459 261/300 Write block 1687 len 32768 data: 2 2 2 4 4 3 3 3
1460 262/300 Write block 1769 len 8192 data: 2 2
1461 263/300 Write block 170 len 16384 data: 3 4 3 3
1462 264/300 Write block 1586 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1463 265/300 Write block 251 len 4096 data: 2
1464 266/300 Flush
1465 267/300 Write block 2377 len 24576 data: 2 3 3 3 3 3
1466 268/300 Read block 1474 len 32768
1467 269/300 Read block 2161 len 24576
1468 270/300 Write block 749 len 28672 data: 2 2 2 2 2 2 2
1469 271/300 Read block 3608 len 24576
1470 272/300 Flush
1471 273/300 Write block 135 len 12288 data: 2 2 2
1472 274/300 Read block 3873 len 4096
1473 275/300 Read block 2440 len 4096
1474 276/300 Flush
1475 277/300 Write block 4234 len 28672 data: 2 2 2 2 2 2 2
1476 278/300 Write block 1787 len 16384 data: 2 2 2 2
1477 279/300 Read block 3403 len 12288
1478 280/300 Write block 3805 len 16384 data: 2 2 2 2
1479 281/300 Flush
1480 282/300 Flush
1481 283/300 Read block 2877 len 40960
1482 284/300 Write block 4195 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1483 285/300 Write block 393 len 16384 data: 2 2 2 2
1484 286/300 Read block 4615 len 16384
1485 287/300 Write block 3909 len 4096 data: 2
1486 288/300 Write block 4046 len 8192 data: 3 3
1487 289/300 Write block 4997 len 12288 data: 2 2 2
1488 290/300 Write block 1495 len 28672 data: 2 2 2 2 2 2 2
1489 291/300 Read block 1943 len 28672
1490 292/300 Read block 3020 len 8192
1491 293/300 Flush
1492 294/300 Write block 188 len 28672 data: 2 2 2 2 3 3 3
1493 295/300 Write block 3767 len 8192 data: 2 2
1494 296/300 Write block 946 len 36864 data: 2 2 2 2 2 2 3 3 3
1495 297/300 Flush
1496 298/300 Write block 4770 len 4096 data: 2
1497 299/300 Flush
1498 300/300 Write block 2498 len 4096 data: 2
1499 301/300 Read block 4406 len 32768
1500 Main task has work to do, go find it
1501 got dsc Start(1)
1502 start 1
1503 Jan 29 05:10:59.941 INFO request completed, latency_us: 177, response_code: 204, uri: /start/cid/1, method: POST, req_id: 488f58e3-d753-4de4-ba38-832223474383, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
1504 Replay: started 1, returned:Ok(())
1505 ----------------------------------------------------------------
1506 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1507 States: Active Offline Active
1508 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1509 1302 Acked Read 7 Done Sent Done false
1510 1303 Acked Read 7 Done Sent Done false
1511 1304 Acked Read 6 Done Sent Done false
1512 1305 Acked Read 7 Done Sent Done false
1513 1306 Acked Flush 0 Done Sent Done false
1514 1307 Acked Write 6 Done Sent Done false
1515 1308 Acked Flush 0 Done Sent Done false
1516 1309 Acked Write 4 Done Sent Done false
1517 1310 Acked Write 2 Done Sent Done false
1518 1311 Acked Write 4 Done Sent Done false
1519 1312 Acked Read 1 Done Sent Done false
1520 1313 Acked Read 7 Done Sent Done false
1521 1314 Acked Write 1 Done Sent Done false
1522 1315 Acked Write 4 Done Sent Done false
1523 1316 Acked Write 8 Done Sent Done false
1524 1317 Acked Flush 0 Done Sent Done false
1525 1318 Acked Write 3 Done Sent Done false
1526 1319 Acked Write 9 Done Sent Done false
1527 1320 Acked Write 10 Done Sent Done false
1528 1321 Acked Read 3 Done Sent Done false
1529 1322 Acked Flush 0 Done Sent Done false
1530 1323 Acked Flush 0 Done Sent Done false
1531 1324 Acked Write 8 Done Sent Done false
1532 1325 Acked Write 10 Done Sent Done false
1533 1326 Acked Read 9 Done Sent Done false
1534 1327 Acked Write 6 Done Sent Done false
1535 1328 Acked Read 6 Done Sent Done false
1536 1329 Acked Write 1 Done Sent Done false
1537 1330 Acked Write 9 Done Sent Done false
1538 1331 Acked Read 6 Done Sent Done false
1539 1332 Acked Read 5 Done Sent Done false
1540 1333 Acked Read 3 Done Sent Done false
1541 1334 Acked Read 2 Done Sent Done false
1542 1335 Acked Write 4 Done Sent Done false
1543 1336 Acked Read 2 Done Sent Done false
1544 1337 Acked Read 10 Done Sent Done false
1545 1338 Acked Write 8 Done Sent Done false
1546 1339 Acked Flush 0 Done Sent Done false
1547 1340 Acked Write 10 Done Sent Done false
1548 1341 Acked Read 10 Done Sent Done false
1549 1342 Acked Read 5 Done Sent Done false
1550 1343 Acked Read 8 Done Sent Done false
1551 1344 Acked Write 1 Done Sent Done false
1552 1345 Acked Read 4 Done Sent Done false
1553 1346 Acked Read 5 Done Sent Done false
1554 1347 Acked Flush 0 Done Sent Done false
1555 1348 Acked Read 2 Done Sent Done false
1556 1349 Acked Flush 0 Done Sent Done false
1557 1350 Acked Flush 0 Done Sent Done false
1558 1351 Acked Write 6 Done Sent Done false
1559 1352 Acked Write 3 Done Sent Done false
1560 1353 Acked Read 4 Done Sent Done false
1561 1354 Acked Read 8 Done Sent Done false
1562 1355 Acked Write 3 Done Sent Done false
1563 1356 Acked Write 10 Done Sent Done false
1564 1357 Acked Read 9 Done Sent Done false
1565 1358 Acked Read 6 Done Sent Done false
1566 1359 Acked Read 5 Done Sent Done false
1567 1360 Acked Read 4 Done Sent Done false
1568 1361 Acked Read 7 Done Sent Done false
1569 1362 Acked Read 2 Done Sent Done false
1570 1363 Acked Read 4 Done Sent Done false
1571 1364 Acked Flush 0 Done Sent Done false
1572 1365 Acked Read 1 Done Sent Done false
1573 1366 Acked Flush 0 Done Sent Done false
1574 1367 Acked Read 8 Done Sent Done false
1575 1368 Acked Read 9 Done Sent Done false
1576 1369 Acked Read 3 Done Sent Done false
1577 1370 Acked Read 9 Done Sent Done false
1578 1371 Acked Read 1 Done Sent Done false
1579 1372 Acked Read 10 Done Sent Done false
1580 1373 Acked Write 10 Done Sent Done false
1581 1374 Acked Write 10 Done Sent Done false
1582 1375 Acked Write 9 Done Sent Done false
1583 1376 Acked Read 8 Done Sent Done false
1584 1377 Acked Flush 0 Done Sent Done false
1585 1378 Acked Flush 0 Done Sent Done false
1586 1379 Acked Read 1 Done Sent Done false
1587 1380 Acked Write 4 Done Sent Done false
1588 1381 Acked Write 5 Done Sent Done false
1589 1382 Acked Write 3 Done Sent Done false
1590 1383 Acked Write 7 Done Sent Done false
1591 1384 Acked Read 2 Done Sent Done false
1592 1385 Acked Read 4 Done Sent Done false
1593 1386 Acked Write 8 Done Sent Done false
1594 1387 Acked Read 1 Done Sent Done false
1595 1388 Acked Write 5 Done Sent Done false
1596 1389 Acked Read 9 Done Sent Done false
1597 1390 Acked Read 2 Done Sent Done false
1598 1391 Acked Read 3 Done Sent Done false
1599 1392 Acked Write 5 Done Sent Done false
1600 1393 Acked Read 6 Done Sent Done false
1601 1394 Acked Write 9 Done Sent Done false
1602 1395 Acked Read 4 Done Sent Done false
1603 1396 Acked Flush 0 Done Sent Done false
1604 1397 Acked Read 3 Done Sent Done false
1605 1398 Acked Write 8 Done Sent Done false
1606 1399 Acked Flush 0 Done Sent Done false
1607 1400 Acked Flush 0 Done Sent Done false
1608 1401 Acked Write 6 Done Sent Done false
1609 1402 Acked Read 9 Done Sent Done false
1610 1403 Acked Write 5 Done Sent Done false
1611 1404 Acked Write 1 Done Sent Done false
1612 1405 Acked Read 10 Done Sent Done false
1613 1406 Acked Read 7 Done Sent Done false
1614 1407 Acked Read 8 Done Sent Done false
1615 1408 Acked Read 7 Done Sent Done false
1616 1409 Acked Write 9 Done Sent Done false
1617 1410 Acked Read 5 Done Sent Done false
1618 1411 Acked Flush 0 Done Sent Done false
1619 1412 Acked Write 5 Done Sent Done false
1620 1413 Acked Read 5 Done Sent Done false
1621 1414 Acked Write 7 Done Sent Done false
1622 1415 Acked Read 8 Done Sent Done false
1623 1416 Acked Write 10 Done Sent Done false
1624 1417 Acked Read 5 Done Sent Done false
1625 1418 Acked Write 10 Done Sent Done false
1626 1419 Acked Write 8 Done Sent Done false
1627 1420 Acked Read 7 Done Sent Done false
1628 1421 Acked Read 8 Done Sent Done false
1629 1422 Acked Read 6 Done Sent Done false
1630 1423 Acked Read 8 Done Sent Done false
1631 1424 Acked Read 10 Done Sent Done false
1632 1425 Acked Read 9 Done Sent Done false
1633 1426 Acked Read 9 Done Sent Done false
1634 1427 Acked Read 7 Done Sent Done false
1635 1428 Acked Read 6 Done Sent Done false
1636 1429 Acked Read 10 Done Sent Done false
1637 1430 Acked Read 9 Done Sent Done false
1638 1431 Acked Read 3 Done Sent Done false
1639 1432 Acked Read 1 Done Sent Done false
1640 1433 Acked Read 4 Done Sent Done false
1641 1434 Acked Write 6 Done Sent Done false
1642 1435 Acked Read 1 Done Sent Done false
1643 1436 Acked Read 2 Done Sent Done false
1644 1437 Acked Read 6 Done Sent Done false
1645 1438 Acked Read 7 Done Sent Done false
1646 1439 Acked Write 3 Done Sent Done false
1647 1440 Acked Read 3 Done Sent Done false
1648 1441 Acked Flush 0 Done Sent Done false
1649 1442 Acked Flush 0 Done Sent Done false
1650 1443 Acked Write 10 Done Sent Done false
1651 1444 Acked Read 6 Done Sent Done false
1652 1445 Acked Read 3 Done Sent Done false
1653 1446 Acked Read 4 Done Sent Done false
1654 1447 Acked Write 4 Done Sent Done false
1655 1448 Acked Read 6 Done Sent Done false
1656 1449 Acked Read 8 Done Sent Done false
1657 1450 Acked Read 7 Done Sent Done false
1658 1451 Acked Write 8 Done Sent Done false
1659 1452 Acked Read 5 Done Sent Done false
1660 1453 Acked Write 2 Done Sent Done false
1661 1454 Acked Write 9 Done Sent Done false
1662 1455 Acked Read 6 Done Sent Done false
1663 1456 Acked Write 9 Done Sent Done false
1664 1457 Acked Read 9 Done Sent Done false
1665 1458 Acked Read 2 Done Sent Done false
1666 1459 Acked Read 5 Done Sent Done false
1667 1460 Acked Flush 0 Done Sent Done false
1668 1461 Acked Write 8 Done Sent Done false
1669 1462 Acked Read 9 Done Sent Done false
1670 1463 Acked Flush 0 Done Sent Done false
1671 1464 Acked Read 9 Done Sent Done false
1672 1465 Acked Write 1 Done Sent Done false
1673 1466 Acked Write 2 Done Sent Done false
1674 1467 Acked Write 6 Done Sent Done false
1675 1468 Acked Read 2 Done Sent Done false
1676 1469 Acked Flush 0 Done Sent Done false
1677 1470 Acked Read 4 Done Sent Done false
1678 1471 Acked Read 3 Done Sent Done false
1679 1472 Acked Read 9 Done Sent Done false
1680 1473 Acked Read 1 Done Sent Done false
1681 1474 Acked Read 10 Done Sent Done false
1682 1475 Acked Read 7 Done Sent Done false
1683 1476 Acked Write 5 Done Sent Done false
1684 1477 Acked Write 7 Done Sent Done false
1685 1478 Acked Read 9 Done Sent Done false
1686 1479 Acked Read 5 Done Sent Done false
1687 1480 Acked Read 6 Done Sent Done false
1688 1481 Acked Flush 0 Done Sent Done false
1689 1482 Acked Flush 0 Done Sent Done false
1690 1483 Acked Write 5 Done Sent Done false
1691 1484 Acked Flush 0 Done Sent Done false
1692 1485 Acked Read 1 Done Sent Done false
1693 1486 Acked Write 4 Done Sent Done false
1694 1487 Acked Write 6 Done Sent Done false
1695 1488 Acked Read 4 Done Sent Done false
1696 1489 Acked Flush 0 Done Sent Done false
1697 1490 Acked Read 3 Done Sent Done false
1698 1491 Acked Flush 0 Done Sent Done false
1699 1492 Acked Read 3 Done Sent Done false
1700 1493 Acked Flush 0 Done Sent Done false
1701 1494 Acked Read 2 Done Sent Done false
1702 1495 Acked Read 1 Done Sent Done false
1703 1496 Acked Read 6 Done Sent Done false
1704 1497 Acked Read 1 Done Sent Done false
1705 1498 Acked Read 1 Done Sent Done false
1706 1499 Acked Write 9 Done Sent Done false
1707 1500 Acked Flush 0 Done Sent Done false
1708 1501 Acked Read 9 Done Sent Done false
1709 1502 Acked Write 2 Done Sent Done false
1710 1503 Acked Write 1 Done Sent Done false
1711 1504 Acked Write 10 Done Sent Done false
1712 1505 Acked Write 4 Done Sent Done false
1713 1506 Acked Read 6 Done Sent Done false
1714 1507 Acked Read 1 Done Sent Done false
1715 1508 Acked Write 8 Done Sent Done false
1716 1509 Acked Read 9 Done Sent Done false
1717 1510 Acked Read 4 Done Sent Done false
1718 1511 Acked Write 5 Done Sent Done false
1719 1512 Acked Write 1 Done Sent Done false
1720 1513 Acked Read 8 Done Sent Done false
1721 1514 Acked Read 5 Done Sent Done false
1722 1515 Acked Flush 0 Done Sent Done false
1723 1516 Acked Write 2 Done Sent Done false
1724 1517 Acked Read 7 Done Sent Done false
1725 1518 Acked Read 9 Done Sent Done false
1726 1519 Acked Read 1 Done Sent Done false
1727 1520 Acked Flush 0 Done Sent Done false
1728 1521 Acked Write 6 Done Sent Done false
1729 1522 Acked Write 9 Done Sent Done false
1730 1523 Acked Write 2 Done Sent Done false
1731 1524 Acked Write 6 Done Sent Done false
1732 1525 Acked Read 8 Done Sent Done false
1733 1526 Acked Read 10 Done Sent Done false
1734 1527 Acked Read 7 Done Sent Done false
1735 1528 Acked Write 4 Done Sent Done false
1736 1529 Acked Write 4 Done Sent Done false
1737 1530 Acked Read 1 Done Sent Done false
1738 1531 Acked Read 4 Done Sent Done false
1739 1532 Acked Write 7 Done Sent Done false
1740 1533 Acked Read 5 Done Sent Done false
1741 1534 Acked Read 4 Done Sent Done false
1742 1535 Acked Write 6 Done Sent Done false
1743 1536 Acked Read 5 Done Sent Done false
1744 1537 Acked Read 5 Done Sent Done false
1745 1538 Acked Write 2 Done Sent Done false
1746 1539 Acked Flush 0 Done Sent Done false
1747 1540 Acked Read 9 Done Sent Done false
1748 1541 Acked Read 10 Done Sent Done false
1749 1542 Acked Write 8 Done Sent Done false
1750 1543 Acked Read 7 Done Sent Done false
1751 1544 Acked Read 3 Done Sent Done false
1752 1545 Acked Read 1 Done Sent Done false
1753 1546 Acked Read 10 Done Sent Done false
1754 1547 Acked Write 7 Done Sent Done false
1755 1548 Acked Write 2 Done Sent Done false
1756 1549 Acked Write 8 Done Sent Done false
1757 1550 Acked Write 9 Done Sent Done false
1758 1551 Acked Write 4 Done Sent Done false
1759 1552 Acked Read 2 Done Sent Done false
1760 1553 Acked Write 6 Done Sent Done false
1761 1554 Acked Read 1 Done Sent Done false
1762 1555 Acked Read 8 Done Sent Done false
1763 1556 Acked Write 5 Done Sent Done false
1764 1557 Acked Write 10 Done Sent Done false
1765 1558 Acked Read 8 Done Sent Done false
1766 1559 Acked Read 6 Done Sent Done false
1767 1560 Acked Write 10 Done Sent Done false
1768 1561 Acked Read 8 Done Sent Done false
1769 1562 Acked Write 8 Done Sent Done false
1770 1563 Acked Write 2 Done Sent Done false
1771 1564 Acked Write 4 Done Sent Done false
1772 1565 Acked Write 10 Done Sent Done false
1773 1566 Acked Write 1 Done Sent Done false
1774 1567 Acked Flush 0 Done Sent Done false
1775 1568 Acked Write 6 Done Sent Done false
1776 1569 Acked Read 8 Done Sent Done false
1777 1570 Acked Read 6 Done Sent Done false
1778 1571 Acked Write 7 Done Sent Done false
1779 1572 Acked Read 6 Done Sent Done false
1780 1573 Acked Flush 0 Done Sent Done false
1781 1574 Acked Write 3 Done Sent Done false
1782 1575 Acked Read 1 Done Sent Done false
1783 1576 Acked Read 1 Done Sent Done false
1784 1577 Acked Flush 0 Done Sent Done false
1785 1578 Acked Write 7 Done Sent Done false
1786 1579 Acked Write 4 Done Sent Done false
1787 1580 Acked Read 3 Done Sent Done false
1788 1581 Acked Write 4 Done Sent Done false
1789 1582 Acked Flush 0 Done Sent Done false
1790 1583 Acked Flush 0 Done Sent Done false
1791 1584 Acked Read 10 Done Sent Done false
1792 1585 Acked Write 10 Done Sent Done false
1793 1586 Acked Write 4 Done Sent Done false
1794 1587 Acked Read 4 Done Sent Done false
1795 1588 Acked Write 1 Done Sent Done false
1796 1589 Acked Write 2 Done Sent Done false
1797 1590 Acked Write 3 Done Sent Done false
1798 1591 Acked Write 7 Done Sent Done false
1799 1592 Acked Read 7 Done Sent Done false
1800 1593 Acked Read 2 Done Sent Done false
1801 1594 Acked Flush 0 Done Sent Done false
1802 1595 Acked Write 7 Done Sent Done false
1803 1596 Acked Write 2 Done Sent Done false
1804 1597 Acked Write 9 Done Sent Done false
1805 1598 Acked Flush 0 Done Sent Done false
1806 1599 Acked Write 1 Done Sent Done false
1807 1600 Acked Flush 0 Done Sent Done false
1808 1601 Acked Write 1 Done Sent Done false
1809 1602 Acked Read 8 Done Sent Done false
1810 STATES DS:0 DS:1 DS:2 TOTAL
1811 Sent 0 301 0 301
1812 Done 301 0 301 602
1813 Skipped 0 0 0 0
1814 Error 0 0 0 0
1815 Last Flush: Some(JobId(1600)) Some(JobId(1301)) Some(JobId(1600))
1816 Downstairs last five completed: 1301 1300 1299 1298 1297
1817 Upstairs last five completed: 1602 1601 1600 1599 1598
1818 CLIENT: Up:0 ds:301 act:2
18192025-01-29T05:11:00.934ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1820 [8820] got start action
1821 [8820] I am going to restart
1822 Starting downstairs at port 8820
1823 Make output file at "/tmp/dsc/downstairs-8820.txt"
1824 [8820][1] reports Starting
1825 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1204)
1826 [8820][1] reports Running
18272025-01-29T05:11:01.936ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
18282025-01-29T05:11:02.938ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
1829 ----------------------------------------------------------------
1830 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1831 States: Active Offline Active
1832 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1833 1302 Acked Read 7 Done Sent Done false
1834 1303 Acked Read 7 Done Sent Done false
1835 1304 Acked Read 6 Done Sent Done false
1836 1305 Acked Read 7 Done Sent Done false
1837 1306 Acked Flush 0 Done Sent Done false
1838 1307 Acked Write 6 Done Sent Done false
1839 1308 Acked Flush 0 Done Sent Done false
1840 1309 Acked Write 4 Done Sent Done false
1841 1310 Acked Write 2 Done Sent Done false
1842 1311 Acked Write 4 Done Sent Done false
1843 1312 Acked Read 1 Done Sent Done false
1844 1313 Acked Read 7 Done Sent Done false
1845 1314 Acked Write 1 Done Sent Done false
1846 1315 Acked Write 4 Done Sent Done false
1847 1316 Acked Write 8 Done Sent Done false
1848 1317 Acked Flush 0 Done Sent Done false
1849 1318 Acked Write 3 Done Sent Done false
1850 1319 Acked Write 9 Done Sent Done false
1851 1320 Acked Write 10 Done Sent Done false
1852 1321 Acked Read 3 Done Sent Done false
1853 1322 Acked Flush 0 Done Sent Done false
1854 1323 Acked Flush 0 Done Sent Done false
1855 1324 Acked Write 8 Done Sent Done false
1856 1325 Acked Write 10 Done Sent Done false
1857 1326 Acked Read 9 Done Sent Done false
1858 1327 Acked Write 6 Done Sent Done false
1859 1328 Acked Read 6 Done Sent Done false
1860 1329 Acked Write 1 Done Sent Done false
1861 1330 Acked Write 9 Done Sent Done false
1862 1331 Acked Read 6 Done Sent Done false
1863 1332 Acked Read 5 Done Sent Done false
1864 1333 Acked Read 3 Done Sent Done false
1865 1334 Acked Read 2 Done Sent Done false
1866 1335 Acked Write 4 Done Sent Done false
1867 1336 Acked Read 2 Done Sent Done false
1868 1337 Acked Read 10 Done Sent Done false
1869 1338 Acked Write 8 Done Sent Done false
1870 1339 Acked Flush 0 Done Sent Done false
1871 1340 Acked Write 10 Done Sent Done false
1872 1341 Acked Read 10 Done Sent Done false
1873 1342 Acked Read 5 Done Sent Done false
1874 1343 Acked Read 8 Done Sent Done false
1875 1344 Acked Write 1 Done Sent Done false
1876 1345 Acked Read 4 Done Sent Done false
1877 1346 Acked Read 5 Done Sent Done false
1878 1347 Acked Flush 0 Done Sent Done false
1879 1348 Acked Read 2 Done Sent Done false
1880 1349 Acked Flush 0 Done Sent Done false
1881 1350 Acked Flush 0 Done Sent Done false
1882 1351 Acked Write 6 Done Sent Done false
1883 1352 Acked Write 3 Done Sent Done false
1884 1353 Acked Read 4 Done Sent Done false
1885 1354 Acked Read 8 Done Sent Done false
1886 1355 Acked Write 3 Done Sent Done false
1887 1356 Acked Write 10 Done Sent Done false
1888 1357 Acked Read 9 Done Sent Done false
1889 1358 Acked Read 6 Done Sent Done false
1890 1359 Acked Read 5 Done Sent Done false
1891 1360 Acked Read 4 Done Sent Done false
1892 1361 Acked Read 7 Done Sent Done false
1893 1362 Acked Read 2 Done Sent Done false
1894 1363 Acked Read 4 Done Sent Done false
1895 1364 Acked Flush 0 Done Sent Done false
1896 1365 Acked Read 1 Done Sent Done false
1897 1366 Acked Flush 0 Done Sent Done false
1898 1367 Acked Read 8 Done Sent Done false
1899 1368 Acked Read 9 Done Sent Done false
1900 1369 Acked Read 3 Done Sent Done false
1901 1370 Acked Read 9 Done Sent Done false
1902 1371 Acked Read 1 Done Sent Done false
1903 1372 Acked Read 10 Done Sent Done false
1904 1373 Acked Write 10 Done Sent Done false
1905 1374 Acked Write 10 Done Sent Done false
1906 1375 Acked Write 9 Done Sent Done false
1907 1376 Acked Read 8 Done Sent Done false
1908 1377 Acked Flush 0 Done Sent Done false
1909 1378 Acked Flush 0 Done Sent Done false
1910 1379 Acked Read 1 Done Sent Done false
1911 1380 Acked Write 4 Done Sent Done false
1912 1381 Acked Write 5 Done Sent Done false
1913 1382 Acked Write 3 Done Sent Done false
1914 1383 Acked Write 7 Done Sent Done false
1915 1384 Acked Read 2 Done Sent Done false
1916 1385 Acked Read 4 Done Sent Done false
1917 1386 Acked Write 8 Done Sent Done false
1918 1387 Acked Read 1 Done Sent Done false
1919 1388 Acked Write 5 Done Sent Done false
1920 1389 Acked Read 9 Done Sent Done false
1921 1390 Acked Read 2 Done Sent Done false
1922 1391 Acked Read 3 Done Sent Done false
1923 1392 Acked Write 5 Done Sent Done false
1924 1393 Acked Read 6 Done Sent Done false
1925 1394 Acked Write 9 Done Sent Done false
1926 1395 Acked Read 4 Done Sent Done false
1927 1396 Acked Flush 0 Done Sent Done false
1928 1397 Acked Read 3 Done Sent Done false
1929 1398 Acked Write 8 Done Sent Done false
1930 1399 Acked Flush 0 Done Sent Done false
1931 1400 Acked Flush 0 Done Sent Done false
1932 1401 Acked Write 6 Done Sent Done false
1933 1402 Acked Read 9 Done Sent Done false
1934 1403 Acked Write 5 Done Sent Done false
1935 1404 Acked Write 1 Done Sent Done false
1936 1405 Acked Read 10 Done Sent Done false
1937 1406 Acked Read 7 Done Sent Done false
1938 1407 Acked Read 8 Done Sent Done false
1939 1408 Acked Read 7 Done Sent Done false
1940 1409 Acked Write 9 Done Sent Done false
1941 1410 Acked Read 5 Done Sent Done false
1942 1411 Acked Flush 0 Done Sent Done false
1943 1412 Acked Write 5 Done Sent Done false
1944 1413 Acked Read 5 Done Sent Done false
1945 1414 Acked Write 7 Done Sent Done false
1946 1415 Acked Read 8 Done Sent Done false
1947 1416 Acked Write 10 Done Sent Done false
1948 1417 Acked Read 5 Done Sent Done false
1949 1418 Acked Write 10 Done Sent Done false
1950 1419 Acked Write 8 Done Sent Done false
1951 1420 Acked Read 7 Done Sent Done false
1952 1421 Acked Read 8 Done Sent Done false
1953 1422 Acked Read 6 Done Sent Done false
1954 1423 Acked Read 8 Done Sent Done false
1955 1424 Acked Read 10 Done Sent Done false
1956 1425 Acked Read 9 Done Sent Done false
1957 1426 Acked Read 9 Done Sent Done false
1958 1427 Acked Read 7 Done Sent Done false
1959 1428 Acked Read 6 Done Sent Done false
1960 1429 Acked Read 10 Done Sent Done false
1961 1430 Acked Read 9 Done Sent Done false
1962 1431 Acked Read 3 Done Sent Done false
1963 1432 Acked Read 1 Done Sent Done false
1964 1433 Acked Read 4 Done Sent Done false
1965 1434 Acked Write 6 Done Sent Done false
1966 1435 Acked Read 1 Done Sent Done false
1967 1436 Acked Read 2 Done Sent Done false
1968 1437 Acked Read 6 Done Sent Done false
1969 1438 Acked Read 7 Done Sent Done false
1970 1439 Acked Write 3 Done Sent Done false
1971 1440 Acked Read 3 Done Sent Done false
1972 1441 Acked Flush 0 Done Sent Done false
1973 1442 Acked Flush 0 Done Sent Done false
1974 1443 Acked Write 10 Done Sent Done false
1975 1444 Acked Read 6 Done Sent Done false
1976 1445 Acked Read 3 Done Sent Done false
1977 1446 Acked Read 4 Done Sent Done false
1978 1447 Acked Write 4 Done Sent Done false
1979 1448 Acked Read 6 Done Sent Done false
1980 1449 Acked Read 8 Done Sent Done false
1981 1450 Acked Read 7 Done Sent Done false
1982 1451 Acked Write 8 Done Sent Done false
1983 1452 Acked Read 5 Done Sent Done false
1984 1453 Acked Write 2 Done Sent Done false
1985 1454 Acked Write 9 Done Sent Done false
1986 1455 Acked Read 6 Done Sent Done false
1987 1456 Acked Write 9 Done Sent Done false
1988 1457 Acked Read 9 Done Sent Done false
1989 1458 Acked Read 2 Done Sent Done false
1990 1459 Acked Read 5 Done Sent Done false
1991 1460 Acked Flush 0 Done Sent Done false
1992 1461 Acked Write 8 Done Sent Done false
1993 1462 Acked Read 9 Done Sent Done false
1994 1463 Acked Flush 0 Done Sent Done false
1995 1464 Acked Read 9 Done Sent Done false
1996 1465 Acked Write 1 Done Sent Done false
1997 1466 Acked Write 2 Done Sent Done false
1998 1467 Acked Write 6 Done Sent Done false
1999 1468 Acked Read 2 Done Sent Done false
2000 1469 Acked Flush 0 Done Sent Done false
2001 1470 Acked Read 4 Done Sent Done false
2002 1471 Acked Read 3 Done Sent Done false
2003 1472 Acked Read 9 Done Sent Done false
2004 1473 Acked Read 1 Done Sent Done false
2005 1474 Acked Read 10 Done Sent Done false
2006 1475 Acked Read 7 Done Sent Done false
2007 1476 Acked Write 5 Done Sent Done false
2008 1477 Acked Write 7 Done Sent Done false
2009 1478 Acked Read 9 Done Sent Done false
2010 1479 Acked Read 5 Done Sent Done false
2011 1480 Acked Read 6 Done Sent Done false
2012 1481 Acked Flush 0 Done Sent Done false
2013 1482 Acked Flush 0 Done Sent Done false
2014 1483 Acked Write 5 Done Sent Done false
2015 1484 Acked Flush 0 Done Sent Done false
2016 1485 Acked Read 1 Done Sent Done false
2017 1486 Acked Write 4 Done Sent Done false
2018 1487 Acked Write 6 Done Sent Done false
2019 1488 Acked Read 4 Done Sent Done false
2020 1489 Acked Flush 0 Done Sent Done false
2021 1490 Acked Read 3 Done Sent Done false
2022 1491 Acked Flush 0 Done Sent Done false
2023 1492 Acked Read 3 Done Sent Done false
2024 1493 Acked Flush 0 Done Sent Done false
2025 1494 Acked Read 2 Done Sent Done false
2026 1495 Acked Read 1 Done Sent Done false
2027 1496 Acked Read 6 Done Sent Done false
2028 1497 Acked Read 1 Done Sent Done false
2029 1498 Acked Read 1 Done Sent Done false
2030 1499 Acked Write 9 Done Sent Done false
2031 1500 Acked Flush 0 Done Sent Done false
2032 1501 Acked Read 9 Done Sent Done false
2033 1502 Acked Write 2 Done Sent Done false
2034 1503 Acked Write 1 Done Sent Done false
2035 1504 Acked Write 10 Done Sent Done false
2036 1505 Acked Write 4 Done Sent Done false
2037 1506 Acked Read 6 Done Sent Done false
2038 1507 Acked Read 1 Done Sent Done false
2039 1508 Acked Write 8 Done Sent Done false
2040 1509 Acked Read 9 Done Sent Done false
2041 1510 Acked Read 4 Done Sent Done false
2042 1511 Acked Write 5 Done Sent Done false
2043 1512 Acked Write 1 Done Sent Done false
2044 1513 Acked Read 8 Done Sent Done false
2045 1514 Acked Read 5 Done Sent Done false
2046 1515 Acked Flush 0 Done Sent Done false
2047 1516 Acked Write 2 Done Sent Done false
2048 1517 Acked Read 7 Done Sent Done false
2049 1518 Acked Read 9 Done Sent Done false
2050 1519 Acked Read 1 Done Sent Done false
2051 1520 Acked Flush 0 Done Sent Done false
2052 1521 Acked Write 6 Done Sent Done false
2053 1522 Acked Write 9 Done Sent Done false
2054 1523 Acked Write 2 Done Sent Done false
2055 1524 Acked Write 6 Done Sent Done false
2056 1525 Acked Read 8 Done Sent Done false
2057 1526 Acked Read 10 Done Sent Done false
2058 1527 Acked Read 7 Done Sent Done false
2059 1528 Acked Write 4 Done Sent Done false
2060 1529 Acked Write 4 Done Sent Done false
2061 1530 Acked Read 1 Done Sent Done false
2062 1531 Acked Read 4 Done Sent Done false
2063 1532 Acked Write 7 Done Sent Done false
2064 1533 Acked Read 5 Done Sent Done false
2065 1534 Acked Read 4 Done Sent Done false
2066 1535 Acked Write 6 Done Sent Done false
2067 1536 Acked Read 5 Done Sent Done false
2068 1537 Acked Read 5 Done Sent Done false
2069 1538 Acked Write 2 Done Sent Done false
2070 1539 Acked Flush 0 Done Sent Done false
2071 1540 Acked Read 9 Done Sent Done false
2072 1541 Acked Read 10 Done Sent Done false
2073 1542 Acked Write 8 Done Sent Done false
2074 1543 Acked Read 7 Done Sent Done false
2075 1544 Acked Read 3 Done Sent Done false
2076 1545 Acked Read 1 Done Sent Done false
2077 1546 Acked Read 10 Done Sent Done false
2078 1547 Acked Write 7 Done Sent Done false
2079 1548 Acked Write 2 Done Sent Done false
2080 1549 Acked Write 8 Done Sent Done false
2081 1550 Acked Write 9 Done Sent Done false
2082 1551 Acked Write 4 Done Sent Done false
2083 1552 Acked Read 2 Done Sent Done false
2084 1553 Acked Write 6 Done Sent Done false
2085 1554 Acked Read 1 Done Sent Done false
2086 1555 Acked Read 8 Done Sent Done false
2087 1556 Acked Write 5 Done Sent Done false
2088 1557 Acked Write 10 Done Sent Done false
2089 1558 Acked Read 8 Done Sent Done false
2090 1559 Acked Read 6 Done Sent Done false
2091 1560 Acked Write 10 Done Sent Done false
2092 1561 Acked Read 8 Done Sent Done false
2093 1562 Acked Write 8 Done Sent Done false
2094 1563 Acked Write 2 Done Sent Done false
2095 1564 Acked Write 4 Done Sent Done false
2096 1565 Acked Write 10 Done Sent Done false
2097 1566 Acked Write 1 Done Sent Done false
2098 1567 Acked Flush 0 Done Sent Done false
2099 1568 Acked Write 6 Done Sent Done false
2100 1569 Acked Read 8 Done Sent Done false
2101 1570 Acked Read 6 Done Sent Done false
2102 1571 Acked Write 7 Done Sent Done false
2103 1572 Acked Read 6 Done Sent Done false
2104 1573 Acked Flush 0 Done Sent Done false
2105 1574 Acked Write 3 Done Sent Done false
2106 1575 Acked Read 1 Done Sent Done false
2107 1576 Acked Read 1 Done Sent Done false
2108 1577 Acked Flush 0 Done Sent Done false
2109 1578 Acked Write 7 Done Sent Done false
2110 1579 Acked Write 4 Done Sent Done false
2111 1580 Acked Read 3 Done Sent Done false
2112 1581 Acked Write 4 Done Sent Done false
2113 1582 Acked Flush 0 Done Sent Done false
2114 1583 Acked Flush 0 Done Sent Done false
2115 1584 Acked Read 10 Done Sent Done false
2116 1585 Acked Write 10 Done Sent Done false
2117 1586 Acked Write 4 Done Sent Done false
2118 1587 Acked Read 4 Done Sent Done false
2119 1588 Acked Write 1 Done Sent Done false
2120 1589 Acked Write 2 Done Sent Done false
2121 1590 Acked Write 3 Done Sent Done false
2122 1591 Acked Write 7 Done Sent Done false
2123 1592 Acked Read 7 Done Sent Done false
2124 1593 Acked Read 2 Done Sent Done false
2125 1594 Acked Flush 0 Done Sent Done false
2126 1595 Acked Write 7 Done Sent Done false
2127 1596 Acked Write 2 Done Sent Done false
2128 1597 Acked Write 9 Done Sent Done false
2129 1598 Acked Flush 0 Done Sent Done false
2130 1599 Acked Write 1 Done Sent Done false
2131 1600 Acked Flush 0 Done Sent Done false
2132 1601 Acked Write 1 Done Sent Done false
2133 1602 Acked Read 8 Done Sent Done false
2134 STATES DS:0 DS:1 DS:2 TOTAL
2135 Sent 0 301 0 301
2136 Done 301 0 301 602
2137 Skipped 0 0 0 0
2138 Error 0 0 0 0
2139 Last Flush: Some(JobId(1600)) Some(JobId(1301)) Some(JobId(1600))
2140 Downstairs last five completed: 1301 1300 1299 1298 1297
2141 Upstairs last five completed: 1602 1601 1600 1599 1598
2142 CLIENT: Up:0 ds:301 act:2
21432025-01-29T05:11:04.942ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21442025-01-29T05:11:05.699ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21452025-01-29T05:11:05.699ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60767) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21462025-01-29T05:11:05.700ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21472025-01-29T05:11:05.700ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21482025-01-29T05:11:05.700ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21492025-01-29T05:11:05.700ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21502025-01-29T05:11:05.700ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21512025-01-29T05:11:05.700ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21522025-01-29T05:11:05.701ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2153 ----------------------------------------------------------------
2154 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2155 Downstairs last five completed: 1603 1602 1601 1600 1599
2156 Upstairs last five completed: 1603 1602 1601 1600 1599
2157 CLIENT: Up:0 ds:0 act:3
2158 Replay: All jobs finished
2159 Main task has work to do, go find it
2160 got dsc Stop(0)
2161 stop 0
2162 Jan 29 05:11:07.948[8810] Got stop action so:false kr:false
2163 INFO request completed, latency_us: 361, response_code: 204, uri: /stop/cid/0, method: POST, req_id: dcfeb62d-d4e9-49eb-bada-258c81e5dfa9, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
2164 Jan 29 05:11:07.948 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/0, method: GET, req_id{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:11:07.948930267Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
2165 : 8faf1b20-497b-4157-b8e8-33c7520eda22, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
21662025-01-29T05:11:07.949ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21672025-01-29T05:11:07.949ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21682025-01-29T05:11:07.949ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2169 [8810] Exited with: ExitStatus(unix_wait_status(9))
2170 [8810][0] reports Exit
2171 Jan 29 05:11:11.951 INFO request completed, latency_us: 354, response_code: 200, uri: /state/cid/0, method: GET, req_id: 667175cf-138e-46b7-b4f1-fdc60fa859e1, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
2172 001/300 Write block 2999 len 32768 data: 2 2 2 3 3 3 3 3
2173 002/300 Flush
2174 003/300 Read block 437 len 12288
2175 004/300 Flush
2176 005/300 Flush
2177 006/300 Write block 2086 len 8192 data: 2 2
2178 007/300 Read block 206 len 12288
2179 008/300 Read block 3313 len 12288
2180 009/300 Write block 1353 len 36864 data: 2 2 2 2 2 2 2 2 2
2181 010/300 Write block 2382 len 4096 data: 4
2182 011/300 Write block 821 len 28672 data: 3 2 2 2 2 2 2
2183 012/300 Write block 688 len 32768 data: 2 3 3 3 3 2 2 2
2184 013/300 Write block 1410 len 12288 data: 2 2 2
2185 014/300 Write block 1067 len 4096 data: 2
2186 015/300 Read block 4075 len 20480
2187 016/300 Read block 3848 len 24576
2188 017/300 Write block 4365 len 36864 data: 3 3 3 2 2 3 3 3 3
2189 018/300 Write block 4120 len 20480 data: 3 3 3 3 3
2190 019/300 Read block 1858 len 40960
2191 020/300 Read block 2074 len 16384
2192 021/300 Read block 2699 len 32768
2193 022/300 Write block 2523 len 32768 data: 2 2 2 2 2 2 2 2
2194 023/300 Flush
2195 024/300 Write block 4810 len 4096 data: 2
2196 025/300 Read block 809 len 40960
2197 026/300 Flush
2198 027/300 Read block 1038 len 8192
2199 028/300 Read block 3647 len 20480
2200 029/300 Read block 3172 len 24576
2201 030/300 Read block 167 len 40960
2202 031/300 Write block 4761 len 24576 data: 2 2 2 2 2 2
2203 032/300 Read block 605 len 4096
2204 033/300 Read block 2485 len 32768
2205 034/300 Read block 4000 len 40960
2206 035/300 Read block 4837 len 24576
2207 036/300 Read block 4353 len 12288
2208 037/300 Read block 3311 len 8192
2209 038/300 Read block 2294 len 32768
2210 039/300 Write block 3259 len 8192 data: 2 2
2211 040/300 Read block 2835 len 4096
2212 041/300 Read block 95 len 16384
2213 042/300 Write block 1389 len 12288 data: 2 2 2
2214 043/300 Write block 4517 len 12288 data: 2 2 2
2215 044/300 Flush
2216 045/300 Flush
2217 046/300 Read block 3882 len 24576
2218 047/300 Read block 4791 len 4096
2219 048/300 Write block 1965 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2220 049/300 Read block 209 len 4096
2221 050/300 Read block 2504 len 8192
2222 051/300 Write block 817 len 12288 data: 3 3 3
2223 052/300 Read block 4970 len 36864
2224 053/300 Write block 3944 len 20480 data: 3 3 3 3 3
2225 054/300 Flush
2226 055/300 Read block 345 len 4096
2227 056/300 Write block 2204 len 12288 data: 2 2 2
2228 057/300 Read block 3073 len 4096
2229 058/300 Read block 3886 len 32768
2230 059/300 Read block 235 len 20480
2231 060/300 Read block 2088 len 24576
2232 061/300 Read block 553 len 40960
2233 062/300 Read block 4557 len 32768
2234 063/300 Read block 3445 len 20480
2235 064/300 Write block 68 len 40960 data: 2 2 2 3 3 3 3 3 3 3
2236 065/300 Read block 3320 len 16384
2237 066/300 Write block 4335 len 4096 data: 2
2238 067/300 Read block 3827 len 20480
2239 068/300 Write block 898 len 32768 data: 2 2 2 2 2 2 2 2
2240 069/300 Write block 4124 len 28672 data: 4 3 3 3 3 2 2
2241 070/300 Write block 4530 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2242 071/300 Read block 2285 len 20480
2243 072/300 Read block 1539 len 36864
2244 073/300 Write block 3544 len 4096 data: 3
2245 074/300 Write block 2889 len 20480 data: 2 2 2 2 2
2246 075/300 Write block 4391 len 24576 data: 2 3 3 3 3 2
2247 076/300 Write block 4795 len 20480 data: 2 2 2 2 2
2248 077/300 Read block 3104 len 16384
2249 078/300 Write block 4101 len 16384 data: 2 2 2 2
2250 079/300 Read block 3021 len 32768
2251 080/300 Read block 2268 len 4096
2252 081/300 Flush
2253 082/300 Read block 4149 len 20480
2254 083/300 Write block 3904 len 8192 data: 2 2
2255 084/300 Read block 585 len 28672
2256 085/300 Read block 2577 len 28672
2257 086/300 Flush
2258 087/300 Flush
2259 088/300 Write block 3370 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2260 089/300 Read block 352 len 8192
2261 090/300 Write block 374 len 36864 data: 2 2 2 2 3 3 3 3 3
2262 091/300 Write block 1783 len 8192 data: 2 2
2263 092/300 Write block 3351 len 8192 data: 2 3
2264 093/300 Read block 2149 len 32768
2265 094/300 Flush
2266 095/300 Write block 1532 len 8192 data: 2 2
2267 096/300 Read block 3048 len 4096
2268 097/300 Read block 1666 len 8192
2269 098/300 Write block 900 len 4096 data: 3
2270 099/300 Read block 1860 len 24576
2271 100/300 Write block 4986 len 28672 data: 2 2 2 2 2 2 2
2272 101/300 Read block 4190 len 24576
2273 102/300 Read block 3169 len 16384
2274 103/300 Flush
2275 104/300 Read block 1730 len 24576
2276 105/300 Read block 2553 len 8192
2277 106/300 Write block 3786 len 40960 data: 2 2 3 3 3 3 3 3 2 2
2278 107/300 Read block 1373 len 36864
2279 108/300 Read block 2703 len 36864
2280 109/300 Write block 223 len 24576 data: 2 2 2 2 2 2
2281 110/300 Flush
2282 111/300 Flush
2283 112/300 Read block 2293 len 36864
2284 113/300 Write block 2573 len 16384 data: 3 3 3 3
2285 114/300 Write block 1713 len 12288 data: 3 2 2
2286 115/300 Write block 801 len 36864 data: 2 2 2 2 2 2 2 2 2
2287 116/300 Read block 2583 len 40960
2288 117/300 Read block 4136 len 24576
2289 118/300 Flush
2290 119/300 Write block 639 len 12288 data: 3 3 3
2291 120/300 Read block 4477 len 36864
2292 121/300 Read block 2650 len 8192
2293 122/300 Flush
2294 123/300 Read block 3558 len 4096
2295 124/300 Write block 4207 len 20480 data: 2 2 2 3 3
2296 125/300 Write block 4974 len 32768 data: 2 2 2 2 2 2 2 2
2297 126/300 Read block 1522 len 16384
2298 127/300 Read block 2310 len 20480
2299 128/300 Write block 2638 len 8192 data: 3 3
2300 129/300 Read block 3928 len 36864
2301 130/300 Read block 2356 len 24576
2302 131/300 Read block 1961 len 20480
2303 132/300 Write block 4812 len 24576 data: 2 2 2 2 2 2
2304 133/300 Read block 1883 len 36864
2305 134/300 Write block 3275 len 28672 data: 2 2 2 2 2 2 2
2306 135/300 Flush
2307 136/300 Read block 381 len 16384
2308 137/300 Read block 692 len 12288
2309 138/300 Write block 3135 len 32768 data: 2 2 2 2 2 2 2 2
2310 139/300 Write block 474 len 40960 data: 2 2 2 2 3 3 3 3 3 3
2311 140/300 Flush
2312 141/300 Write block 1725 len 12288 data: 2 2 2
2313 142/300 Read block 1191 len 36864
2314 143/300 Read block 3001 len 20480
2315 144/300 Write block 3575 len 28672 data: 2 2 2 2 2 2 2
2316 145/300 Write block 2380 len 24576 data: 4 4 5 3 3 3
2317 146/300 Read block 3355 len 32768
2318 147/300 Write block 4724 len 8192 data: 2 2
2319 148/300 Write block 4810 len 36864 data: 3 2 3 3 3 3 3 3 2
2320 149/300 Read block 895 len 8192
2321 150/300 Read block 3300 len 24576
2322 151/300 Write block 4025 len 16384 data: 3 3 3 3
2323 152/300 Read block 2155 len 24576
2324 153/300 Write block 1794 len 28672 data: 2 2 2 2 2 2 2
2325 154/300 Write block 1381 len 28672 data: 2 2 2 2 2 2 2
2326 155/300 Read block 2508 len 36864
2327 156/300 Write block 3046 len 36864 data: 2 2 2 2 2 2 2 2 2
2328 157/300 Read block 469 len 4096
2329 158/300 Write block 1817 len 28672 data: 2 2 2 2 2 2 2
2330 159/300 Read block 2320 len 28672
2331 160/300 Read block 4503 len 20480
2332 161/300 Flush
2333 162/300 Write block 577 len 4096 data: 2
2334 163/300 Write block 2736 len 16384 data: 2 2 2 2
2335 164/300 Read block 3303 len 20480
2336 165/300 Read block 2066 len 32768
2337 166/300 Write block 3389 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2338 167/300 Write block 2600 len 36864 data: 2 3 3 3 3 3 3 3 3
2339 168/300 Write block 1870 len 28672 data: 2 2 2 2 2 2 2
2340 169/300 Read block 4162 len 20480
2341 170/300 Write block 3203 len 32768 data: 2 2 2 2 2 2 2 2
2342 171/300 Read block 4440 len 32768
2343 172/300 Write block 3835 len 20480 data: 2 2 2 2 2
2344 173/300 Write block 1747 len 40960 data: 3 3 3 3 3 3 3 3 2 2
2345 174/300 Read block 207 len 32768
2346 175/300 Write block 4757 len 28672 data: 2 2 2 2 3 3 3
2347 176/300 Write block 3658 len 4096 data: 2
2348 177/300 Write block 80 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2349 178/300 Write block 3003 len 28672 data: 4 4 4 4 3 2 2
2350 179/300 Read block 4705 len 12288
2351 180/300 Write block 2812 len 16384 data: 2 2 2 2
2352 181/300 Write block 2906 len 24576 data: 2 2 2 2 2 2
2353 182/300 Read block 560 len 28672
2354 183/300 Write block 2942 len 40960 data: 2 2 3 3 4 4 4 3 3 2
2355 184/300 Read block 4280 len 4096
2356 185/300 Write block 4308 len 28672 data: 2 2 2 2 2 2 2
2357 186/300 Write block 1261 len 28672 data: 2 2 2 2 2 2 2
2358 187/300 Read block 1033 len 20480
2359 188/300 Read block 322 len 28672
2360 189/300 Read block 3820 len 20480
2361 190/300 Read block 1862 len 12288
2362 191/300 Read block 4596 len 12288
2363 192/300 Write block 2465 len 24576 data: 2 2 2 2 2 3
2364 193/300 Write block 2448 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2365 194/300 Flush
2366 195/300 Read block 2425 len 24576
2367 196/300 Read block 442 len 28672
2368 197/300 Read block 2994 len 16384
2369 198/300 Read block 213 len 36864
2370 199/300 Read block 91 len 20480
2371 200/300 Read block 3107 len 16384
2372 201/300 Write block 414 len 24576 data: 2 2 2 2 2 2
2373 202/300 Read block 273 len 36864
2374 203/300 Write block 4110 len 36864 data: 3 2 2 2 2 2 2 2 2
2375 204/300 Write block 4829 len 4096 data: 2
2376 205/300 Read block 3419 len 28672
2377 206/300 Read block 1213 len 8192
2378 207/300 Read block 3361 len 40960
2379 208/300 Write block 1326 len 12288 data: 2 2 2
2380 209/300 Read block 2429 len 36864
2381 210/300 Read block 3445 len 8192
2382 211/300 Write block 4140 len 20480 data: 3 3 2 2 2
2383 212/300 Read block 1410 len 16384
2384 213/300 Read block 2726 len 8192
2385 214/300 Read block 2691 len 32768
2386 215/300 Write block 4955 len 40960 data: 3 3 3 3 3 2 2 2 2 2
2387 216/300 Read block 2471 len 36864
2388 217/300 Read block 3626 len 12288
2389 218/300 Flush
2390 219/300 Write block 3874 len 20480 data: 2 2 2 2 2
2391 220/300 Write block 3954 len 8192 data: 3 3
2392 221/300 Read block 4913 len 12288
2393 222/300 Read block 2987 len 4096
2394 223/300 Write block 1501 len 8192 data: 3 3
2395 224/300 Read block 1266 len 32768
2396 225/300 Write block 3056 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2397 226/300 Flush
2398 227/300 Write block 1935 len 24576 data: 2 2 2 2 2 2
2399 228/300 Write block 3147 len 12288 data: 2 2 2
2400 229/300 Read block 2165 len 24576
2401 230/300 Flush
2402 231/300 Read block 2907 len 16384
2403 232/300 Read block 4757 len 28672
2404 233/300 Read block 3320 len 12288
2405 234/300 Read block 137 len 32768
2406 235/300 Read block 3979 len 32768
2407 236/300 Read block 2719 len 20480
2408 237/300 Flush
2409 238/300 Write block 2198 len 8192 data: 2 2
2410 239/300 Read block 3855 len 28672
2411 240/300 Write block 1224 len 8192 data: 2 2
2412 241/300 Flush
2413 242/300 Write block 2103 len 24576 data: 2 2 2 2 2 2
2414 243/300 Write block 3974 len 36864 data: 2 2 2 2 2 2 2 2 2
2415 244/300 Read block 2896 len 8192
2416 245/300 Read block 3905 len 32768
2417 246/300 Read block 4443 len 16384
2418 247/300 Read block 2640 len 20480
2419 248/300 Write block 556 len 4096 data: 2
2420 249/300 Read block 4160 len 40960
2421 250/300 Read block 4616 len 4096
2422 251/300 Write block 4313 len 24576 data: 3 3 2 2 2 2
2423 252/300 Read block 871 len 28672
2424 253/300 Write block 1039 len 40960 data: 2 2 3 3 3 3 3 3 3 3
2425 254/300 Flush
2426 255/300 Flush
2427 256/300 Write block 350 len 16384 data: 2 2 2 2
2428 257/300 Read block 756 len 40960
2429 258/300 Flush
2430 259/300 Read block 3512 len 4096
2431 260/300 Read block 654 len 40960
2432 261/300 Read block 4700 len 28672
2433 262/300 Write block 2637 len 12288 data: 3 4 4
2434 263/300 Read block 1069 len 20480
2435 264/300 Write block 3798 len 4096 data: 2
2436 265/300 Write block 3997 len 24576 data: 2 2 2 2 2 2
2437 266/300 Read block 2 len 40960
2438 267/300 Read block 4076 len 4096
2439 268/300 Flush
2440 269/300 Read block 4873 len 28672
2441 270/300 Read block 311 len 24576
2442 271/300 Write block 59 len 32768 data: 2 2 2 2 2 2 2 2
2443 272/300 Write block 1804 len 28672 data: 2 2 2 2 2 2 2
2444 273/300 Read block 3593 len 36864
2445 274/300 Read block 2116 len 24576
2446 275/300 Write block 2203 len 40960 data: 2 3 3 3 2 2 2 2 2 2
2447 276/300 Read block 1357 len 40960
2448 277/300 Flush
2449 278/300 Write block 3154 len 20480 data: 2 2 2 2 2
2450 279/300 Read block 4952 len 4096
2451 280/300 Flush
2452 281/300 Write block 267 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2453 282/300 Write block 4733 len 16384 data: 2 2 2 2
2454 283/300 Write block 4857 len 36864 data: 2 2 3 3 3 3 3 3 2
2455 284/300 Read block 2263 len 28672
2456 285/300 Read block 2669 len 20480
2457 286/300 Write block 3130 len 36864 data: 2 2 2 2 2 3 3 3 3
2458 287/300 Read block 3788 len 20480
2459 288/300 Write block 1982 len 20480 data: 2 2 2 2 2
2460 289/300 Read block 3168 len 12288
2461 290/300 Read block 1782 len 24576
2462 291/300 Read block 638 len 32768
2463 292/300 Write block 3498 len 36864 data: 2 3 3 3 3 3 3 3 3
2464 293/300 Write block 842 len 28672 data: 2 2 2 2 2 2 2
2465 294/300 Read block 4249 len 8192
2466 295/300 Read block 1204 len 4096
2467 296/300 Read block 3583 len 28672
2468 297/300 Read block 4671 len 32768
2469 298/300 Read block 3565 len 8192
2470 299/300 Write block 3741 len 16384 data: 2 2 2 2
2471 300/300 Write block 2256 len 32768 data: 2 2 2 3 3 3 3 3
2472 301/300 Write block 3317 len 32768 data: 2 2 2 2 2 2 2 2
2473 Main task has work to do, go find it
2474 got dsc Start(0)
2475 start 0
2476 Jan 29 05:11:12.195 INFO request completed, latency_us: 254, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0c627fa5-180e-4634-a078-123faf804487, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
2477 Replay: started 0, returned:Ok(())
2478 ----------------------------------------------------------------
2479 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2480 States: Offline Active Active
2481 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2482 1604 Acked Write 8 Sent Done Done false
2483 1605 Acked Flush 0 Sent Done Done false
2484 1606 Acked Read 3 Sent Done Done false
2485 1607 Acked Flush 0 Sent Done Done false
2486 1608 Acked Flush 0 Sent Done Done false
2487 1609 Acked Write 2 Sent Done Done false
2488 1610 Acked Read 3 Sent Done Done false
2489 1611 Acked Read 3 Sent Done Done false
2490 1612 Acked Write 9 Sent Done Done false
2491 1613 Acked Write 1 Sent Done Done false
2492 1614 Acked Write 7 Sent Done Done false
2493 1615 Acked Write 8 Sent Done Done false
2494 1616 Acked Write 3 Sent Done Done false
2495 1617 Acked Write 1 Sent Done Done false
2496 1618 Acked Read 5 Sent Done Done false
2497 1619 Acked Read 6 Sent Done Done false
2498 1620 Acked Write 9 Sent Done Done false
2499 1621 Acked Write 5 Sent Done Done false
2500 1622 Acked Read 10 Sent Done Done false
2501 1623 Acked Read 4 Sent Done Done false
2502 1624 Acked Read 8 Sent Done Done false
2503 1625 Acked Write 8 Sent Done Done false
2504 1626 Acked Flush 0 Sent Done Done false
2505 1627 Acked Write 1 Sent Done Done false
2506 1628 Acked Read 10 Sent Done Done false
2507 1629 Acked Flush 0 Sent Done Done false
2508 1630 Acked Read 2 Sent Done Done false
2509 1631 Acked Read 5 Sent Done Done false
2510 1632 Acked Read 6 Sent Done Done false
2511 1633 Acked Read 10 Sent Done Done false
2512 1634 Acked Write 6 Sent Done Done false
2513 1635 Acked Read 1 Sent Done Done false
2514 1636 Acked Read 8 Sent Done Done false
2515 1637 Acked Read 10 Sent Done Done false
2516 1638 Acked Read 6 Sent Done Done false
2517 1639 Acked Read 3 Sent Done Done false
2518 1640 Acked Read 2 Sent Done Done false
2519 1641 Acked Read 8 Sent Done Done false
2520 1642 Acked Write 2 Sent Done Done false
2521 1643 Acked Read 1 Sent Done Done false
2522 1644 Acked Read 4 Sent Done Done false
2523 1645 Acked Write 3 Sent Done Done false
2524 1646 Acked Write 3 Sent Done Done false
2525 1647 Acked Flush 0 Sent Done Done false
2526 1648 Acked Flush 0 Sent Done Done false
2527 1649 Acked Read 6 Sent Done Done false
2528 1650 Acked Read 1 Sent Done Done false
2529 1651 Acked Write 10 Sent Done Done false
2530 1652 Acked Read 1 Sent Done Done false
2531 1653 Acked Read 2 Sent Done Done false
2532 1654 Acked Write 3 Sent Done Done false
2533 1655 Acked Read 9 Sent Done Done false
2534 1656 Acked Write 5 Sent Done Done false
2535 1657 Acked Flush 0 Sent Done Done false
2536 1658 Acked Read 1 Sent Done Done false
2537 1659 Acked Write 3 Sent Done Done false
2538 1660 Acked Read 1 Sent Done Done false
2539 1661 Acked Read 8 Sent Done Done false
2540 1662 Acked Read 5 Sent Done Done false
2541 1663 Acked Read 6 Sent Done Done false
2542 1664 Acked Read 10 Sent Done Done false
2543 1665 Acked Read 8 Sent Done Done false
2544 1666 Acked Read 5 Sent Done Done false
2545 1667 Acked Write 10 Sent Done Done false
2546 1668 Acked Read 4 Sent Done Done false
2547 1669 Acked Write 1 Sent Done Done false
2548 1670 Acked Read 5 Sent Done Done false
2549 1671 Acked Write 8 Sent Done Done false
2550 1672 Acked Write 7 Sent Done Done false
2551 1673 Acked Write 10 Sent Done Done false
2552 1674 Acked Read 5 Sent Done Done false
2553 1675 Acked Read 9 Sent Done Done false
2554 1676 Acked Write 1 Sent Done Done false
2555 1677 Acked Write 5 Sent Done Done false
2556 1678 Acked Write 6 Sent Done Done false
2557 1679 Acked Write 5 Sent Done Done false
2558 1680 Acked Read 4 Sent Done Done false
2559 1681 Acked Write 4 Sent Done Done false
2560 1682 Acked Read 8 Sent Done Done false
2561 1683 Acked Read 1 Sent Done Done false
2562 1684 Acked Flush 0 Sent Done Done false
2563 1685 Acked Read 5 Sent Done Done false
2564 1686 Acked Write 2 Sent Done Done false
2565 1687 Acked Read 7 Sent Done Done false
2566 1688 Acked Read 7 Sent Done Done false
2567 1689 Acked Flush 0 Sent Done Done false
2568 1690 Acked Flush 0 Sent Done Done false
2569 1691 Acked Write 10 Sent Done Done false
2570 1692 Acked Read 2 Sent Done Done false
2571 1693 Acked Write 9 Sent Done Done false
2572 1694 Acked Write 2 Sent Done Done false
2573 1695 Acked Write 2 Sent Done Done false
2574 1696 Acked Read 8 Sent Done Done false
2575 1697 Acked Flush 0 Sent Done Done false
2576 1698 Acked Write 2 Sent Done Done false
2577 1699 Acked Read 1 Sent Done Done false
2578 1700 Acked Read 2 Sent Done Done false
2579 1701 Acked Write 1 Sent Done Done false
2580 1702 Acked Read 6 Sent Done Done false
2581 1703 Acked Write 7 Sent Done Done false
2582 1704 Acked Read 6 Sent Done Done false
2583 1705 Acked Read 4 Sent Done Done false
2584 1706 Acked Flush 0 Sent Done Done false
2585 1707 Acked Read 6 Sent Done Done false
2586 1708 Acked Read 2 Sent Done Done false
2587 1709 Acked Write 10 Sent Done Done false
2588 1710 Acked Read 9 Sent Done Done false
2589 1711 Acked Read 9 Sent Done Done false
2590 1712 Acked Write 6 Sent Done Done false
2591 1713 Acked Flush 0 Sent Done Done false
2592 1714 Acked Flush 0 Sent Done Done false
2593 1715 Acked Read 9 Sent Done Done false
2594 1716 Acked Write 4 Sent Done Done false
2595 1717 Acked Write 3 Sent Done Done false
2596 1718 Acked Write 9 Sent Done Done false
2597 1719 Acked Read 10 Sent Done Done false
2598 1720 Acked Read 6 Sent Done Done false
2599 1721 Acked Flush 0 Sent Done Done false
2600 1722 Acked Write 3 Sent Done Done false
2601 1723 Acked Read 9 Sent Done Done false
2602 1724 Acked Read 2 Sent Done Done false
2603 1725 Acked Flush 0 Sent Done Done false
2604 1726 Acked Read 1 Sent Done Done false
2605 1727 Acked Write 5 Sent Done Done false
2606 1728 Acked Write 8 Sent Done Done false
2607 1729 Acked Read 4 Sent Done Done false
2608 1730 Acked Read 5 Sent Done Done false
2609 1731 Acked Write 2 Sent Done Done false
2610 1732 Acked Read 9 Sent Done Done false
2611 1733 Acked Read 6 Sent Done Done false
2612 1734 Acked Read 5 Sent Done Done false
2613 1735 Acked Write 6 Sent Done Done false
2614 1736 Acked Read 9 Sent Done Done false
2615 1737 Acked Write 7 Sent Done Done false
2616 1738 Acked Flush 0 Sent Done Done false
2617 1739 Acked Read 4 Sent Done Done false
2618 1740 Acked Read 3 Sent Done Done false
2619 1741 Acked Write 8 Sent Done Done false
2620 1742 Acked Write 10 Sent Done Done false
2621 1743 Acked Flush 0 Sent Done Done false
2622 1744 Acked Write 3 Sent Done Done false
2623 1745 Acked Read 9 Sent Done Done false
2624 1746 Acked Read 5 Sent Done Done false
2625 1747 Acked Write 7 Sent Done Done false
2626 1748 Acked Write 6 Sent Done Done false
2627 1749 Acked Read 8 Sent Done Done false
2628 1750 Acked Write 2 Sent Done Done false
2629 1751 Acked Write 9 Sent Done Done false
2630 1752 Acked Read 2 Sent Done Done false
2631 1753 Acked Read 6 Sent Done Done false
2632 1754 Acked Write 4 Sent Done Done false
2633 1755 Acked Read 6 Sent Done Done false
2634 1756 Acked Write 7 Sent Done Done false
2635 1757 Acked Write 7 Sent Done Done false
2636 1758 Acked Read 9 Sent Done Done false
2637 1759 Acked Write 9 Sent Done Done false
2638 1760 Acked Read 1 Sent Done Done false
2639 1761 Acked Write 7 Sent Done Done false
2640 1762 Acked Read 7 Sent Done Done false
2641 1763 Acked Read 5 Sent Done Done false
2642 1764 Acked Flush 0 Sent Done Done false
2643 1765 Acked Write 1 Sent Done Done false
2644 1766 Acked Write 4 Sent Done Done false
2645 1767 Acked Read 5 Sent Done Done false
2646 1768 Acked Read 8 Sent Done Done false
2647 1769 Acked Write 10 Sent Done Done false
2648 1770 Acked Write 9 Sent Done Done false
2649 1771 Acked Write 7 Sent Done Done false
2650 1772 Acked Read 5 Sent Done Done false
2651 1773 Acked Write 8 Sent Done Done false
2652 1774 Acked Read 8 Sent Done Done false
2653 1775 Acked Write 5 Sent Done Done false
2654 1776 Acked Write 10 Sent Done Done false
2655 1777 Acked Read 8 Sent Done Done false
2656 1778 Acked Write 7 Sent Done Done false
2657 1779 Acked Write 1 Sent Done Done false
2658 1780 Acked Write 10 Sent Done Done false
2659 1781 Acked Write 7 Sent Done Done false
2660 1782 Acked Read 3 Sent Done Done false
2661 1783 Acked Write 4 Sent Done Done false
2662 1784 Acked Write 6 Sent Done Done false
2663 1785 Acked Read 7 Sent Done Done false
2664 1786 Acked Write 10 Sent Done Done false
2665 1787 Acked Read 1 Sent Done Done false
2666 1788 Acked Write 7 Sent Done Done false
2667 1789 Acked Write 7 Sent Done Done false
2668 1790 Acked Read 5 Sent Done Done false
2669 1791 Acked Read 7 Sent Done Done false
2670 1792 Acked Read 5 Sent Done Done false
2671 1793 Acked Read 3 Sent Done Done false
2672 1794 Acked Read 3 Sent Done Done false
2673 1795 Acked Write 6 Sent Done Done false
2674 1796 Acked Write 10 Sent Done Done false
2675 1797 Acked Flush 0 Sent Done Done false
2676 1798 Acked Read 6 Sent Done Done false
2677 1799 Acked Read 7 Sent Done Done false
2678 1800 Acked Read 4 Sent Done Done false
2679 1801 Acked Read 9 Sent Done Done false
2680 1802 Acked Read 5 Sent Done Done false
2681 1803 Acked Read 4 Sent Done Done false
2682 1804 Acked Write 6 Sent Done Done false
2683 1805 Acked Read 9 Sent Done Done false
2684 1806 Acked Write 9 Sent Done Done false
2685 1807 Acked Write 1 Sent Done Done false
2686 1808 Acked Read 7 Sent Done Done false
2687 1809 Acked Read 2 Sent Done Done false
2688 1810 Acked Read 10 Sent Done Done false
2689 1811 Acked Write 3 Sent Done Done false
2690 1812 Acked Read 9 Sent Done Done false
2691 1813 Acked Read 2 Sent Done Done false
2692 1814 Acked Write 5 Sent Done Done false
2693 1815 Acked Read 4 Sent Done Done false
2694 1816 Acked Read 2 Sent Done Done false
2695 1817 Acked Read 8 Sent Done Done false
2696 1818 Acked Write 10 Sent Done Done false
2697 1819 Acked Read 9 Sent Done Done false
2698 1820 Acked Read 3 Sent Done Done false
2699 1821 Acked Flush 0 Sent Done Done false
2700 1822 Acked Write 5 Sent Done Done false
2701 1823 Acked Write 2 Sent Done Done false
2702 1824 Acked Read 3 Sent Done Done false
2703 1825 Acked Read 1 Sent Done Done false
2704 1826 Acked Write 2 Sent Done Done false
2705 1827 Acked Read 8 Sent Done Done false
2706 1828 Acked Write 10 Sent Done Done false
2707 1829 Acked Flush 0 Sent Done Done false
2708 1830 Acked Write 6 Sent Done Done false
2709 1831 Acked Write 3 Sent Done Done false
2710 1832 Acked Read 6 Sent Done Done false
2711 1833 Acked Flush 0 Sent Done Done false
2712 1834 Acked Read 4 Sent Done Done false
2713 1835 Acked Read 7 Sent Done Done false
2714 1836 Acked Read 3 Sent Done Done false
2715 1837 Acked Read 8 Sent Done Done false
2716 1838 Acked Read 8 Sent Done Done false
2717 1839 Acked Read 5 Sent Done Done false
2718 1840 Acked Flush 0 Sent Done Done false
2719 1841 Acked Write 2 Sent Done Done false
2720 1842 Acked Read 7 Sent Done Done false
2721 1843 Acked Write 2 Sent Done Done false
2722 1844 Acked Flush 0 Sent Done Done false
2723 1845 Acked Write 6 Sent Done Done false
2724 1846 Acked Write 9 Sent Done Done false
2725 1847 Acked Read 2 Sent Done Done false
2726 1848 Acked Read 8 Sent Done Done false
2727 1849 Acked Read 4 Sent Done Done false
2728 1850 Acked Read 5 Sent Done Done false
2729 1851 Acked Write 1 Sent Done Done false
2730 1852 Acked Read 10 Sent Done Done false
2731 1853 Acked Read 1 Sent Done Done false
2732 1854 Acked Write 6 Sent Done Done false
2733 1855 Acked Read 7 Sent Done Done false
2734 1856 Acked Write 10 Sent Done Done false
2735 1857 Acked Flush 0 Sent Done Done false
2736 1858 Acked Flush 0 Sent Done Done false
2737 1859 Acked Write 4 Sent Done Done false
2738 1860 Acked Read 10 Sent Done Done false
2739 1861 Acked Flush 0 Sent Done Done false
2740 1862 Acked Read 1 Sent Done Done false
2741 1863 Acked Read 10 Sent Done Done false
2742 1864 Acked Read 7 Sent Done Done false
2743 1865 Acked Write 3 Sent Done Done false
2744 1866 Acked Read 5 Sent Done Done false
2745 1867 Acked Write 1 Sent Done Done false
2746 1868 Acked Write 6 Sent Done Done false
2747 1869 Acked Read 10 Sent Done Done false
2748 1870 Acked Read 1 Sent Done Done false
2749 1871 Acked Flush 0 Sent Done Done false
2750 1872 Acked Read 7 Sent Done Done false
2751 1873 Acked Read 6 Sent Done Done false
2752 1874 Acked Write 8 Sent Done Done false
2753 1875 Acked Write 7 Sent Done Done false
2754 1876 Acked Read 9 Sent Done Done false
2755 1877 Acked Read 6 Sent Done Done false
2756 1878 Acked Write 10 Sent Done Done false
2757 1879 Acked Read 10 Sent Done Done false
2758 1880 Acked Flush 0 Sent Done Done false
2759 1881 Acked Write 5 Sent Done Done false
2760 1882 Acked Read 1 Sent Done Done false
2761 1883 Acked Flush 0 Sent Done Done false
2762 1884 Acked Write 10 Sent Done Done false
2763 1885 Acked Write 4 Sent Done Done false
2764 1886 Acked Write 9 Sent Done Done false
2765 1887 Acked Read 7 Sent Done Done false
2766 1888 Acked Read 5 Sent Done Done false
2767 1889 Acked Write 9 Sent Done Done false
2768 1890 Acked Read 5 Sent Done Done false
2769 1891 Acked Write 5 Sent Done Done false
2770 1892 Acked Read 3 Sent Done Done false
2771 1893 Acked Read 6 Sent Done Done false
2772 1894 Acked Read 8 Sent Done Done false
2773 1895 Acked Write 9 Sent Done Done false
2774 1896 Acked Write 7 Sent Done Done false
2775 1897 Acked Read 2 Sent Done Done false
2776 1898 Acked Read 1 Sent Done Done false
2777 1899 Acked Read 7 Sent Done Done false
2778 1900 Acked Read 8 Sent Done Done false
2779 1901 Acked Read 2 Sent Done Done false
2780 1902 Acked Write 4 Sent Done Done false
2781 1903 Acked Write 8 Sent Done Done false
2782 1904 Acked Write 8 Sent Done Sent false
2783 STATES DS:0 DS:1 DS:2 TOTAL
2784 Sent 301 0 1 302
2785 Done 0 301 300 601
2786 Skipped 0 0 0 0
2787 Error 0 0 0 0
2788 Last Flush: Some(JobId(1603)) Some(JobId(1883)) Some(JobId(1883))
2789 Downstairs last five completed: 1603 1602 1601 1600 1599
2790 Upstairs last five completed: 1904 1903 1902 1901 1900
2791 CLIENT: Up:0 ds:301 act:2
27922025-01-29T05:11:12.956ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
27932025-01-29T05:11:13.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2794 [8810] got start action
2795 [8810] I am going to restart
2796 Starting downstairs at port 8810
2797 Make output file at "/tmp/dsc/downstairs-8810.txt"
2798 [8810][0] reports Starting
2799 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1205)
2800 [8810][0] reports Running
28012025-01-29T05:11:14.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
2802 ----------------------------------------------------------------
2803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2804 States: Offline Active Active
2805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2806 1604 Acked Write 8 Sent Done Done false
2807 1605 Acked Flush 0 Sent Done Done false
2808 1606 Acked Read 3 Sent Done Done false
2809 1607 Acked Flush 0 Sent Done Done false
2810 1608 Acked Flush 0 Sent Done Done false
2811 1609 Acked Write 2 Sent Done Done false
2812 1610 Acked Read 3 Sent Done Done false
2813 1611 Acked Read 3 Sent Done Done false
2814 1612 Acked Write 9 Sent Done Done false
2815 1613 Acked Write 1 Sent Done Done false
2816 1614 Acked Write 7 Sent Done Done false
2817 1615 Acked Write 8 Sent Done Done false
2818 1616 Acked Write 3 Sent Done Done false
2819 1617 Acked Write 1 Sent Done Done false
2820 1618 Acked Read 5 Sent Done Done false
2821 1619 Acked Read 6 Sent Done Done false
2822 1620 Acked Write 9 Sent Done Done false
2823 1621 Acked Write 5 Sent Done Done false
2824 1622 Acked Read 10 Sent Done Done false
2825 1623 Acked Read 4 Sent Done Done false
2826 1624 Acked Read 8 Sent Done Done false
2827 1625 Acked Write 8 Sent Done Done false
2828 1626 Acked Flush 0 Sent Done Done false
2829 1627 Acked Write 1 Sent Done Done false
2830 1628 Acked Read 10 Sent Done Done false
2831 1629 Acked Flush 0 Sent Done Done false
2832 1630 Acked Read 2 Sent Done Done false
2833 1631 Acked Read 5 Sent Done Done false
2834 1632 Acked Read 6 Sent Done Done false
2835 1633 Acked Read 10 Sent Done Done false
2836 1634 Acked Write 6 Sent Done Done false
2837 1635 Acked Read 1 Sent Done Done false
2838 1636 Acked Read 8 Sent Done Done false
2839 1637 Acked Read 10 Sent Done Done false
2840 1638 Acked Read 6 Sent Done Done false
2841 1639 Acked Read 3 Sent Done Done false
2842 1640 Acked Read 2 Sent Done Done false
2843 1641 Acked Read 8 Sent Done Done false
2844 1642 Acked Write 2 Sent Done Done false
2845 1643 Acked Read 1 Sent Done Done false
2846 1644 Acked Read 4 Sent Done Done false
2847 1645 Acked Write 3 Sent Done Done false
2848 1646 Acked Write 3 Sent Done Done false
2849 1647 Acked Flush 0 Sent Done Done false
2850 1648 Acked Flush 0 Sent Done Done false
2851 1649 Acked Read 6 Sent Done Done false
2852 1650 Acked Read 1 Sent Done Done false
2853 1651 Acked Write 10 Sent Done Done false
2854 1652 Acked Read 1 Sent Done Done false
2855 1653 Acked Read 2 Sent Done Done false
2856 1654 Acked Write 3 Sent Done Done false
2857 1655 Acked Read 9 Sent Done Done false
2858 1656 Acked Write 5 Sent Done Done false
2859 1657 Acked Flush 0 Sent Done Done false
2860 1658 Acked Read 1 Sent Done Done false
2861 1659 Acked Write 3 Sent Done Done false
2862 1660 Acked Read 1 Sent Done Done false
2863 1661 Acked Read 8 Sent Done Done false
2864 1662 Acked Read 5 Sent Done Done false
2865 1663 Acked Read 6 Sent Done Done false
2866 1664 Acked Read 10 Sent Done Done false
2867 1665 Acked Read 8 Sent Done Done false
2868 1666 Acked Read 5 Sent Done Done false
2869 1667 Acked Write 10 Sent Done Done false
2870 1668 Acked Read 4 Sent Done Done false
2871 1669 Acked Write 1 Sent Done Done false
2872 1670 Acked Read 5 Sent Done Done false
2873 1671 Acked Write 8 Sent Done Done false
2874 1672 Acked Write 7 Sent Done Done false
2875 1673 Acked Write 10 Sent Done Done false
2876 1674 Acked Read 5 Sent Done Done false
2877 1675 Acked Read 9 Sent Done Done false
2878 1676 Acked Write 1 Sent Done Done false
2879 1677 Acked Write 5 Sent Done Done false
2880 1678 Acked Write 6 Sent Done Done false
2881 1679 Acked Write 5 Sent Done Done false
2882 1680 Acked Read 4 Sent Done Done false
2883 1681 Acked Write 4 Sent Done Done false
2884 1682 Acked Read 8 Sent Done Done false
2885 1683 Acked Read 1 Sent Done Done false
2886 1684 Acked Flush 0 Sent Done Done false
2887 1685 Acked Read 5 Sent Done Done false
2888 1686 Acked Write 2 Sent Done Done false
2889 1687 Acked Read 7 Sent Done Done false
2890 1688 Acked Read 7 Sent Done Done false
2891 1689 Acked Flush 0 Sent Done Done false
2892 1690 Acked Flush 0 Sent Done Done false
2893 1691 Acked Write 10 Sent Done Done false
2894 1692 Acked Read 2 Sent Done Done false
2895 1693 Acked Write 9 Sent Done Done false
2896 1694 Acked Write 2 Sent Done Done false
2897 1695 Acked Write 2 Sent Done Done false
2898 1696 Acked Read 8 Sent Done Done false
2899 1697 Acked Flush 0 Sent Done Done false
2900 1698 Acked Write 2 Sent Done Done false
2901 1699 Acked Read 1 Sent Done Done false
2902 1700 Acked Read 2 Sent Done Done false
2903 1701 Acked Write 1 Sent Done Done false
2904 1702 Acked Read 6 Sent Done Done false
2905 1703 Acked Write 7 Sent Done Done false
2906 1704 Acked Read 6 Sent Done Done false
2907 1705 Acked Read 4 Sent Done Done false
2908 1706 Acked Flush 0 Sent Done Done false
2909 1707 Acked Read 6 Sent Done Done false
2910 1708 Acked Read 2 Sent Done Done false
2911 1709 Acked Write 10 Sent Done Done false
2912 1710 Acked Read 9 Sent Done Done false
2913 1711 Acked Read 9 Sent Done Done false
2914 1712 Acked Write 6 Sent Done Done false
2915 1713 Acked Flush 0 Sent Done Done false
2916 1714 Acked Flush 0 Sent Done Done false
2917 1715 Acked Read 9 Sent Done Done false
2918 1716 Acked Write 4 Sent Done Done false
2919 1717 Acked Write 3 Sent Done Done false
2920 1718 Acked Write 9 Sent Done Done false
2921 1719 Acked Read 10 Sent Done Done false
2922 1720 Acked Read 6 Sent Done Done false
2923 1721 Acked Flush 0 Sent Done Done false
2924 1722 Acked Write 3 Sent Done Done false
2925 1723 Acked Read 9 Sent Done Done false
2926 1724 Acked Read 2 Sent Done Done false
2927 1725 Acked Flush 0 Sent Done Done false
2928 1726 Acked Read 1 Sent Done Done false
2929 1727 Acked Write 5 Sent Done Done false
2930 1728 Acked Write 8 Sent Done Done false
2931 1729 Acked Read 4 Sent Done Done false
2932 1730 Acked Read 5 Sent Done Done false
2933 1731 Acked Write 2 Sent Done Done false
2934 1732 Acked Read 9 Sent Done Done false
2935 1733 Acked Read 6 Sent Done Done false
2936 1734 Acked Read 5 Sent Done Done false
2937 1735 Acked Write 6 Sent Done Done false
2938 1736 Acked Read 9 Sent Done Done false
2939 1737 Acked Write 7 Sent Done Done false
2940 1738 Acked Flush 0 Sent Done Done false
2941 1739 Acked Read 4 Sent Done Done false
2942 1740 Acked Read 3 Sent Done Done false
2943 1741 Acked Write 8 Sent Done Done false
2944 1742 Acked Write 10 Sent Done Done false
2945 1743 Acked Flush 0 Sent Done Done false
2946 1744 Acked Write 3 Sent Done Done false
2947 1745 Acked Read 9 Sent Done Done false
2948 1746 Acked Read 5 Sent Done Done false
2949 1747 Acked Write 7 Sent Done Done false
2950 1748 Acked Write 6 Sent Done Done false
2951 1749 Acked Read 8 Sent Done Done false
2952 1750 Acked Write 2 Sent Done Done false
2953 1751 Acked Write 9 Sent Done Done false
2954 1752 Acked Read 2 Sent Done Done false
2955 1753 Acked Read 6 Sent Done Done false
2956 1754 Acked Write 4 Sent Done Done false
2957 1755 Acked Read 6 Sent Done Done false
2958 1756 Acked Write 7 Sent Done Done false
2959 1757 Acked Write 7 Sent Done Done false
2960 1758 Acked Read 9 Sent Done Done false
2961 1759 Acked Write 9 Sent Done Done false
2962 1760 Acked Read 1 Sent Done Done false
2963 1761 Acked Write 7 Sent Done Done false
2964 1762 Acked Read 7 Sent Done Done false
2965 1763 Acked Read 5 Sent Done Done false
2966 1764 Acked Flush 0 Sent Done Done false
2967 1765 Acked Write 1 Sent Done Done false
2968 1766 Acked Write 4 Sent Done Done false
2969 1767 Acked Read 5 Sent Done Done false
2970 1768 Acked Read 8 Sent Done Done false
2971 1769 Acked Write 10 Sent Done Done false
2972 1770 Acked Write 9 Sent Done Done false
2973 1771 Acked Write 7 Sent Done Done false
2974 1772 Acked Read 5 Sent Done Done false
2975 1773 Acked Write 8 Sent Done Done false
2976 1774 Acked Read 8 Sent Done Done false
2977 1775 Acked Write 5 Sent Done Done false
2978 1776 Acked Write 10 Sent Done Done false
2979 1777 Acked Read 8 Sent Done Done false
2980 1778 Acked Write 7 Sent Done Done false
2981 1779 Acked Write 1 Sent Done Done false
2982 1780 Acked Write 10 Sent Done Done false
2983 1781 Acked Write 7 Sent Done Done false
2984 1782 Acked Read 3 Sent Done Done false
2985 1783 Acked Write 4 Sent Done Done false
2986 1784 Acked Write 6 Sent Done Done false
2987 1785 Acked Read 7 Sent Done Done false
2988 1786 Acked Write 10 Sent Done Done false
2989 1787 Acked Read 1 Sent Done Done false
2990 1788 Acked Write 7 Sent Done Done false
2991 1789 Acked Write 7 Sent Done Done false
2992 1790 Acked Read 5 Sent Done Done false
2993 1791 Acked Read 7 Sent Done Done false
2994 1792 Acked Read 5 Sent Done Done false
2995 1793 Acked Read 3 Sent Done Done false
2996 1794 Acked Read 3 Sent Done Done false
2997 1795 Acked Write 6 Sent Done Done false
2998 1796 Acked Write 10 Sent Done Done false
2999 1797 Acked Flush 0 Sent Done Done false
3000 1798 Acked Read 6 Sent Done Done false
3001 1799 Acked Read 7 Sent Done Done false
3002 1800 Acked Read 4 Sent Done Done false
3003 1801 Acked Read 9 Sent Done Done false
3004 1802 Acked Read 5 Sent Done Done false
3005 1803 Acked Read 4 Sent Done Done false
3006 1804 Acked Write 6 Sent Done Done false
3007 1805 Acked Read 9 Sent Done Done false
3008 1806 Acked Write 9 Sent Done Done false
3009 1807 Acked Write 1 Sent Done Done false
3010 1808 Acked Read 7 Sent Done Done false
3011 1809 Acked Read 2 Sent Done Done false
3012 1810 Acked Read 10 Sent Done Done false
3013 1811 Acked Write 3 Sent Done Done false
3014 1812 Acked Read 9 Sent Done Done false
3015 1813 Acked Read 2 Sent Done Done false
3016 1814 Acked Write 5 Sent Done Done false
3017 1815 Acked Read 4 Sent Done Done false
3018 1816 Acked Read 2 Sent Done Done false
3019 1817 Acked Read 8 Sent Done Done false
3020 1818 Acked Write 10 Sent Done Done false
3021 1819 Acked Read 9 Sent Done Done false
3022 1820 Acked Read 3 Sent Done Done false
3023 1821 Acked Flush 0 Sent Done Done false
3024 1822 Acked Write 5 Sent Done Done false
3025 1823 Acked Write 2 Sent Done Done false
3026 1824 Acked Read 3 Sent Done Done false
3027 1825 Acked Read 1 Sent Done Done false
3028 1826 Acked Write 2 Sent Done Done false
3029 1827 Acked Read 8 Sent Done Done false
3030 1828 Acked Write 10 Sent Done Done false
3031 1829 Acked Flush 0 Sent Done Done false
3032 1830 Acked Write 6 Sent Done Done false
3033 1831 Acked Write 3 Sent Done Done false
3034 1832 Acked Read 6 Sent Done Done false
3035 1833 Acked Flush 0 Sent Done Done false
3036 1834 Acked Read 4 Sent Done Done false
3037 1835 Acked Read 7 Sent Done Done false
3038 1836 Acked Read 3 Sent Done Done false
3039 1837 Acked Read 8 Sent Done Done false
3040 1838 Acked Read 8 Sent Done Done false
3041 1839 Acked Read 5 Sent Done Done false
3042 1840 Acked Flush 0 Sent Done Done false
3043 1841 Acked Write 2 Sent Done Done false
3044 1842 Acked Read 7 Sent Done Done false
3045 1843 Acked Write 2 Sent Done Done false
3046 1844 Acked Flush 0 Sent Done Done false
3047 1845 Acked Write 6 Sent Done Done false
3048 1846 Acked Write 9 Sent Done Done false
3049 1847 Acked Read 2 Sent Done Done false
3050 1848 Acked Read 8 Sent Done Done false
3051 1849 Acked Read 4 Sent Done Done false
3052 1850 Acked Read 5 Sent Done Done false
3053 1851 Acked Write 1 Sent Done Done false
3054 1852 Acked Read 10 Sent Done Done false
3055 1853 Acked Read 1 Sent Done Done false
3056 1854 Acked Write 6 Sent Done Done false
3057 1855 Acked Read 7 Sent Done Done false
3058 1856 Acked Write 10 Sent Done Done false
3059 1857 Acked Flush 0 Sent Done Done false
3060 1858 Acked Flush 0 Sent Done Done false
3061 1859 Acked Write 4 Sent Done Done false
3062 1860 Acked Read 10 Sent Done Done false
3063 1861 Acked Flush 0 Sent Done Done false
3064 1862 Acked Read 1 Sent Done Done false
3065 1863 Acked Read 10 Sent Done Done false
3066 1864 Acked Read 7 Sent Done Done false
3067 1865 Acked Write 3 Sent Done Done false
3068 1866 Acked Read 5 Sent Done Done false
3069 1867 Acked Write 1 Sent Done Done false
3070 1868 Acked Write 6 Sent Done Done false
3071 1869 Acked Read 10 Sent Done Done false
3072 1870 Acked Read 1 Sent Done Done false
3073 1871 Acked Flush 0 Sent Done Done false
3074 1872 Acked Read 7 Sent Done Done false
3075 1873 Acked Read 6 Sent Done Done false
3076 1874 Acked Write 8 Sent Done Done false
3077 1875 Acked Write 7 Sent Done Done false
3078 1876 Acked Read 9 Sent Done Done false
3079 1877 Acked Read 6 Sent Done Done false
3080 1878 Acked Write 10 Sent Done Done false
3081 1879 Acked Read 10 Sent Done Done false
3082 1880 Acked Flush 0 Sent Done Done false
3083 1881 Acked Write 5 Sent Done Done false
3084 1882 Acked Read 1 Sent Done Done false
3085 1883 Acked Flush 0 Sent Done Done false
3086 1884 Acked Write 10 Sent Done Done false
3087 1885 Acked Write 4 Sent Done Done false
3088 1886 Acked Write 9 Sent Done Done false
3089 1887 Acked Read 7 Sent Done Done false
3090 1888 Acked Read 5 Sent Done Done false
3091 1889 Acked Write 9 Sent Done Done false
3092 1890 Acked Read 5 Sent Done Done false
3093 1891 Acked Write 5 Sent Done Done false
3094 1892 Acked Read 3 Sent Done Done false
3095 1893 Acked Read 6 Sent Done Done false
3096 1894 Acked Read 8 Sent Done Done false
3097 1895 Acked Write 9 Sent Done Done false
3098 1896 Acked Write 7 Sent Done Done false
3099 1897 Acked Read 2 Sent Done Done false
3100 1898 Acked Read 1 Sent Done Done false
3101 1899 Acked Read 7 Sent Done Done false
3102 1900 Acked Read 8 Sent Done Done false
3103 1901 Acked Read 2 Sent Done Done false
3104 1902 Acked Write 4 Sent Done Done false
3105 1903 Acked Write 8 Sent Done Done false
3106 1904 Acked Write 8 Sent Done Done false
3107 STATES DS:0 DS:1 DS:2 TOTAL
3108 Sent 301 0 0 301
3109 Done 0 301 301 602
3110 Skipped 0 0 0 0
3111 Error 0 0 0 0
3112 Last Flush: Some(JobId(1603)) Some(JobId(1883)) Some(JobId(1883))
3113 Downstairs last five completed: 1603 1602 1601 1600 1599
3114 Upstairs last five completed: 1904 1903 1902 1901 1900
3115 CLIENT: Up:0 ds:301 act:2
31162025-01-29T05:11:16.961ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31172025-01-29T05:11:17.950ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31182025-01-29T05:11:17.950ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59543) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31192025-01-29T05:11:17.951ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31202025-01-29T05:11:17.952ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31212025-01-29T05:11:17.952ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31222025-01-29T05:11:17.952ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31232025-01-29T05:11:17.952ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31242025-01-29T05:11:17.952ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31252025-01-29T05:11:17.952ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
3126 ----------------------------------------------------------------
3127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3128 Downstairs last five completed: 1905 1904 1903 1902 1901
3129 Upstairs last five completed: 1905 1904 1903 1902 1901
3130 CLIENT: Up:0 ds:0 act:3
3131 Replay: All jobs finished
3132 Main task has work to do, go find it
3133 got dsc Stop(0)
3134 stop 0
3135 Jan 29 05:11:20.200 INFO request completed[8810] Got stop action so:false kr:false
3136 , latency_us: 372, response_code: 204, uri: /stop/cid/0, method: POST, req_id: fb71371e-e4f6-484e-b786-58526024189d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
31372025-01-29T05:11:20.200ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31382025-01-29T05:11:20.201ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31392025-01-29T05:11:20.201ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
3140 Jan 29 05:11:20.201 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/0, method: GET, req_id: ff7dc147-7671-42f4-b455-24bcf6060db7, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:11:20.201153998Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
3141 
3142 [8810] Exited with: ExitStatus(unix_wait_status(9))
3143 [8810][0] reports Exit
3144 Jan 29 05:11:24.202 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: 248756ae-a990-4235-b2cc-955ccc2c3cd9, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
3145 001/300 Read block 3980 len 28672
3146 002/300 Read block 3275 len 28672
3147 003/300 Read block 3323 len 12288
3148 004/300 Read block 990 len 8192
3149 005/300 Write block 2261 len 16384 data: 4 4 4 3
3150 006/300 Flush
3151 007/300 Write block 2147 len 20480 data: 2 2 2 2 2
3152 008/300 Read block 2967 len 40960
3153 009/300 Read block 47 len 40960
3154 010/300 Read block 185 len 16384
3155 011/300 Write block 4188 len 16384 data: 2 2 2 2
3156 012/300 Read block 1255 len 32768
3157 013/300 Read block 3811 len 12288
3158 014/300 Write block 217 len 4096 data: 4
3159 015/300 Read block 3824 len 16384
3160 016/300 Read block 1779 len 40960
3161 017/300 Read block 265 len 20480
3162 018/300 Read block 1905 len 4096
3163 019/300 Read block 3763 len 28672
3164 020/300 Read block 4598 len 16384
3165 021/300 Read block 1771 len 36864
3166 022/300 Read block 2546 len 8192
3167 023/300 Read block 703 len 16384
3168 024/300 Write block 3429 len 8192 data: 2 2
3169 025/300 Read block 1766 len 8192
3170 026/300 Read block 155 len 32768
3171 027/300 Read block 98 len 16384
3172 028/300 Read block 161 len 8192
3173 029/300 Read block 3972 len 28672
3174 030/300 Write block 1122 len 36864 data: 2 2 2 2 2 2 2 2 2
3175 031/300 Write block 4547 len 24576 data: 2 2 2 2 2 2
3176 032/300 Read block 3262 len 28672
3177 033/300 Flush
3178 034/300 Read block 716 len 24576
3179 035/300 Read block 1720 len 24576
3180 036/300 Flush
3181 037/300 Flush
3182 038/300 Write block 4165 len 36864 data: 2 2 2 2 2 2 2 2 2
3183 039/300 Read block 273 len 12288
3184 040/300 Read block 3412 len 40960
3185 041/300 Read block 1353 len 24576
3186 042/300 Read block 380 len 28672
3187 043/300 Write block 2008 len 20480 data: 3 3 3 3 3
3188 044/300 Read block 3899 len 4096
3189 045/300 Read block 454 len 12288
3190 046/300 Read block 4620 len 32768
3191 047/300 Read block 3787 len 12288
3192 048/300 Read block 1975 len 28672
3193 049/300 Write block 1546 len 28672 data: 2 2 2 2 2 2 2
3194 050/300 Write block 4838 len 20480 data: 3 3 3 3 3
3195 051/300 Read block 3195 len 28672
3196 052/300 Read block 2377 len 24576
3197 053/300 Write block 1334 len 36864 data: 2 2 2 2 2 2 2 2 2
3198 054/300 Write block 3460 len 12288 data: 2 2 2
3199 055/300 Write block 1583 len 40960 data: 2 2 2 3 3 3 3 3 3 3
3200 056/300 Write block 3863 len 32768 data: 2 2 2 2 2 2 2 2
3201 057/300 Write block 542 len 36864 data: 2 2 2 2 2 2 2 2 2
3202 058/300 Write block 705 len 36864 data: 2 2 2 2 2 2 2 2 2
3203 059/300 Write block 2672 len 12288 data: 2 3 3
3204 060/300 Read block 3103 len 28672
3205 061/300 Write block 1527 len 32768 data: 2 2 2 2 2 3 3 2
3206 062/300 Write block 2103 len 32768 data: 3 3 3 3 3 3 2 2
3207 063/300 Read block 1908 len 24576
3208 064/300 Read block 3233 len 4096
3209 065/300 Read block 4105 len 40960
3210 066/300 Write block 2109 len 20480 data: 3 3 4 4 3
3211 067/300 Read block 1530 len 24576
3212 068/300 Flush
3213 069/300 Write block 237 len 20480 data: 3 3 3 3 4
3214 070/300 Read block 4028 len 16384
3215 071/300 Read block 2196 len 4096
3216 072/300 Write block 2138 len 28672 data: 3 3 3 3 2 2 2
3217 073/300 Flush
3218 074/300 Read block 4412 len 24576
3219 075/300 Read block 603 len 20480
3220 076/300 Write block 4379 len 36864 data: 2 2 2 2 2 2 2 2 2
3221 077/300 Flush
3222 078/300 Read block 996 len 8192
3223 079/300 Write block 2235 len 8192 data: 2 2
3224 080/300 Write block 1242 len 16384 data: 2 2 2 2
3225 081/300 Write block 3519 len 24576 data: 3 3 2 2 2 2
3226 082/300 Read block 238 len 24576
3227 083/300 Read block 1004 len 12288
3228 084/300 Read block 2587 len 12288
3229 085/300 Flush
3230 086/300 Flush
3231 087/300 Read block 1906 len 8192
3232 088/300 Read block 1952 len 16384
3233 089/300 Write block 2852 len 36864 data: 2 2 2 2 2 2 2 2 2
3234 090/300 Write block 2241 len 24576 data: 2 2 2 2 2 2
3235 091/300 Write block 2425 len 8192 data: 3 3
3236 092/300 Write block 2230 len 32768 data: 2 2 2 2 2 3 3 2
3237 093/300 Read block 1389 len 4096
3238 094/300 Read block 3669 len 24576
3239 095/300 Write block 1951 len 28672 data: 3 3 2 2 2 2 2
3240 096/300 Read block 4087 len 40960
3241 097/300 Read block 1340 len 8192
3242 098/300 Write block 4095 len 36864 data: 2 2 2 2 2 2 3 3 3
3243 099/300 Write block 4313 len 16384 data: 4 4 3 3
3244 100/300 Write block 4548 len 8192 data: 3 3
3245 101/300 Read block 1233 len 40960
3246 102/300 Read block 1492 len 24576
3247 103/300 Write block 2734 len 40960 data: 2 2 3 3 3 3 2 2 2 2
3248 104/300 Write block 4589 len 24576 data: 2 2 2 2 2 2
3249 105/300 Read block 1901 len 32768
3250 106/300 Write block 2222 len 8192 data: 3 2
3251 107/300 Write block 2755 len 32768 data: 3 3 2 2 2 2 2 2
3252 108/300 Flush
3253 109/300 Read block 1673 len 36864
3254 110/300 Write block 1773 len 32768 data: 2 2 2 2 2 2 2 2
3255 111/300 Flush
3256 112/300 Flush
3257 113/300 Read block 3944 len 32768
3258 114/300 Read block 1236 len 36864
3259 115/300 Write block 3246 len 36864 data: 2 2 2 2 2 2 2 2 2
3260 116/300 Write block 3875 len 16384 data: 3 3 3 3
3261 117/300 Write block 4374 len 32768 data: 3 2 2 2 2 3 3 3
3262 118/300 Write block 3231 len 12288 data: 2 2 2
3263 119/300 Write block 2499 len 20480 data: 2 2 2 2 2
3264 120/300 Write block 2618 len 36864 data: 2 2 2 2 2 2 2 2 2
3265 121/300 Read block 4170 len 16384
3266 122/300 Write block 3732 len 32768 data: 2 2 2 2 2 2 2 2
3267 123/300 Write block 4469 len 16384 data: 2 2 2 2
3268 124/300 Write block 2051 len 20480 data: 2 2 2 2 2
3269 125/300 Write block 4025 len 12288 data: 4 4 4
3270 126/300 Read block 1893 len 12288
3271 127/300 Write block 1538 len 20480 data: 2 2 2 2 2
3272 128/300 Read block 3187 len 24576
3273 129/300 Write block 1884 len 32768 data: 2 2 2 2 2 2 2 3
3274 130/300 Write block 468 len 12288 data: 3 2 2
3275 131/300 Write block 3655 len 16384 data: 2 2 2 3
3276 132/300 Write block 1983 len 36864 data: 3 3 3 3 2 2 2 2 2
3277 133/300 Read block 783 len 32768
3278 134/300 Write block 3820 len 32768 data: 2 2 2 2 2 2 2 2
3279 135/300 Write block 745 len 4096 data: 3
3280 136/300 Read block 2909 len 12288
3281 137/300 Read block 3986 len 40960
3282 138/300 Read block 803 len 8192
3283 139/300 Read block 4943 len 12288
3284 140/300 Write block 1764 len 12288 data: 2 2 2
3285 141/300 Read block 3020 len 28672
3286 142/300 Write block 3103 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3287 143/300 Read block 3397 len 36864
3288 144/300 Write block 978 len 8192 data: 2 2
3289 145/300 Write block 979 len 8192 data: 3 3
3290 146/300 Read block 3790 len 4096
3291 147/300 Write block 149 len 8192 data: 2 2
3292 148/300 Write block 2659 len 32768 data: 3 3 3 3 3 2 2 2
3293 149/300 Write block 4643 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3294 150/300 Write block 3721 len 12288 data: 2 2 2
3295 151/300 Write block 294 len 36864 data: 3 2 2 2 2 2 2 2 2
3296 152/300 Write block 4046 len 8192 data: 4 4
3297 153/300 Read block 4922 len 24576
3298 154/300 Write block 3270 len 16384 data: 2 2 2 2
3299 155/300 Write block 4094 len 12288 data: 2 3 3
3300 156/300 Read block 2955 len 20480
3301 157/300 Write block 255 len 24576 data: 2 2 2 2 2 2
3302 158/300 Read block 3510 len 36864
3303 159/300 Read block 4917 len 4096
3304 160/300 Write block 502 len 24576 data: 3 3 3 3 3 3
3305 161/300 Read block 2748 len 8192
3306 162/300 Flush
3307 163/300 Write block 3576 len 16384 data: 3 3 3 3
3308 164/300 Read block 159 len 12288
3309 165/300 Write block 707 len 12288 data: 3 3 3
3310 166/300 Read block 2977 len 28672
3311 167/300 Read block 3882 len 20480
3312 168/300 Write block 2837 len 16384 data: 2 2 2 2
3313 169/300 Write block 2400 len 4096 data: 2
3314 170/300 Write block 3830 len 24576 data: 2 2 2 2 2 3
3315 171/300 Write block 755 len 28672 data: 3 2 2 2 2 2 2
3316 172/300 Read block 2650 len 36864
3317 173/300 Write block 4580 len 24576 data: 2 2 2 2 2 2
3318 174/300 Write block 3521 len 8192 data: 3 3
3319 175/300 Read block 2033 len 4096
3320 176/300 Write block 647 len 28672 data: 2 2 2 2 2 2 2
3321 177/300 Read block 2229 len 32768
3322 178/300 Read block 3838 len 12288
3323 179/300 Read block 4182 len 4096
3324 180/300 Read block 3239 len 28672
3325 181/300 Read block 4801 len 28672
3326 182/300 Read block 33 len 32768
3327 183/300 Write block 2154 len 36864 data: 2 2 2 2 2 2 2 2 2
3328 184/300 Read block 3599 len 16384
3329 185/300 Read block 2486 len 16384
3330 186/300 Read block 4807 len 16384
3331 187/300 Write block 778 len 24576 data: 2 2 2 2 2 2
3332 188/300 Read block 2327 len 32768
3333 189/300 Read block 4670 len 40960
3334 190/300 Write block 4552 len 20480 data: 3 2 2 2 2
3335 191/300 Write block 4290 len 20480 data: 2 2 3 3 4
3336 192/300 Read block 2677 len 40960
3337 193/300 Write block 2066 len 12288 data: 2 2 2
3338 194/300 Read block 3418 len 4096
3339 195/300 Read block 1887 len 36864
3340 196/300 Read block 4864 len 32768
3341 197/300 Write block 3702 len 24576 data: 2 2 2 2 2 2
3342 198/300 Write block 2412 len 12288 data: 2 2 2
3343 199/300 Read block 3487 len 4096
3344 200/300 Write block 38 len 4096 data: 2
3345 201/300 Read block 3574 len 12288
3346 202/300 Read block 3228 len 8192
3347 203/300 Read block 3699 len 4096
3348 204/300 Write block 4169 len 8192 data: 3 3
3349 205/300 Read block 3945 len 8192
3350 206/300 Write block 2976 len 36864 data: 3 3 3 3 2 2 2 2 2
3351 207/300 Read block 634 len 12288
3352 208/300 Read block 4882 len 12288
3353 209/300 Read block 2650 len 32768
3354 210/300 Read block 778 len 20480
3355 211/300 Write block 4739 len 16384 data: 2 2 2 2
3356 212/300 Write block 3105 len 24576 data: 3 3 3 3 3 3
3357 213/300 Read block 385 len 16384
3358 214/300 Write block 3970 len 40960 data: 3 3 2 2 3 3 3 3 3 3
3359 215/300 Flush
3360 216/300 Write block 2912 len 16384 data: 2 2 2 2
3361 217/300 Write block 1416 len 4096 data: 2
3362 218/300 Read block 3972 len 24576
3363 219/300 Read block 1630 len 8192
3364 220/300 Read block 4537 len 16384
3365 221/300 Write block 4476 len 8192 data: 3 3
3366 222/300 Write block 1107 len 8192 data: 2 2
3367 223/300 Write block 1907 len 36864 data: 3 2 2 2 2 2 2 2 2
3368 224/300 Read block 2908 len 32768
3369 225/300 Write block 965 len 24576 data: 3 3 4 4 4 4
3370 226/300 Write block 3671 len 16384 data: 2 2 2 2
3371 227/300 Read block 1442 len 36864
3372 228/300 Read block 768 len 24576
3373 229/300 Read block 1300 len 16384
3374 230/300 Write block 1559 len 20480 data: 3 3 2 2 2
3375 231/300 Write block 1703 len 8192 data: 2 3
3376 232/300 Read block 2671 len 32768
3377 233/300 Write block 4274 len 16384 data: 2 2 2 3
3378 234/300 Flush
3379 235/300 Write block 1442 len 32768 data: 2 3 3 3 3 3 4 4
3380 236/300 Write block 656 len 36864 data: 2 2 2 2 2 2 2 2 2
3381 237/300 Read block 4668 len 16384
3382 238/300 Write block 3649 len 24576 data: 2 2 2 2 2 2
3383 239/300 Write block 1441 len 12288 data: 2 3 4
3384 240/300 Write block 1420 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3385 241/300 Read block 1706 len 32768
3386 242/300 Read block 2914 len 4096
3387 243/300 Read block 4124 len 28672
3388 244/300 Flush
3389 245/300 Write block 1109 len 16384 data: 2 2 2 2
3390 246/300 Flush
3391 247/300 Read block 4016 len 36864
3392 248/300 Write block 2746 len 32768 data: 2 2 3 3 3 3 3 3
3393 249/300 Write block 4453 len 12288 data: 3 3 3
3394 250/300 Read block 1433 len 36864
3395 251/300 Read block 1147 len 20480
3396 252/300 Read block 2366 len 16384
3397 253/300 Write block 188 len 36864 data: 3 3 3 3 4 4 4 2 2
3398 254/300 Read block 3768 len 28672
3399 255/300 Write block 4357 len 32768 data: 2 2 2 2 2 3 3 3
3400 256/300 Write block 4275 len 40960 data: 3 3 4 3 2 3 3 3 3 3
3401 257/300 Read block 1423 len 40960
3402 258/300 Read block 4929 len 40960
3403 259/300 Read block 4931 len 24576
3404 260/300 Flush
3405 261/300 Write block 330 len 16384 data: 2 2 2 2
3406 262/300 Read block 4035 len 12288
3407 263/300 Read block 234 len 36864
3408 264/300 Read block 2537 len 4096
3409 265/300 Read block 1530 len 8192
3410 266/300 Read block 1771 len 4096
3411 267/300 Read block 3515 len 36864
3412 268/300 Read block 702 len 20480
3413 269/300 Read block 888 len 20480
3414 270/300 Read block 2231 len 8192
3415 271/300 Write block 1100 len 4096 data: 2
3416 272/300 Write block 752 len 24576 data: 3 3 3 4 3 3
3417 273/300 Write block 2107 len 8192 data: 4 4
3418 274/300 Read block 4938 len 20480
3419 275/300 Write block 82 len 28672 data: 3 3 3 3 3 3 3
3420 276/300 Write block 4174 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3421 277/300 Read block 2617 len 8192
3422 278/300 Write block 996 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3423 279/300 Read block 3452 len 8192
3424 280/300 Read block 2665 len 28672
3425 281/300 Write block 1009 len 20480 data: 2 2 2 2 2
3426 282/300 Read block 2858 len 20480
3427 283/300 Write block 3514 len 4096 data: 3
3428 284/300 Read block 3030 len 24576
3429 285/300 Read block 1332 len 24576
3430 286/300 Read block 1443 len 24576
3431 287/300 Write block 2607 len 20480 data: 4 4 4 3 3
3432 288/300 Read block 2759 len 20480
3433 289/300 Read block 4962 len 28672
3434 290/300 Write block 1620 len 36864 data: 2 2 3 3 3 3 3 3 3
3435 291/300 Flush
3436 292/300 Write block 4041 len 32768 data: 2 3 3 3 3 5 5 3
3437 293/300 Write block 2492 len 4096 data: 3
3438 294/300 Read block 3260 len 32768
3439 295/300 Read block 186 len 12288
3440 296/300 Read block 4011 len 40960
3441 297/300 Read block 4346 len 24576
3442 298/300 Read block 1863 len 20480
3443 299/300 Flush
3444 300/300 Read block 760 len 8192
3445 301/300 Read block 4392 len 16384
3446 Main task has work to do, go find it
3447 got dsc Start(0)
3448 start 0
3449 Jan 29 05:11:24.441 INFO request completed, latency_us: 187, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7e54eb04-2b36-4d63-b4ec-716636adfd32, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
3450 Replay: started 0, returned:Ok(())
3451 ----------------------------------------------------------------
3452 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3453 States: Offline Active Active
3454 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3455 1906 Acked Read 7 Sent Done Done false
3456 1907 Acked Read 7 Sent Done Done false
3457 1908 Acked Read 3 Sent Done Done false
3458 1909 Acked Read 2 Sent Done Done false
3459 1910 Acked Write 4 Sent Done Done false
3460 1911 Acked Flush 0 Sent Done Done false
3461 1912 Acked Write 5 Sent Done Done false
3462 1913 Acked Read 10 Sent Done Done false
3463 1914 Acked Read 10 Sent Done Done false
3464 1915 Acked Read 4 Sent Done Done false
3465 1916 Acked Write 4 Sent Done Done false
3466 1917 Acked Read 8 Sent Done Done false
3467 1918 Acked Read 3 Sent Done Done false
3468 1919 Acked Write 1 Sent Done Done false
3469 1920 Acked Read 4 Sent Done Done false
3470 1921 Acked Read 10 Sent Done Done false
3471 1922 Acked Read 5 Sent Done Done false
3472 1923 Acked Read 1 Sent Done Done false
3473 1924 Acked Read 7 Sent Done Done false
3474 1925 Acked Read 4 Sent Done Done false
3475 1926 Acked Read 9 Sent Done Done false
3476 1927 Acked Read 2 Sent Done Done false
3477 1928 Acked Read 4 Sent Done Done false
3478 1929 Acked Write 2 Sent Done Done false
3479 1930 Acked Read 2 Sent Done Done false
3480 1931 Acked Read 8 Sent Done Done false
3481 1932 Acked Read 4 Sent Done Done false
3482 1933 Acked Read 2 Sent Done Done false
3483 1934 Acked Read 7 Sent Done Done false
3484 1935 Acked Write 9 Sent Done Done false
3485 1936 Acked Write 6 Sent Done Done false
3486 1937 Acked Read 7 Sent Done Done false
3487 1938 Acked Flush 0 Sent Done Done false
3488 1939 Acked Read 6 Sent Done Done false
3489 1940 Acked Read 6 Sent Done Done false
3490 1941 Acked Flush 0 Sent Done Done false
3491 1942 Acked Flush 0 Sent Done Done false
3492 1943 Acked Write 9 Sent Done Done false
3493 1944 Acked Read 3 Sent Done Done false
3494 1945 Acked Read 10 Sent Done Done false
3495 1946 Acked Read 6 Sent Done Done false
3496 1947 Acked Read 7 Sent Done Done false
3497 1948 Acked Write 5 Sent Done Done false
3498 1949 Acked Read 1 Sent Done Done false
3499 1950 Acked Read 3 Sent Done Done false
3500 1951 Acked Read 8 Sent Done Done false
3501 1952 Acked Read 3 Sent Done Done false
3502 1953 Acked Read 7 Sent Done Done false
3503 1954 Acked Write 7 Sent Done Done false
3504 1955 Acked Write 5 Sent Done Done false
3505 1956 Acked Read 7 Sent Done Done false
3506 1957 Acked Read 6 Sent Done Done false
3507 1958 Acked Write 9 Sent Done Done false
3508 1959 Acked Write 3 Sent Done Done false
3509 1960 Acked Write 10 Sent Done Done false
3510 1961 Acked Write 8 Sent Done Done false
3511 1962 Acked Write 9 Sent Done Done false
3512 1963 Acked Write 9 Sent Done Done false
3513 1964 Acked Write 3 Sent Done Done false
3514 1965 Acked Read 7 Sent Done Done false
3515 1966 Acked Write 8 Sent Done Done false
3516 1967 Acked Write 8 Sent Done Done false
3517 1968 Acked Read 6 Sent Done Done false
3518 1969 Acked Read 1 Sent Done Done false
3519 1970 Acked Read 10 Sent Done Done false
3520 1971 Acked Write 5 Sent Done Done false
3521 1972 Acked Read 6 Sent Done Done false
3522 1973 Acked Flush 0 Sent Done Done false
3523 1974 Acked Write 5 Sent Done Done false
3524 1975 Acked Read 4 Sent Done Done false
3525 1976 Acked Read 1 Sent Done Done false
3526 1977 Acked Write 7 Sent Done Done false
3527 1978 Acked Flush 0 Sent Done Done false
3528 1979 Acked Read 6 Sent Done Done false
3529 1980 Acked Read 5 Sent Done Done false
3530 1981 Acked Write 9 Sent Done Done false
3531 1982 Acked Flush 0 Sent Done Done false
3532 1983 Acked Read 2 Sent Done Done false
3533 1984 Acked Write 2 Sent Done Done false
3534 1985 Acked Write 4 Sent Done Done false
3535 1986 Acked Write 6 Sent Done Done false
3536 1987 Acked Read 6 Sent Done Done false
3537 1988 Acked Read 3 Sent Done Done false
3538 1989 Acked Read 3 Sent Done Done false
3539 1990 Acked Flush 0 Sent Done Done false
3540 1991 Acked Flush 0 Sent Done Done false
3541 1992 Acked Read 2 Sent Done Done false
3542 1993 Acked Read 4 Sent Done Done false
3543 1994 Acked Write 9 Sent Done Done false
3544 1995 Acked Write 6 Sent Done Done false
3545 1996 Acked Write 2 Sent Done Done false
3546 1997 Acked Write 8 Sent Done Done false
3547 1998 Acked Read 1 Sent Done Done false
3548 1999 Acked Read 6 Sent Done Done false
3549 2000 Acked Write 7 Sent Done Done false
3550 2001 Acked Read 10 Sent Done Done false
3551 2002 Acked Read 2 Sent Done Done false
3552 2003 Acked Write 9 Sent Done Done false
3553 2004 Acked Write 4 Sent Done Done false
3554 2005 Acked Write 2 Sent Done Done false
3555 2006 Acked Read 10 Sent Done Done false
3556 2007 Acked Read 6 Sent Done Done false
3557 2008 Acked Write 10 Sent Done Done false
3558 2009 Acked Write 6 Sent Done Done false
3559 2010 Acked Read 8 Sent Done Done false
3560 2011 Acked Write 2 Sent Done Done false
3561 2012 Acked Write 8 Sent Done Done false
3562 2013 Acked Flush 0 Sent Done Done false
3563 2014 Acked Read 9 Sent Done Done false
3564 2015 Acked Write 8 Sent Done Done false
3565 2016 Acked Flush 0 Sent Done Done false
3566 2017 Acked Flush 0 Sent Done Done false
3567 2018 Acked Read 8 Sent Done Done false
3568 2019 Acked Read 9 Sent Done Done false
3569 2020 Acked Write 9 Sent Done Done false
3570 2021 Acked Write 4 Sent Done Done false
3571 2022 Acked Write 8 Sent Done Done false
3572 2023 Acked Write 3 Sent Done Done false
3573 2024 Acked Write 5 Sent Done Done false
3574 2025 Acked Write 9 Sent Done Done false
3575 2026 Acked Read 4 Sent Done Done false
3576 2027 Acked Write 8 Sent Done Done false
3577 2028 Acked Write 4 Sent Done Done false
3578 2029 Acked Write 5 Sent Done Done false
3579 2030 Acked Write 3 Sent Done Done false
3580 2031 Acked Read 3 Sent Done Done false
3581 2032 Acked Write 5 Sent Done Done false
3582 2033 Acked Read 6 Sent Done Done false
3583 2034 Acked Write 8 Sent Done Done false
3584 2035 Acked Write 3 Sent Done Done false
3585 2036 Acked Write 4 Sent Done Done false
3586 2037 Acked Write 9 Sent Done Done false
3587 2038 Acked Read 8 Sent Done Done false
3588 2039 Acked Write 8 Sent Done Done false
3589 2040 Acked Write 1 Sent Done Done false
3590 2041 Acked Read 3 Sent Done Done false
3591 2042 Acked Read 10 Sent Done Done false
3592 2043 Acked Read 2 Sent Done Done false
3593 2044 Acked Read 3 Sent Done Done false
3594 2045 Acked Write 3 Sent Done Done false
3595 2046 Acked Read 7 Sent Done Done false
3596 2047 Acked Write 10 Sent Done Done false
3597 2048 Acked Read 9 Sent Done Done false
3598 2049 Acked Write 2 Sent Done Done false
3599 2050 Acked Write 2 Sent Done Done false
3600 2051 Acked Read 1 Sent Done Done false
3601 2052 Acked Write 2 Sent Done Done false
3602 2053 Acked Write 8 Sent Done Done false
3603 2054 Acked Write 10 Sent Done Done false
3604 2055 Acked Write 3 Sent Done Done false
3605 2056 Acked Write 9 Sent Done Done false
3606 2057 Acked Write 2 Sent Done Done false
3607 2058 Acked Read 6 Sent Done Done false
3608 2059 Acked Write 4 Sent Done Done false
3609 2060 Acked Write 3 Sent Done Done false
3610 2061 Acked Read 5 Sent Done Done false
3611 2062 Acked Write 6 Sent Done Done false
3612 2063 Acked Read 9 Sent Done Done false
3613 2064 Acked Read 1 Sent Done Done false
3614 2065 Acked Write 6 Sent Done Done false
3615 2066 Acked Read 2 Sent Done Done false
3616 2067 Acked Flush 0 Sent Done Done false
3617 2068 Acked Write 4 Sent Done Done false
3618 2069 Acked Read 3 Sent Done Done false
3619 2070 Acked Write 3 Sent Done Done false
3620 2071 Acked Read 7 Sent Done Done false
3621 2072 Acked Read 5 Sent Done Done false
3622 2073 Acked Write 4 Sent Done Done false
3623 2074 Acked Write 1 Sent Done Done false
3624 2075 Acked Write 6 Sent Done Done false
3625 2076 Acked Write 7 Sent Done Done false
3626 2077 Acked Read 9 Sent Done Done false
3627 2078 Acked Write 6 Sent Done Done false
3628 2079 Acked Write 2 Sent Done Done false
3629 2080 Acked Read 1 Sent Done Done false
3630 2081 Acked Write 7 Sent Done Done false
3631 2082 Acked Read 8 Sent Done Done false
3632 2083 Acked Read 3 Sent Done Done false
3633 2084 Acked Read 1 Sent Done Done false
3634 2085 Acked Read 7 Sent Done Done false
3635 2086 Acked Read 7 Sent Done Done false
3636 2087 Acked Read 8 Sent Done Done false
3637 2088 Acked Write 9 Sent Done Done false
3638 2089 Acked Read 4 Sent Done Done false
3639 2090 Acked Read 4 Sent Done Done false
3640 2091 Acked Read 4 Sent Done Done false
3641 2092 Acked Write 6 Sent Done Done false
3642 2093 Acked Read 8 Sent Done Done false
3643 2094 Acked Read 10 Sent Done Done false
3644 2095 Acked Write 5 Sent Done Done false
3645 2096 Acked Write 5 Sent Done Done false
3646 2097 Acked Read 10 Sent Done Done false
3647 2098 Acked Write 3 Sent Done Done false
3648 2099 Acked Read 1 Sent Done Done false
3649 2100 Acked Read 9 Sent Done Done false
3650 2101 Acked Read 8 Sent Done Done false
3651 2102 Acked Write 6 Sent Done Done false
3652 2103 Acked Write 3 Sent Done Done false
3653 2104 Acked Read 1 Sent Done Done false
3654 2105 Acked Write 1 Sent Done Done false
3655 2106 Acked Read 3 Sent Done Done false
3656 2107 Acked Read 2 Sent Done Done false
3657 2108 Acked Read 1 Sent Done Done false
3658 2109 Acked Write 2 Sent Done Done false
3659 2110 Acked Read 2 Sent Done Done false
3660 2111 Acked Write 9 Sent Done Done false
3661 2112 Acked Read 3 Sent Done Done false
3662 2113 Acked Read 3 Sent Done Done false
3663 2114 Acked Read 8 Sent Done Done false
3664 2115 Acked Read 5 Sent Done Done false
3665 2116 Acked Write 4 Sent Done Done false
3666 2117 Acked Write 6 Sent Done Done false
3667 2118 Acked Read 4 Sent Done Done false
3668 2119 Acked Write 10 Sent Done Done false
3669 2120 Acked Flush 0 Sent Done Done false
3670 2121 Acked Write 4 Sent Done Done false
3671 2122 Acked Write 1 Sent Done Done false
3672 2123 Acked Read 6 Sent Done Done false
3673 2124 Acked Read 2 Sent Done Done false
3674 2125 Acked Read 4 Sent Done Done false
3675 2126 Acked Write 2 Sent Done Done false
3676 2127 Acked Write 2 Sent Done Done false
3677 2128 Acked Write 9 Sent Done Done false
3678 2129 Acked Read 8 Sent Done Done false
3679 2130 Acked Write 6 Sent Done Done false
3680 2131 Acked Write 4 Sent Done Done false
3681 2132 Acked Read 9 Sent Done Done false
3682 2133 Acked Read 6 Sent Done Done false
3683 2134 Acked Read 4 Sent Done Done false
3684 2135 Acked Write 5 Sent Done Done false
3685 2136 Acked Write 2 Sent Done Done false
3686 2137 Acked Read 8 Sent Done Done false
3687 2138 Acked Write 4 Sent Done Done false
3688 2139 Acked Flush 0 Sent Done Done false
3689 2140 Acked Write 8 Sent Done Done false
3690 2141 Acked Write 9 Sent Done Done false
3691 2142 Acked Read 4 Sent Done Done false
3692 2143 Acked Write 6 Sent Done Done false
3693 2144 Acked Write 3 Sent Done Done false
3694 2145 Acked Write 10 Sent Done Done false
3695 2146 Acked Read 8 Sent Done Done false
3696 2147 Acked Read 1 Sent Done Done false
3697 2148 Acked Read 7 Sent Done Done false
3698 2149 Acked Flush 0 Sent Done Done false
3699 2150 Acked Write 4 Sent Done Done false
3700 2151 Acked Flush 0 Sent Done Done false
3701 2152 Acked Read 9 Sent Done Done false
3702 2153 Acked Write 8 Sent Done Done false
3703 2154 Acked Write 3 Sent Done Done false
3704 2155 Acked Read 9 Sent Done Done false
3705 2156 Acked Read 5 Sent Done Done false
3706 2157 Acked Read 4 Sent Done Done false
3707 2158 Acked Write 9 Sent Done Done false
3708 2159 Acked Read 7 Sent Done Done false
3709 2160 Acked Write 8 Sent Done Done false
3710 2161 Acked Write 10 Sent Done Done false
3711 2162 Acked Read 10 Sent Done Done false
3712 2163 Acked Read 10 Sent Done Done false
3713 2164 Acked Read 6 Sent Done Done false
3714 2165 Acked Flush 0 Sent Done Done false
3715 2166 Acked Write 4 Sent Done Done false
3716 2167 Acked Read 3 Sent Done Done false
3717 2168 Acked Read 9 Sent Done Done false
3718 2169 Acked Read 1 Sent Done Done false
3719 2170 Acked Read 2 Sent Done Done false
3720 2171 Acked Read 1 Sent Done Done false
3721 2172 Acked Read 9 Sent Done Done false
3722 2173 Acked Read 5 Sent Done Done false
3723 2174 Acked Read 5 Sent Done Done false
3724 2175 Acked Read 2 Sent Done Done false
3725 2176 Acked Write 1 Sent Done Done false
3726 2177 Acked Write 6 Sent Done Done false
3727 2178 Acked Write 2 Sent Done Done false
3728 2179 Acked Read 5 Sent Done Done false
3729 2180 Acked Write 7 Sent Done Done false
3730 2181 Acked Write 10 Sent Done Done false
3731 2182 Acked Read 2 Sent Done Done false
3732 2183 Acked Write 10 Sent Done Done false
3733 2184 Acked Read 2 Sent Done Done false
3734 2185 Acked Read 7 Sent Done Done false
3735 2186 Acked Write 5 Sent Done Done false
3736 2187 Acked Read 5 Sent Done Done false
3737 2188 Acked Write 1 Sent Done Done false
3738 2189 Acked Read 6 Sent Done Done false
3739 2190 Acked Read 6 Sent Done Done false
3740 2191 Acked Read 6 Sent Done Done false
3741 2192 Acked Write 5 Sent Done Done false
3742 2193 Acked Read 5 Sent Done Done false
3743 2194 Acked Read 7 Sent Done Done false
3744 2195 Acked Write 9 Sent Done Done false
3745 2196 Acked Flush 0 Sent Done Done false
3746 2197 Acked Write 8 Sent Done Done false
3747 2198 Acked Write 1 Sent Done Done false
3748 2199 Acked Read 8 Sent Done Done false
3749 2200 Acked Read 3 Sent Done Done false
3750 2201 Acked Read 10 Sent Done Done false
3751 2202 Acked Read 6 Sent Done Done false
3752 2203 Acked Read 5 Sent Done Done false
3753 2204 Acked Flush 0 Sent Done Done false
3754 2205 Acked Read 2 Sent Done Done false
3755 2206 Acked Read 4 Sent Done Done false
3756 STATES DS:0 DS:1 DS:2 TOTAL
3757 Sent 301 0 0 301
3758 Done 0 301 301 602
3759 Skipped 0 0 0 0
3760 Error 0 0 0 0
3761 Last Flush: Some(JobId(1905)) Some(JobId(2204)) Some(JobId(2204))
3762 Downstairs last five completed: 1905 1904 1903 1902 1901
3763 Upstairs last five completed: 2206 2205 2204 2203 2202
3764 CLIENT: Up:0 ds:301 act:2
37652025-01-29T05:11:24.976ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37662025-01-29T05:11:26.206ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
3767 [8810] got start action
3768 [8810] I am going to restart
3769 Starting downstairs at port 8810
3770 Make output file at "/tmp/dsc/downstairs-8810.txt"
3771 [8810][0] reports Starting
3772 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1206)
3773 [8810][0] reports Running
37742025-01-29T05:11:26.979ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37752025-01-29T05:11:27.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
3776 ----------------------------------------------------------------
3777 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3778 States: Offline Active Active
3779 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3780 1906 Acked Read 7 Sent Done Done false
3781 1907 Acked Read 7 Sent Done Done false
3782 1908 Acked Read 3 Sent Done Done false
3783 1909 Acked Read 2 Sent Done Done false
3784 1910 Acked Write 4 Sent Done Done false
3785 1911 Acked Flush 0 Sent Done Done false
3786 1912 Acked Write 5 Sent Done Done false
3787 1913 Acked Read 10 Sent Done Done false
3788 1914 Acked Read 10 Sent Done Done false
3789 1915 Acked Read 4 Sent Done Done false
3790 1916 Acked Write 4 Sent Done Done false
3791 1917 Acked Read 8 Sent Done Done false
3792 1918 Acked Read 3 Sent Done Done false
3793 1919 Acked Write 1 Sent Done Done false
3794 1920 Acked Read 4 Sent Done Done false
3795 1921 Acked Read 10 Sent Done Done false
3796 1922 Acked Read 5 Sent Done Done false
3797 1923 Acked Read 1 Sent Done Done false
3798 1924 Acked Read 7 Sent Done Done false
3799 1925 Acked Read 4 Sent Done Done false
3800 1926 Acked Read 9 Sent Done Done false
3801 1927 Acked Read 2 Sent Done Done false
3802 1928 Acked Read 4 Sent Done Done false
3803 1929 Acked Write 2 Sent Done Done false
3804 1930 Acked Read 2 Sent Done Done false
3805 1931 Acked Read 8 Sent Done Done false
3806 1932 Acked Read 4 Sent Done Done false
3807 1933 Acked Read 2 Sent Done Done false
3808 1934 Acked Read 7 Sent Done Done false
3809 1935 Acked Write 9 Sent Done Done false
3810 1936 Acked Write 6 Sent Done Done false
3811 1937 Acked Read 7 Sent Done Done false
3812 1938 Acked Flush 0 Sent Done Done false
3813 1939 Acked Read 6 Sent Done Done false
3814 1940 Acked Read 6 Sent Done Done false
3815 1941 Acked Flush 0 Sent Done Done false
3816 1942 Acked Flush 0 Sent Done Done false
3817 1943 Acked Write 9 Sent Done Done false
3818 1944 Acked Read 3 Sent Done Done false
3819 1945 Acked Read 10 Sent Done Done false
3820 1946 Acked Read 6 Sent Done Done false
3821 1947 Acked Read 7 Sent Done Done false
3822 1948 Acked Write 5 Sent Done Done false
3823 1949 Acked Read 1 Sent Done Done false
3824 1950 Acked Read 3 Sent Done Done false
3825 1951 Acked Read 8 Sent Done Done false
3826 1952 Acked Read 3 Sent Done Done false
3827 1953 Acked Read 7 Sent Done Done false
3828 1954 Acked Write 7 Sent Done Done false
3829 1955 Acked Write 5 Sent Done Done false
3830 1956 Acked Read 7 Sent Done Done false
3831 1957 Acked Read 6 Sent Done Done false
3832 1958 Acked Write 9 Sent Done Done false
3833 1959 Acked Write 3 Sent Done Done false
3834 1960 Acked Write 10 Sent Done Done false
3835 1961 Acked Write 8 Sent Done Done false
3836 1962 Acked Write 9 Sent Done Done false
3837 1963 Acked Write 9 Sent Done Done false
3838 1964 Acked Write 3 Sent Done Done false
3839 1965 Acked Read 7 Sent Done Done false
3840 1966 Acked Write 8 Sent Done Done false
3841 1967 Acked Write 8 Sent Done Done false
3842 1968 Acked Read 6 Sent Done Done false
3843 1969 Acked Read 1 Sent Done Done false
3844 1970 Acked Read 10 Sent Done Done false
3845 1971 Acked Write 5 Sent Done Done false
3846 1972 Acked Read 6 Sent Done Done false
3847 1973 Acked Flush 0 Sent Done Done false
3848 1974 Acked Write 5 Sent Done Done false
3849 1975 Acked Read 4 Sent Done Done false
3850 1976 Acked Read 1 Sent Done Done false
3851 1977 Acked Write 7 Sent Done Done false
3852 1978 Acked Flush 0 Sent Done Done false
3853 1979 Acked Read 6 Sent Done Done false
3854 1980 Acked Read 5 Sent Done Done false
3855 1981 Acked Write 9 Sent Done Done false
3856 1982 Acked Flush 0 Sent Done Done false
3857 1983 Acked Read 2 Sent Done Done false
3858 1984 Acked Write 2 Sent Done Done false
3859 1985 Acked Write 4 Sent Done Done false
3860 1986 Acked Write 6 Sent Done Done false
3861 1987 Acked Read 6 Sent Done Done false
3862 1988 Acked Read 3 Sent Done Done false
3863 1989 Acked Read 3 Sent Done Done false
3864 1990 Acked Flush 0 Sent Done Done false
3865 1991 Acked Flush 0 Sent Done Done false
3866 1992 Acked Read 2 Sent Done Done false
3867 1993 Acked Read 4 Sent Done Done false
3868 1994 Acked Write 9 Sent Done Done false
3869 1995 Acked Write 6 Sent Done Done false
3870 1996 Acked Write 2 Sent Done Done false
3871 1997 Acked Write 8 Sent Done Done false
3872 1998 Acked Read 1 Sent Done Done false
3873 1999 Acked Read 6 Sent Done Done false
3874 2000 Acked Write 7 Sent Done Done false
3875 2001 Acked Read 10 Sent Done Done false
3876 2002 Acked Read 2 Sent Done Done false
3877 2003 Acked Write 9 Sent Done Done false
3878 2004 Acked Write 4 Sent Done Done false
3879 2005 Acked Write 2 Sent Done Done false
3880 2006 Acked Read 10 Sent Done Done false
3881 2007 Acked Read 6 Sent Done Done false
3882 2008 Acked Write 10 Sent Done Done false
3883 2009 Acked Write 6 Sent Done Done false
3884 2010 Acked Read 8 Sent Done Done false
3885 2011 Acked Write 2 Sent Done Done false
3886 2012 Acked Write 8 Sent Done Done false
3887 2013 Acked Flush 0 Sent Done Done false
3888 2014 Acked Read 9 Sent Done Done false
3889 2015 Acked Write 8 Sent Done Done false
3890 2016 Acked Flush 0 Sent Done Done false
3891 2017 Acked Flush 0 Sent Done Done false
3892 2018 Acked Read 8 Sent Done Done false
3893 2019 Acked Read 9 Sent Done Done false
3894 2020 Acked Write 9 Sent Done Done false
3895 2021 Acked Write 4 Sent Done Done false
3896 2022 Acked Write 8 Sent Done Done false
3897 2023 Acked Write 3 Sent Done Done false
3898 2024 Acked Write 5 Sent Done Done false
3899 2025 Acked Write 9 Sent Done Done false
3900 2026 Acked Read 4 Sent Done Done false
3901 2027 Acked Write 8 Sent Done Done false
3902 2028 Acked Write 4 Sent Done Done false
3903 2029 Acked Write 5 Sent Done Done false
3904 2030 Acked Write 3 Sent Done Done false
3905 2031 Acked Read 3 Sent Done Done false
3906 2032 Acked Write 5 Sent Done Done false
3907 2033 Acked Read 6 Sent Done Done false
3908 2034 Acked Write 8 Sent Done Done false
3909 2035 Acked Write 3 Sent Done Done false
3910 2036 Acked Write 4 Sent Done Done false
3911 2037 Acked Write 9 Sent Done Done false
3912 2038 Acked Read 8 Sent Done Done false
3913 2039 Acked Write 8 Sent Done Done false
3914 2040 Acked Write 1 Sent Done Done false
3915 2041 Acked Read 3 Sent Done Done false
3916 2042 Acked Read 10 Sent Done Done false
3917 2043 Acked Read 2 Sent Done Done false
3918 2044 Acked Read 3 Sent Done Done false
3919 2045 Acked Write 3 Sent Done Done false
3920 2046 Acked Read 7 Sent Done Done false
3921 2047 Acked Write 10 Sent Done Done false
3922 2048 Acked Read 9 Sent Done Done false
3923 2049 Acked Write 2 Sent Done Done false
3924 2050 Acked Write 2 Sent Done Done false
3925 2051 Acked Read 1 Sent Done Done false
3926 2052 Acked Write 2 Sent Done Done false
3927 2053 Acked Write 8 Sent Done Done false
3928 2054 Acked Write 10 Sent Done Done false
3929 2055 Acked Write 3 Sent Done Done false
3930 2056 Acked Write 9 Sent Done Done false
3931 2057 Acked Write 2 Sent Done Done false
3932 2058 Acked Read 6 Sent Done Done false
3933 2059 Acked Write 4 Sent Done Done false
3934 2060 Acked Write 3 Sent Done Done false
3935 2061 Acked Read 5 Sent Done Done false
3936 2062 Acked Write 6 Sent Done Done false
3937 2063 Acked Read 9 Sent Done Done false
3938 2064 Acked Read 1 Sent Done Done false
3939 2065 Acked Write 6 Sent Done Done false
3940 2066 Acked Read 2 Sent Done Done false
3941 2067 Acked Flush 0 Sent Done Done false
3942 2068 Acked Write 4 Sent Done Done false
3943 2069 Acked Read 3 Sent Done Done false
3944 2070 Acked Write 3 Sent Done Done false
3945 2071 Acked Read 7 Sent Done Done false
3946 2072 Acked Read 5 Sent Done Done false
3947 2073 Acked Write 4 Sent Done Done false
3948 2074 Acked Write 1 Sent Done Done false
3949 2075 Acked Write 6 Sent Done Done false
3950 2076 Acked Write 7 Sent Done Done false
3951 2077 Acked Read 9 Sent Done Done false
3952 2078 Acked Write 6 Sent Done Done false
3953 2079 Acked Write 2 Sent Done Done false
3954 2080 Acked Read 1 Sent Done Done false
3955 2081 Acked Write 7 Sent Done Done false
3956 2082 Acked Read 8 Sent Done Done false
3957 2083 Acked Read 3 Sent Done Done false
3958 2084 Acked Read 1 Sent Done Done false
3959 2085 Acked Read 7 Sent Done Done false
3960 2086 Acked Read 7 Sent Done Done false
3961 2087 Acked Read 8 Sent Done Done false
3962 2088 Acked Write 9 Sent Done Done false
3963 2089 Acked Read 4 Sent Done Done false
3964 2090 Acked Read 4 Sent Done Done false
3965 2091 Acked Read 4 Sent Done Done false
3966 2092 Acked Write 6 Sent Done Done false
3967 2093 Acked Read 8 Sent Done Done false
3968 2094 Acked Read 10 Sent Done Done false
3969 2095 Acked Write 5 Sent Done Done false
3970 2096 Acked Write 5 Sent Done Done false
3971 2097 Acked Read 10 Sent Done Done false
3972 2098 Acked Write 3 Sent Done Done false
3973 2099 Acked Read 1 Sent Done Done false
3974 2100 Acked Read 9 Sent Done Done false
3975 2101 Acked Read 8 Sent Done Done false
3976 2102 Acked Write 6 Sent Done Done false
3977 2103 Acked Write 3 Sent Done Done false
3978 2104 Acked Read 1 Sent Done Done false
3979 2105 Acked Write 1 Sent Done Done false
3980 2106 Acked Read 3 Sent Done Done false
3981 2107 Acked Read 2 Sent Done Done false
3982 2108 Acked Read 1 Sent Done Done false
3983 2109 Acked Write 2 Sent Done Done false
3984 2110 Acked Read 2 Sent Done Done false
3985 2111 Acked Write 9 Sent Done Done false
3986 2112 Acked Read 3 Sent Done Done false
3987 2113 Acked Read 3 Sent Done Done false
3988 2114 Acked Read 8 Sent Done Done false
3989 2115 Acked Read 5 Sent Done Done false
3990 2116 Acked Write 4 Sent Done Done false
3991 2117 Acked Write 6 Sent Done Done false
3992 2118 Acked Read 4 Sent Done Done false
3993 2119 Acked Write 10 Sent Done Done false
3994 2120 Acked Flush 0 Sent Done Done false
3995 2121 Acked Write 4 Sent Done Done false
3996 2122 Acked Write 1 Sent Done Done false
3997 2123 Acked Read 6 Sent Done Done false
3998 2124 Acked Read 2 Sent Done Done false
3999 2125 Acked Read 4 Sent Done Done false
4000 2126 Acked Write 2 Sent Done Done false
4001 2127 Acked Write 2 Sent Done Done false
4002 2128 Acked Write 9 Sent Done Done false
4003 2129 Acked Read 8 Sent Done Done false
4004 2130 Acked Write 6 Sent Done Done false
4005 2131 Acked Write 4 Sent Done Done false
4006 2132 Acked Read 9 Sent Done Done false
4007 2133 Acked Read 6 Sent Done Done false
4008 2134 Acked Read 4 Sent Done Done false
4009 2135 Acked Write 5 Sent Done Done false
4010 2136 Acked Write 2 Sent Done Done false
4011 2137 Acked Read 8 Sent Done Done false
4012 2138 Acked Write 4 Sent Done Done false
4013 2139 Acked Flush 0 Sent Done Done false
4014 2140 Acked Write 8 Sent Done Done false
4015 2141 Acked Write 9 Sent Done Done false
4016 2142 Acked Read 4 Sent Done Done false
4017 2143 Acked Write 6 Sent Done Done false
4018 2144 Acked Write 3 Sent Done Done false
4019 2145 Acked Write 10 Sent Done Done false
4020 2146 Acked Read 8 Sent Done Done false
4021 2147 Acked Read 1 Sent Done Done false
4022 2148 Acked Read 7 Sent Done Done false
4023 2149 Acked Flush 0 Sent Done Done false
4024 2150 Acked Write 4 Sent Done Done false
4025 2151 Acked Flush 0 Sent Done Done false
4026 2152 Acked Read 9 Sent Done Done false
4027 2153 Acked Write 8 Sent Done Done false
4028 2154 Acked Write 3 Sent Done Done false
4029 2155 Acked Read 9 Sent Done Done false
4030 2156 Acked Read 5 Sent Done Done false
4031 2157 Acked Read 4 Sent Done Done false
4032 2158 Acked Write 9 Sent Done Done false
4033 2159 Acked Read 7 Sent Done Done false
4034 2160 Acked Write 8 Sent Done Done false
4035 2161 Acked Write 10 Sent Done Done false
4036 2162 Acked Read 10 Sent Done Done false
4037 2163 Acked Read 10 Sent Done Done false
4038 2164 Acked Read 6 Sent Done Done false
4039 2165 Acked Flush 0 Sent Done Done false
4040 2166 Acked Write 4 Sent Done Done false
4041 2167 Acked Read 3 Sent Done Done false
4042 2168 Acked Read 9 Sent Done Done false
4043 2169 Acked Read 1 Sent Done Done false
4044 2170 Acked Read 2 Sent Done Done false
4045 2171 Acked Read 1 Sent Done Done false
4046 2172 Acked Read 9 Sent Done Done false
4047 2173 Acked Read 5 Sent Done Done false
4048 2174 Acked Read 5 Sent Done Done false
4049 2175 Acked Read 2 Sent Done Done false
4050 2176 Acked Write 1 Sent Done Done false
4051 2177 Acked Write 6 Sent Done Done false
4052 2178 Acked Write 2 Sent Done Done false
4053 2179 Acked Read 5 Sent Done Done false
4054 2180 Acked Write 7 Sent Done Done false
4055 2181 Acked Write 10 Sent Done Done false
4056 2182 Acked Read 2 Sent Done Done false
4057 2183 Acked Write 10 Sent Done Done false
4058 2184 Acked Read 2 Sent Done Done false
4059 2185 Acked Read 7 Sent Done Done false
4060 2186 Acked Write 5 Sent Done Done false
4061 2187 Acked Read 5 Sent Done Done false
4062 2188 Acked Write 1 Sent Done Done false
4063 2189 Acked Read 6 Sent Done Done false
4064 2190 Acked Read 6 Sent Done Done false
4065 2191 Acked Read 6 Sent Done Done false
4066 2192 Acked Write 5 Sent Done Done false
4067 2193 Acked Read 5 Sent Done Done false
4068 2194 Acked Read 7 Sent Done Done false
4069 2195 Acked Write 9 Sent Done Done false
4070 2196 Acked Flush 0 Sent Done Done false
4071 2197 Acked Write 8 Sent Done Done false
4072 2198 Acked Write 1 Sent Done Done false
4073 2199 Acked Read 8 Sent Done Done false
4074 2200 Acked Read 3 Sent Done Done false
4075 2201 Acked Read 10 Sent Done Done false
4076 2202 Acked Read 6 Sent Done Done false
4077 2203 Acked Read 5 Sent Done Done false
4078 2204 Acked Flush 0 Sent Done Done false
4079 2205 Acked Read 2 Sent Done Done false
4080 2206 Acked Read 4 Sent Done Done false
4081 STATES DS:0 DS:1 DS:2 TOTAL
4082 Sent 301 0 0 301
4083 Done 0 301 301 602
4084 Skipped 0 0 0 0
4085 Error 0 0 0 0
4086 Last Flush: Some(JobId(1905)) Some(JobId(2204)) Some(JobId(2204))
4087 Downstairs last five completed: 1905 1904 1903 1902 1901
4088 Upstairs last five completed: 2206 2205 2204 2203 2202
4089 CLIENT: Up:0 ds:301 act:2
40902025-01-29T05:11:28.982ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40912025-01-29T05:11:29.984ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40922025-01-29T05:11:30.202ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40932025-01-29T05:11:30.202ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52375) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40942025-01-29T05:11:30.203ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40952025-01-29T05:11:30.204ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40962025-01-29T05:11:30.204ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40972025-01-29T05:11:30.204ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40982025-01-29T05:11:30.204ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
40992025-01-29T05:11:30.204ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
41002025-01-29T05:11:30.204ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
4101 ----------------------------------------------------------------
4102 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4103 Downstairs last five completed: 2207 2206 2205 2204 2203
4104 Upstairs last five completed: 2207 2206 2205 2204 2203
4105 CLIENT: Up:0 ds:0 act:3
4106 Replay: All jobs finished
4107 Main task has work to do, go find it
4108 got dsc Stop(1)
4109 stop 1
4110 Jan 29 05:11:32.446 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6b8b9264-f635-4f1a-801f-f75c5be4f0a9, remote_addr: 127.0[8820] Got stop action so:false kr:false
4111 .0.1:46995, local_addr: 127.0.0.1:9998
41122025-01-29T05:11:32.447ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
41132025-01-29T05:11:32.447ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
41142025-01-29T05:11:32.447ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
41152025-01-29T05:11:32.447ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
4116 Jan 29 05:11:32.447 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/1, method: GET, req_id: b3fbbe5d-d4b8-4634-ac3c-b8bc0280a4e1, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
4117 [8820] Exited with: ExitStatus(unix_wait_status(9))
4118 [8820][1] reports Exit
4119 Jan 29 05:11:36.450 INFO request completed, latency_us: 376, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7b011dca-e397-4aab-998a-0d32b383a0f0, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
4120 001/300 Read block 1515 len 40960
4121 002/300 Write block 1850 len 20480 data: 2 2 2 2 2
4122 003/300 Flush
4123 004/300 Read block 1417 len 24576
4124 005/300 Write block 1029 len 20480 data: 3 2 2 2 2
4125 006/300 Write block 2731 len 36864 data: 2 2 2 3 3 4 4 4 4
4126 007/300 Read block 1503 len 24576
4127 008/300 Read block 2239 len 28672
4128 009/300 Read block 3450 len 32768
4129 010/300 Read block 1883 len 8192
4130 011/300 Flush
4131 012/300 Read block 451 len 24576
4132 013/300 Read block 2879 len 24576
4133 014/300 Read block 1283 len 4096
4134 015/300 Read block 390 len 4096
4135 016/300 Read block 60 len 32768
4136 017/300 Write block 4320 len 28672 data: 2 2 2 2 2 2 2
4137 018/300 Write block 1950 len 8192 data: 3 4
4138 019/300 Write block 3463 len 32768 data: 2 2 2 2 2 2 2 2
4139 020/300 Read block 3243 len 20480
4140 021/300 Read block 1351 len 32768
4141 022/300 Flush
4142 023/300 Write block 4410 len 36864 data: 2 2 2 2 2 2 2 2 2
4143 024/300 Read block 3390 len 32768
4144 025/300 Read block 4234 len 36864
4145 026/300 Write block 2229 len 40960 data: 2 3 3 3 3 3 4 4 3 2
4146 027/300 Read block 1191 len 36864
4147 028/300 Read block 4593 len 40960
4148 029/300 Write block 1040 len 24576 data: 3 4 4 4 4 4
4149 030/300 Write block 1827 len 16384 data: 2 2 2 2
4150 031/300 Read block 3464 len 12288
4151 032/300 Write block 3017 len 12288 data: 2 3 3
4152 033/300 Read block 3371 len 20480
4153 034/300 Read block 4371 len 8192
4154 035/300 Read block 2451 len 12288
4155 036/300 Flush
4156 037/300 Write block 1603 len 4096 data: 2
4157 038/300 Read block 1298 len 4096
4158 039/300 Read block 3603 len 28672
4159 040/300 Read block 2862 len 32768
4160 041/300 Read block 3934 len 36864
4161 042/300 Read block 3513 len 24576
4162 043/300 Write block 1000 len 24576 data: 3 3 3 3 3 3
4163 044/300 Write block 1973 len 28672 data: 3 3 2 2 2 2 2
4164 045/300 Read block 1164 len 16384
4165 046/300 Read block 301 len 12288
4166 047/300 Write block 1168 len 20480 data: 2 2 2 2 2
4167 048/300 Read block 1463 len 40960
4168 049/300 Read block 3883 len 40960
4169 050/300 Read block 2597 len 8192
4170 051/300 Write block 2832 len 8192 data: 3 3
4171 052/300 Flush
4172 053/300 Write block 4957 len 12288 data: 4 4 4
4173 054/300 Write block 1373 len 24576 data: 3 3 3 2 2 2
4174 055/300 Read block 1467 len 4096
4175 056/300 Read block 1393 len 8192
4176 057/300 Read block 4848 len 32768
4177 058/300 Write block 654 len 28672 data: 2 2 3 3 3 3 3
4178 059/300 Read block 4679 len 20480
4179 060/300 Read block 89 len 32768
4180 061/300 Read block 2598 len 24576
4181 062/300 Read block 146 len 36864
4182 063/300 Write block 3628 len 24576 data: 2 2 2 2 2 2
4183 064/300 Write block 4718 len 8192 data: 2 2
4184 065/300 Read block 2184 len 32768
4185 066/300 Read block 1100 len 8192
4186 067/300 Write block 2695 len 20480 data: 2 2 2 2 2
4187 068/300 Read block 641 len 36864
4188 069/300 Read block 1834 len 24576
4189 070/300 Read block 1923 len 16384
4190 071/300 Write block 4149 len 4096 data: 2
4191 072/300 Read block 1854 len 4096
4192 073/300 Write block 3141 len 4096 data: 3
4193 074/300 Read block 1729 len 32768
4194 075/300 Write block 4808 len 16384 data: 2 2 4 3
4195 076/300 Read block 2210 len 8192
4196 077/300 Flush
4197 078/300 Read block 1090 len 36864
4198 079/300 Write block 3854 len 28672 data: 2 2 2 2 2 2 2
4199 080/300 Flush
4200 081/300 Read block 365 len 36864
4201 082/300 Flush
4202 083/300 Read block 4054 len 8192
4203 084/300 Read block 1247 len 40960
4204 085/300 Write block 4252 len 36864 data: 3 3 3 3 3 3 3 2 2
4205 086/300 Write block 4380 len 36864 data: 4 4 3 3 3 3 3 3 2
4206 087/300 Write block 1251 len 32768 data: 2 2 2 2 2 2 2 2
4207 088/300 Read block 3598 len 40960
4208 089/300 Write block 104 len 36864 data: 2 2 2 2 2 2 2 2 2
4209 090/300 Write block 3228 len 40960 data: 2 2 2 3 3 3 2 2 2 2
4210 091/300 Read block 2610 len 16384
4211 092/300 Read block 593 len 24576
4212 093/300 Flush
4213 094/300 Read block 4920 len 28672
4214 095/300 Read block 3594 len 12288
4215 096/300 Read block 4431 len 8192
4216 097/300 Read block 314 len 32768
4217 098/300 Write block 3126 len 28672 data: 2 2 2 2 3 3 3
4218 099/300 Read block 2206 len 4096
4219 100/300 Write block 2790 len 28672 data: 3 2 2 2 2 2 2
4220 101/300 Read block 92 len 24576
4221 102/300 Read block 2982 len 32768
4222 103/300 Read block 3742 len 16384
4223 104/300 Write block 1682 len 16384 data: 3 2 2 2
4224 105/300 Write block 1771 len 24576 data: 2 2 3 3 3 3
4225 106/300 Write block 2742 len 8192 data: 3 3
4226 107/300 Read block 2323 len 40960
4227 108/300 Write block 1172 len 12288 data: 3 2 2
4228 109/300 Write block 2028 len 32768 data: 2 2 2 2 2 2 2 2
4229 110/300 Read block 4766 len 40960
4230 111/300 Write block 1108 len 32768 data: 3 3 3 3 3 2 2 2
4231 112/300 Write block 4008 len 24576 data: 3 3 3 3 3 2
4232 113/300 Read block 4911 len 20480
4233 114/300 Read block 4515 len 8192
4234 115/300 Write block 2964 len 36864 data: 3 3 3 3 2 2 2 2 2
4235 116/300 Write block 2006 len 24576 data: 3 2 4 4 4 4
4236 117/300 Read block 4293 len 28672
4237 118/300 Read block 1909 len 32768
4238 119/300 Write block 2663 len 12288 data: 4 3 3
4239 120/300 Write block 1918 len 24576 data: 2 2 2 2 2 2
4240 121/300 Write block 158 len 36864 data: 2 2 2 2 2 2 2 2 2
4241 122/300 Write block 3899 len 28672 data: 2 2 2 2 2 3 3
4242 123/300 Flush
4243 124/300 Write block 1501 len 16384 data: 4 4 3 3
4244 125/300 Write block 3373 len 4096 data: 3
4245 126/300 Read block 1097 len 28672
4246 127/300 Write block 3527 len 24576 data: 2 2 2 2 2 2
4247 128/300 Write block 1296 len 28672 data: 2 2 2 2 2 2 2
4248 129/300 Read block 2210 len 28672
4249 130/300 Read block 3962 len 24576
4250 131/300 Read block 3030 len 36864
4251 132/300 Read block 3454 len 20480
4252 133/300 Write block 2297 len 24576 data: 3 2 2 2 2 2
4253 134/300 Write block 1593 len 8192 data: 3 3
4254 135/300 Write block 408 len 16384 data: 2 2 2 2
4255 136/300 Read block 2701 len 16384
4256 137/300 Flush
4257 138/300 Write block 4419 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4258 139/300 Read block 446 len 32768
4259 140/300 Write block 744 len 16384 data: 3 4 2 2
4260 141/300 Flush
4261 142/300 Write block 1019 len 36864 data: 2 2 2 2 2 2 2 2 2
4262 143/300 Read block 2137 len 20480
4263 144/300 Flush
4264 145/300 Read block 439 len 24576
4265 146/300 Write block 2822 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4266 147/300 Read block 3202 len 4096
4267 148/300 Write block 1283 len 32768 data: 2 2 2 2 2 2 2 2
4268 149/300 Write block 984 len 4096 data: 2
4269 150/300 Read block 1567 len 36864
4270 151/300 Write block 1167 len 36864 data: 2 3 3 3 3 4 3 3 2
4271 152/300 Write block 4575 len 4096 data: 2
4272 153/300 Read block 1038 len 32768
4273 154/300 Write block 3896 len 36864 data: 2 2 2 3 3 3 3 3 4
4274 155/300 Read block 4984 len 4096
4275 156/300 Write block 466 len 32768 data: 2 3 4 3 3 2 2 2
4276 157/300 Read block 876 len 20480
4277 158/300 Read block 382 len 16384
4278 159/300 Write block 4685 len 16384 data: 3 3 2 2
4279 160/300 Read block 3229 len 8192
4280 161/300 Write block 4515 len 20480 data: 2 2 3 3 3
4281 162/300 Read block 3932 len 40960
4282 163/300 Read block 1451 len 28672
4283 164/300 Read block 2934 len 4096
4284 165/300 Read block 879 len 40960
4285 166/300 Flush
4286 167/300 Flush
4287 168/300 Flush
4288 169/300 Read block 1257 len 32768
4289 170/300 Read block 533 len 24576
4290 171/300 Read block 502 len 28672
4291 172/300 Write block 2346 len 28672 data: 3 3 2 2 2 2 2
4292 173/300 Write block 3357 len 32768 data: 2 2 2 2 2 2 2 3
4293 174/300 Write block 4555 len 12288 data: 3 3 2
4294 175/300 Write block 4102 len 20480 data: 4 4 3 2 2
4295 176/300 Write block 2425 len 40960 data: 4 4 3 2 2 2 2 2 2 2
4296 177/300 Write block 1758 len 16384 data: 2 2 2 2
4297 178/300 Read block 2942 len 24576
4298 179/300 Write block 3578 len 4096 data: 4
4299 180/300 Read block 2733 len 12288
4300 181/300 Write block 4612 len 16384 data: 3 3 2 2
4301 182/300 Write block 1661 len 4096 data: 2
4302 183/300 Read block 955 len 16384
4303 184/300 Read block 3268 len 40960
4304 185/300 Read block 4392 len 4096
4305 186/300 Write block 510 len 32768 data: 3 2 2 2 2 3 3 3
4306 187/300 Read block 1386 len 8192
4307 188/300 Read block 2548 len 36864
4308 189/300 Write block 909 len 28672 data: 2 2 2 2 3 3 3
4309 190/300 Write block 687 len 32768 data: 2 3 4 4 4 4 3 3
4310 191/300 Write block 2001 len 32768 data: 4 4 3 3 3 4 3 5
4311 192/300 Read block 4011 len 24576
4312 193/300 Read block 579 len 24576
4313 194/300 Flush
4314 195/300 Read block 3226 len 20480
4315 196/300 Write block 4533 len 8192 data: 3 3
4316 197/300 Write block 698 len 36864 data: 2 2 2 2 2 2 2 3 3
4317 198/300 Flush
4318 199/300 Write block 3235 len 12288 data: 3 3 3
4319 200/300 Flush
4320 201/300 Write block 4584 len 8192 data: 3 3
4321 202/300 Write block 1129 len 16384 data: 3 3 2 2
4322 203/300 Write block 2356 len 36864 data: 2 2 2 2 2 2 2 2 2
4323 204/300 Flush
4324 205/300 Read block 3168 len 40960
4325 206/300 Read block 544 len 20480
4326 207/300 Flush
4327 208/300 Write block 3470 len 20480 data: 3 2 2 2 2
4328 209/300 Read block 633 len 20480
4329 210/300 Flush
4330 211/300 Read block 4077 len 32768
4331 212/300 Write block 4597 len 32768 data: 2 2 2 3 3 3 3 3
4332 213/300 Write block 4625 len 28672 data: 2 2 2 2 2 2 2
4333 214/300 Read block 3960 len 32768
4334 215/300 Read block 2591 len 4096
4335 216/300 Read block 1174 len 24576
4336 217/300 Write block 116 len 32768 data: 2 2 2 2 2 2 2 2
4337 218/300 Read block 2326 len 36864
4338 219/300 Read block 2727 len 4096
4339 220/300 Write block 1923 len 20480 data: 3 2 2 2 2
4340 221/300 Flush
4341 222/300 Read block 2033 len 20480
4342 223/300 Flush
4343 224/300 Write block 4167 len 32768 data: 3 3 4 4 3 3 3 3
4344 225/300 Write block 3892 len 20480 data: 2 2 2 2 3
4345 226/300 Read block 2048 len 8192
4346 227/300 Write block 1790 len 12288 data: 3 2 2
4347 228/300 Read block 4599 len 12288
4348 229/300 Read block 3936 len 28672
4349 230/300 Read block 1765 len 32768
4350 231/300 Write block 1171 len 36864 data: 4 5 4 4 3 2 2 2 2
4351 232/300 Read block 912 len 8192
4352 233/300 Flush
4353 234/300 Write block 2865 len 12288 data: 2 2 2
4354 235/300 Read block 4707 len 16384
4355 236/300 Write block 4785 len 28672 data: 2 2 2 2 2 2 2
4356 237/300 Write block 1022 len 20480 data: 3 3 3 3 3
4357 238/300 Write block 1299 len 36864 data: 3 3 3 3 2 2 2 2 2
4358 239/300 Write block 3635 len 32768 data: 2 2 2 2 2 2 3 3
4359 240/300 Read block 3337 len 36864
4360 241/300 Write block 2809 len 20480 data: 2 2 2 3 3
4361 242/300 Write block 1299 len 20480 data: 4 4 4 4 3
4362 243/300 Write block 2757 len 28672 data: 3 3 3 3 3 3 2
4363 244/300 Flush
4364 245/300 Read block 2176 len 36864
4365 246/300 Read block 3816 len 12288
4366 247/300 Read block 694 len 32768
4367 248/300 Read block 3310 len 40960
4368 249/300 Read block 2372 len 12288
4369 250/300 Read block 4245 len 16384
4370 251/300 Read block 1278 len 24576
4371 252/300 Read block 3290 len 28672
4372 253/300 Write block 2096 len 20480 data: 2 2 2 2 2
4373 254/300 Read block 3842 len 12288
4374 255/300 Read block 4764 len 32768
4375 256/300 Read block 433 len 32768
4376 257/300 Flush
4377 258/300 Read block 227 len 24576
4378 259/300 Flush
4379 260/300 Flush
4380 261/300 Flush
4381 262/300 Write block 1416 len 24576 data: 3 2 2 2 3 3
4382 263/300 Read block 211 len 20480
4383 264/300 Write block 3118 len 24576 data: 3 3 3 2 2 2
4384 265/300 Read block 2582 len 20480
4385 266/300 Flush
4386 267/300 Write block 111 len 20480 data: 3 3 2 2 3
4387 268/300 Flush
4388 269/300 Write block 4201 len 24576 data: 4 4 4 4 3 2
4389 270/300 Flush
4390 271/300 Write block 2029 len 28672 data: 3 3 3 3 3 3 3
4391 272/300 Write block 1889 len 20480 data: 3 3 4 3 3
4392 273/300 Read block 4473 len 8192
4393 274/300 Write block 3730 len 28672 data: 2 2 3 3 3 3 3
4394 275/300 Read block 4226 len 32768
4395 276/300 Write block 1553 len 12288 data: 2 2 2
4396 277/300 Write block 3090 len 36864 data: 2 2 2 2 2 2 2 2 2
4397 278/300 Read block 1729 len 24576
4398 279/300 Flush
4399 280/300 Flush
4400 281/300 Write block 3904 len 40960 data: 5 4 2 2 2 3 2 2 2 2
4401 282/300 Read block 3854 len 28672
4402 283/300 Read block 3861 len 32768
4403 284/300 Read block 821 len 8192
4404 285/300 Write block 4305 len 8192 data: 2 2
4405 286/300 Write block 2923 len 40960 data: 2 2 2 2 2 2 2 2 2 3
4406 287/300 Write block 3728 len 12288 data: 2 2 3
4407 288/300 Read block 3151 len 4096
4408 289/300 Write block 3580 len 12288 data: 3 3 2
4409 290/300 Write block 3533 len 12288 data: 2 2 2
4410 291/300 Write block 736 len 20480 data: 3 3 3 3 3
4411 292/300 Write block 2569 len 20480 data: 2 2 2 2 4
4412 293/300 Write block 3195 len 20480 data: 2 2 2 2 2
4413 294/300 Read block 2434 len 40960
4414 295/300 Read block 299 len 4096
4415 296/300 Write block 1594 len 4096 data: 4
4416 297/300 Write block 3657 len 8192 data: 3 4
4417 298/300 Write block 234 len 36864 data: 2 2 3 4 4 4 4 5 4
4418 299/300 Read block 2979 len 12288
4419 300/300 Read block 2368 len 40960
4420 301/300 Write block 4498 len 40960 data: 2 2 2 2 3 4 4 4 3 3
4421 Main task has work to do, go find it
4422 got dsc Start(1)
4423 start 1
4424 Jan 29 05:11:36.691 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9aa65022-1be4-4d84-9879-1586133ea689, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
4425 Replay: started 1, returned:Ok(())
4426 ----------------------------------------------------------------
4427 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4428 States: Active Offline Active
4429 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4430 2208 Acked Read 10 Done Sent Done false
4431 2209 Acked Write 5 Done Sent Done false
4432 2210 Acked Flush 0 Done Sent Done false
4433 2211 Acked Read 6 Done Sent Done false
4434 2212 Acked Write 5 Done Sent Done false
4435 2213 Acked Write 9 Done Sent Done false
4436 2214 Acked Read 6 Done Sent Done false
4437 2215 Acked Read 7 Done Sent Done false
4438 2216 Acked Read 8 Done Sent Done false
4439 2217 Acked Read 2 Done Sent Done false
4440 2218 Acked Flush 0 Done Sent Done false
4441 2219 Acked Read 6 Done Sent Done false
4442 2220 Acked Read 6 Done Sent Done false
4443 2221 Acked Read 1 Done Sent Done false
4444 2222 Acked Read 1 Done Sent Done false
4445 2223 Acked Read 8 Done Sent Done false
4446 2224 Acked Write 7 Done Sent Done false
4447 2225 Acked Write 2 Done Sent Done false
4448 2226 Acked Write 8 Done Sent Done false
4449 2227 Acked Read 5 Done Sent Done false
4450 2228 Acked Read 8 Done Sent Done false
4451 2229 Acked Flush 0 Done Sent Done false
4452 2230 Acked Write 9 Done Sent Done false
4453 2231 Acked Read 8 Done Sent Done false
4454 2232 Acked Read 9 Done Sent Done false
4455 2233 Acked Write 10 Done Sent Done false
4456 2234 Acked Read 9 Done Sent Done false
4457 2235 Acked Read 10 Done Sent Done false
4458 2236 Acked Write 6 Done Sent Done false
4459 2237 Acked Write 4 Done Sent Done false
4460 2238 Acked Read 3 Done Sent Done false
4461 2239 Acked Write 3 Done Sent Done false
4462 2240 Acked Read 5 Done Sent Done false
4463 2241 Acked Read 2 Done Sent Done false
4464 2242 Acked Read 3 Done Sent Done false
4465 2243 Acked Flush 0 Done Sent Done false
4466 2244 Acked Write 1 Done Sent Done false
4467 2245 Acked Read 1 Done Sent Done false
4468 2246 Acked Read 7 Done Sent Done false
4469 2247 Acked Read 8 Done Sent Done false
4470 2248 Acked Read 9 Done Sent Done false
4471 2249 Acked Read 6 Done Sent Done false
4472 2250 Acked Write 6 Done Sent Done false
4473 2251 Acked Write 7 Done Sent Done false
4474 2252 Acked Read 4 Done Sent Done false
4475 2253 Acked Read 3 Done Sent Done false
4476 2254 Acked Write 5 Done Sent Done false
4477 2255 Acked Read 10 Done Sent Done false
4478 2256 Acked Read 10 Done Sent Done false
4479 2257 Acked Read 2 Done Sent Done false
4480 2258 Acked Write 2 Done Sent Done false
4481 2259 Acked Flush 0 Done Sent Done false
4482 2260 Acked Write 3 Done Sent Done false
4483 2261 Acked Write 6 Done Sent Done false
4484 2262 Acked Read 1 Done Sent Done false
4485 2263 Acked Read 2 Done Sent Done false
4486 2264 Acked Read 8 Done Sent Done false
4487 2265 Acked Write 7 Done Sent Done false
4488 2266 Acked Read 5 Done Sent Done false
4489 2267 Acked Read 8 Done Sent Done false
4490 2268 Acked Read 6 Done Sent Done false
4491 2269 Acked Read 9 Done Sent Done false
4492 2270 Acked Write 6 Done Sent Done false
4493 2271 Acked Write 2 Done Sent Done false
4494 2272 Acked Read 8 Done Sent Done false
4495 2273 Acked Read 2 Done Sent Done false
4496 2274 Acked Write 5 Done Sent Done false
4497 2275 Acked Read 9 Done Sent Done false
4498 2276 Acked Read 6 Done Sent Done false
4499 2277 Acked Read 4 Done Sent Done false
4500 2278 Acked Write 1 Done Sent Done false
4501 2279 Acked Read 1 Done Sent Done false
4502 2280 Acked Write 1 Done Sent Done false
4503 2281 Acked Read 8 Done Sent Done false
4504 2282 Acked Write 4 Done Sent Done false
4505 2283 Acked Read 2 Done Sent Done false
4506 2284 Acked Flush 0 Done Sent Done false
4507 2285 Acked Read 9 Done Sent Done false
4508 2286 Acked Write 7 Done Sent Done false
4509 2287 Acked Flush 0 Done Sent Done false
4510 2288 Acked Read 9 Done Sent Done false
4511 2289 Acked Flush 0 Done Sent Done false
4512 2290 Acked Read 2 Done Sent Done false
4513 2291 Acked Read 10 Done Sent Done false
4514 2292 Acked Write 9 Done Sent Done false
4515 2293 Acked Write 9 Done Sent Done false
4516 2294 Acked Write 8 Done Sent Done false
4517 2295 Acked Read 10 Done Sent Done false
4518 2296 Acked Write 9 Done Sent Done false
4519 2297 Acked Write 10 Done Sent Done false
4520 2298 Acked Read 4 Done Sent Done false
4521 2299 Acked Read 6 Done Sent Done false
4522 2300 Acked Flush 0 Done Sent Done false
4523 2301 Acked Read 7 Done Sent Done false
4524 2302 Acked Read 3 Done Sent Done false
4525 2303 Acked Read 2 Done Sent Done false
4526 2304 Acked Read 8 Done Sent Done false
4527 2305 Acked Write 7 Done Sent Done false
4528 2306 Acked Read 1 Done Sent Done false
4529 2307 Acked Write 7 Done Sent Done false
4530 2308 Acked Read 6 Done Sent Done false
4531 2309 Acked Read 8 Done Sent Done false
4532 2310 Acked Read 4 Done Sent Done false
4533 2311 Acked Write 4 Done Sent Done false
4534 2312 Acked Write 6 Done Sent Done false
4535 2313 Acked Write 2 Done Sent Done false
4536 2314 Acked Read 10 Done Sent Done false
4537 2315 Acked Write 3 Done Sent Done false
4538 2316 Acked Write 8 Done Sent Done false
4539 2317 Acked Read 10 Done Sent Done false
4540 2318 Acked Write 8 Done Sent Done false
4541 2319 Acked Write 6 Done Sent Done false
4542 2320 Acked Read 5 Done Sent Done false
4543 2321 Acked Read 2 Done Sent Done false
4544 2322 Acked Write 9 Done Sent Done false
4545 2323 Acked Write 6 Done Sent Done false
4546 2324 Acked Read 7 Done Sent Done false
4547 2325 Acked Read 8 Done Sent Done false
4548 2326 Acked Write 3 Done Sent Done false
4549 2327 Acked Write 6 Done Sent Done false
4550 2328 Acked Write 9 Done Sent Done false
4551 2329 Acked Write 7 Done Sent Done false
4552 2330 Acked Flush 0 Done Sent Done false
4553 2331 Acked Write 4 Done Sent Done false
4554 2332 Acked Write 1 Done Sent Done false
4555 2333 Acked Read 7 Done Sent Done false
4556 2334 Acked Write 6 Done Sent Done false
4557 2335 Acked Write 7 Done Sent Done false
4558 2336 Acked Read 7 Done Sent Done false
4559 2337 Acked Read 6 Done Sent Done false
4560 2338 Acked Read 9 Done Sent Done false
4561 2339 Acked Read 5 Done Sent Done false
4562 2340 Acked Write 6 Done Sent Done false
4563 2341 Acked Write 2 Done Sent Done false
4564 2342 Acked Write 4 Done Sent Done false
4565 2343 Acked Read 4 Done Sent Done false
4566 2344 Acked Flush 0 Done Sent Done false
4567 2345 Acked Write 10 Done Sent Done false
4568 2346 Acked Read 8 Done Sent Done false
4569 2347 Acked Write 4 Done Sent Done false
4570 2348 Acked Flush 0 Done Sent Done false
4571 2349 Acked Write 9 Done Sent Done false
4572 2350 Acked Read 5 Done Sent Done false
4573 2351 Acked Flush 0 Done Sent Done false
4574 2352 Acked Read 6 Done Sent Done false
4575 2353 Acked Write 10 Done Sent Done false
4576 2354 Acked Read 1 Done Sent Done false
4577 2355 Acked Write 8 Done Sent Done false
4578 2356 Acked Write 1 Done Sent Done false
4579 2357 Acked Read 9 Done Sent Done false
4580 2358 Acked Write 9 Done Sent Done false
4581 2359 Acked Write 1 Done Sent Done false
4582 2360 Acked Read 8 Done Sent Done false
4583 2361 Acked Write 9 Done Sent Done false
4584 2362 Acked Read 1 Done Sent Done false
4585 2363 Acked Write 8 Done Sent Done false
4586 2364 Acked Read 5 Done Sent Done false
4587 2365 Acked Read 4 Done Sent Done false
4588 2366 Acked Write 4 Done Sent Done false
4589 2367 Acked Read 2 Done Sent Done false
4590 2368 Acked Write 5 Done Sent Done false
4591 2369 Acked Read 10 Done Sent Done false
4592 2370 Acked Read 7 Done Sent Done false
4593 2371 Acked Read 1 Done Sent Done false
4594 2372 Acked Read 10 Done Sent Done false
4595 2373 Acked Flush 0 Done Sent Done false
4596 2374 Acked Flush 0 Done Sent Done false
4597 2375 Acked Flush 0 Done Sent Done false
4598 2376 Acked Read 8 Done Sent Done false
4599 2377 Acked Read 6 Done Sent Done false
4600 2378 Acked Read 7 Done Sent Done false
4601 2379 Acked Write 7 Done Sent Done false
4602 2380 Acked Write 8 Done Sent Done false
4603 2381 Acked Write 3 Done Sent Done false
4604 2382 Acked Write 5 Done Sent Done false
4605 2383 Acked Write 10 Done Sent Done false
4606 2384 Acked Write 4 Done Sent Done false
4607 2385 Acked Read 6 Done Sent Done false
4608 2386 Acked Write 1 Done Sent Done false
4609 2387 Acked Read 3 Done Sent Done false
4610 2388 Acked Write 4 Done Sent Done false
4611 2389 Acked Write 1 Done Sent Done false
4612 2390 Acked Read 4 Done Sent Done false
4613 2391 Acked Read 10 Done Sent Done false
4614 2392 Acked Read 1 Done Sent Done false
4615 2393 Acked Write 8 Done Sent Done false
4616 2394 Acked Read 2 Done Sent Done false
4617 2395 Acked Read 9 Done Sent Done false
4618 2396 Acked Write 7 Done Sent Done false
4619 2397 Acked Write 8 Done Sent Done false
4620 2398 Acked Write 8 Done Sent Done false
4621 2399 Acked Read 6 Done Sent Done false
4622 2400 Acked Read 6 Done Sent Done false
4623 2401 Acked Flush 0 Done Sent Done false
4624 2402 Acked Read 5 Done Sent Done false
4625 2403 Acked Write 2 Done Sent Done false
4626 2404 Acked Write 9 Done Sent Done false
4627 2405 Acked Flush 0 Done Sent Done false
4628 2406 Acked Write 3 Done Sent Done false
4629 2407 Acked Flush 0 Done Sent Done false
4630 2408 Acked Write 2 Done Sent Done false
4631 2409 Acked Write 4 Done Sent Done false
4632 2410 Acked Write 9 Done Sent Done false
4633 2411 Acked Flush 0 Done Sent Done false
4634 2412 Acked Read 10 Done Sent Done false
4635 2413 Acked Read 5 Done Sent Done false
4636 2414 Acked Flush 0 Done Sent Done false
4637 2415 Acked Write 5 Done Sent Done false
4638 2416 Acked Read 5 Done Sent Done false
4639 2417 Acked Flush 0 Done Sent Done false
4640 2418 Acked Read 8 Done Sent Done false
4641 2419 Acked Write 8 Done Sent Done false
4642 2420 Acked Write 7 Done Sent Done false
4643 2421 Acked Read 8 Done Sent Done false
4644 2422 Acked Read 1 Done Sent Done false
4645 2423 Acked Read 6 Done Sent Done false
4646 2424 Acked Write 8 Done Sent Done false
4647 2425 Acked Read 9 Done Sent Done false
4648 2426 Acked Read 1 Done Sent Done false
4649 2427 Acked Write 5 Done Sent Done false
4650 2428 Acked Flush 0 Done Sent Done false
4651 2429 Acked Read 5 Done Sent Done false
4652 2430 Acked Flush 0 Done Sent Done false
4653 2431 Acked Write 8 Done Sent Done false
4654 2432 Acked Write 5 Done Sent Done false
4655 2433 Acked Read 2 Done Sent Done false
4656 2434 Acked Write 3 Done Sent Done false
4657 2435 Acked Read 3 Done Sent Done false
4658 2436 Acked Read 7 Done Sent Done false
4659 2437 Acked Read 8 Done Sent Done false
4660 2438 Acked Write 9 Done Sent Done false
4661 2439 Acked Read 2 Done Sent Done false
4662 2440 Acked Flush 0 Done Sent Done false
4663 2441 Acked Write 3 Done Sent Done false
4664 2442 Acked Read 4 Done Sent Done false
4665 2443 Acked Write 7 Done Sent Done false
4666 2444 Acked Write 5 Done Sent Done false
4667 2445 Acked Write 9 Done Sent Done false
4668 2446 Acked Write 8 Done Sent Done false
4669 2447 Acked Read 9 Done Sent Done false
4670 2448 Acked Write 5 Done Sent Done false
4671 2449 Acked Write 5 Done Sent Done false
4672 2450 Acked Write 7 Done Sent Done false
4673 2451 Acked Flush 0 Done Sent Done false
4674 2452 Acked Read 9 Done Sent Done false
4675 2453 Acked Read 3 Done Sent Done false
4676 2454 Acked Read 8 Done Sent Done false
4677 2455 Acked Read 10 Done Sent Done false
4678 2456 Acked Read 3 Done Sent Done false
4679 2457 Acked Read 4 Done Sent Done false
4680 2458 Acked Read 6 Done Sent Done false
4681 2459 Acked Read 7 Done Sent Done false
4682 2460 Acked Write 5 Done Sent Done false
4683 2461 Acked Read 3 Done Sent Done false
4684 2462 Acked Read 8 Done Sent Done false
4685 2463 Acked Read 8 Done Sent Done false
4686 2464 Acked Flush 0 Done Sent Done false
4687 2465 Acked Read 6 Done Sent Done false
4688 2466 Acked Flush 0 Done Sent Done false
4689 2467 Acked Flush 0 Done Sent Done false
4690 2468 Acked Flush 0 Done Sent Done false
4691 2469 Acked Write 6 Done Sent Done false
4692 2470 Acked Read 5 Done Sent Done false
4693 2471 Acked Write 6 Done Sent Done false
4694 2472 Acked Read 5 Done Sent Done false
4695 2473 Acked Flush 0 Done Sent Done false
4696 2474 Acked Write 5 Done Sent Done false
4697 2475 Acked Flush 0 Done Sent Done false
4698 2476 Acked Write 6 Done Sent Done false
4699 2477 Acked Flush 0 Done Sent Done false
4700 2478 Acked Write 7 Done Sent Done false
4701 2479 Acked Write 5 Done Sent Done false
4702 2480 Acked Read 2 Done Sent Done false
4703 2481 Acked Write 7 Done Sent Done false
4704 2482 Acked Read 8 Done Sent Done false
4705 2483 Acked Write 3 Done Sent Done false
4706 2484 Acked Write 9 Done Sent Done false
4707 2485 Acked Read 6 Done Sent Done false
4708 2486 Acked Flush 0 Done Sent Done false
4709 2487 Acked Flush 0 Done Sent Done false
4710 2488 Acked Write 10 Done Sent Done false
4711 2489 Acked Read 7 Done Sent Done false
4712 2490 Acked Read 8 Done Sent Done false
4713 2491 Acked Read 2 Done Sent Done false
4714 2492 Acked Write 2 Done Sent Done false
4715 2493 Acked Write 10 Done Sent Done false
4716 2494 Acked Write 3 Done Sent Done false
4717 2495 Acked Read 1 Done Sent Done false
4718 2496 Acked Write 3 Done Sent Done false
4719 2497 Acked Write 3 Done Sent Done false
4720 2498 Acked Write 5 Done Sent Done false
4721 2499 Acked Write 5 Done Sent Done false
4722 2500 Acked Write 5 Done Sent Done false
4723 2501 Acked Read 10 Done Sent Done false
4724 2502 Acked Read 1 Done Sent Done false
4725 2503 Acked Write 1 Done Sent Done false
4726 2504 Acked Write 2 Done Sent Done false
4727 2505 Acked Write 9 Done Sent Done false
4728 2506 Acked Read 3 Done Sent Done false
4729 2507 Acked Read 10 Done Sent Done false
4730 2508 Acked Write 10 Done Sent Done false
4731 STATES DS:0 DS:1 DS:2 TOTAL
4732 Sent 0 301 0 301
4733 Done 301 0 301 602
4734 Skipped 0 0 0 0
4735 Error 0 0 0 0
4736 Last Flush: Some(JobId(2487)) Some(JobId(2207)) Some(JobId(2487))
4737 Downstairs last five completed: 2207 2206 2205 2204 2203
4738 Upstairs last five completed: 2508 2507 2506 2505 2504
4739 CLIENT: Up:0 ds:301 act:2
47402025-01-29T05:11:37.193ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
47412025-01-29T05:11:37.996ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
4742 [8820] got start action
4743 [8820] I am going to restart
4744 Starting downstairs at port 8820
4745 Make output file at "/tmp/dsc/downstairs-8820.txt"
4746 [8820][1] reports Starting
4747 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1207)
4748 [8820][1] reports Running
47492025-01-29T05:11:38.998ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
47502025-01-29T05:11:40.000ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
4751 ----------------------------------------------------------------
4752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4753 States: Active Offline Active
4754 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4755 2208 Acked Read 10 Done Sent Done false
4756 2209 Acked Write 5 Done Sent Done false
4757 2210 Acked Flush 0 Done Sent Done false
4758 2211 Acked Read 6 Done Sent Done false
4759 2212 Acked Write 5 Done Sent Done false
4760 2213 Acked Write 9 Done Sent Done false
4761 2214 Acked Read 6 Done Sent Done false
4762 2215 Acked Read 7 Done Sent Done false
4763 2216 Acked Read 8 Done Sent Done false
4764 2217 Acked Read 2 Done Sent Done false
4765 2218 Acked Flush 0 Done Sent Done false
4766 2219 Acked Read 6 Done Sent Done false
4767 2220 Acked Read 6 Done Sent Done false
4768 2221 Acked Read 1 Done Sent Done false
4769 2222 Acked Read 1 Done Sent Done false
4770 2223 Acked Read 8 Done Sent Done false
4771 2224 Acked Write 7 Done Sent Done false
4772 2225 Acked Write 2 Done Sent Done false
4773 2226 Acked Write 8 Done Sent Done false
4774 2227 Acked Read 5 Done Sent Done false
4775 2228 Acked Read 8 Done Sent Done false
4776 2229 Acked Flush 0 Done Sent Done false
4777 2230 Acked Write 9 Done Sent Done false
4778 2231 Acked Read 8 Done Sent Done false
4779 2232 Acked Read 9 Done Sent Done false
4780 2233 Acked Write 10 Done Sent Done false
4781 2234 Acked Read 9 Done Sent Done false
4782 2235 Acked Read 10 Done Sent Done false
4783 2236 Acked Write 6 Done Sent Done false
4784 2237 Acked Write 4 Done Sent Done false
4785 2238 Acked Read 3 Done Sent Done false
4786 2239 Acked Write 3 Done Sent Done false
4787 2240 Acked Read 5 Done Sent Done false
4788 2241 Acked Read 2 Done Sent Done false
4789 2242 Acked Read 3 Done Sent Done false
4790 2243 Acked Flush 0 Done Sent Done false
4791 2244 Acked Write 1 Done Sent Done false
4792 2245 Acked Read 1 Done Sent Done false
4793 2246 Acked Read 7 Done Sent Done false
4794 2247 Acked Read 8 Done Sent Done false
4795 2248 Acked Read 9 Done Sent Done false
4796 2249 Acked Read 6 Done Sent Done false
4797 2250 Acked Write 6 Done Sent Done false
4798 2251 Acked Write 7 Done Sent Done false
4799 2252 Acked Read 4 Done Sent Done false
4800 2253 Acked Read 3 Done Sent Done false
4801 2254 Acked Write 5 Done Sent Done false
4802 2255 Acked Read 10 Done Sent Done false
4803 2256 Acked Read 10 Done Sent Done false
4804 2257 Acked Read 2 Done Sent Done false
4805 2258 Acked Write 2 Done Sent Done false
4806 2259 Acked Flush 0 Done Sent Done false
4807 2260 Acked Write 3 Done Sent Done false
4808 2261 Acked Write 6 Done Sent Done false
4809 2262 Acked Read 1 Done Sent Done false
4810 2263 Acked Read 2 Done Sent Done false
4811 2264 Acked Read 8 Done Sent Done false
4812 2265 Acked Write 7 Done Sent Done false
4813 2266 Acked Read 5 Done Sent Done false
4814 2267 Acked Read 8 Done Sent Done false
4815 2268 Acked Read 6 Done Sent Done false
4816 2269 Acked Read 9 Done Sent Done false
4817 2270 Acked Write 6 Done Sent Done false
4818 2271 Acked Write 2 Done Sent Done false
4819 2272 Acked Read 8 Done Sent Done false
4820 2273 Acked Read 2 Done Sent Done false
4821 2274 Acked Write 5 Done Sent Done false
4822 2275 Acked Read 9 Done Sent Done false
4823 2276 Acked Read 6 Done Sent Done false
4824 2277 Acked Read 4 Done Sent Done false
4825 2278 Acked Write 1 Done Sent Done false
4826 2279 Acked Read 1 Done Sent Done false
4827 2280 Acked Write 1 Done Sent Done false
4828 2281 Acked Read 8 Done Sent Done false
4829 2282 Acked Write 4 Done Sent Done false
4830 2283 Acked Read 2 Done Sent Done false
4831 2284 Acked Flush 0 Done Sent Done false
4832 2285 Acked Read 9 Done Sent Done false
4833 2286 Acked Write 7 Done Sent Done false
4834 2287 Acked Flush 0 Done Sent Done false
4835 2288 Acked Read 9 Done Sent Done false
4836 2289 Acked Flush 0 Done Sent Done false
4837 2290 Acked Read 2 Done Sent Done false
4838 2291 Acked Read 10 Done Sent Done false
4839 2292 Acked Write 9 Done Sent Done false
4840 2293 Acked Write 9 Done Sent Done false
4841 2294 Acked Write 8 Done Sent Done false
4842 2295 Acked Read 10 Done Sent Done false
4843 2296 Acked Write 9 Done Sent Done false
4844 2297 Acked Write 10 Done Sent Done false
4845 2298 Acked Read 4 Done Sent Done false
4846 2299 Acked Read 6 Done Sent Done false
4847 2300 Acked Flush 0 Done Sent Done false
4848 2301 Acked Read 7 Done Sent Done false
4849 2302 Acked Read 3 Done Sent Done false
4850 2303 Acked Read 2 Done Sent Done false
4851 2304 Acked Read 8 Done Sent Done false
4852 2305 Acked Write 7 Done Sent Done false
4853 2306 Acked Read 1 Done Sent Done false
4854 2307 Acked Write 7 Done Sent Done false
4855 2308 Acked Read 6 Done Sent Done false
4856 2309 Acked Read 8 Done Sent Done false
4857 2310 Acked Read 4 Done Sent Done false
4858 2311 Acked Write 4 Done Sent Done false
4859 2312 Acked Write 6 Done Sent Done false
4860 2313 Acked Write 2 Done Sent Done false
4861 2314 Acked Read 10 Done Sent Done false
4862 2315 Acked Write 3 Done Sent Done false
4863 2316 Acked Write 8 Done Sent Done false
4864 2317 Acked Read 10 Done Sent Done false
4865 2318 Acked Write 8 Done Sent Done false
4866 2319 Acked Write 6 Done Sent Done false
4867 2320 Acked Read 5 Done Sent Done false
4868 2321 Acked Read 2 Done Sent Done false
4869 2322 Acked Write 9 Done Sent Done false
4870 2323 Acked Write 6 Done Sent Done false
4871 2324 Acked Read 7 Done Sent Done false
4872 2325 Acked Read 8 Done Sent Done false
4873 2326 Acked Write 3 Done Sent Done false
4874 2327 Acked Write 6 Done Sent Done false
4875 2328 Acked Write 9 Done Sent Done false
4876 2329 Acked Write 7 Done Sent Done false
4877 2330 Acked Flush 0 Done Sent Done false
4878 2331 Acked Write 4 Done Sent Done false
4879 2332 Acked Write 1 Done Sent Done false
4880 2333 Acked Read 7 Done Sent Done false
4881 2334 Acked Write 6 Done Sent Done false
4882 2335 Acked Write 7 Done Sent Done false
4883 2336 Acked Read 7 Done Sent Done false
4884 2337 Acked Read 6 Done Sent Done false
4885 2338 Acked Read 9 Done Sent Done false
4886 2339 Acked Read 5 Done Sent Done false
4887 2340 Acked Write 6 Done Sent Done false
4888 2341 Acked Write 2 Done Sent Done false
4889 2342 Acked Write 4 Done Sent Done false
4890 2343 Acked Read 4 Done Sent Done false
4891 2344 Acked Flush 0 Done Sent Done false
4892 2345 Acked Write 10 Done Sent Done false
4893 2346 Acked Read 8 Done Sent Done false
4894 2347 Acked Write 4 Done Sent Done false
4895 2348 Acked Flush 0 Done Sent Done false
4896 2349 Acked Write 9 Done Sent Done false
4897 2350 Acked Read 5 Done Sent Done false
4898 2351 Acked Flush 0 Done Sent Done false
4899 2352 Acked Read 6 Done Sent Done false
4900 2353 Acked Write 10 Done Sent Done false
4901 2354 Acked Read 1 Done Sent Done false
4902 2355 Acked Write 8 Done Sent Done false
4903 2356 Acked Write 1 Done Sent Done false
4904 2357 Acked Read 9 Done Sent Done false
4905 2358 Acked Write 9 Done Sent Done false
4906 2359 Acked Write 1 Done Sent Done false
4907 2360 Acked Read 8 Done Sent Done false
4908 2361 Acked Write 9 Done Sent Done false
4909 2362 Acked Read 1 Done Sent Done false
4910 2363 Acked Write 8 Done Sent Done false
4911 2364 Acked Read 5 Done Sent Done false
4912 2365 Acked Read 4 Done Sent Done false
4913 2366 Acked Write 4 Done Sent Done false
4914 2367 Acked Read 2 Done Sent Done false
4915 2368 Acked Write 5 Done Sent Done false
4916 2369 Acked Read 10 Done Sent Done false
4917 2370 Acked Read 7 Done Sent Done false
4918 2371 Acked Read 1 Done Sent Done false
4919 2372 Acked Read 10 Done Sent Done false
4920 2373 Acked Flush 0 Done Sent Done false
4921 2374 Acked Flush 0 Done Sent Done false
4922 2375 Acked Flush 0 Done Sent Done false
4923 2376 Acked Read 8 Done Sent Done false
4924 2377 Acked Read 6 Done Sent Done false
4925 2378 Acked Read 7 Done Sent Done false
4926 2379 Acked Write 7 Done Sent Done false
4927 2380 Acked Write 8 Done Sent Done false
4928 2381 Acked Write 3 Done Sent Done false
4929 2382 Acked Write 5 Done Sent Done false
4930 2383 Acked Write 10 Done Sent Done false
4931 2384 Acked Write 4 Done Sent Done false
4932 2385 Acked Read 6 Done Sent Done false
4933 2386 Acked Write 1 Done Sent Done false
4934 2387 Acked Read 3 Done Sent Done false
4935 2388 Acked Write 4 Done Sent Done false
4936 2389 Acked Write 1 Done Sent Done false
4937 2390 Acked Read 4 Done Sent Done false
4938 2391 Acked Read 10 Done Sent Done false
4939 2392 Acked Read 1 Done Sent Done false
4940 2393 Acked Write 8 Done Sent Done false
4941 2394 Acked Read 2 Done Sent Done false
4942 2395 Acked Read 9 Done Sent Done false
4943 2396 Acked Write 7 Done Sent Done false
4944 2397 Acked Write 8 Done Sent Done false
4945 2398 Acked Write 8 Done Sent Done false
4946 2399 Acked Read 6 Done Sent Done false
4947 2400 Acked Read 6 Done Sent Done false
4948 2401 Acked Flush 0 Done Sent Done false
4949 2402 Acked Read 5 Done Sent Done false
4950 2403 Acked Write 2 Done Sent Done false
4951 2404 Acked Write 9 Done Sent Done false
4952 2405 Acked Flush 0 Done Sent Done false
4953 2406 Acked Write 3 Done Sent Done false
4954 2407 Acked Flush 0 Done Sent Done false
4955 2408 Acked Write 2 Done Sent Done false
4956 2409 Acked Write 4 Done Sent Done false
4957 2410 Acked Write 9 Done Sent Done false
4958 2411 Acked Flush 0 Done Sent Done false
4959 2412 Acked Read 10 Done Sent Done false
4960 2413 Acked Read 5 Done Sent Done false
4961 2414 Acked Flush 0 Done Sent Done false
4962 2415 Acked Write 5 Done Sent Done false
4963 2416 Acked Read 5 Done Sent Done false
4964 2417 Acked Flush 0 Done Sent Done false
4965 2418 Acked Read 8 Done Sent Done false
4966 2419 Acked Write 8 Done Sent Done false
4967 2420 Acked Write 7 Done Sent Done false
4968 2421 Acked Read 8 Done Sent Done false
4969 2422 Acked Read 1 Done Sent Done false
4970 2423 Acked Read 6 Done Sent Done false
4971 2424 Acked Write 8 Done Sent Done false
4972 2425 Acked Read 9 Done Sent Done false
4973 2426 Acked Read 1 Done Sent Done false
4974 2427 Acked Write 5 Done Sent Done false
4975 2428 Acked Flush 0 Done Sent Done false
4976 2429 Acked Read 5 Done Sent Done false
4977 2430 Acked Flush 0 Done Sent Done false
4978 2431 Acked Write 8 Done Sent Done false
4979 2432 Acked Write 5 Done Sent Done false
4980 2433 Acked Read 2 Done Sent Done false
4981 2434 Acked Write 3 Done Sent Done false
4982 2435 Acked Read 3 Done Sent Done false
4983 2436 Acked Read 7 Done Sent Done false
4984 2437 Acked Read 8 Done Sent Done false
4985 2438 Acked Write 9 Done Sent Done false
4986 2439 Acked Read 2 Done Sent Done false
4987 2440 Acked Flush 0 Done Sent Done false
4988 2441 Acked Write 3 Done Sent Done false
4989 2442 Acked Read 4 Done Sent Done false
4990 2443 Acked Write 7 Done Sent Done false
4991 2444 Acked Write 5 Done Sent Done false
4992 2445 Acked Write 9 Done Sent Done false
4993 2446 Acked Write 8 Done Sent Done false
4994 2447 Acked Read 9 Done Sent Done false
4995 2448 Acked Write 5 Done Sent Done false
4996 2449 Acked Write 5 Done Sent Done false
4997 2450 Acked Write 7 Done Sent Done false
4998 2451 Acked Flush 0 Done Sent Done false
4999 2452 Acked Read 9 Done Sent Done false
5000 2453 Acked Read 3 Done Sent Done false
5001 2454 Acked Read 8 Done Sent Done false
5002 2455 Acked Read 10 Done Sent Done false
5003 2456 Acked Read 3 Done Sent Done false
5004 2457 Acked Read 4 Done Sent Done false
5005 2458 Acked Read 6 Done Sent Done false
5006 2459 Acked Read 7 Done Sent Done false
5007 2460 Acked Write 5 Done Sent Done false
5008 2461 Acked Read 3 Done Sent Done false
5009 2462 Acked Read 8 Done Sent Done false
5010 2463 Acked Read 8 Done Sent Done false
5011 2464 Acked Flush 0 Done Sent Done false
5012 2465 Acked Read 6 Done Sent Done false
5013 2466 Acked Flush 0 Done Sent Done false
5014 2467 Acked Flush 0 Done Sent Done false
5015 2468 Acked Flush 0 Done Sent Done false
5016 2469 Acked Write 6 Done Sent Done false
5017 2470 Acked Read 5 Done Sent Done false
5018 2471 Acked Write 6 Done Sent Done false
5019 2472 Acked Read 5 Done Sent Done false
5020 2473 Acked Flush 0 Done Sent Done false
5021 2474 Acked Write 5 Done Sent Done false
5022 2475 Acked Flush 0 Done Sent Done false
5023 2476 Acked Write 6 Done Sent Done false
5024 2477 Acked Flush 0 Done Sent Done false
5025 2478 Acked Write 7 Done Sent Done false
5026 2479 Acked Write 5 Done Sent Done false
5027 2480 Acked Read 2 Done Sent Done false
5028 2481 Acked Write 7 Done Sent Done false
5029 2482 Acked Read 8 Done Sent Done false
5030 2483 Acked Write 3 Done Sent Done false
5031 2484 Acked Write 9 Done Sent Done false
5032 2485 Acked Read 6 Done Sent Done false
5033 2486 Acked Flush 0 Done Sent Done false
5034 2487 Acked Flush 0 Done Sent Done false
5035 2488 Acked Write 10 Done Sent Done false
5036 2489 Acked Read 7 Done Sent Done false
5037 2490 Acked Read 8 Done Sent Done false
5038 2491 Acked Read 2 Done Sent Done false
5039 2492 Acked Write 2 Done Sent Done false
5040 2493 Acked Write 10 Done Sent Done false
5041 2494 Acked Write 3 Done Sent Done false
5042 2495 Acked Read 1 Done Sent Done false
5043 2496 Acked Write 3 Done Sent Done false
5044 2497 Acked Write 3 Done Sent Done false
5045 2498 Acked Write 5 Done Sent Done false
5046 2499 Acked Write 5 Done Sent Done false
5047 2500 Acked Write 5 Done Sent Done false
5048 2501 Acked Read 10 Done Sent Done false
5049 2502 Acked Read 1 Done Sent Done false
5050 2503 Acked Write 1 Done Sent Done false
5051 2504 Acked Write 2 Done Sent Done false
5052 2505 Acked Write 9 Done Sent Done false
5053 2506 Acked Read 3 Done Sent Done false
5054 2507 Acked Read 10 Done Sent Done false
5055 2508 Acked Write 10 Done Sent Done false
5056 STATES DS:0 DS:1 DS:2 TOTAL
5057 Sent 0 301 0 301
5058 Done 301 0 301 602
5059 Skipped 0 0 0 0
5060 Error 0 0 0 0
5061 Last Flush: Some(JobId(2487)) Some(JobId(2207)) Some(JobId(2487))
5062 Downstairs last five completed: 2207 2206 2205 2204 2203
5063 Upstairs last five completed: 2508 2507 2506 2505 2504
5064 CLIENT: Up:0 ds:301 act:2
50652025-01-29T05:11:41.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50662025-01-29T05:11:42.004ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50672025-01-29T05:11:42.448ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50682025-01-29T05:11:42.448ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61677) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50692025-01-29T05:11:42.449ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50702025-01-29T05:11:42.450ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50712025-01-29T05:11:42.450ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50722025-01-29T05:11:42.450ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50732025-01-29T05:11:42.450ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50742025-01-29T05:11:42.450ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50752025-01-29T05:11:42.450ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
5076 ----------------------------------------------------------------
5077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5078 Downstairs last five completed: 2509 2508 2507 2506 2505
5079 Upstairs last five completed: 2509 2508 2507 2506 2505
5080 CLIENT: Up:0 ds:0 act:3
5081 Replay: All jobs finished
5082 Main task has work to do, go find it
5083 got dsc Stop(0)
5084 stop 0
5085 Jan 29 05:11:44.696 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
5086 353, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7c40cd2c-767d-4f2a-adcb-e9e7a64f8490, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
50872025-01-29T05:11:44.697ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50882025-01-29T05:11:44.697ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
50892025-01-29T05:11:44.697ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
5090 Jan 29 05:11:44.697 INFO request completed, latency_us: 106, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9368220d-c3b3-4bca-bd86-2840e8afb0b0, remote_addr: 127.0.0.1:46995, local_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:11:44.697443178Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
5091 .0.1:9998
5092 [8810] Exited with: ExitStatus(unix_wait_status(9))
5093 [8810][0] reports Exit
5094 Jan 29 05:11:48.698 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5f4842ca-5192-41f1-aee4-7567a2d41436, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
5095 001/300 Read block 3662 len 16384
5096 002/300 Write block 4488 len 24576 data: 2 2 2 2 2 2
5097 003/300 Flush
5098 004/300 Write block 2102 len 8192 data: 2 4
5099 005/300 Flush
5100 006/300 Read block 362 len 36864
5101 007/300 Read block 2921 len 8192
5102 008/300 Write block 4350 len 20480 data: 2 2 2 2 2
5103 009/300 Read block 4365 len 4096
5104 010/300 Read block 4389 len 36864
5105 011/300 Write block 4603 len 20480 data: 4 4 2 2 2
5106 012/300 Flush
5107 013/300 Write block 4356 len 4096 data: 2
5108 014/300 Read block 1922 len 12288
5109 015/300 Read block 3856 len 4096
5110 016/300 Read block 2360 len 12288
5111 017/300 Read block 3555 len 20480
5112 018/300 Write block 2278 len 12288 data: 2 2 2
5113 019/300 Write block 1337 len 20480 data: 3 3 3 3 3
5114 020/300 Flush
5115 021/300 Read block 2101 len 32768
5116 022/300 Read block 4779 len 8192
5117 023/300 Write block 2139 len 40960 data: 4 4 4 3 3 3 2 2 3 3
5118 024/300 Read block 3492 len 36864
5119 025/300 Read block 2968 len 36864
5120 026/300 Write block 4661 len 24576 data: 2 2 2 2 2 2
5121 027/300 Read block 4967 len 36864
5122 028/300 Write block 2141 len 36864 data: 5 4 4 4 3 3 4 4 3
5123 029/300 Write block 2240 len 36864 data: 2 3 3 3 3 3 3 2 2
5124 030/300 Read block 3709 len 20480
5125 031/300 Read block 2053 len 16384
5126 032/300 Read block 225 len 12288
5127 033/300 Write block 178 len 36864 data: 3 2 2 2 2 2 2 2 2
5128 034/300 Read block 3664 len 16384
5129 035/300 Read block 2400 len 4096
5130 036/300 Write block 4176 len 4096 data: 3
5131 037/300 Read block 1260 len 28672
5132 038/300 Write block 4459 len 16384 data: 3 3 3 3
5133 039/300 Read block 3659 len 4096
5134 040/300 Write block 2084 len 8192 data: 2 2
5135 041/300 Write block 4238 len 16384 data: 3 3 3 2
5136 042/300 Write block 725 len 32768 data: 2 2 2 2 2 2 3 3
5137 043/300 Read block 1533 len 40960
5138 044/300 Read block 2694 len 12288
5139 045/300 Read block 2237 len 28672
5140 046/300 Flush
5141 047/300 Read block 3075 len 24576
5142 048/300 Flush
5143 049/300 Read block 310 len 40960
5144 050/300 Read block 3296 len 20480
5145 051/300 Read block 4020 len 8192
5146 052/300 Read block 3192 len 4096
5147 053/300 Read block 2620 len 36864
5148 054/300 Read block 288 len 28672
5149 055/300 Read block 3418 len 36864
5150 056/300 Write block 507 len 36864 data: 4 3 3 4 3 3 3 3 4
5151 057/300 Write block 4451 len 28672 data: 3 3 4 4 4 3 3
5152 058/300 Write block 3478 len 24576 data: 2 2 2 2 2 2
5153 059/300 Read block 4162 len 20480
5154 060/300 Read block 3569 len 32768
5155 061/300 Read block 400 len 24576
5156 062/300 Read block 3343 len 20480
5157 063/300 Read block 741 len 36864
5158 064/300 Read block 4121 len 16384
5159 065/300 Write block 473 len 36864 data: 3 3 3 3 3 4 4 4 4
5160 066/300 Read block 1061 len 36864
5161 067/300 Write block 2692 len 40960 data: 2 2 2 3 3 3 3 3 2 2
5162 068/300 Write block 388 len 16384 data: 2 2 2 2
5163 069/300 Read block 4330 len 40960
5164 070/300 Write block 3608 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5165 071/300 Read block 1912 len 20480
5166 072/300 Read block 10 len 40960
5167 073/300 Read block 2252 len 4096
5168 074/300 Read block 1784 len 32768
5169 075/300 Flush
5170 076/300 Write block 322 len 32768 data: 2 2 2 2 2 2 2 2
5171 077/300 Write block 1680 len 24576 data: 3 3 4 3 3 3
5172 078/300 Write block 4800 len 12288 data: 2 2 3
5173 079/300 Read block 552 len 40960
5174 080/300 Read block 2413 len 28672
5175 081/300 Write block 2266 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5176 082/300 Read block 1354 len 36864
5177 083/300 Write block 1326 len 12288 data: 3 3 3
5178 084/300 Flush
5179 085/300 Read block 372 len 40960
5180 086/300 Write block 3848 len 20480 data: 2 2 2 2 2
5181 087/300 Write block 4928 len 36864 data: 2 2 2 2 2 2 2 2 2
5182 088/300 Write block 3765 len 32768 data: 2 2 3 3 2 2 3 3
5183 089/300 Read block 1683 len 8192
5184 090/300 Write block 4884 len 28672 data: 2 2 2 2 2 2 2
5185 091/300 Read block 4169 len 36864
5186 092/300 Read block 3772 len 8192
5187 093/300 Flush
5188 094/300 Write block 2751 len 40960 data: 4 4 4 3 4 4 4 4 4 4
5189 095/300 Read block 1006 len 8192
5190 096/300 Write block 4719 len 8192 data: 3 2
5191 097/300 Read block 666 len 20480
5192 098/300 Flush
5193 099/300 Read block 2856 len 40960
5194 100/300 Read block 4561 len 28672
5195 101/300 Read block 2576 len 40960
5196 102/300 Write block 3144 len 40960 data: 2 2 2 3 3 3 2 2 2 2
5197 103/300 Flush
5198 104/300 Read block 3511 len 8192
5199 105/300 Write block 3590 len 16384 data: 2 2 2 2
5200 106/300 Flush
5201 107/300 Read block 1629 len 12288
5202 108/300 Write block 1248 len 16384 data: 2 2 2 3
5203 109/300 Read block 3446 len 16384
5204 110/300 Read block 3154 len 36864
5205 111/300 Write block 2147 len 20480 data: 5 5 4 3 3
5206 112/300 Write block 766 len 4096 data: 2
5207 113/300 Read block 293 len 20480
5208 114/300 Read block 351 len 8192
5209 115/300 Write block 1008 len 24576 data: 2 3 3 3 3 3
5210 116/300 Read block 2241 len 40960
5211 117/300 Write block 420 len 8192 data: 2 2
5212 118/300 Write block 3474 len 12288 data: 3 2 2
5213 119/300 Write block 2438 len 28672 data: 3 2 2 2 2 2 2
5214 120/300 Read block 3212 len 28672
5215 121/300 Flush
5216 122/300 Read block 3763 len 36864
5217 123/300 Write block 3801 len 32768 data: 2 2 2 2 3 3 3 3
5218 124/300 Read block 1101 len 8192
5219 125/300 Read block 3080 len 8192
5220 126/300 Read block 1283 len 36864
5221 127/300 Flush
5222 128/300 Write block 1637 len 12288 data: 2 2 2
5223 129/300 Write block 4427 len 8192 data: 3 3
5224 130/300 Write block 847 len 36864 data: 3 3 2 2 2 2 2 2 2
5225 131/300 Read block 777 len 16384
5226 132/300 Read block 429 len 12288
5227 133/300 Read block 304 len 24576
5228 134/300 Read block 2167 len 40960
5229 135/300 Write block 3226 len 36864 data: 2 2 3 3 3 4 4 4 3
5230 136/300 Read block 1843 len 20480
5231 137/300 Write block 4279 len 40960 data: 3 4 4 4 4 4 2 2 2 2
5232 138/300 Read block 211 len 4096
5233 139/300 Write block 2089 len 24576 data: 2 2 2 2 2 2
5234 140/300 Read block 925 len 28672
5235 141/300 Read block 1724 len 16384
5236 142/300 Read block 3078 len 12288
5237 143/300 Read block 1083 len 20480
5238 144/300 Read block 790 len 12288
5239 145/300 Read block 965 len 40960
5240 146/300 Write block 3129 len 40960 data: 3 4 4 4 3 3 4 4 4 4
5241 147/300 Write block 4743 len 36864 data: 2 2 2 2 2 2 2 2 2
5242 148/300 Write block 3118 len 8192 data: 4 4
5243 149/300 Read block 3145 len 20480
5244 150/300 Flush
5245 151/300 Read block 4914 len 20480
5246 152/300 Write block 3476 len 36864 data: 3 2 3 3 3 3 3 3 2
5247 153/300 Flush
5248 154/300 Read block 4089 len 16384
5249 155/300 Write block 1860 len 36864 data: 2 2 2 2 2 2 2 2 2
5250 156/300 Read block 2139 len 20480
5251 157/300 Write block 3110 len 28672 data: 4 3 3 2 2 3 3
5252 158/300 Write block 285 len 4096 data: 3
5253 159/300 Read block 690 len 40960
5254 160/300 Write block 3716 len 4096 data: 3
5255 161/300 Write block 1035 len 8192 data: 2 2
5256 162/300 Read block 3097 len 4096
5257 163/300 Read block 2556 len 24576
5258 164/300 Read block 3094 len 32768
5259 165/300 Write block 4630 len 36864 data: 3 3 2 2 2 2 2 2 3
5260 166/300 Read block 167 len 24576
5261 167/300 Write block 4219 len 36864 data: 3 4 3 3 3 3 3 2 3
5262 168/300 Read block 3647 len 4096
5263 169/300 Write block 1560 len 20480 data: 4 3 3 3 2
5264 170/300 Write block 4967 len 36864 data: 2 2 2 2 2 2 2 3 3
5265 171/300 Flush
5266 172/300 Write block 1083 len 16384 data: 2 2 2 2
5267 173/300 Read block 48 len 36864
5268 174/300 Read block 4670 len 4096
5269 175/300 Write block 1099 len 24576 data: 2 3 2 2 2 2
5270 176/300 Read block 4702 len 24576
5271 177/300 Read block 17 len 36864
5272 178/300 Write block 4730 len 4096 data: 2
5273 179/300 Read block 1868 len 12288
5274 180/300 Read block 1508 len 24576
5275 181/300 Read block 4988 len 20480
5276 182/300 Write block 255 len 12288 data: 3 3 3
5277 183/300 Read block 2032 len 24576
5278 184/300 Read block 406 len 40960
5279 185/300 Write block 2679 len 12288 data: 2 2 2
5280 186/300 Read block 626 len 24576
5281 187/300 Flush
5282 188/300 Read block 2035 len 4096
5283 189/300 Write block 1083 len 4096 data: 3
5284 190/300 Write block 955 len 16384 data: 3 2 2 2
5285 191/300 Write block 4401 len 16384 data: 2 2 2 2
5286 192/300 Read block 827 len 32768
5287 193/300 Read block 2176 len 8192
5288 194/300 Read block 1111 len 40960
5289 195/300 Write block 744 len 32768 data: 4 5 3 3 2 3 3 3
5290 196/300 Read block 939 len 36864
5291 197/300 Read block 1601 len 32768
5292 198/300 Write block 4418 len 8192 data: 3 3
5293 199/300 Read block 2936 len 4096
5294 200/300 Write block 4986 len 20480 data: 3 3 3 3 3
5295 201/300 Read block 3319 len 28672
5296 202/300 Flush
5297 203/300 Read block 4767 len 12288
5298 204/300 Flush
5299 205/300 Write block 1299 len 28672 data: 5 5 5 5 4 3 3
5300 206/300 Read block 1373 len 12288
5301 207/300 Write block 3947 len 36864 data: 4 4 3 3 2 2 2 4 4
5302 208/300 Write block 749 len 8192 data: 4 4
5303 209/300 Write block 1955 len 12288 data: 3 3 3
5304 210/300 Write block 586 len 36864 data: 2 3 3 4 4 3 3 3 3
5305 211/300 Read block 1645 len 8192
5306 212/300 Read block 831 len 4096
5307 213/300 Flush
5308 214/300 Read block 4930 len 28672
5309 215/300 Read block 4324 len 8192
5310 216/300 Flush
5311 217/300 Read block 452 len 24576
5312 218/300 Read block 3464 len 4096
5313 219/300 Read block 1297 len 16384
5314 220/300 Read block 3348 len 28672
5315 221/300 Write block 4652 len 28672 data: 3 2 2 2 2 2 2
5316 222/300 Flush
5317 223/300 Write block 446 len 12288 data: 2 2 2
5318 224/300 Read block 2981 len 8192
5319 225/300 Read block 3015 len 24576
5320 226/300 Read block 3489 len 12288
5321 227/300 Write block 507 len 8192 data: 5 4
5322 228/300 Write block 1133 len 24576 data: 2 2 2 2 2 2
5323 229/300 Flush
5324 230/300 Write block 3430 len 28672 data: 3 2 2 2 2 2 2
5325 231/300 Write block 2923 len 36864 data: 3 3 3 3 3 3 3 3 3
5326 232/300 Read block 4585 len 24576
5327 233/300 Write block 3752 len 4096 data: 3
5328 234/300 Write block 4569 len 8192 data: 2 2
5329 235/300 Write block 2855 len 36864 data: 3 3 3 3 3 3 2 2 2
5330 236/300 Read block 3250 len 16384
5331 237/300 Read block 1385 len 32768
5332 238/300 Read block 180 len 16384
5333 239/300 Flush
5334 240/300 Read block 4381 len 40960
5335 241/300 Read block 731 len 40960
5336 242/300 Read block 2243 len 32768
5337 243/300 Read block 3827 len 16384
5338 244/300 Write block 1160 len 24576 data: 2 2 2 2 2 2
5339 245/300 Write block 808 len 36864 data: 3 3 2 2 3 3 3 3 3
5340 246/300 Read block 2211 len 20480
5341 247/300 Write block 778 len 28672 data: 3 3 3 3 3 3 2
5342 248/300 Read block 3234 len 12288
5343 249/300 Read block 3759 len 8192
5344 250/300 Write block 4217 len 36864 data: 2 2 4 5 4 4 4 4 4
5345 251/300 Write block 2163 len 4096 data: 2
5346 252/300 Read block 1096 len 8192
5347 253/300 Read block 4184 len 20480
5348 254/300 Write block 2576 len 20480 data: 4 3 3 2 2
5349 255/300 Flush
5350 256/300 Read block 3494 len 24576
5351 257/300 Flush
5352 258/300 Read block 4815 len 20480
5353 259/300 Write block 2005 len 8192 data: 4 5
5354 260/300 Read block 3332 len 20480
5355 261/300 Read block 238 len 24576
5356 262/300 Write block 1811 len 12288 data: 2 2 2
5357 263/300 Write block 4526 len 4096 data: 2
5358 264/300 Read block 4244 len 16384
5359 265/300 Write block 931 len 12288 data: 2 2 2
5360 266/300 Write block 853 len 20480 data: 3 3 3 2 2
5361 267/300 Read block 1436 len 32768
5362 268/300 Read block 133 len 40960
5363 269/300 Write block 1339 len 36864 data: 4 4 4 3 2 2 2 2 2
5364 270/300 Read block 385 len 4096
5365 271/300 Read block 3601 len 24576
5366 272/300 Flush
5367 273/300 Read block 199 len 36864
5368 274/300 Read block 966 len 20480
5369 275/300 Read block 823 len 16384
5370 276/300 Flush
5371 277/300 Write block 118 len 32768 data: 3 3 3 3 3 3 2 2
5372 278/300 Read block 3307 len 24576
5373 279/300 Read block 4540 len 40960
5374 280/300 Read block 1636 len 4096
5375 281/300 Read block 2894 len 12288
5376 282/300 Flush
5377 283/300 Write block 4525 len 28672 data: 2 3 2 3 3 3 3
5378 284/300 Write block 3202 len 4096 data: 2
5379 285/300 Read block 4686 len 24576
5380 286/300 Write block 2003 len 4096 data: 4
5381 287/300 Write block 2972 len 20480 data: 3 2 2 2 4
5382 288/300 Write block 3766 len 20480 data: 3 4 4 3 3
5383 289/300 Read block 1031 len 36864
5384 290/300 Write block 1152 len 4096 data: 2
5385 291/300 Write block 4895 len 24576 data: 2 2 2 2 2 2
5386 292/300 Write block 3089 len 24576 data: 2 3 3 3 3 3
5387 293/300 Write block 4627 len 20480 data: 3 3 3 4 4
5388 294/300 Read block 3538 len 8192
5389 295/300 Read block 1130 len 40960
5390 296/300 Write block 1973 len 8192 data: 4 4
5391 297/300 Write block 87 len 20480 data: 4 4 3 2 2
5392 298/300 Read block 863 len 28672
5393 299/300 Read block 1808 len 4096
5394 300/300 Read block 3878 len 8192
5395 301/300 Read block 760 len 8192
5396 Main task has work to do, go find it
5397 got dsc Start(0)
5398 start 0
5399 Jan 29 05:11:48.943 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0da3b8df-d32c-46b6-9452-6003626fb071, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
5400 Replay: started 0, returned:Ok(())
5401 ----------------------------------------------------------------
5402 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5403 States: Offline Active Active
5404 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5405 2510 Acked Read 4 Sent Done Done false
5406 2511 Acked Write 6 Sent Done Done false
5407 2512 Acked Flush 0 Sent Done Done false
5408 2513 Acked Write 2 Sent Done Done false
5409 2514 Acked Flush 0 Sent Done Done false
5410 2515 Acked Read 9 Sent Done Done false
5411 2516 Acked Read 2 Sent Done Done false
5412 2517 Acked Write 5 Sent Done Done false
5413 2518 Acked Read 1 Sent Done Done false
5414 2519 Acked Read 9 Sent Done Done false
5415 2520 Acked Write 5 Sent Done Done false
5416 2521 Acked Flush 0 Sent Done Done false
5417 2522 Acked Write 1 Sent Done Done false
5418 2523 Acked Read 3 Sent Done Done false
5419 2524 Acked Read 1 Sent Done Done false
5420 2525 Acked Read 3 Sent Done Done false
5421 2526 Acked Read 5 Sent Done Done false
5422 2527 Acked Write 3 Sent Done Done false
5423 2528 Acked Write 5 Sent Done Done false
5424 2529 Acked Flush 0 Sent Done Done false
5425 2530 Acked Read 8 Sent Done Done false
5426 2531 Acked Read 2 Sent Done Done false
5427 2532 Acked Write 10 Sent Done Done false
5428 2533 Acked Read 9 Sent Done Done false
5429 2534 Acked Read 9 Sent Done Done false
5430 2535 Acked Write 6 Sent Done Done false
5431 2536 Acked Read 9 Sent Done Done false
5432 2537 Acked Write 9 Sent Done Done false
5433 2538 Acked Write 9 Sent Done Done false
5434 2539 Acked Read 5 Sent Done Done false
5435 2540 Acked Read 4 Sent Done Done false
5436 2541 Acked Read 3 Sent Done Done false
5437 2542 Acked Write 9 Sent Done Done false
5438 2543 Acked Read 4 Sent Done Done false
5439 2544 Acked Read 1 Sent Done Done false
5440 2545 Acked Write 1 Sent Done Done false
5441 2546 Acked Read 7 Sent Done Done false
5442 2547 Acked Write 4 Sent Done Done false
5443 2548 Acked Read 1 Sent Done Done false
5444 2549 Acked Write 2 Sent Done Done false
5445 2550 Acked Write 4 Sent Done Done false
5446 2551 Acked Write 8 Sent Done Done false
5447 2552 Acked Read 10 Sent Done Done false
5448 2553 Acked Read 3 Sent Done Done false
5449 2554 Acked Read 7 Sent Done Done false
5450 2555 Acked Flush 0 Sent Done Done false
5451 2556 Acked Read 6 Sent Done Done false
5452 2557 Acked Flush 0 Sent Done Done false
5453 2558 Acked Read 10 Sent Done Done false
5454 2559 Acked Read 5 Sent Done Done false
5455 2560 Acked Read 2 Sent Done Done false
5456 2561 Acked Read 1 Sent Done Done false
5457 2562 Acked Read 9 Sent Done Done false
5458 2563 Acked Read 7 Sent Done Done false
5459 2564 Acked Read 9 Sent Done Done false
5460 2565 Acked Write 9 Sent Done Done false
5461 2566 Acked Write 7 Sent Done Done false
5462 2567 Acked Write 6 Sent Done Done false
5463 2568 Acked Read 5 Sent Done Done false
5464 2569 Acked Read 8 Sent Done Done false
5465 2570 Acked Read 6 Sent Done Done false
5466 2571 Acked Read 5 Sent Done Done false
5467 2572 Acked Read 9 Sent Done Done false
5468 2573 Acked Read 4 Sent Done Done false
5469 2574 Acked Write 9 Sent Done Done false
5470 2575 Acked Read 9 Sent Done Done false
5471 2576 Acked Write 10 Sent Done Done false
5472 2577 Acked Write 4 Sent Done Done false
5473 2578 Acked Read 10 Sent Done Done false
5474 2579 Acked Write 10 Sent Done Done false
5475 2580 Acked Read 5 Sent Done Done false
5476 2581 Acked Read 10 Sent Done Done false
5477 2582 Acked Read 1 Sent Done Done false
5478 2583 Acked Read 8 Sent Done Done false
5479 2584 Acked Flush 0 Sent Done Done false
5480 2585 Acked Write 8 Sent Done Done false
5481 2586 Acked Write 6 Sent Done Done false
5482 2587 Acked Write 3 Sent Done Done false
5483 2588 Acked Read 10 Sent Done Done false
5484 2589 Acked Read 7 Sent Done Done false
5485 2590 Acked Write 10 Sent Done Done false
5486 2591 Acked Read 9 Sent Done Done false
5487 2592 Acked Write 3 Sent Done Done false
5488 2593 Acked Flush 0 Sent Done Done false
5489 2594 Acked Read 10 Sent Done Done false
5490 2595 Acked Write 5 Sent Done Done false
5491 2596 Acked Write 9 Sent Done Done false
5492 2597 Acked Write 8 Sent Done Done false
5493 2598 Acked Read 2 Sent Done Done false
5494 2599 Acked Write 7 Sent Done Done false
5495 2600 Acked Read 9 Sent Done Done false
5496 2601 Acked Read 2 Sent Done Done false
5497 2602 Acked Flush 0 Sent Done Done false
5498 2603 Acked Write 10 Sent Done Done false
5499 2604 Acked Read 2 Sent Done Done false
5500 2605 Acked Write 2 Sent Done Done false
5501 2606 Acked Read 5 Sent Done Done false
5502 2607 Acked Flush 0 Sent Done Done false
5503 2608 Acked Read 10 Sent Done Done false
5504 2609 Acked Read 7 Sent Done Done false
5505 2610 Acked Read 10 Sent Done Done false
5506 2611 Acked Write 10 Sent Done Done false
5507 2612 Acked Flush 0 Sent Done Done false
5508 2613 Acked Read 2 Sent Done Done false
5509 2614 Acked Write 4 Sent Done Done false
5510 2615 Acked Flush 0 Sent Done Done false
5511 2616 Acked Read 3 Sent Done Done false
5512 2617 Acked Write 4 Sent Done Done false
5513 2618 Acked Read 4 Sent Done Done false
5514 2619 Acked Read 9 Sent Done Done false
5515 2620 Acked Write 5 Sent Done Done false
5516 2621 Acked Write 1 Sent Done Done false
5517 2622 Acked Read 5 Sent Done Done false
5518 2623 Acked Read 2 Sent Done Done false
5519 2624 Acked Write 6 Sent Done Done false
5520 2625 Acked Read 10 Sent Done Done false
5521 2626 Acked Write 2 Sent Done Done false
5522 2627 Acked Write 3 Sent Done Done false
5523 2628 Acked Write 7 Sent Done Done false
5524 2629 Acked Read 7 Sent Done Done false
5525 2630 Acked Flush 0 Sent Done Done false
5526 2631 Acked Read 9 Sent Done Done false
5527 2632 Acked Write 8 Sent Done Done false
5528 2633 Acked Read 2 Sent Done Done false
5529 2634 Acked Read 2 Sent Done Done false
5530 2635 Acked Read 9 Sent Done Done false
5531 2636 Acked Flush 0 Sent Done Done false
5532 2637 Acked Write 3 Sent Done Done false
5533 2638 Acked Write 2 Sent Done Done false
5534 2639 Acked Write 9 Sent Done Done false
5535 2640 Acked Read 4 Sent Done Done false
5536 2641 Acked Read 3 Sent Done Done false
5537 2642 Acked Read 6 Sent Done Done false
5538 2643 Acked Read 10 Sent Done Done false
5539 2644 Acked Write 9 Sent Done Done false
5540 2645 Acked Read 5 Sent Done Done false
5541 2646 Acked Write 10 Sent Done Done false
5542 2647 Acked Read 1 Sent Done Done false
5543 2648 Acked Write 6 Sent Done Done false
5544 2649 Acked Read 7 Sent Done Done false
5545 2650 Acked Read 4 Sent Done Done false
5546 2651 Acked Read 3 Sent Done Done false
5547 2652 Acked Read 5 Sent Done Done false
5548 2653 Acked Read 3 Sent Done Done false
5549 2654 Acked Read 10 Sent Done Done false
5550 2655 Acked Write 10 Sent Done Done false
5551 2656 Acked Write 9 Sent Done Done false
5552 2657 Acked Write 2 Sent Done Done false
5553 2658 Acked Read 5 Sent Done Done false
5554 2659 Acked Flush 0 Sent Done Done false
5555 2660 Acked Read 5 Sent Done Done false
5556 2661 Acked Write 9 Sent Done Done false
5557 2662 Acked Flush 0 Sent Done Done false
5558 2663 Acked Read 4 Sent Done Done false
5559 2664 Acked Write 9 Sent Done Done false
5560 2665 Acked Read 5 Sent Done Done false
5561 2666 Acked Write 7 Sent Done Done false
5562 2667 Acked Write 1 Sent Done Done false
5563 2668 Acked Read 10 Sent Done Done false
5564 2669 Acked Write 1 Sent Done Done false
5565 2670 Acked Write 2 Sent Done Done false
5566 2671 Acked Read 1 Sent Done Done false
5567 2672 Acked Read 6 Sent Done Done false
5568 2673 Acked Read 8 Sent Done Done false
5569 2674 Acked Write 9 Sent Done Done false
5570 2675 Acked Read 6 Sent Done Done false
5571 2676 Acked Write 9 Sent Done Done false
5572 2677 Acked Read 1 Sent Done Done false
5573 2678 Acked Write 5 Sent Done Done false
5574 2679 Acked Write 9 Sent Done Done false
5575 2680 Acked Flush 0 Sent Done Done false
5576 2681 Acked Write 4 Sent Done Done false
5577 2682 Acked Read 9 Sent Done Done false
5578 2683 Acked Read 1 Sent Done Done false
5579 2684 Acked Write 6 Sent Done Done false
5580 2685 Acked Read 6 Sent Done Done false
5581 2686 Acked Read 9 Sent Done Done false
5582 2687 Acked Write 1 Sent Done Done false
5583 2688 Acked Read 3 Sent Done Done false
5584 2689 Acked Read 6 Sent Done Done false
5585 2690 Acked Read 5 Sent Done Done false
5586 2691 Acked Write 3 Sent Done Done false
5587 2692 Acked Read 6 Sent Done Done false
5588 2693 Acked Read 10 Sent Done Done false
5589 2694 Acked Write 3 Sent Done Done false
5590 2695 Acked Read 6 Sent Done Done false
5591 2696 Acked Flush 0 Sent Done Done false
5592 2697 Acked Read 1 Sent Done Done false
5593 2698 Acked Write 1 Sent Done Done false
5594 2699 Acked Write 4 Sent Done Done false
5595 2700 Acked Write 4 Sent Done Done false
5596 2701 Acked Read 8 Sent Done Done false
5597 2702 Acked Read 2 Sent Done Done false
5598 2703 Acked Read 10 Sent Done Done false
5599 2704 Acked Write 8 Sent Done Done false
5600 2705 Acked Read 9 Sent Done Done false
5601 2706 Acked Read 8 Sent Done Done false
5602 2707 Acked Write 2 Sent Done Done false
5603 2708 Acked Read 1 Sent Done Done false
5604 2709 Acked Write 5 Sent Done Done false
5605 2710 Acked Read 7 Sent Done Done false
5606 2711 Acked Flush 0 Sent Done Done false
5607 2712 Acked Read 3 Sent Done Done false
5608 2713 Acked Flush 0 Sent Done Done false
5609 2714 Acked Write 7 Sent Done Done false
5610 2715 Acked Read 3 Sent Done Done false
5611 2716 Acked Write 9 Sent Done Done false
5612 2717 Acked Write 2 Sent Done Done false
5613 2718 Acked Write 3 Sent Done Done false
5614 2719 Acked Write 9 Sent Done Done false
5615 2720 Acked Read 2 Sent Done Done false
5616 2721 Acked Read 1 Sent Done Done false
5617 2722 Acked Flush 0 Sent Done Done false
5618 2723 Acked Read 7 Sent Done Done false
5619 2724 Acked Read 2 Sent Done Done false
5620 2725 Acked Flush 0 Sent Done Done false
5621 2726 Acked Read 6 Sent Done Done false
5622 2727 Acked Read 1 Sent Done Done false
5623 2728 Acked Read 4 Sent Done Done false
5624 2729 Acked Read 7 Sent Done Done false
5625 2730 Acked Write 7 Sent Done Done false
5626 2731 Acked Flush 0 Sent Done Done false
5627 2732 Acked Write 3 Sent Done Done false
5628 2733 Acked Read 2 Sent Done Done false
5629 2734 Acked Read 6 Sent Done Done false
5630 2735 Acked Read 3 Sent Done Done false
5631 2736 Acked Write 2 Sent Done Done false
5632 2737 Acked Write 6 Sent Done Done false
5633 2738 Acked Flush 0 Sent Done Done false
5634 2739 Acked Write 7 Sent Done Done false
5635 2740 Acked Write 9 Sent Done Done false
5636 2741 Acked Read 6 Sent Done Done false
5637 2742 Acked Write 1 Sent Done Done false
5638 2743 Acked Write 2 Sent Done Done false
5639 2744 Acked Write 9 Sent Done Done false
5640 2745 Acked Read 4 Sent Done Done false
5641 2746 Acked Read 8 Sent Done Done false
5642 2747 Acked Read 4 Sent Done Done false
5643 2748 Acked Flush 0 Sent Done Done false
5644 2749 Acked Read 10 Sent Done Done false
5645 2750 Acked Read 10 Sent Done Done false
5646 2751 Acked Read 8 Sent Done Done false
5647 2752 Acked Read 4 Sent Done Done false
5648 2753 Acked Write 6 Sent Done Done false
5649 2754 Acked Write 9 Sent Done Done false
5650 2755 Acked Read 5 Sent Done Done false
5651 2756 Acked Write 7 Sent Done Done false
5652 2757 Acked Read 3 Sent Done Done false
5653 2758 Acked Read 2 Sent Done Done false
5654 2759 Acked Write 9 Sent Done Done false
5655 2760 Acked Write 1 Sent Done Done false
5656 2761 Acked Read 2 Sent Done Done false
5657 2762 Acked Read 5 Sent Done Done false
5658 2763 Acked Write 5 Sent Done Done false
5659 2764 Acked Flush 0 Sent Done Done false
5660 2765 Acked Read 6 Sent Done Done false
5661 2766 Acked Flush 0 Sent Done Done false
5662 2767 Acked Read 5 Sent Done Done false
5663 2768 Acked Write 2 Sent Done Done false
5664 2769 Acked Read 5 Sent Done Done false
5665 2770 Acked Read 6 Sent Done Done false
5666 2771 Acked Write 3 Sent Done Done false
5667 2772 Acked Write 1 Sent Done Done false
5668 2773 Acked Read 4 Sent Done Done false
5669 2774 Acked Write 3 Sent Done Done false
5670 2775 Acked Write 5 Sent Done Done false
5671 2776 Acked Read 8 Sent Done Done false
5672 2777 Acked Read 10 Sent Done Done false
5673 2778 Acked Write 9 Sent Done Done false
5674 2779 Acked Read 1 Sent Done Done false
5675 2780 Acked Read 6 Sent Done Done false
5676 2781 Acked Flush 0 Sent Done Done false
5677 2782 Acked Read 9 Sent Done Done false
5678 2783 Acked Read 5 Sent Done Done false
5679 2784 Acked Read 4 Sent Done Done false
5680 2785 Acked Flush 0 Sent Done Done false
5681 2786 Acked Write 8 Sent Done Done false
5682 2787 Acked Read 6 Sent Done Done false
5683 2788 Acked Read 10 Sent Done Done false
5684 2789 Acked Read 1 Sent Done Done false
5685 2790 Acked Read 3 Sent Done Done false
5686 2791 Acked Flush 0 Sent Done Done false
5687 2792 Acked Write 7 Sent Done Done false
5688 2793 Acked Write 1 Sent Done Done false
5689 2794 Acked Read 6 Sent Done Done false
5690 2795 Acked Write 1 Sent Done Done false
5691 2796 Acked Write 5 Sent Done Done false
5692 2797 Acked Write 5 Sent Done Done false
5693 2798 Acked Read 9 Sent Done Done false
5694 2799 Acked Write 1 Sent Done Done false
5695 2800 Acked Write 6 Sent Done Done false
5696 2801 Acked Write 6 Sent Done Done false
5697 2802 Acked Write 5 Sent Done Done false
5698 2803 Acked Read 2 Sent Done Done false
5699 2804 Acked Read 10 Sent Done Done false
5700 2805 Acked Write 2 Sent Done Done false
5701 2806 Acked Write 5 Sent Done Done false
5702 2807 Acked Read 7 Sent Done Done false
5703 2808 Acked Read 1 Sent Done Done false
5704 2809 Acked Read 2 Sent Done Done false
5705 2810 Acked Read 2 Sent Done Done false
5706 STATES DS:0 DS:1 DS:2 TOTAL
5707 Sent 301 0 0 301
5708 Done 0 301 301 602
5709 Skipped 0 0 0 0
5710 Error 0 0 0 0
5711 Last Flush: Some(JobId(2509)) Some(JobId(2791)) Some(JobId(2791))
5712 Downstairs last five completed: 2509 2508 2507 2506 2505
5713 Upstairs last five completed: 2810 2809 2808 2807 2806
5714 CLIENT: Up:0 ds:301 act:2
57152025-01-29T05:11:49.445ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
57162025-01-29T05:11:50.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
57172025-01-29T05:11:51.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
57182025-01-29T05:11:52.022ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
5719 ----------------------------------------------------------------
5720 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5721 States: Offline Active Active
5722 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5723 2510 Acked Read 4 Sent Done Done false
5724 2511 Acked Write 6 Sent Done Done false
5725 2512 Acked Flush 0 Sent Done Done false
5726 2513 Acked Write 2 Sent Done Done false
5727 2514 Acked Flush 0 Sent Done Done false
5728 2515 Acked Read 9 Sent Done Done false
5729 2516 Acked Read 2 Sent Done Done false
5730 2517 Acked Write 5 Sent Done Done false
5731 2518 Acked Read 1 Sent Done Done false
5732 2519 Acked Read 9 Sent Done Done false
5733 2520 Acked Write 5 Sent Done Done false
5734 2521 Acked Flush 0 Sent Done Done false
5735 2522 Acked Write 1 Sent Done Done false
5736 2523 Acked Read 3 Sent Done Done false
5737 2524 Acked Read 1 Sent Done Done false
5738 2525 Acked Read 3 Sent Done Done false
5739 2526 Acked Read 5 Sent Done Done false
5740 2527 Acked Write 3 Sent Done Done false
5741 2528 Acked Write 5 Sent Done Done false
5742 2529 Acked Flush 0 Sent Done Done false
5743 2530 Acked Read 8 Sent Done Done false
5744 2531 Acked Read 2 Sent Done Done false
5745 2532 Acked Write 10 Sent Done Done false
5746 2533 Acked Read 9 Sent Done Done false
5747 2534 Acked Read 9 Sent Done Done false
5748 2535 Acked Write 6 Sent Done Done false
5749 2536 Acked Read 9 Sent Done Done false
5750 2537 Acked Write 9 Sent Done Done false
5751 2538 Acked Write 9 Sent Done Done false
5752 2539 Acked Read 5 Sent Done Done false
5753 2540 Acked Read 4 Sent Done Done false
5754 2541 Acked Read 3 Sent Done Done false
5755 2542 Acked Write 9 Sent Done Done false
5756 2543 Acked Read 4 Sent Done Done false
5757 2544 Acked Read 1 Sent Done Done false
5758 2545 Acked Write 1 Sent Done Done false
5759 2546 Acked Read 7 Sent Done Done false
5760 2547 Acked Write 4 Sent Done Done false
5761 2548 Acked Read 1 Sent Done Done false
5762 2549 Acked Write 2 Sent Done Done false
5763 2550 Acked Write 4 Sent Done Done false
5764 2551 Acked Write 8 Sent Done Done false
5765 2552 Acked Read 10 Sent Done Done false
5766 2553 Acked Read 3 Sent Done Done false
5767 2554 Acked Read 7 Sent Done Done false
5768 2555 Acked Flush 0 Sent Done Done false
5769 2556 Acked Read 6 Sent Done Done false
5770 2557 Acked Flush 0 Sent Done Done false
5771 2558 Acked Read 10 Sent Done Done false
5772 2559 Acked Read 5 Sent Done Done false
5773 2560 Acked Read 2 Sent Done Done false
5774 2561 Acked Read 1 Sent Done Done false
5775 2562 Acked Read 9 Sent Done Done false
5776 2563 Acked Read 7 Sent Done Done false
5777 2564 Acked Read 9 Sent Done Done false
5778 2565 Acked Write 9 Sent Done Done false
5779 2566 Acked Write 7 Sent Done Done false
5780 2567 Acked Write 6 Sent Done Done false
5781 2568 Acked Read 5 Sent Done Done false
5782 2569 Acked Read 8 Sent Done Done false
5783 2570 Acked Read 6 Sent Done Done false
5784 2571 Acked Read 5 Sent Done Done false
5785 2572 Acked Read 9 Sent Done Done false
5786 2573 Acked Read 4 Sent Done Done false
5787 2574 Acked Write 9 Sent Done Done false
5788 2575 Acked Read 9 Sent Done Done false
5789 2576 Acked Write 10 Sent Done Done false
5790 2577 Acked Write 4 Sent Done Done false
5791 2578 Acked Read 10 Sent Done Done false
5792 2579 Acked Write 10 Sent Done Done false
5793 2580 Acked Read 5 Sent Done Done false
5794 2581 Acked Read 10 Sent Done Done false
5795 2582 Acked Read 1 Sent Done Done false
5796 2583 Acked Read 8 Sent Done Done false
5797 2584 Acked Flush 0 Sent Done Done false
5798 2585 Acked Write 8 Sent Done Done false
5799 2586 Acked Write 6 Sent Done Done false
5800 2587 Acked Write 3 Sent Done Done false
5801 2588 Acked Read 10 Sent Done Done false
5802 2589 Acked Read 7 Sent Done Done false
5803 2590 Acked Write 10 Sent Done Done false
5804 2591 Acked Read 9 Sent Done Done false
5805 2592 Acked Write 3 Sent Done Done false
5806 2593 Acked Flush 0 Sent Done Done false
5807 2594 Acked Read 10 Sent Done Done false
5808 2595 Acked Write 5 Sent Done Done false
5809 2596 Acked Write 9 Sent Done Done false
5810 2597 Acked Write 8 Sent Done Done false
5811 2598 Acked Read 2 Sent Done Done false
5812 2599 Acked Write 7 Sent Done Done false
5813 2600 Acked Read 9 Sent Done Done false
5814 2601 Acked Read 2 Sent Done Done false
5815 2602 Acked Flush 0 Sent Done Done false
5816 2603 Acked Write 10 Sent Done Done false
5817 2604 Acked Read 2 Sent Done Done false
5818 2605 Acked Write 2 Sent Done Done false
5819 2606 Acked Read 5 Sent Done Done false
5820 2607 Acked Flush 0 Sent Done Done false
5821 2608 Acked Read 10 Sent Done Done false
5822 2609 Acked Read 7 Sent Done Done false
5823 2610 Acked Read 10 Sent Done Done false
5824 2611 Acked Write 10 Sent Done Done false
5825 2612 Acked Flush 0 Sent Done Done false
5826 2613 Acked Read 2 Sent Done Done false
5827 2614 Acked Write 4 Sent Done Done false
5828 2615 Acked Flush 0 Sent Done Done false
5829 2616 Acked Read 3 Sent Done Done false
5830 2617 Acked Write 4 Sent Done Done false
5831 2618 Acked Read 4 Sent Done Done false
5832 2619 Acked Read 9 Sent Done Done false
5833 2620 Acked Write 5 Sent Done Done false
5834 2621 Acked Write 1 Sent Done Done false
5835 2622 Acked Read 5 Sent Done Done false
5836 2623 Acked Read 2 Sent Done Done false
5837 2624 Acked Write 6 Sent Done Done false
5838 2625 Acked Read 10 Sent Done Done false
5839 2626 Acked Write 2 Sent Done Done false
5840 2627 Acked Write 3 Sent Done Done false
5841 2628 Acked Write 7 Sent Done Done false
5842 2629 Acked Read 7 Sent Done Done false
5843 2630 Acked Flush 0 Sent Done Done false
5844 2631 Acked Read 9 Sent Done Done false
5845 2632 Acked Write 8 Sent Done Done false
5846 2633 Acked Read 2 Sent Done Done false
5847 2634 Acked Read 2 Sent Done Done false
5848 2635 Acked Read 9 Sent Done Done false
5849 2636 Acked Flush 0 Sent Done Done false
5850 2637 Acked Write 3 Sent Done Done false
5851 2638 Acked Write 2 Sent Done Done false
5852 2639 Acked Write 9 Sent Done Done false
5853 2640 Acked Read 4 Sent Done Done false
5854 2641 Acked Read 3 Sent Done Done false
5855 2642 Acked Read 6 Sent Done Done false
5856 2643 Acked Read 10 Sent Done Done false
5857 2644 Acked Write 9 Sent Done Done false
5858 2645 Acked Read 5 Sent Done Done false
5859 2646 Acked Write 10 Sent Done Done false
5860 2647 Acked Read 1 Sent Done Done false
5861 2648 Acked Write 6 Sent Done Done false
5862 2649 Acked Read 7 Sent Done Done false
5863 2650 Acked Read 4 Sent Done Done false
5864 2651 Acked Read 3 Sent Done Done false
5865 2652 Acked Read 5 Sent Done Done false
5866 2653 Acked Read 3 Sent Done Done false
5867 2654 Acked Read 10 Sent Done Done false
5868 2655 Acked Write 10 Sent Done Done false
5869 2656 Acked Write 9 Sent Done Done false
5870 2657 Acked Write 2 Sent Done Done false
5871 2658 Acked Read 5 Sent Done Done false
5872 2659 Acked Flush 0 Sent Done Done false
5873 2660 Acked Read 5 Sent Done Done false
5874 2661 Acked Write 9 Sent Done Done false
5875 2662 Acked Flush 0 Sent Done Done false
5876 2663 Acked Read 4 Sent Done Done false
5877 2664 Acked Write 9 Sent Done Done false
5878 2665 Acked Read 5 Sent Done Done false
5879 2666 Acked Write 7 Sent Done Done false
5880 2667 Acked Write 1 Sent Done Done false
5881 2668 Acked Read 10 Sent Done Done false
5882 2669 Acked Write 1 Sent Done Done false
5883 2670 Acked Write 2 Sent Done Done false
5884 2671 Acked Read 1 Sent Done Done false
5885 2672 Acked Read 6 Sent Done Done false
5886 2673 Acked Read 8 Sent Done Done false
5887 2674 Acked Write 9 Sent Done Done false
5888 2675 Acked Read 6 Sent Done Done false
5889 2676 Acked Write 9 Sent Done Done false
5890 2677 Acked Read 1 Sent Done Done false
5891 2678 Acked Write 5 Sent Done Done false
5892 2679 Acked Write 9 Sent Done Done false
5893 2680 Acked Flush 0 Sent Done Done false
5894 2681 Acked Write 4 Sent Done Done false
5895 2682 Acked Read 9 Sent Done Done false
5896 2683 Acked Read 1 Sent Done Done false
5897 2684 Acked Write 6 Sent Done Done false
5898 2685 Acked Read 6 Sent Done Done false
5899 2686 Acked Read 9 Sent Done Done false
5900 2687 Acked Write 1 Sent Done Done false
5901 2688 Acked Read 3 Sent Done Done false
5902 2689 Acked Read 6 Sent Done Done false
5903 2690 Acked Read 5 Sent Done Done false
5904 2691 Acked Write 3 Sent Done Done false
5905 2692 Acked Read 6 Sent Done Done false
5906 2693 Acked Read 10 Sent Done Done false
5907 2694 Acked Write 3 Sent Done Done false
5908 2695 Acked Read 6 Sent Done Done false
5909 2696 Acked Flush 0 Sent Done Done false
5910 2697 Acked Read 1 Sent Done Done false
5911 2698 Acked Write 1 Sent Done Done false
5912 2699 Acked Write 4 Sent Done Done false
5913 2700 Acked Write 4 Sent Done Done false
5914 2701 Acked Read 8 Sent Done Done false
5915 2702 Acked Read 2 Sent Done Done false
5916 2703 Acked Read 10 Sent Done Done false
5917 2704 Acked Write 8 Sent Done Done false
5918 2705 Acked Read 9 Sent Done Done false
5919 2706 Acked Read 8 Sent Done Done false
5920 2707 Acked Write 2 Sent Done Done false
5921 2708 Acked Read 1 Sent Done Done false
5922 2709 Acked Write 5 Sent Done Done false
5923 2710 Acked Read 7 Sent Done Done false
5924 2711 Acked Flush 0 Sent Done Done false
5925 2712 Acked Read 3 Sent Done Done false
5926 2713 Acked Flush 0 Sent Done Done false
5927 2714 Acked Write 7 Sent Done Done false
5928 2715 Acked Read 3 Sent Done Done false
5929 2716 Acked Write 9 Sent Done Done false
5930 2717 Acked Write 2 Sent Done Done false
5931 2718 Acked Write 3 Sent Done Done false
5932 2719 Acked Write 9 Sent Done Done false
5933 2720 Acked Read 2 Sent Done Done false
5934 2721 Acked Read 1 Sent Done Done false
5935 2722 Acked Flush 0 Sent Done Done false
5936 2723 Acked Read 7 Sent Done Done false
5937 2724 Acked Read 2 Sent Done Done false
5938 2725 Acked Flush 0 Sent Done Done false
5939 2726 Acked Read 6 Sent Done Done false
5940 2727 Acked Read 1 Sent Done Done false
5941 2728 Acked Read 4 Sent Done Done false
5942 2729 Acked Read 7 Sent Done Done false
5943 2730 Acked Write 7 Sent Done Done false
5944 2731 Acked Flush 0 Sent Done Done false
5945 2732 Acked Write 3 Sent Done Done false
5946 2733 Acked Read 2 Sent Done Done false
5947 2734 Acked Read 6 Sent Done Done false
5948 2735 Acked Read 3 Sent Done Done false
5949 2736 Acked Write 2 Sent Done Done false
5950 2737 Acked Write 6 Sent Done Done false
5951 2738 Acked Flush 0 Sent Done Done false
5952 2739 Acked Write 7 Sent Done Done false
5953 2740 Acked Write 9 Sent Done Done false
5954 2741 Acked Read 6 Sent Done Done false
5955 2742 Acked Write 1 Sent Done Done false
5956 2743 Acked Write 2 Sent Done Done false
5957 2744 Acked Write 9 Sent Done Done false
5958 2745 Acked Read 4 Sent Done Done false
5959 2746 Acked Read 8 Sent Done Done false
5960 2747 Acked Read 4 Sent Done Done false
5961 2748 Acked Flush 0 Sent Done Done false
5962 2749 Acked Read 10 Sent Done Done false
5963 2750 Acked Read 10 Sent Done Done false
5964 2751 Acked Read 8 Sent Done Done false
5965 2752 Acked Read 4 Sent Done Done false
5966 2753 Acked Write 6 Sent Done Done false
5967 2754 Acked Write 9 Sent Done Done false
5968 2755 Acked Read 5 Sent Done Done false
5969 2756 Acked Write 7 Sent Done Done false
5970 2757 Acked Read 3 Sent Done Done false
5971 2758 Acked Read 2 Sent Done Done false
5972 2759 Acked Write 9 Sent Done Done false
5973 2760 Acked Write 1 Sent Done Done false
5974 2761 Acked Read 2 Sent Done Done false
5975 2762 Acked Read 5 Sent Done Done false
5976 2763 Acked Write 5 Sent Done Done false
5977 2764 Acked Flush 0 Sent Done Done false
5978 2765 Acked Read 6 Sent Done Done false
5979 2766 Acked Flush 0 Sent Done Done false
5980 2767 Acked Read 5 Sent Done Done false
5981 2768 Acked Write 2 Sent Done Done false
5982 2769 Acked Read 5 Sent Done Done false
5983 2770 Acked Read 6 Sent Done Done false
5984 2771 Acked Write 3 Sent Done Done false
5985 2772 Acked Write 1 Sent Done Done false
5986 2773 Acked Read 4 Sent Done Done false
5987 2774 Acked Write 3 Sent Done Done false
5988 2775 Acked Write 5 Sent Done Done false
5989 2776 Acked Read 8 Sent Done Done false
5990 2777 Acked Read 10 Sent Done Done false
5991 2778 Acked Write 9 Sent Done Done false
5992 2779 Acked Read 1 Sent Done Done false
5993 2780 Acked Read 6 Sent Done Done false
5994 2781 Acked Flush 0 Sent Done Done false
5995 2782 Acked Read 9 Sent Done Done false
5996 2783 Acked Read 5 Sent Done Done false
5997 2784 Acked Read 4 Sent Done Done false
5998 2785 Acked Flush 0 Sent Done Done false
5999 2786 Acked Write 8 Sent Done Done false
6000 2787 Acked Read 6 Sent Done Done false
6001 2788 Acked Read 10 Sent Done Done false
6002 2789 Acked Read 1 Sent Done Done false
6003 2790 Acked Read 3 Sent Done Done false
6004 2791 Acked Flush 0 Sent Done Done false
6005 2792 Acked Write 7 Sent Done Done false
6006 2793 Acked Write 1 Sent Done Done false
6007 2794 Acked Read 6 Sent Done Done false
6008 2795 Acked Write 1 Sent Done Done false
6009 2796 Acked Write 5 Sent Done Done false
6010 2797 Acked Write 5 Sent Done Done false
6011 2798 Acked Read 9 Sent Done Done false
6012 2799 Acked Write 1 Sent Done Done false
6013 2800 Acked Write 6 Sent Done Done false
6014 2801 Acked Write 6 Sent Done Done false
6015 2802 Acked Write 5 Sent Done Done false
6016 2803 Acked Read 2 Sent Done Done false
6017 2804 Acked Read 10 Sent Done Done false
6018 2805 Acked Write 2 Sent Done Done false
6019 2806 Acked Write 5 Sent Done Done false
6020 2807 Acked Read 7 Sent Done Done false
6021 2808 Acked Read 1 Sent Done Done false
6022 2809 Acked Read 2 Sent Done Done false
6023 2810 Acked Read 2 Sent Done Done false
6024 STATES DS:0 DS:1 DS:2 TOTAL
6025 Sent 301 0 0 301
6026 Done 0 301 301 602
6027 Skipped 0 0 0 0
6028 Error 0 0 0 0
6029 Last Flush: Some(JobId(2509)) Some(JobId(2791)) Some(JobId(2791))
6030 Downstairs last five completed: 2509 2508 2507 2506 2505
6031 Upstairs last five completed: 2810 2809 2808 2807 2806
6032 CLIENT: Up:0 ds:301 act:2
60332025-01-29T05:11:53.448ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60342025-01-29T05:11:54.025ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60352025-01-29T05:11:54.698ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60362025-01-29T05:11:54.698ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60372025-01-29T05:11:54.698ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60382025-01-29T05:11:54.698ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60392025-01-29T05:11:54.698ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60402025-01-29T05:11:55.198ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
60412025-01-29T05:11:56.027ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
6042 ----------------------------------------------------------------
6043 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6044 States: Offline Active Active
6045 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6046 2510 Acked Read 4 Sent Done Done false
6047 2511 Acked Write 6 Sent Done Done false
6048 2512 Acked Flush 0 Sent Done Done false
6049 2513 Acked Write 2 Sent Done Done false
6050 2514 Acked Flush 0 Sent Done Done false
6051 2515 Acked Read 9 Sent Done Done false
6052 2516 Acked Read 2 Sent Done Done false
6053 2517 Acked Write 5 Sent Done Done false
6054 2518 Acked Read 1 Sent Done Done false
6055 2519 Acked Read 9 Sent Done Done false
6056 2520 Acked Write 5 Sent Done Done false
6057 2521 Acked Flush 0 Sent Done Done false
6058 2522 Acked Write 1 Sent Done Done false
6059 2523 Acked Read 3 Sent Done Done false
6060 2524 Acked Read 1 Sent Done Done false
6061 2525 Acked Read 3 Sent Done Done false
6062 2526 Acked Read 5 Sent Done Done false
6063 2527 Acked Write 3 Sent Done Done false
6064 2528 Acked Write 5 Sent Done Done false
6065 2529 Acked Flush 0 Sent Done Done false
6066 2530 Acked Read 8 Sent Done Done false
6067 2531 Acked Read 2 Sent Done Done false
6068 2532 Acked Write 10 Sent Done Done false
6069 2533 Acked Read 9 Sent Done Done false
6070 2534 Acked Read 9 Sent Done Done false
6071 2535 Acked Write 6 Sent Done Done false
6072 2536 Acked Read 9 Sent Done Done false
6073 2537 Acked Write 9 Sent Done Done false
6074 2538 Acked Write 9 Sent Done Done false
6075 2539 Acked Read 5 Sent Done Done false
6076 2540 Acked Read 4 Sent Done Done false
6077 2541 Acked Read 3 Sent Done Done false
6078 2542 Acked Write 9 Sent Done Done false
6079 2543 Acked Read 4 Sent Done Done false
6080 2544 Acked Read 1 Sent Done Done false
6081 2545 Acked Write 1 Sent Done Done false
6082 2546 Acked Read 7 Sent Done Done false
6083 2547 Acked Write 4 Sent Done Done false
6084 2548 Acked Read 1 Sent Done Done false
6085 2549 Acked Write 2 Sent Done Done false
6086 2550 Acked Write 4 Sent Done Done false
6087 2551 Acked Write 8 Sent Done Done false
6088 2552 Acked Read 10 Sent Done Done false
6089 2553 Acked Read 3 Sent Done Done false
6090 2554 Acked Read 7 Sent Done Done false
6091 2555 Acked Flush 0 Sent Done Done false
6092 2556 Acked Read 6 Sent Done Done false
6093 2557 Acked Flush 0 Sent Done Done false
6094 2558 Acked Read 10 Sent Done Done false
6095 2559 Acked Read 5 Sent Done Done false
6096 2560 Acked Read 2 Sent Done Done false
6097 2561 Acked Read 1 Sent Done Done false
6098 2562 Acked Read 9 Sent Done Done false
6099 2563 Acked Read 7 Sent Done Done false
6100 2564 Acked Read 9 Sent Done Done false
6101 2565 Acked Write 9 Sent Done Done false
6102 2566 Acked Write 7 Sent Done Done false
6103 2567 Acked Write 6 Sent Done Done false
6104 2568 Acked Read 5 Sent Done Done false
6105 2569 Acked Read 8 Sent Done Done false
6106 2570 Acked Read 6 Sent Done Done false
6107 2571 Acked Read 5 Sent Done Done false
6108 2572 Acked Read 9 Sent Done Done false
6109 2573 Acked Read 4 Sent Done Done false
6110 2574 Acked Write 9 Sent Done Done false
6111 2575 Acked Read 9 Sent Done Done false
6112 2576 Acked Write 10 Sent Done Done false
6113 2577 Acked Write 4 Sent Done Done false
6114 2578 Acked Read 10 Sent Done Done false
6115 2579 Acked Write 10 Sent Done Done false
6116 2580 Acked Read 5 Sent Done Done false
6117 2581 Acked Read 10 Sent Done Done false
6118 2582 Acked Read 1 Sent Done Done false
6119 2583 Acked Read 8 Sent Done Done false
6120 2584 Acked Flush 0 Sent Done Done false
6121 2585 Acked Write 8 Sent Done Done false
6122 2586 Acked Write 6 Sent Done Done false
6123 2587 Acked Write 3 Sent Done Done false
6124 2588 Acked Read 10 Sent Done Done false
6125 2589 Acked Read 7 Sent Done Done false
6126 2590 Acked Write 10 Sent Done Done false
6127 2591 Acked Read 9 Sent Done Done false
6128 2592 Acked Write 3 Sent Done Done false
6129 2593 Acked Flush 0 Sent Done Done false
6130 2594 Acked Read 10 Sent Done Done false
6131 2595 Acked Write 5 Sent Done Done false
6132 2596 Acked Write 9 Sent Done Done false
6133 2597 Acked Write 8 Sent Done Done false
6134 2598 Acked Read 2 Sent Done Done false
6135 2599 Acked Write 7 Sent Done Done false
6136 2600 Acked Read 9 Sent Done Done false
6137 2601 Acked Read 2 Sent Done Done false
6138 2602 Acked Flush 0 Sent Done Done false
6139 2603 Acked Write 10 Sent Done Done false
6140 2604 Acked Read 2 Sent Done Done false
6141 2605 Acked Write 2 Sent Done Done false
6142 2606 Acked Read 5 Sent Done Done false
6143 2607 Acked Flush 0 Sent Done Done false
6144 2608 Acked Read 10 Sent Done Done false
6145 2609 Acked Read 7 Sent Done Done false
6146 2610 Acked Read 10 Sent Done Done false
6147 2611 Acked Write 10 Sent Done Done false
6148 2612 Acked Flush 0 Sent Done Done false
6149 2613 Acked Read 2 Sent Done Done false
6150 2614 Acked Write 4 Sent Done Done false
6151 2615 Acked Flush 0 Sent Done Done false
6152 2616 Acked Read 3 Sent Done Done false
6153 2617 Acked Write 4 Sent Done Done false
6154 2618 Acked Read 4 Sent Done Done false
6155 2619 Acked Read 9 Sent Done Done false
6156 2620 Acked Write 5 Sent Done Done false
6157 2621 Acked Write 1 Sent Done Done false
6158 2622 Acked Read 5 Sent Done Done false
6159 2623 Acked Read 2 Sent Done Done false
6160 2624 Acked Write 6 Sent Done Done false
6161 2625 Acked Read 10 Sent Done Done false
6162 2626 Acked Write 2 Sent Done Done false
6163 2627 Acked Write 3 Sent Done Done false
6164 2628 Acked Write 7 Sent Done Done false
6165 2629 Acked Read 7 Sent Done Done false
6166 2630 Acked Flush 0 Sent Done Done false
6167 2631 Acked Read 9 Sent Done Done false
6168 2632 Acked Write 8 Sent Done Done false
6169 2633 Acked Read 2 Sent Done Done false
6170 2634 Acked Read 2 Sent Done Done false
6171 2635 Acked Read 9 Sent Done Done false
6172 2636 Acked Flush 0 Sent Done Done false
6173 2637 Acked Write 3 Sent Done Done false
6174 2638 Acked Write 2 Sent Done Done false
6175 2639 Acked Write 9 Sent Done Done false
6176 2640 Acked Read 4 Sent Done Done false
6177 2641 Acked Read 3 Sent Done Done false
6178 2642 Acked Read 6 Sent Done Done false
6179 2643 Acked Read 10 Sent Done Done false
6180 2644 Acked Write 9 Sent Done Done false
6181 2645 Acked Read 5 Sent Done Done false
6182 2646 Acked Write 10 Sent Done Done false
6183 2647 Acked Read 1 Sent Done Done false
6184 2648 Acked Write 6 Sent Done Done false
6185 2649 Acked Read 7 Sent Done Done false
6186 2650 Acked Read 4 Sent Done Done false
6187 2651 Acked Read 3 Sent Done Done false
6188 2652 Acked Read 5 Sent Done Done false
6189 2653 Acked Read 3 Sent Done Done false
6190 2654 Acked Read 10 Sent Done Done false
6191 2655 Acked Write 10 Sent Done Done false
6192 2656 Acked Write 9 Sent Done Done false
6193 2657 Acked Write 2 Sent Done Done false
6194 2658 Acked Read 5 Sent Done Done false
6195 2659 Acked Flush 0 Sent Done Done false
6196 2660 Acked Read 5 Sent Done Done false
6197 2661 Acked Write 9 Sent Done Done false
6198 2662 Acked Flush 0 Sent Done Done false
6199 2663 Acked Read 4 Sent Done Done false
6200 2664 Acked Write 9 Sent Done Done false
6201 2665 Acked Read 5 Sent Done Done false
6202 2666 Acked Write 7 Sent Done Done false
6203 2667 Acked Write 1 Sent Done Done false
6204 2668 Acked Read 10 Sent Done Done false
6205 2669 Acked Write 1 Sent Done Done false
6206 2670 Acked Write 2 Sent Done Done false
6207 2671 Acked Read 1 Sent Done Done false
6208 2672 Acked Read 6 Sent Done Done false
6209 2673 Acked Read 8 Sent Done Done false
6210 2674 Acked Write 9 Sent Done Done false
6211 2675 Acked Read 6 Sent Done Done false
6212 2676 Acked Write 9 Sent Done Done false
6213 2677 Acked Read 1 Sent Done Done false
6214 2678 Acked Write 5 Sent Done Done false
6215 2679 Acked Write 9 Sent Done Done false
6216 2680 Acked Flush 0 Sent Done Done false
6217 2681 Acked Write 4 Sent Done Done false
6218 2682 Acked Read 9 Sent Done Done false
6219 2683 Acked Read 1 Sent Done Done false
6220 2684 Acked Write 6 Sent Done Done false
6221 2685 Acked Read 6 Sent Done Done false
6222 2686 Acked Read 9 Sent Done Done false
6223 2687 Acked Write 1 Sent Done Done false
6224 2688 Acked Read 3 Sent Done Done false
6225 2689 Acked Read 6 Sent Done Done false
6226 2690 Acked Read 5 Sent Done Done false
6227 2691 Acked Write 3 Sent Done Done false
6228 2692 Acked Read 6 Sent Done Done false
6229 2693 Acked Read 10 Sent Done Done false
6230 2694 Acked Write 3 Sent Done Done false
6231 2695 Acked Read 6 Sent Done Done false
6232 2696 Acked Flush 0 Sent Done Done false
6233 2697 Acked Read 1 Sent Done Done false
6234 2698 Acked Write 1 Sent Done Done false
6235 2699 Acked Write 4 Sent Done Done false
6236 2700 Acked Write 4 Sent Done Done false
6237 2701 Acked Read 8 Sent Done Done false
6238 2702 Acked Read 2 Sent Done Done false
6239 2703 Acked Read 10 Sent Done Done false
6240 2704 Acked Write 8 Sent Done Done false
6241 2705 Acked Read 9 Sent Done Done false
6242 2706 Acked Read 8 Sent Done Done false
6243 2707 Acked Write 2 Sent Done Done false
6244 2708 Acked Read 1 Sent Done Done false
6245 2709 Acked Write 5 Sent Done Done false
6246 2710 Acked Read 7 Sent Done Done false
6247 2711 Acked Flush 0 Sent Done Done false
6248 2712 Acked Read 3 Sent Done Done false
6249 2713 Acked Flush 0 Sent Done Done false
6250 2714 Acked Write 7 Sent Done Done false
6251 2715 Acked Read 3 Sent Done Done false
6252 2716 Acked Write 9 Sent Done Done false
6253 2717 Acked Write 2 Sent Done Done false
6254 2718 Acked Write 3 Sent Done Done false
6255 2719 Acked Write 9 Sent Done Done false
6256 2720 Acked Read 2 Sent Done Done false
6257 2721 Acked Read 1 Sent Done Done false
6258 2722 Acked Flush 0 Sent Done Done false
6259 2723 Acked Read 7 Sent Done Done false
6260 2724 Acked Read 2 Sent Done Done false
6261 2725 Acked Flush 0 Sent Done Done false
6262 2726 Acked Read 6 Sent Done Done false
6263 2727 Acked Read 1 Sent Done Done false
6264 2728 Acked Read 4 Sent Done Done false
6265 2729 Acked Read 7 Sent Done Done false
6266 2730 Acked Write 7 Sent Done Done false
6267 2731 Acked Flush 0 Sent Done Done false
6268 2732 Acked Write 3 Sent Done Done false
6269 2733 Acked Read 2 Sent Done Done false
6270 2734 Acked Read 6 Sent Done Done false
6271 2735 Acked Read 3 Sent Done Done false
6272 2736 Acked Write 2 Sent Done Done false
6273 2737 Acked Write 6 Sent Done Done false
6274 2738 Acked Flush 0 Sent Done Done false
6275 2739 Acked Write 7 Sent Done Done false
6276 2740 Acked Write 9 Sent Done Done false
6277 2741 Acked Read 6 Sent Done Done false
6278 2742 Acked Write 1 Sent Done Done false
6279 2743 Acked Write 2 Sent Done Done false
6280 2744 Acked Write 9 Sent Done Done false
6281 2745 Acked Read 4 Sent Done Done false
6282 2746 Acked Read 8 Sent Done Done false
6283 2747 Acked Read 4 Sent Done Done false
6284 2748 Acked Flush 0 Sent Done Done false
6285 2749 Acked Read 10 Sent Done Done false
6286 2750 Acked Read 10 Sent Done Done false
6287 2751 Acked Read 8 Sent Done Done false
6288 2752 Acked Read 4 Sent Done Done false
6289 2753 Acked Write 6 Sent Done Done false
6290 2754 Acked Write 9 Sent Done Done false
6291 2755 Acked Read 5 Sent Done Done false
6292 2756 Acked Write 7 Sent Done Done false
6293 2757 Acked Read 3 Sent Done Done false
6294 2758 Acked Read 2 Sent Done Done false
6295 2759 Acked Write 9 Sent Done Done false
6296 2760 Acked Write 1 Sent Done Done false
6297 2761 Acked Read 2 Sent Done Done false
6298 2762 Acked Read 5 Sent Done Done false
6299 2763 Acked Write 5 Sent Done Done false
6300 2764 Acked Flush 0 Sent Done Done false
6301 2765 Acked Read 6 Sent Done Done false
6302 2766 Acked Flush 0 Sent Done Done false
6303 2767 Acked Read 5 Sent Done Done false
6304 2768 Acked Write 2 Sent Done Done false
6305 2769 Acked Read 5 Sent Done Done false
6306 2770 Acked Read 6 Sent Done Done false
6307 2771 Acked Write 3 Sent Done Done false
6308 2772 Acked Write 1 Sent Done Done false
6309 2773 Acked Read 4 Sent Done Done false
6310 2774 Acked Write 3 Sent Done Done false
6311 2775 Acked Write 5 Sent Done Done false
6312 2776 Acked Read 8 Sent Done Done false
6313 2777 Acked Read 10 Sent Done Done false
6314 2778 Acked Write 9 Sent Done Done false
6315 2779 Acked Read 1 Sent Done Done false
6316 2780 Acked Read 6 Sent Done Done false
6317 2781 Acked Flush 0 Sent Done Done false
6318 2782 Acked Read 9 Sent Done Done false
6319 2783 Acked Read 5 Sent Done Done false
6320 2784 Acked Read 4 Sent Done Done false
6321 2785 Acked Flush 0 Sent Done Done false
6322 2786 Acked Write 8 Sent Done Done false
6323 2787 Acked Read 6 Sent Done Done false
6324 2788 Acked Read 10 Sent Done Done false
6325 2789 Acked Read 1 Sent Done Done false
6326 2790 Acked Read 3 Sent Done Done false
6327 2791 Acked Flush 0 Sent Done Done false
6328 2792 Acked Write 7 Sent Done Done false
6329 2793 Acked Write 1 Sent Done Done false
6330 2794 Acked Read 6 Sent Done Done false
6331 2795 Acked Write 1 Sent Done Done false
6332 2796 Acked Write 5 Sent Done Done false
6333 2797 Acked Write 5 Sent Done Done false
6334 2798 Acked Read 9 Sent Done Done false
6335 2799 Acked Write 1 Sent Done Done false
6336 2800 Acked Write 6 Sent Done Done false
6337 2801 Acked Write 6 Sent Done Done false
6338 2802 Acked Write 5 Sent Done Done false
6339 2803 Acked Read 2 Sent Done Done false
6340 2804 Acked Read 10 Sent Done Done false
6341 2805 Acked Write 2 Sent Done Done false
6342 2806 Acked Write 5 Sent Done Done false
6343 2807 Acked Read 7 Sent Done Done false
6344 2808 Acked Read 1 Sent Done Done false
6345 2809 Acked Read 2 Sent Done Done false
6346 2810 Acked Read 2 Sent Done Done false
6347 STATES DS:0 DS:1 DS:2 TOTAL
6348 Sent 301 0 0 301
6349 Done 0 301 301 602
6350 Skipped 0 0 0 0
6351 Error 0 0 0 0
6352 Last Flush: Some(JobId(2509)) Some(JobId(2791)) Some(JobId(2791))
6353 Downstairs last five completed: 2509 2508 2507 2506 2505
6354 Upstairs last five completed: 2810 2809 2808 2807 2806
6355 CLIENT: Up:0 ds:301 act:2
63562025-01-29T05:11:57.450ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
63572025-01-29T05:11:58.032ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
6358 [8810] got start action
6359 [8810] I am going to restart
6360 Starting downstairs at port 8810
6361 Make output file at "/tmp/dsc/downstairs-8810.txt"
6362 [8810][0] reports Starting
6363 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1208)
6364 [8810][0] reports Running
63652025-01-29T05:11:59.034ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
63662025-01-29T05:12:00.036ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
6367 ----------------------------------------------------------------
6368 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6369 States: Offline Active Active
6370 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6371 2510 Acked Read 4 Sent Done Done false
6372 2511 Acked Write 6 Sent Done Done false
6373 2512 Acked Flush 0 Sent Done Done false
6374 2513 Acked Write 2 Sent Done Done false
6375 2514 Acked Flush 0 Sent Done Done false
6376 2515 Acked Read 9 Sent Done Done false
6377 2516 Acked Read 2 Sent Done Done false
6378 2517 Acked Write 5 Sent Done Done false
6379 2518 Acked Read 1 Sent Done Done false
6380 2519 Acked Read 9 Sent Done Done false
6381 2520 Acked Write 5 Sent Done Done false
6382 2521 Acked Flush 0 Sent Done Done false
6383 2522 Acked Write 1 Sent Done Done false
6384 2523 Acked Read 3 Sent Done Done false
6385 2524 Acked Read 1 Sent Done Done false
6386 2525 Acked Read 3 Sent Done Done false
6387 2526 Acked Read 5 Sent Done Done false
6388 2527 Acked Write 3 Sent Done Done false
6389 2528 Acked Write 5 Sent Done Done false
6390 2529 Acked Flush 0 Sent Done Done false
6391 2530 Acked Read 8 Sent Done Done false
6392 2531 Acked Read 2 Sent Done Done false
6393 2532 Acked Write 10 Sent Done Done false
6394 2533 Acked Read 9 Sent Done Done false
6395 2534 Acked Read 9 Sent Done Done false
6396 2535 Acked Write 6 Sent Done Done false
6397 2536 Acked Read 9 Sent Done Done false
6398 2537 Acked Write 9 Sent Done Done false
6399 2538 Acked Write 9 Sent Done Done false
6400 2539 Acked Read 5 Sent Done Done false
6401 2540 Acked Read 4 Sent Done Done false
6402 2541 Acked Read 3 Sent Done Done false
6403 2542 Acked Write 9 Sent Done Done false
6404 2543 Acked Read 4 Sent Done Done false
6405 2544 Acked Read 1 Sent Done Done false
6406 2545 Acked Write 1 Sent Done Done false
6407 2546 Acked Read 7 Sent Done Done false
6408 2547 Acked Write 4 Sent Done Done false
6409 2548 Acked Read 1 Sent Done Done false
6410 2549 Acked Write 2 Sent Done Done false
6411 2550 Acked Write 4 Sent Done Done false
6412 2551 Acked Write 8 Sent Done Done false
6413 2552 Acked Read 10 Sent Done Done false
6414 2553 Acked Read 3 Sent Done Done false
6415 2554 Acked Read 7 Sent Done Done false
6416 2555 Acked Flush 0 Sent Done Done false
6417 2556 Acked Read 6 Sent Done Done false
6418 2557 Acked Flush 0 Sent Done Done false
6419 2558 Acked Read 10 Sent Done Done false
6420 2559 Acked Read 5 Sent Done Done false
6421 2560 Acked Read 2 Sent Done Done false
6422 2561 Acked Read 1 Sent Done Done false
6423 2562 Acked Read 9 Sent Done Done false
6424 2563 Acked Read 7 Sent Done Done false
6425 2564 Acked Read 9 Sent Done Done false
6426 2565 Acked Write 9 Sent Done Done false
6427 2566 Acked Write 7 Sent Done Done false
6428 2567 Acked Write 6 Sent Done Done false
6429 2568 Acked Read 5 Sent Done Done false
6430 2569 Acked Read 8 Sent Done Done false
6431 2570 Acked Read 6 Sent Done Done false
6432 2571 Acked Read 5 Sent Done Done false
6433 2572 Acked Read 9 Sent Done Done false
6434 2573 Acked Read 4 Sent Done Done false
6435 2574 Acked Write 9 Sent Done Done false
6436 2575 Acked Read 9 Sent Done Done false
6437 2576 Acked Write 10 Sent Done Done false
6438 2577 Acked Write 4 Sent Done Done false
6439 2578 Acked Read 10 Sent Done Done false
6440 2579 Acked Write 10 Sent Done Done false
6441 2580 Acked Read 5 Sent Done Done false
6442 2581 Acked Read 10 Sent Done Done false
6443 2582 Acked Read 1 Sent Done Done false
6444 2583 Acked Read 8 Sent Done Done false
6445 2584 Acked Flush 0 Sent Done Done false
6446 2585 Acked Write 8 Sent Done Done false
6447 2586 Acked Write 6 Sent Done Done false
6448 2587 Acked Write 3 Sent Done Done false
6449 2588 Acked Read 10 Sent Done Done false
6450 2589 Acked Read 7 Sent Done Done false
6451 2590 Acked Write 10 Sent Done Done false
6452 2591 Acked Read 9 Sent Done Done false
6453 2592 Acked Write 3 Sent Done Done false
6454 2593 Acked Flush 0 Sent Done Done false
6455 2594 Acked Read 10 Sent Done Done false
6456 2595 Acked Write 5 Sent Done Done false
6457 2596 Acked Write 9 Sent Done Done false
6458 2597 Acked Write 8 Sent Done Done false
6459 2598 Acked Read 2 Sent Done Done false
6460 2599 Acked Write 7 Sent Done Done false
6461 2600 Acked Read 9 Sent Done Done false
6462 2601 Acked Read 2 Sent Done Done false
6463 2602 Acked Flush 0 Sent Done Done false
6464 2603 Acked Write 10 Sent Done Done false
6465 2604 Acked Read 2 Sent Done Done false
6466 2605 Acked Write 2 Sent Done Done false
6467 2606 Acked Read 5 Sent Done Done false
6468 2607 Acked Flush 0 Sent Done Done false
6469 2608 Acked Read 10 Sent Done Done false
6470 2609 Acked Read 7 Sent Done Done false
6471 2610 Acked Read 10 Sent Done Done false
6472 2611 Acked Write 10 Sent Done Done false
6473 2612 Acked Flush 0 Sent Done Done false
6474 2613 Acked Read 2 Sent Done Done false
6475 2614 Acked Write 4 Sent Done Done false
6476 2615 Acked Flush 0 Sent Done Done false
6477 2616 Acked Read 3 Sent Done Done false
6478 2617 Acked Write 4 Sent Done Done false
6479 2618 Acked Read 4 Sent Done Done false
6480 2619 Acked Read 9 Sent Done Done false
6481 2620 Acked Write 5 Sent Done Done false
6482 2621 Acked Write 1 Sent Done Done false
6483 2622 Acked Read 5 Sent Done Done false
6484 2623 Acked Read 2 Sent Done Done false
6485 2624 Acked Write 6 Sent Done Done false
6486 2625 Acked Read 10 Sent Done Done false
6487 2626 Acked Write 2 Sent Done Done false
6488 2627 Acked Write 3 Sent Done Done false
6489 2628 Acked Write 7 Sent Done Done false
6490 2629 Acked Read 7 Sent Done Done false
6491 2630 Acked Flush 0 Sent Done Done false
6492 2631 Acked Read 9 Sent Done Done false
6493 2632 Acked Write 8 Sent Done Done false
6494 2633 Acked Read 2 Sent Done Done false
6495 2634 Acked Read 2 Sent Done Done false
6496 2635 Acked Read 9 Sent Done Done false
6497 2636 Acked Flush 0 Sent Done Done false
6498 2637 Acked Write 3 Sent Done Done false
6499 2638 Acked Write 2 Sent Done Done false
6500 2639 Acked Write 9 Sent Done Done false
6501 2640 Acked Read 4 Sent Done Done false
6502 2641 Acked Read 3 Sent Done Done false
6503 2642 Acked Read 6 Sent Done Done false
6504 2643 Acked Read 10 Sent Done Done false
6505 2644 Acked Write 9 Sent Done Done false
6506 2645 Acked Read 5 Sent Done Done false
6507 2646 Acked Write 10 Sent Done Done false
6508 2647 Acked Read 1 Sent Done Done false
6509 2648 Acked Write 6 Sent Done Done false
6510 2649 Acked Read 7 Sent Done Done false
6511 2650 Acked Read 4 Sent Done Done false
6512 2651 Acked Read 3 Sent Done Done false
6513 2652 Acked Read 5 Sent Done Done false
6514 2653 Acked Read 3 Sent Done Done false
6515 2654 Acked Read 10 Sent Done Done false
6516 2655 Acked Write 10 Sent Done Done false
6517 2656 Acked Write 9 Sent Done Done false
6518 2657 Acked Write 2 Sent Done Done false
6519 2658 Acked Read 5 Sent Done Done false
6520 2659 Acked Flush 0 Sent Done Done false
6521 2660 Acked Read 5 Sent Done Done false
6522 2661 Acked Write 9 Sent Done Done false
6523 2662 Acked Flush 0 Sent Done Done false
6524 2663 Acked Read 4 Sent Done Done false
6525 2664 Acked Write 9 Sent Done Done false
6526 2665 Acked Read 5 Sent Done Done false
6527 2666 Acked Write 7 Sent Done Done false
6528 2667 Acked Write 1 Sent Done Done false
6529 2668 Acked Read 10 Sent Done Done false
6530 2669 Acked Write 1 Sent Done Done false
6531 2670 Acked Write 2 Sent Done Done false
6532 2671 Acked Read 1 Sent Done Done false
6533 2672 Acked Read 6 Sent Done Done false
6534 2673 Acked Read 8 Sent Done Done false
6535 2674 Acked Write 9 Sent Done Done false
6536 2675 Acked Read 6 Sent Done Done false
6537 2676 Acked Write 9 Sent Done Done false
6538 2677 Acked Read 1 Sent Done Done false
6539 2678 Acked Write 5 Sent Done Done false
6540 2679 Acked Write 9 Sent Done Done false
6541 2680 Acked Flush 0 Sent Done Done false
6542 2681 Acked Write 4 Sent Done Done false
6543 2682 Acked Read 9 Sent Done Done false
6544 2683 Acked Read 1 Sent Done Done false
6545 2684 Acked Write 6 Sent Done Done false
6546 2685 Acked Read 6 Sent Done Done false
6547 2686 Acked Read 9 Sent Done Done false
6548 2687 Acked Write 1 Sent Done Done false
6549 2688 Acked Read 3 Sent Done Done false
6550 2689 Acked Read 6 Sent Done Done false
6551 2690 Acked Read 5 Sent Done Done false
6552 2691 Acked Write 3 Sent Done Done false
6553 2692 Acked Read 6 Sent Done Done false
6554 2693 Acked Read 10 Sent Done Done false
6555 2694 Acked Write 3 Sent Done Done false
6556 2695 Acked Read 6 Sent Done Done false
6557 2696 Acked Flush 0 Sent Done Done false
6558 2697 Acked Read 1 Sent Done Done false
6559 2698 Acked Write 1 Sent Done Done false
6560 2699 Acked Write 4 Sent Done Done false
6561 2700 Acked Write 4 Sent Done Done false
6562 2701 Acked Read 8 Sent Done Done false
6563 2702 Acked Read 2 Sent Done Done false
6564 2703 Acked Read 10 Sent Done Done false
6565 2704 Acked Write 8 Sent Done Done false
6566 2705 Acked Read 9 Sent Done Done false
6567 2706 Acked Read 8 Sent Done Done false
6568 2707 Acked Write 2 Sent Done Done false
6569 2708 Acked Read 1 Sent Done Done false
6570 2709 Acked Write 5 Sent Done Done false
6571 2710 Acked Read 7 Sent Done Done false
6572 2711 Acked Flush 0 Sent Done Done false
6573 2712 Acked Read 3 Sent Done Done false
6574 2713 Acked Flush 0 Sent Done Done false
6575 2714 Acked Write 7 Sent Done Done false
6576 2715 Acked Read 3 Sent Done Done false
6577 2716 Acked Write 9 Sent Done Done false
6578 2717 Acked Write 2 Sent Done Done false
6579 2718 Acked Write 3 Sent Done Done false
6580 2719 Acked Write 9 Sent Done Done false
6581 2720 Acked Read 2 Sent Done Done false
6582 2721 Acked Read 1 Sent Done Done false
6583 2722 Acked Flush 0 Sent Done Done false
6584 2723 Acked Read 7 Sent Done Done false
6585 2724 Acked Read 2 Sent Done Done false
6586 2725 Acked Flush 0 Sent Done Done false
6587 2726 Acked Read 6 Sent Done Done false
6588 2727 Acked Read 1 Sent Done Done false
6589 2728 Acked Read 4 Sent Done Done false
6590 2729 Acked Read 7 Sent Done Done false
6591 2730 Acked Write 7 Sent Done Done false
6592 2731 Acked Flush 0 Sent Done Done false
6593 2732 Acked Write 3 Sent Done Done false
6594 2733 Acked Read 2 Sent Done Done false
6595 2734 Acked Read 6 Sent Done Done false
6596 2735 Acked Read 3 Sent Done Done false
6597 2736 Acked Write 2 Sent Done Done false
6598 2737 Acked Write 6 Sent Done Done false
6599 2738 Acked Flush 0 Sent Done Done false
6600 2739 Acked Write 7 Sent Done Done false
6601 2740 Acked Write 9 Sent Done Done false
6602 2741 Acked Read 6 Sent Done Done false
6603 2742 Acked Write 1 Sent Done Done false
6604 2743 Acked Write 2 Sent Done Done false
6605 2744 Acked Write 9 Sent Done Done false
6606 2745 Acked Read 4 Sent Done Done false
6607 2746 Acked Read 8 Sent Done Done false
6608 2747 Acked Read 4 Sent Done Done false
6609 2748 Acked Flush 0 Sent Done Done false
6610 2749 Acked Read 10 Sent Done Done false
6611 2750 Acked Read 10 Sent Done Done false
6612 2751 Acked Read 8 Sent Done Done false
6613 2752 Acked Read 4 Sent Done Done false
6614 2753 Acked Write 6 Sent Done Done false
6615 2754 Acked Write 9 Sent Done Done false
6616 2755 Acked Read 5 Sent Done Done false
6617 2756 Acked Write 7 Sent Done Done false
6618 2757 Acked Read 3 Sent Done Done false
6619 2758 Acked Read 2 Sent Done Done false
6620 2759 Acked Write 9 Sent Done Done false
6621 2760 Acked Write 1 Sent Done Done false
6622 2761 Acked Read 2 Sent Done Done false
6623 2762 Acked Read 5 Sent Done Done false
6624 2763 Acked Write 5 Sent Done Done false
6625 2764 Acked Flush 0 Sent Done Done false
6626 2765 Acked Read 6 Sent Done Done false
6627 2766 Acked Flush 0 Sent Done Done false
6628 2767 Acked Read 5 Sent Done Done false
6629 2768 Acked Write 2 Sent Done Done false
6630 2769 Acked Read 5 Sent Done Done false
6631 2770 Acked Read 6 Sent Done Done false
6632 2771 Acked Write 3 Sent Done Done false
6633 2772 Acked Write 1 Sent Done Done false
6634 2773 Acked Read 4 Sent Done Done false
6635 2774 Acked Write 3 Sent Done Done false
6636 2775 Acked Write 5 Sent Done Done false
6637 2776 Acked Read 8 Sent Done Done false
6638 2777 Acked Read 10 Sent Done Done false
6639 2778 Acked Write 9 Sent Done Done false
6640 2779 Acked Read 1 Sent Done Done false
6641 2780 Acked Read 6 Sent Done Done false
6642 2781 Acked Flush 0 Sent Done Done false
6643 2782 Acked Read 9 Sent Done Done false
6644 2783 Acked Read 5 Sent Done Done false
6645 2784 Acked Read 4 Sent Done Done false
6646 2785 Acked Flush 0 Sent Done Done false
6647 2786 Acked Write 8 Sent Done Done false
6648 2787 Acked Read 6 Sent Done Done false
6649 2788 Acked Read 10 Sent Done Done false
6650 2789 Acked Read 1 Sent Done Done false
6651 2790 Acked Read 3 Sent Done Done false
6652 2791 Acked Flush 0 Sent Done Done false
6653 2792 Acked Write 7 Sent Done Done false
6654 2793 Acked Write 1 Sent Done Done false
6655 2794 Acked Read 6 Sent Done Done false
6656 2795 Acked Write 1 Sent Done Done false
6657 2796 Acked Write 5 Sent Done Done false
6658 2797 Acked Write 5 Sent Done Done false
6659 2798 Acked Read 9 Sent Done Done false
6660 2799 Acked Write 1 Sent Done Done false
6661 2800 Acked Write 6 Sent Done Done false
6662 2801 Acked Write 6 Sent Done Done false
6663 2802 Acked Write 5 Sent Done Done false
6664 2803 Acked Read 2 Sent Done Done false
6665 2804 Acked Read 10 Sent Done Done false
6666 2805 Acked Write 2 Sent Done Done false
6667 2806 Acked Write 5 Sent Done Done false
6668 2807 Acked Read 7 Sent Done Done false
6669 2808 Acked Read 1 Sent Done Done false
6670 2809 Acked Read 2 Sent Done Done false
6671 2810 Acked Read 2 Sent Done Done false
6672 STATES DS:0 DS:1 DS:2 TOTAL
6673 Sent 301 0 0 301
6674 Done 0 301 301 602
6675 Skipped 0 0 0 0
6676 Error 0 0 0 0
6677 Last Flush: Some(JobId(2509)) Some(JobId(2791)) Some(JobId(2791))
6678 Downstairs last five completed: 2509 2508 2507 2506 2505
6679 Upstairs last five completed: 2810 2809 2808 2807 2806
6680 CLIENT: Up:0 ds:301 act:2
66812025-01-29T05:12:01.453ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66822025-01-29T05:12:02.039ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66832025-01-29T05:12:03.041ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66842025-01-29T05:12:04.043ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66852025-01-29T05:12:04.698ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66862025-01-29T05:12:04.699ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33390) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66872025-01-29T05:12:04.700ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66882025-01-29T05:12:04.700ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66892025-01-29T05:12:04.700ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66902025-01-29T05:12:04.700ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66912025-01-29T05:12:04.700ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66922025-01-29T05:12:04.700ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
66932025-01-29T05:12:04.701ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
6694 ----------------------------------------------------------------
6695 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:19
6696 States: Active Active Active
6697 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6698 2792 Acked Write 7 Done Done Done true
6699 2793 Acked Write 1 Done Done Done true
6700 2794 Acked Read 6 Done Done Done true
6701 2795 Acked Write 1 Done Done Done true
6702 2796 Acked Write 5 Done Done Done true
6703 2797 Acked Write 5 Done Done Done true
6704 2798 Acked Read 9 Done Done Done true
6705 2799 Acked Write 1 Done Done Done true
6706 2800 Acked Write 6 Done Done Done true
6707 2801 Acked Write 6 Done Done Done true
6708 2802 Acked Write 5 Done Done Done true
6709 2803 Acked Read 2 Done Done Done true
6710 2804 Acked Read 10 Done Done Done true
6711 2805 Acked Write 2 Done Done Done true
6712 2806 Acked Write 5 Done Done Done true
6713 2807 Acked Read 7 Done Done Done true
6714 2808 Acked Read 1 Done Done Done true
6715 2809 Acked Read 2 Done Done Done true
6716 2810 Acked Read 2 Done Done Done true
6717 STATES DS:0 DS:1 DS:2 TOTAL
6718 Sent 0 0 0 0
6719 Done 19 19 19 57
6720 Skipped 0 0 0 0
6721 Error 0 0 0 0
6722 Last Flush: Some(JobId(2791)) Some(JobId(2791)) Some(JobId(2791))
6723 Downstairs last five completed: 2791 2790 2789 2788 2787
6724 Upstairs last five completed: 2810 2809 2808 2807 2806
6725 CLIENT: Up:0 ds:19 act:3
6726 ----------------------------------------------------------------
6727 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6728 Downstairs last five completed: 2811 2810 2809 2808 2807
6729 Upstairs last five completed: 2811 2810 2809 2808 2807
6730 CLIENT: Up:0 ds:0 act:3
6731 Replay: All jobs finished
6732 Main task has work to do, go find it
6733 got dsc Stop(1)
6734 stop 1
6735 Jan 29 05:12:08.954 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: [8820] Got stop action so:false kr:false
6736 53f2ea1d-d308-4c4e-b281-903e22ddd87e, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
6737 Jan 29 05:12:08.955 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9d9767b4-0858-46a2-913c-ef7d5c2dba69, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
67382025-01-29T05:12:08.955ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
67392025-01-29T05:12:08.955ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
67402025-01-29T05:12:08.955ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
67412025-01-29T05:12:08.955ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
6742 [8820] Exited with: ExitStatus(unix_wait_status(9))
6743 [8820][1] reports Exit
6744 Jan 29 05:12:12.958 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/1, method: GET, req_id: 24bfd444-4801-4c03-8103-4f25c2e1fa3a, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
6745 001/300 Write block 1996 len 40960 data: 2 2 3 4 4 5 5 5 4 5
6746 002/300 Read block 2973 len 36864
6747 003/300 Write block 2079 len 32768 data: 2 2 2 2 2 3 3 3
6748 004/300 Write block 4743 len 8192 data: 3 3
6749 005/300 Read block 4098 len 32768
6750 006/300 Read block 2940 len 24576
6751 007/300 Read block 154 len 12288
6752 008/300 Write block 2580 len 4096 data: 3
6753 009/300 Write block 4136 len 24576 data: 2 2 3 3 4 4
6754 010/300 Flush
6755 011/300 Read block 2198 len 20480
6756 012/300 Read block 3199 len 32768
6757 013/300 Write block 2881 len 16384 data: 3 3 2 2
6758 014/300 Read block 3786 len 12288
6759 015/300 Read block 2463 len 16384
6760 016/300 Read block 3923 len 32768
6761 017/300 Write block 123 len 20480 data: 4 3 3 2 2
6762 018/300 Read block 3290 len 28672
6763 019/300 Write block 12 len 40960 data: 2 2 3 3 3 3 2 2 2 2
6764 020/300 Write block 3169 len 40960 data: 3 3 3 3 3 2 2 2 2 2
6765 021/300 Flush
6766 022/300 Read block 1154 len 16384
6767 023/300 Flush
6768 024/300 Flush
6769 025/300 Read block 3356 len 16384
6770 026/300 Write block 426 len 24576 data: 2 2 2 2 2 2
6771 027/300 Write block 2050 len 4096 data: 2
6772 028/300 Write block 4693 len 40960 data: 2 2 2 2 3 3 3 3 3 3
6773 029/300 Read block 333 len 36864
6774 030/300 Write block 135 len 24576 data: 3 3 3 2 2 2
6775 031/300 Write block 4547 len 16384 data: 3 4 4 3
6776 032/300 Read block 534 len 20480
6777 033/300 Read block 4507 len 16384
6778 034/300 Write block 4984 len 12288 data: 2 2 4
6779 035/300 Write block 3568 len 16384 data: 2 2 2 2
6780 036/300 Write block 3676 len 40960 data: 2 3 3 3 3 3 3 2 2 2
6781 037/300 Write block 502 len 24576 data: 4 4 4 4 4 6
6782 038/300 Write block 1 len 8192 data: 2 2
6783 039/300 Read block 2656 len 20480
6784 040/300 Read block 530 len 20480
6785 041/300 Flush
6786 042/300 Read block 735 len 12288
6787 043/300 Read block 2520 len 40960
6788 044/300 Read block 413 len 20480
6789 045/300 Read block 4994 len 4096
6790 046/300 Read block 4977 len 16384
6791 047/300 Read block 1485 len 4096
6792 048/300 Read block 1496 len 36864
6793 049/300 Write block 3420 len 20480 data: 4 4 4 4 2
6794 050/300 Read block 1676 len 20480
6795 051/300 Write block 3368 len 16384 data: 2 2 3 3
6796 052/300 Flush
6797 053/300 Flush
6798 054/300 Read block 254 len 4096
6799 055/300 Read block 1737 len 12288
6800 056/300 Flush
6801 057/300 Write block 195 len 16384 data: 3 3 2 2
6802 058/300 Write block 3096 len 4096 data: 3
6803 059/300 Read block 3453 len 16384
6804 060/300 Read block 3619 len 32768
6805 061/300 Read block 1591 len 28672
6806 062/300 Read block 3782 len 40960
6807 063/300 Read block 686 len 16384
6808 064/300 Write block 2438 len 36864 data: 4 3 3 3 3 3 3 2 2
6809 065/300 Read block 2687 len 24576
6810 066/300 Write block 659 len 40960 data: 4 4 3 3 3 3 2 2 2 2
6811 067/300 Flush
6812 068/300 Write block 1775 len 20480 data: 4 4 3 3 3
6813 069/300 Read block 1938 len 20480
6814 070/300 Read block 4751 len 24576
6815 071/300 Write block 4411 len 12288 data: 3 3 3
6816 072/300 Read block 2491 len 40960
6817 073/300 Read block 2118 len 28672
6818 074/300 Read block 1642 len 4096
6819 075/300 Write block 4740 len 24576 data: 3 3 3 4 4 3
6820 076/300 Write block 715 len 20480 data: 2 2 2 2 2
6821 077/300 Read block 4368 len 36864
6822 078/300 Write block 2759 len 16384 data: 5 5 4 4
6823 079/300 Flush
6824 080/300 Write block 1632 len 32768 data: 3 3 3 2 2 3 3 3
6825 081/300 Read block 2679 len 32768
6826 082/300 Read block 1499 len 32768
6827 083/300 Flush
6828 084/300 Read block 2562 len 24576
6829 085/300 Write block 23 len 8192 data: 2 2
6830 086/300 Read block 2617 len 32768
6831 087/300 Write block 1370 len 12288 data: 3 3 3
6832 088/300 Write block 2319 len 32768 data: 3 3 2 3 3 2 2 2
6833 089/300 Write block 4323 len 36864 data: 3 3 3 3 2 2 2 2 2
6834 090/300 Write block 230 len 40960 data: 2 2 2 2 3 3 4 5 5 5
6835 091/300 Write block 4986 len 40960 data: 5 4 4 4 4 3 3 2 2 2
6836 092/300 Write block 2005 len 20480 data: 6 6 4 6 5
6837 093/300 Write block 3834 len 12288 data: 3 4 3
6838 094/300 Read block 2468 len 32768
6839 095/300 Write block 358 len 24576 data: 2 2 2 2 2 2
6840 096/300 Read block 2493 len 12288
6841 097/300 Read block 4035 len 4096
6842 098/300 Write block 3531 len 28672 data: 3 3 3 3 3 2 2
6843 099/300 Write block 3075 len 12288 data: 3 3 3
6844 100/300 Write block 1805 len 8192 data: 3 3
6845 101/300 Read block 3664 len 40960
6846 102/300 Flush
6847 103/300 Write block 4898 len 32768 data: 3 3 3 2 2 2 2 2
6848 104/300 Read block 1267 len 8192
6849 105/300 Write block 1269 len 24576 data: 2 2 2 3 3 2
6850 106/300 Read block 3598 len 4096
6851 107/300 Read block 2530 len 12288
6852 108/300 Write block 1873 len 40960 data: 3 3 3 3 2 2 2 2 2 2
6853 109/300 Read block 3748 len 8192
6854 110/300 Write block 1059 len 20480 data: 2 2 2 2 2
6855 111/300 Write block 1453 len 36864 data: 2 2 2 2 2 2 2 2 2
6856 112/300 Read block 4470 len 16384
6857 113/300 Read block 3663 len 8192
6858 114/300 Write block 2803 len 12288 data: 2 2 2
6859 115/300 Write block 2868 len 8192 data: 2 2
6860 116/300 Read block 3768 len 8192
6861 117/300 Read block 126 len 28672
6862 118/300 Write block 2071 len 4096 data: 2
6863 119/300 Read block 1820 len 4096
6864 120/300 Read block 3434 len 40960
6865 121/300 Flush
6866 122/300 Write block 1836 len 4096 data: 2
6867 123/300 Write block 1724 len 36864 data: 2 3 3 3 2 2 2 2 2
6868 124/300 Write block 3037 len 8192 data: 2 3
6869 125/300 Write block 3991 len 24576 data: 3 3 3 3 3 2
6870 126/300 Read block 2120 len 24576
6871 127/300 Write block 4878 len 12288 data: 2 2 2
6872 128/300 Write block 1369 len 4096 data: 3
6873 129/300 Read block 3663 len 36864
6874 130/300 Flush
6875 131/300 Read block 1045 len 8192
6876 132/300 Read block 1587 len 32768
6877 133/300 Write block 2809 len 32768 data: 3 3 3 4 4 3 3 2
6878 134/300 Read block 1058 len 12288
6879 135/300 Write block 1092 len 24576 data: 2 2 2 2 2 2
6880 136/300 Read block 4422 len 36864
6881 137/300 Read block 1289 len 12288
6882 138/300 Read block 4243 len 16384
6883 139/300 Read block 4529 len 20480
6884 140/300 Read block 4402 len 8192
6885 141/300 Read block 874 len 8192
6886 142/300 Read block 4888 len 36864
6887 143/300 Read block 1160 len 20480
6888 144/300 Write block 4870 len 32768 data: 2 2 2 2 2 2 2 2
6889 145/300 Flush
6890 146/300 Read block 4846 len 12288
6891 147/300 Write block 1432 len 36864 data: 3 3 3 3 3 2 2 2 2
6892 148/300 Write block 1166 len 32768 data: 2 3 4 4 4 5 6 5
6893 149/300 Read block 1870 len 40960
6894 150/300 Write block 2341 len 8192 data: 4 4
6895 151/300 Write block 2208 len 20480 data: 3 3 3 3 3
6896 152/300 Write block 3977 len 28672 data: 4 4 4 3 3 3 2
6897 153/300 Read block 377 len 24576
6898 154/300 Flush
6899 155/300 Read block 3702 len 20480
6900 156/300 Write block 3699 len 8192 data: 2 2
6901 157/300 Read block 4606 len 28672
6902 158/300 Write block 2960 len 28672 data: 3 3 3 3 4 4 4
6903 159/300 Read block 1688 len 28672
6904 160/300 Write block 3122 len 12288 data: 3 3 2
6905 161/300 Write block 3140 len 36864 data: 3 4 3 2 3 3 3 4 4
6906 162/300 Read block 1421 len 16384
6907 163/300 Write block 2057 len 24576 data: 2 2 2 2 2 2
6908 164/300 Read block 4191 len 8192
6909 165/300 Read block 518 len 8192
6910 166/300 Write block 3764 len 40960 data: 2 3 4 5 5 4 4 4 4 3
6911 167/300 Write block 314 len 40960 data: 3 3 3 2 2 2 2 2 3 3
6912 168/300 Write block 4145 len 24576 data: 2 2 2 2 3 2
6913 169/300 Write block 3969 len 28672 data: 2 4 4 3 3 4 4
6914 170/300 Read block 254 len 32768
6915 171/300 Read block 4142 len 16384
6916 172/300 Read block 373 len 28672
6917 173/300 Flush
6918 174/300 Write block 196 len 20480 data: 4 3 3 2 2
6919 175/300 Write block 159 len 24576 data: 3 3 3 3 3 3
6920 176/300 Read block 4500 len 32768
6921 177/300 Write block 169 len 8192 data: 3 4
6922 178/300 Read block 990 len 36864
6923 179/300 Write block 4532 len 16384 data: 3 4 4 3
6924 180/300 Read block 4108 len 24576
6925 181/300 Read block 3232 len 32768
6926 182/300 Write block 39 len 24576 data: 2 2 2 2 2 3
6927 183/300 Write block 1992 len 4096 data: 2
6928 184/300 Read block 2905 len 12288
6929 185/300 Read block 3356 len 12288
6930 186/300 Write block 595 len 16384 data: 3 3 3 3
6931 187/300 Read block 4588 len 24576
6932 188/300 Write block 2717 len 8192 data: 2 2
6933 189/300 Write block 4449 len 32768 data: 3 3 4 4 5 5 5 4
6934 190/300 Write block 4143 len 16384 data: 3 3 3 3
6935 191/300 Read block 4004 len 40960
6936 192/300 Read block 1518 len 16384
6937 193/300 Write block 4386 len 8192 data: 4 4
6938 194/300 Write block 861 len 8192 data: 2 2
6939 195/300 Write block 4500 len 4096 data: 3
6940 196/300 Write block 4433 len 36864 data: 2 2 2 2 2 2 2 2 2
6941 197/300 Write block 1193 len 36864 data: 2 2 2 2 2 2 2 2 2
6942 198/300 Read block 3604 len 24576
6943 199/300 Read block 3686 len 24576
6944 200/300 Flush
6945 201/300 Write block 3868 len 32768 data: 3 3 3 2 2 2 3 4
6946 202/300 Read block 562 len 20480
6947 203/300 Read block 3949 len 8192
6948 204/300 Read block 2410 len 20480
6949 205/300 Read block 3772 len 32768
6950 206/300 Write block 4249 len 4096 data: 2
6951 207/300 Read block 1429 len 16384
6952 208/300 Write block 4100 len 20480 data: 3 4 5 5 4
6953 209/300 Read block 2539 len 28672
6954 210/300 Write block 2771 len 40960 data: 2 2 2 2 2 2 2 2 4 4
6955 211/300 Read block 3600 len 24576
6956 212/300 Read block 324 len 12288
6957 213/300 Write block 668 len 40960 data: 3 2 2 2 2 2 2 2 2 2
6958 214/300 Read block 534 len 12288
6959 215/300 Read block 4629 len 28672
6960 216/300 Read block 4634 len 4096
6961 217/300 Write block 511 len 12288 data: 4 4 4
6962 218/300 Write block 2469 len 36864 data: 3 4 3 3 3 2 2 2 2
6963 219/300 Write block 1385 len 28672 data: 3 3 3 2 3 3 3
6964 220/300 Write block 2500 len 36864 data: 3 3 3 3 2 2 2 2 2
6965 221/300 Write block 4272 len 12288 data: 2 2 3
6966 222/300 Flush
6967 223/300 Write block 4947 len 36864 data: 2 2 2 2 2 2 3 3 4
6968 224/300 Read block 3552 len 36864
6969 225/300 Write block 2793 len 4096 data: 3
6970 226/300 Write block 1693 len 4096 data: 4
6971 227/300 Read block 3467 len 32768
6972 228/300 Write block 4991 len 16384 data: 4 4 3 3
6973 229/300 Read block 3410 len 20480
6974 230/300 Read block 3243 len 12288
6975 231/300 Flush
6976 232/300 Read block 4869 len 24576
6977 233/300 Flush
6978 234/300 Read block 757 len 8192
6979 235/300 Read block 1542 len 28672
6980 236/300 Write block 4329 len 12288 data: 3 3 3
6981 237/300 Read block 2699 len 32768
6982 238/300 Read block 4447 len 24576
6983 239/300 Read block 4925 len 40960
6984 240/300 Read block 3864 len 12288
6985 241/300 Write block 3899 len 40960 data: 4 4 4 4 4 6 5 3 3 3
6986 242/300 Read block 3542 len 28672
6987 243/300 Read block 1055 len 36864
6988 244/300 Flush
6989 245/300 Write block 1881 len 40960 data: 3 3 2 3 3 3 3 3 4 4
6990 246/300 Flush
6991 247/300 Read block 421 len 8192
6992 248/300 Read block 3919 len 40960
6993 249/300 Read block 2618 len 8192
6994 250/300 Flush
6995 251/300 Write block 4123 len 16384 data: 4 5 4 4
6996 252/300 Read block 722 len 20480
6997 253/300 Flush
6998 254/300 Read block 1535 len 12288
6999 255/300 Write block 2409 len 32768 data: 3 2 2 3 3 3 2 3
7000 256/300 Write block 1967 len 16384 data: 3 3 3 3
7001 257/300 Write block 1122 len 28672 data: 3 3 3 3 3 3 3
7002 258/300 Read block 1853 len 32768
7003 259/300 Read block 1396 len 20480
7004 260/300 Write block 4296 len 36864 data: 3 2 2 2 2 2 2 2 2
7005 261/300 Read block 470 len 4096
7006 262/300 Read block 926 len 12288
7007 263/300 Read block 4192 len 20480
7008 264/300 Read block 773 len 40960
7009 265/300 Read block 2971 len 32768
7010 266/300 Read block 2964 len 4096
7011 267/300 Write block 2 len 32768 data: 3 2 3 3 2 2 2 2
7012 268/300 Read block 243 len 20480
7013 269/300 Write block 4395 len 20480 data: 4 3 2 2 2
7014 270/300 Read block 4195 len 8192
7015 271/300 Read block 2542 len 24576
7016 272/300 Write block 2207 len 36864 data: 3 4 4 4 4 4 2 2 2
7017 273/300 Write block 1338 len 28672 data: 4 5 5 5 4 3 3
7018 274/300 Read block 2894 len 28672
7019 275/300 Read block 104 len 4096
7020 276/300 Read block 4717 len 12288
7021 277/300 Read block 2590 len 12288
7022 278/300 Write block 144 len 28672 data: 3 2 2 2 2 3 3
7023 279/300 Read block 25 len 32768
7024 280/300 Write block 3763 len 32768 data: 2 3 4 5 6 6 5 5
7025 281/300 Write block 2092 len 28672 data: 3 3 3 2 3 3 3
7026 282/300 Read block 2486 len 40960
7027 283/300 Write block 404 len 36864 data: 2 2 2 2 3 3 3 3 2
7028 284/300 Flush
7029 285/300 Read block 2914 len 36864
7030 286/300 Write block 2166 len 8192 data: 2 2
7031 287/300 Write block 3331 len 36864 data: 2 2 2 2 2 2 2 2 2
7032 288/300 Write block 4840 len 36864 data: 4 4 4 3 4 4 3 2 2
7033 289/300 Read block 1780 len 16384
7034 290/300 Read block 2990 len 8192
7035 291/300 Write block 736 len 40960 data: 4 4 4 4 4 3 3 3 5 6
7036 292/300 Read block 933 len 32768
7037 293/300 Write block 3983 len 40960 data: 3 2 2 2 2 2 3 3 4 4
7038 294/300 Read block 1526 len 4096
7039 295/300 Read block 355 len 4096
7040 296/300 Write block 877 len 32768 data: 3 3 3 3 3 3 3 3
7041 297/300 Write block 2350 len 36864 data: 3 3 3 2 2 2 3 3 3
7042 298/300 Flush
7043 299/300 Read block 3941 len 4096
7044 300/300 Write block 1591 len 12288 data: 4 4 4
7045 301/300 Read block 3414 len 24576
7046 Main task has work to do, go find it
7047 got dsc Start(1)
7048 start 1
7049 Jan 29 05:12:13.198 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/1, method: POST, req_id: e56738ec-19c0-4dbc-b6b2-d35e4581250a, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
7050 Replay: started 1, returned:Ok(())
7051 ----------------------------------------------------------------
7052 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7053 States: Active Offline Active
7054 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7055 2812 Acked Write 10 Done Sent Done false
7056 2813 Acked Read 9 Done Sent Done false
7057 2814 Acked Write 8 Done Sent Done false
7058 2815 Acked Write 2 Done Sent Done false
7059 2816 Acked Read 8 Done Sent Done false
7060 2817 Acked Read 6 Done Sent Done false
7061 2818 Acked Read 3 Done Sent Done false
7062 2819 Acked Write 1 Done Sent Done false
7063 2820 Acked Write 6 Done Sent Done false
7064 2821 Acked Flush 0 Done Sent Done false
7065 2822 Acked Read 5 Done Sent Done false
7066 2823 Acked Read 8 Done Sent Done false
7067 2824 Acked Write 4 Done Sent Done false
7068 2825 Acked Read 3 Done Sent Done false
7069 2826 Acked Read 4 Done Sent Done false
7070 2827 Acked Read 8 Done Sent Done false
7071 2828 Acked Write 5 Done Sent Done false
7072 2829 Acked Read 7 Done Sent Done false
7073 2830 Acked Write 10 Done Sent Done false
7074 2831 Acked Write 10 Done Sent Done false
7075 2832 Acked Flush 0 Done Sent Done false
7076 2833 Acked Read 4 Done Sent Done false
7077 2834 Acked Flush 0 Done Sent Done false
7078 2835 Acked Flush 0 Done Sent Done false
7079 2836 Acked Read 4 Done Sent Done false
7080 2837 Acked Write 6 Done Sent Done false
7081 2838 Acked Write 1 Done Sent Done false
7082 2839 Acked Write 10 Done Sent Done false
7083 2840 Acked Read 9 Done Sent Done false
7084 2841 Acked Write 6 Done Sent Done false
7085 2842 Acked Write 4 Done Sent Done false
7086 2843 Acked Read 5 Done Sent Done false
7087 2844 Acked Read 4 Done Sent Done false
7088 2845 Acked Write 3 Done Sent Done false
7089 2846 Acked Write 4 Done Sent Done false
7090 2847 Acked Write 10 Done Sent Done false
7091 2848 Acked Write 6 Done Sent Done false
7092 2849 Acked Write 2 Done Sent Done false
7093 2850 Acked Read 5 Done Sent Done false
7094 2851 Acked Read 5 Done Sent Done false
7095 2852 Acked Flush 0 Done Sent Done false
7096 2853 Acked Read 3 Done Sent Done false
7097 2854 Acked Read 10 Done Sent Done false
7098 2855 Acked Read 5 Done Sent Done false
7099 2856 Acked Read 1 Done Sent Done false
7100 2857 Acked Read 4 Done Sent Done false
7101 2858 Acked Read 1 Done Sent Done false
7102 2859 Acked Read 9 Done Sent Done false
7103 2860 Acked Write 5 Done Sent Done false
7104 2861 Acked Read 5 Done Sent Done false
7105 2862 Acked Write 4 Done Sent Done false
7106 2863 Acked Flush 0 Done Sent Done false
7107 2864 Acked Flush 0 Done Sent Done false
7108 2865 Acked Read 1 Done Sent Done false
7109 2866 Acked Read 3 Done Sent Done false
7110 2867 Acked Flush 0 Done Sent Done false
7111 2868 Acked Write 4 Done Sent Done false
7112 2869 Acked Write 1 Done Sent Done false
7113 2870 Acked Read 4 Done Sent Done false
7114 2871 Acked Read 8 Done Sent Done false
7115 2872 Acked Read 7 Done Sent Done false
7116 2873 Acked Read 10 Done Sent Done false
7117 2874 Acked Read 4 Done Sent Done false
7118 2875 Acked Write 9 Done Sent Done false
7119 2876 Acked Read 6 Done Sent Done false
7120 2877 Acked Write 10 Done Sent Done false
7121 2878 Acked Flush 0 Done Sent Done false
7122 2879 Acked Write 5 Done Sent Done false
7123 2880 Acked Read 5 Done Sent Done false
7124 2881 Acked Read 6 Done Sent Done false
7125 2882 Acked Write 3 Done Sent Done false
7126 2883 Acked Read 10 Done Sent Done false
7127 2884 Acked Read 7 Done Sent Done false
7128 2885 Acked Read 1 Done Sent Done false
7129 2886 Acked Write 6 Done Sent Done false
7130 2887 Acked Write 5 Done Sent Done false
7131 2888 Acked Read 9 Done Sent Done false
7132 2889 Acked Write 4 Done Sent Done false
7133 2890 Acked Flush 0 Done Sent Done false
7134 2891 Acked Write 8 Done Sent Done false
7135 2892 Acked Read 8 Done Sent Done false
7136 2893 Acked Read 8 Done Sent Done false
7137 2894 Acked Flush 0 Done Sent Done false
7138 2895 Acked Read 6 Done Sent Done false
7139 2896 Acked Write 2 Done Sent Done false
7140 2897 Acked Read 8 Done Sent Done false
7141 2898 Acked Write 3 Done Sent Done false
7142 2899 Acked Write 8 Done Sent Done false
7143 2900 Acked Write 9 Done Sent Done false
7144 2901 Acked Write 10 Done Sent Done false
7145 2902 Acked Write 10 Done Sent Done false
7146 2903 Acked Write 5 Done Sent Done false
7147 2904 Acked Write 3 Done Sent Done false
7148 2905 Acked Read 8 Done Sent Done false
7149 2906 Acked Write 6 Done Sent Done false
7150 2907 Acked Read 3 Done Sent Done false
7151 2908 Acked Read 1 Done Sent Done false
7152 2909 Acked Write 7 Done Sent Done false
7153 2910 Acked Write 3 Done Sent Done false
7154 2911 Acked Write 2 Done Sent Done false
7155 2912 Acked Read 10 Done Sent Done false
7156 2913 Acked Flush 0 Done Sent Done false
7157 2914 Acked Write 8 Done Sent Done false
7158 2915 Acked Read 2 Done Sent Done false
7159 2916 Acked Write 6 Done Sent Done false
7160 2917 Acked Read 1 Done Sent Done false
7161 2918 Acked Read 3 Done Sent Done false
7162 2919 Acked Write 10 Done Sent Done false
7163 2920 Acked Read 2 Done Sent Done false
7164 2921 Acked Write 5 Done Sent Done false
7165 2922 Acked Write 9 Done Sent Done false
7166 2923 Acked Read 4 Done Sent Done false
7167 2924 Acked Read 2 Done Sent Done false
7168 2925 Acked Write 3 Done Sent Done false
7169 2926 Acked Write 2 Done Sent Done false
7170 2927 Acked Read 2 Done Sent Done false
7171 2928 Acked Read 7 Done Sent Done false
7172 2929 Acked Write 1 Done Sent Done false
7173 2930 Acked Read 1 Done Sent Done false
7174 2931 Acked Read 10 Done Sent Done false
7175 2932 Acked Flush 0 Done Sent Done false
7176 2933 Acked Write 1 Done Sent Done false
7177 2934 Acked Write 9 Done Sent Done false
7178 2935 Acked Write 2 Done Sent Done false
7179 2936 Acked Write 6 Done Sent Done false
7180 2937 Acked Read 6 Done Sent Done false
7181 2938 Acked Write 3 Done Sent Done false
7182 2939 Acked Write 1 Done Sent Done false
7183 2940 Acked Read 9 Done Sent Done false
7184 2941 Acked Flush 0 Done Sent Done false
7185 2942 Acked Read 2 Done Sent Done false
7186 2943 Acked Read 8 Done Sent Done false
7187 2944 Acked Write 8 Done Sent Done false
7188 2945 Acked Read 3 Done Sent Done false
7189 2946 Acked Write 6 Done Sent Done false
7190 2947 Acked Read 9 Done Sent Done false
7191 2948 Acked Read 3 Done Sent Done false
7192 2949 Acked Read 4 Done Sent Done false
7193 2950 Acked Read 5 Done Sent Done false
7194 2951 Acked Read 2 Done Sent Done false
7195 2952 Acked Read 2 Done Sent Done false
7196 2953 Acked Read 9 Done Sent Done false
7197 2954 Acked Read 5 Done Sent Done false
7198 2955 Acked Write 8 Done Sent Done false
7199 2956 Acked Flush 0 Done Sent Done false
7200 2957 Acked Read 3 Done Sent Done false
7201 2958 Acked Write 9 Done Sent Done false
7202 2959 Acked Write 8 Done Sent Done false
7203 2960 Acked Read 10 Done Sent Done false
7204 2961 Acked Write 2 Done Sent Done false
7205 2962 Acked Write 5 Done Sent Done false
7206 2963 Acked Write 7 Done Sent Done false
7207 2964 Acked Read 6 Done Sent Done false
7208 2965 Acked Flush 0 Done Sent Done false
7209 2966 Acked Read 5 Done Sent Done false
7210 2967 Acked Write 2 Done Sent Done false
7211 2968 Acked Read 7 Done Sent Done false
7212 2969 Acked Write 7 Done Sent Done false
7213 2970 Acked Read 7 Done Sent Done false
7214 2971 Acked Write 3 Done Sent Done false
7215 2972 Acked Write 9 Done Sent Done false
7216 2973 Acked Read 4 Done Sent Done false
7217 2974 Acked Write 6 Done Sent Done false
7218 2975 Acked Read 2 Done Sent Done false
7219 2976 Acked Read 2 Done Sent Done false
7220 2977 Acked Write 10 Done Sent Done false
7221 2978 Acked Write 10 Done Sent Done false
7222 2979 Acked Write 6 Done Sent Done false
7223 2980 Acked Write 7 Done Sent Done false
7224 2981 Acked Read 8 Done Sent Done false
7225 2982 Acked Read 4 Done Sent Done false
7226 2983 Acked Read 7 Done Sent Done false
7227 2984 Acked Flush 0 Done Sent Done false
7228 2985 Acked Write 5 Done Sent Done false
7229 2986 Acked Write 6 Done Sent Done false
7230 2987 Acked Read 8 Done Sent Done false
7231 2988 Acked Write 2 Done Sent Done false
7232 2989 Acked Read 9 Done Sent Done false
7233 2990 Acked Write 4 Done Sent Done false
7234 2991 Acked Read 6 Done Sent Done false
7235 2992 Acked Read 8 Done Sent Done false
7236 2993 Acked Write 6 Done Sent Done false
7237 2994 Acked Write 1 Done Sent Done false
7238 2995 Acked Read 3 Done Sent Done false
7239 2996 Acked Read 3 Done Sent Done false
7240 2997 Acked Write 4 Done Sent Done false
7241 2998 Acked Read 6 Done Sent Done false
7242 2999 Acked Write 2 Done Sent Done false
7243 3000 Acked Write 8 Done Sent Done false
7244 3001 Acked Write 4 Done Sent Done false
7245 3002 Acked Read 10 Done Sent Done false
7246 3003 Acked Read 4 Done Sent Done false
7247 3004 Acked Write 2 Done Sent Done false
7248 3005 Acked Write 2 Done Sent Done false
7249 3006 Acked Write 1 Done Sent Done false
7250 3007 Acked Write 9 Done Sent Done false
7251 3008 Acked Write 9 Done Sent Done false
7252 3009 Acked Read 6 Done Sent Done false
7253 3010 Acked Read 6 Done Sent Done false
7254 3011 Acked Flush 0 Done Sent Done false
7255 3012 Acked Write 8 Done Sent Done false
7256 3013 Acked Read 5 Done Sent Done false
7257 3014 Acked Read 2 Done Sent Done false
7258 3015 Acked Read 5 Done Sent Done false
7259 3016 Acked Read 8 Done Sent Done false
7260 3017 Acked Write 1 Done Sent Done false
7261 3018 Acked Read 4 Done Sent Done false
7262 3019 Acked Write 5 Done Sent Done false
7263 3020 Acked Read 7 Done Sent Done false
7264 3021 Acked Write 10 Done Sent Done false
7265 3022 Acked Read 6 Done Sent Done false
7266 3023 Acked Read 3 Done Sent Done false
7267 3024 Acked Write 10 Done Sent Done false
7268 3025 Acked Read 3 Done Sent Done false
7269 3026 Acked Read 7 Done Sent Done false
7270 3027 Acked Read 1 Done Sent Done false
7271 3028 Acked Write 3 Done Sent Done false
7272 3029 Acked Write 9 Done Sent Done false
7273 3030 Acked Write 7 Done Sent Done false
7274 3031 Acked Write 9 Done Sent Done false
7275 3032 Acked Write 3 Done Sent Done false
7276 3033 Acked Flush 0 Done Sent Done false
7277 3034 Acked Write 9 Done Sent Done false
7278 3035 Acked Read 9 Done Sent Done false
7279 3036 Acked Write 1 Done Sent Done false
7280 3037 Acked Write 1 Done Sent Done false
7281 3038 Acked Read 8 Done Sent Done false
7282 3039 Acked Write 4 Done Sent Done false
7283 3040 Acked Read 5 Done Sent Done false
7284 3041 Acked Read 3 Done Sent Done false
7285 3042 Acked Flush 0 Done Sent Done false
7286 3043 Acked Read 6 Done Sent Done false
7287 3044 Acked Flush 0 Done Sent Done false
7288 3045 Acked Read 2 Done Sent Done false
7289 3046 Acked Read 7 Done Sent Done false
7290 3047 Acked Write 3 Done Sent Done false
7291 3048 Acked Read 8 Done Sent Done false
7292 3049 Acked Read 6 Done Sent Done false
7293 3050 Acked Read 10 Done Sent Done false
7294 3051 Acked Read 3 Done Sent Done false
7295 3052 Acked Write 10 Done Sent Done false
7296 3053 Acked Read 7 Done Sent Done false
7297 3054 Acked Read 9 Done Sent Done false
7298 3055 Acked Flush 0 Done Sent Done false
7299 3056 Acked Write 10 Done Sent Done false
7300 3057 Acked Flush 0 Done Sent Done false
7301 3058 Acked Read 2 Done Sent Done false
7302 3059 Acked Read 10 Done Sent Done false
7303 3060 Acked Read 2 Done Sent Done false
7304 3061 Acked Flush 0 Done Sent Done false
7305 3062 Acked Write 4 Done Sent Done false
7306 3063 Acked Read 5 Done Sent Done false
7307 3064 Acked Flush 0 Done Sent Done false
7308 3065 Acked Read 3 Done Sent Done false
7309 3066 Acked Write 8 Done Sent Done false
7310 3067 Acked Write 4 Done Sent Done false
7311 3068 Acked Write 7 Done Sent Done false
7312 3069 Acked Read 8 Done Sent Done false
7313 3070 Acked Read 5 Done Sent Done false
7314 3071 Acked Write 9 Done Sent Done false
7315 3072 Acked Read 1 Done Sent Done false
7316 3073 Acked Read 3 Done Sent Done false
7317 3074 Acked Read 5 Done Sent Done false
7318 3075 Acked Read 10 Done Sent Done false
7319 3076 Acked Read 8 Done Sent Done false
7320 3077 Acked Read 1 Done Sent Done false
7321 3078 Acked Write 8 Done Sent Done false
7322 3079 Acked Read 5 Done Sent Done false
7323 3080 Acked Write 5 Done Sent Done false
7324 3081 Acked Read 2 Done Sent Done false
7325 3082 Acked Read 6 Done Sent Done false
7326 3083 Acked Write 9 Done Sent Done false
7327 3084 Acked Write 7 Done Sent Done false
7328 3085 Acked Read 7 Done Sent Done false
7329 3086 Acked Read 1 Done Sent Done false
7330 3087 Acked Read 3 Done Sent Done false
7331 3088 Acked Read 3 Done Sent Done false
7332 3089 Acked Write 7 Done Sent Done false
7333 3090 Acked Read 8 Done Sent Done false
7334 3091 Acked Write 8 Done Sent Done false
7335 3092 Acked Write 7 Done Sent Done false
7336 3093 Acked Read 10 Done Sent Done false
7337 3094 Acked Write 9 Done Sent Done false
7338 3095 Acked Flush 0 Done Sent Done false
7339 3096 Acked Read 9 Done Sent Done false
7340 3097 Acked Write 2 Done Sent Done false
7341 3098 Acked Write 9 Done Sent Done false
7342 3099 Acked Write 9 Done Sent Done false
7343 3100 Acked Read 4 Done Sent Done false
7344 3101 Acked Read 2 Done Sent Done false
7345 3102 Acked Write 10 Done Sent Done false
7346 3103 Acked Read 8 Done Sent Done false
7347 3104 Acked Write 10 Done Sent Done false
7348 3105 Acked Read 1 Done Sent Done false
7349 3106 Acked Read 1 Done Sent Done false
7350 3107 Acked Write 8 Done Sent Done false
7351 3108 Acked Write 9 Done Sent Done false
7352 3109 Acked Flush 0 Done Sent Done false
7353 3110 Acked Read 1 Done Sent Done false
7354 3111 Acked Write 3 Done Sent Done false
7355 3112 Acked Read 6 Done Sent Done false
7356 STATES DS:0 DS:1 DS:2 TOTAL
7357 Sent 0 301 0 301
7358 Done 301 0 301 602
7359 Skipped 0 0 0 0
7360 Error 0 0 0 0
7361 Last Flush: Some(JobId(3109)) Some(JobId(2811)) Some(JobId(3109))
7362 Downstairs last five completed: 2811 2810 2809 2808 2807
7363 Upstairs last five completed: 3112 3111 3110 3109 3108
7364 CLIENT: Up:0 ds:301 act:2
73652025-01-29T05:12:14.060ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
73662025-01-29T05:12:15.203ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
73672025-01-29T05:12:16.064ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
73682025-01-29T05:12:17.066ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
7369 ----------------------------------------------------------------
7370 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7371 States: Active Offline Active
7372 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7373 2812 Acked Write 10 Done Sent Done false
7374 2813 Acked Read 9 Done Sent Done false
7375 2814 Acked Write 8 Done Sent Done false
7376 2815 Acked Write 2 Done Sent Done false
7377 2816 Acked Read 8 Done Sent Done false
7378 2817 Acked Read 6 Done Sent Done false
7379 2818 Acked Read 3 Done Sent Done false
7380 2819 Acked Write 1 Done Sent Done false
7381 2820 Acked Write 6 Done Sent Done false
7382 2821 Acked Flush 0 Done Sent Done false
7383 2822 Acked Read 5 Done Sent Done false
7384 2823 Acked Read 8 Done Sent Done false
7385 2824 Acked Write 4 Done Sent Done false
7386 2825 Acked Read 3 Done Sent Done false
7387 2826 Acked Read 4 Done Sent Done false
7388 2827 Acked Read 8 Done Sent Done false
7389 2828 Acked Write 5 Done Sent Done false
7390 2829 Acked Read 7 Done Sent Done false
7391 2830 Acked Write 10 Done Sent Done false
7392 2831 Acked Write 10 Done Sent Done false
7393 2832 Acked Flush 0 Done Sent Done false
7394 2833 Acked Read 4 Done Sent Done false
7395 2834 Acked Flush 0 Done Sent Done false
7396 2835 Acked Flush 0 Done Sent Done false
7397 2836 Acked Read 4 Done Sent Done false
7398 2837 Acked Write 6 Done Sent Done false
7399 2838 Acked Write 1 Done Sent Done false
7400 2839 Acked Write 10 Done Sent Done false
7401 2840 Acked Read 9 Done Sent Done false
7402 2841 Acked Write 6 Done Sent Done false
7403 2842 Acked Write 4 Done Sent Done false
7404 2843 Acked Read 5 Done Sent Done false
7405 2844 Acked Read 4 Done Sent Done false
7406 2845 Acked Write 3 Done Sent Done false
7407 2846 Acked Write 4 Done Sent Done false
7408 2847 Acked Write 10 Done Sent Done false
7409 2848 Acked Write 6 Done Sent Done false
7410 2849 Acked Write 2 Done Sent Done false
7411 2850 Acked Read 5 Done Sent Done false
7412 2851 Acked Read 5 Done Sent Done false
7413 2852 Acked Flush 0 Done Sent Done false
7414 2853 Acked Read 3 Done Sent Done false
7415 2854 Acked Read 10 Done Sent Done false
7416 2855 Acked Read 5 Done Sent Done false
7417 2856 Acked Read 1 Done Sent Done false
7418 2857 Acked Read 4 Done Sent Done false
7419 2858 Acked Read 1 Done Sent Done false
7420 2859 Acked Read 9 Done Sent Done false
7421 2860 Acked Write 5 Done Sent Done false
7422 2861 Acked Read 5 Done Sent Done false
7423 2862 Acked Write 4 Done Sent Done false
7424 2863 Acked Flush 0 Done Sent Done false
7425 2864 Acked Flush 0 Done Sent Done false
7426 2865 Acked Read 1 Done Sent Done false
7427 2866 Acked Read 3 Done Sent Done false
7428 2867 Acked Flush 0 Done Sent Done false
7429 2868 Acked Write 4 Done Sent Done false
7430 2869 Acked Write 1 Done Sent Done false
7431 2870 Acked Read 4 Done Sent Done false
7432 2871 Acked Read 8 Done Sent Done false
7433 2872 Acked Read 7 Done Sent Done false
7434 2873 Acked Read 10 Done Sent Done false
7435 2874 Acked Read 4 Done Sent Done false
7436 2875 Acked Write 9 Done Sent Done false
7437 2876 Acked Read 6 Done Sent Done false
7438 2877 Acked Write 10 Done Sent Done false
7439 2878 Acked Flush 0 Done Sent Done false
7440 2879 Acked Write 5 Done Sent Done false
7441 2880 Acked Read 5 Done Sent Done false
7442 2881 Acked Read 6 Done Sent Done false
7443 2882 Acked Write 3 Done Sent Done false
7444 2883 Acked Read 10 Done Sent Done false
7445 2884 Acked Read 7 Done Sent Done false
7446 2885 Acked Read 1 Done Sent Done false
7447 2886 Acked Write 6 Done Sent Done false
7448 2887 Acked Write 5 Done Sent Done false
7449 2888 Acked Read 9 Done Sent Done false
7450 2889 Acked Write 4 Done Sent Done false
7451 2890 Acked Flush 0 Done Sent Done false
7452 2891 Acked Write 8 Done Sent Done false
7453 2892 Acked Read 8 Done Sent Done false
7454 2893 Acked Read 8 Done Sent Done false
7455 2894 Acked Flush 0 Done Sent Done false
7456 2895 Acked Read 6 Done Sent Done false
7457 2896 Acked Write 2 Done Sent Done false
7458 2897 Acked Read 8 Done Sent Done false
7459 2898 Acked Write 3 Done Sent Done false
7460 2899 Acked Write 8 Done Sent Done false
7461 2900 Acked Write 9 Done Sent Done false
7462 2901 Acked Write 10 Done Sent Done false
7463 2902 Acked Write 10 Done Sent Done false
7464 2903 Acked Write 5 Done Sent Done false
7465 2904 Acked Write 3 Done Sent Done false
7466 2905 Acked Read 8 Done Sent Done false
7467 2906 Acked Write 6 Done Sent Done false
7468 2907 Acked Read 3 Done Sent Done false
7469 2908 Acked Read 1 Done Sent Done false
7470 2909 Acked Write 7 Done Sent Done false
7471 2910 Acked Write 3 Done Sent Done false
7472 2911 Acked Write 2 Done Sent Done false
7473 2912 Acked Read 10 Done Sent Done false
7474 2913 Acked Flush 0 Done Sent Done false
7475 2914 Acked Write 8 Done Sent Done false
7476 2915 Acked Read 2 Done Sent Done false
7477 2916 Acked Write 6 Done Sent Done false
7478 2917 Acked Read 1 Done Sent Done false
7479 2918 Acked Read 3 Done Sent Done false
7480 2919 Acked Write 10 Done Sent Done false
7481 2920 Acked Read 2 Done Sent Done false
7482 2921 Acked Write 5 Done Sent Done false
7483 2922 Acked Write 9 Done Sent Done false
7484 2923 Acked Read 4 Done Sent Done false
7485 2924 Acked Read 2 Done Sent Done false
7486 2925 Acked Write 3 Done Sent Done false
7487 2926 Acked Write 2 Done Sent Done false
7488 2927 Acked Read 2 Done Sent Done false
7489 2928 Acked Read 7 Done Sent Done false
7490 2929 Acked Write 1 Done Sent Done false
7491 2930 Acked Read 1 Done Sent Done false
7492 2931 Acked Read 10 Done Sent Done false
7493 2932 Acked Flush 0 Done Sent Done false
7494 2933 Acked Write 1 Done Sent Done false
7495 2934 Acked Write 9 Done Sent Done false
7496 2935 Acked Write 2 Done Sent Done false
7497 2936 Acked Write 6 Done Sent Done false
7498 2937 Acked Read 6 Done Sent Done false
7499 2938 Acked Write 3 Done Sent Done false
7500 2939 Acked Write 1 Done Sent Done false
7501 2940 Acked Read 9 Done Sent Done false
7502 2941 Acked Flush 0 Done Sent Done false
7503 2942 Acked Read 2 Done Sent Done false
7504 2943 Acked Read 8 Done Sent Done false
7505 2944 Acked Write 8 Done Sent Done false
7506 2945 Acked Read 3 Done Sent Done false
7507 2946 Acked Write 6 Done Sent Done false
7508 2947 Acked Read 9 Done Sent Done false
7509 2948 Acked Read 3 Done Sent Done false
7510 2949 Acked Read 4 Done Sent Done false
7511 2950 Acked Read 5 Done Sent Done false
7512 2951 Acked Read 2 Done Sent Done false
7513 2952 Acked Read 2 Done Sent Done false
7514 2953 Acked Read 9 Done Sent Done false
7515 2954 Acked Read 5 Done Sent Done false
7516 2955 Acked Write 8 Done Sent Done false
7517 2956 Acked Flush 0 Done Sent Done false
7518 2957 Acked Read 3 Done Sent Done false
7519 2958 Acked Write 9 Done Sent Done false
7520 2959 Acked Write 8 Done Sent Done false
7521 2960 Acked Read 10 Done Sent Done false
7522 2961 Acked Write 2 Done Sent Done false
7523 2962 Acked Write 5 Done Sent Done false
7524 2963 Acked Write 7 Done Sent Done false
7525 2964 Acked Read 6 Done Sent Done false
7526 2965 Acked Flush 0 Done Sent Done false
7527 2966 Acked Read 5 Done Sent Done false
7528 2967 Acked Write 2 Done Sent Done false
7529 2968 Acked Read 7 Done Sent Done false
7530 2969 Acked Write 7 Done Sent Done false
7531 2970 Acked Read 7 Done Sent Done false
7532 2971 Acked Write 3 Done Sent Done false
7533 2972 Acked Write 9 Done Sent Done false
7534 2973 Acked Read 4 Done Sent Done false
7535 2974 Acked Write 6 Done Sent Done false
7536 2975 Acked Read 2 Done Sent Done false
7537 2976 Acked Read 2 Done Sent Done false
7538 2977 Acked Write 10 Done Sent Done false
7539 2978 Acked Write 10 Done Sent Done false
7540 2979 Acked Write 6 Done Sent Done false
7541 2980 Acked Write 7 Done Sent Done false
7542 2981 Acked Read 8 Done Sent Done false
7543 2982 Acked Read 4 Done Sent Done false
7544 2983 Acked Read 7 Done Sent Done false
7545 2984 Acked Flush 0 Done Sent Done false
7546 2985 Acked Write 5 Done Sent Done false
7547 2986 Acked Write 6 Done Sent Done false
7548 2987 Acked Read 8 Done Sent Done false
7549 2988 Acked Write 2 Done Sent Done false
7550 2989 Acked Read 9 Done Sent Done false
7551 2990 Acked Write 4 Done Sent Done false
7552 2991 Acked Read 6 Done Sent Done false
7553 2992 Acked Read 8 Done Sent Done false
7554 2993 Acked Write 6 Done Sent Done false
7555 2994 Acked Write 1 Done Sent Done false
7556 2995 Acked Read 3 Done Sent Done false
7557 2996 Acked Read 3 Done Sent Done false
7558 2997 Acked Write 4 Done Sent Done false
7559 2998 Acked Read 6 Done Sent Done false
7560 2999 Acked Write 2 Done Sent Done false
7561 3000 Acked Write 8 Done Sent Done false
7562 3001 Acked Write 4 Done Sent Done false
7563 3002 Acked Read 10 Done Sent Done false
7564 3003 Acked Read 4 Done Sent Done false
7565 3004 Acked Write 2 Done Sent Done false
7566 3005 Acked Write 2 Done Sent Done false
7567 3006 Acked Write 1 Done Sent Done false
7568 3007 Acked Write 9 Done Sent Done false
7569 3008 Acked Write 9 Done Sent Done false
7570 3009 Acked Read 6 Done Sent Done false
7571 3010 Acked Read 6 Done Sent Done false
7572 3011 Acked Flush 0 Done Sent Done false
7573 3012 Acked Write 8 Done Sent Done false
7574 3013 Acked Read 5 Done Sent Done false
7575 3014 Acked Read 2 Done Sent Done false
7576 3015 Acked Read 5 Done Sent Done false
7577 3016 Acked Read 8 Done Sent Done false
7578 3017 Acked Write 1 Done Sent Done false
7579 3018 Acked Read 4 Done Sent Done false
7580 3019 Acked Write 5 Done Sent Done false
7581 3020 Acked Read 7 Done Sent Done false
7582 3021 Acked Write 10 Done Sent Done false
7583 3022 Acked Read 6 Done Sent Done false
7584 3023 Acked Read 3 Done Sent Done false
7585 3024 Acked Write 10 Done Sent Done false
7586 3025 Acked Read 3 Done Sent Done false
7587 3026 Acked Read 7 Done Sent Done false
7588 3027 Acked Read 1 Done Sent Done false
7589 3028 Acked Write 3 Done Sent Done false
7590 3029 Acked Write 9 Done Sent Done false
7591 3030 Acked Write 7 Done Sent Done false
7592 3031 Acked Write 9 Done Sent Done false
7593 3032 Acked Write 3 Done Sent Done false
7594 3033 Acked Flush 0 Done Sent Done false
7595 3034 Acked Write 9 Done Sent Done false
7596 3035 Acked Read 9 Done Sent Done false
7597 3036 Acked Write 1 Done Sent Done false
7598 3037 Acked Write 1 Done Sent Done false
7599 3038 Acked Read 8 Done Sent Done false
7600 3039 Acked Write 4 Done Sent Done false
7601 3040 Acked Read 5 Done Sent Done false
7602 3041 Acked Read 3 Done Sent Done false
7603 3042 Acked Flush 0 Done Sent Done false
7604 3043 Acked Read 6 Done Sent Done false
7605 3044 Acked Flush 0 Done Sent Done false
7606 3045 Acked Read 2 Done Sent Done false
7607 3046 Acked Read 7 Done Sent Done false
7608 3047 Acked Write 3 Done Sent Done false
7609 3048 Acked Read 8 Done Sent Done false
7610 3049 Acked Read 6 Done Sent Done false
7611 3050 Acked Read 10 Done Sent Done false
7612 3051 Acked Read 3 Done Sent Done false
7613 3052 Acked Write 10 Done Sent Done false
7614 3053 Acked Read 7 Done Sent Done false
7615 3054 Acked Read 9 Done Sent Done false
7616 3055 Acked Flush 0 Done Sent Done false
7617 3056 Acked Write 10 Done Sent Done false
7618 3057 Acked Flush 0 Done Sent Done false
7619 3058 Acked Read 2 Done Sent Done false
7620 3059 Acked Read 10 Done Sent Done false
7621 3060 Acked Read 2 Done Sent Done false
7622 3061 Acked Flush 0 Done Sent Done false
7623 3062 Acked Write 4 Done Sent Done false
7624 3063 Acked Read 5 Done Sent Done false
7625 3064 Acked Flush 0 Done Sent Done false
7626 3065 Acked Read 3 Done Sent Done false
7627 3066 Acked Write 8 Done Sent Done false
7628 3067 Acked Write 4 Done Sent Done false
7629 3068 Acked Write 7 Done Sent Done false
7630 3069 Acked Read 8 Done Sent Done false
7631 3070 Acked Read 5 Done Sent Done false
7632 3071 Acked Write 9 Done Sent Done false
7633 3072 Acked Read 1 Done Sent Done false
7634 3073 Acked Read 3 Done Sent Done false
7635 3074 Acked Read 5 Done Sent Done false
7636 3075 Acked Read 10 Done Sent Done false
7637 3076 Acked Read 8 Done Sent Done false
7638 3077 Acked Read 1 Done Sent Done false
7639 3078 Acked Write 8 Done Sent Done false
7640 3079 Acked Read 5 Done Sent Done false
7641 3080 Acked Write 5 Done Sent Done false
7642 3081 Acked Read 2 Done Sent Done false
7643 3082 Acked Read 6 Done Sent Done false
7644 3083 Acked Write 9 Done Sent Done false
7645 3084 Acked Write 7 Done Sent Done false
7646 3085 Acked Read 7 Done Sent Done false
7647 3086 Acked Read 1 Done Sent Done false
7648 3087 Acked Read 3 Done Sent Done false
7649 3088 Acked Read 3 Done Sent Done false
7650 3089 Acked Write 7 Done Sent Done false
7651 3090 Acked Read 8 Done Sent Done false
7652 3091 Acked Write 8 Done Sent Done false
7653 3092 Acked Write 7 Done Sent Done false
7654 3093 Acked Read 10 Done Sent Done false
7655 3094 Acked Write 9 Done Sent Done false
7656 3095 Acked Flush 0 Done Sent Done false
7657 3096 Acked Read 9 Done Sent Done false
7658 3097 Acked Write 2 Done Sent Done false
7659 3098 Acked Write 9 Done Sent Done false
7660 3099 Acked Write 9 Done Sent Done false
7661 3100 Acked Read 4 Done Sent Done false
7662 3101 Acked Read 2 Done Sent Done false
7663 3102 Acked Write 10 Done Sent Done false
7664 3103 Acked Read 8 Done Sent Done false
7665 3104 Acked Write 10 Done Sent Done false
7666 3105 Acked Read 1 Done Sent Done false
7667 3106 Acked Read 1 Done Sent Done false
7668 3107 Acked Write 8 Done Sent Done false
7669 3108 Acked Write 9 Done Sent Done false
7670 3109 Acked Flush 0 Done Sent Done false
7671 3110 Acked Read 1 Done Sent Done false
7672 3111 Acked Write 3 Done Sent Done false
7673 3112 Acked Read 6 Done Sent Done false
7674 STATES DS:0 DS:1 DS:2 TOTAL
7675 Sent 0 301 0 301
7676 Done 301 0 301 602
7677 Skipped 0 0 0 0
7678 Error 0 0 0 0
7679 Last Flush: Some(JobId(3109)) Some(JobId(2811)) Some(JobId(3109))
7680 Downstairs last five completed: 2811 2810 2809 2808 2807
7681 Upstairs last five completed: 3112 3111 3110 3109 3108
7682 CLIENT: Up:0 ds:301 act:2
76832025-01-29T05:12:18.066ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76842025-01-29T05:12:18.956ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76852025-01-29T05:12:18.957ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76862025-01-29T05:12:18.957ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76872025-01-29T05:12:18.957ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76882025-01-29T05:12:18.957ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
7689 [8820] got start action
7690 [8820] I am going to restart
7691 Starting downstairs at port 8820
7692 Make output file at "/tmp/dsc/downstairs-8820.txt"
7693 [8820][1] reports Starting
7694 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1209)
7695 [8820][1] reports Running
76962025-01-29T05:12:19.458ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76972025-01-29T05:12:20.203ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
76982025-01-29T05:12:21.070ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
7699 ----------------------------------------------------------------
7700 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7701 States: Active Offline Active
7702 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7703 2812 Acked Write 10 Done Sent Done false
7704 2813 Acked Read 9 Done Sent Done false
7705 2814 Acked Write 8 Done Sent Done false
7706 2815 Acked Write 2 Done Sent Done false
7707 2816 Acked Read 8 Done Sent Done false
7708 2817 Acked Read 6 Done Sent Done false
7709 2818 Acked Read 3 Done Sent Done false
7710 2819 Acked Write 1 Done Sent Done false
7711 2820 Acked Write 6 Done Sent Done false
7712 2821 Acked Flush 0 Done Sent Done false
7713 2822 Acked Read 5 Done Sent Done false
7714 2823 Acked Read 8 Done Sent Done false
7715 2824 Acked Write 4 Done Sent Done false
7716 2825 Acked Read 3 Done Sent Done false
7717 2826 Acked Read 4 Done Sent Done false
7718 2827 Acked Read 8 Done Sent Done false
7719 2828 Acked Write 5 Done Sent Done false
7720 2829 Acked Read 7 Done Sent Done false
7721 2830 Acked Write 10 Done Sent Done false
7722 2831 Acked Write 10 Done Sent Done false
7723 2832 Acked Flush 0 Done Sent Done false
7724 2833 Acked Read 4 Done Sent Done false
7725 2834 Acked Flush 0 Done Sent Done false
7726 2835 Acked Flush 0 Done Sent Done false
7727 2836 Acked Read 4 Done Sent Done false
7728 2837 Acked Write 6 Done Sent Done false
7729 2838 Acked Write 1 Done Sent Done false
7730 2839 Acked Write 10 Done Sent Done false
7731 2840 Acked Read 9 Done Sent Done false
7732 2841 Acked Write 6 Done Sent Done false
7733 2842 Acked Write 4 Done Sent Done false
7734 2843 Acked Read 5 Done Sent Done false
7735 2844 Acked Read 4 Done Sent Done false
7736 2845 Acked Write 3 Done Sent Done false
7737 2846 Acked Write 4 Done Sent Done false
7738 2847 Acked Write 10 Done Sent Done false
7739 2848 Acked Write 6 Done Sent Done false
7740 2849 Acked Write 2 Done Sent Done false
7741 2850 Acked Read 5 Done Sent Done false
7742 2851 Acked Read 5 Done Sent Done false
7743 2852 Acked Flush 0 Done Sent Done false
7744 2853 Acked Read 3 Done Sent Done false
7745 2854 Acked Read 10 Done Sent Done false
7746 2855 Acked Read 5 Done Sent Done false
7747 2856 Acked Read 1 Done Sent Done false
7748 2857 Acked Read 4 Done Sent Done false
7749 2858 Acked Read 1 Done Sent Done false
7750 2859 Acked Read 9 Done Sent Done false
7751 2860 Acked Write 5 Done Sent Done false
7752 2861 Acked Read 5 Done Sent Done false
7753 2862 Acked Write 4 Done Sent Done false
7754 2863 Acked Flush 0 Done Sent Done false
7755 2864 Acked Flush 0 Done Sent Done false
7756 2865 Acked Read 1 Done Sent Done false
7757 2866 Acked Read 3 Done Sent Done false
7758 2867 Acked Flush 0 Done Sent Done false
7759 2868 Acked Write 4 Done Sent Done false
7760 2869 Acked Write 1 Done Sent Done false
7761 2870 Acked Read 4 Done Sent Done false
7762 2871 Acked Read 8 Done Sent Done false
7763 2872 Acked Read 7 Done Sent Done false
7764 2873 Acked Read 10 Done Sent Done false
7765 2874 Acked Read 4 Done Sent Done false
7766 2875 Acked Write 9 Done Sent Done false
7767 2876 Acked Read 6 Done Sent Done false
7768 2877 Acked Write 10 Done Sent Done false
7769 2878 Acked Flush 0 Done Sent Done false
7770 2879 Acked Write 5 Done Sent Done false
7771 2880 Acked Read 5 Done Sent Done false
7772 2881 Acked Read 6 Done Sent Done false
7773 2882 Acked Write 3 Done Sent Done false
7774 2883 Acked Read 10 Done Sent Done false
7775 2884 Acked Read 7 Done Sent Done false
7776 2885 Acked Read 1 Done Sent Done false
7777 2886 Acked Write 6 Done Sent Done false
7778 2887 Acked Write 5 Done Sent Done false
7779 2888 Acked Read 9 Done Sent Done false
7780 2889 Acked Write 4 Done Sent Done false
7781 2890 Acked Flush 0 Done Sent Done false
7782 2891 Acked Write 8 Done Sent Done false
7783 2892 Acked Read 8 Done Sent Done false
7784 2893 Acked Read 8 Done Sent Done false
7785 2894 Acked Flush 0 Done Sent Done false
7786 2895 Acked Read 6 Done Sent Done false
7787 2896 Acked Write 2 Done Sent Done false
7788 2897 Acked Read 8 Done Sent Done false
7789 2898 Acked Write 3 Done Sent Done false
7790 2899 Acked Write 8 Done Sent Done false
7791 2900 Acked Write 9 Done Sent Done false
7792 2901 Acked Write 10 Done Sent Done false
7793 2902 Acked Write 10 Done Sent Done false
7794 2903 Acked Write 5 Done Sent Done false
7795 2904 Acked Write 3 Done Sent Done false
7796 2905 Acked Read 8 Done Sent Done false
7797 2906 Acked Write 6 Done Sent Done false
7798 2907 Acked Read 3 Done Sent Done false
7799 2908 Acked Read 1 Done Sent Done false
7800 2909 Acked Write 7 Done Sent Done false
7801 2910 Acked Write 3 Done Sent Done false
7802 2911 Acked Write 2 Done Sent Done false
7803 2912 Acked Read 10 Done Sent Done false
7804 2913 Acked Flush 0 Done Sent Done false
7805 2914 Acked Write 8 Done Sent Done false
7806 2915 Acked Read 2 Done Sent Done false
7807 2916 Acked Write 6 Done Sent Done false
7808 2917 Acked Read 1 Done Sent Done false
7809 2918 Acked Read 3 Done Sent Done false
7810 2919 Acked Write 10 Done Sent Done false
7811 2920 Acked Read 2 Done Sent Done false
7812 2921 Acked Write 5 Done Sent Done false
7813 2922 Acked Write 9 Done Sent Done false
7814 2923 Acked Read 4 Done Sent Done false
7815 2924 Acked Read 2 Done Sent Done false
7816 2925 Acked Write 3 Done Sent Done false
7817 2926 Acked Write 2 Done Sent Done false
7818 2927 Acked Read 2 Done Sent Done false
7819 2928 Acked Read 7 Done Sent Done false
7820 2929 Acked Write 1 Done Sent Done false
7821 2930 Acked Read 1 Done Sent Done false
7822 2931 Acked Read 10 Done Sent Done false
7823 2932 Acked Flush 0 Done Sent Done false
7824 2933 Acked Write 1 Done Sent Done false
7825 2934 Acked Write 9 Done Sent Done false
7826 2935 Acked Write 2 Done Sent Done false
7827 2936 Acked Write 6 Done Sent Done false
7828 2937 Acked Read 6 Done Sent Done false
7829 2938 Acked Write 3 Done Sent Done false
7830 2939 Acked Write 1 Done Sent Done false
7831 2940 Acked Read 9 Done Sent Done false
7832 2941 Acked Flush 0 Done Sent Done false
7833 2942 Acked Read 2 Done Sent Done false
7834 2943 Acked Read 8 Done Sent Done false
7835 2944 Acked Write 8 Done Sent Done false
7836 2945 Acked Read 3 Done Sent Done false
7837 2946 Acked Write 6 Done Sent Done false
7838 2947 Acked Read 9 Done Sent Done false
7839 2948 Acked Read 3 Done Sent Done false
7840 2949 Acked Read 4 Done Sent Done false
7841 2950 Acked Read 5 Done Sent Done false
7842 2951 Acked Read 2 Done Sent Done false
7843 2952 Acked Read 2 Done Sent Done false
7844 2953 Acked Read 9 Done Sent Done false
7845 2954 Acked Read 5 Done Sent Done false
7846 2955 Acked Write 8 Done Sent Done false
7847 2956 Acked Flush 0 Done Sent Done false
7848 2957 Acked Read 3 Done Sent Done false
7849 2958 Acked Write 9 Done Sent Done false
7850 2959 Acked Write 8 Done Sent Done false
7851 2960 Acked Read 10 Done Sent Done false
7852 2961 Acked Write 2 Done Sent Done false
7853 2962 Acked Write 5 Done Sent Done false
7854 2963 Acked Write 7 Done Sent Done false
7855 2964 Acked Read 6 Done Sent Done false
7856 2965 Acked Flush 0 Done Sent Done false
7857 2966 Acked Read 5 Done Sent Done false
7858 2967 Acked Write 2 Done Sent Done false
7859 2968 Acked Read 7 Done Sent Done false
7860 2969 Acked Write 7 Done Sent Done false
7861 2970 Acked Read 7 Done Sent Done false
7862 2971 Acked Write 3 Done Sent Done false
7863 2972 Acked Write 9 Done Sent Done false
7864 2973 Acked Read 4 Done Sent Done false
7865 2974 Acked Write 6 Done Sent Done false
7866 2975 Acked Read 2 Done Sent Done false
7867 2976 Acked Read 2 Done Sent Done false
7868 2977 Acked Write 10 Done Sent Done false
7869 2978 Acked Write 10 Done Sent Done false
7870 2979 Acked Write 6 Done Sent Done false
7871 2980 Acked Write 7 Done Sent Done false
7872 2981 Acked Read 8 Done Sent Done false
7873 2982 Acked Read 4 Done Sent Done false
7874 2983 Acked Read 7 Done Sent Done false
7875 2984 Acked Flush 0 Done Sent Done false
7876 2985 Acked Write 5 Done Sent Done false
7877 2986 Acked Write 6 Done Sent Done false
7878 2987 Acked Read 8 Done Sent Done false
7879 2988 Acked Write 2 Done Sent Done false
7880 2989 Acked Read 9 Done Sent Done false
7881 2990 Acked Write 4 Done Sent Done false
7882 2991 Acked Read 6 Done Sent Done false
7883 2992 Acked Read 8 Done Sent Done false
7884 2993 Acked Write 6 Done Sent Done false
7885 2994 Acked Write 1 Done Sent Done false
7886 2995 Acked Read 3 Done Sent Done false
7887 2996 Acked Read 3 Done Sent Done false
7888 2997 Acked Write 4 Done Sent Done false
7889 2998 Acked Read 6 Done Sent Done false
7890 2999 Acked Write 2 Done Sent Done false
7891 3000 Acked Write 8 Done Sent Done false
7892 3001 Acked Write 4 Done Sent Done false
7893 3002 Acked Read 10 Done Sent Done false
7894 3003 Acked Read 4 Done Sent Done false
7895 3004 Acked Write 2 Done Sent Done false
7896 3005 Acked Write 2 Done Sent Done false
7897 3006 Acked Write 1 Done Sent Done false
7898 3007 Acked Write 9 Done Sent Done false
7899 3008 Acked Write 9 Done Sent Done false
7900 3009 Acked Read 6 Done Sent Done false
7901 3010 Acked Read 6 Done Sent Done false
7902 3011 Acked Flush 0 Done Sent Done false
7903 3012 Acked Write 8 Done Sent Done false
7904 3013 Acked Read 5 Done Sent Done false
7905 3014 Acked Read 2 Done Sent Done false
7906 3015 Acked Read 5 Done Sent Done false
7907 3016 Acked Read 8 Done Sent Done false
7908 3017 Acked Write 1 Done Sent Done false
7909 3018 Acked Read 4 Done Sent Done false
7910 3019 Acked Write 5 Done Sent Done false
7911 3020 Acked Read 7 Done Sent Done false
7912 3021 Acked Write 10 Done Sent Done false
7913 3022 Acked Read 6 Done Sent Done false
7914 3023 Acked Read 3 Done Sent Done false
7915 3024 Acked Write 10 Done Sent Done false
7916 3025 Acked Read 3 Done Sent Done false
7917 3026 Acked Read 7 Done Sent Done false
7918 3027 Acked Read 1 Done Sent Done false
7919 3028 Acked Write 3 Done Sent Done false
7920 3029 Acked Write 9 Done Sent Done false
7921 3030 Acked Write 7 Done Sent Done false
7922 3031 Acked Write 9 Done Sent Done false
7923 3032 Acked Write 3 Done Sent Done false
7924 3033 Acked Flush 0 Done Sent Done false
7925 3034 Acked Write 9 Done Sent Done false
7926 3035 Acked Read 9 Done Sent Done false
7927 3036 Acked Write 1 Done Sent Done false
7928 3037 Acked Write 1 Done Sent Done false
7929 3038 Acked Read 8 Done Sent Done false
7930 3039 Acked Write 4 Done Sent Done false
7931 3040 Acked Read 5 Done Sent Done false
7932 3041 Acked Read 3 Done Sent Done false
7933 3042 Acked Flush 0 Done Sent Done false
7934 3043 Acked Read 6 Done Sent Done false
7935 3044 Acked Flush 0 Done Sent Done false
7936 3045 Acked Read 2 Done Sent Done false
7937 3046 Acked Read 7 Done Sent Done false
7938 3047 Acked Write 3 Done Sent Done false
7939 3048 Acked Read 8 Done Sent Done false
7940 3049 Acked Read 6 Done Sent Done false
7941 3050 Acked Read 10 Done Sent Done false
7942 3051 Acked Read 3 Done Sent Done false
7943 3052 Acked Write 10 Done Sent Done false
7944 3053 Acked Read 7 Done Sent Done false
7945 3054 Acked Read 9 Done Sent Done false
7946 3055 Acked Flush 0 Done Sent Done false
7947 3056 Acked Write 10 Done Sent Done false
7948 3057 Acked Flush 0 Done Sent Done false
7949 3058 Acked Read 2 Done Sent Done false
7950 3059 Acked Read 10 Done Sent Done false
7951 3060 Acked Read 2 Done Sent Done false
7952 3061 Acked Flush 0 Done Sent Done false
7953 3062 Acked Write 4 Done Sent Done false
7954 3063 Acked Read 5 Done Sent Done false
7955 3064 Acked Flush 0 Done Sent Done false
7956 3065 Acked Read 3 Done Sent Done false
7957 3066 Acked Write 8 Done Sent Done false
7958 3067 Acked Write 4 Done Sent Done false
7959 3068 Acked Write 7 Done Sent Done false
7960 3069 Acked Read 8 Done Sent Done false
7961 3070 Acked Read 5 Done Sent Done false
7962 3071 Acked Write 9 Done Sent Done false
7963 3072 Acked Read 1 Done Sent Done false
7964 3073 Acked Read 3 Done Sent Done false
7965 3074 Acked Read 5 Done Sent Done false
7966 3075 Acked Read 10 Done Sent Done false
7967 3076 Acked Read 8 Done Sent Done false
7968 3077 Acked Read 1 Done Sent Done false
7969 3078 Acked Write 8 Done Sent Done false
7970 3079 Acked Read 5 Done Sent Done false
7971 3080 Acked Write 5 Done Sent Done false
7972 3081 Acked Read 2 Done Sent Done false
7973 3082 Acked Read 6 Done Sent Done false
7974 3083 Acked Write 9 Done Sent Done false
7975 3084 Acked Write 7 Done Sent Done false
7976 3085 Acked Read 7 Done Sent Done false
7977 3086 Acked Read 1 Done Sent Done false
7978 3087 Acked Read 3 Done Sent Done false
7979 3088 Acked Read 3 Done Sent Done false
7980 3089 Acked Write 7 Done Sent Done false
7981 3090 Acked Read 8 Done Sent Done false
7982 3091 Acked Write 8 Done Sent Done false
7983 3092 Acked Write 7 Done Sent Done false
7984 3093 Acked Read 10 Done Sent Done false
7985 3094 Acked Write 9 Done Sent Done false
7986 3095 Acked Flush 0 Done Sent Done false
7987 3096 Acked Read 9 Done Sent Done false
7988 3097 Acked Write 2 Done Sent Done false
7989 3098 Acked Write 9 Done Sent Done false
7990 3099 Acked Write 9 Done Sent Done false
7991 3100 Acked Read 4 Done Sent Done false
7992 3101 Acked Read 2 Done Sent Done false
7993 3102 Acked Write 10 Done Sent Done false
7994 3103 Acked Read 8 Done Sent Done false
7995 3104 Acked Write 10 Done Sent Done false
7996 3105 Acked Read 1 Done Sent Done false
7997 3106 Acked Read 1 Done Sent Done false
7998 3107 Acked Write 8 Done Sent Done false
7999 3108 Acked Write 9 Done Sent Done false
8000 3109 Acked Flush 0 Done Sent Done false
8001 3110 Acked Read 1 Done Sent Done false
8002 3111 Acked Write 3 Done Sent Done false
8003 3112 Acked Read 6 Done Sent Done false
8004 STATES DS:0 DS:1 DS:2 TOTAL
8005 Sent 0 301 0 301
8006 Done 301 0 301 602
8007 Skipped 0 0 0 0
8008 Error 0 0 0 0
8009 Last Flush: Some(JobId(3109)) Some(JobId(2811)) Some(JobId(3109))
8010 Downstairs last five completed: 2811 2810 2809 2808 2807
8011 Upstairs last five completed: 3112 3111 3110 3109 3108
8012 CLIENT: Up:0 ds:301 act:2
80132025-01-29T05:12:22.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
80142025-01-29T05:12:23.074ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
80152025-01-29T05:12:24.076ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
8016 ----------------------------------------------------------------
8017 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8018 States: Active Offline Active
8019 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8020 2812 Acked Write 10 Done Sent Done false
8021 2813 Acked Read 9 Done Sent Done false
8022 2814 Acked Write 8 Done Sent Done false
8023 2815 Acked Write 2 Done Sent Done false
8024 2816 Acked Read 8 Done Sent Done false
8025 2817 Acked Read 6 Done Sent Done false
8026 2818 Acked Read 3 Done Sent Done false
8027 2819 Acked Write 1 Done Sent Done false
8028 2820 Acked Write 6 Done Sent Done false
8029 2821 Acked Flush 0 Done Sent Done false
8030 2822 Acked Read 5 Done Sent Done false
8031 2823 Acked Read 8 Done Sent Done false
8032 2824 Acked Write 4 Done Sent Done false
8033 2825 Acked Read 3 Done Sent Done false
8034 2826 Acked Read 4 Done Sent Done false
8035 2827 Acked Read 8 Done Sent Done false
8036 2828 Acked Write 5 Done Sent Done false
8037 2829 Acked Read 7 Done Sent Done false
8038 2830 Acked Write 10 Done Sent Done false
8039 2831 Acked Write 10 Done Sent Done false
8040 2832 Acked Flush 0 Done Sent Done false
8041 2833 Acked Read 4 Done Sent Done false
8042 2834 Acked Flush 0 Done Sent Done false
8043 2835 Acked Flush 0 Done Sent Done false
8044 2836 Acked Read 4 Done Sent Done false
8045 2837 Acked Write 6 Done Sent Done false
8046 2838 Acked Write 1 Done Sent Done false
8047 2839 Acked Write 10 Done Sent Done false
8048 2840 Acked Read 9 Done Sent Done false
8049 2841 Acked Write 6 Done Sent Done false
8050 2842 Acked Write 4 Done Sent Done false
8051 2843 Acked Read 5 Done Sent Done false
8052 2844 Acked Read 4 Done Sent Done false
8053 2845 Acked Write 3 Done Sent Done false
8054 2846 Acked Write 4 Done Sent Done false
8055 2847 Acked Write 10 Done Sent Done false
8056 2848 Acked Write 6 Done Sent Done false
8057 2849 Acked Write 2 Done Sent Done false
8058 2850 Acked Read 5 Done Sent Done false
8059 2851 Acked Read 5 Done Sent Done false
8060 2852 Acked Flush 0 Done Sent Done false
8061 2853 Acked Read 3 Done Sent Done false
8062 2854 Acked Read 10 Done Sent Done false
8063 2855 Acked Read 5 Done Sent Done false
8064 2856 Acked Read 1 Done Sent Done false
8065 2857 Acked Read 4 Done Sent Done false
8066 2858 Acked Read 1 Done Sent Done false
8067 2859 Acked Read 9 Done Sent Done false
8068 2860 Acked Write 5 Done Sent Done false
8069 2861 Acked Read 5 Done Sent Done false
8070 2862 Acked Write 4 Done Sent Done false
8071 2863 Acked Flush 0 Done Sent Done false
8072 2864 Acked Flush 0 Done Sent Done false
8073 2865 Acked Read 1 Done Sent Done false
8074 2866 Acked Read 3 Done Sent Done false
8075 2867 Acked Flush 0 Done Sent Done false
8076 2868 Acked Write 4 Done Sent Done false
8077 2869 Acked Write 1 Done Sent Done false
8078 2870 Acked Read 4 Done Sent Done false
8079 2871 Acked Read 8 Done Sent Done false
8080 2872 Acked Read 7 Done Sent Done false
8081 2873 Acked Read 10 Done Sent Done false
8082 2874 Acked Read 4 Done Sent Done false
8083 2875 Acked Write 9 Done Sent Done false
8084 2876 Acked Read 6 Done Sent Done false
8085 2877 Acked Write 10 Done Sent Done false
8086 2878 Acked Flush 0 Done Sent Done false
8087 2879 Acked Write 5 Done Sent Done false
8088 2880 Acked Read 5 Done Sent Done false
8089 2881 Acked Read 6 Done Sent Done false
8090 2882 Acked Write 3 Done Sent Done false
8091 2883 Acked Read 10 Done Sent Done false
8092 2884 Acked Read 7 Done Sent Done false
8093 2885 Acked Read 1 Done Sent Done false
8094 2886 Acked Write 6 Done Sent Done false
8095 2887 Acked Write 5 Done Sent Done false
8096 2888 Acked Read 9 Done Sent Done false
8097 2889 Acked Write 4 Done Sent Done false
8098 2890 Acked Flush 0 Done Sent Done false
8099 2891 Acked Write 8 Done Sent Done false
8100 2892 Acked Read 8 Done Sent Done false
8101 2893 Acked Read 8 Done Sent Done false
8102 2894 Acked Flush 0 Done Sent Done false
8103 2895 Acked Read 6 Done Sent Done false
8104 2896 Acked Write 2 Done Sent Done false
8105 2897 Acked Read 8 Done Sent Done false
8106 2898 Acked Write 3 Done Sent Done false
8107 2899 Acked Write 8 Done Sent Done false
8108 2900 Acked Write 9 Done Sent Done false
8109 2901 Acked Write 10 Done Sent Done false
8110 2902 Acked Write 10 Done Sent Done false
8111 2903 Acked Write 5 Done Sent Done false
8112 2904 Acked Write 3 Done Sent Done false
8113 2905 Acked Read 8 Done Sent Done false
8114 2906 Acked Write 6 Done Sent Done false
8115 2907 Acked Read 3 Done Sent Done false
8116 2908 Acked Read 1 Done Sent Done false
8117 2909 Acked Write 7 Done Sent Done false
8118 2910 Acked Write 3 Done Sent Done false
8119 2911 Acked Write 2 Done Sent Done false
8120 2912 Acked Read 10 Done Sent Done false
8121 2913 Acked Flush 0 Done Sent Done false
8122 2914 Acked Write 8 Done Sent Done false
8123 2915 Acked Read 2 Done Sent Done false
8124 2916 Acked Write 6 Done Sent Done false
8125 2917 Acked Read 1 Done Sent Done false
8126 2918 Acked Read 3 Done Sent Done false
8127 2919 Acked Write 10 Done Sent Done false
8128 2920 Acked Read 2 Done Sent Done false
8129 2921 Acked Write 5 Done Sent Done false
8130 2922 Acked Write 9 Done Sent Done false
8131 2923 Acked Read 4 Done Sent Done false
8132 2924 Acked Read 2 Done Sent Done false
8133 2925 Acked Write 3 Done Sent Done false
8134 2926 Acked Write 2 Done Sent Done false
8135 2927 Acked Read 2 Done Sent Done false
8136 2928 Acked Read 7 Done Sent Done false
8137 2929 Acked Write 1 Done Sent Done false
8138 2930 Acked Read 1 Done Sent Done false
8139 2931 Acked Read 10 Done Sent Done false
8140 2932 Acked Flush 0 Done Sent Done false
8141 2933 Acked Write 1 Done Sent Done false
8142 2934 Acked Write 9 Done Sent Done false
8143 2935 Acked Write 2 Done Sent Done false
8144 2936 Acked Write 6 Done Sent Done false
8145 2937 Acked Read 6 Done Sent Done false
8146 2938 Acked Write 3 Done Sent Done false
8147 2939 Acked Write 1 Done Sent Done false
8148 2940 Acked Read 9 Done Sent Done false
8149 2941 Acked Flush 0 Done Sent Done false
8150 2942 Acked Read 2 Done Sent Done false
8151 2943 Acked Read 8 Done Sent Done false
8152 2944 Acked Write 8 Done Sent Done false
8153 2945 Acked Read 3 Done Sent Done false
8154 2946 Acked Write 6 Done Sent Done false
8155 2947 Acked Read 9 Done Sent Done false
8156 2948 Acked Read 3 Done Sent Done false
8157 2949 Acked Read 4 Done Sent Done false
8158 2950 Acked Read 5 Done Sent Done false
8159 2951 Acked Read 2 Done Sent Done false
8160 2952 Acked Read 2 Done Sent Done false
8161 2953 Acked Read 9 Done Sent Done false
8162 2954 Acked Read 5 Done Sent Done false
8163 2955 Acked Write 8 Done Sent Done false
8164 2956 Acked Flush 0 Done Sent Done false
8165 2957 Acked Read 3 Done Sent Done false
8166 2958 Acked Write 9 Done Sent Done false
8167 2959 Acked Write 8 Done Sent Done false
8168 2960 Acked Read 10 Done Sent Done false
8169 2961 Acked Write 2 Done Sent Done false
8170 2962 Acked Write 5 Done Sent Done false
8171 2963 Acked Write 7 Done Sent Done false
8172 2964 Acked Read 6 Done Sent Done false
8173 2965 Acked Flush 0 Done Sent Done false
8174 2966 Acked Read 5 Done Sent Done false
8175 2967 Acked Write 2 Done Sent Done false
8176 2968 Acked Read 7 Done Sent Done false
8177 2969 Acked Write 7 Done Sent Done false
8178 2970 Acked Read 7 Done Sent Done false
8179 2971 Acked Write 3 Done Sent Done false
8180 2972 Acked Write 9 Done Sent Done false
8181 2973 Acked Read 4 Done Sent Done false
8182 2974 Acked Write 6 Done Sent Done false
8183 2975 Acked Read 2 Done Sent Done false
8184 2976 Acked Read 2 Done Sent Done false
8185 2977 Acked Write 10 Done Sent Done false
8186 2978 Acked Write 10 Done Sent Done false
8187 2979 Acked Write 6 Done Sent Done false
8188 2980 Acked Write 7 Done Sent Done false
8189 2981 Acked Read 8 Done Sent Done false
8190 2982 Acked Read 4 Done Sent Done false
8191 2983 Acked Read 7 Done Sent Done false
8192 2984 Acked Flush 0 Done Sent Done false
8193 2985 Acked Write 5 Done Sent Done false
8194 2986 Acked Write 6 Done Sent Done false
8195 2987 Acked Read 8 Done Sent Done false
8196 2988 Acked Write 2 Done Sent Done false
8197 2989 Acked Read 9 Done Sent Done false
8198 2990 Acked Write 4 Done Sent Done false
8199 2991 Acked Read 6 Done Sent Done false
8200 2992 Acked Read 8 Done Sent Done false
8201 2993 Acked Write 6 Done Sent Done false
8202 2994 Acked Write 1 Done Sent Done false
8203 2995 Acked Read 3 Done Sent Done false
8204 2996 Acked Read 3 Done Sent Done false
8205 2997 Acked Write 4 Done Sent Done false
8206 2998 Acked Read 6 Done Sent Done false
8207 2999 Acked Write 2 Done Sent Done false
8208 3000 Acked Write 8 Done Sent Done false
8209 3001 Acked Write 4 Done Sent Done false
8210 3002 Acked Read 10 Done Sent Done false
8211 3003 Acked Read 4 Done Sent Done false
8212 3004 Acked Write 2 Done Sent Done false
8213 3005 Acked Write 2 Done Sent Done false
8214 3006 Acked Write 1 Done Sent Done false
8215 3007 Acked Write 9 Done Sent Done false
8216 3008 Acked Write 9 Done Sent Done false
8217 3009 Acked Read 6 Done Sent Done false
8218 3010 Acked Read 6 Done Sent Done false
8219 3011 Acked Flush 0 Done Sent Done false
8220 3012 Acked Write 8 Done Sent Done false
8221 3013 Acked Read 5 Done Sent Done false
8222 3014 Acked Read 2 Done Sent Done false
8223 3015 Acked Read 5 Done Sent Done false
8224 3016 Acked Read 8 Done Sent Done false
8225 3017 Acked Write 1 Done Sent Done false
8226 3018 Acked Read 4 Done Sent Done false
8227 3019 Acked Write 5 Done Sent Done false
8228 3020 Acked Read 7 Done Sent Done false
8229 3021 Acked Write 10 Done Sent Done false
8230 3022 Acked Read 6 Done Sent Done false
8231 3023 Acked Read 3 Done Sent Done false
8232 3024 Acked Write 10 Done Sent Done false
8233 3025 Acked Read 3 Done Sent Done false
8234 3026 Acked Read 7 Done Sent Done false
8235 3027 Acked Read 1 Done Sent Done false
8236 3028 Acked Write 3 Done Sent Done false
8237 3029 Acked Write 9 Done Sent Done false
8238 3030 Acked Write 7 Done Sent Done false
8239 3031 Acked Write 9 Done Sent Done false
8240 3032 Acked Write 3 Done Sent Done false
8241 3033 Acked Flush 0 Done Sent Done false
8242 3034 Acked Write 9 Done Sent Done false
8243 3035 Acked Read 9 Done Sent Done false
8244 3036 Acked Write 1 Done Sent Done false
8245 3037 Acked Write 1 Done Sent Done false
8246 3038 Acked Read 8 Done Sent Done false
8247 3039 Acked Write 4 Done Sent Done false
8248 3040 Acked Read 5 Done Sent Done false
8249 3041 Acked Read 3 Done Sent Done false
8250 3042 Acked Flush 0 Done Sent Done false
8251 3043 Acked Read 6 Done Sent Done false
8252 3044 Acked Flush 0 Done Sent Done false
8253 3045 Acked Read 2 Done Sent Done false
8254 3046 Acked Read 7 Done Sent Done false
8255 3047 Acked Write 3 Done Sent Done false
8256 3048 Acked Read 8 Done Sent Done false
8257 3049 Acked Read 6 Done Sent Done false
8258 3050 Acked Read 10 Done Sent Done false
8259 3051 Acked Read 3 Done Sent Done false
8260 3052 Acked Write 10 Done Sent Done false
8261 3053 Acked Read 7 Done Sent Done false
8262 3054 Acked Read 9 Done Sent Done false
8263 3055 Acked Flush 0 Done Sent Done false
8264 3056 Acked Write 10 Done Sent Done false
8265 3057 Acked Flush 0 Done Sent Done false
8266 3058 Acked Read 2 Done Sent Done false
8267 3059 Acked Read 10 Done Sent Done false
8268 3060 Acked Read 2 Done Sent Done false
8269 3061 Acked Flush 0 Done Sent Done false
8270 3062 Acked Write 4 Done Sent Done false
8271 3063 Acked Read 5 Done Sent Done false
8272 3064 Acked Flush 0 Done Sent Done false
8273 3065 Acked Read 3 Done Sent Done false
8274 3066 Acked Write 8 Done Sent Done false
8275 3067 Acked Write 4 Done Sent Done false
8276 3068 Acked Write 7 Done Sent Done false
8277 3069 Acked Read 8 Done Sent Done false
8278 3070 Acked Read 5 Done Sent Done false
8279 3071 Acked Write 9 Done Sent Done false
8280 3072 Acked Read 1 Done Sent Done false
8281 3073 Acked Read 3 Done Sent Done false
8282 3074 Acked Read 5 Done Sent Done false
8283 3075 Acked Read 10 Done Sent Done false
8284 3076 Acked Read 8 Done Sent Done false
8285 3077 Acked Read 1 Done Sent Done false
8286 3078 Acked Write 8 Done Sent Done false
8287 3079 Acked Read 5 Done Sent Done false
8288 3080 Acked Write 5 Done Sent Done false
8289 3081 Acked Read 2 Done Sent Done false
8290 3082 Acked Read 6 Done Sent Done false
8291 3083 Acked Write 9 Done Sent Done false
8292 3084 Acked Write 7 Done Sent Done false
8293 3085 Acked Read 7 Done Sent Done false
8294 3086 Acked Read 1 Done Sent Done false
8295 3087 Acked Read 3 Done Sent Done false
8296 3088 Acked Read 3 Done Sent Done false
8297 3089 Acked Write 7 Done Sent Done false
8298 3090 Acked Read 8 Done Sent Done false
8299 3091 Acked Write 8 Done Sent Done false
8300 3092 Acked Write 7 Done Sent Done false
8301 3093 Acked Read 10 Done Sent Done false
8302 3094 Acked Write 9 Done Sent Done false
8303 3095 Acked Flush 0 Done Sent Done false
8304 3096 Acked Read 9 Done Sent Done false
8305 3097 Acked Write 2 Done Sent Done false
8306 3098 Acked Write 9 Done Sent Done false
8307 3099 Acked Write 9 Done Sent Done false
8308 3100 Acked Read 4 Done Sent Done false
8309 3101 Acked Read 2 Done Sent Done false
8310 3102 Acked Write 10 Done Sent Done false
8311 3103 Acked Read 8 Done Sent Done false
8312 3104 Acked Write 10 Done Sent Done false
8313 3105 Acked Read 1 Done Sent Done false
8314 3106 Acked Read 1 Done Sent Done false
8315 3107 Acked Write 8 Done Sent Done false
8316 3108 Acked Write 9 Done Sent Done false
8317 3109 Acked Flush 0 Done Sent Done false
8318 3110 Acked Read 1 Done Sent Done false
8319 3111 Acked Write 3 Done Sent Done false
8320 3112 Acked Read 6 Done Sent Done false
8321 STATES DS:0 DS:1 DS:2 TOTAL
8322 Sent 0 301 0 301
8323 Done 301 0 301 602
8324 Skipped 0 0 0 0
8325 Error 0 0 0 0
8326 Last Flush: Some(JobId(3109)) Some(JobId(2811)) Some(JobId(3109))
8327 Downstairs last five completed: 2811 2810 2809 2808 2807
8328 Upstairs last five completed: 3112 3111 3110 3109 3108
8329 CLIENT: Up:0 ds:301 act:2
83302025-01-29T05:12:26.080ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83312025-01-29T05:12:27.082ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83322025-01-29T05:12:28.084ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83332025-01-29T05:12:28.958ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83342025-01-29T05:12:28.958ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39908) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83352025-01-29T05:12:28.959ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83362025-01-29T05:12:28.960ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83372025-01-29T05:12:28.960ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83382025-01-29T05:12:28.960ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83392025-01-29T05:12:28.960ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83402025-01-29T05:12:28.960ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83412025-01-29T05:12:28.960ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
8342 ----------------------------------------------------------------
8343 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
8344 States: Active Active Active
8345 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8346 3110 Acked Read 1 Done Done Done true
8347 3111 Acked Write 3 Done Done Done true
8348 3112 Acked Read 6 Done Done Done true
8349 STATES DS:0 DS:1 DS:2 TOTAL
8350 Sent 0 0 0 0
8351 Done 3 3 3 9
8352 Skipped 0 0 0 0
8353 Error 0 0 0 0
8354 Last Flush: Some(JobId(3109)) Some(JobId(3109)) Some(JobId(3109))
8355 Downstairs last five completed: 3109 3108 3107 3106 3105
8356 Upstairs last five completed: 3112 3111 3110 3109 3108
8357 CLIENT: Up:0 ds:3 act:3
8358 ----------------------------------------------------------------
8359 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8360 Downstairs last five completed: 3113 3112 3111 3110 3109
8361 Upstairs last five completed: 3113 3112 3111 3110 3109
8362 CLIENT: Up:0 ds:0 act:3
8363 Replay: All jobs finished
8364 Main task has work to do, go find it
8365 got dsc Stop(1)
8366 stop 1
8367 Jan 29 05:12:33.210 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
8368 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b92a744b-fde7-40f7-bc2f-13187691da1f, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
8369 Jan 29 05:12:33.210 INFO request completed, latency_us: 129, response_code: 200, uri: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:12:33.210969639Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
8370 /state/cid/1, method: GET, req_id: 0450dfb4-df0a-40c0-82f4-02a34ce66949, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:12:33.211037043Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
8371 .1:9998
83722025-01-29T05:12:33.211ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
83732025-01-29T05:12:33.211ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
8374 [8820] Exited with: ExitStatus(unix_wait_status(9))
8375 [8820][1] reports Exit
8376 Jan 29 05:12:37.212 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/1, method: GET, req_id: 001b6806-3e84-4bb5-b68c-f5fd2eca8f37, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
8377 001/300 Write block 1856 len 32768 data: 2 2 2 3 3 3 3 3
8378 002/300 Read block 2304 len 12288
8379 003/300 Write block 3619 len 8192 data: 2 2
8380 004/300 Read block 4213 len 16384
8381 005/300 Write block 3077 len 20480 data: 4 3 3 4 3
8382 006/300 Read block 2814 len 36864
8383 007/300 Read block 4034 len 24576
8384 008/300 Flush
8385 009/300 Read block 3706 len 28672
8386 010/300 Read block 2434 len 4096
8387 011/300 Read block 1712 len 4096
8388 012/300 Read block 4400 len 24576
8389 013/300 Read block 794 len 24576
8390 014/300 Read block 1296 len 12288
8391 015/300 Write block 3726 len 16384 data: 2 2 3 3
8392 016/300 Read block 3521 len 16384
8393 017/300 Read block 4953 len 24576
8394 018/300 Flush
8395 019/300 Read block 485 len 4096
8396 020/300 Write block 2497 len 36864 data: 2 3 3 4 4 4 4 3 3
8397 021/300 Flush
8398 022/300 Write block 2214 len 32768 data: 3 3 2 2 2 3 3 3
8399 023/300 Read block 1946 len 24576
8400 024/300 Write block 4599 len 12288 data: 3 4 4
8401 025/300 Flush
8402 026/300 Read block 1174 len 12288
8403 027/300 Write block 4078 len 4096 data: 2
8404 028/300 Read block 2694 len 28672
8405 029/300 Read block 4603 len 4096
8406 030/300 Write block 1687 len 20480 data: 3 3 3 5 5
8407 031/300 Read block 3575 len 24576
8408 032/300 Write block 3439 len 32768 data: 4 3 3 3 3 3 3 3
8409 033/300 Write block 77 len 20480 data: 4 3 3 3 3
8410 034/300 Read block 4934 len 20480
8411 035/300 Write block 307 len 4096 data: 2
8412 036/300 Write block 4400 len 40960 data: 2 3 3 3 3 2 2 2 2 2
8413 037/300 Write block 481 len 28672 data: 5 4 4 3 3 3 3
8414 038/300 Write block 3993 len 32768 data: 4 4 4 3 3 3 3 3
8415 039/300 Read block 2016 len 36864
8416 040/300 Read block 1578 len 24576
8417 041/300 Read block 1184 len 16384
8418 042/300 Write block 4350 len 4096 data: 3
8419 043/300 Write block 871 len 24576 data: 3 3 3 3 2 2
8420 044/300 Read block 1038 len 28672
8421 045/300 Write block 4779 len 16384 data: 4 4 3 3
8422 046/300 Read block 272 len 8192
8423 047/300 Write block 1720 len 16384 data: 2 2 2 2
8424 048/300 Read block 2518 len 4096
8425 049/300 Write block 4316 len 8192 data: 4 3
8426 050/300 Write block 556 len 20480 data: 3 2 2 2 2
8427 051/300 Read block 1659 len 16384
8428 052/300 Write block 2912 len 40960 data: 3 3 3 3 2 2 2 2 2 2
8429 053/300 Write block 4610 len 24576 data: 3 3 4 4 3 3
8430 054/300 Flush
8431 055/300 Read block 2290 len 40960
8432 056/300 Write block 4496 len 28672 data: 2 2 3 3 4 3 4
8433 057/300 Write block 1256 len 20480 data: 3 3 3 2 2
8434 058/300 Write block 491 len 40960 data: 3 3 3 3 2 2 2 2 2 2
8435 059/300 Write block 515 len 28672 data: 5 4 4 3 3 3 3
8436 060/300 Read block 1690 len 36864
8437 061/300 Read block 1475 len 36864
8438 062/300 Write block 3936 len 24576 data: 2 2 2 2 2 2
8439 063/300 Write block 3379 len 24576 data: 3 2 2 3 2 2
8440 064/300 Read block 1253 len 32768
8441 065/300 Read block 2588 len 4096
8442 066/300 Write block 242 len 40960 data: 5 4 4 4 3 3 2 2 2 3
8443 067/300 Write block 1679 len 32768 data: 3 4 4 5 4 4 4 2
8444 068/300 Write block 2983 len 20480 data: 3 3 2 2 2
8445 069/300 Flush
8446 070/300 Read block 3358 len 8192
8447 071/300 Flush
8448 072/300 Write block 4485 len 8192 data: 2 2
8449 073/300 Read block 4964 len 40960
8450 074/300 Flush
8451 075/300 Read block 3697 len 4096
8452 076/300 Write block 4692 len 20480 data: 2 3 3 3 3
8453 077/300 Write block 945 len 40960 data: 2 3 3 3 3 3 3 4 4 4
8454 078/300 Write block 2113 len 20480 data: 4 3 2 2 2
8455 079/300 Write block 2072 len 4096 data: 2
8456 080/300 Read block 123 len 24576
8457 081/300 Read block 3190 len 36864
8458 082/300 Write block 2815 len 32768 data: 4 3 2 2 2 2 2 3
8459 083/300 Write block 1176 len 4096 data: 3
8460 084/300 Read block 613 len 32768
8461 085/300 Write block 4659 len 20480 data: 2 2 3 3 3
8462 086/300 Read block 4255 len 32768
8463 087/300 Write block 2333 len 28672 data: 3 3 3 3 4 4 3
8464 088/300 Read block 4761 len 32768
8465 089/300 Read block 4609 len 4096
8466 090/300 Write block 1087 len 4096 data: 2
8467 091/300 Write block 1117 len 12288 data: 2 2 2
8468 092/300 Read block 677 len 28672
8469 093/300 Read block 2701 len 20480
8470 094/300 Read block 1678 len 40960
8471 095/300 Write block 4329 len 12288 data: 4 4 4
8472 096/300 Read block 1381 len 12288
8473 097/300 Write block 2747 len 40960 data: 3 4 4 4 5 5 5 4 5 5
8474 098/300 Read block 3547 len 28672
8475 099/300 Read block 735 len 32768
8476 100/300 Write block 3173 len 40960 data: 4 3 3 3 3 3 2 2 2 2
8477 101/300 Read block 4914 len 36864
8478 102/300 Flush
8479 103/300 Write block 3752 len 8192 data: 4 3
8480 104/300 Write block 490 len 36864 data: 2 4 4 4 4 3 3 3 3
8481 105/300 Write block 2733 len 40960 data: 3 4 4 5 5 5 5 3 3 4
8482 106/300 Flush
8483 107/300 Read block 2826 len 28672
8484 108/300 Read block 848 len 4096
8485 109/300 Write block 3236 len 32768 data: 4 4 2 2 2 2 2 2
8486 110/300 Read block 656 len 36864
8487 111/300 Write block 2455 len 24576 data: 3 3 3 2 2 2
8488 112/300 Write block 1687 len 20480 data: 4 4 4 6 6
8489 113/300 Write block 2716 len 16384 data: 2 3 3 2
8490 114/300 Read block 694 len 32768
8491 115/300 Read block 3956 len 8192
8492 116/300 Read block 830 len 8192
8493 117/300 Read block 493 len 28672
8494 118/300 Read block 1281 len 32768
8495 119/300 Read block 586 len 32768
8496 120/300 Read block 521 len 24576
8497 121/300 Read block 270 len 24576
8498 122/300 Write block 4992 len 16384 data: 5 4 4 3
8499 123/300 Read block 1724 len 40960
8500 124/300 Write block 4602 len 16384 data: 4 5 5 3
8501 125/300 Write block 834 len 28672 data: 2 2 2 2 2 2 2
8502 126/300 Read block 253 len 16384
8503 127/300 Flush
8504 128/300 Write block 3367 len 40960 data: 2 3 3 4 4 3 4 3 3 3
8505 129/300 Read block 2756 len 16384
8506 130/300 Flush
8507 131/300 Write block 3494 len 20480 data: 2 2 2 2 3
8508 132/300 Read block 3441 len 40960
8509 133/300 Read block 284 len 8192
8510 134/300 Read block 1778 len 4096
8511 135/300 Flush
8512 136/300 Flush
8513 137/300 Write block 4098 len 36864 data: 3 3 4 5 6 6 5 3 3
8514 138/300 Read block 3589 len 8192
8515 139/300 Write block 3849 len 16384 data: 3 3 3 3
8516 140/300 Write block 235 len 4096 data: 4
8517 141/300 Write block 3574 len 36864 data: 2 3 4 4 5 4 4 4 3
8518 142/300 Read block 142 len 28672
8519 143/300 Flush
8520 144/300 Write block 3841 len 28672 data: 2 2 2 2 2 2 2
8521 145/300 Write block 3972 len 12288 data: 4 4 5
8522 146/300 Read block 4368 len 28672
8523 147/300 Read block 684 len 12288
8524 148/300 Read block 4063 len 28672
8525 149/300 Write block 2837 len 28672 data: 3 3 3 3 2 2 2
8526 150/300 Read block 2151 len 28672
8527 151/300 Read block 2850 len 40960
8528 152/300 Flush
8529 153/300 Read block 4007 len 8192
8530 154/300 Write block 2976 len 40960 data: 5 4 4 4 3 3 3 4 4 3
8531 155/300 Read block 2272 len 24576
8532 156/300 Read block 3483 len 12288
8533 157/300 Read block 4809 len 24576
8534 158/300 Flush
8535 159/300 Read block 4984 len 32768
8536 160/300 Write block 968 len 20480 data: 5 5 5 4 4
8537 161/300 Write block 395 len 12288 data: 3 3 3
8538 162/300 Read block 4254 len 40960
8539 163/300 Read block 2263 len 8192
8540 164/300 Flush
8541 165/300 Write block 2975 len 16384 data: 3 6 5 5
8542 166/300 Read block 3632 len 4096
8543 167/300 Write block 4000 len 20480 data: 4 3 3 2 3
8544 168/300 Read block 2295 len 36864
8545 169/300 Write block 2434 len 20480 data: 3 3 3 3 5
8546 170/300 Read block 3339 len 28672
8547 171/300 Read block 2059 len 24576
8548 172/300 Write block 2547 len 36864 data: 2 2 2 2 2 2 3 3 3
8549 173/300 Write block 4134 len 36864 data: 2 2 3 3 4 4 5 5 3
8550 174/300 Read block 253 len 32768
8551 175/300 Flush
8552 176/300 Flush
8553 177/300 Write block 4408 len 28672 data: 3 3 3 4 4 4 3
8554 178/300 Write block 3732 len 8192 data: 4 4
8555 179/300 Write block 1049 len 16384 data: 3 3 2 2
8556 180/300 Read block 632 len 4096
8557 181/300 Write block 3570 len 32768 data: 3 3 2 2 3 4 5 5
8558 182/300 Write block 29 len 40960 data: 2 2 2 2 2 2 2 2 2 3
8559 183/300 Read block 3915 len 4096
8560 184/300 Write block 3589 len 12288 data: 2 3 3
8561 185/300 Write block 2818 len 40960 data: 3 3 3 3 4 3 3 3 3 3
8562 186/300 Flush
8563 187/300 Read block 4902 len 24576
8564 188/300 Write block 137 len 36864 data: 4 3 3 3 3 3 2 4 3
8565 189/300 Read block 468 len 36864
8566 190/300 Read block 2683 len 20480
8567 191/300 Write block 2468 len 8192 data: 3 4
8568 192/300 Read block 3523 len 12288
8569 193/300 Write block 960 len 16384 data: 2 3 3 3
8570 194/300 Read block 4921 len 32768
8571 195/300 Flush
8572 196/300 Read block 2926 len 20480
8573 197/300 Write block 2486 len 16384 data: 3 2 2 3
8574 198/300 Read block 3743 len 12288
8575 199/300 Read block 2528 len 4096
8576 200/300 Read block 1333 len 24576
8577 201/300 Read block 20 len 16384
8578 202/300 Flush
8579 203/300 Read block 2517 len 36864
8580 204/300 Read block 1288 len 20480
8581 205/300 Write block 1594 len 28672 data: 5 3 2 2 2 2 2
8582 206/300 Write block 4777 len 32768 data: 4 4 5 5 4 4 3 2
8583 207/300 Read block 1071 len 16384
8584 208/300 Read block 500 len 40960
8585 209/300 Read block 1560 len 32768
8586 210/300 Read block 4421 len 24576
8587 211/300 Read block 2779 len 36864
8588 212/300 Read block 3925 len 16384
8589 213/300 Read block 3979 len 24576
8590 214/300 Flush
8591 215/300 Read block 682 len 24576
8592 216/300 Write block 1863 len 24576 data: 4 3 3 3 3 3
8593 217/300 Write block 3879 len 24576 data: 2 2 2 2 2 2
8594 218/300 Read block 1515 len 8192
8595 219/300 Flush
8596 220/300 Read block 4877 len 32768
8597 221/300 Write block 787 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8598 222/300 Write block 4205 len 20480 data: 4 3 3 3 3
8599 223/300 Read block 841 len 12288
8600 224/300 Flush
8601 225/300 Flush
8602 226/300 Read block 4123 len 8192
8603 227/300 Write block 3796 len 12288 data: 2 2 3
8604 228/300 Flush
8605 229/300 Write block 1593 len 32768 data: 5 6 4 3 3 3 3 3
8606 230/300 Write block 3384 len 8192 data: 3 2
8607 231/300 Write block 4391 len 20480 data: 3 4 4 4 5
8608 232/300 Read block 1018 len 40960
8609 233/300 Flush
8610 234/300 Write block 2109 len 32768 data: 4 4 5 5 5 4 3 3
8611 235/300 Write block 237 len 16384 data: 6 6 6 5
8612 236/300 Write block 3467 len 20480 data: 3 3 3 4 3
8613 237/300 Read block 2621 len 8192
8614 238/300 Flush
8615 239/300 Write block 502 len 32768 data: 5 5 5 5 5 7 5 4
8616 240/300 Read block 3244 len 24576
8617 241/300 Flush
8618 242/300 Read block 745 len 36864
8619 243/300 Read block 2396 len 20480
8620 244/300 Write block 880 len 4096 data: 4
8621 245/300 Flush
8622 246/300 Read block 1040 len 32768
8623 247/300 Write block 1218 len 36864 data: 3 3 3 2 2 2 3 3 2
8624 248/300 Read block 3109 len 36864
8625 249/300 Read block 4982 len 40960
8626 250/300 Read block 1930 len 28672
8627 251/300 Read block 2415 len 20480
8628 252/300 Read block 4867 len 28672
8629 253/300 Read block 3336 len 16384
8630 254/300 Write block 3525 len 8192 data: 2 2
8631 255/300 Write block 4387 len 40960 data: 5 3 2 2 4 5 5 5 6 4
8632 256/300 Write block 4770 len 28672 data: 3 2 2 3 3 3 4
8633 257/300 Read block 4478 len 20480
8634 258/300 Write block 2669 len 36864 data: 2 2 2 3 4 4 3 3 2
8635 259/300 Write block 347 len 8192 data: 2 2
8636 260/300 Write block 4510 len 12288 data: 2 2 2
8637 261/300 Write block 661 len 28672 data: 4 4 4 4 3 3 3
8638 262/300 Read block 3306 len 28672
8639 263/300 Write block 4398 len 40960 data: 3 3 3 4 4 4 4 3 3 3
8640 264/300 Write block 4975 len 40960 data: 4 3 3 3 3 3 3 2 2 3
8641 265/300 Read block 801 len 4096
8642 266/300 Read block 51 len 28672
8643 267/300 Write block 3117 len 36864 data: 3 5 5 4 3 4 4 3 2
8644 268/300 Read block 4778 len 40960
8645 269/300 Read block 635 len 8192
8646 270/300 Read block 1635 len 20480
8647 271/300 Read block 613 len 24576
8648 272/300 Write block 3228 len 4096 data: 4
8649 273/300 Write block 3459 len 20480 data: 2 3 3 3 3
8650 274/300 Write block 4663 len 40960 data: 4 3 3 3 2 2 2 2 2 2
8651 275/300 Write block 550 len 12288 data: 3 2 2
8652 276/300 Read block 4133 len 24576
8653 277/300 Write block 1674 len 8192 data: 4 4
8654 278/300 Write block 4636 len 36864 data: 3 3 4 3 3 3 3 4 4
8655 279/300 Write block 3402 len 32768 data: 3 3 3 3 3 3 3 3
8656 280/300 Read block 375 len 20480
8657 281/300 Read block 3869 len 36864
8658 282/300 Read block 3090 len 32768
8659 283/300 Read block 325 len 40960
8660 284/300 Write block 861 len 16384 data: 3 3 2 2
8661 285/300 Read block 4859 len 36864
8662 286/300 Write block 1936 len 24576 data: 3 3 3 3 3 2
8663 287/300 Write block 1281 len 32768 data: 2 2 3 3 3 3 3 3
8664 288/300 Write block 1624 len 36864 data: 4 4 4 4 4 3 3 3 4
8665 289/300 Read block 3530 len 36864
8666 290/300 Write block 4549 len 16384 data: 5 4 3 4
8667 291/300 Read block 4137 len 40960
8668 292/300 Write block 169 len 28672 data: 4 5 5 4 4 3 3
8669 293/300 Read block 248 len 40960
8670 294/300 Read block 3370 len 20480
8671 295/300 Read block 4417 len 8192
8672 296/300 Flush
8673 297/300 Write block 1840 len 16384 data: 3 3 3 3
8674 298/300 Write block 2199 len 12288 data: 3 2 2
8675 299/300 Read block 1611 len 12288
8676 300/300 Read block 1576 len 36864
8677 301/300 Write block 104 len 4096 data: 3
8678 Main task has work to do, go find it
8679 got dsc Start(1)
8680 start 1
8681 Jan 29 05:12:37.455 INFO request completed, latency_us: 186, response_code: 204, uri: /start/cid/1, method: POST, req_id: f12bf2db-9ce7-4016-8f55-309cdb0c5e78, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
8682 Replay: started 1, returned:Ok(())
8683 ----------------------------------------------------------------
8684 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8685 States: Active Offline Active
8686 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8687 3114 Acked Write 8 Done Sent Done false
8688 3115 Acked Read 3 Done Sent Done false
8689 3116 Acked Write 2 Done Sent Done false
8690 3117 Acked Read 4 Done Sent Done false
8691 3118 Acked Write 5 Done Sent Done false
8692 3119 Acked Read 9 Done Sent Done false
8693 3120 Acked Read 6 Done Sent Done false
8694 3121 Acked Flush 0 Done Sent Done false
8695 3122 Acked Read 7 Done Sent Done false
8696 3123 Acked Read 1 Done Sent Done false
8697 3124 Acked Read 1 Done Sent Done false
8698 3125 Acked Read 6 Done Sent Done false
8699 3126 Acked Read 6 Done Sent Done false
8700 3127 Acked Read 3 Done Sent Done false
8701 3128 Acked Write 4 Done Sent Done false
8702 3129 Acked Read 4 Done Sent Done false
8703 3130 Acked Read 6 Done Sent Done false
8704 3131 Acked Flush 0 Done Sent Done false
8705 3132 Acked Read 1 Done Sent Done false
8706 3133 Acked Write 9 Done Sent Done false
8707 3134 Acked Flush 0 Done Sent Done false
8708 3135 Acked Write 8 Done Sent Done false
8709 3136 Acked Read 6 Done Sent Done false
8710 3137 Acked Write 3 Done Sent Done false
8711 3138 Acked Flush 0 Done Sent Done false
8712 3139 Acked Read 3 Done Sent Done false
8713 3140 Acked Write 1 Done Sent Done false
8714 3141 Acked Read 7 Done Sent Done false
8715 3142 Acked Read 1 Done Sent Done false
8716 3143 Acked Write 5 Done Sent Done false
8717 3144 Acked Read 6 Done Sent Done false
8718 3145 Acked Write 8 Done Sent Done false
8719 3146 Acked Write 5 Done Sent Done false
8720 3147 Acked Read 5 Done Sent Done false
8721 3148 Acked Write 1 Done Sent Done false
8722 3149 Acked Write 10 Done Sent Done false
8723 3150 Acked Write 7 Done Sent Done false
8724 3151 Acked Write 8 Done Sent Done false
8725 3152 Acked Read 9 Done Sent Done false
8726 3153 Acked Read 6 Done Sent Done false
8727 3154 Acked Read 4 Done Sent Done false
8728 3155 Acked Write 1 Done Sent Done false
8729 3156 Acked Write 6 Done Sent Done false
8730 3157 Acked Read 7 Done Sent Done false
8731 3158 Acked Write 4 Done Sent Done false
8732 3159 Acked Read 2 Done Sent Done false
8733 3160 Acked Write 4 Done Sent Done false
8734 3161 Acked Read 1 Done Sent Done false
8735 3162 Acked Write 2 Done Sent Done false
8736 3163 Acked Write 5 Done Sent Done false
8737 3164 Acked Read 4 Done Sent Done false
8738 3165 Acked Write 10 Done Sent Done false
8739 3166 Acked Write 6 Done Sent Done false
8740 3167 Acked Flush 0 Done Sent Done false
8741 3168 Acked Read 10 Done Sent Done false
8742 3169 Acked Write 7 Done Sent Done false
8743 3170 Acked Write 5 Done Sent Done false
8744 3171 Acked Write 10 Done Sent Done false
8745 3172 Acked Write 7 Done Sent Done false
8746 3173 Acked Read 9 Done Sent Done false
8747 3174 Acked Read 9 Done Sent Done false
8748 3175 Acked Write 6 Done Sent Done false
8749 3176 Acked Write 6 Done Sent Done false
8750 3177 Acked Read 8 Done Sent Done false
8751 3178 Acked Read 1 Done Sent Done false
8752 3179 Acked Write 10 Done Sent Done false
8753 3180 Acked Write 8 Done Sent Done false
8754 3181 Acked Write 5 Done Sent Done false
8755 3182 Acked Flush 0 Done Sent Done false
8756 3183 Acked Read 2 Done Sent Done false
8757 3184 Acked Flush 0 Done Sent Done false
8758 3185 Acked Write 2 Done Sent Done false
8759 3186 Acked Read 10 Done Sent Done false
8760 3187 Acked Flush 0 Done Sent Done false
8761 3188 Acked Read 1 Done Sent Done false
8762 3189 Acked Write 5 Done Sent Done false
8763 3190 Acked Write 10 Done Sent Done false
8764 3191 Acked Write 5 Done Sent Done false
8765 3192 Acked Write 1 Done Sent Done false
8766 3193 Acked Read 6 Done Sent Done false
8767 3194 Acked Read 9 Done Sent Done false
8768 3195 Acked Write 8 Done Sent Done false
8769 3196 Acked Write 1 Done Sent Done false
8770 3197 Acked Read 8 Done Sent Done false
8771 3198 Acked Write 5 Done Sent Done false
8772 3199 Acked Read 8 Done Sent Done false
8773 3200 Acked Write 7 Done Sent Done false
8774 3201 Acked Read 8 Done Sent Done false
8775 3202 Acked Read 1 Done Sent Done false
8776 3203 Acked Write 1 Done Sent Done false
8777 3204 Acked Write 3 Done Sent Done false
8778 3205 Acked Read 7 Done Sent Done false
8779 3206 Acked Read 5 Done Sent Done false
8780 3207 Acked Read 10 Done Sent Done false
8781 3208 Acked Write 3 Done Sent Done false
8782 3209 Acked Read 3 Done Sent Done false
8783 3210 Acked Write 10 Done Sent Done false
8784 3211 Acked Read 7 Done Sent Done false
8785 3212 Acked Read 8 Done Sent Done false
8786 3213 Acked Write 10 Done Sent Done false
8787 3214 Acked Read 9 Done Sent Done false
8788 3215 Acked Flush 0 Done Sent Done false
8789 3216 Acked Write 2 Done Sent Done false
8790 3217 Acked Write 9 Done Sent Done false
8791 3218 Acked Write 10 Done Sent Done false
8792 3219 Acked Flush 0 Done Sent Done false
8793 3220 Acked Read 7 Done Sent Done false
8794 3221 Acked Read 1 Done Sent Done false
8795 3222 Acked Write 8 Done Sent Done false
8796 3223 Acked Read 9 Done Sent Done false
8797 3224 Acked Write 6 Done Sent Done false
8798 3225 Acked Write 5 Done Sent Done false
8799 3226 Acked Write 4 Done Sent Done false
8800 3227 Acked Read 8 Done Sent Done false
8801 3228 Acked Read 2 Done Sent Done false
8802 3229 Acked Read 2 Done Sent Done false
8803 3230 Acked Read 7 Done Sent Done false
8804 3231 Acked Read 8 Done Sent Done false
8805 3232 Acked Read 8 Done Sent Done false
8806 3233 Acked Read 6 Done Sent Done false
8807 3234 Acked Read 6 Done Sent Done false
8808 3235 Acked Write 4 Done Sent Done false
8809 3236 Acked Read 10 Done Sent Done false
8810 3237 Acked Write 4 Done Sent Done false
8811 3238 Acked Write 7 Done Sent Done false
8812 3239 Acked Read 4 Done Sent Done false
8813 3240 Acked Flush 0 Done Sent Done false
8814 3241 Acked Write 10 Done Sent Done false
8815 3242 Acked Read 4 Done Sent Done false
8816 3243 Acked Flush 0 Done Sent Done false
8817 3244 Acked Write 5 Done Sent Done false
8818 3245 Acked Read 10 Done Sent Done false
8819 3246 Acked Read 2 Done Sent Done false
8820 3247 Acked Read 1 Done Sent Done false
8821 3248 Acked Flush 0 Done Sent Done false
8822 3249 Acked Flush 0 Done Sent Done false
8823 3250 Acked Write 9 Done Sent Done false
8824 3251 Acked Read 2 Done Sent Done false
8825 3252 Acked Write 4 Done Sent Done false
8826 3253 Acked Write 1 Done Sent Done false
8827 3254 Acked Write 9 Done Sent Done false
8828 3255 Acked Read 7 Done Sent Done false
8829 3256 Acked Flush 0 Done Sent Done false
8830 3257 Acked Write 7 Done Sent Done false
8831 3258 Acked Write 3 Done Sent Done false
8832 3259 Acked Read 7 Done Sent Done false
8833 3260 Acked Read 3 Done Sent Done false
8834 3261 Acked Read 7 Done Sent Done false
8835 3262 Acked Write 7 Done Sent Done false
8836 3263 Acked Read 7 Done Sent Done false
8837 3264 Acked Read 10 Done Sent Done false
8838 3265 Acked Flush 0 Done Sent Done false
8839 3266 Acked Read 2 Done Sent Done false
8840 3267 Acked Write 10 Done Sent Done false
8841 3268 Acked Read 6 Done Sent Done false
8842 3269 Acked Read 3 Done Sent Done false
8843 3270 Acked Read 6 Done Sent Done false
8844 3271 Acked Flush 0 Done Sent Done false
8845 3272 Acked Read 8 Done Sent Done false
8846 3273 Acked Write 5 Done Sent Done false
8847 3274 Acked Write 3 Done Sent Done false
8848 3275 Acked Read 10 Done Sent Done false
8849 3276 Acked Read 2 Done Sent Done false
8850 3277 Acked Flush 0 Done Sent Done false
8851 3278 Acked Write 4 Done Sent Done false
8852 3279 Acked Read 1 Done Sent Done false
8853 3280 Acked Write 5 Done Sent Done false
8854 3281 Acked Read 9 Done Sent Done false
8855 3282 Acked Write 5 Done Sent Done false
8856 3283 Acked Read 7 Done Sent Done false
8857 3284 Acked Read 6 Done Sent Done false
8858 3285 Acked Write 9 Done Sent Done false
8859 3286 Acked Write 9 Done Sent Done false
8860 3287 Acked Read 8 Done Sent Done false
8861 3288 Acked Flush 0 Done Sent Done false
8862 3289 Acked Flush 0 Done Sent Done false
8863 3290 Acked Write 7 Done Sent Done false
8864 3291 Acked Write 2 Done Sent Done false
8865 3292 Acked Write 4 Done Sent Done false
8866 3293 Acked Read 1 Done Sent Done false
8867 3294 Acked Write 8 Done Sent Done false
8868 3295 Acked Write 10 Done Sent Done false
8869 3296 Acked Read 1 Done Sent Done false
8870 3297 Acked Write 3 Done Sent Done false
8871 3298 Acked Write 10 Done Sent Done false
8872 3299 Acked Flush 0 Done Sent Done false
8873 3300 Acked Read 6 Done Sent Done false
8874 3301 Acked Write 9 Done Sent Done false
8875 3302 Acked Read 9 Done Sent Done false
8876 3303 Acked Read 5 Done Sent Done false
8877 3304 Acked Write 2 Done Sent Done false
8878 3305 Acked Read 3 Done Sent Done false
8879 3306 Acked Write 4 Done Sent Done false
8880 3307 Acked Read 8 Done Sent Done false
8881 3308 Acked Flush 0 Done Sent Done false
8882 3309 Acked Read 5 Done Sent Done false
8883 3310 Acked Write 4 Done Sent Done false
8884 3311 Acked Read 3 Done Sent Done false
8885 3312 Acked Read 1 Done Sent Done false
8886 3313 Acked Read 6 Done Sent Done false
8887 3314 Acked Read 4 Done Sent Done false
8888 3315 Acked Flush 0 Done Sent Done false
8889 3316 Acked Read 9 Done Sent Done false
8890 3317 Acked Read 5 Done Sent Done false
8891 3318 Acked Write 7 Done Sent Done false
8892 3319 Acked Write 8 Done Sent Done false
8893 3320 Acked Read 4 Done Sent Done false
8894 3321 Acked Read 10 Done Sent Done false
8895 3322 Acked Read 8 Done Sent Done false
8896 3323 Acked Read 6 Done Sent Done false
8897 3324 Acked Read 9 Done Sent Done false
8898 3325 Acked Read 4 Done Sent Done false
8899 3326 Acked Read 6 Done Sent Done false
8900 3327 Acked Flush 0 Done Sent Done false
8901 3328 Acked Read 6 Done Sent Done false
8902 3329 Acked Write 6 Done Sent Done false
8903 3330 Acked Write 6 Done Sent Done false
8904 3331 Acked Read 2 Done Sent Done false
8905 3332 Acked Flush 0 Done Sent Done false
8906 3333 Acked Read 8 Done Sent Done false
8907 3334 Acked Write 10 Done Sent Done false
8908 3335 Acked Write 5 Done Sent Done false
8909 3336 Acked Read 3 Done Sent Done false
8910 3337 Acked Flush 0 Done Sent Done false
8911 3338 Acked Flush 0 Done Sent Done false
8912 3339 Acked Read 2 Done Sent Done false
8913 3340 Acked Write 3 Done Sent Done false
8914 3341 Acked Flush 0 Done Sent Done false
8915 3342 Acked Write 8 Done Sent Done false
8916 3343 Acked Write 2 Done Sent Done false
8917 3344 Acked Write 5 Done Sent Done false
8918 3345 Acked Read 10 Done Sent Done false
8919 3346 Acked Flush 0 Done Sent Done false
8920 3347 Acked Write 8 Done Sent Done false
8921 3348 Acked Write 4 Done Sent Done false
8922 3349 Acked Write 5 Done Sent Done false
8923 3350 Acked Read 2 Done Sent Done false
8924 3351 Acked Flush 0 Done Sent Done false
8925 3352 Acked Write 8 Done Sent Done false
8926 3353 Acked Read 6 Done Sent Done false
8927 3354 Acked Flush 0 Done Sent Done false
8928 3355 Acked Read 9 Done Sent Done false
8929 3356 Acked Read 5 Done Sent Done false
8930 3357 Acked Write 1 Done Sent Done false
8931 3358 Acked Flush 0 Done Sent Done false
8932 3359 Acked Read 8 Done Sent Done false
8933 3360 Acked Write 9 Done Sent Done false
8934 3361 Acked Read 9 Done Sent Done false
8935 3362 Acked Read 10 Done Sent Done false
8936 3363 Acked Read 7 Done Sent Done false
8937 3364 Acked Read 5 Done Sent Done false
8938 3365 Acked Read 7 Done Sent Done false
8939 3366 Acked Read 4 Done Sent Done false
8940 3367 Acked Write 2 Done Sent Done false
8941 3368 Acked Write 10 Done Sent Done false
8942 3369 Acked Write 7 Done Sent Done false
8943 3370 Acked Read 5 Done Sent Done false
8944 3371 Acked Write 9 Done Sent Done false
8945 3372 Acked Write 2 Done Sent Done false
8946 3373 Acked Write 3 Done Sent Done false
8947 3374 Acked Write 7 Done Sent Done false
8948 3375 Acked Read 7 Done Sent Done false
8949 3376 Acked Write 10 Done Sent Done false
8950 3377 Acked Write 10 Done Sent Done false
8951 3378 Acked Read 1 Done Sent Done false
8952 3379 Acked Read 7 Done Sent Done false
8953 3380 Acked Write 9 Done Sent Done false
8954 3381 Acked Read 10 Done Sent Done false
8955 3382 Acked Read 2 Done Sent Done false
8956 3383 Acked Read 5 Done Sent Done false
8957 3384 Acked Read 6 Done Sent Done false
8958 3385 Acked Write 1 Done Sent Done false
8959 3386 Acked Write 5 Done Sent Done false
8960 3387 Acked Write 10 Done Sent Done false
8961 3388 Acked Write 3 Done Sent Done false
8962 3389 Acked Read 6 Done Sent Done false
8963 3390 Acked Write 2 Done Sent Done false
8964 3391 Acked Write 9 Done Sent Done false
8965 3392 Acked Write 8 Done Sent Done false
8966 3393 Acked Read 5 Done Sent Done false
8967 3394 Acked Read 9 Done Sent Done false
8968 3395 Acked Read 8 Done Sent Done false
8969 3396 Acked Read 10 Done Sent Done false
8970 3397 Acked Write 4 Done Sent Done false
8971 3398 Acked Read 9 Done Sent Done false
8972 3399 Acked Write 6 Done Sent Done false
8973 3400 Acked Write 8 Done Sent Done false
8974 3401 Acked Write 9 Done Sent Done false
8975 3402 Acked Read 9 Done Sent Done false
8976 3403 Acked Write 4 Done Sent Done false
8977 3404 Acked Read 10 Done Sent Done false
8978 3405 Acked Write 7 Done Sent Done false
8979 3406 Acked Read 10 Done Sent Done false
8980 3407 Acked Read 5 Done Sent Done false
8981 3408 Acked Read 2 Done Sent Done false
8982 3409 Acked Flush 0 Done Sent Done false
8983 3410 Acked Write 4 Done Sent Done false
8984 3411 Acked Write 3 Done Sent Done false
8985 3412 Acked Read 3 Done Sent Done false
8986 3413 Acked Read 9 Done Sent Done false
8987 3414 Acked Write 1 Done Sent Done false
8988 STATES DS:0 DS:1 DS:2 TOTAL
8989 Sent 0 301 0 301
8990 Done 301 0 301 602
8991 Skipped 0 0 0 0
8992 Error 0 0 0 0
8993 Last Flush: Some(JobId(3409)) Some(JobId(3113)) Some(JobId(3409))
8994 Downstairs last five completed: 3113 3112 3111 3110 3109
8995 Upstairs last five completed: 3414 3413 3412 3411 3410
8996 CLIENT: Up:0 ds:301 act:2
89972025-01-29T05:12:38.103ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
89982025-01-29T05:12:39.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
89992025-01-29T05:12:40.208ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
90002025-01-29T05:12:41.108ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
9001 ----------------------------------------------------------------
9002 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9003 States: Active Offline Active
9004 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9005 3114 Acked Write 8 Done Sent Done false
9006 3115 Acked Read 3 Done Sent Done false
9007 3116 Acked Write 2 Done Sent Done false
9008 3117 Acked Read 4 Done Sent Done false
9009 3118 Acked Write 5 Done Sent Done false
9010 3119 Acked Read 9 Done Sent Done false
9011 3120 Acked Read 6 Done Sent Done false
9012 3121 Acked Flush 0 Done Sent Done false
9013 3122 Acked Read 7 Done Sent Done false
9014 3123 Acked Read 1 Done Sent Done false
9015 3124 Acked Read 1 Done Sent Done false
9016 3125 Acked Read 6 Done Sent Done false
9017 3126 Acked Read 6 Done Sent Done false
9018 3127 Acked Read 3 Done Sent Done false
9019 3128 Acked Write 4 Done Sent Done false
9020 3129 Acked Read 4 Done Sent Done false
9021 3130 Acked Read 6 Done Sent Done false
9022 3131 Acked Flush 0 Done Sent Done false
9023 3132 Acked Read 1 Done Sent Done false
9024 3133 Acked Write 9 Done Sent Done false
9025 3134 Acked Flush 0 Done Sent Done false
9026 3135 Acked Write 8 Done Sent Done false
9027 3136 Acked Read 6 Done Sent Done false
9028 3137 Acked Write 3 Done Sent Done false
9029 3138 Acked Flush 0 Done Sent Done false
9030 3139 Acked Read 3 Done Sent Done false
9031 3140 Acked Write 1 Done Sent Done false
9032 3141 Acked Read 7 Done Sent Done false
9033 3142 Acked Read 1 Done Sent Done false
9034 3143 Acked Write 5 Done Sent Done false
9035 3144 Acked Read 6 Done Sent Done false
9036 3145 Acked Write 8 Done Sent Done false
9037 3146 Acked Write 5 Done Sent Done false
9038 3147 Acked Read 5 Done Sent Done false
9039 3148 Acked Write 1 Done Sent Done false
9040 3149 Acked Write 10 Done Sent Done false
9041 3150 Acked Write 7 Done Sent Done false
9042 3151 Acked Write 8 Done Sent Done false
9043 3152 Acked Read 9 Done Sent Done false
9044 3153 Acked Read 6 Done Sent Done false
9045 3154 Acked Read 4 Done Sent Done false
9046 3155 Acked Write 1 Done Sent Done false
9047 3156 Acked Write 6 Done Sent Done false
9048 3157 Acked Read 7 Done Sent Done false
9049 3158 Acked Write 4 Done Sent Done false
9050 3159 Acked Read 2 Done Sent Done false
9051 3160 Acked Write 4 Done Sent Done false
9052 3161 Acked Read 1 Done Sent Done false
9053 3162 Acked Write 2 Done Sent Done false
9054 3163 Acked Write 5 Done Sent Done false
9055 3164 Acked Read 4 Done Sent Done false
9056 3165 Acked Write 10 Done Sent Done false
9057 3166 Acked Write 6 Done Sent Done false
9058 3167 Acked Flush 0 Done Sent Done false
9059 3168 Acked Read 10 Done Sent Done false
9060 3169 Acked Write 7 Done Sent Done false
9061 3170 Acked Write 5 Done Sent Done false
9062 3171 Acked Write 10 Done Sent Done false
9063 3172 Acked Write 7 Done Sent Done false
9064 3173 Acked Read 9 Done Sent Done false
9065 3174 Acked Read 9 Done Sent Done false
9066 3175 Acked Write 6 Done Sent Done false
9067 3176 Acked Write 6 Done Sent Done false
9068 3177 Acked Read 8 Done Sent Done false
9069 3178 Acked Read 1 Done Sent Done false
9070 3179 Acked Write 10 Done Sent Done false
9071 3180 Acked Write 8 Done Sent Done false
9072 3181 Acked Write 5 Done Sent Done false
9073 3182 Acked Flush 0 Done Sent Done false
9074 3183 Acked Read 2 Done Sent Done false
9075 3184 Acked Flush 0 Done Sent Done false
9076 3185 Acked Write 2 Done Sent Done false
9077 3186 Acked Read 10 Done Sent Done false
9078 3187 Acked Flush 0 Done Sent Done false
9079 3188 Acked Read 1 Done Sent Done false
9080 3189 Acked Write 5 Done Sent Done false
9081 3190 Acked Write 10 Done Sent Done false
9082 3191 Acked Write 5 Done Sent Done false
9083 3192 Acked Write 1 Done Sent Done false
9084 3193 Acked Read 6 Done Sent Done false
9085 3194 Acked Read 9 Done Sent Done false
9086 3195 Acked Write 8 Done Sent Done false
9087 3196 Acked Write 1 Done Sent Done false
9088 3197 Acked Read 8 Done Sent Done false
9089 3198 Acked Write 5 Done Sent Done false
9090 3199 Acked Read 8 Done Sent Done false
9091 3200 Acked Write 7 Done Sent Done false
9092 3201 Acked Read 8 Done Sent Done false
9093 3202 Acked Read 1 Done Sent Done false
9094 3203 Acked Write 1 Done Sent Done false
9095 3204 Acked Write 3 Done Sent Done false
9096 3205 Acked Read 7 Done Sent Done false
9097 3206 Acked Read 5 Done Sent Done false
9098 3207 Acked Read 10 Done Sent Done false
9099 3208 Acked Write 3 Done Sent Done false
9100 3209 Acked Read 3 Done Sent Done false
9101 3210 Acked Write 10 Done Sent Done false
9102 3211 Acked Read 7 Done Sent Done false
9103 3212 Acked Read 8 Done Sent Done false
9104 3213 Acked Write 10 Done Sent Done false
9105 3214 Acked Read 9 Done Sent Done false
9106 3215 Acked Flush 0 Done Sent Done false
9107 3216 Acked Write 2 Done Sent Done false
9108 3217 Acked Write 9 Done Sent Done false
9109 3218 Acked Write 10 Done Sent Done false
9110 3219 Acked Flush 0 Done Sent Done false
9111 3220 Acked Read 7 Done Sent Done false
9112 3221 Acked Read 1 Done Sent Done false
9113 3222 Acked Write 8 Done Sent Done false
9114 3223 Acked Read 9 Done Sent Done false
9115 3224 Acked Write 6 Done Sent Done false
9116 3225 Acked Write 5 Done Sent Done false
9117 3226 Acked Write 4 Done Sent Done false
9118 3227 Acked Read 8 Done Sent Done false
9119 3228 Acked Read 2 Done Sent Done false
9120 3229 Acked Read 2 Done Sent Done false
9121 3230 Acked Read 7 Done Sent Done false
9122 3231 Acked Read 8 Done Sent Done false
9123 3232 Acked Read 8 Done Sent Done false
9124 3233 Acked Read 6 Done Sent Done false
9125 3234 Acked Read 6 Done Sent Done false
9126 3235 Acked Write 4 Done Sent Done false
9127 3236 Acked Read 10 Done Sent Done false
9128 3237 Acked Write 4 Done Sent Done false
9129 3238 Acked Write 7 Done Sent Done false
9130 3239 Acked Read 4 Done Sent Done false
9131 3240 Acked Flush 0 Done Sent Done false
9132 3241 Acked Write 10 Done Sent Done false
9133 3242 Acked Read 4 Done Sent Done false
9134 3243 Acked Flush 0 Done Sent Done false
9135 3244 Acked Write 5 Done Sent Done false
9136 3245 Acked Read 10 Done Sent Done false
9137 3246 Acked Read 2 Done Sent Done false
9138 3247 Acked Read 1 Done Sent Done false
9139 3248 Acked Flush 0 Done Sent Done false
9140 3249 Acked Flush 0 Done Sent Done false
9141 3250 Acked Write 9 Done Sent Done false
9142 3251 Acked Read 2 Done Sent Done false
9143 3252 Acked Write 4 Done Sent Done false
9144 3253 Acked Write 1 Done Sent Done false
9145 3254 Acked Write 9 Done Sent Done false
9146 3255 Acked Read 7 Done Sent Done false
9147 3256 Acked Flush 0 Done Sent Done false
9148 3257 Acked Write 7 Done Sent Done false
9149 3258 Acked Write 3 Done Sent Done false
9150 3259 Acked Read 7 Done Sent Done false
9151 3260 Acked Read 3 Done Sent Done false
9152 3261 Acked Read 7 Done Sent Done false
9153 3262 Acked Write 7 Done Sent Done false
9154 3263 Acked Read 7 Done Sent Done false
9155 3264 Acked Read 10 Done Sent Done false
9156 3265 Acked Flush 0 Done Sent Done false
9157 3266 Acked Read 2 Done Sent Done false
9158 3267 Acked Write 10 Done Sent Done false
9159 3268 Acked Read 6 Done Sent Done false
9160 3269 Acked Read 3 Done Sent Done false
9161 3270 Acked Read 6 Done Sent Done false
9162 3271 Acked Flush 0 Done Sent Done false
9163 3272 Acked Read 8 Done Sent Done false
9164 3273 Acked Write 5 Done Sent Done false
9165 3274 Acked Write 3 Done Sent Done false
9166 3275 Acked Read 10 Done Sent Done false
9167 3276 Acked Read 2 Done Sent Done false
9168 3277 Acked Flush 0 Done Sent Done false
9169 3278 Acked Write 4 Done Sent Done false
9170 3279 Acked Read 1 Done Sent Done false
9171 3280 Acked Write 5 Done Sent Done false
9172 3281 Acked Read 9 Done Sent Done false
9173 3282 Acked Write 5 Done Sent Done false
9174 3283 Acked Read 7 Done Sent Done false
9175 3284 Acked Read 6 Done Sent Done false
9176 3285 Acked Write 9 Done Sent Done false
9177 3286 Acked Write 9 Done Sent Done false
9178 3287 Acked Read 8 Done Sent Done false
9179 3288 Acked Flush 0 Done Sent Done false
9180 3289 Acked Flush 0 Done Sent Done false
9181 3290 Acked Write 7 Done Sent Done false
9182 3291 Acked Write 2 Done Sent Done false
9183 3292 Acked Write 4 Done Sent Done false
9184 3293 Acked Read 1 Done Sent Done false
9185 3294 Acked Write 8 Done Sent Done false
9186 3295 Acked Write 10 Done Sent Done false
9187 3296 Acked Read 1 Done Sent Done false
9188 3297 Acked Write 3 Done Sent Done false
9189 3298 Acked Write 10 Done Sent Done false
9190 3299 Acked Flush 0 Done Sent Done false
9191 3300 Acked Read 6 Done Sent Done false
9192 3301 Acked Write 9 Done Sent Done false
9193 3302 Acked Read 9 Done Sent Done false
9194 3303 Acked Read 5 Done Sent Done false
9195 3304 Acked Write 2 Done Sent Done false
9196 3305 Acked Read 3 Done Sent Done false
9197 3306 Acked Write 4 Done Sent Done false
9198 3307 Acked Read 8 Done Sent Done false
9199 3308 Acked Flush 0 Done Sent Done false
9200 3309 Acked Read 5 Done Sent Done false
9201 3310 Acked Write 4 Done Sent Done false
9202 3311 Acked Read 3 Done Sent Done false
9203 3312 Acked Read 1 Done Sent Done false
9204 3313 Acked Read 6 Done Sent Done false
9205 3314 Acked Read 4 Done Sent Done false
9206 3315 Acked Flush 0 Done Sent Done false
9207 3316 Acked Read 9 Done Sent Done false
9208 3317 Acked Read 5 Done Sent Done false
9209 3318 Acked Write 7 Done Sent Done false
9210 3319 Acked Write 8 Done Sent Done false
9211 3320 Acked Read 4 Done Sent Done false
9212 3321 Acked Read 10 Done Sent Done false
9213 3322 Acked Read 8 Done Sent Done false
9214 3323 Acked Read 6 Done Sent Done false
9215 3324 Acked Read 9 Done Sent Done false
9216 3325 Acked Read 4 Done Sent Done false
9217 3326 Acked Read 6 Done Sent Done false
9218 3327 Acked Flush 0 Done Sent Done false
9219 3328 Acked Read 6 Done Sent Done false
9220 3329 Acked Write 6 Done Sent Done false
9221 3330 Acked Write 6 Done Sent Done false
9222 3331 Acked Read 2 Done Sent Done false
9223 3332 Acked Flush 0 Done Sent Done false
9224 3333 Acked Read 8 Done Sent Done false
9225 3334 Acked Write 10 Done Sent Done false
9226 3335 Acked Write 5 Done Sent Done false
9227 3336 Acked Read 3 Done Sent Done false
9228 3337 Acked Flush 0 Done Sent Done false
9229 3338 Acked Flush 0 Done Sent Done false
9230 3339 Acked Read 2 Done Sent Done false
9231 3340 Acked Write 3 Done Sent Done false
9232 3341 Acked Flush 0 Done Sent Done false
9233 3342 Acked Write 8 Done Sent Done false
9234 3343 Acked Write 2 Done Sent Done false
9235 3344 Acked Write 5 Done Sent Done false
9236 3345 Acked Read 10 Done Sent Done false
9237 3346 Acked Flush 0 Done Sent Done false
9238 3347 Acked Write 8 Done Sent Done false
9239 3348 Acked Write 4 Done Sent Done false
9240 3349 Acked Write 5 Done Sent Done false
9241 3350 Acked Read 2 Done Sent Done false
9242 3351 Acked Flush 0 Done Sent Done false
9243 3352 Acked Write 8 Done Sent Done false
9244 3353 Acked Read 6 Done Sent Done false
9245 3354 Acked Flush 0 Done Sent Done false
9246 3355 Acked Read 9 Done Sent Done false
9247 3356 Acked Read 5 Done Sent Done false
9248 3357 Acked Write 1 Done Sent Done false
9249 3358 Acked Flush 0 Done Sent Done false
9250 3359 Acked Read 8 Done Sent Done false
9251 3360 Acked Write 9 Done Sent Done false
9252 3361 Acked Read 9 Done Sent Done false
9253 3362 Acked Read 10 Done Sent Done false
9254 3363 Acked Read 7 Done Sent Done false
9255 3364 Acked Read 5 Done Sent Done false
9256 3365 Acked Read 7 Done Sent Done false
9257 3366 Acked Read 4 Done Sent Done false
9258 3367 Acked Write 2 Done Sent Done false
9259 3368 Acked Write 10 Done Sent Done false
9260 3369 Acked Write 7 Done Sent Done false
9261 3370 Acked Read 5 Done Sent Done false
9262 3371 Acked Write 9 Done Sent Done false
9263 3372 Acked Write 2 Done Sent Done false
9264 3373 Acked Write 3 Done Sent Done false
9265 3374 Acked Write 7 Done Sent Done false
9266 3375 Acked Read 7 Done Sent Done false
9267 3376 Acked Write 10 Done Sent Done false
9268 3377 Acked Write 10 Done Sent Done false
9269 3378 Acked Read 1 Done Sent Done false
9270 3379 Acked Read 7 Done Sent Done false
9271 3380 Acked Write 9 Done Sent Done false
9272 3381 Acked Read 10 Done Sent Done false
9273 3382 Acked Read 2 Done Sent Done false
9274 3383 Acked Read 5 Done Sent Done false
9275 3384 Acked Read 6 Done Sent Done false
9276 3385 Acked Write 1 Done Sent Done false
9277 3386 Acked Write 5 Done Sent Done false
9278 3387 Acked Write 10 Done Sent Done false
9279 3388 Acked Write 3 Done Sent Done false
9280 3389 Acked Read 6 Done Sent Done false
9281 3390 Acked Write 2 Done Sent Done false
9282 3391 Acked Write 9 Done Sent Done false
9283 3392 Acked Write 8 Done Sent Done false
9284 3393 Acked Read 5 Done Sent Done false
9285 3394 Acked Read 9 Done Sent Done false
9286 3395 Acked Read 8 Done Sent Done false
9287 3396 Acked Read 10 Done Sent Done false
9288 3397 Acked Write 4 Done Sent Done false
9289 3398 Acked Read 9 Done Sent Done false
9290 3399 Acked Write 6 Done Sent Done false
9291 3400 Acked Write 8 Done Sent Done false
9292 3401 Acked Write 9 Done Sent Done false
9293 3402 Acked Read 9 Done Sent Done false
9294 3403 Acked Write 4 Done Sent Done false
9295 3404 Acked Read 10 Done Sent Done false
9296 3405 Acked Write 7 Done Sent Done false
9297 3406 Acked Read 10 Done Sent Done false
9298 3407 Acked Read 5 Done Sent Done false
9299 3408 Acked Read 2 Done Sent Done false
9300 3409 Acked Flush 0 Done Sent Done false
9301 3410 Acked Write 4 Done Sent Done false
9302 3411 Acked Write 3 Done Sent Done false
9303 3412 Acked Read 3 Done Sent Done false
9304 3413 Acked Read 9 Done Sent Done false
9305 3414 Acked Write 1 Done Sent Done false
9306 STATES DS:0 DS:1 DS:2 TOTAL
9307 Sent 0 301 0 301
9308 Done 301 0 301 602
9309 Skipped 0 0 0 0
9310 Error 0 0 0 0
9311 Last Flush: Some(JobId(3409)) Some(JobId(3113)) Some(JobId(3409))
9312 Downstairs last five completed: 3113 3112 3111 3110 3109
9313 Upstairs last five completed: 3414 3413 3412 3411 3410
9314 CLIENT: Up:0 ds:301 act:2
93152025-01-29T05:12:42.109ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93162025-01-29T05:12:43.111ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93172025-01-29T05:12:43.212ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93182025-01-29T05:12:43.212ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93192025-01-29T05:12:43.212ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93202025-01-29T05:12:43.212ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93212025-01-29T05:12:43.212ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93222025-01-29T05:12:44.113ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
93232025-01-29T05:12:45.208ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
9324 [8820] got start action
9325 [8820] I am going to restart
9326 Starting downstairs at port 8820
9327 Make output file at "/tmp/dsc/downstairs-8820.txt"
9328 [8820][1] reports Starting
9329 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1210)
9330 [8820][1] reports Running
9331 ----------------------------------------------------------------
9332 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9333 States: Active Offline Active
9334 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9335 3114 Acked Write 8 Done Sent Done false
9336 3115 Acked Read 3 Done Sent Done false
9337 3116 Acked Write 2 Done Sent Done false
9338 3117 Acked Read 4 Done Sent Done false
9339 3118 Acked Write 5 Done Sent Done false
9340 3119 Acked Read 9 Done Sent Done false
9341 3120 Acked Read 6 Done Sent Done false
9342 3121 Acked Flush 0 Done Sent Done false
9343 3122 Acked Read 7 Done Sent Done false
9344 3123 Acked Read 1 Done Sent Done false
9345 3124 Acked Read 1 Done Sent Done false
9346 3125 Acked Read 6 Done Sent Done false
9347 3126 Acked Read 6 Done Sent Done false
9348 3127 Acked Read 3 Done Sent Done false
9349 3128 Acked Write 4 Done Sent Done false
9350 3129 Acked Read 4 Done Sent Done false
9351 3130 Acked Read 6 Done Sent Done false
9352 3131 Acked Flush 0 Done Sent Done false
9353 3132 Acked Read 1 Done Sent Done false
9354 3133 Acked Write 9 Done Sent Done false
9355 3134 Acked Flush 0 Done Sent Done false
9356 3135 Acked Write 8 Done Sent Done false
9357 3136 Acked Read 6 Done Sent Done false
9358 3137 Acked Write 3 Done Sent Done false
9359 3138 Acked Flush 0 Done Sent Done false
9360 3139 Acked Read 3 Done Sent Done false
9361 3140 Acked Write 1 Done Sent Done false
9362 3141 Acked Read 7 Done Sent Done false
9363 3142 Acked Read 1 Done Sent Done false
9364 3143 Acked Write 5 Done Sent Done false
9365 3144 Acked Read 6 Done Sent Done false
9366 3145 Acked Write 8 Done Sent Done false
9367 3146 Acked Write 5 Done Sent Done false
9368 3147 Acked Read 5 Done Sent Done false
9369 3148 Acked Write 1 Done Sent Done false
9370 3149 Acked Write 10 Done Sent Done false
9371 3150 Acked Write 7 Done Sent Done false
9372 3151 Acked Write 8 Done Sent Done false
9373 3152 Acked Read 9 Done Sent Done false
9374 3153 Acked Read 6 Done Sent Done false
9375 3154 Acked Read 4 Done Sent Done false
9376 3155 Acked Write 1 Done Sent Done false
9377 3156 Acked Write 6 Done Sent Done false
9378 3157 Acked Read 7 Done Sent Done false
9379 3158 Acked Write 4 Done Sent Done false
9380 3159 Acked Read 2 Done Sent Done false
9381 3160 Acked Write 4 Done Sent Done false
9382 3161 Acked Read 1 Done Sent Done false
9383 3162 Acked Write 2 Done Sent Done false
9384 3163 Acked Write 5 Done Sent Done false
9385 3164 Acked Read 4 Done Sent Done false
9386 3165 Acked Write 10 Done Sent Done false
9387 3166 Acked Write 6 Done Sent Done false
9388 3167 Acked Flush 0 Done Sent Done false
9389 3168 Acked Read 10 Done Sent Done false
9390 3169 Acked Write 7 Done Sent Done false
9391 3170 Acked Write 5 Done Sent Done false
9392 3171 Acked Write 10 Done Sent Done false
9393 3172 Acked Write 7 Done Sent Done false
9394 3173 Acked Read 9 Done Sent Done false
9395 3174 Acked Read 9 Done Sent Done false
9396 3175 Acked Write 6 Done Sent Done false
9397 3176 Acked Write 6 Done Sent Done false
9398 3177 Acked Read 8 Done Sent Done false
9399 3178 Acked Read 1 Done Sent Done false
9400 3179 Acked Write 10 Done Sent Done false
9401 3180 Acked Write 8 Done Sent Done false
9402 3181 Acked Write 5 Done Sent Done false
9403 3182 Acked Flush 0 Done Sent Done false
9404 3183 Acked Read 2 Done Sent Done false
9405 3184 Acked Flush 0 Done Sent Done false
9406 3185 Acked Write 2 Done Sent Done false
9407 3186 Acked Read 10 Done Sent Done false
9408 3187 Acked Flush 0 Done Sent Done false
9409 3188 Acked Read 1 Done Sent Done false
9410 3189 Acked Write 5 Done Sent Done false
9411 3190 Acked Write 10 Done Sent Done false
9412 3191 Acked Write 5 Done Sent Done false
9413 3192 Acked Write 1 Done Sent Done false
9414 3193 Acked Read 6 Done Sent Done false
9415 3194 Acked Read 9 Done Sent Done false
9416 3195 Acked Write 8 Done Sent Done false
9417 3196 Acked Write 1 Done Sent Done false
9418 3197 Acked Read 8 Done Sent Done false
9419 3198 Acked Write 5 Done Sent Done false
9420 3199 Acked Read 8 Done Sent Done false
9421 3200 Acked Write 7 Done Sent Done false
9422 3201 Acked Read 8 Done Sent Done false
9423 3202 Acked Read 1 Done Sent Done false
9424 3203 Acked Write 1 Done Sent Done false
9425 3204 Acked Write 3 Done Sent Done false
9426 3205 Acked Read 7 Done Sent Done false
9427 3206 Acked Read 5 Done Sent Done false
9428 3207 Acked Read 10 Done Sent Done false
9429 3208 Acked Write 3 Done Sent Done false
9430 3209 Acked Read 3 Done Sent Done false
9431 3210 Acked Write 10 Done Sent Done false
9432 3211 Acked Read 7 Done Sent Done false
9433 3212 Acked Read 8 Done Sent Done false
9434 3213 Acked Write 10 Done Sent Done false
9435 3214 Acked Read 9 Done Sent Done false
9436 3215 Acked Flush 0 Done Sent Done false
9437 3216 Acked Write 2 Done Sent Done false
9438 3217 Acked Write 9 Done Sent Done false
9439 3218 Acked Write 10 Done Sent Done false
9440 3219 Acked Flush 0 Done Sent Done false
9441 3220 Acked Read 7 Done Sent Done false
9442 3221 Acked Read 1 Done Sent Done false
9443 3222 Acked Write 8 Done Sent Done false
9444 3223 Acked Read 9 Done Sent Done false
9445 3224 Acked Write 6 Done Sent Done false
9446 3225 Acked Write 5 Done Sent Done false
9447 3226 Acked Write 4 Done Sent Done false
9448 3227 Acked Read 8 Done Sent Done false
9449 3228 Acked Read 2 Done Sent Done false
9450 3229 Acked Read 2 Done Sent Done false
9451 3230 Acked Read 7 Done Sent Done false
9452 3231 Acked Read 8 Done Sent Done false
9453 3232 Acked Read 8 Done Sent Done false
9454 3233 Acked Read 6 Done Sent Done false
9455 3234 Acked Read 6 Done Sent Done false
9456 3235 Acked Write 4 Done Sent Done false
9457 3236 Acked Read 10 Done Sent Done false
9458 3237 Acked Write 4 Done Sent Done false
9459 3238 Acked Write 7 Done Sent Done false
9460 3239 Acked Read 4 Done Sent Done false
9461 3240 Acked Flush 0 Done Sent Done false
9462 3241 Acked Write 10 Done Sent Done false
9463 3242 Acked Read 4 Done Sent Done false
9464 3243 Acked Flush 0 Done Sent Done false
9465 3244 Acked Write 5 Done Sent Done false
9466 3245 Acked Read 10 Done Sent Done false
9467 3246 Acked Read 2 Done Sent Done false
9468 3247 Acked Read 1 Done Sent Done false
9469 3248 Acked Flush 0 Done Sent Done false
9470 3249 Acked Flush 0 Done Sent Done false
9471 3250 Acked Write 9 Done Sent Done false
9472 3251 Acked Read 2 Done Sent Done false
9473 3252 Acked Write 4 Done Sent Done false
9474 3253 Acked Write 1 Done Sent Done false
9475 3254 Acked Write 9 Done Sent Done false
9476 3255 Acked Read 7 Done Sent Done false
9477 3256 Acked Flush 0 Done Sent Done false
9478 3257 Acked Write 7 Done Sent Done false
9479 3258 Acked Write 3 Done Sent Done false
9480 3259 Acked Read 7 Done Sent Done false
9481 3260 Acked Read 3 Done Sent Done false
9482 3261 Acked Read 7 Done Sent Done false
9483 3262 Acked Write 7 Done Sent Done false
9484 3263 Acked Read 7 Done Sent Done false
9485 3264 Acked Read 10 Done Sent Done false
9486 3265 Acked Flush 0 Done Sent Done false
9487 3266 Acked Read 2 Done Sent Done false
9488 3267 Acked Write 10 Done Sent Done false
9489 3268 Acked Read 6 Done Sent Done false
9490 3269 Acked Read 3 Done Sent Done false
9491 3270 Acked Read 6 Done Sent Done false
9492 3271 Acked Flush 0 Done Sent Done false
9493 3272 Acked Read 8 Done Sent Done false
9494 3273 Acked Write 5 Done Sent Done false
9495 3274 Acked Write 3 Done Sent Done false
9496 3275 Acked Read 10 Done Sent Done false
9497 3276 Acked Read 2 Done Sent Done false
9498 3277 Acked Flush 0 Done Sent Done false
9499 3278 Acked Write 4 Done Sent Done false
9500 3279 Acked Read 1 Done Sent Done false
9501 3280 Acked Write 5 Done Sent Done false
9502 3281 Acked Read 9 Done Sent Done false
9503 3282 Acked Write 5 Done Sent Done false
9504 3283 Acked Read 7 Done Sent Done false
9505 3284 Acked Read 6 Done Sent Done false
9506 3285 Acked Write 9 Done Sent Done false
9507 3286 Acked Write 9 Done Sent Done false
9508 3287 Acked Read 8 Done Sent Done false
9509 3288 Acked Flush 0 Done Sent Done false
9510 3289 Acked Flush 0 Done Sent Done false
9511 3290 Acked Write 7 Done Sent Done false
9512 3291 Acked Write 2 Done Sent Done false
9513 3292 Acked Write 4 Done Sent Done false
9514 3293 Acked Read 1 Done Sent Done false
9515 3294 Acked Write 8 Done Sent Done false
9516 3295 Acked Write 10 Done Sent Done false
9517 3296 Acked Read 1 Done Sent Done false
9518 3297 Acked Write 3 Done Sent Done false
9519 3298 Acked Write 10 Done Sent Done false
9520 3299 Acked Flush 0 Done Sent Done false
9521 3300 Acked Read 6 Done Sent Done false
9522 3301 Acked Write 9 Done Sent Done false
9523 3302 Acked Read 9 Done Sent Done false
9524 3303 Acked Read 5 Done Sent Done false
9525 3304 Acked Write 2 Done Sent Done false
9526 3305 Acked Read 3 Done Sent Done false
9527 3306 Acked Write 4 Done Sent Done false
9528 3307 Acked Read 8 Done Sent Done false
9529 3308 Acked Flush 0 Done Sent Done false
9530 3309 Acked Read 5 Done Sent Done false
9531 3310 Acked Write 4 Done Sent Done false
9532 3311 Acked Read 3 Done Sent Done false
9533 3312 Acked Read 1 Done Sent Done false
9534 3313 Acked Read 6 Done Sent Done false
9535 3314 Acked Read 4 Done Sent Done false
9536 3315 Acked Flush 0 Done Sent Done false
9537 3316 Acked Read 9 Done Sent Done false
9538 3317 Acked Read 5 Done Sent Done false
9539 3318 Acked Write 7 Done Sent Done false
9540 3319 Acked Write 8 Done Sent Done false
9541 3320 Acked Read 4 Done Sent Done false
9542 3321 Acked Read 10 Done Sent Done false
9543 3322 Acked Read 8 Done Sent Done false
9544 3323 Acked Read 6 Done Sent Done false
9545 3324 Acked Read 9 Done Sent Done false
9546 3325 Acked Read 4 Done Sent Done false
9547 3326 Acked Read 6 Done Sent Done false
9548 3327 Acked Flush 0 Done Sent Done false
9549 3328 Acked Read 6 Done Sent Done false
9550 3329 Acked Write 6 Done Sent Done false
9551 3330 Acked Write 6 Done Sent Done false
9552 3331 Acked Read 2 Done Sent Done false
9553 3332 Acked Flush 0 Done Sent Done false
9554 3333 Acked Read 8 Done Sent Done false
9555 3334 Acked Write 10 Done Sent Done false
9556 3335 Acked Write 5 Done Sent Done false
9557 3336 Acked Read 3 Done Sent Done false
9558 3337 Acked Flush 0 Done Sent Done false
9559 3338 Acked Flush 0 Done Sent Done false
9560 3339 Acked Read 2 Done Sent Done false
9561 3340 Acked Write 3 Done Sent Done false
9562 3341 Acked Flush 0 Done Sent Done false
9563 3342 Acked Write 8 Done Sent Done false
9564 3343 Acked Write 2 Done Sent Done false
9565 3344 Acked Write 5 Done Sent Done false
9566 3345 Acked Read 10 Done Sent Done false
9567 3346 Acked Flush 0 Done Sent Done false
9568 3347 Acked Write 8 Done Sent Done false
9569 3348 Acked Write 4 Done Sent Done false
9570 3349 Acked Write 5 Done Sent Done false
9571 3350 Acked Read 2 Done Sent Done false
9572 3351 Acked Flush 0 Done Sent Done false
9573 3352 Acked Write 8 Done Sent Done false
9574 3353 Acked Read 6 Done Sent Done false
9575 3354 Acked Flush 0 Done Sent Done false
9576 3355 Acked Read 9 Done Sent Done false
9577 3356 Acked Read 5 Done Sent Done false
9578 3357 Acked Write 1 Done Sent Done false
9579 3358 Acked Flush 0 Done Sent Done false
9580 3359 Acked Read 8 Done Sent Done false
9581 3360 Acked Write 9 Done Sent Done false
9582 3361 Acked Read 9 Done Sent Done false
9583 3362 Acked Read 10 Done Sent Done false
9584 3363 Acked Read 7 Done Sent Done false
9585 3364 Acked Read 5 Done Sent Done false
9586 3365 Acked Read 7 Done Sent Done false
9587 3366 Acked Read 4 Done Sent Done false
9588 3367 Acked Write 2 Done Sent Done false
9589 3368 Acked Write 10 Done Sent Done false
9590 3369 Acked Write 7 Done Sent Done false
9591 3370 Acked Read 5 Done Sent Done false
9592 3371 Acked Write 9 Done Sent Done false
9593 3372 Acked Write 2 Done Sent Done false
9594 3373 Acked Write 3 Done Sent Done false
9595 3374 Acked Write 7 Done Sent Done false
9596 3375 Acked Read 7 Done Sent Done false
9597 3376 Acked Write 10 Done Sent Done false
9598 3377 Acked Write 10 Done Sent Done false
9599 3378 Acked Read 1 Done Sent Done false
9600 3379 Acked Read 7 Done Sent Done false
9601 3380 Acked Write 9 Done Sent Done false
9602 3381 Acked Read 10 Done Sent Done false
9603 3382 Acked Read 2 Done Sent Done false
9604 3383 Acked Read 5 Done Sent Done false
9605 3384 Acked Read 6 Done Sent Done false
9606 3385 Acked Write 1 Done Sent Done false
9607 3386 Acked Write 5 Done Sent Done false
9608 3387 Acked Write 10 Done Sent Done false
9609 3388 Acked Write 3 Done Sent Done false
9610 3389 Acked Read 6 Done Sent Done false
9611 3390 Acked Write 2 Done Sent Done false
9612 3391 Acked Write 9 Done Sent Done false
9613 3392 Acked Write 8 Done Sent Done false
9614 3393 Acked Read 5 Done Sent Done false
9615 3394 Acked Read 9 Done Sent Done false
9616 3395 Acked Read 8 Done Sent Done false
9617 3396 Acked Read 10 Done Sent Done false
9618 3397 Acked Write 4 Done Sent Done false
9619 3398 Acked Read 9 Done Sent Done false
9620 3399 Acked Write 6 Done Sent Done false
9621 3400 Acked Write 8 Done Sent Done false
9622 3401 Acked Write 9 Done Sent Done false
9623 3402 Acked Read 9 Done Sent Done false
9624 3403 Acked Write 4 Done Sent Done false
9625 3404 Acked Read 10 Done Sent Done false
9626 3405 Acked Write 7 Done Sent Done false
9627 3406 Acked Read 10 Done Sent Done false
9628 3407 Acked Read 5 Done Sent Done false
9629 3408 Acked Read 2 Done Sent Done false
9630 3409 Acked Flush 0 Done Sent Done false
9631 3410 Acked Write 4 Done Sent Done false
9632 3411 Acked Write 3 Done Sent Done false
9633 3412 Acked Read 3 Done Sent Done false
9634 3413 Acked Read 9 Done Sent Done false
9635 3414 Acked Write 1 Done Sent Done false
9636 STATES DS:0 DS:1 DS:2 TOTAL
9637 Sent 0 301 0 301
9638 Done 301 0 301 602
9639 Skipped 0 0 0 0
9640 Error 0 0 0 0
9641 Last Flush: Some(JobId(3409)) Some(JobId(3113)) Some(JobId(3409))
9642 Downstairs last five completed: 3113 3112 3111 3110 3109
9643 Upstairs last five completed: 3414 3413 3412 3411 3410
9644 CLIENT: Up:0 ds:301 act:2
96452025-01-29T05:12:46.116ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
96462025-01-29T05:12:47.118ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
96472025-01-29T05:12:48.120ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
96482025-01-29T05:12:49.122ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
9649 ----------------------------------------------------------------
9650 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9651 States: Active Offline Active
9652 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9653 3114 Acked Write 8 Done Sent Done false
9654 3115 Acked Read 3 Done Sent Done false
9655 3116 Acked Write 2 Done Sent Done false
9656 3117 Acked Read 4 Done Sent Done false
9657 3118 Acked Write 5 Done Sent Done false
9658 3119 Acked Read 9 Done Sent Done false
9659 3120 Acked Read 6 Done Sent Done false
9660 3121 Acked Flush 0 Done Sent Done false
9661 3122 Acked Read 7 Done Sent Done false
9662 3123 Acked Read 1 Done Sent Done false
9663 3124 Acked Read 1 Done Sent Done false
9664 3125 Acked Read 6 Done Sent Done false
9665 3126 Acked Read 6 Done Sent Done false
9666 3127 Acked Read 3 Done Sent Done false
9667 3128 Acked Write 4 Done Sent Done false
9668 3129 Acked Read 4 Done Sent Done false
9669 3130 Acked Read 6 Done Sent Done false
9670 3131 Acked Flush 0 Done Sent Done false
9671 3132 Acked Read 1 Done Sent Done false
9672 3133 Acked Write 9 Done Sent Done false
9673 3134 Acked Flush 0 Done Sent Done false
9674 3135 Acked Write 8 Done Sent Done false
9675 3136 Acked Read 6 Done Sent Done false
9676 3137 Acked Write 3 Done Sent Done false
9677 3138 Acked Flush 0 Done Sent Done false
9678 3139 Acked Read 3 Done Sent Done false
9679 3140 Acked Write 1 Done Sent Done false
9680 3141 Acked Read 7 Done Sent Done false
9681 3142 Acked Read 1 Done Sent Done false
9682 3143 Acked Write 5 Done Sent Done false
9683 3144 Acked Read 6 Done Sent Done false
9684 3145 Acked Write 8 Done Sent Done false
9685 3146 Acked Write 5 Done Sent Done false
9686 3147 Acked Read 5 Done Sent Done false
9687 3148 Acked Write 1 Done Sent Done false
9688 3149 Acked Write 10 Done Sent Done false
9689 3150 Acked Write 7 Done Sent Done false
9690 3151 Acked Write 8 Done Sent Done false
9691 3152 Acked Read 9 Done Sent Done false
9692 3153 Acked Read 6 Done Sent Done false
9693 3154 Acked Read 4 Done Sent Done false
9694 3155 Acked Write 1 Done Sent Done false
9695 3156 Acked Write 6 Done Sent Done false
9696 3157 Acked Read 7 Done Sent Done false
9697 3158 Acked Write 4 Done Sent Done false
9698 3159 Acked Read 2 Done Sent Done false
9699 3160 Acked Write 4 Done Sent Done false
9700 3161 Acked Read 1 Done Sent Done false
9701 3162 Acked Write 2 Done Sent Done false
9702 3163 Acked Write 5 Done Sent Done false
9703 3164 Acked Read 4 Done Sent Done false
9704 3165 Acked Write 10 Done Sent Done false
9705 3166 Acked Write 6 Done Sent Done false
9706 3167 Acked Flush 0 Done Sent Done false
9707 3168 Acked Read 10 Done Sent Done false
9708 3169 Acked Write 7 Done Sent Done false
9709 3170 Acked Write 5 Done Sent Done false
9710 3171 Acked Write 10 Done Sent Done false
9711 3172 Acked Write 7 Done Sent Done false
9712 3173 Acked Read 9 Done Sent Done false
9713 3174 Acked Read 9 Done Sent Done false
9714 3175 Acked Write 6 Done Sent Done false
9715 3176 Acked Write 6 Done Sent Done false
9716 3177 Acked Read 8 Done Sent Done false
9717 3178 Acked Read 1 Done Sent Done false
9718 3179 Acked Write 10 Done Sent Done false
9719 3180 Acked Write 8 Done Sent Done false
9720 3181 Acked Write 5 Done Sent Done false
9721 3182 Acked Flush 0 Done Sent Done false
9722 3183 Acked Read 2 Done Sent Done false
9723 3184 Acked Flush 0 Done Sent Done false
9724 3185 Acked Write 2 Done Sent Done false
9725 3186 Acked Read 10 Done Sent Done false
9726 3187 Acked Flush 0 Done Sent Done false
9727 3188 Acked Read 1 Done Sent Done false
9728 3189 Acked Write 5 Done Sent Done false
9729 3190 Acked Write 10 Done Sent Done false
9730 3191 Acked Write 5 Done Sent Done false
9731 3192 Acked Write 1 Done Sent Done false
9732 3193 Acked Read 6 Done Sent Done false
9733 3194 Acked Read 9 Done Sent Done false
9734 3195 Acked Write 8 Done Sent Done false
9735 3196 Acked Write 1 Done Sent Done false
9736 3197 Acked Read 8 Done Sent Done false
9737 3198 Acked Write 5 Done Sent Done false
9738 3199 Acked Read 8 Done Sent Done false
9739 3200 Acked Write 7 Done Sent Done false
9740 3201 Acked Read 8 Done Sent Done false
9741 3202 Acked Read 1 Done Sent Done false
9742 3203 Acked Write 1 Done Sent Done false
9743 3204 Acked Write 3 Done Sent Done false
9744 3205 Acked Read 7 Done Sent Done false
9745 3206 Acked Read 5 Done Sent Done false
9746 3207 Acked Read 10 Done Sent Done false
9747 3208 Acked Write 3 Done Sent Done false
9748 3209 Acked Read 3 Done Sent Done false
9749 3210 Acked Write 10 Done Sent Done false
9750 3211 Acked Read 7 Done Sent Done false
9751 3212 Acked Read 8 Done Sent Done false
9752 3213 Acked Write 10 Done Sent Done false
9753 3214 Acked Read 9 Done Sent Done false
9754 3215 Acked Flush 0 Done Sent Done false
9755 3216 Acked Write 2 Done Sent Done false
9756 3217 Acked Write 9 Done Sent Done false
9757 3218 Acked Write 10 Done Sent Done false
9758 3219 Acked Flush 0 Done Sent Done false
9759 3220 Acked Read 7 Done Sent Done false
9760 3221 Acked Read 1 Done Sent Done false
9761 3222 Acked Write 8 Done Sent Done false
9762 3223 Acked Read 9 Done Sent Done false
9763 3224 Acked Write 6 Done Sent Done false
9764 3225 Acked Write 5 Done Sent Done false
9765 3226 Acked Write 4 Done Sent Done false
9766 3227 Acked Read 8 Done Sent Done false
9767 3228 Acked Read 2 Done Sent Done false
9768 3229 Acked Read 2 Done Sent Done false
9769 3230 Acked Read 7 Done Sent Done false
9770 3231 Acked Read 8 Done Sent Done false
9771 3232 Acked Read 8 Done Sent Done false
9772 3233 Acked Read 6 Done Sent Done false
9773 3234 Acked Read 6 Done Sent Done false
9774 3235 Acked Write 4 Done Sent Done false
9775 3236 Acked Read 10 Done Sent Done false
9776 3237 Acked Write 4 Done Sent Done false
9777 3238 Acked Write 7 Done Sent Done false
9778 3239 Acked Read 4 Done Sent Done false
9779 3240 Acked Flush 0 Done Sent Done false
9780 3241 Acked Write 10 Done Sent Done false
9781 3242 Acked Read 4 Done Sent Done false
9782 3243 Acked Flush 0 Done Sent Done false
9783 3244 Acked Write 5 Done Sent Done false
9784 3245 Acked Read 10 Done Sent Done false
9785 3246 Acked Read 2 Done Sent Done false
9786 3247 Acked Read 1 Done Sent Done false
9787 3248 Acked Flush 0 Done Sent Done false
9788 3249 Acked Flush 0 Done Sent Done false
9789 3250 Acked Write 9 Done Sent Done false
9790 3251 Acked Read 2 Done Sent Done false
9791 3252 Acked Write 4 Done Sent Done false
9792 3253 Acked Write 1 Done Sent Done false
9793 3254 Acked Write 9 Done Sent Done false
9794 3255 Acked Read 7 Done Sent Done false
9795 3256 Acked Flush 0 Done Sent Done false
9796 3257 Acked Write 7 Done Sent Done false
9797 3258 Acked Write 3 Done Sent Done false
9798 3259 Acked Read 7 Done Sent Done false
9799 3260 Acked Read 3 Done Sent Done false
9800 3261 Acked Read 7 Done Sent Done false
9801 3262 Acked Write 7 Done Sent Done false
9802 3263 Acked Read 7 Done Sent Done false
9803 3264 Acked Read 10 Done Sent Done false
9804 3265 Acked Flush 0 Done Sent Done false
9805 3266 Acked Read 2 Done Sent Done false
9806 3267 Acked Write 10 Done Sent Done false
9807 3268 Acked Read 6 Done Sent Done false
9808 3269 Acked Read 3 Done Sent Done false
9809 3270 Acked Read 6 Done Sent Done false
9810 3271 Acked Flush 0 Done Sent Done false
9811 3272 Acked Read 8 Done Sent Done false
9812 3273 Acked Write 5 Done Sent Done false
9813 3274 Acked Write 3 Done Sent Done false
9814 3275 Acked Read 10 Done Sent Done false
9815 3276 Acked Read 2 Done Sent Done false
9816 3277 Acked Flush 0 Done Sent Done false
9817 3278 Acked Write 4 Done Sent Done false
9818 3279 Acked Read 1 Done Sent Done false
9819 3280 Acked Write 5 Done Sent Done false
9820 3281 Acked Read 9 Done Sent Done false
9821 3282 Acked Write 5 Done Sent Done false
9822 3283 Acked Read 7 Done Sent Done false
9823 3284 Acked Read 6 Done Sent Done false
9824 3285 Acked Write 9 Done Sent Done false
9825 3286 Acked Write 9 Done Sent Done false
9826 3287 Acked Read 8 Done Sent Done false
9827 3288 Acked Flush 0 Done Sent Done false
9828 3289 Acked Flush 0 Done Sent Done false
9829 3290 Acked Write 7 Done Sent Done false
9830 3291 Acked Write 2 Done Sent Done false
9831 3292 Acked Write 4 Done Sent Done false
9832 3293 Acked Read 1 Done Sent Done false
9833 3294 Acked Write 8 Done Sent Done false
9834 3295 Acked Write 10 Done Sent Done false
9835 3296 Acked Read 1 Done Sent Done false
9836 3297 Acked Write 3 Done Sent Done false
9837 3298 Acked Write 10 Done Sent Done false
9838 3299 Acked Flush 0 Done Sent Done false
9839 3300 Acked Read 6 Done Sent Done false
9840 3301 Acked Write 9 Done Sent Done false
9841 3302 Acked Read 9 Done Sent Done false
9842 3303 Acked Read 5 Done Sent Done false
9843 3304 Acked Write 2 Done Sent Done false
9844 3305 Acked Read 3 Done Sent Done false
9845 3306 Acked Write 4 Done Sent Done false
9846 3307 Acked Read 8 Done Sent Done false
9847 3308 Acked Flush 0 Done Sent Done false
9848 3309 Acked Read 5 Done Sent Done false
9849 3310 Acked Write 4 Done Sent Done false
9850 3311 Acked Read 3 Done Sent Done false
9851 3312 Acked Read 1 Done Sent Done false
9852 3313 Acked Read 6 Done Sent Done false
9853 3314 Acked Read 4 Done Sent Done false
9854 3315 Acked Flush 0 Done Sent Done false
9855 3316 Acked Read 9 Done Sent Done false
9856 3317 Acked Read 5 Done Sent Done false
9857 3318 Acked Write 7 Done Sent Done false
9858 3319 Acked Write 8 Done Sent Done false
9859 3320 Acked Read 4 Done Sent Done false
9860 3321 Acked Read 10 Done Sent Done false
9861 3322 Acked Read 8 Done Sent Done false
9862 3323 Acked Read 6 Done Sent Done false
9863 3324 Acked Read 9 Done Sent Done false
9864 3325 Acked Read 4 Done Sent Done false
9865 3326 Acked Read 6 Done Sent Done false
9866 3327 Acked Flush 0 Done Sent Done false
9867 3328 Acked Read 6 Done Sent Done false
9868 3329 Acked Write 6 Done Sent Done false
9869 3330 Acked Write 6 Done Sent Done false
9870 3331 Acked Read 2 Done Sent Done false
9871 3332 Acked Flush 0 Done Sent Done false
9872 3333 Acked Read 8 Done Sent Done false
9873 3334 Acked Write 10 Done Sent Done false
9874 3335 Acked Write 5 Done Sent Done false
9875 3336 Acked Read 3 Done Sent Done false
9876 3337 Acked Flush 0 Done Sent Done false
9877 3338 Acked Flush 0 Done Sent Done false
9878 3339 Acked Read 2 Done Sent Done false
9879 3340 Acked Write 3 Done Sent Done false
9880 3341 Acked Flush 0 Done Sent Done false
9881 3342 Acked Write 8 Done Sent Done false
9882 3343 Acked Write 2 Done Sent Done false
9883 3344 Acked Write 5 Done Sent Done false
9884 3345 Acked Read 10 Done Sent Done false
9885 3346 Acked Flush 0 Done Sent Done false
9886 3347 Acked Write 8 Done Sent Done false
9887 3348 Acked Write 4 Done Sent Done false
9888 3349 Acked Write 5 Done Sent Done false
9889 3350 Acked Read 2 Done Sent Done false
9890 3351 Acked Flush 0 Done Sent Done false
9891 3352 Acked Write 8 Done Sent Done false
9892 3353 Acked Read 6 Done Sent Done false
9893 3354 Acked Flush 0 Done Sent Done false
9894 3355 Acked Read 9 Done Sent Done false
9895 3356 Acked Read 5 Done Sent Done false
9896 3357 Acked Write 1 Done Sent Done false
9897 3358 Acked Flush 0 Done Sent Done false
9898 3359 Acked Read 8 Done Sent Done false
9899 3360 Acked Write 9 Done Sent Done false
9900 3361 Acked Read 9 Done Sent Done false
9901 3362 Acked Read 10 Done Sent Done false
9902 3363 Acked Read 7 Done Sent Done false
9903 3364 Acked Read 5 Done Sent Done false
9904 3365 Acked Read 7 Done Sent Done false
9905 3366 Acked Read 4 Done Sent Done false
9906 3367 Acked Write 2 Done Sent Done false
9907 3368 Acked Write 10 Done Sent Done false
9908 3369 Acked Write 7 Done Sent Done false
9909 3370 Acked Read 5 Done Sent Done false
9910 3371 Acked Write 9 Done Sent Done false
9911 3372 Acked Write 2 Done Sent Done false
9912 3373 Acked Write 3 Done Sent Done false
9913 3374 Acked Write 7 Done Sent Done false
9914 3375 Acked Read 7 Done Sent Done false
9915 3376 Acked Write 10 Done Sent Done false
9916 3377 Acked Write 10 Done Sent Done false
9917 3378 Acked Read 1 Done Sent Done false
9918 3379 Acked Read 7 Done Sent Done false
9919 3380 Acked Write 9 Done Sent Done false
9920 3381 Acked Read 10 Done Sent Done false
9921 3382 Acked Read 2 Done Sent Done false
9922 3383 Acked Read 5 Done Sent Done false
9923 3384 Acked Read 6 Done Sent Done false
9924 3385 Acked Write 1 Done Sent Done false
9925 3386 Acked Write 5 Done Sent Done false
9926 3387 Acked Write 10 Done Sent Done false
9927 3388 Acked Write 3 Done Sent Done false
9928 3389 Acked Read 6 Done Sent Done false
9929 3390 Acked Write 2 Done Sent Done false
9930 3391 Acked Write 9 Done Sent Done false
9931 3392 Acked Write 8 Done Sent Done false
9932 3393 Acked Read 5 Done Sent Done false
9933 3394 Acked Read 9 Done Sent Done false
9934 3395 Acked Read 8 Done Sent Done false
9935 3396 Acked Read 10 Done Sent Done false
9936 3397 Acked Write 4 Done Sent Done false
9937 3398 Acked Read 9 Done Sent Done false
9938 3399 Acked Write 6 Done Sent Done false
9939 3400 Acked Write 8 Done Sent Done false
9940 3401 Acked Write 9 Done Sent Done false
9941 3402 Acked Read 9 Done Sent Done false
9942 3403 Acked Write 4 Done Sent Done false
9943 3404 Acked Read 10 Done Sent Done false
9944 3405 Acked Write 7 Done Sent Done false
9945 3406 Acked Read 10 Done Sent Done false
9946 3407 Acked Read 5 Done Sent Done false
9947 3408 Acked Read 2 Done Sent Done false
9948 3409 Acked Flush 0 Done Sent Done false
9949 3410 Acked Write 4 Done Sent Done false
9950 3411 Acked Write 3 Done Sent Done false
9951 3412 Acked Read 3 Done Sent Done false
9952 3413 Acked Read 9 Done Sent Done false
9953 3414 Acked Write 1 Done Sent Done false
9954 STATES DS:0 DS:1 DS:2 TOTAL
9955 Sent 0 301 0 301
9956 Done 301 0 301 602
9957 Skipped 0 0 0 0
9958 Error 0 0 0 0
9959 Last Flush: Some(JobId(3409)) Some(JobId(3113)) Some(JobId(3409))
9960 Downstairs last five completed: 3113 3112 3111 3110 3109
9961 Upstairs last five completed: 3414 3413 3412 3411 3410
9962 CLIENT: Up:0 ds:301 act:2
99632025-01-29T05:12:50.209ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99642025-01-29T05:12:51.124ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99652025-01-29T05:12:52.125ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99662025-01-29T05:12:53.125ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99672025-01-29T05:12:53.214ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99682025-01-29T05:12:53.214ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63025) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99692025-01-29T05:12:53.215ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99702025-01-29T05:12:53.215ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99712025-01-29T05:12:53.215ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99722025-01-29T05:12:53.215ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99732025-01-29T05:12:53.215ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99742025-01-29T05:12:53.215ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
99752025-01-29T05:12:53.215ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
9976 ----------------------------------------------------------------
9977 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
9978 States: Active Active Active
9979 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9980 3410 Acked Write 4 Done Done Done true
9981 3411 Acked Write 3 Done Done Done true
9982 3412 Acked Read 3 Done Done Done true
9983 3413 Acked Read 9 Done Done Done true
9984 3414 Acked Write 1 Done Done Done true
9985 STATES DS:0 DS:1 DS:2 TOTAL
9986 Sent 0 0 0 0
9987 Done 5 5 5 15
9988 Skipped 0 0 0 0
9989 Error 0 0 0 0
9990 Last Flush: Some(JobId(3409)) Some(JobId(3409)) Some(JobId(3409))
9991 Downstairs last five completed: 3409 3408 3407 3406 3405
9992 Upstairs last five completed: 3414 3413 3412 3411 3410
9993 CLIENT: Up:0 ds:5 act:3
9994 ----------------------------------------------------------------
9995 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9996 Downstairs last five completed: 3415 3414 3413 3412 3411
9997 Upstairs last five completed: 3415 3414 3413 3412 3411
9998 CLIENT: Up:0 ds:0 act:3
9999 Replay: All jobs finished
10000 Main task has work to do, go find it
10001 got dsc Stop(0)
10002 stop 0
10003 Jan 29 05:12:57.466 INFO request completed, [8810] Got stop action so:false kr:false
10004 latency_us: 349, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7e5221f8-fc19-4385-a117-32c6cb3dd51e, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
100052025-01-29T05:12:57.467ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
100062025-01-29T05:12:57.467ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10007 Jan 29 05:12:57.467 {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:12:57.467488348Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
10008 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/0, method: GET, req_id: 99e08e06-ffbe-4e09-9c8a-a7708bd02e15, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
100092025-01-29T05:12:57.467ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10010 [8810] Exited with: ExitStatus(unix_wait_status(9))
10011 [8810][0] reports Exit
10012 Jan 29 05:13:01.470 INFO request completed, latency_us: 314, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0e11178a-a423-40d5-a75b-909cdb3213e9, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
10013 001/300 Read block 656 len 4096
10014 002/300 Flush
10015 003/300 Read block 843 len 16384
10016 004/300 Write block 2442 len 12288 data: 4 4 4
10017 005/300 Read block 2326 len 40960
10018 006/300 Read block 1475 len 40960
10019 007/300 Read block 686 len 12288
10020 008/300 Write block 1508 len 8192 data: 2 2
10021 009/300 Write block 2014 len 28672 data: 4 4 3 2 2 2 2
10022 010/300 Read block 4544 len 40960
10023 011/300 Write block 375 len 4096 data: 3
10024 012/300 Write block 4585 len 8192 data: 4 2
10025 013/300 Write block 1581 len 24576 data: 2 2 3 3 3 4
10026 014/300 Write block 1621 len 16384 data: 3 4 4 5
10027 015/300 Read block 3836 len 12288
10028 016/300 Read block 3081 len 24576
10029 017/300 Write block 454 len 32768 data: 2 2 2 2 2 2 2 2
10030 018/300 Write block 2462 len 8192 data: 2 2
10031 019/300 Read block 448 len 40960
10032 020/300 Read block 3779 len 28672
10033 021/300 Read block 3384 len 40960
10034 022/300 Write block 3928 len 4096 data: 2
10035 023/300 Write block 3428 len 12288 data: 2 3 4
10036 024/300 Read block 965 len 28672
10037 025/300 Write block 4686 len 28672 data: 4 3 3 2 2 2 3
10038 026/300 Read block 2929 len 40960
10039 027/300 Read block 2479 len 24576
10040 028/300 Read block 561 len 40960
10041 029/300 Write block 1940 len 4096 data: 4
10042 030/300 Read block 807 len 4096
10043 031/300 Read block 2952 len 16384
10044 032/300 Read block 2970 len 36864
10045 033/300 Read block 4137 len 8192
10046 034/300 Read block 3143 len 32768
10047 035/300 Write block 650 len 16384 data: 3 3 3 3
10048 036/300 Read block 68 len 40960
10049 037/300 Read block 3161 len 20480
10050 038/300 Read block 4515 len 12288
10051 039/300 Read block 4960 len 24576
10052 040/300 Read block 4121 len 36864
10053 041/300 Read block 3507 len 40960
10054 042/300 Read block 128 len 12288
10055 043/300 Read block 3521 len 12288
10056 044/300 Read block 173 len 4096
10057 045/300 Read block 369 len 20480
10058 046/300 Write block 4391 len 28672 data: 5 6 6 6 7 5 3
10059 047/300 Write block 582 len 16384 data: 2 2 2 2
10060 048/300 Write block 2369 len 36864 data: 2 2 2 2 2 2 2 2 3
10061 049/300 Read block 1634 len 16384
10062 050/300 Flush
10063 051/300 Read block 1209 len 32768
10064 052/300 Flush
10065 053/300 Write block 2961 len 12288 data: 4 4 4
10066 054/300 Read block 4118 len 20480
10067 055/300 Write block 4597 len 16384 data: 3 3 4 5
10068 056/300 Read block 359 len 4096
10069 057/300 Read block 4694 len 16384
10070 058/300 Write block 4182 len 40960 data: 3 3 2 2 2 2 3 3 3 3
10071 059/300 Read block 3285 len 32768
10072 060/300 Read block 945 len 8192
10073 061/300 Read block 2875 len 16384
10074 062/300 Write block 1421 len 32768 data: 4 3 3 3 3 3 3 3
10075 063/300 Write block 2136 len 4096 data: 3
10076 064/300 Write block 542 len 32768 data: 3 3 3 3 3 3 3 3
10077 065/300 Read block 1892 len 16384
10078 066/300 Read block 1858 len 4096
10079 067/300 Read block 4359 len 40960
10080 068/300 Write block 322 len 4096 data: 4
10081 069/300 Read block 3125 len 16384
10082 070/300 Read block 3936 len 28672
10083 071/300 Write block 2756 len 32768 data: 6 5 5 6 6 5 5 3
10084 072/300 Read block 3459 len 32768
10085 073/300 Write block 224 len 12288 data: 3 3 3
10086 074/300 Read block 1216 len 28672
10087 075/300 Write block 862 len 36864 data: 4 3 3 3 3 3 3 3 3
10088 076/300 Write block 3322 len 32768 data: 3 3 3 2 2 2 2 2
10089 077/300 Write block 4886 len 40960 data: 3 3 3 3 3 2 2 2 2 3
10090 078/300 Read block 2262 len 12288
10091 079/300 Read block 4016 len 4096
10092 080/300 Read block 1166 len 8192
10093 081/300 Write block 2413 len 24576 data: 4 4 3 4 3 3
10094 082/300 Write block 1278 len 36864 data: 2 2 2 3 3 4 4 4 4
10095 083/300 Read block 4843 len 4096
10096 084/300 Read block 4501 len 20480
10097 085/300 Write block 3749 len 8192 data: 3 3
10098 086/300 Read block 2654 len 8192
10099 087/300 Write block 2837 len 28672 data: 4 4 4 4 3 3 3
10100 088/300 Read block 3009 len 8192
10101 089/300 Read block 3847 len 28672
10102 090/300 Read block 206 len 24576
10103 091/300 Read block 1787 len 20480
10104 092/300 Read block 2227 len 12288
10105 093/300 Write block 3719 len 8192 data: 3 3
10106 094/300 Write block 4722 len 12288 data: 2 2 3
10107 095/300 Read block 1573 len 24576
10108 096/300 Write block 1881 len 36864 data: 4 4 3 4 4 4 4 4 5
10109 097/300 Write block 1659 len 8192 data: 2 2
10110 098/300 Flush
10111 099/300 Read block 2737 len 28672
10112 100/300 Read block 301 len 8192
10113 101/300 Write block 4874 len 8192 data: 3 3
10114 102/300 Write block 1695 len 20480 data: 3 3 3 3 3
10115 103/300 Write block 54 len 24576 data: 3 3 3 2 2 3
10116 104/300 Read block 1226 len 40960
10117 105/300 Read block 942 len 8192
10118 106/300 Read block 4838 len 36864
10119 107/300 Write block 2567 len 16384 data: 2 2 3 3
10120 108/300 Read block 1242 len 20480
10121 109/300 Write block 3345 len 28672 data: 2 2 2 2 2 2 3
10122 110/300 Write block 2276 len 8192 data: 2 2
10123 111/300 Read block 938 len 12288
10124 112/300 Read block 2906 len 4096
10125 113/300 Write block 1255 len 8192 data: 3 4
10126 114/300 Write block 774 len 32768 data: 2 2 2 2 4 4 4 4
10127 115/300 Write block 4741 len 32768 data: 4 4 5 5 4 3 3 3
10128 116/300 Read block 1405 len 12288
10129 117/300 Flush
10130 118/300 Write block 1076 len 32768 data: 2 2 2 2 2 2 2 4
10131 119/300 Read block 2337 len 28672
10132 120/300 Read block 1664 len 36864
10133 121/300 Write block 3936 len 40960 data: 3 3 3 3 3 3 3 3 4 4
10134 122/300 Flush
10135 123/300 Read block 1308 len 4096
10136 124/300 Write block 4193 len 4096 data: 2
10137 125/300 Read block 2672 len 8192
10138 126/300 Write block 1911 len 12288 data: 3 3 3
10139 127/300 Write block 110 len 40960 data: 3 4 4 3 3 4 3 3 4 4
10140 128/300 Read block 4384 len 20480
10141 129/300 Write block 2663 len 28672 data: 5 4 4 3 2 2 3
10142 130/300 Read block 1859 len 36864
10143 131/300 Write block 1621 len 16384 data: 4 5 5 6
10144 132/300 Write block 390 len 32768 data: 3 3 2 3 3 4 4 4
10145 133/300 Write block 1862 len 32768 data: 4 5 4 4 4 4 4 2
10146 134/300 Read block 1392 len 4096
10147 135/300 Read block 1836 len 16384
10148 136/300 Write block 1388 len 4096 data: 3
10149 137/300 Write block 425 len 20480 data: 2 3 3 3 3
10150 138/300 Flush
10151 139/300 Write block 2727 len 36864 data: 2 2 2 2 3 3 4 5 5
10152 140/300 Flush
10153 141/300 Read block 922 len 16384
10154 142/300 Read block 3020 len 8192
10155 143/300 Write block 4148 len 36864 data: 3 4 3 2 2 2 2 2 2
10156 144/300 Write block 1625 len 20480 data: 5 5 5 5 4
10157 145/300 Read block 3012 len 32768
10158 146/300 Write block 1316 len 12288 data: 3 3 2
10159 147/300 Write block 2521 len 20480 data: 2 2 3 3 3
10160 148/300 Read block 891 len 12288
10161 149/300 Read block 2407 len 8192
10162 150/300 Read block 1568 len 40960
10163 151/300 Read block 1225 len 40960
10164 152/300 Read block 81 len 16384
10165 153/300 Read block 2926 len 8192
10166 154/300 Read block 4240 len 24576
10167 155/300 Write block 4768 len 32768 data: 2 2 4 3 3 4 4 4
10168 156/300 Read block 1828 len 16384
10169 157/300 Write block 4709 len 32768 data: 2 2 2 2 2 2 2 2
10170 158/300 Read block 4162 len 28672
10171 159/300 Read block 4404 len 8192
10172 160/300 Read block 4541 len 40960
10173 161/300 Read block 765 len 8192
10174 162/300 Read block 2399 len 20480
10175 163/300 Flush
10176 164/300 Write block 127 len 28672 data: 3 2 2 2 2 2 2
10177 165/300 Read block 1778 len 4096
10178 166/300 Read block 2678 len 20480
10179 167/300 Write block 1296 len 36864 data: 3 3 3 6 6 6 6 5 4
10180 168/300 Read block 1685 len 20480
10181 169/300 Read block 3783 len 16384
10182 170/300 Read block 3253 len 8192
10183 171/300 Read block 3076 len 8192
10184 172/300 Read block 761 len 16384
10185 173/300 Read block 2155 len 40960
10186 174/300 Read block 1045 len 4096
10187 175/300 Flush
10188 176/300 Read block 3605 len 8192
10189 177/300 Read block 1639 len 4096
10190 178/300 Read block 1153 len 12288
10191 179/300 Read block 3933 len 32768
10192 180/300 Read block 1750 len 20480
10193 181/300 Read block 4895 len 8192
10194 182/300 Read block 2914 len 16384
10195 183/300 Write block 945 len 36864 data: 3 4 4 4 4 4 4 5 5
10196 184/300 Write block 4575 len 8192 data: 3 2
10197 185/300 Read block 1863 len 16384
10198 186/300 Flush
10199 187/300 Read block 3683 len 40960
10200 188/300 Read block 3535 len 40960
10201 189/300 Read block 3925 len 20480
10202 190/300 Write block 843 len 24576 data: 3 3 3 3 4 4
10203 191/300 Read block 76 len 32768
10204 192/300 Flush
10205 193/300 Write block 2615 len 4096 data: 3
10206 194/300 Write block 3486 len 4096 data: 2
10207 195/300 Write block 3418 len 40960 data: 4 4 5 5 5 5 3 2 2 2
10208 196/300 Flush
10209 197/300 Read block 3212 len 32768
10210 198/300 Write block 2309 len 40960 data: 2 3 3 3 3 4 4 4 4 3
10211 199/300 Read block 4245 len 36864
10212 200/300 Read block 4805 len 36864
10213 201/300 Flush
10214 202/300 Write block 253 len 12288 data: 2 2 4
10215 203/300 Write block 1249 len 20480 data: 3 3 4 3 3
10216 204/300 Read block 4470 len 24576
10217 205/300 Read block 4809 len 32768
10218 206/300 Read block 4392 len 28672
10219 207/300 Read block 15 len 20480
10220 208/300 Read block 1208 len 16384
10221 209/300 Read block 3803 len 20480
10222 210/300 Flush
10223 211/300 Read block 1997 len 20480
10224 212/300 Flush
10225 213/300 Read block 4681 len 36864
10226 214/300 Read block 1799 len 4096
10227 215/300 Read block 4984 len 8192
10228 216/300 Flush
10229 217/300 Read block 534 len 4096
10230 218/300 Read block 4924 len 4096
10231 219/300 Flush
10232 220/300 Write block 1042 len 20480 data: 5 5 5 5 4
10233 221/300 Read block 1366 len 12288
10234 222/300 Flush
10235 223/300 Write block 572 len 28672 data: 2 2 2 2 2 3 2
10236 224/300 Write block 454 len 28672 data: 3 3 3 3 3 3 3
10237 225/300 Write block 1143 len 4096 data: 2
10238 226/300 Write block 1233 len 16384 data: 2 2 2 2
10239 227/300 Write block 4391 len 16384 data: 6 7 7 7
10240 228/300 Flush
10241 229/300 Write block 4333 len 8192 data: 2 2
10242 230/300 Read block 3915 len 28672
10243 231/300 Read block 1563 len 4096
10244 232/300 Write block 1361 len 20480 data: 3 2 2 2 2
10245 233/300 Write block 4031 len 16384 data: 2 2 2 2
10246 234/300 Read block 630 len 8192
10247 235/300 Read block 3087 len 20480
10248 236/300 Read block 4506 len 8192
10249 237/300 Read block 436 len 20480
10250 238/300 Read block 4819 len 28672
10251 239/300 Write block 767 len 8192 data: 2 2
10252 240/300 Write block 3684 len 16384 data: 3 3 2 2
10253 241/300 Write block 3177 len 24576 data: 4 4 3 3 3 3
10254 242/300 Flush
10255 243/300 Read block 287 len 20480
10256 244/300 Write block 3642 len 12288 data: 4 3 3
10257 245/300 Read block 3858 len 40960
10258 246/300 Write block 1626 len 32768 data: 6 6 6 5 4 4 5 4
10259 247/300 Write block 1087 len 8192 data: 3 2
10260 248/300 Read block 4987 len 24576
10261 249/300 Read block 3263 len 24576
10262 250/300 Read block 1198 len 24576
10263 251/300 Write block 3481 len 16384 data: 4 4 4 3
10264 252/300 Write block 1033 len 12288 data: 3 2 3
10265 253/300 Write block 548 len 28672 data: 4 4 4 3 3 2 2
10266 254/300 Flush
10267 255/300 Read block 1827 len 8192
10268 256/300 Write block 751 len 24576 data: 4 4 4 4 5 4
10269 257/300 Write block 1358 len 40960 data: 3 3 3 4 3 3 3 3 2 2
10270 258/300 Read block 2318 len 8192
10271 259/300 Read block 2284 len 36864
10272 260/300 Read block 1957 len 24576
10273 261/300 Read block 2260 len 4096
10274 262/300 Read block 1221 len 20480
10275 263/300 Read block 2394 len 36864
10276 264/300 Read block 2486 len 16384
10277 265/300 Write block 2780 len 16384 data: 5 4 4 4
10278 266/300 Write block 2793 len 36864 data: 4 3 3 3 2 2 2 2 2
10279 267/300 Read block 4539 len 36864
10280 268/300 Read block 1838 len 28672
10281 269/300 Read block 2591 len 20480
10282 270/300 Write block 2020 len 4096 data: 3
10283 271/300 Read block 1083 len 4096
10284 272/300 Write block 1485 len 4096 data: 3
10285 273/300 Write block 2635 len 40960 data: 3 3 4 5 5 3 3 3 4 4
10286 274/300 Read block 592 len 8192
10287 275/300 Write block 2810 len 12288 data: 4 4 5
10288 276/300 Write block 4336 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10289 277/300 Read block 2497 len 8192
10290 278/300 Read block 131 len 28672
10291 279/300 Read block 2551 len 20480
10292 280/300 Write block 49 len 16384 data: 4 4 4 3
10293 281/300 Write block 3458 len 24576 data: 2 3 4 4 4 4
10294 282/300 Write block 3380 len 32768 data: 3 3 4 3 4 3 2 2
10295 283/300 Read block 4881 len 20480
10296 284/300 Flush
10297 285/300 Flush
10298 286/300 Read block 514 len 20480
10299 287/300 Write block 2382 len 12288 data: 6 4 4
10300 288/300 Read block 1532 len 40960
10301 289/300 Write block 2710 len 8192 data: 2 2
10302 290/300 Write block 3699 len 16384 data: 3 3 2 3
10303 291/300 Read block 4430 len 28672
10304 292/300 Write block 2368 len 40960 data: 2 3 3 3 3 3 3 3 3 4
10305 293/300 Write block 4701 len 28672 data: 4 4 3 3 3 2 2
10306 294/300 Write block 3605 len 40960 data: 2 2 2 3 3 3 3 3 3 3
10307 295/300 Write block 838 len 28672 data: 3 3 3 2 3 4 4
10308 296/300 Read block 1728 len 40960
10309 297/300 Read block 2015 len 20480
10310 298/300 Write block 744 len 28672 data: 6 7 4 4 3 5 5
10311 299/300 Read block 2750 len 16384
10312 300/300 Read block 4294 len 20480
10313 301/300 Read block 4886 len 32768
10314 Main task has work to do, go find it
10315 got dsc Start(0)
10316 start 0
10317 Jan 29 05:13:01.711 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/0, method: POST, req_id: 020583c6-a8ad-4a83-a81e-823006915ffd, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
10318 Replay: started 0, returned:Ok(())
10319 ----------------------------------------------------------------
10320 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10321 States: Offline Active Active
10322 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10323 3416 Acked Read 1 Sent Done Done false
10324 3417 Acked Flush 0 Sent Done Done false
10325 3418 Acked Read 4 Sent Done Done false
10326 3419 Acked Write 3 Sent Done Done false
10327 3420 Acked Read 10 Sent Done Done false
10328 3421 Acked Read 10 Sent Done Done false
10329 3422 Acked Read 3 Sent Done Done false
10330 3423 Acked Write 2 Sent Done Done false
10331 3424 Acked Write 7 Sent Done Done false
10332 3425 Acked Read 10 Sent Done Done false
10333 3426 Acked Write 1 Sent Done Done false
10334 3427 Acked Write 2 Sent Done Done false
10335 3428 Acked Write 6 Sent Done Done false
10336 3429 Acked Write 4 Sent Done Done false
10337 3430 Acked Read 3 Sent Done Done false
10338 3431 Acked Read 6 Sent Done Done false
10339 3432 Acked Write 8 Sent Done Done false
10340 3433 Acked Write 2 Sent Done Done false
10341 3434 Acked Read 10 Sent Done Done false
10342 3435 Acked Read 7 Sent Done Done false
10343 3436 Acked Read 10 Sent Done Done false
10344 3437 Acked Write 1 Sent Done Done false
10345 3438 Acked Write 3 Sent Done Done false
10346 3439 Acked Read 7 Sent Done Done false
10347 3440 Acked Write 7 Sent Done Done false
10348 3441 Acked Read 10 Sent Done Done false
10349 3442 Acked Read 6 Sent Done Done false
10350 3443 Acked Read 10 Sent Done Done false
10351 3444 Acked Write 1 Sent Done Done false
10352 3445 Acked Read 1 Sent Done Done false
10353 3446 Acked Read 4 Sent Done Done false
10354 3447 Acked Read 9 Sent Done Done false
10355 3448 Acked Read 2 Sent Done Done false
10356 3449 Acked Read 8 Sent Done Done false
10357 3450 Acked Write 4 Sent Done Done false
10358 3451 Acked Read 10 Sent Done Done false
10359 3452 Acked Read 5 Sent Done Done false
10360 3453 Acked Read 3 Sent Done Done false
10361 3454 Acked Read 6 Sent Done Done false
10362 3455 Acked Read 9 Sent Done Done false
10363 3456 Acked Read 10 Sent Done Done false
10364 3457 Acked Read 3 Sent Done Done false
10365 3458 Acked Read 3 Sent Done Done false
10366 3459 Acked Read 1 Sent Done Done false
10367 3460 Acked Read 5 Sent Done Done false
10368 3461 Acked Write 7 Sent Done Done false
10369 3462 Acked Write 4 Sent Done Done false
10370 3463 Acked Write 9 Sent Done Done false
10371 3464 Acked Read 4 Sent Done Done false
10372 3465 Acked Flush 0 Sent Done Done false
10373 3466 Acked Read 8 Sent Done Done false
10374 3467 Acked Flush 0 Sent Done Done false
10375 3468 Acked Write 3 Sent Done Done false
10376 3469 Acked Read 5 Sent Done Done false
10377 3470 Acked Write 4 Sent Done Done false
10378 3471 Acked Read 1 Sent Done Done false
10379 3472 Acked Read 4 Sent Done Done false
10380 3473 Acked Write 10 Sent Done Done false
10381 3474 Acked Read 8 Sent Done Done false
10382 3475 Acked Read 2 Sent Done Done false
10383 3476 Acked Read 4 Sent Done Done false
10384 3477 Acked Write 8 Sent Done Done false
10385 3478 Acked Write 1 Sent Done Done false
10386 3479 Acked Write 8 Sent Done Done false
10387 3480 Acked Read 4 Sent Done Done false
10388 3481 Acked Read 1 Sent Done Done false
10389 3482 Acked Read 10 Sent Done Done false
10390 3483 Acked Write 1 Sent Done Done false
10391 3484 Acked Read 4 Sent Done Done false
10392 3485 Acked Read 7 Sent Done Done false
10393 3486 Acked Write 8 Sent Done Done false
10394 3487 Acked Read 8 Sent Done Done false
10395 3488 Acked Write 3 Sent Done Done false
10396 3489 Acked Read 7 Sent Done Done false
10397 3490 Acked Write 9 Sent Done Done false
10398 3491 Acked Write 8 Sent Done Done false
10399 3492 Acked Write 10 Sent Done Done false
10400 3493 Acked Read 3 Sent Done Done false
10401 3494 Acked Read 1 Sent Done Done false
10402 3495 Acked Read 2 Sent Done Done false
10403 3496 Acked Write 6 Sent Done Done false
10404 3497 Acked Write 9 Sent Done Done false
10405 3498 Acked Read 1 Sent Done Done false
10406 3499 Acked Read 5 Sent Done Done false
10407 3500 Acked Write 2 Sent Done Done false
10408 3501 Acked Read 2 Sent Done Done false
10409 3502 Acked Write 7 Sent Done Done false
10410 3503 Acked Read 2 Sent Done Done false
10411 3504 Acked Read 7 Sent Done Done false
10412 3505 Acked Read 6 Sent Done Done false
10413 3506 Acked Read 5 Sent Done Done false
10414 3507 Acked Read 3 Sent Done Done false
10415 3508 Acked Write 2 Sent Done Done false
10416 3509 Acked Write 3 Sent Done Done false
10417 3510 Acked Read 6 Sent Done Done false
10418 3511 Acked Write 9 Sent Done Done false
10419 3512 Acked Write 2 Sent Done Done false
10420 3513 Acked Flush 0 Sent Done Done false
10421 3514 Acked Read 7 Sent Done Done false
10422 3515 Acked Read 2 Sent Done Done false
10423 3516 Acked Write 2 Sent Done Done false
10424 3517 Acked Write 5 Sent Done Done false
10425 3518 Acked Write 6 Sent Done Done false
10426 3519 Acked Read 10 Sent Done Done false
10427 3520 Acked Read 2 Sent Done Done false
10428 3521 Acked Read 9 Sent Done Done false
10429 3522 Acked Write 4 Sent Done Done false
10430 3523 Acked Read 5 Sent Done Done false
10431 3524 Acked Write 7 Sent Done Done false
10432 3525 Acked Write 2 Sent Done Done false
10433 3526 Acked Read 3 Sent Done Done false
10434 3527 Acked Read 1 Sent Done Done false
10435 3528 Acked Write 2 Sent Done Done false
10436 3529 Acked Write 8 Sent Done Done false
10437 3530 Acked Write 8 Sent Done Done false
10438 3531 Acked Read 3 Sent Done Done false
10439 3532 Acked Flush 0 Sent Done Done false
10440 3533 Acked Write 8 Sent Done Done false
10441 3534 Acked Read 7 Sent Done Done false
10442 3535 Acked Read 9 Sent Done Done false
10443 3536 Acked Write 10 Sent Done Done false
10444 3537 Acked Flush 0 Sent Done Done false
10445 3538 Acked Read 1 Sent Done Done false
10446 3539 Acked Write 1 Sent Done Done false
10447 3540 Acked Read 2 Sent Done Done false
10448 3541 Acked Write 3 Sent Done Done false
10449 3542 Acked Write 10 Sent Done Done false
10450 3543 Acked Read 5 Sent Done Done false
10451 3544 Acked Write 7 Sent Done Done false
10452 3545 Acked Read 9 Sent Done Done false
10453 3546 Acked Write 4 Sent Done Done false
10454 3547 Acked Write 8 Sent Done Done false
10455 3548 Acked Write 8 Sent Done Done false
10456 3549 Acked Read 1 Sent Done Done false
10457 3550 Acked Read 4 Sent Done Done false
10458 3551 Acked Write 1 Sent Done Done false
10459 3552 Acked Write 5 Sent Done Done false
10460 3553 Acked Flush 0 Sent Done Done false
10461 3554 Acked Write 9 Sent Done Done false
10462 3555 Acked Flush 0 Sent Done Done false
10463 3556 Acked Read 4 Sent Done Done false
10464 3557 Acked Read 2 Sent Done Done false
10465 3558 Acked Write 9 Sent Done Done false
10466 3559 Acked Write 5 Sent Done Done false
10467 3560 Acked Read 8 Sent Done Done false
10468 3561 Acked Write 3 Sent Done Done false
10469 3562 Acked Write 5 Sent Done Done false
10470 3563 Acked Read 3 Sent Done Done false
10471 3564 Acked Read 2 Sent Done Done false
10472 3565 Acked Read 10 Sent Done Done false
10473 3566 Acked Read 10 Sent Done Done false
10474 3567 Acked Read 4 Sent Done Done false
10475 3568 Acked Read 2 Sent Done Done false
10476 3569 Acked Read 6 Sent Done Done false
10477 3570 Acked Write 8 Sent Done Done false
10478 3571 Acked Read 4 Sent Done Done false
10479 3572 Acked Write 8 Sent Done Done false
10480 3573 Acked Read 7 Sent Done Done false
10481 3574 Acked Read 2 Sent Done Done false
10482 3575 Acked Read 10 Sent Done Done false
10483 3576 Acked Read 2 Sent Done Done false
10484 3577 Acked Read 5 Sent Done Done false
10485 3578 Acked Flush 0 Sent Done Done false
10486 3579 Acked Write 7 Sent Done Done false
10487 3580 Acked Read 1 Sent Done Done false
10488 3581 Acked Read 5 Sent Done Done false
10489 3582 Acked Write 9 Sent Done Done false
10490 3583 Acked Read 5 Sent Done Done false
10491 3584 Acked Read 4 Sent Done Done false
10492 3585 Acked Read 2 Sent Done Done false
10493 3586 Acked Read 2 Sent Done Done false
10494 3587 Acked Read 4 Sent Done Done false
10495 3588 Acked Read 10 Sent Done Done false
10496 3589 Acked Read 1 Sent Done Done false
10497 3590 Acked Flush 0 Sent Done Done false
10498 3591 Acked Read 2 Sent Done Done false
10499 3592 Acked Read 1 Sent Done Done false
10500 3593 Acked Read 3 Sent Done Done false
10501 3594 Acked Read 8 Sent Done Done false
10502 3595 Acked Read 5 Sent Done Done false
10503 3596 Acked Read 2 Sent Done Done false
10504 3597 Acked Read 4 Sent Done Done false
10505 3598 Acked Write 9 Sent Done Done false
10506 3599 Acked Write 2 Sent Done Done false
10507 3600 Acked Read 4 Sent Done Done false
10508 3601 Acked Flush 0 Sent Done Done false
10509 3602 Acked Read 10 Sent Done Done false
10510 3603 Acked Read 10 Sent Done Done false
10511 3604 Acked Read 5 Sent Done Done false
10512 3605 Acked Write 6 Sent Done Done false
10513 3606 Acked Read 8 Sent Done Done false
10514 3607 Acked Flush 0 Sent Done Done false
10515 3608 Acked Write 1 Sent Done Done false
10516 3609 Acked Write 1 Sent Done Done false
10517 3610 Acked Write 10 Sent Done Done false
10518 3611 Acked Flush 0 Sent Done Done false
10519 3612 Acked Read 8 Sent Done Done false
10520 3613 Acked Write 10 Sent Done Done false
10521 3614 Acked Read 9 Sent Done Done false
10522 3615 Acked Read 9 Sent Done Done false
10523 3616 Acked Flush 0 Sent Done Done false
10524 3617 Acked Write 3 Sent Done Done false
10525 3618 Acked Write 5 Sent Done Done false
10526 3619 Acked Read 6 Sent Done Done false
10527 3620 Acked Read 8 Sent Done Done false
10528 3621 Acked Read 7 Sent Done Done false
10529 3622 Acked Read 5 Sent Done Done false
10530 3623 Acked Read 4 Sent Done Done false
10531 3624 Acked Read 5 Sent Done Done false
10532 3625 Acked Flush 0 Sent Done Done false
10533 3626 Acked Read 5 Sent Done Done false
10534 3627 Acked Flush 0 Sent Done Done false
10535 3628 Acked Read 9 Sent Done Done false
10536 3629 Acked Read 1 Sent Done Done false
10537 3630 Acked Read 2 Sent Done Done false
10538 3631 Acked Flush 0 Sent Done Done false
10539 3632 Acked Read 1 Sent Done Done false
10540 3633 Acked Read 1 Sent Done Done false
10541 3634 Acked Flush 0 Sent Done Done false
10542 3635 Acked Write 5 Sent Done Done false
10543 3636 Acked Read 3 Sent Done Done false
10544 3637 Acked Flush 0 Sent Done Done false
10545 3638 Acked Write 7 Sent Done Done false
10546 3639 Acked Write 7 Sent Done Done false
10547 3640 Acked Write 1 Sent Done Done false
10548 3641 Acked Write 4 Sent Done Done false
10549 3642 Acked Write 4 Sent Done Done false
10550 3643 Acked Flush 0 Sent Done Done false
10551 3644 Acked Write 2 Sent Done Done false
10552 3645 Acked Read 7 Sent Done Done false
10553 3646 Acked Read 1 Sent Done Done false
10554 3647 Acked Write 5 Sent Done Done false
10555 3648 Acked Write 4 Sent Done Done false
10556 3649 Acked Read 2 Sent Done Done false
10557 3650 Acked Read 5 Sent Done Done false
10558 3651 Acked Read 2 Sent Done Done false
10559 3652 Acked Read 5 Sent Done Done false
10560 3653 Acked Read 7 Sent Done Done false
10561 3654 Acked Write 2 Sent Done Done false
10562 3655 Acked Write 4 Sent Done Done false
10563 3656 Acked Write 6 Sent Done Done false
10564 3657 Acked Flush 0 Sent Done Done false
10565 3658 Acked Read 5 Sent Done Done false
10566 3659 Acked Write 3 Sent Done Done false
10567 3660 Acked Read 10 Sent Done Done false
10568 3661 Acked Write 8 Sent Done Done false
10569 3662 Acked Write 2 Sent Done Done false
10570 3663 Acked Read 6 Sent Done Done false
10571 3664 Acked Read 6 Sent Done Done false
10572 3665 Acked Read 6 Sent Done Done false
10573 3666 Acked Write 4 Sent Done Done false
10574 3667 Acked Write 3 Sent Done Done false
10575 3668 Acked Write 7 Sent Done Done false
10576 3669 Acked Flush 0 Sent Done Done false
10577 3670 Acked Read 2 Sent Done Done false
10578 3671 Acked Write 6 Sent Done Done false
10579 3672 Acked Write 10 Sent Done Done false
10580 3673 Acked Read 2 Sent Done Done false
10581 3674 Acked Read 9 Sent Done Done false
10582 3675 Acked Read 6 Sent Done Done false
10583 3676 Acked Read 1 Sent Done Done false
10584 3677 Acked Read 5 Sent Done Done false
10585 3678 Acked Read 9 Sent Done Done false
10586 3679 Acked Read 4 Sent Done Done false
10587 3680 Acked Write 4 Sent Done Done false
10588 3681 Acked Write 9 Sent Done Done false
10589 3682 Acked Read 9 Sent Done Done false
10590 3683 Acked Read 7 Sent Done Done false
10591 3684 Acked Read 5 Sent Done Done false
10592 3685 Acked Write 1 Sent Done Done false
10593 3686 Acked Read 1 Sent Done Done false
10594 3687 Acked Write 1 Sent Done Done false
10595 3688 Acked Write 10 Sent Done Done false
10596 3689 Acked Read 2 Sent Done Done false
10597 3690 Acked Write 3 Sent Done Done false
10598 3691 Acked Write 10 Sent Done Done false
10599 3692 Acked Read 2 Sent Done Done false
10600 3693 Acked Read 7 Sent Done Done false
10601 3694 Acked Read 5 Sent Done Done false
10602 3695 Acked Write 4 Sent Done Done false
10603 3696 Acked Write 6 Sent Done Done false
10604 3697 Acked Write 8 Sent Done Done false
10605 3698 Acked Read 5 Sent Done Done false
10606 3699 Acked Flush 0 Sent Done Done false
10607 3700 Acked Flush 0 Sent Done Done false
10608 3701 Acked Read 5 Sent Done Done false
10609 3702 Acked Write 3 Sent Done Done false
10610 3703 Acked Read 10 Sent Done Done false
10611 3704 Acked Write 2 Sent Done Done false
10612 3705 Acked Write 4 Sent Done Done false
10613 3706 Acked Read 7 Sent Done Done false
10614 3707 Acked Write 10 Sent Done Done false
10615 3708 Acked Write 7 Sent Done Done false
10616 3709 Acked Write 10 Sent Done Done false
10617 3710 Acked Write 7 Sent Done Done false
10618 3711 Acked Read 10 Sent Done Done false
10619 3712 Acked Read 5 Sent Done Done false
10620 3713 Acked Write 7 Sent Done Done false
10621 3714 Acked Read 4 Sent Done Done false
10622 3715 Acked Read 5 Sent Done Done false
10623 3716 Acked Read 8 Sent Done Done false
10624 STATES DS:0 DS:1 DS:2 TOTAL
10625 Sent 301 0 0 301
10626 Done 0 301 301 602
10627 Skipped 0 0 0 0
10628 Error 0 0 0 0
10629 Last Flush: Some(JobId(3415)) Some(JobId(3700)) Some(JobId(3700))
10630 Downstairs last five completed: 3415 3414 3413 3412 3411
10631 Upstairs last five completed: 3716 3715 3714 3713 3712
10632 CLIENT: Up:0 ds:301 act:2
106332025-01-29T05:13:02.213ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
106342025-01-29T05:13:03.141ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10635 [8810] got start action
10636 [8810] I am going to restart
10637 Starting downstairs at port 8810
10638 Make output file at "/tmp/dsc/downstairs-8810.txt"
10639 [8810][0] reports Starting
10640 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211)
10641 [8810][0] reports Running
106422025-01-29T05:13:04.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
106432025-01-29T05:13:05.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10644 ----------------------------------------------------------------
10645 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10646 States: Offline Active Active
10647 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10648 3416 Acked Read 1 Sent Done Done false
10649 3417 Acked Flush 0 Sent Done Done false
10650 3418 Acked Read 4 Sent Done Done false
10651 3419 Acked Write 3 Sent Done Done false
10652 3420 Acked Read 10 Sent Done Done false
10653 3421 Acked Read 10 Sent Done Done false
10654 3422 Acked Read 3 Sent Done Done false
10655 3423 Acked Write 2 Sent Done Done false
10656 3424 Acked Write 7 Sent Done Done false
10657 3425 Acked Read 10 Sent Done Done false
10658 3426 Acked Write 1 Sent Done Done false
10659 3427 Acked Write 2 Sent Done Done false
10660 3428 Acked Write 6 Sent Done Done false
10661 3429 Acked Write 4 Sent Done Done false
10662 3430 Acked Read 3 Sent Done Done false
10663 3431 Acked Read 6 Sent Done Done false
10664 3432 Acked Write 8 Sent Done Done false
10665 3433 Acked Write 2 Sent Done Done false
10666 3434 Acked Read 10 Sent Done Done false
10667 3435 Acked Read 7 Sent Done Done false
10668 3436 Acked Read 10 Sent Done Done false
10669 3437 Acked Write 1 Sent Done Done false
10670 3438 Acked Write 3 Sent Done Done false
10671 3439 Acked Read 7 Sent Done Done false
10672 3440 Acked Write 7 Sent Done Done false
10673 3441 Acked Read 10 Sent Done Done false
10674 3442 Acked Read 6 Sent Done Done false
10675 3443 Acked Read 10 Sent Done Done false
10676 3444 Acked Write 1 Sent Done Done false
10677 3445 Acked Read 1 Sent Done Done false
10678 3446 Acked Read 4 Sent Done Done false
10679 3447 Acked Read 9 Sent Done Done false
10680 3448 Acked Read 2 Sent Done Done false
10681 3449 Acked Read 8 Sent Done Done false
10682 3450 Acked Write 4 Sent Done Done false
10683 3451 Acked Read 10 Sent Done Done false
10684 3452 Acked Read 5 Sent Done Done false
10685 3453 Acked Read 3 Sent Done Done false
10686 3454 Acked Read 6 Sent Done Done false
10687 3455 Acked Read 9 Sent Done Done false
10688 3456 Acked Read 10 Sent Done Done false
10689 3457 Acked Read 3 Sent Done Done false
10690 3458 Acked Read 3 Sent Done Done false
10691 3459 Acked Read 1 Sent Done Done false
10692 3460 Acked Read 5 Sent Done Done false
10693 3461 Acked Write 7 Sent Done Done false
10694 3462 Acked Write 4 Sent Done Done false
10695 3463 Acked Write 9 Sent Done Done false
10696 3464 Acked Read 4 Sent Done Done false
10697 3465 Acked Flush 0 Sent Done Done false
10698 3466 Acked Read 8 Sent Done Done false
10699 3467 Acked Flush 0 Sent Done Done false
10700 3468 Acked Write 3 Sent Done Done false
10701 3469 Acked Read 5 Sent Done Done false
10702 3470 Acked Write 4 Sent Done Done false
10703 3471 Acked Read 1 Sent Done Done false
10704 3472 Acked Read 4 Sent Done Done false
10705 3473 Acked Write 10 Sent Done Done false
10706 3474 Acked Read 8 Sent Done Done false
10707 3475 Acked Read 2 Sent Done Done false
10708 3476 Acked Read 4 Sent Done Done false
10709 3477 Acked Write 8 Sent Done Done false
10710 3478 Acked Write 1 Sent Done Done false
10711 3479 Acked Write 8 Sent Done Done false
10712 3480 Acked Read 4 Sent Done Done false
10713 3481 Acked Read 1 Sent Done Done false
10714 3482 Acked Read 10 Sent Done Done false
10715 3483 Acked Write 1 Sent Done Done false
10716 3484 Acked Read 4 Sent Done Done false
10717 3485 Acked Read 7 Sent Done Done false
10718 3486 Acked Write 8 Sent Done Done false
10719 3487 Acked Read 8 Sent Done Done false
10720 3488 Acked Write 3 Sent Done Done false
10721 3489 Acked Read 7 Sent Done Done false
10722 3490 Acked Write 9 Sent Done Done false
10723 3491 Acked Write 8 Sent Done Done false
10724 3492 Acked Write 10 Sent Done Done false
10725 3493 Acked Read 3 Sent Done Done false
10726 3494 Acked Read 1 Sent Done Done false
10727 3495 Acked Read 2 Sent Done Done false
10728 3496 Acked Write 6 Sent Done Done false
10729 3497 Acked Write 9 Sent Done Done false
10730 3498 Acked Read 1 Sent Done Done false
10731 3499 Acked Read 5 Sent Done Done false
10732 3500 Acked Write 2 Sent Done Done false
10733 3501 Acked Read 2 Sent Done Done false
10734 3502 Acked Write 7 Sent Done Done false
10735 3503 Acked Read 2 Sent Done Done false
10736 3504 Acked Read 7 Sent Done Done false
10737 3505 Acked Read 6 Sent Done Done false
10738 3506 Acked Read 5 Sent Done Done false
10739 3507 Acked Read 3 Sent Done Done false
10740 3508 Acked Write 2 Sent Done Done false
10741 3509 Acked Write 3 Sent Done Done false
10742 3510 Acked Read 6 Sent Done Done false
10743 3511 Acked Write 9 Sent Done Done false
10744 3512 Acked Write 2 Sent Done Done false
10745 3513 Acked Flush 0 Sent Done Done false
10746 3514 Acked Read 7 Sent Done Done false
10747 3515 Acked Read 2 Sent Done Done false
10748 3516 Acked Write 2 Sent Done Done false
10749 3517 Acked Write 5 Sent Done Done false
10750 3518 Acked Write 6 Sent Done Done false
10751 3519 Acked Read 10 Sent Done Done false
10752 3520 Acked Read 2 Sent Done Done false
10753 3521 Acked Read 9 Sent Done Done false
10754 3522 Acked Write 4 Sent Done Done false
10755 3523 Acked Read 5 Sent Done Done false
10756 3524 Acked Write 7 Sent Done Done false
10757 3525 Acked Write 2 Sent Done Done false
10758 3526 Acked Read 3 Sent Done Done false
10759 3527 Acked Read 1 Sent Done Done false
10760 3528 Acked Write 2 Sent Done Done false
10761 3529 Acked Write 8 Sent Done Done false
10762 3530 Acked Write 8 Sent Done Done false
10763 3531 Acked Read 3 Sent Done Done false
10764 3532 Acked Flush 0 Sent Done Done false
10765 3533 Acked Write 8 Sent Done Done false
10766 3534 Acked Read 7 Sent Done Done false
10767 3535 Acked Read 9 Sent Done Done false
10768 3536 Acked Write 10 Sent Done Done false
10769 3537 Acked Flush 0 Sent Done Done false
10770 3538 Acked Read 1 Sent Done Done false
10771 3539 Acked Write 1 Sent Done Done false
10772 3540 Acked Read 2 Sent Done Done false
10773 3541 Acked Write 3 Sent Done Done false
10774 3542 Acked Write 10 Sent Done Done false
10775 3543 Acked Read 5 Sent Done Done false
10776 3544 Acked Write 7 Sent Done Done false
10777 3545 Acked Read 9 Sent Done Done false
10778 3546 Acked Write 4 Sent Done Done false
10779 3547 Acked Write 8 Sent Done Done false
10780 3548 Acked Write 8 Sent Done Done false
10781 3549 Acked Read 1 Sent Done Done false
10782 3550 Acked Read 4 Sent Done Done false
10783 3551 Acked Write 1 Sent Done Done false
10784 3552 Acked Write 5 Sent Done Done false
10785 3553 Acked Flush 0 Sent Done Done false
10786 3554 Acked Write 9 Sent Done Done false
10787 3555 Acked Flush 0 Sent Done Done false
10788 3556 Acked Read 4 Sent Done Done false
10789 3557 Acked Read 2 Sent Done Done false
10790 3558 Acked Write 9 Sent Done Done false
10791 3559 Acked Write 5 Sent Done Done false
10792 3560 Acked Read 8 Sent Done Done false
10793 3561 Acked Write 3 Sent Done Done false
10794 3562 Acked Write 5 Sent Done Done false
10795 3563 Acked Read 3 Sent Done Done false
10796 3564 Acked Read 2 Sent Done Done false
10797 3565 Acked Read 10 Sent Done Done false
10798 3566 Acked Read 10 Sent Done Done false
10799 3567 Acked Read 4 Sent Done Done false
10800 3568 Acked Read 2 Sent Done Done false
10801 3569 Acked Read 6 Sent Done Done false
10802 3570 Acked Write 8 Sent Done Done false
10803 3571 Acked Read 4 Sent Done Done false
10804 3572 Acked Write 8 Sent Done Done false
10805 3573 Acked Read 7 Sent Done Done false
10806 3574 Acked Read 2 Sent Done Done false
10807 3575 Acked Read 10 Sent Done Done false
10808 3576 Acked Read 2 Sent Done Done false
10809 3577 Acked Read 5 Sent Done Done false
10810 3578 Acked Flush 0 Sent Done Done false
10811 3579 Acked Write 7 Sent Done Done false
10812 3580 Acked Read 1 Sent Done Done false
10813 3581 Acked Read 5 Sent Done Done false
10814 3582 Acked Write 9 Sent Done Done false
10815 3583 Acked Read 5 Sent Done Done false
10816 3584 Acked Read 4 Sent Done Done false
10817 3585 Acked Read 2 Sent Done Done false
10818 3586 Acked Read 2 Sent Done Done false
10819 3587 Acked Read 4 Sent Done Done false
10820 3588 Acked Read 10 Sent Done Done false
10821 3589 Acked Read 1 Sent Done Done false
10822 3590 Acked Flush 0 Sent Done Done false
10823 3591 Acked Read 2 Sent Done Done false
10824 3592 Acked Read 1 Sent Done Done false
10825 3593 Acked Read 3 Sent Done Done false
10826 3594 Acked Read 8 Sent Done Done false
10827 3595 Acked Read 5 Sent Done Done false
10828 3596 Acked Read 2 Sent Done Done false
10829 3597 Acked Read 4 Sent Done Done false
10830 3598 Acked Write 9 Sent Done Done false
10831 3599 Acked Write 2 Sent Done Done false
10832 3600 Acked Read 4 Sent Done Done false
10833 3601 Acked Flush 0 Sent Done Done false
10834 3602 Acked Read 10 Sent Done Done false
10835 3603 Acked Read 10 Sent Done Done false
10836 3604 Acked Read 5 Sent Done Done false
10837 3605 Acked Write 6 Sent Done Done false
10838 3606 Acked Read 8 Sent Done Done false
10839 3607 Acked Flush 0 Sent Done Done false
10840 3608 Acked Write 1 Sent Done Done false
10841 3609 Acked Write 1 Sent Done Done false
10842 3610 Acked Write 10 Sent Done Done false
10843 3611 Acked Flush 0 Sent Done Done false
10844 3612 Acked Read 8 Sent Done Done false
10845 3613 Acked Write 10 Sent Done Done false
10846 3614 Acked Read 9 Sent Done Done false
10847 3615 Acked Read 9 Sent Done Done false
10848 3616 Acked Flush 0 Sent Done Done false
10849 3617 Acked Write 3 Sent Done Done false
10850 3618 Acked Write 5 Sent Done Done false
10851 3619 Acked Read 6 Sent Done Done false
10852 3620 Acked Read 8 Sent Done Done false
10853 3621 Acked Read 7 Sent Done Done false
10854 3622 Acked Read 5 Sent Done Done false
10855 3623 Acked Read 4 Sent Done Done false
10856 3624 Acked Read 5 Sent Done Done false
10857 3625 Acked Flush 0 Sent Done Done false
10858 3626 Acked Read 5 Sent Done Done false
10859 3627 Acked Flush 0 Sent Done Done false
10860 3628 Acked Read 9 Sent Done Done false
10861 3629 Acked Read 1 Sent Done Done false
10862 3630 Acked Read 2 Sent Done Done false
10863 3631 Acked Flush 0 Sent Done Done false
10864 3632 Acked Read 1 Sent Done Done false
10865 3633 Acked Read 1 Sent Done Done false
10866 3634 Acked Flush 0 Sent Done Done false
10867 3635 Acked Write 5 Sent Done Done false
10868 3636 Acked Read 3 Sent Done Done false
10869 3637 Acked Flush 0 Sent Done Done false
10870 3638 Acked Write 7 Sent Done Done false
10871 3639 Acked Write 7 Sent Done Done false
10872 3640 Acked Write 1 Sent Done Done false
10873 3641 Acked Write 4 Sent Done Done false
10874 3642 Acked Write 4 Sent Done Done false
10875 3643 Acked Flush 0 Sent Done Done false
10876 3644 Acked Write 2 Sent Done Done false
10877 3645 Acked Read 7 Sent Done Done false
10878 3646 Acked Read 1 Sent Done Done false
10879 3647 Acked Write 5 Sent Done Done false
10880 3648 Acked Write 4 Sent Done Done false
10881 3649 Acked Read 2 Sent Done Done false
10882 3650 Acked Read 5 Sent Done Done false
10883 3651 Acked Read 2 Sent Done Done false
10884 3652 Acked Read 5 Sent Done Done false
10885 3653 Acked Read 7 Sent Done Done false
10886 3654 Acked Write 2 Sent Done Done false
10887 3655 Acked Write 4 Sent Done Done false
10888 3656 Acked Write 6 Sent Done Done false
10889 3657 Acked Flush 0 Sent Done Done false
10890 3658 Acked Read 5 Sent Done Done false
10891 3659 Acked Write 3 Sent Done Done false
10892 3660 Acked Read 10 Sent Done Done false
10893 3661 Acked Write 8 Sent Done Done false
10894 3662 Acked Write 2 Sent Done Done false
10895 3663 Acked Read 6 Sent Done Done false
10896 3664 Acked Read 6 Sent Done Done false
10897 3665 Acked Read 6 Sent Done Done false
10898 3666 Acked Write 4 Sent Done Done false
10899 3667 Acked Write 3 Sent Done Done false
10900 3668 Acked Write 7 Sent Done Done false
10901 3669 Acked Flush 0 Sent Done Done false
10902 3670 Acked Read 2 Sent Done Done false
10903 3671 Acked Write 6 Sent Done Done false
10904 3672 Acked Write 10 Sent Done Done false
10905 3673 Acked Read 2 Sent Done Done false
10906 3674 Acked Read 9 Sent Done Done false
10907 3675 Acked Read 6 Sent Done Done false
10908 3676 Acked Read 1 Sent Done Done false
10909 3677 Acked Read 5 Sent Done Done false
10910 3678 Acked Read 9 Sent Done Done false
10911 3679 Acked Read 4 Sent Done Done false
10912 3680 Acked Write 4 Sent Done Done false
10913 3681 Acked Write 9 Sent Done Done false
10914 3682 Acked Read 9 Sent Done Done false
10915 3683 Acked Read 7 Sent Done Done false
10916 3684 Acked Read 5 Sent Done Done false
10917 3685 Acked Write 1 Sent Done Done false
10918 3686 Acked Read 1 Sent Done Done false
10919 3687 Acked Write 1 Sent Done Done false
10920 3688 Acked Write 10 Sent Done Done false
10921 3689 Acked Read 2 Sent Done Done false
10922 3690 Acked Write 3 Sent Done Done false
10923 3691 Acked Write 10 Sent Done Done false
10924 3692 Acked Read 2 Sent Done Done false
10925 3693 Acked Read 7 Sent Done Done false
10926 3694 Acked Read 5 Sent Done Done false
10927 3695 Acked Write 4 Sent Done Done false
10928 3696 Acked Write 6 Sent Done Done false
10929 3697 Acked Write 8 Sent Done Done false
10930 3698 Acked Read 5 Sent Done Done false
10931 3699 Acked Flush 0 Sent Done Done false
10932 3700 Acked Flush 0 Sent Done Done false
10933 3701 Acked Read 5 Sent Done Done false
10934 3702 Acked Write 3 Sent Done Done false
10935 3703 Acked Read 10 Sent Done Done false
10936 3704 Acked Write 2 Sent Done Done false
10937 3705 Acked Write 4 Sent Done Done false
10938 3706 Acked Read 7 Sent Done Done false
10939 3707 Acked Write 10 Sent Done Done false
10940 3708 Acked Write 7 Sent Done Done false
10941 3709 Acked Write 10 Sent Done Done false
10942 3710 Acked Write 7 Sent Done Done false
10943 3711 Acked Read 10 Sent Done Done false
10944 3712 Acked Read 5 Sent Done Done false
10945 3713 Acked Write 7 Sent Done Done false
10946 3714 Acked Read 4 Sent Done Done false
10947 3715 Acked Read 5 Sent Done Done false
10948 3716 Acked Read 8 Sent Done Done false
10949 STATES DS:0 DS:1 DS:2 TOTAL
10950 Sent 301 0 0 301
10951 Done 0 301 301 602
10952 Skipped 0 0 0 0
10953 Error 0 0 0 0
10954 Last Flush: Some(JobId(3415)) Some(JobId(3700)) Some(JobId(3700))
10955 Downstairs last five completed: 3415 3414 3413 3412 3411
10956 Upstairs last five completed: 3716 3715 3714 3713 3712
10957 CLIENT: Up:0 ds:301 act:2
109582025-01-29T05:13:06.215ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109592025-01-29T05:13:07.147ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109602025-01-29T05:13:07.468ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109612025-01-29T05:13:07.468ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:54495) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109622025-01-29T05:13:07.469ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109632025-01-29T05:13:07.469ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109642025-01-29T05:13:07.469ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109652025-01-29T05:13:07.469ZINFOcrucible: send last flush ID to this DS: Some(JobId(3415)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109662025-01-29T05:13:07.469ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109672025-01-29T05:13:07.469ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3415)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109682025-01-29T05:13:07.470ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3415)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10969 ----------------------------------------------------------------
10970 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10971 Downstairs last five completed: 3717 3716 3715 3714 3713
10972 Upstairs last five completed: 3717 3716 3715 3714 3713
10973 CLIENT: Up:0 ds:0 act:3
10974 Replay: All jobs finished
10975 Main task has work to do, go find it
10976 got dsc Stop(0)
10977 stop 0
10978 Jan 29 05:13:09.716 INFO request completed, latency_us[8810] Got stop action so:false kr:false
10979 : 365, response_code: 204, uri: /stop/cid/0, method: POST, req_id: e17ec415-84bd-4ea2-b229-7e3e9fef264a, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
10980 Jan 29 05:13:09.717 INFO request completed, latency_us: 111, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6b1bec8a-598c-4e60-9ec9-66914de36a32, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
109812025-01-29T05:13:09.717ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109822025-01-29T05:13:09.717ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109832025-01-29T05:13:09.717ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
109842025-01-29T05:13:09.717ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
10985 [8810] Exited with: ExitStatus(unix_wait_status(9))
10986 [8810][0] reports Exit
10987 Jan 29 05:13:13.720 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4f595e5a-fdc9-44c4-bd90-109e77e985bb, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
10988 001/300 Read block 1342 len 12288
10989 002/300 Read block 4227 len 32768
10990 003/300 Read block 1649 len 8192
10991 004/300 Read block 1482 len 16384
10992 005/300 Read block 4066 len 12288
10993 006/300 Read block 3942 len 32768
10994 007/300 Write block 4599 len 36864 data: 5 6 5 5 6 6 4 3 3
10995 008/300 Write block 1983 len 40960 data: 4 4 4 4 3 3 3 3 3 3
10996 009/300 Read block 4622 len 8192
10997 010/300 Read block 4157 len 32768
10998 011/300 Read block 1004 len 24576
10999 012/300 Read block 2219 len 8192
11000 013/300 Read block 763 len 36864
11001 014/300 Write block 1090 len 16384 data: 2 2 3 3
11002 015/300 Write block 1263 len 12288 data: 3 3 3
11003 016/300 Read block 2184 len 4096
11004 017/300 Read block 1775 len 20480
11005 018/300 Read block 1915 len 40960
11006 019/300 Write block 3194 len 32768 data: 2 3 3 3 3 3 2 2
11007 020/300 Write block 732 len 32768 data: 4 3 3 3 5 5 5 5
11008 021/300 Write block 1961 len 20480 data: 2 2 2 2 3
11009 022/300 Flush
11010 023/300 Flush
11011 024/300 Flush
11012 025/300 Read block 3938 len 20480
11013 026/300 Write block 890 len 24576 data: 2 3 3 2 2 2
11014 027/300 Flush
11015 028/300 Read block 4820 len 16384
11016 029/300 Read block 1797 len 16384
11017 030/300 Read block 2777 len 32768
11018 031/300 Flush
11019 032/300 Write block 1487 len 24576 data: 2 2 2 2 2 2
11020 033/300 Write block 2954 len 4096 data: 3
11021 034/300 Read block 1225 len 12288
11022 035/300 Read block 4891 len 20480
11023 036/300 Read block 2438 len 4096
11024 037/300 Write block 4596 len 40960 data: 2 4 4 6 7 6 6 7 7 5
11025 038/300 Read block 513 len 40960
11026 039/300 Write block 765 len 12288 data: 2 3 3
11027 040/300 Read block 4247 len 32768
11028 041/300 Write block 2011 len 12288 data: 5 4 3
11029 042/300 Write block 1148 len 40960 data: 2 2 2 2 3 2 2 2 2 2
11030 043/300 Write block 740 len 40960 data: 5 4 4 4 7 8 5 5 4 6
11031 044/300 Write block 2707 len 16384 data: 2 2 2 3
11032 045/300 Write block 2252 len 8192 data: 2 2
11033 046/300 Write block 3216 len 20480 data: 3 3 3 3 3
11034 047/300 Write block 4147 len 28672 data: 3 4 5 4 3 3 3
11035 048/300 Read block 1912 len 20480
11036 049/300 Read block 173 len 40960
11037 050/300 Write block 3825 len 32768 data: 3 3 3 2 2 3 3 3
11038 051/300 Read block 123 len 32768
11039 052/300 Read block 1119 len 24576
11040 053/300 Flush
11041 054/300 Write block 2199 len 16384 data: 4 3 3 2
11042 055/300 Write block 4697 len 8192 data: 4 4
11043 056/300 Write block 1365 len 4096 data: 4
11044 057/300 Write block 3396 len 16384 data: 3 3 3 2
11045 058/300 Write block 2423 len 24576 data: 3 3 5 5 4 3
11046 059/300 Write block 4001 len 28672 data: 4 4 3 4 3 3 3
11047 060/300 Read block 1356 len 12288
11048 061/300 Read block 496 len 20480
11049 062/300 Read block 2472 len 8192
11050 063/300 Read block 3212 len 20480
11051 064/300 Write block 4006 len 24576 data: 4 4 4 4 4 4
11052 065/300 Write block 4636 len 40960 data: 4 4 5 4 4 4 4 5 5 4
11053 066/300 Write block 1683 len 12288 data: 5 5 5
11054 067/300 Write block 3136 len 20480 data: 5 5 5 3 4
11055 068/300 Write block 1600 len 4096 data: 4
11056 069/300 Write block 807 len 36864 data: 3 4 4 3 3 4 4 4 4
11057 070/300 Read block 2261 len 8192
11058 071/300 Read block 3288 len 20480
11059 072/300 Read block 4284 len 36864
11060 073/300 Flush
11061 074/300 Read block 4483 len 20480
11062 075/300 Write block 1241 len 28672 data: 2 3 3 3 3 2 2
11063 076/300 Flush
11064 077/300 Write block 1538 len 8192 data: 3 3
11065 078/300 Read block 2514 len 12288
11066 079/300 Write block 3430 len 16384 data: 5 3 3 3
11067 080/300 Write block 846 len 24576 data: 4 5 5 3 3 3
11068 081/300 Read block 1490 len 28672
11069 082/300 Write block 2946 len 12288 data: 5 5 5
11070 083/300 Read block 3946 len 16384
11071 084/300 Read block 613 len 24576
11072 085/300 Read block 1938 len 24576
11073 086/300 Read block 1201 len 16384
11074 087/300 Flush
11075 088/300 Read block 3366 len 8192
11076 089/300 Read block 4640 len 20480
11077 090/300 Write block 2669 len 12288 data: 4 3 3
11078 091/300 Write block 382 len 36864 data: 4 2 2 2 2 2 3 3 4
11079 092/300 Write block 1373 len 8192 data: 4 4
11080 093/300 Write block 3821 len 16384 data: 3 3 3 3
11081 094/300 Read block 1171 len 40960
11082 095/300 Write block 569 len 36864 data: 2 2 2 3 3 3 3 3 4
11083 096/300 Write block 2751 len 20480 data: 6 6 6 5 6
11084 097/300 Read block 4572 len 24576
11085 098/300 Write block 742 len 16384 data: 5 5 8 9
11086 099/300 Read block 2785 len 8192
11087 100/300 Write block 3987 len 16384 data: 3 3 4 4
11088 101/300 Write block 4655 len 24576 data: 3 3 3 3 3 3
11089 102/300 Read block 781 len 20480
11090 103/300 Read block 918 len 40960
11091 104/300 Read block 3327 len 40960
11092 105/300 Write block 2745 len 16384 data: 2 3 4 5
11093 106/300 Read block 1801 len 12288
11094 107/300 Read block 3538 len 40960
11095 108/300 Read block 3803 len 4096
11096 109/300 Write block 4871 len 20480 data: 3 3 3 4 4
11097 110/300 Write block 4830 len 20480 data: 3 3 3 3 3
11098 111/300 Flush
11099 112/300 Read block 2125 len 16384
11100 113/300 Flush
11101 114/300 Flush
11102 115/300 Read block 4440 len 40960
11103 116/300 Write block 757 len 40960 data: 4 3 3 3 3 2 2 2 3 4
11104 117/300 Write block 95 len 28672 data: 2 2 2 2 2 2 2
11105 118/300 Write block 363 len 28672 data: 3 2 2 2 2 2 2
11106 119/300 Write block 1551 len 20480 data: 3 3 3 3 3
11107 120/300 Write block 111 len 20480 data: 5 5 4 4 5
11108 121/300 Write block 3827 len 40960 data: 4 3 3 4 4 4 3 4 5 4
11109 122/300 Write block 4537 len 36864 data: 3 3 3 2 2 2 2 2 2
11110 123/300 Write block 4293 len 28672 data: 4 5 4 4 3 3 3
11111 124/300 Read block 2889 len 32768
11112 125/300 Read block 752 len 40960
11113 126/300 Write block 3228 len 40960 data: 5 4 4 5 5 5 4 4 5 5
11114 127/300 Write block 2798 len 4096 data: 3
11115 128/300 Read block 4283 len 4096
11116 129/300 Read block 2180 len 36864
11117 130/300 Read block 2385 len 16384
11118 131/300 Write block 2916 len 24576 data: 3 3 3 3 3 3
11119 132/300 Write block 2493 len 12288 data: 3 2 2
11120 133/300 Write block 3656 len 8192 data: 3 4
11121 134/300 Read block 1429 len 12288
11122 135/300 Read block 1678 len 40960
11123 136/300 Read block 2898 len 4096
11124 137/300 Read block 177 len 20480
11125 138/300 Read block 4922 len 4096
11126 139/300 Read block 1588 len 16384
11127 140/300 Read block 2569 len 16384
11128 141/300 Write block 1607 len 36864 data: 2 2 3 3 3 2 2 2 2
11129 142/300 Read block 2671 len 40960
11130 143/300 Read block 4271 len 4096
11131 144/300 Write block 3485 len 24576 data: 2 3 2 2 2 2
11132 145/300 Read block 138 len 4096
11133 146/300 Read block 1172 len 36864
11134 147/300 Read block 851 len 8192
11135 148/300 Read block 1708 len 24576
11136 149/300 Write block 770 len 12288 data: 2 2 2
11137 150/300 Write block 419 len 36864 data: 3 3 3 2 2 2 3 4 4
11138 151/300 Read block 4590 len 16384
11139 152/300 Read block 4971 len 40960
11140 153/300 Read block 2412 len 12288
11141 154/300 Write block 3378 len 28672 data: 3 4 4 4 5 4 5
11142 155/300 Read block 3883 len 40960
11143 156/300 Flush
11144 157/300 Read block 1729 len 20480
11145 158/300 Write block 4929 len 8192 data: 3 3
11146 159/300 Write block 3841 len 36864 data: 3 3 3 3 3 3 3 3 4
11147 160/300 Write block 1479 len 4096 data: 2
11148 161/300 Flush
11149 162/300 Read block 2077 len 40960
11150 163/300 Read block 760 len 12288
11151 164/300 Read block 4979 len 4096
11152 165/300 Read block 3491 len 40960
11153 166/300 Write block 331 len 16384 data: 3 3 3 2
11154 167/300 Write block 1068 len 28672 data: 2 2 2 2 2 2 2
11155 168/300 Read block 1981 len 28672
11156 169/300 Flush
11157 170/300 Read block 4113 len 16384
11158 171/300 Flush
11159 172/300 Write block 4439 len 4096 data: 3
11160 173/300 Write block 952 len 24576 data: 6 6 5 4 3 3
11161 174/300 Read block 1184 len 4096
11162 175/300 Read block 1343 len 32768
11163 176/300 Read block 2228 len 8192
11164 177/300 Write block 3509 len 32768 data: 2 2 2 2 3 4 3 3
11165 178/300 Write block 4886 len 4096 data: 4
11166 179/300 Read block 760 len 36864
11167 180/300 Write block 868 len 24576 data: 4 4 4 4 4 4
11168 181/300 Write block 2939 len 12288 data: 2 2 2
11169 182/300 Read block 3170 len 40960
11170 183/300 Write block 1524 len 36864 data: 2 2 2 3 3 3 3 3 4
11171 184/300 Read block 1621 len 32768
11172 185/300 Read block 4724 len 32768
11173 186/300 Write block 279 len 32768 data: 3 3 3 3 3 3 4 4
11174 187/300 Read block 3312 len 12288
11175 188/300 Flush
11176 189/300 Read block 2070 len 40960
11177 190/300 Write block 2454 len 20480 data: 3 4 4 4 3
11178 191/300 Read block 186 len 8192
11179 192/300 Write block 3110 len 24576 data: 5 4 4 3 3 4
11180 193/300 Read block 2902 len 28672
11181 194/300 Write block 4509 len 4096 data: 2
11182 195/300 Write block 811 len 40960 data: 4 5 5 5 5 4 4 4 4 3
11183 196/300 Write block 3376 len 40960 data: 4 3 4 5 5 5 6 5 6 4
11184 197/300 Read block 2926 len 16384
11185 198/300 Flush
11186 199/300 Write block 635 len 40960 data: 2 2 2 3 4 4 4 3 3 3
11187 200/300 Read block 2156 len 40960
11188 201/300 Write block 4701 len 40960 data: 5 5 4 4 4 3 3 2 3 3
11189 202/300 Write block 4647 len 28672 data: 4 4 3 3 3 4 3
11190 203/300 Write block 2608 len 40960 data: 5 5 4 4 3 3 3 4 3 2
11191 204/300 Write block 880 len 32768 data: 5 4 4 4 4 3 2 2
11192 205/300 Write block 4882 len 36864 data: 2 2 3 3 5 4 4 4 4
11193 206/300 Read block 3798 len 12288
11194 207/300 Write block 439 len 40960 data: 3 2 2 2 2 2 2 3 3 3
11195 208/300 Read block 1372 len 24576
11196 209/300 Read block 2631 len 32768
11197 210/300 Read block 11 len 4096
11198 211/300 Write block 1280 len 28672 data: 3 4 4 5 5 5 5
11199 212/300 Read block 149 len 40960
11200 213/300 Write block 4117 len 16384 data: 3 3 3 4
11201 214/300 Write block 2150 len 36864 data: 4 4 2 2 3 3 3 3 3
11202 215/300 Write block 637 len 28672 data: 3 4 5 5 5 4 4
11203 216/300 Write block 792 len 32768 data: 3 3 3 3 3 2 2 2
11204 217/300 Write block 546 len 20480 data: 4 4 5 5 5
11205 218/300 Write block 4767 len 24576 data: 2 3 3 5 4 4
11206 219/300 Flush
11207 220/300 Read block 2962 len 28672
11208 221/300 Write block 44 len 20480 data: 4 3 3 3 4
11209 222/300 Write block 4821 len 28672 data: 2 2 2 2 2 2 2
11210 223/300 Read block 1443 len 24576
11211 224/300 Read block 2861 len 28672
11212 225/300 Write block 295 len 40960 data: 3 3 3 3 3 3 3 3 2 2
11213 226/300 Write block 4286 len 12288 data: 3 3 3
11214 227/300 Read block 4245 len 20480
11215 228/300 Write block 1711 len 32768 data: 3 3 4 3 3 2 2 2
11216 229/300 Read block 800 len 20480
11217 230/300 Read block 1010 len 4096
11218 231/300 Read block 662 len 20480
11219 232/300 Read block 4552 len 28672
11220 233/300 Read block 4658 len 32768
11221 234/300 Write block 2476 len 28672 data: 3 3 2 2 2 2 2
11222 235/300 Write block 3219 len 28672 data: 4 4 3 3 3 3 3
11223 236/300 Flush
11224 237/300 Read block 1888 len 20480
11225 238/300 Read block 1588 len 20480
11226 239/300 Write block 1413 len 24576 data: 2 2 2 4 3 3
11227 240/300 Flush
11228 241/300 Write block 4171 len 24576 data: 4 4 4 4 3 4
11229 242/300 Flush
11230 243/300 Write block 1521 len 12288 data: 3 2 2
11231 244/300 Read block 68 len 8192
11232 245/300 Read block 3968 len 8192
11233 246/300 Read block 3866 len 8192
11234 247/300 Flush
11235 248/300 Write block 2658 len 8192 data: 2 4
11236 249/300 Read block 2862 len 32768
11237 250/300 Write block 2522 len 32768 data: 3 4 4 4 3 3 3 3
11238 251/300 Write block 95 len 4096 data: 3
11239 252/300 Read block 3208 len 8192
11240 253/300 Read block 4455 len 28672
11241 254/300 Read block 2898 len 8192
11242 255/300 Write block 4454 len 4096 data: 6
11243 256/300 Read block 2777 len 28672
11244 257/300 Write block 95 len 36864 data: 4 3 3 3 3 3 3 2 2
11245 258/300 Write block 4567 len 20480 data: 2 2 3 3 2
11246 259/300 Read block 1975 len 24576
11247 260/300 Read block 4486 len 16384
11248 261/300 Write block 4452 len 28672 data: 5 6 7 6 5 4 3
11249 262/300 Write block 3220 len 4096 data: 5
11250 263/300 Write block 3078 len 32768 data: 4 4 5 4 3 3 2 2
11251 264/300 Read block 3409 len 40960
11252 265/300 Write block 1902 len 40960 data: 2 2 2 2 3 4 3 3 3 4
11253 266/300 Read block 3461 len 20480
11254 267/300 Write block 4198 len 40960 data: 3 3 4 5 5 5 5 5 4 4
11255 268/300 Read block 2312 len 24576
11256 269/300 Write block 226 len 32768 data: 4 3 3 2 3 3 3 3
11257 270/300 Write block 4 len 8192 data: 4 4
11258 271/300 Read block 981 len 32768
11259 272/300 Write block 2171 len 12288 data: 2 2 2
11260 273/300 Write block 51 len 36864 data: 5 4 3 4 4 4 3 3 4
11261 274/300 Write block 3507 len 4096 data: 2
11262 275/300 Write block 634 len 28672 data: 2 3 3 4 5 6 6
11263 276/300 Flush
11264 277/300 Flush
11265 278/300 Flush
11266 279/300 Write block 1937 len 32768 data: 4 4 4 5 3 2 2 2
11267 280/300 Write block 4733 len 36864 data: 3 3 3 3 2 2 3 4 5
11268 281/300 Read block 4088 len 28672
11269 282/300 Read block 869 len 28672
11270 283/300 Read block 4188 len 8192
11271 284/300 Read block 3110 len 8192
11272 285/300 Write block 388 len 4096 data: 4
11273 286/300 Flush
11274 287/300 Write block 1641 len 12288 data: 2 2 2
11275 288/300 Write block 2452 len 28672 data: 3 3 4 5 5 5 4
11276 289/300 Read block 4194 len 4096
11277 290/300 Read block 2460 len 20480
11278 291/300 Read block 3922 len 16384
11279 292/300 Read block 1118 len 12288
11280 293/300 Write block 1497 len 28672 data: 3 3 3 3 5 5 4
11281 294/300 Read block 1528 len 28672
11282 295/300 Read block 3986 len 8192
11283 296/300 Flush
11284 297/300 Read block 3050 len 8192
11285 298/300 Write block 1315 len 20480 data: 3 4 4 3 2
11286 299/300 Write block 449 len 36864 data: 2 2 2 2 2 4 4 4 4
11287 300/300 Write block 4197 len 16384 data: 3 4 4 5
11288 301/300 Read block 4490 len 24576
11289 Main task has work to do, go find it
11290 got dsc Start(0)
11291 start 0
11292 Jan 29 05:13:13.952 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3a40e0e2-d719-4850-bd7e-55fc2efd1b58, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
11293 Replay: started 0, returned:Ok(())
11294 ----------------------------------------------------------------
11295 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11296 States: Offline Active Active
11297 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11298 3718 Acked Read 3 Sent Done Done false
11299 3719 Acked Read 8 Sent Done Done false
11300 3720 Acked Read 2 Sent Done Done false
11301 3721 Acked Read 4 Sent Done Done false
11302 3722 Acked Read 3 Sent Done Done false
11303 3723 Acked Read 8 Sent Done Done false
11304 3724 Acked Write 9 Sent Done Done false
11305 3725 Acked Write 10 Sent Done Done false
11306 3726 Acked Read 2 Sent Done Done false
11307 3727 Acked Read 8 Sent Done Done false
11308 3728 Acked Read 6 Sent Done Done false
11309 3729 Acked Read 2 Sent Done Done false
11310 3730 Acked Read 9 Sent Done Done false
11311 3731 Acked Write 4 Sent Done Done false
11312 3732 Acked Write 3 Sent Done Done false
11313 3733 Acked Read 1 Sent Done Done false
11314 3734 Acked Read 5 Sent Done Done false
11315 3735 Acked Read 10 Sent Done Done false
11316 3736 Acked Write 8 Sent Done Done false
11317 3737 Acked Write 8 Sent Done Done false
11318 3738 Acked Write 5 Sent Done Done false
11319 3739 Acked Flush 0 Sent Done Done false
11320 3740 Acked Flush 0 Sent Done Done false
11321 3741 Acked Flush 0 Sent Done Done false
11322 3742 Acked Read 5 Sent Done Done false
11323 3743 Acked Write 6 Sent Done Done false
11324 3744 Acked Flush 0 Sent Done Done false
11325 3745 Acked Read 4 Sent Done Done false
11326 3746 Acked Read 4 Sent Done Done false
11327 3747 Acked Read 8 Sent Done Done false
11328 3748 Acked Flush 0 Sent Done Done false
11329 3749 Acked Write 6 Sent Done Done false
11330 3750 Acked Write 1 Sent Done Done false
11331 3751 Acked Read 3 Sent Done Done false
11332 3752 Acked Read 5 Sent Done Done false
11333 3753 Acked Read 1 Sent Done Done false
11334 3754 Acked Write 10 Sent Done Done false
11335 3755 Acked Read 10 Sent Done Done false
11336 3756 Acked Write 3 Sent Done Done false
11337 3757 Acked Read 8 Sent Done Done false
11338 3758 Acked Write 3 Sent Done Done false
11339 3759 Acked Write 10 Sent Done Done false
11340 3760 Acked Write 10 Sent Done Done false
11341 3761 Acked Write 4 Sent Done Done false
11342 3762 Acked Write 2 Sent Done Done false
11343 3763 Acked Write 5 Sent Done Done false
11344 3764 Acked Write 7 Sent Done Done false
11345 3765 Acked Read 5 Sent Done Done false
11346 3766 Acked Read 10 Sent Done Done false
11347 3767 Acked Write 8 Sent Done Done false
11348 3768 Acked Read 8 Sent Done Done false
11349 3769 Acked Read 6 Sent Done Done false
11350 3770 Acked Flush 0 Sent Done Done false
11351 3771 Acked Write 4 Sent Done Done false
11352 3772 Acked Write 2 Sent Done Done false
11353 3773 Acked Write 1 Sent Done Done false
11354 3774 Acked Write 4 Sent Done Done false
11355 3775 Acked Write 6 Sent Done Done false
11356 3776 Acked Write 7 Sent Done Done false
11357 3777 Acked Read 3 Sent Done Done false
11358 3778 Acked Read 5 Sent Done Done false
11359 3779 Acked Read 2 Sent Done Done false
11360 3780 Acked Read 5 Sent Done Done false
11361 3781 Acked Write 6 Sent Done Done false
11362 3782 Acked Write 10 Sent Done Done false
11363 3783 Acked Write 3 Sent Done Done false
11364 3784 Acked Write 5 Sent Done Done false
11365 3785 Acked Write 1 Sent Done Done false
11366 3786 Acked Write 9 Sent Done Done false
11367 3787 Acked Read 2 Sent Done Done false
11368 3788 Acked Read 5 Sent Done Done false
11369 3789 Acked Read 9 Sent Done Done false
11370 3790 Acked Flush 0 Sent Done Done false
11371 3791 Acked Read 5 Sent Done Done false
11372 3792 Acked Write 7 Sent Done Done false
11373 3793 Acked Flush 0 Sent Done Done false
11374 3794 Acked Write 2 Sent Done Done false
11375 3795 Acked Read 3 Sent Done Done false
11376 3796 Acked Write 4 Sent Done Done false
11377 3797 Acked Write 6 Sent Done Done false
11378 3798 Acked Read 7 Sent Done Done false
11379 3799 Acked Write 3 Sent Done Done false
11380 3800 Acked Read 4 Sent Done Done false
11381 3801 Acked Read 6 Sent Done Done false
11382 3802 Acked Read 6 Sent Done Done false
11383 3803 Acked Read 4 Sent Done Done false
11384 3804 Acked Flush 0 Sent Done Done false
11385 3805 Acked Read 2 Sent Done Done false
11386 3806 Acked Read 5 Sent Done Done false
11387 3807 Acked Write 3 Sent Done Done false
11388 3808 Acked Write 9 Sent Done Done false
11389 3809 Acked Write 2 Sent Done Done false
11390 3810 Acked Write 4 Sent Done Done false
11391 3811 Acked Read 10 Sent Done Done false
11392 3812 Acked Write 9 Sent Done Done false
11393 3813 Acked Write 5 Sent Done Done false
11394 3814 Acked Read 6 Sent Done Done false
11395 3815 Acked Write 4 Sent Done Done false
11396 3816 Acked Read 2 Sent Done Done false
11397 3817 Acked Write 4 Sent Done Done false
11398 3818 Acked Write 6 Sent Done Done false
11399 3819 Acked Read 5 Sent Done Done false
11400 3820 Acked Read 10 Sent Done Done false
11401 3821 Acked Read 10 Sent Done Done false
11402 3822 Acked Write 4 Sent Done Done false
11403 3823 Acked Read 3 Sent Done Done false
11404 3824 Acked Read 10 Sent Done Done false
11405 3825 Acked Read 1 Sent Done Done false
11406 3826 Acked Write 5 Sent Done Done false
11407 3827 Acked Write 5 Sent Done Done false
11408 3828 Acked Flush 0 Sent Done Done false
11409 3829 Acked Read 4 Sent Done Done false
11410 3830 Acked Flush 0 Sent Done Done false
11411 3831 Acked Flush 0 Sent Done Done false
11412 3832 Acked Read 10 Sent Done Done false
11413 3833 Acked Write 10 Sent Done Done false
11414 3834 Acked Write 7 Sent Done Done false
11415 3835 Acked Write 7 Sent Done Done false
11416 3836 Acked Write 5 Sent Done Done false
11417 3837 Acked Write 5 Sent Done Done false
11418 3838 Acked Write 10 Sent Done Done false
11419 3839 Acked Write 9 Sent Done Done false
11420 3840 Acked Write 7 Sent Done Done false
11421 3841 Acked Read 8 Sent Done Done false
11422 3842 Acked Read 10 Sent Done Done false
11423 3843 Acked Write 10 Sent Done Done false
11424 3844 Acked Write 1 Sent Done Done false
11425 3845 Acked Read 1 Sent Done Done false
11426 3846 Acked Read 9 Sent Done Done false
11427 3847 Acked Read 4 Sent Done Done false
11428 3848 Acked Write 6 Sent Done Done false
11429 3849 Acked Write 3 Sent Done Done false
11430 3850 Acked Write 2 Sent Done Done false
11431 3851 Acked Read 3 Sent Done Done false
11432 3852 Acked Read 10 Sent Done Done false
11433 3853 Acked Read 1 Sent Done Done false
11434 3854 Acked Read 5 Sent Done Done false
11435 3855 Acked Read 1 Sent Done Done false
11436 3856 Acked Read 4 Sent Done Done false
11437 3857 Acked Read 4 Sent Done Done false
11438 3858 Acked Write 9 Sent Done Done false
11439 3859 Acked Read 10 Sent Done Done false
11440 3860 Acked Read 1 Sent Done Done false
11441 3861 Acked Write 6 Sent Done Done false
11442 3862 Acked Read 1 Sent Done Done false
11443 3863 Acked Read 9 Sent Done Done false
11444 3864 Acked Read 2 Sent Done Done false
11445 3865 Acked Read 6 Sent Done Done false
11446 3866 Acked Write 3 Sent Done Done false
11447 3867 Acked Write 9 Sent Done Done false
11448 3868 Acked Read 4 Sent Done Done false
11449 3869 Acked Read 10 Sent Done Done false
11450 3870 Acked Read 3 Sent Done Done false
11451 3871 Acked Write 7 Sent Done Done false
11452 3872 Acked Read 10 Sent Done Done false
11453 3873 Acked Flush 0 Sent Done Done false
11454 3874 Acked Read 5 Sent Done Done false
11455 3875 Acked Write 2 Sent Done Done false
11456 3876 Acked Write 9 Sent Done Done false
11457 3877 Acked Write 1 Sent Done Done false
11458 3878 Acked Flush 0 Sent Done Done false
11459 3879 Acked Read 10 Sent Done Done false
11460 3880 Acked Read 3 Sent Done Done false
11461 3881 Acked Read 1 Sent Done Done false
11462 3882 Acked Read 10 Sent Done Done false
11463 3883 Acked Write 4 Sent Done Done false
11464 3884 Acked Write 7 Sent Done Done false
11465 3885 Acked Read 7 Sent Done Done false
11466 3886 Acked Flush 0 Sent Done Done false
11467 3887 Acked Read 4 Sent Done Done false
11468 3888 Acked Flush 0 Sent Done Done false
11469 3889 Acked Write 1 Sent Done Done false
11470 3890 Acked Write 6 Sent Done Done false
11471 3891 Acked Read 1 Sent Done Done false
11472 3892 Acked Read 8 Sent Done Done false
11473 3893 Acked Read 2 Sent Done Done false
11474 3894 Acked Write 8 Sent Done Done false
11475 3895 Acked Write 1 Sent Done Done false
11476 3896 Acked Read 9 Sent Done Done false
11477 3897 Acked Write 6 Sent Done Done false
11478 3898 Acked Write 3 Sent Done Done false
11479 3899 Acked Read 10 Sent Done Done false
11480 3900 Acked Write 9 Sent Done Done false
11481 3901 Acked Read 8 Sent Done Done false
11482 3902 Acked Read 8 Sent Done Done false
11483 3903 Acked Write 8 Sent Done Done false
11484 3904 Acked Read 3 Sent Done Done false
11485 3905 Acked Flush 0 Sent Done Done false
11486 3906 Acked Read 10 Sent Done Done false
11487 3907 Acked Write 5 Sent Done Done false
11488 3908 Acked Read 2 Sent Done Done false
11489 3909 Acked Write 6 Sent Done Done false
11490 3910 Acked Read 7 Sent Done Done false
11491 3911 Acked Write 1 Sent Done Done false
11492 3912 Acked Write 10 Sent Done Done false
11493 3913 Acked Write 10 Sent Done Done false
11494 3914 Acked Read 4 Sent Done Done false
11495 3915 Acked Flush 0 Sent Done Done false
11496 3916 Acked Write 10 Sent Done Done false
11497 3917 Acked Read 10 Sent Done Done false
11498 3918 Acked Write 10 Sent Done Done false
11499 3919 Acked Write 7 Sent Done Done false
11500 3920 Acked Write 10 Sent Done Done false
11501 3921 Acked Write 8 Sent Done Done false
11502 3922 Acked Write 9 Sent Done Done false
11503 3923 Acked Read 3 Sent Done Done false
11504 3924 Acked Write 10 Sent Done Done false
11505 3925 Acked Read 6 Sent Done Done false
11506 3926 Acked Read 8 Sent Done Done false
11507 3927 Acked Read 1 Sent Done Done false
11508 3928 Acked Write 7 Sent Done Done false
11509 3929 Acked Read 10 Sent Done Done false
11510 3930 Acked Write 4 Sent Done Done false
11511 3931 Acked Write 9 Sent Done Done false
11512 3932 Acked Write 7 Sent Done Done false
11513 3933 Acked Write 8 Sent Done Done false
11514 3934 Acked Write 5 Sent Done Done false
11515 3935 Acked Write 6 Sent Done Done false
11516 3936 Acked Flush 0 Sent Done Done false
11517 3937 Acked Read 7 Sent Done Done false
11518 3938 Acked Write 5 Sent Done Done false
11519 3939 Acked Write 7 Sent Done Done false
11520 3940 Acked Read 6 Sent Done Done false
11521 3941 Acked Read 7 Sent Done Done false
11522 3942 Acked Write 10 Sent Done Done false
11523 3943 Acked Write 3 Sent Done Done false
11524 3944 Acked Read 5 Sent Done Done false
11525 3945 Acked Write 8 Sent Done Done false
11526 3946 Acked Read 5 Sent Done Done false
11527 3947 Acked Read 1 Sent Done Done false
11528 3948 Acked Read 5 Sent Done Done false
11529 3949 Acked Read 7 Sent Done Done false
11530 3950 Acked Read 8 Sent Done Done false
11531 3951 Acked Write 7 Sent Done Done false
11532 3952 Acked Write 7 Sent Done Done false
11533 3953 Acked Flush 0 Sent Done Done false
11534 3954 Acked Read 5 Sent Done Done false
11535 3955 Acked Read 5 Sent Done Done false
11536 3956 Acked Write 6 Sent Done Done false
11537 3957 Acked Flush 0 Sent Done Done false
11538 3958 Acked Write 6 Sent Done Done false
11539 3959 Acked Flush 0 Sent Done Done false
11540 3960 Acked Write 3 Sent Done Done false
11541 3961 Acked Read 2 Sent Done Done false
11542 3962 Acked Read 2 Sent Done Done false
11543 3963 Acked Read 2 Sent Done Done false
11544 3964 Acked Flush 0 Sent Done Done false
11545 3965 Acked Write 2 Sent Done Done false
11546 3966 Acked Read 8 Sent Done Done false
11547 3967 Acked Write 8 Sent Done Done false
11548 3968 Acked Write 1 Sent Done Done false
11549 3969 Acked Read 2 Sent Done Done false
11550 3970 Acked Read 7 Sent Done Done false
11551 3971 Acked Read 2 Sent Done Done false
11552 3972 Acked Write 1 Sent Done Done false
11553 3973 Acked Read 7 Sent Done Done false
11554 3974 Acked Write 9 Sent Done Done false
11555 3975 Acked Write 5 Sent Done Done false
11556 3976 Acked Read 6 Sent Done Done false
11557 3977 Acked Read 4 Sent Done Done false
11558 3978 Acked Write 7 Sent Done Done false
11559 3979 Acked Write 1 Sent Done Done false
11560 3980 Acked Write 8 Sent Done Done false
11561 3981 Acked Read 10 Sent Done Done false
11562 3982 Acked Write 10 Sent Done Done false
11563 3983 Acked Read 5 Sent Done Done false
11564 3984 Acked Write 10 Sent Done Done false
11565 3985 Acked Read 6 Sent Done Done false
11566 3986 Acked Write 8 Sent Done Done false
11567 3987 Acked Write 2 Sent Done Done false
11568 3988 Acked Read 8 Sent Done Done false
11569 3989 Acked Write 3 Sent Done Done false
11570 3990 Acked Write 9 Sent Done Done false
11571 3991 Acked Write 1 Sent Done Done false
11572 3992 Acked Write 7 Sent Done Done false
11573 3993 Acked Flush 0 Sent Done Done false
11574 3994 Acked Flush 0 Sent Done Done false
11575 3995 Acked Flush 0 Sent Done Done false
11576 3996 Acked Write 8 Sent Done Done false
11577 3997 Acked Write 9 Sent Done Done false
11578 3998 Acked Read 7 Sent Done Done false
11579 3999 Acked Read 7 Sent Done Done false
11580 4000 Acked Read 2 Sent Done Done false
11581 4001 Acked Read 2 Sent Done Done false
11582 4002 Acked Write 1 Sent Done Done false
11583 4003 Acked Flush 0 Sent Done Done false
11584 4004 Acked Write 3 Sent Done Done false
11585 4005 Acked Write 7 Sent Done Done false
11586 4006 Acked Read 1 Sent Done Done false
11587 4007 Acked Read 5 Sent Done Done false
11588 4008 Acked Read 4 Sent Done Done false
11589 4009 Acked Read 3 Sent Done Done false
11590 4010 Acked Write 7 Sent Done Done false
11591 4011 Acked Read 7 Sent Done Done false
11592 4012 Acked Read 2 Sent Done Done false
11593 4013 Acked Flush 0 Sent Done Done false
11594 4014 Acked Read 2 Sent Done Done false
11595 4015 Acked Write 5 Sent Done Done false
11596 4016 Acked Write 9 Sent Done Done false
11597 4017 Acked Write 4 Sent Done Done false
11598 4018 Acked Read 6 Sent Done Done false
11599 STATES DS:0 DS:1 DS:2 TOTAL
11600 Sent 301 0 0 301
11601 Done 0 301 301 602
11602 Skipped 0 0 0 0
11603 Error 0 0 0 0
11604 Last Flush: Some(JobId(3717)) Some(JobId(4013)) Some(JobId(4013))
11605 Downstairs last five completed: 3717 3716 3715 3714 3713
11606 Upstairs last five completed: 4018 4017 4016 4015 4014
11607 CLIENT: Up:0 ds:301 act:2
116082025-01-29T05:13:14.454ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
116092025-01-29T05:13:15.160ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11610 [8810] got start action
11611 [8810] I am going to restart
11612 Starting downstairs at port 8810
11613 Make output file at "/tmp/dsc/downstairs-8810.txt"
11614 [8810][0] reports Starting
11615 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212)
11616 [8810][0] reports Running
116172025-01-29T05:13:16.163ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
116182025-01-29T05:13:17.165ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11619 ----------------------------------------------------------------
11620 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11621 States: Offline Active Active
11622 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11623 3718 Acked Read 3 Sent Done Done false
11624 3719 Acked Read 8 Sent Done Done false
11625 3720 Acked Read 2 Sent Done Done false
11626 3721 Acked Read 4 Sent Done Done false
11627 3722 Acked Read 3 Sent Done Done false
11628 3723 Acked Read 8 Sent Done Done false
11629 3724 Acked Write 9 Sent Done Done false
11630 3725 Acked Write 10 Sent Done Done false
11631 3726 Acked Read 2 Sent Done Done false
11632 3727 Acked Read 8 Sent Done Done false
11633 3728 Acked Read 6 Sent Done Done false
11634 3729 Acked Read 2 Sent Done Done false
11635 3730 Acked Read 9 Sent Done Done false
11636 3731 Acked Write 4 Sent Done Done false
11637 3732 Acked Write 3 Sent Done Done false
11638 3733 Acked Read 1 Sent Done Done false
11639 3734 Acked Read 5 Sent Done Done false
11640 3735 Acked Read 10 Sent Done Done false
11641 3736 Acked Write 8 Sent Done Done false
11642 3737 Acked Write 8 Sent Done Done false
11643 3738 Acked Write 5 Sent Done Done false
11644 3739 Acked Flush 0 Sent Done Done false
11645 3740 Acked Flush 0 Sent Done Done false
11646 3741 Acked Flush 0 Sent Done Done false
11647 3742 Acked Read 5 Sent Done Done false
11648 3743 Acked Write 6 Sent Done Done false
11649 3744 Acked Flush 0 Sent Done Done false
11650 3745 Acked Read 4 Sent Done Done false
11651 3746 Acked Read 4 Sent Done Done false
11652 3747 Acked Read 8 Sent Done Done false
11653 3748 Acked Flush 0 Sent Done Done false
11654 3749 Acked Write 6 Sent Done Done false
11655 3750 Acked Write 1 Sent Done Done false
11656 3751 Acked Read 3 Sent Done Done false
11657 3752 Acked Read 5 Sent Done Done false
11658 3753 Acked Read 1 Sent Done Done false
11659 3754 Acked Write 10 Sent Done Done false
11660 3755 Acked Read 10 Sent Done Done false
11661 3756 Acked Write 3 Sent Done Done false
11662 3757 Acked Read 8 Sent Done Done false
11663 3758 Acked Write 3 Sent Done Done false
11664 3759 Acked Write 10 Sent Done Done false
11665 3760 Acked Write 10 Sent Done Done false
11666 3761 Acked Write 4 Sent Done Done false
11667 3762 Acked Write 2 Sent Done Done false
11668 3763 Acked Write 5 Sent Done Done false
11669 3764 Acked Write 7 Sent Done Done false
11670 3765 Acked Read 5 Sent Done Done false
11671 3766 Acked Read 10 Sent Done Done false
11672 3767 Acked Write 8 Sent Done Done false
11673 3768 Acked Read 8 Sent Done Done false
11674 3769 Acked Read 6 Sent Done Done false
11675 3770 Acked Flush 0 Sent Done Done false
11676 3771 Acked Write 4 Sent Done Done false
11677 3772 Acked Write 2 Sent Done Done false
11678 3773 Acked Write 1 Sent Done Done false
11679 3774 Acked Write 4 Sent Done Done false
11680 3775 Acked Write 6 Sent Done Done false
11681 3776 Acked Write 7 Sent Done Done false
11682 3777 Acked Read 3 Sent Done Done false
11683 3778 Acked Read 5 Sent Done Done false
11684 3779 Acked Read 2 Sent Done Done false
11685 3780 Acked Read 5 Sent Done Done false
11686 3781 Acked Write 6 Sent Done Done false
11687 3782 Acked Write 10 Sent Done Done false
11688 3783 Acked Write 3 Sent Done Done false
11689 3784 Acked Write 5 Sent Done Done false
11690 3785 Acked Write 1 Sent Done Done false
11691 3786 Acked Write 9 Sent Done Done false
11692 3787 Acked Read 2 Sent Done Done false
11693 3788 Acked Read 5 Sent Done Done false
11694 3789 Acked Read 9 Sent Done Done false
11695 3790 Acked Flush 0 Sent Done Done false
11696 3791 Acked Read 5 Sent Done Done false
11697 3792 Acked Write 7 Sent Done Done false
11698 3793 Acked Flush 0 Sent Done Done false
11699 3794 Acked Write 2 Sent Done Done false
11700 3795 Acked Read 3 Sent Done Done false
11701 3796 Acked Write 4 Sent Done Done false
11702 3797 Acked Write 6 Sent Done Done false
11703 3798 Acked Read 7 Sent Done Done false
11704 3799 Acked Write 3 Sent Done Done false
11705 3800 Acked Read 4 Sent Done Done false
11706 3801 Acked Read 6 Sent Done Done false
11707 3802 Acked Read 6 Sent Done Done false
11708 3803 Acked Read 4 Sent Done Done false
11709 3804 Acked Flush 0 Sent Done Done false
11710 3805 Acked Read 2 Sent Done Done false
11711 3806 Acked Read 5 Sent Done Done false
11712 3807 Acked Write 3 Sent Done Done false
11713 3808 Acked Write 9 Sent Done Done false
11714 3809 Acked Write 2 Sent Done Done false
11715 3810 Acked Write 4 Sent Done Done false
11716 3811 Acked Read 10 Sent Done Done false
11717 3812 Acked Write 9 Sent Done Done false
11718 3813 Acked Write 5 Sent Done Done false
11719 3814 Acked Read 6 Sent Done Done false
11720 3815 Acked Write 4 Sent Done Done false
11721 3816 Acked Read 2 Sent Done Done false
11722 3817 Acked Write 4 Sent Done Done false
11723 3818 Acked Write 6 Sent Done Done false
11724 3819 Acked Read 5 Sent Done Done false
11725 3820 Acked Read 10 Sent Done Done false
11726 3821 Acked Read 10 Sent Done Done false
11727 3822 Acked Write 4 Sent Done Done false
11728 3823 Acked Read 3 Sent Done Done false
11729 3824 Acked Read 10 Sent Done Done false
11730 3825 Acked Read 1 Sent Done Done false
11731 3826 Acked Write 5 Sent Done Done false
11732 3827 Acked Write 5 Sent Done Done false
11733 3828 Acked Flush 0 Sent Done Done false
11734 3829 Acked Read 4 Sent Done Done false
11735 3830 Acked Flush 0 Sent Done Done false
11736 3831 Acked Flush 0 Sent Done Done false
11737 3832 Acked Read 10 Sent Done Done false
11738 3833 Acked Write 10 Sent Done Done false
11739 3834 Acked Write 7 Sent Done Done false
11740 3835 Acked Write 7 Sent Done Done false
11741 3836 Acked Write 5 Sent Done Done false
11742 3837 Acked Write 5 Sent Done Done false
11743 3838 Acked Write 10 Sent Done Done false
11744 3839 Acked Write 9 Sent Done Done false
11745 3840 Acked Write 7 Sent Done Done false
11746 3841 Acked Read 8 Sent Done Done false
11747 3842 Acked Read 10 Sent Done Done false
11748 3843 Acked Write 10 Sent Done Done false
11749 3844 Acked Write 1 Sent Done Done false
11750 3845 Acked Read 1 Sent Done Done false
11751 3846 Acked Read 9 Sent Done Done false
11752 3847 Acked Read 4 Sent Done Done false
11753 3848 Acked Write 6 Sent Done Done false
11754 3849 Acked Write 3 Sent Done Done false
11755 3850 Acked Write 2 Sent Done Done false
11756 3851 Acked Read 3 Sent Done Done false
11757 3852 Acked Read 10 Sent Done Done false
11758 3853 Acked Read 1 Sent Done Done false
11759 3854 Acked Read 5 Sent Done Done false
11760 3855 Acked Read 1 Sent Done Done false
11761 3856 Acked Read 4 Sent Done Done false
11762 3857 Acked Read 4 Sent Done Done false
11763 3858 Acked Write 9 Sent Done Done false
11764 3859 Acked Read 10 Sent Done Done false
11765 3860 Acked Read 1 Sent Done Done false
11766 3861 Acked Write 6 Sent Done Done false
11767 3862 Acked Read 1 Sent Done Done false
11768 3863 Acked Read 9 Sent Done Done false
11769 3864 Acked Read 2 Sent Done Done false
11770 3865 Acked Read 6 Sent Done Done false
11771 3866 Acked Write 3 Sent Done Done false
11772 3867 Acked Write 9 Sent Done Done false
11773 3868 Acked Read 4 Sent Done Done false
11774 3869 Acked Read 10 Sent Done Done false
11775 3870 Acked Read 3 Sent Done Done false
11776 3871 Acked Write 7 Sent Done Done false
11777 3872 Acked Read 10 Sent Done Done false
11778 3873 Acked Flush 0 Sent Done Done false
11779 3874 Acked Read 5 Sent Done Done false
11780 3875 Acked Write 2 Sent Done Done false
11781 3876 Acked Write 9 Sent Done Done false
11782 3877 Acked Write 1 Sent Done Done false
11783 3878 Acked Flush 0 Sent Done Done false
11784 3879 Acked Read 10 Sent Done Done false
11785 3880 Acked Read 3 Sent Done Done false
11786 3881 Acked Read 1 Sent Done Done false
11787 3882 Acked Read 10 Sent Done Done false
11788 3883 Acked Write 4 Sent Done Done false
11789 3884 Acked Write 7 Sent Done Done false
11790 3885 Acked Read 7 Sent Done Done false
11791 3886 Acked Flush 0 Sent Done Done false
11792 3887 Acked Read 4 Sent Done Done false
11793 3888 Acked Flush 0 Sent Done Done false
11794 3889 Acked Write 1 Sent Done Done false
11795 3890 Acked Write 6 Sent Done Done false
11796 3891 Acked Read 1 Sent Done Done false
11797 3892 Acked Read 8 Sent Done Done false
11798 3893 Acked Read 2 Sent Done Done false
11799 3894 Acked Write 8 Sent Done Done false
11800 3895 Acked Write 1 Sent Done Done false
11801 3896 Acked Read 9 Sent Done Done false
11802 3897 Acked Write 6 Sent Done Done false
11803 3898 Acked Write 3 Sent Done Done false
11804 3899 Acked Read 10 Sent Done Done false
11805 3900 Acked Write 9 Sent Done Done false
11806 3901 Acked Read 8 Sent Done Done false
11807 3902 Acked Read 8 Sent Done Done false
11808 3903 Acked Write 8 Sent Done Done false
11809 3904 Acked Read 3 Sent Done Done false
11810 3905 Acked Flush 0 Sent Done Done false
11811 3906 Acked Read 10 Sent Done Done false
11812 3907 Acked Write 5 Sent Done Done false
11813 3908 Acked Read 2 Sent Done Done false
11814 3909 Acked Write 6 Sent Done Done false
11815 3910 Acked Read 7 Sent Done Done false
11816 3911 Acked Write 1 Sent Done Done false
11817 3912 Acked Write 10 Sent Done Done false
11818 3913 Acked Write 10 Sent Done Done false
11819 3914 Acked Read 4 Sent Done Done false
11820 3915 Acked Flush 0 Sent Done Done false
11821 3916 Acked Write 10 Sent Done Done false
11822 3917 Acked Read 10 Sent Done Done false
11823 3918 Acked Write 10 Sent Done Done false
11824 3919 Acked Write 7 Sent Done Done false
11825 3920 Acked Write 10 Sent Done Done false
11826 3921 Acked Write 8 Sent Done Done false
11827 3922 Acked Write 9 Sent Done Done false
11828 3923 Acked Read 3 Sent Done Done false
11829 3924 Acked Write 10 Sent Done Done false
11830 3925 Acked Read 6 Sent Done Done false
11831 3926 Acked Read 8 Sent Done Done false
11832 3927 Acked Read 1 Sent Done Done false
11833 3928 Acked Write 7 Sent Done Done false
11834 3929 Acked Read 10 Sent Done Done false
11835 3930 Acked Write 4 Sent Done Done false
11836 3931 Acked Write 9 Sent Done Done false
11837 3932 Acked Write 7 Sent Done Done false
11838 3933 Acked Write 8 Sent Done Done false
11839 3934 Acked Write 5 Sent Done Done false
11840 3935 Acked Write 6 Sent Done Done false
11841 3936 Acked Flush 0 Sent Done Done false
11842 3937 Acked Read 7 Sent Done Done false
11843 3938 Acked Write 5 Sent Done Done false
11844 3939 Acked Write 7 Sent Done Done false
11845 3940 Acked Read 6 Sent Done Done false
11846 3941 Acked Read 7 Sent Done Done false
11847 3942 Acked Write 10 Sent Done Done false
11848 3943 Acked Write 3 Sent Done Done false
11849 3944 Acked Read 5 Sent Done Done false
11850 3945 Acked Write 8 Sent Done Done false
11851 3946 Acked Read 5 Sent Done Done false
11852 3947 Acked Read 1 Sent Done Done false
11853 3948 Acked Read 5 Sent Done Done false
11854 3949 Acked Read 7 Sent Done Done false
11855 3950 Acked Read 8 Sent Done Done false
11856 3951 Acked Write 7 Sent Done Done false
11857 3952 Acked Write 7 Sent Done Done false
11858 3953 Acked Flush 0 Sent Done Done false
11859 3954 Acked Read 5 Sent Done Done false
11860 3955 Acked Read 5 Sent Done Done false
11861 3956 Acked Write 6 Sent Done Done false
11862 3957 Acked Flush 0 Sent Done Done false
11863 3958 Acked Write 6 Sent Done Done false
11864 3959 Acked Flush 0 Sent Done Done false
11865 3960 Acked Write 3 Sent Done Done false
11866 3961 Acked Read 2 Sent Done Done false
11867 3962 Acked Read 2 Sent Done Done false
11868 3963 Acked Read 2 Sent Done Done false
11869 3964 Acked Flush 0 Sent Done Done false
11870 3965 Acked Write 2 Sent Done Done false
11871 3966 Acked Read 8 Sent Done Done false
11872 3967 Acked Write 8 Sent Done Done false
11873 3968 Acked Write 1 Sent Done Done false
11874 3969 Acked Read 2 Sent Done Done false
11875 3970 Acked Read 7 Sent Done Done false
11876 3971 Acked Read 2 Sent Done Done false
11877 3972 Acked Write 1 Sent Done Done false
11878 3973 Acked Read 7 Sent Done Done false
11879 3974 Acked Write 9 Sent Done Done false
11880 3975 Acked Write 5 Sent Done Done false
11881 3976 Acked Read 6 Sent Done Done false
11882 3977 Acked Read 4 Sent Done Done false
11883 3978 Acked Write 7 Sent Done Done false
11884 3979 Acked Write 1 Sent Done Done false
11885 3980 Acked Write 8 Sent Done Done false
11886 3981 Acked Read 10 Sent Done Done false
11887 3982 Acked Write 10 Sent Done Done false
11888 3983 Acked Read 5 Sent Done Done false
11889 3984 Acked Write 10 Sent Done Done false
11890 3985 Acked Read 6 Sent Done Done false
11891 3986 Acked Write 8 Sent Done Done false
11892 3987 Acked Write 2 Sent Done Done false
11893 3988 Acked Read 8 Sent Done Done false
11894 3989 Acked Write 3 Sent Done Done false
11895 3990 Acked Write 9 Sent Done Done false
11896 3991 Acked Write 1 Sent Done Done false
11897 3992 Acked Write 7 Sent Done Done false
11898 3993 Acked Flush 0 Sent Done Done false
11899 3994 Acked Flush 0 Sent Done Done false
11900 3995 Acked Flush 0 Sent Done Done false
11901 3996 Acked Write 8 Sent Done Done false
11902 3997 Acked Write 9 Sent Done Done false
11903 3998 Acked Read 7 Sent Done Done false
11904 3999 Acked Read 7 Sent Done Done false
11905 4000 Acked Read 2 Sent Done Done false
11906 4001 Acked Read 2 Sent Done Done false
11907 4002 Acked Write 1 Sent Done Done false
11908 4003 Acked Flush 0 Sent Done Done false
11909 4004 Acked Write 3 Sent Done Done false
11910 4005 Acked Write 7 Sent Done Done false
11911 4006 Acked Read 1 Sent Done Done false
11912 4007 Acked Read 5 Sent Done Done false
11913 4008 Acked Read 4 Sent Done Done false
11914 4009 Acked Read 3 Sent Done Done false
11915 4010 Acked Write 7 Sent Done Done false
11916 4011 Acked Read 7 Sent Done Done false
11917 4012 Acked Read 2 Sent Done Done false
11918 4013 Acked Flush 0 Sent Done Done false
11919 4014 Acked Read 2 Sent Done Done false
11920 4015 Acked Write 5 Sent Done Done false
11921 4016 Acked Write 9 Sent Done Done false
11922 4017 Acked Write 4 Sent Done Done false
11923 4018 Acked Read 6 Sent Done Done false
11924 STATES DS:0 DS:1 DS:2 TOTAL
11925 Sent 301 0 0 301
11926 Done 0 301 301 602
11927 Skipped 0 0 0 0
11928 Error 0 0 0 0
11929 Last Flush: Some(JobId(3717)) Some(JobId(4013)) Some(JobId(4013))
11930 Downstairs last five completed: 3717 3716 3715 3714 3713
11931 Upstairs last five completed: 4018 4017 4016 4015 4014
11932 CLIENT: Up:0 ds:301 act:2
119332025-01-29T05:13:19.169ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119342025-01-29T05:13:19.719ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119352025-01-29T05:13:19.720ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58153) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119362025-01-29T05:13:19.721ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119372025-01-29T05:13:19.721ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119382025-01-29T05:13:19.721ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119392025-01-29T05:13:19.721ZINFOcrucible: send last flush ID to this DS: Some(JobId(3717)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119402025-01-29T05:13:19.721ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119412025-01-29T05:13:19.721ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3717)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119422025-01-29T05:13:19.721ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3717)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11943 ----------------------------------------------------------------
11944 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11945 Downstairs last five completed: 4019 4018 4017 4016 4015
11946 Upstairs last five completed: 4019 4018 4017 4016 4015
11947 CLIENT: Up:0 ds:0 act:3
11948 Replay: All jobs finished
11949 Main task has work to do, go find it
11950 got dsc Stop(1)
11951 stop 1
11952 [8820] Got stop action so:false kr:false
11953 Jan 29 05:13:21.958 INFO request completed, latency_us: 319, response_code: 204, uri: /stop/cid/1, method: POST, req_id: d31fcb25-f120-46fe-84fa-4e5c652332de, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
119542025-01-29T05:13:21.959ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119552025-01-29T05:13:21.959ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
119562025-01-29T05:13:21.959ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
11957 Jan 29 05:13:21.959 INFO request completed, latency_us: 136, response_code: 200, uri: /state/cid/1, method: GET, req_id: cca73976-6757-4d66-b889-2310e272bcd3, remote_addr: 127.0.0.1:46995, local_addr: 127{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:13:21.959820657Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
11958 .0.0.1:9998
11959 [8820] Exited with: ExitStatus(unix_wait_status(9))
11960 [8820][1] reports Exit
11961 Jan 29 05:13:25.962 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: c0fb7513-922c-4461-be40-467723aab93d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
11962 001/300 Write block 1402 len 24576 data: 3 3 3 2 2 2
11963 002/300 Write block 2885 len 32768 data: 2 2 2 2 3 3 3 3
11964 003/300 Write block 2711 len 16384 data: 3 2 2 2
11965 004/300 Write block 4781 len 20480 data: 5 5 4 3 3
11966 005/300 Read block 1686 len 28672
11967 006/300 Read block 3369 len 24576
11968 007/300 Read block 3331 len 24576
11969 008/300 Write block 1770 len 8192 data: 3 3
11970 009/300 Write block 1792 len 16384 data: 3 2 3 3
11971 010/300 Write block 1562 len 40960 data: 4 4 3 2 2 2 2 2 2 2
11972 011/300 Read block 885 len 32768
11973 012/300 Read block 1181 len 40960
11974 013/300 Read block 3709 len 16384
11975 014/300 Flush
11976 015/300 Read block 4196 len 40960
11977 016/300 Read block 3409 len 32768
11978 017/300 Read block 1895 len 4096
11979 018/300 Write block 2998 len 24576 data: 2 3 3 3 4 5
11980 019/300 Write block 3520 len 40960 data: 4 4 4 3 3 3 3 3 3 3
11981 020/300 Read block 3688 len 12288
11982 021/300 Write block 2903 len 16384 data: 2 2 2 3
11983 022/300 Read block 1546 len 24576
11984 023/300 Flush
11985 024/300 Read block 150 len 16384
11986 025/300 Read block 597 len 12288
11987 026/300 Read block 4355 len 24576
11988 027/300 Read block 92 len 20480
11989 028/300 Write block 772 len 28672 data: 3 2 3 3 3 3 5
11990 029/300 Read block 3711 len 36864
11991 030/300 Write block 18 len 28672 data: 3 3 3 3 2 3 3
11992 031/300 Read block 2636 len 32768
11993 032/300 Flush
11994 033/300 Write block 201 len 8192 data: 2 2
11995 034/300 Read block 1894 len 20480
11996 035/300 Write block 2158 len 8192 data: 4 3
11997 036/300 Read block 4016 len 8192
11998 037/300 Read block 4636 len 12288
11999 038/300 Read block 159 len 28672
12000 039/300 Write block 4176 len 4096 data: 5
12001 040/300 Read block 2252 len 12288
12002 041/300 Write block 1285 len 20480 data: 6 6 4 4 3
12003 042/300 Read block 4393 len 4096
12004 043/300 Read block 3848 len 40960
12005 044/300 Read block 2837 len 40960
12006 045/300 Write block 2547 len 12288 data: 3 3 3
12007 046/300 Write block 642 len 32768 data: 5 5 4 2 2 3 3 3
12008 047/300 Write block 366 len 40960 data: 3 3 3 3 2 2 2 2 3 4
12009 048/300 Flush
12010 049/300 Write block 1280 len 40960 data: 4 5 5 6 6 7 7 5 5 4
12011 050/300 Read block 694 len 20480
12012 051/300 Read block 892 len 24576
12013 052/300 Read block 294 len 36864
12014 053/300 Write block 4611 len 8192 data: 4 5
12015 054/300 Write block 1882 len 28672 data: 5 4 5 5 5 5 5
12016 055/300 Read block 4889 len 8192
12017 056/300 Read block 4337 len 32768
12018 057/300 Write block 4083 len 40960 data: 2 2 2 2 2 2 2 2 2 2
12019 058/300 Read block 3113 len 16384
12020 059/300 Read block 2960 len 40960
12021 060/300 Read block 379 len 12288
12022 061/300 Flush
12023 062/300 Write block 118 len 40960 data: 5 5 4 4 4 5 4 4 3 4
12024 063/300 Write block 119 len 36864 data: 6 5 5 5 6 5 5 4 5
12025 064/300 Read block 835 len 8192
12026 065/300 Read block 4369 len 32768
12027 066/300 Read block 4401 len 36864
12028 067/300 Read block 3862 len 12288
12029 068/300 Read block 1119 len 28672
12030 069/300 Read block 3907 len 8192
12031 070/300 Write block 277 len 8192 data: 2 2
12032 071/300 Write block 1025 len 40960 data: 4 4 3 3 4 3 3 3 4 3
12033 072/300 Read block 4313 len 32768
12034 073/300 Write block 3154 len 12288 data: 3 3 3
12035 074/300 Read block 1866 len 20480
12036 075/300 Read block 1684 len 16384
12037 076/300 Read block 2777 len 20480
12038 077/300 Flush
12039 078/300 Read block 1379 len 36864
12040 079/300 Write block 791 len 28672 data: 3 4 4 4 4 4 3
12041 080/300 Flush
12042 081/300 Write block 2713 len 20480 data: 3 3 2 3 4
12043 082/300 Flush
12044 083/300 Flush
12045 084/300 Write block 3626 len 20480 data: 2 2 3 3 3
12046 085/300 Read block 2778 len 20480
12047 086/300 Read block 2394 len 28672
12048 087/300 Write block 1967 len 28672 data: 4 4 4 4 3 3 5
12049 088/300 Read block 3136 len 32768
12050 089/300 Write block 3855 len 16384 data: 3 3 3 3
12051 090/300 Write block 1774 len 36864 data: 4 5 5 4 4 4 3 2 2
12052 091/300 Write block 1695 len 24576 data: 4 4 4 4 4 2
12053 092/300 Write block 4181 len 16384 data: 3 4 4 3
12054 093/300 Read block 4504 len 8192
12055 094/300 Read block 2878 len 4096
12056 095/300 Write block 3643 len 32768 data: 4 4 3 2 2 2 3 3
12057 096/300 Read block 3277 len 20480
12058 097/300 Read block 2907 len 4096
12059 098/300 Flush
12060 099/300 Read block 1569 len 24576
12061 100/300 Write block 1953 len 16384 data: 3 3 4 4
12062 101/300 Write block 467 len 20480 data: 4 5 4 4 3
12063 102/300 Read block 119 len 8192
12064 103/300 Read block 4727 len 24576
12065 104/300 Write block 2073 len 8192 data: 2 2
12066 105/300 Write block 884 len 20480 data: 5 4 3 3 2
12067 106/300 Write block 312 len 4096 data: 3
12068 107/300 Read block 3616 len 20480
12069 108/300 Read block 3163 len 40960
12070 109/300 Read block 3387 len 8192
12071 110/300 Read block 2123 len 40960
12072 111/300 Write block 1688 len 24576 data: 5 5 7 7 4 5
12073 112/300 Write block 4396 len 16384 data: 6 4 4 4
12074 113/300 Flush
12075 114/300 Read block 3067 len 8192
12076 115/300 Read block 3901 len 20480
12077 116/300 Read block 1300 len 24576
12078 117/300 Read block 1270 len 16384
12079 118/300 Write block 2250 len 4096 data: 2
12080 119/300 Read block 4228 len 20480
12081 120/300 Read block 2925 len 8192
12082 121/300 Write block 2668 len 28672 data: 3 5 4 4 4 5 5
12083 122/300 Read block 373 len 16384
12084 123/300 Read block 4537 len 20480
12085 124/300 Read block 2219 len 24576
12086 125/300 Read block 3236 len 28672
12087 126/300 Write block 2474 len 20480 data: 3 3 4 4 3
12088 127/300 Read block 2229 len 28672
12089 128/300 Read block 3372 len 36864
12090 129/300 Write block 4902 len 4096 data: 3
12091 130/300 Write block 1682 len 12288 data: 6 6 6
12092 131/300 Write block 890 len 24576 data: 3 4 4 3 3 3
12093 132/300 Write block 3975 len 32768 data: 5 4 5 5 5 4 4 4
12094 133/300 Flush
12095 134/300 Write block 3936 len 28672 data: 4 4 4 4 4 4 4
12096 135/300 Read block 4554 len 20480
12097 136/300 Read block 3057 len 40960
12098 137/300 Read block 4326 len 20480
12099 138/300 Write block 551 len 36864 data: 4 4 3 3 2 4 3 3 3
12100 139/300 Read block 389 len 4096
12101 140/300 Write block 4589 len 28672 data: 3 3 3 3 3 3 2
12102 141/300 Write block 4356 len 20480 data: 3 3 3 3 3
12103 142/300 Flush
12104 143/300 Read block 554 len 24576
12105 144/300 Read block 2542 len 32768
12106 145/300 Read block 256 len 16384
12107 146/300 Write block 4615 len 40960 data: 4 2 2 2 2 2 2 2 2 2
12108 147/300 Write block 3219 len 20480 data: 5 6 4 4 4
12109 148/300 Read block 1239 len 4096
12110 149/300 Read block 3060 len 8192
12111 150/300 Write block 796 len 28672 data: 5 4 3 3 2 3 3
12112 151/300 Read block 860 len 4096
12113 152/300 Read block 2148 len 8192
12114 153/300 Read block 4145 len 32768
12115 154/300 Write block 67 len 32768 data: 2 3 3 3 4 4 4 4
12116 155/300 Write block 1742 len 4096 data: 2
12117 156/300 Read block 3821 len 12288
12118 157/300 Write block 2718 len 36864 data: 4 3 2 2 2 2 2 2 2
12119 158/300 Flush
12120 159/300 Read block 323 len 8192
12121 160/300 Read block 439 len 40960
12122 161/300 Write block 1937 len 28672 data: 5 5 5 6 4 3 3
12123 162/300 Flush
12124 163/300 Read block 935 len 32768
12125 164/300 Read block 417 len 24576
12126 165/300 Read block 4230 len 20480
12127 166/300 Write block 3933 len 40960 data: 2 2 2 5 5 5 5 5 5 5
12128 167/300 Read block 3049 len 8192
12129 168/300 Write block 537 len 20480 data: 2 2 2 2 2
12130 169/300 Write block 4527 len 36864 data: 3 4 4 4 4 4 5 5 4
12131 170/300 Write block 446 len 8192 data: 4 4
12132 171/300 Read block 825 len 8192
12133 172/300 Write block 2067 len 12288 data: 3 3 2
12134 173/300 Flush
12135 174/300 Read block 2705 len 8192
12136 175/300 Read block 3136 len 8192
12137 176/300 Read block 4504 len 32768
12138 177/300 Write block 3806 len 40960 data: 4 4 4 2 2 2 2 2 2 2
12139 178/300 Write block 4735 len 16384 data: 4 4 3 3
12140 179/300 Write block 1184 len 40960 data: 2 2 2 2 2 2 2 2 2 3
12141 180/300 Read block 2582 len 28672
12142 181/300 Read block 657 len 4096
12143 182/300 Write block 2481 len 24576 data: 3 3 2 2 2 4
12144 183/300 Flush
12145 184/300 Read block 187 len 12288
12146 185/300 Write block 1348 len 4096 data: 2
12147 186/300 Flush
12148 187/300 Write block 343 len 32768 data: 2 2 2 2 3 3 2 3
12149 188/300 Write block 1921 len 8192 data: 3 3
12150 189/300 Read block 2393 len 28672
12151 190/300 Write block 2170 len 12288 data: 2 3 3
12152 191/300 Read block 3288 len 40960
12153 192/300 Read block 3823 len 8192
12154 193/300 Read block 2802 len 16384
12155 194/300 Read block 2643 len 8192
12156 195/300 Read block 105 len 28672
12157 196/300 Read block 2168 len 16384
12158 197/300 Read block 341 len 28672
12159 198/300 Read block 1555 len 8192
12160 199/300 Read block 3689 len 8192
12161 200/300 Read block 844 len 12288
12162 201/300 Read block 2654 len 16384
12163 202/300 Read block 3258 len 16384
12164 203/300 Read block 4841 len 12288
12165 204/300 Write block 277 len 8192 data: 3 3
12166 205/300 Read block 2385 len 16384
12167 206/300 Read block 4476 len 8192
12168 207/300 Read block 2742 len 28672
12169 208/300 Read block 577 len 36864
12170 209/300 Flush
12171 210/300 Read block 3832 len 24576
12172 211/300 Read block 3692 len 4096
12173 212/300 Write block 2701 len 24576 data: 3 2 2 2 2 2
12174 213/300 Read block 3626 len 40960
12175 214/300 Read block 4119 len 20480
12176 215/300 Read block 1940 len 24576
12177 216/300 Read block 331 len 28672
12178 217/300 Write block 2830 len 32768 data: 4 4 4 4 3 3 3 5
12179 218/300 Write block 3376 len 36864 data: 5 4 5 6 6 6 7 6 7
12180 219/300 Flush
12181 220/300 Read block 4985 len 36864
12182 221/300 Write block 164 len 40960 data: 4 3 3 2 2 5 6 6 5 5
12183 222/300 Write block 1682 len 40960 data: 7 7 7 6 3 5 6 6 8 8
12184 223/300 Write block 3609 len 4096 data: 4
12185 224/300 Write block 4771 len 32768 data: 5 5 5 5 5 5 5 5
12186 225/300 Write block 1939 len 40960 data: 6 7 5 4 4 3 2 2 2 2
12187 226/300 Read block 4939 len 16384
12188 227/300 Write block 3038 len 16384 data: 4 3 3 3
12189 228/300 Write block 1123 len 20480 data: 4 4 4 4 4
12190 229/300 Write block 4964 len 16384 data: 3 2 2 3
12191 230/300 Read block 969 len 36864
12192 231/300 Write block 965 len 16384 data: 4 4 5 6
12193 232/300 Write block 3605 len 8192 data: 3 3
12194 233/300 Write block 2140 len 12288 data: 5 6 5
12195 234/300 Read block 3654 len 4096
12196 235/300 Read block 4938 len 28672
12197 236/300 Write block 80 len 28672 data: 4 4 4 4 4 4 4
12198 237/300 Write block 4856 len 20480 data: 2 3 3 4 4
12199 238/300 Read block 545 len 40960
12200 239/300 Flush
12201 240/300 Read block 1148 len 32768
12202 241/300 Write block 3888 len 16384 data: 2 2 2 2
12203 242/300 Read block 4183 len 40960
12204 243/300 Write block 4136 len 16384 data: 4 4 5 5
12205 244/300 Read block 647 len 8192
12206 245/300 Write block 3749 len 40960 data: 4 4 3 5 4 4 4 4 5 4
12207 246/300 Read block 850 len 24576
12208 247/300 Write block 1880 len 20480 data: 3 5 6 5 6
12209 248/300 Write block 3646 len 24576 data: 3 3 3 4 4 3
12210 249/300 Write block 2995 len 32768 data: 2 2 2 3 4 4 4 5
12211 250/300 Read block 974 len 40960
12212 251/300 Write block 4957 len 24576 data: 5 5 5 3 3 3
12213 252/300 Write block 2505 len 4096 data: 4
12214 253/300 Write block 3351 len 4096 data: 4
12215 254/300 Write block 238 len 36864 data: 7 7 6 6 6 5 5 5 4
12216 255/300 Write block 4989 len 16384 data: 5 5 5 6
12217 256/300 Write block 2083 len 24576 data: 3 4 4 4 3 2
12218 257/300 Read block 2743 len 20480
12219 258/300 Write block 1247 len 4096 data: 3
12220 259/300 Write block 13 len 20480 data: 3 4 4 4 4
12221 260/300 Write block 845 len 32768 data: 4 5 6 6 4 4 4 3
12222 261/300 Flush
12223 262/300 Write block 121 len 28672 data: 6 6 7 6 6 5 6
12224 263/300 Read block 3304 len 4096
12225 264/300 Write block 967 len 32768 data: 6 7 6 6 5 5 4 3
12226 265/300 Read block 111 len 8192
12227 266/300 Write block 1367 len 20480 data: 3 2 4 4 4
12228 267/300 Read block 4151 len 12288
12229 268/300 Write block 4805 len 36864 data: 3 3 3 3 3 5 4 4 4
12230 269/300 Write block 4033 len 40960 data: 3 3 2 2 2 2 2 2 3 4
12231 270/300 Read block 2333 len 16384
12232 271/300 Read block 4233 len 32768
12233 272/300 Write block 3609 len 28672 data: 5 4 4 4 4 4 3
12234 273/300 Read block 3849 len 16384
12235 274/300 Write block 4669 len 8192 data: 3 3
12236 275/300 Read block 835 len 32768
12237 276/300 Write block 4876 len 4096 data: 3
12238 277/300 Write block 287 len 40960 data: 4 3 3 3 3 3 3 4 4 4
12239 278/300 Read block 2192 len 40960
12240 279/300 Write block 573 len 40960 data: 4 4 4 4 5 3 2 2 2 3
12241 280/300 Read block 3926 len 16384
12242 281/300 Read block 4325 len 40960
12243 282/300 Write block 4490 len 36864 data: 3 3 3 3 2 2 3 3 4
12244 283/300 Read block 3884 len 32768
12245 284/300 Write block 3942 len 32768 data: 6 4 5 5 4 5 5 4
12246 285/300 Write block 492 len 36864 data: 5 5 5 4 4 4 4 3 3
12247 286/300 Read block 51 len 4096
12248 287/300 Write block 777 len 12288 data: 4 6 5
12249 288/300 Flush
12250 289/300 Read block 4887 len 36864
12251 290/300 Read block 4850 len 40960
12252 291/300 Write block 4080 len 20480 data: 2 2 2 3 3
12253 292/300 Read block 2214 len 36864
12254 293/300 Read block 2361 len 40960
12255 294/300 Write block 204 len 4096 data: 2
12256 295/300 Read block 643 len 28672
12257 296/300 Read block 3130 len 36864
12258 297/300 Write block 695 len 20480 data: 3 2 2 3 3
12259 298/300 Read block 2393 len 32768
12260 299/300 Read block 2664 len 24576
12261 300/300 Write block 124 len 4096 data: 7
12262 301/300 Write block 1567 len 8192 data: 3 3
12263 Main task has work to do, go find it
12264 got dsc Start(1)
12265 start 1
12266 Jan 29 05:13:26.206 INFO request completed, latency_us: 191, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3f14892f-28f4-42a1-91e9-c2caebe9c2b4, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
12267 Replay: started 1, returned:Ok(())
12268 ----------------------------------------------------------------
12269 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12270 States: Active Offline Active
12271 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12272 4020 Acked Write 6 Done Sent Done false
12273 4021 Acked Write 8 Done Sent Done false
12274 4022 Acked Write 4 Done Sent Done false
12275 4023 Acked Write 5 Done Sent Done false
12276 4024 Acked Read 7 Done Sent Done false
12277 4025 Acked Read 6 Done Sent Done false
12278 4026 Acked Read 6 Done Sent Done false
12279 4027 Acked Write 2 Done Sent Done false
12280 4028 Acked Write 4 Done Sent Done false
12281 4029 Acked Write 10 Done Sent Done false
12282 4030 Acked Read 8 Done Sent Done false
12283 4031 Acked Read 10 Done Sent Done false
12284 4032 Acked Read 4 Done Sent Done false
12285 4033 Acked Flush 0 Done Sent Done false
12286 4034 Acked Read 10 Done Sent Done false
12287 4035 Acked Read 8 Done Sent Done false
12288 4036 Acked Read 1 Done Sent Done false
12289 4037 Acked Write 6 Done Sent Done false
12290 4038 Acked Write 10 Done Sent Done false
12291 4039 Acked Read 3 Done Sent Done false
12292 4040 Acked Write 4 Done Sent Done false
12293 4041 Acked Read 6 Done Sent Done false
12294 4042 Acked Flush 0 Done Sent Done false
12295 4043 Acked Read 4 Done Sent Done false
12296 4044 Acked Read 3 Done Sent Done false
12297 4045 Acked Read 6 Done Sent Done false
12298 4046 Acked Read 5 Done Sent Done false
12299 4047 Acked Write 7 Done Sent Done false
12300 4048 Acked Read 9 Done Sent Done false
12301 4049 Acked Write 7 Done Sent Done false
12302 4050 Acked Read 8 Done Sent Done false
12303 4051 Acked Flush 0 Done Sent Done false
12304 4052 Acked Write 2 Done Sent Done false
12305 4053 Acked Read 5 Done Sent Done false
12306 4054 Acked Write 2 Done Sent Done false
12307 4055 Acked Read 2 Done Sent Done false
12308 4056 Acked Read 3 Done Sent Done false
12309 4057 Acked Read 7 Done Sent Done false
12310 4058 Acked Write 1 Done Sent Done false
12311 4059 Acked Read 3 Done Sent Done false
12312 4060 Acked Write 5 Done Sent Done false
12313 4061 Acked Read 1 Done Sent Done false
12314 4062 Acked Read 10 Done Sent Done false
12315 4063 Acked Read 10 Done Sent Done false
12316 4064 Acked Write 3 Done Sent Done false
12317 4065 Acked Write 8 Done Sent Done false
12318 4066 Acked Write 10 Done Sent Done false
12319 4067 Acked Flush 0 Done Sent Done false
12320 4068 Acked Write 10 Done Sent Done false
12321 4069 Acked Read 5 Done Sent Done false
12322 4070 Acked Read 6 Done Sent Done false
12323 4071 Acked Read 9 Done Sent Done false
12324 4072 Acked Write 2 Done Sent Done false
12325 4073 Acked Write 7 Done Sent Done false
12326 4074 Acked Read 2 Done Sent Done false
12327 4075 Acked Read 8 Done Sent Done false
12328 4076 Acked Write 10 Done Sent Done false
12329 4077 Acked Read 4 Done Sent Done false
12330 4078 Acked Read 10 Done Sent Done false
12331 4079 Acked Read 3 Done Sent Done false
12332 4080 Acked Flush 0 Done Sent Done false
12333 4081 Acked Write 10 Done Sent Done false
12334 4082 Acked Write 9 Done Sent Done false
12335 4083 Acked Read 2 Done Sent Done false
12336 4084 Acked Read 8 Done Sent Done false
12337 4085 Acked Read 9 Done Sent Done false
12338 4086 Acked Read 3 Done Sent Done false
12339 4087 Acked Read 7 Done Sent Done false
12340 4088 Acked Read 2 Done Sent Done false
12341 4089 Acked Write 2 Done Sent Done false
12342 4090 Acked Write 10 Done Sent Done false
12343 4091 Acked Read 8 Done Sent Done false
12344 4092 Acked Write 3 Done Sent Done false
12345 4093 Acked Read 5 Done Sent Done false
12346 4094 Acked Read 4 Done Sent Done false
12347 4095 Acked Read 5 Done Sent Done false
12348 4096 Acked Flush 0 Done Sent Done false
12349 4097 Acked Read 9 Done Sent Done false
12350 4098 Acked Write 7 Done Sent Done false
12351 4099 Acked Flush 0 Done Sent Done false
12352 4100 Acked Write 5 Done Sent Done false
12353 4101 Acked Flush 0 Done Sent Done false
12354 4102 Acked Flush 0 Done Sent Done false
12355 4103 Acked Write 5 Done Sent Done false
12356 4104 Acked Read 5 Done Sent Done false
12357 4105 Acked Read 7 Done Sent Done false
12358 4106 Acked Write 7 Done Sent Done false
12359 4107 Acked Read 8 Done Sent Done false
12360 4108 Acked Write 4 Done Sent Done false
12361 4109 Acked Write 9 Done Sent Done false
12362 4110 Acked Write 6 Done Sent Done false
12363 4111 Acked Write 4 Done Sent Done false
12364 4112 Acked Read 2 Done Sent Done false
12365 4113 Acked Read 1 Done Sent Done false
12366 4114 Acked Write 8 Done Sent Done false
12367 4115 Acked Read 5 Done Sent Done false
12368 4116 Acked Read 1 Done Sent Done false
12369 4117 Acked Flush 0 Done Sent Done false
12370 4118 Acked Read 6 Done Sent Done false
12371 4119 Acked Write 4 Done Sent Done false
12372 4120 Acked Write 5 Done Sent Done false
12373 4121 Acked Read 2 Done Sent Done false
12374 4122 Acked Read 6 Done Sent Done false
12375 4123 Acked Write 2 Done Sent Done false
12376 4124 Acked Write 5 Done Sent Done false
12377 4125 Acked Write 1 Done Sent Done false
12378 4126 Acked Read 5 Done Sent Done false
12379 4127 Acked Read 10 Done Sent Done false
12380 4128 Acked Read 2 Done Sent Done false
12381 4129 Acked Read 10 Done Sent Done false
12382 4130 Acked Write 6 Done Sent Done false
12383 4131 Acked Write 4 Done Sent Done false
12384 4132 Acked Flush 0 Done Sent Done false
12385 4133 Acked Read 2 Done Sent Done false
12386 4134 Acked Read 5 Done Sent Done false
12387 4135 Acked Read 6 Done Sent Done false
12388 4136 Acked Read 4 Done Sent Done false
12389 4137 Acked Write 1 Done Sent Done false
12390 4138 Acked Read 5 Done Sent Done false
12391 4139 Acked Read 2 Done Sent Done false
12392 4140 Acked Write 7 Done Sent Done false
12393 4141 Acked Read 4 Done Sent Done false
12394 4142 Acked Read 5 Done Sent Done false
12395 4143 Acked Read 6 Done Sent Done false
12396 4144 Acked Read 7 Done Sent Done false
12397 4145 Acked Write 5 Done Sent Done false
12398 4146 Acked Read 7 Done Sent Done false
12399 4147 Acked Read 9 Done Sent Done false
12400 4148 Acked Write 1 Done Sent Done false
12401 4149 Acked Write 3 Done Sent Done false
12402 4150 Acked Write 6 Done Sent Done false
12403 4151 Acked Write 8 Done Sent Done false
12404 4152 Acked Flush 0 Done Sent Done false
12405 4153 Acked Write 7 Done Sent Done false
12406 4154 Acked Read 5 Done Sent Done false
12407 4155 Acked Read 10 Done Sent Done false
12408 4156 Acked Read 5 Done Sent Done false
12409 4157 Acked Write 9 Done Sent Done false
12410 4158 Acked Read 1 Done Sent Done false
12411 4159 Acked Write 7 Done Sent Done false
12412 4160 Acked Write 5 Done Sent Done false
12413 4161 Acked Flush 0 Done Sent Done false
12414 4162 Acked Read 6 Done Sent Done false
12415 4163 Acked Read 8 Done Sent Done false
12416 4164 Acked Read 4 Done Sent Done false
12417 4165 Acked Write 10 Done Sent Done false
12418 4166 Acked Write 5 Done Sent Done false
12419 4167 Acked Read 1 Done Sent Done false
12420 4168 Acked Read 2 Done Sent Done false
12421 4169 Acked Write 7 Done Sent Done false
12422 4170 Acked Read 1 Done Sent Done false
12423 4171 Acked Read 2 Done Sent Done false
12424 4172 Acked Read 8 Done Sent Done false
12425 4173 Acked Write 8 Done Sent Done false
12426 4174 Acked Write 1 Done Sent Done false
12427 4175 Acked Read 3 Done Sent Done false
12428 4176 Acked Write 9 Done Sent Done false
12429 4177 Acked Flush 0 Done Sent Done false
12430 4178 Acked Read 2 Done Sent Done false
12431 4179 Acked Read 10 Done Sent Done false
12432 4180 Acked Write 7 Done Sent Done false
12433 4181 Acked Flush 0 Done Sent Done false
12434 4182 Acked Read 8 Done Sent Done false
12435 4183 Acked Read 6 Done Sent Done false
12436 4184 Acked Read 5 Done Sent Done false
12437 4185 Acked Write 10 Done Sent Done false
12438 4186 Acked Read 2 Done Sent Done false
12439 4187 Acked Write 5 Done Sent Done false
12440 4188 Acked Write 9 Done Sent Done false
12441 4189 Acked Write 2 Done Sent Done false
12442 4190 Acked Read 2 Done Sent Done false
12443 4191 Acked Write 3 Done Sent Done false
12444 4192 Acked Flush 0 Done Sent Done false
12445 4193 Acked Read 2 Done Sent Done false
12446 4194 Acked Read 2 Done Sent Done false
12447 4195 Acked Read 8 Done Sent Done false
12448 4196 Acked Write 10 Done Sent Done false
12449 4197 Acked Write 4 Done Sent Done false
12450 4198 Acked Write 10 Done Sent Done false
12451 4199 Acked Read 7 Done Sent Done false
12452 4200 Acked Read 1 Done Sent Done false
12453 4201 Acked Write 6 Done Sent Done false
12454 4202 Acked Flush 0 Done Sent Done false
12455 4203 Acked Read 3 Done Sent Done false
12456 4204 Acked Write 1 Done Sent Done false
12457 4205 Acked Flush 0 Done Sent Done false
12458 4206 Acked Write 8 Done Sent Done false
12459 4207 Acked Write 2 Done Sent Done false
12460 4208 Acked Read 7 Done Sent Done false
12461 4209 Acked Write 3 Done Sent Done false
12462 4210 Acked Read 10 Done Sent Done false
12463 4211 Acked Read 2 Done Sent Done false
12464 4212 Acked Read 4 Done Sent Done false
12465 4213 Acked Read 2 Done Sent Done false
12466 4214 Acked Read 7 Done Sent Done false
12467 4215 Acked Read 4 Done Sent Done false
12468 4216 Acked Read 7 Done Sent Done false
12469 4217 Acked Read 2 Done Sent Done false
12470 4218 Acked Read 2 Done Sent Done false
12471 4219 Acked Read 3 Done Sent Done false
12472 4220 Acked Read 4 Done Sent Done false
12473 4221 Acked Read 4 Done Sent Done false
12474 4222 Acked Read 3 Done Sent Done false
12475 4223 Acked Write 2 Done Sent Done false
12476 4224 Acked Read 4 Done Sent Done false
12477 4225 Acked Read 2 Done Sent Done false
12478 4226 Acked Read 7 Done Sent Done false
12479 4227 Acked Read 9 Done Sent Done false
12480 4228 Acked Flush 0 Done Sent Done false
12481 4229 Acked Read 6 Done Sent Done false
12482 4230 Acked Read 1 Done Sent Done false
12483 4231 Acked Write 6 Done Sent Done false
12484 4232 Acked Read 10 Done Sent Done false
12485 4233 Acked Read 5 Done Sent Done false
12486 4234 Acked Read 6 Done Sent Done false
12487 4235 Acked Read 7 Done Sent Done false
12488 4236 Acked Write 8 Done Sent Done false
12489 4237 Acked Write 9 Done Sent Done false
12490 4238 Acked Flush 0 Done Sent Done false
12491 4239 Acked Read 9 Done Sent Done false
12492 4240 Acked Write 10 Done Sent Done false
12493 4241 Acked Write 10 Done Sent Done false
12494 4242 Acked Write 1 Done Sent Done false
12495 4243 Acked Write 8 Done Sent Done false
12496 4244 Acked Write 10 Done Sent Done false
12497 4245 Acked Read 4 Done Sent Done false
12498 4246 Acked Write 4 Done Sent Done false
12499 4247 Acked Write 5 Done Sent Done false
12500 4248 Acked Write 4 Done Sent Done false
12501 4249 Acked Read 9 Done Sent Done false
12502 4250 Acked Write 4 Done Sent Done false
12503 4251 Acked Write 2 Done Sent Done false
12504 4252 Acked Write 3 Done Sent Done false
12505 4253 Acked Read 1 Done Sent Done false
12506 4254 Acked Read 7 Done Sent Done false
12507 4255 Acked Write 7 Done Sent Done false
12508 4256 Acked Write 5 Done Sent Done false
12509 4257 Acked Read 10 Done Sent Done false
12510 4258 Acked Flush 0 Done Sent Done false
12511 4259 Acked Read 8 Done Sent Done false
12512 4260 Acked Write 4 Done Sent Done false
12513 4261 Acked Read 10 Done Sent Done false
12514 4262 Acked Write 4 Done Sent Done false
12515 4263 Acked Read 2 Done Sent Done false
12516 4264 Acked Write 10 Done Sent Done false
12517 4265 Acked Read 6 Done Sent Done false
12518 4266 Acked Write 5 Done Sent Done false
12519 4267 Acked Write 6 Done Sent Done false
12520 4268 Acked Write 8 Done Sent Done false
12521 4269 Acked Read 10 Done Sent Done false
12522 4270 Acked Write 6 Done Sent Done false
12523 4271 Acked Write 1 Done Sent Done false
12524 4272 Acked Write 1 Done Sent Done false
12525 4273 Acked Write 9 Done Sent Done false
12526 4274 Acked Write 4 Done Sent Done false
12527 4275 Acked Write 6 Done Sent Done false
12528 4276 Acked Read 5 Done Sent Done false
12529 4277 Acked Write 1 Done Sent Done false
12530 4278 Acked Write 5 Done Sent Done false
12531 4279 Acked Write 8 Done Sent Done false
12532 4280 Acked Flush 0 Done Sent Done false
12533 4281 Acked Write 7 Done Sent Done false
12534 4282 Acked Read 1 Done Sent Done false
12535 4283 Acked Write 8 Done Sent Done false
12536 4284 Acked Read 2 Done Sent Done false
12537 4285 Acked Write 5 Done Sent Done false
12538 4286 Acked Read 3 Done Sent Done false
12539 4287 Acked Write 9 Done Sent Done false
12540 4288 Acked Write 10 Done Sent Done false
12541 4289 Acked Read 4 Done Sent Done false
12542 4290 Acked Read 8 Done Sent Done false
12543 4291 Acked Write 7 Done Sent Done false
12544 4292 Acked Read 4 Done Sent Done false
12545 4293 Acked Write 2 Done Sent Done false
12546 4294 Acked Read 8 Done Sent Done false
12547 4295 Acked Write 1 Done Sent Done false
12548 4296 Acked Write 10 Done Sent Done false
12549 4297 Acked Read 10 Done Sent Done false
12550 4298 Acked Write 10 Done Sent Done false
12551 4299 Acked Read 4 Done Sent Done false
12552 4300 Acked Read 10 Done Sent Done false
12553 4301 Acked Write 9 Done Sent Done false
12554 4302 Acked Read 8 Done Sent Done false
12555 4303 Acked Write 8 Done Sent Done false
12556 4304 Acked Write 9 Done Sent Done false
12557 4305 Acked Read 1 Done Sent Done false
12558 4306 Acked Write 3 Done Sent Done false
12559 4307 Acked Flush 0 Done Sent Done false
12560 4308 Acked Read 9 Done Sent Done false
12561 4309 Acked Read 10 Done Sent Done false
12562 4310 Acked Write 5 Done Sent Done false
12563 4311 Acked Read 9 Done Sent Done false
12564 4312 Acked Read 10 Done Sent Done false
12565 4313 Acked Write 1 Done Sent Done false
12566 4314 Acked Read 7 Done Sent Done false
12567 4315 Acked Read 9 Done Sent Done false
12568 4316 Acked Write 5 Done Sent Done false
12569 4317 Acked Read 8 Done Sent Done false
12570 4318 Acked Read 6 Done Sent Done false
12571 4319 Acked Write 1 Done Sent Done false
12572 4320 Acked Write 2 Done Sent Done false
12573 STATES DS:0 DS:1 DS:2 TOTAL
12574 Sent 0 301 0 301
12575 Done 301 0 301 602
12576 Skipped 0 0 0 0
12577 Error 0 0 0 0
12578 Last Flush: Some(JobId(4307)) Some(JobId(4019)) Some(JobId(4307))
12579 Downstairs last five completed: 4019 4018 4017 4016 4015
12580 Upstairs last five completed: 4320 4319 4318 4317 4316
12581 CLIENT: Up:0 ds:301 act:2
125822025-01-29T05:13:27.183ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
125832025-01-29T05:13:28.185ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
125842025-01-29T05:13:29.187ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
12585 [8820] got start action
12586 [8820] I am going to restart
12587 Starting downstairs at port 8820
12588 Make output file at "/tmp/dsc/downstairs-8820.txt"
12589 [8820][1] reports Starting
12590 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
12591 [8820][1] reports Running
12592 ----------------------------------------------------------------
12593 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12594 States: Active Offline Active
12595 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12596 4020 Acked Write 6 Done Sent Done false
12597 4021 Acked Write 8 Done Sent Done false
12598 4022 Acked Write 4 Done Sent Done false
12599 4023 Acked Write 5 Done Sent Done false
12600 4024 Acked Read 7 Done Sent Done false
12601 4025 Acked Read 6 Done Sent Done false
12602 4026 Acked Read 6 Done Sent Done false
12603 4027 Acked Write 2 Done Sent Done false
12604 4028 Acked Write 4 Done Sent Done false
12605 4029 Acked Write 10 Done Sent Done false
12606 4030 Acked Read 8 Done Sent Done false
12607 4031 Acked Read 10 Done Sent Done false
12608 4032 Acked Read 4 Done Sent Done false
12609 4033 Acked Flush 0 Done Sent Done false
12610 4034 Acked Read 10 Done Sent Done false
12611 4035 Acked Read 8 Done Sent Done false
12612 4036 Acked Read 1 Done Sent Done false
12613 4037 Acked Write 6 Done Sent Done false
12614 4038 Acked Write 10 Done Sent Done false
12615 4039 Acked Read 3 Done Sent Done false
12616 4040 Acked Write 4 Done Sent Done false
12617 4041 Acked Read 6 Done Sent Done false
12618 4042 Acked Flush 0 Done Sent Done false
12619 4043 Acked Read 4 Done Sent Done false
12620 4044 Acked Read 3 Done Sent Done false
12621 4045 Acked Read 6 Done Sent Done false
12622 4046 Acked Read 5 Done Sent Done false
12623 4047 Acked Write 7 Done Sent Done false
12624 4048 Acked Read 9 Done Sent Done false
12625 4049 Acked Write 7 Done Sent Done false
12626 4050 Acked Read 8 Done Sent Done false
12627 4051 Acked Flush 0 Done Sent Done false
12628 4052 Acked Write 2 Done Sent Done false
12629 4053 Acked Read 5 Done Sent Done false
12630 4054 Acked Write 2 Done Sent Done false
12631 4055 Acked Read 2 Done Sent Done false
12632 4056 Acked Read 3 Done Sent Done false
12633 4057 Acked Read 7 Done Sent Done false
12634 4058 Acked Write 1 Done Sent Done false
12635 4059 Acked Read 3 Done Sent Done false
12636 4060 Acked Write 5 Done Sent Done false
12637 4061 Acked Read 1 Done Sent Done false
12638 4062 Acked Read 10 Done Sent Done false
12639 4063 Acked Read 10 Done Sent Done false
12640 4064 Acked Write 3 Done Sent Done false
12641 4065 Acked Write 8 Done Sent Done false
12642 4066 Acked Write 10 Done Sent Done false
12643 4067 Acked Flush 0 Done Sent Done false
12644 4068 Acked Write 10 Done Sent Done false
12645 4069 Acked Read 5 Done Sent Done false
12646 4070 Acked Read 6 Done Sent Done false
12647 4071 Acked Read 9 Done Sent Done false
12648 4072 Acked Write 2 Done Sent Done false
12649 4073 Acked Write 7 Done Sent Done false
12650 4074 Acked Read 2 Done Sent Done false
12651 4075 Acked Read 8 Done Sent Done false
12652 4076 Acked Write 10 Done Sent Done false
12653 4077 Acked Read 4 Done Sent Done false
12654 4078 Acked Read 10 Done Sent Done false
12655 4079 Acked Read 3 Done Sent Done false
12656 4080 Acked Flush 0 Done Sent Done false
12657 4081 Acked Write 10 Done Sent Done false
12658 4082 Acked Write 9 Done Sent Done false
12659 4083 Acked Read 2 Done Sent Done false
12660 4084 Acked Read 8 Done Sent Done false
12661 4085 Acked Read 9 Done Sent Done false
12662 4086 Acked Read 3 Done Sent Done false
12663 4087 Acked Read 7 Done Sent Done false
12664 4088 Acked Read 2 Done Sent Done false
12665 4089 Acked Write 2 Done Sent Done false
12666 4090 Acked Write 10 Done Sent Done false
12667 4091 Acked Read 8 Done Sent Done false
12668 4092 Acked Write 3 Done Sent Done false
12669 4093 Acked Read 5 Done Sent Done false
12670 4094 Acked Read 4 Done Sent Done false
12671 4095 Acked Read 5 Done Sent Done false
12672 4096 Acked Flush 0 Done Sent Done false
12673 4097 Acked Read 9 Done Sent Done false
12674 4098 Acked Write 7 Done Sent Done false
12675 4099 Acked Flush 0 Done Sent Done false
12676 4100 Acked Write 5 Done Sent Done false
12677 4101 Acked Flush 0 Done Sent Done false
12678 4102 Acked Flush 0 Done Sent Done false
12679 4103 Acked Write 5 Done Sent Done false
12680 4104 Acked Read 5 Done Sent Done false
12681 4105 Acked Read 7 Done Sent Done false
12682 4106 Acked Write 7 Done Sent Done false
12683 4107 Acked Read 8 Done Sent Done false
12684 4108 Acked Write 4 Done Sent Done false
12685 4109 Acked Write 9 Done Sent Done false
12686 4110 Acked Write 6 Done Sent Done false
12687 4111 Acked Write 4 Done Sent Done false
12688 4112 Acked Read 2 Done Sent Done false
12689 4113 Acked Read 1 Done Sent Done false
12690 4114 Acked Write 8 Done Sent Done false
12691 4115 Acked Read 5 Done Sent Done false
12692 4116 Acked Read 1 Done Sent Done false
12693 4117 Acked Flush 0 Done Sent Done false
12694 4118 Acked Read 6 Done Sent Done false
12695 4119 Acked Write 4 Done Sent Done false
12696 4120 Acked Write 5 Done Sent Done false
12697 4121 Acked Read 2 Done Sent Done false
12698 4122 Acked Read 6 Done Sent Done false
12699 4123 Acked Write 2 Done Sent Done false
12700 4124 Acked Write 5 Done Sent Done false
12701 4125 Acked Write 1 Done Sent Done false
12702 4126 Acked Read 5 Done Sent Done false
12703 4127 Acked Read 10 Done Sent Done false
12704 4128 Acked Read 2 Done Sent Done false
12705 4129 Acked Read 10 Done Sent Done false
12706 4130 Acked Write 6 Done Sent Done false
12707 4131 Acked Write 4 Done Sent Done false
12708 4132 Acked Flush 0 Done Sent Done false
12709 4133 Acked Read 2 Done Sent Done false
12710 4134 Acked Read 5 Done Sent Done false
12711 4135 Acked Read 6 Done Sent Done false
12712 4136 Acked Read 4 Done Sent Done false
12713 4137 Acked Write 1 Done Sent Done false
12714 4138 Acked Read 5 Done Sent Done false
12715 4139 Acked Read 2 Done Sent Done false
12716 4140 Acked Write 7 Done Sent Done false
12717 4141 Acked Read 4 Done Sent Done false
12718 4142 Acked Read 5 Done Sent Done false
12719 4143 Acked Read 6 Done Sent Done false
12720 4144 Acked Read 7 Done Sent Done false
12721 4145 Acked Write 5 Done Sent Done false
12722 4146 Acked Read 7 Done Sent Done false
12723 4147 Acked Read 9 Done Sent Done false
12724 4148 Acked Write 1 Done Sent Done false
12725 4149 Acked Write 3 Done Sent Done false
12726 4150 Acked Write 6 Done Sent Done false
12727 4151 Acked Write 8 Done Sent Done false
12728 4152 Acked Flush 0 Done Sent Done false
12729 4153 Acked Write 7 Done Sent Done false
12730 4154 Acked Read 5 Done Sent Done false
12731 4155 Acked Read 10 Done Sent Done false
12732 4156 Acked Read 5 Done Sent Done false
12733 4157 Acked Write 9 Done Sent Done false
12734 4158 Acked Read 1 Done Sent Done false
12735 4159 Acked Write 7 Done Sent Done false
12736 4160 Acked Write 5 Done Sent Done false
12737 4161 Acked Flush 0 Done Sent Done false
12738 4162 Acked Read 6 Done Sent Done false
12739 4163 Acked Read 8 Done Sent Done false
12740 4164 Acked Read 4 Done Sent Done false
12741 4165 Acked Write 10 Done Sent Done false
12742 4166 Acked Write 5 Done Sent Done false
12743 4167 Acked Read 1 Done Sent Done false
12744 4168 Acked Read 2 Done Sent Done false
12745 4169 Acked Write 7 Done Sent Done false
12746 4170 Acked Read 1 Done Sent Done false
12747 4171 Acked Read 2 Done Sent Done false
12748 4172 Acked Read 8 Done Sent Done false
12749 4173 Acked Write 8 Done Sent Done false
12750 4174 Acked Write 1 Done Sent Done false
12751 4175 Acked Read 3 Done Sent Done false
12752 4176 Acked Write 9 Done Sent Done false
12753 4177 Acked Flush 0 Done Sent Done false
12754 4178 Acked Read 2 Done Sent Done false
12755 4179 Acked Read 10 Done Sent Done false
12756 4180 Acked Write 7 Done Sent Done false
12757 4181 Acked Flush 0 Done Sent Done false
12758 4182 Acked Read 8 Done Sent Done false
12759 4183 Acked Read 6 Done Sent Done false
12760 4184 Acked Read 5 Done Sent Done false
12761 4185 Acked Write 10 Done Sent Done false
12762 4186 Acked Read 2 Done Sent Done false
12763 4187 Acked Write 5 Done Sent Done false
12764 4188 Acked Write 9 Done Sent Done false
12765 4189 Acked Write 2 Done Sent Done false
12766 4190 Acked Read 2 Done Sent Done false
12767 4191 Acked Write 3 Done Sent Done false
12768 4192 Acked Flush 0 Done Sent Done false
12769 4193 Acked Read 2 Done Sent Done false
12770 4194 Acked Read 2 Done Sent Done false
12771 4195 Acked Read 8 Done Sent Done false
12772 4196 Acked Write 10 Done Sent Done false
12773 4197 Acked Write 4 Done Sent Done false
12774 4198 Acked Write 10 Done Sent Done false
12775 4199 Acked Read 7 Done Sent Done false
12776 4200 Acked Read 1 Done Sent Done false
12777 4201 Acked Write 6 Done Sent Done false
12778 4202 Acked Flush 0 Done Sent Done false
12779 4203 Acked Read 3 Done Sent Done false
12780 4204 Acked Write 1 Done Sent Done false
12781 4205 Acked Flush 0 Done Sent Done false
12782 4206 Acked Write 8 Done Sent Done false
12783 4207 Acked Write 2 Done Sent Done false
12784 4208 Acked Read 7 Done Sent Done false
12785 4209 Acked Write 3 Done Sent Done false
12786 4210 Acked Read 10 Done Sent Done false
12787 4211 Acked Read 2 Done Sent Done false
12788 4212 Acked Read 4 Done Sent Done false
12789 4213 Acked Read 2 Done Sent Done false
12790 4214 Acked Read 7 Done Sent Done false
12791 4215 Acked Read 4 Done Sent Done false
12792 4216 Acked Read 7 Done Sent Done false
12793 4217 Acked Read 2 Done Sent Done false
12794 4218 Acked Read 2 Done Sent Done false
12795 4219 Acked Read 3 Done Sent Done false
12796 4220 Acked Read 4 Done Sent Done false
12797 4221 Acked Read 4 Done Sent Done false
12798 4222 Acked Read 3 Done Sent Done false
12799 4223 Acked Write 2 Done Sent Done false
12800 4224 Acked Read 4 Done Sent Done false
12801 4225 Acked Read 2 Done Sent Done false
12802 4226 Acked Read 7 Done Sent Done false
12803 4227 Acked Read 9 Done Sent Done false
12804 4228 Acked Flush 0 Done Sent Done false
12805 4229 Acked Read 6 Done Sent Done false
12806 4230 Acked Read 1 Done Sent Done false
12807 4231 Acked Write 6 Done Sent Done false
12808 4232 Acked Read 10 Done Sent Done false
12809 4233 Acked Read 5 Done Sent Done false
12810 4234 Acked Read 6 Done Sent Done false
12811 4235 Acked Read 7 Done Sent Done false
12812 4236 Acked Write 8 Done Sent Done false
12813 4237 Acked Write 9 Done Sent Done false
12814 4238 Acked Flush 0 Done Sent Done false
12815 4239 Acked Read 9 Done Sent Done false
12816 4240 Acked Write 10 Done Sent Done false
12817 4241 Acked Write 10 Done Sent Done false
12818 4242 Acked Write 1 Done Sent Done false
12819 4243 Acked Write 8 Done Sent Done false
12820 4244 Acked Write 10 Done Sent Done false
12821 4245 Acked Read 4 Done Sent Done false
12822 4246 Acked Write 4 Done Sent Done false
12823 4247 Acked Write 5 Done Sent Done false
12824 4248 Acked Write 4 Done Sent Done false
12825 4249 Acked Read 9 Done Sent Done false
12826 4250 Acked Write 4 Done Sent Done false
12827 4251 Acked Write 2 Done Sent Done false
12828 4252 Acked Write 3 Done Sent Done false
12829 4253 Acked Read 1 Done Sent Done false
12830 4254 Acked Read 7 Done Sent Done false
12831 4255 Acked Write 7 Done Sent Done false
12832 4256 Acked Write 5 Done Sent Done false
12833 4257 Acked Read 10 Done Sent Done false
12834 4258 Acked Flush 0 Done Sent Done false
12835 4259 Acked Read 8 Done Sent Done false
12836 4260 Acked Write 4 Done Sent Done false
12837 4261 Acked Read 10 Done Sent Done false
12838 4262 Acked Write 4 Done Sent Done false
12839 4263 Acked Read 2 Done Sent Done false
12840 4264 Acked Write 10 Done Sent Done false
12841 4265 Acked Read 6 Done Sent Done false
12842 4266 Acked Write 5 Done Sent Done false
12843 4267 Acked Write 6 Done Sent Done false
12844 4268 Acked Write 8 Done Sent Done false
12845 4269 Acked Read 10 Done Sent Done false
12846 4270 Acked Write 6 Done Sent Done false
12847 4271 Acked Write 1 Done Sent Done false
12848 4272 Acked Write 1 Done Sent Done false
12849 4273 Acked Write 9 Done Sent Done false
12850 4274 Acked Write 4 Done Sent Done false
12851 4275 Acked Write 6 Done Sent Done false
12852 4276 Acked Read 5 Done Sent Done false
12853 4277 Acked Write 1 Done Sent Done false
12854 4278 Acked Write 5 Done Sent Done false
12855 4279 Acked Write 8 Done Sent Done false
12856 4280 Acked Flush 0 Done Sent Done false
12857 4281 Acked Write 7 Done Sent Done false
12858 4282 Acked Read 1 Done Sent Done false
12859 4283 Acked Write 8 Done Sent Done false
12860 4284 Acked Read 2 Done Sent Done false
12861 4285 Acked Write 5 Done Sent Done false
12862 4286 Acked Read 3 Done Sent Done false
12863 4287 Acked Write 9 Done Sent Done false
12864 4288 Acked Write 10 Done Sent Done false
12865 4289 Acked Read 4 Done Sent Done false
12866 4290 Acked Read 8 Done Sent Done false
12867 4291 Acked Write 7 Done Sent Done false
12868 4292 Acked Read 4 Done Sent Done false
12869 4293 Acked Write 2 Done Sent Done false
12870 4294 Acked Read 8 Done Sent Done false
12871 4295 Acked Write 1 Done Sent Done false
12872 4296 Acked Write 10 Done Sent Done false
12873 4297 Acked Read 10 Done Sent Done false
12874 4298 Acked Write 10 Done Sent Done false
12875 4299 Acked Read 4 Done Sent Done false
12876 4300 Acked Read 10 Done Sent Done false
12877 4301 Acked Write 9 Done Sent Done false
12878 4302 Acked Read 8 Done Sent Done false
12879 4303 Acked Write 8 Done Sent Done false
12880 4304 Acked Write 9 Done Sent Done false
12881 4305 Acked Read 1 Done Sent Done false
12882 4306 Acked Write 3 Done Sent Done false
12883 4307 Acked Flush 0 Done Sent Done false
12884 4308 Acked Read 9 Done Sent Done false
12885 4309 Acked Read 10 Done Sent Done false
12886 4310 Acked Write 5 Done Sent Done false
12887 4311 Acked Read 9 Done Sent Done false
12888 4312 Acked Read 10 Done Sent Done false
12889 4313 Acked Write 1 Done Sent Done false
12890 4314 Acked Read 7 Done Sent Done false
12891 4315 Acked Read 9 Done Sent Done false
12892 4316 Acked Write 5 Done Sent Done false
12893 4317 Acked Read 8 Done Sent Done false
12894 4318 Acked Read 6 Done Sent Done false
12895 4319 Acked Write 1 Done Sent Done false
12896 4320 Acked Write 2 Done Sent Done false
12897 STATES DS:0 DS:1 DS:2 TOTAL
12898 Sent 0 301 0 301
12899 Done 301 0 301 602
12900 Skipped 0 0 0 0
12901 Error 0 0 0 0
12902 Last Flush: Some(JobId(4307)) Some(JobId(4019)) Some(JobId(4307))
12903 Downstairs last five completed: 4019 4018 4017 4016 4015
12904 Upstairs last five completed: 4320 4319 4318 4317 4316
12905 CLIENT: Up:0 ds:301 act:2
129062025-01-29T05:13:31.191ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129072025-01-29T05:13:31.960ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129082025-01-29T05:13:31.961ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43831) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129092025-01-29T05:13:31.962ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129102025-01-29T05:13:31.962ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129112025-01-29T05:13:31.962ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129122025-01-29T05:13:31.962ZINFOcrucible: send last flush ID to this DS: Some(JobId(4019)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129132025-01-29T05:13:31.962ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129142025-01-29T05:13:31.962ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4019)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129152025-01-29T05:13:31.962ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4019)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
12916 ----------------------------------------------------------------
12917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12918 Downstairs last five completed: 4321 4320 4319 4318 4317
12919 Upstairs last five completed: 4321 4320 4319 4318 4317
12920 CLIENT: Up:0 ds:0 act:3
12921 Replay: All jobs finished
12922 Main task has work to do, go find it
12923 got dsc Stop(0)
12924 stop 0
12925 [8810] Got stop action so:false kr:false
12926 Jan 29 05:13:34.212 INFO request completed, latency_us: 334, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 55fd1aa5-de0f-47aa-a81a-3daf4778894e, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
129272025-01-29T05:13:34.212ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
12928 Jan 29 05:13:34.212 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/0, method: GET, req_id: 129f905f-e28d-4200-a320-2b42605c3356, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
129292025-01-29T05:13:34.212ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129302025-01-29T05:13:34.212ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
129312025-01-29T05:13:34.213ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
12932 [8810] Exited with: ExitStatus(unix_wait_status(9))
12933 [8810][0] reports Exit
12934 Jan 29 05:13:38.214 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/0, method: GET, req_id: ec8fe3bc-9ef1-445c-b74e-37269bd1133b, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
12935 001/300 Flush
12936 002/300 Write block 2068 len 4096 data: 4
12937 003/300 Read block 3796 len 20480
12938 004/300 Write block 539 len 32768 data: 3 3 3 4 4 4 4 5
12939 005/300 Write block 3235 len 12288 data: 5 6 6
12940 006/300 Read block 105 len 28672
12941 007/300 Write block 366 len 20480 data: 4 4 4 4 3
12942 008/300 Read block 379 len 16384
12943 009/300 Write block 2864 len 32768 data: 2 3 3 3 3 3 2 2
12944 010/300 Write block 3503 len 12288 data: 4 4 4
12945 011/300 Write block 1758 len 12288 data: 3 3 3
12946 012/300 Read block 1876 len 40960
12947 013/300 Flush
12948 014/300 Read block 3051 len 16384
12949 015/300 Read block 4856 len 28672
12950 016/300 Read block 4450 len 36864
12951 017/300 Read block 4563 len 16384
12952 018/300 Read block 1580 len 8192
12953 019/300 Read block 1701 len 24576
12954 020/300 Write block 1653 len 4096 data: 2
12955 021/300 Write block 740 len 20480 data: 6 5 6 6 9
12956 022/300 Read block 2107 len 40960
12957 023/300 Write block 2695 len 36864 data: 4 4 4 4 4 3 4 3 3
12958 024/300 Read block 3483 len 4096
12959 025/300 Write block 2034 len 20480 data: 4 4 2 2 2
12960 026/300 Flush
12961 027/300 Read block 3349 len 28672
12962 028/300 Read block 4826 len 36864
12963 029/300 Write block 1320 len 4096 data: 2
12964 030/300 Read block 1727 len 36864
12965 031/300 Read block 1157 len 16384
12966 032/300 Read block 387 len 8192
12967 033/300 Flush
12968 034/300 Write block 2020 len 32768 data: 4 2 2 2 2 2 2 2
12969 035/300 Read block 455 len 36864
12970 036/300 Read block 4506 len 8192
12971 037/300 Flush
12972 038/300 Read block 4478 len 4096
12973 039/300 Write block 1773 len 36864 data: 4 5 6 6 5 5 5 4 3
12974 040/300 Read block 1756 len 4096
12975 041/300 Read block 2507 len 4096
12976 042/300 Read block 3567 len 16384
12977 043/300 Read block 4045 len 36864
12978 044/300 Flush
12979 045/300 Read block 4463 len 4096
12980 046/300 Flush
12981 047/300 Write block 3701 len 8192 data: 3 4
12982 048/300 Read block 3571 len 12288
12983 049/300 Write block 2140 len 12288 data: 6 7 6
12984 050/300 Read block 953 len 12288
12985 051/300 Read block 2881 len 20480
12986 052/300 Read block 4377 len 12288
12987 053/300 Read block 3773 len 32768
12988 054/300 Read block 698 len 36864
12989 055/300 Flush
12990 056/300 Write block 1823 len 8192 data: 3 2
12991 057/300 Write block 670 len 40960 data: 3 3 3 3 3 3 3 3 2 2
12992 058/300 Read block 1572 len 40960
12993 059/300 Flush
12994 060/300 Read block 1065 len 28672
12995 061/300 Read block 1086 len 32768
12996 062/300 Read block 3446 len 20480
12997 063/300 Write block 1298 len 24576 data: 4 7 7 7 7 6
12998 064/300 Write block 3572 len 28672 data: 3 3 4 5 6 6 6
12999 065/300 Read block 4627 len 28672
13000 066/300 Write block 2061 len 12288 data: 3 3 2
13001 067/300 Read block 2092 len 12288
13002 068/300 Read block 4977 len 24576
13003 069/300 Read block 793 len 24576
13004 070/300 Read block 2616 len 28672
13005 071/300 Read block 4009 len 32768
13006 072/300 Write block 1948 len 8192 data: 3 3
13007 073/300 Read block 4506 len 36864
13008 074/300 Write block 491 len 40960 data: 5 6 6 6 5 5 5 5 4 4
13009 075/300 Write block 1991 len 12288 data: 4 4 2
13010 076/300 Write block 1119 len 12288 data: 3 2 2
13011 077/300 Write block 2411 len 8192 data: 3 4
13012 078/300 Write block 4766 len 12288 data: 3 3 4
13013 079/300 Write block 841 len 32768 data: 3 4 5 5 5 6 7 7
13014 080/300 Read block 4480 len 16384
13015 081/300 Read block 565 len 40960
13016 082/300 Read block 3850 len 40960
13017 083/300 Read block 1245 len 28672
13018 084/300 Write block 2302 len 40960 data: 3 2 2 2 2 2 2 3 4 4
13019 085/300 Read block 260 len 8192
13020 086/300 Read block 3855 len 12288
13021 087/300 Read block 3533 len 36864
13022 088/300 Read block 1751 len 4096
13023 089/300 Flush
13024 090/300 Read block 301 len 4096
13025 091/300 Read block 3955 len 4096
13026 092/300 Read block 4190 len 24576
13027 093/300 Read block 3887 len 36864
13028 094/300 Read block 2924 len 4096
13029 095/300 Flush
13030 096/300 Write block 1554 len 12288 data: 4 4 2
13031 097/300 Write block 3631 len 8192 data: 3 3
13032 098/300 Flush
13033 099/300 Read block 1227 len 8192
13034 100/300 Read block 1230 len 20480
13035 101/300 Write block 4712 len 24576 data: 3 3 3 3 3 2
13036 102/300 Flush
13037 103/300 Write block 334 len 28672 data: 3 2 2 2 2 2 2
13038 104/300 Read block 264 len 4096
13039 105/300 Read block 2516 len 20480
13040 106/300 Write block 128 len 4096 data: 3
13041 107/300 Read block 2962 len 12288
13042 108/300 Read block 378 len 36864
13043 109/300 Write block 2984 len 36864 data: 5 4 3 3 2 2 2 2 2
13044 110/300 Read block 3369 len 36864
13045 111/300 Read block 4473 len 16384
13046 112/300 Read block 1828 len 16384
13047 113/300 Write block 3671 len 16384 data: 3 3 3 3
13048 114/300 Write block 2141 len 20480 data: 8 7 5 5 4
13049 115/300 Read block 3556 len 24576
13050 116/300 Write block 3724 len 4096 data: 2
13051 117/300 Read block 1271 len 32768
13052 118/300 Read block 1474 len 24576
13053 119/300 Read block 27 len 28672
13054 120/300 Read block 1047 len 40960
13055 121/300 Read block 1087 len 12288
13056 122/300 Write block 402 len 40960 data: 2 2 3 3 3 3 4 4 4 4
13057 123/300 Read block 2796 len 4096
13058 124/300 Read block 4088 len 12288
13059 125/300 Read block 1897 len 4096
13060 126/300 Flush
13061 127/300 Write block 27 len 12288 data: 2 2 3
13062 128/300 Write block 1792 len 28672 data: 4 3 4 4 3 3 3
13063 129/300 Write block 4498 len 12288 data: 5 4 5
13064 130/300 Read block 1060 len 28672
13065 131/300 Read block 3122 len 32768
13066 132/300 Write block 1841 len 32768 data: 4 4 4 3 3 3 3 2
13067 133/300 Flush
13068 134/300 Write block 3541 len 24576 data: 2 2 2 4 3 3
13069 135/300 Read block 1771 len 20480
13070 136/300 Write block 634 len 12288 data: 3 4 4
13071 137/300 Read block 1939 len 20480
13072 138/300 Read block 3000 len 36864
13073 139/300 Read block 4906 len 20480
13074 140/300 Write block 1651 len 28672 data: 2 2 3 2 2 2 2
13075 141/300 Read block 3151 len 12288
13076 142/300 Read block 3802 len 28672
13077 143/300 Read block 4776 len 32768
13078 144/300 Read block 4607 len 20480
13079 145/300 Write block 1612 len 24576 data: 3 3 3 3 2 3
13080 146/300 Write block 4736 len 4096 data: 5
13081 147/300 Read block 1016 len 28672
13082 148/300 Write block 4539 len 24576 data: 4 3 3 3 3 3
13083 149/300 Read block 654 len 28672
13084 150/300 Read block 324 len 16384
13085 151/300 Flush
13086 152/300 Write block 1175 len 8192 data: 4 4
13087 153/300 Write block 4144 len 8192 data: 4 4
13088 154/300 Read block 2072 len 12288
13089 155/300 Read block 4237 len 8192
13090 156/300 Write block 2466 len 24576 data: 3 3 4 5 5 4
13091 157/300 Read block 3008 len 28672
13092 158/300 Flush
13093 159/300 Read block 4370 len 4096
13094 160/300 Read block 695 len 28672
13095 161/300 Read block 4811 len 40960
13096 162/300 Read block 819 len 24576
13097 163/300 Read block 3651 len 12288
13098 164/300 Write block 4715 len 8192 data: 4 4
13099 165/300 Write block 1908 len 16384 data: 4 4 4 5
13100 166/300 Read block 4343 len 24576
13101 167/300 Write block 2946 len 40960 data: 6 6 6 4 4 3 2 3 4 3
13102 168/300 Write block 2297 len 4096 data: 4
13103 169/300 Write block 3776 len 20480 data: 3 3 3 3 3
13104 170/300 Write block 4388 len 12288 data: 4 3 3
13105 171/300 Write block 3723 len 16384 data: 3 3 2 3
13106 172/300 Read block 1544 len 20480
13107 173/300 Read block 3629 len 32768
13108 174/300 Read block 698 len 12288
13109 175/300 Write block 4975 len 20480 data: 5 4 4 4 4
13110 176/300 Read block 2323 len 20480
13111 177/300 Read block 1742 len 4096
13112 178/300 Read block 3443 len 12288
13113 179/300 Write block 1041 len 28672 data: 5 6 6 6 6 5 4
13114 180/300 Write block 4882 len 40960 data: 3 3 4 4 6 5 5 5 5 3
13115 181/300 Read block 627 len 20480
13116 182/300 Read block 654 len 8192
13117 183/300 Flush
13118 184/300 Read block 1769 len 36864
13119 185/300 Write block 4226 len 24576 data: 3 4 3 3 2 2
13120 186/300 Write block 2406 len 32768 data: 3 3 3 4 3 4 5 5
13121 187/300 Write block 3348 len 8192 data: 3 3
13122 188/300 Flush
13123 189/300 Write block 3640 len 4096 data: 3
13124 190/300 Read block 41 len 40960
13125 191/300 Write block 2985 len 36864 data: 5 4 4 3 3 3 3 3 2
13126 192/300 Read block 1823 len 4096
13127 193/300 Write block 2805 len 32768 data: 3 2 2 2 4 5 5 6
13128 194/300 Read block 2315 len 32768
13129 195/300 Write block 4487 len 4096 data: 2
13130 196/300 Read block 3432 len 16384
13131 197/300 Flush
13132 198/300 Read block 2807 len 40960
13133 199/300 Read block 2806 len 28672
13134 200/300 Read block 75 len 32768
13135 201/300 Write block 1241 len 8192 data: 3 4
13136 202/300 Read block 1198 len 40960
13137 203/300 Write block 2758 len 40960 data: 6 7 7 6 6 4 2 2 2 2
13138 204/300 Write block 2026 len 36864 data: 3 3 3 4 4 4 4 4 5
13139 205/300 Write block 422 len 12288 data: 3 3 3
13140 206/300 Write block 2155 len 4096 data: 4
13141 207/300 Write block 2780 len 36864 data: 6 5 5 5 4 3 3 3 2
13142 208/300 Write block 1928 len 40960 data: 2 2 2 3 2 2 2 3 4 6
13143 209/300 Write block 2925 len 36864 data: 4 4 4 4 4 4 4 4 2
13144 210/300 Flush
13145 211/300 Read block 553 len 40960
13146 212/300 Write block 1762 len 32768 data: 2 2 3 3 3 2 2 3
13147 213/300 Flush
13148 214/300 Write block 1398 len 16384 data: 3 3 3 3
13149 215/300 Write block 4327 len 36864 data: 3 3 5 5 5 2 3 3 3
13150 216/300 Write block 2170 len 24576 data: 3 4 4 3 2 2
13151 217/300 Write block 4474 len 40960 data: 3 3 4 4 3 3 3 3 2 2
13152 218/300 Write block 3042 len 20480 data: 2 2 2 2 3
13153 219/300 Write block 338 len 28672 data: 3 3 3 2 2 3 3
13154 220/300 Write block 481 len 40960 data: 6 5 5 4 4 4 4 2 2 3
13155 221/300 Read block 3705 len 36864
13156 222/300 Flush
13157 223/300 Flush
13158 224/300 Read block 2543 len 36864
13159 225/300 Read block 3539 len 36864
13160 226/300 Write block 2447 len 36864 data: 2 3 3 3 3 4 4 5 6
13161 227/300 Read block 2023 len 8192
13162 228/300 Read block 1521 len 24576
13163 229/300 Write block 60 len 12288 data: 3 3 3
13164 230/300 Write block 619 len 32768 data: 2 2 2 2 2 2 2 2
13165 231/300 Read block 1695 len 16384
13166 232/300 Read block 2983 len 20480
13167 233/300 Read block 868 len 28672
13168 234/300 Read block 598 len 12288
13169 235/300 Read block 3287 len 24576
13170 236/300 Read block 1022 len 16384
13171 237/300 Write block 3762 len 32768 data: 2 3 4 5 6 7 7 6
13172 238/300 Read block 1347 len 36864
13173 239/300 Write block 1969 len 12288 data: 5 5 4
13174 240/300 Write block 3230 len 32768 data: 5 6 6 6 5 6 7 7
13175 241/300 Write block 3200 len 20480 data: 3 3 3 3 3
13176 242/300 Read block 1791 len 28672
13177 243/300 Read block 4121 len 36864
13178 244/300 Read block 4574 len 20480
13179 245/300 Write block 4750 len 36864 data: 3 3 2 2 2 2 2 3 3
13180 246/300 Read block 3515 len 12288
13181 247/300 Write block 3122 len 32768 data: 5 5 4 3 3 3 3 4
13182 248/300 Read block 30 len 16384
13183 249/300 Write block 3386 len 24576 data: 3 3 2 3 3 3
13184 250/300 Write block 2885 len 16384 data: 3 3 3 3
13185 251/300 Write block 2976 len 20480 data: 7 6 6 5 4
13186 252/300 Read block 4960 len 32768
13187 253/300 Flush
13188 254/300 Read block 1170 len 8192
13189 255/300 Read block 1700 len 36864
13190 256/300 Write block 4012 len 8192 data: 4 3
13191 257/300 Write block 2119 len 32768 data: 2 2 2 2 2 2 2 2
13192 258/300 Read block 4161 len 16384
13193 259/300 Read block 755 len 16384
13194 260/300 Read block 2691 len 12288
13195 261/300 Read block 3349 len 12288
13196 262/300 Read block 2340 len 24576
13197 263/300 Read block 4650 len 24576
13198 264/300 Read block 3343 len 24576
13199 265/300 Read block 1369 len 16384
13200 266/300 Read block 4315 len 40960
13201 267/300 Write block 4478 len 12288 data: 4 4 4
13202 268/300 Read block 158 len 24576
13203 269/300 Write block 4398 len 28672 data: 5 5 4 5 5 5 5
13204 270/300 Write block 1685 len 36864 data: 7 4 6 7 7 9 9 5 6
13205 271/300 Write block 3008 len 20480 data: 3 3 2 2 2
13206 272/300 Read block 923 len 12288
13207 273/300 Write block 3774 len 16384 data: 3 3 4 4
13208 274/300 Read block 3110 len 24576
13209 275/300 Read block 3696 len 36864
13210 276/300 Flush
13211 277/300 Flush
13212 278/300 Read block 802 len 40960
13213 279/300 Write block 1838 len 4096 data: 3
13214 280/300 Write block 1328 len 24576 data: 4 2 2 2 2 2
13215 281/300 Read block 577 len 4096
13216 282/300 Read block 635 len 40960
13217 283/300 Flush
13218 284/300 Write block 4541 len 12288 data: 4 4 4
13219 285/300 Write block 1805 len 40960 data: 4 4 3 3 3 3 3 3 3 2
13220 286/300 Write block 1008 len 40960 data: 3 4 4 4 4 4 2 2 2 2
13221 287/300 Write block 263 len 24576 data: 2 2 2 2 3 3
13222 288/300 Write block 4080 len 16384 data: 3 3 3 4
13223 289/300 Write block 577 len 24576 data: 6 4 3 3 3 4
13224 290/300 Read block 2101 len 32768
13225 291/300 Read block 1555 len 24576
13226 292/300 Write block 1649 len 40960 data: 2 2 3 3 4 3 3 3 3 2
13227 293/300 Read block 3466 len 32768
13228 294/300 Read block 3204 len 8192
13229 295/300 Read block 3125 len 32768
13230 296/300 Flush
13231 297/300 Write block 4554 len 12288 data: 3 4 4
13232 298/300 Flush
13233 299/300 Read block 4079 len 20480
13234 300/300 Write block 344 len 36864 data: 4 3 3 4 4 3 4 3 3
13235 301/300 Write block 2835 len 4096 data: 4
13236 Main task has work to do, go find it
13237 got dsc Start(0)
13238 start 0
13239 Jan 29 05:13:38.458 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1ead515c-0b1a-4d10-915c-9475db97dc8c, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
13240 Replay: started 0, returned:Ok(())
13241 ----------------------------------------------------------------
13242 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13243 States: Offline Active Active
13244 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13245 4322 Acked Flush 0 Sent Done Done false
13246 4323 Acked Write 1 Sent Done Done false
13247 4324 Acked Read 5 Sent Done Done false
13248 4325 Acked Write 8 Sent Done Done false
13249 4326 Acked Write 3 Sent Done Done false
13250 4327 Acked Read 7 Sent Done Done false
13251 4328 Acked Write 5 Sent Done Done false
13252 4329 Acked Read 4 Sent Done Done false
13253 4330 Acked Write 8 Sent Done Done false
13254 4331 Acked Write 3 Sent Done Done false
13255 4332 Acked Write 3 Sent Done Done false
13256 4333 Acked Read 10 Sent Done Done false
13257 4334 Acked Flush 0 Sent Done Done false
13258 4335 Acked Read 4 Sent Done Done false
13259 4336 Acked Read 7 Sent Done Done false
13260 4337 Acked Read 9 Sent Done Done false
13261 4338 Acked Read 4 Sent Done Done false
13262 4339 Acked Read 2 Sent Done Done false
13263 4340 Acked Read 6 Sent Done Done false
13264 4341 Acked Write 1 Sent Done Done false
13265 4342 Acked Write 5 Sent Done Done false
13266 4343 Acked Read 10 Sent Done Done false
13267 4344 Acked Write 9 Sent Done Done false
13268 4345 Acked Read 1 Sent Done Done false
13269 4346 Acked Write 5 Sent Done Done false
13270 4347 Acked Flush 0 Sent Done Done false
13271 4348 Acked Read 7 Sent Done Done false
13272 4349 Acked Read 9 Sent Done Done false
13273 4350 Acked Write 1 Sent Done Done false
13274 4351 Acked Read 9 Sent Done Done false
13275 4352 Acked Read 4 Sent Done Done false
13276 4353 Acked Read 2 Sent Done Done false
13277 4354 Acked Flush 0 Sent Done Done false
13278 4355 Acked Write 8 Sent Done Done false
13279 4356 Acked Read 9 Sent Done Done false
13280 4357 Acked Read 2 Sent Done Done false
13281 4358 Acked Flush 0 Sent Done Done false
13282 4359 Acked Read 1 Sent Done Done false
13283 4360 Acked Write 9 Sent Done Done false
13284 4361 Acked Read 1 Sent Done Done false
13285 4362 Acked Read 1 Sent Done Done false
13286 4363 Acked Read 4 Sent Done Done false
13287 4364 Acked Read 9 Sent Done Done false
13288 4365 Acked Flush 0 Sent Done Done false
13289 4366 Acked Read 1 Sent Done Done false
13290 4367 Acked Flush 0 Sent Done Done false
13291 4368 Acked Write 2 Sent Done Done false
13292 4369 Acked Read 3 Sent Done Done false
13293 4370 Acked Write 3 Sent Done Done false
13294 4371 Acked Read 3 Sent Done Done false
13295 4372 Acked Read 5 Sent Done Done false
13296 4373 Acked Read 3 Sent Done Done false
13297 4374 Acked Read 8 Sent Done Done false
13298 4375 Acked Read 9 Sent Done Done false
13299 4376 Acked Flush 0 Sent Done Done false
13300 4377 Acked Write 2 Sent Done Done false
13301 4378 Acked Write 10 Sent Done Done false
13302 4379 Acked Read 10 Sent Done Done false
13303 4380 Acked Flush 0 Sent Done Done false
13304 4381 Acked Read 7 Sent Done Done false
13305 4382 Acked Read 8 Sent Done Done false
13306 4383 Acked Read 5 Sent Done Done false
13307 4384 Acked Write 6 Sent Done Done false
13308 4385 Acked Write 7 Sent Done Done false
13309 4386 Acked Read 7 Sent Done Done false
13310 4387 Acked Write 3 Sent Done Done false
13311 4388 Acked Read 3 Sent Done Done false
13312 4389 Acked Read 6 Sent Done Done false
13313 4390 Acked Read 6 Sent Done Done false
13314 4391 Acked Read 7 Sent Done Done false
13315 4392 Acked Read 8 Sent Done Done false
13316 4393 Acked Write 2 Sent Done Done false
13317 4394 Acked Read 9 Sent Done Done false
13318 4395 Acked Write 10 Sent Done Done false
13319 4396 Acked Write 3 Sent Done Done false
13320 4397 Acked Write 3 Sent Done Done false
13321 4398 Acked Write 2 Sent Done Done false
13322 4399 Acked Write 3 Sent Done Done false
13323 4400 Acked Write 8 Sent Done Done false
13324 4401 Acked Read 4 Sent Done Done false
13325 4402 Acked Read 10 Sent Done Done false
13326 4403 Acked Read 10 Sent Done Done false
13327 4404 Acked Read 7 Sent Done Done false
13328 4405 Acked Write 10 Sent Done Done false
13329 4406 Acked Read 2 Sent Done Done false
13330 4407 Acked Read 3 Sent Done Done false
13331 4408 Acked Read 9 Sent Done Done false
13332 4409 Acked Read 1 Sent Done Done false
13333 4410 Acked Flush 0 Sent Done Done false
13334 4411 Acked Read 1 Sent Done Done false
13335 4412 Acked Read 1 Sent Done Done false
13336 4413 Acked Read 6 Sent Done Done false
13337 4414 Acked Read 9 Sent Done Done false
13338 4415 Acked Read 1 Sent Done Done false
13339 4416 Acked Flush 0 Sent Done Done false
13340 4417 Acked Write 3 Sent Done Done false
13341 4418 Acked Write 2 Sent Done Done false
13342 4419 Acked Flush 0 Sent Done Done false
13343 4420 Acked Read 2 Sent Done Done false
13344 4421 Acked Read 5 Sent Done Done false
13345 4422 Acked Write 6 Sent Done Done false
13346 4423 Acked Flush 0 Sent Done Done false
13347 4424 Acked Write 7 Sent Done Done false
13348 4425 Acked Read 1 Sent Done Done false
13349 4426 Acked Read 5 Sent Done Done false
13350 4427 Acked Write 1 Sent Done Done false
13351 4428 Acked Read 3 Sent Done Done false
13352 4429 Acked Read 9 Sent Done Done false
13353 4430 Acked Write 9 Sent Done Done false
13354 4431 Acked Read 9 Sent Done Done false
13355 4432 Acked Read 4 Sent Done Done false
13356 4433 Acked Read 4 Sent Done Done false
13357 4434 Acked Write 4 Sent Done Done false
13358 4435 Acked Write 5 Sent Done Done false
13359 4436 Acked Read 6 Sent Done Done false
13360 4437 Acked Write 1 Sent Done Done false
13361 4438 Acked Read 8 Sent Done Done false
13362 4439 Acked Read 6 Sent Done Done false
13363 4440 Acked Read 7 Sent Done Done false
13364 4441 Acked Read 10 Sent Done Done false
13365 4442 Acked Read 3 Sent Done Done false
13366 4443 Acked Write 10 Sent Done Done false
13367 4444 Acked Read 1 Sent Done Done false
13368 4445 Acked Read 3 Sent Done Done false
13369 4446 Acked Read 1 Sent Done Done false
13370 4447 Acked Flush 0 Sent Done Done false
13371 4448 Acked Write 3 Sent Done Done false
13372 4449 Acked Write 7 Sent Done Done false
13373 4450 Acked Write 3 Sent Done Done false
13374 4451 Acked Read 7 Sent Done Done false
13375 4452 Acked Read 8 Sent Done Done false
13376 4453 Acked Write 8 Sent Done Done false
13377 4454 Acked Flush 0 Sent Done Done false
13378 4455 Acked Write 6 Sent Done Done false
13379 4456 Acked Read 5 Sent Done Done false
13380 4457 Acked Write 3 Sent Done Done false
13381 4458 Acked Read 5 Sent Done Done false
13382 4459 Acked Read 9 Sent Done Done false
13383 4460 Acked Read 5 Sent Done Done false
13384 4461 Acked Write 7 Sent Done Done false
13385 4462 Acked Read 3 Sent Done Done false
13386 4463 Acked Read 7 Sent Done Done false
13387 4464 Acked Read 8 Sent Done Done false
13388 4465 Acked Read 5 Sent Done Done false
13389 4466 Acked Write 6 Sent Done Done false
13390 4467 Acked Write 1 Sent Done Done false
13391 4468 Acked Read 7 Sent Done Done false
13392 4469 Acked Write 6 Sent Done Done false
13393 4470 Acked Read 7 Sent Done Done false
13394 4471 Acked Read 4 Sent Done Done false
13395 4472 Acked Flush 0 Sent Done Done false
13396 4473 Acked Write 2 Sent Done Done false
13397 4474 Acked Write 2 Sent Done Done false
13398 4475 Acked Read 3 Sent Done Done false
13399 4476 Acked Read 2 Sent Done Done false
13400 4477 Acked Write 6 Sent Done Done false
13401 4478 Acked Read 7 Sent Done Done false
13402 4479 Acked Flush 0 Sent Done Done false
13403 4480 Acked Read 1 Sent Done Done false
13404 4481 Acked Read 7 Sent Done Done false
13405 4482 Acked Read 10 Sent Done Done false
13406 4483 Acked Read 6 Sent Done Done false
13407 4484 Acked Read 3 Sent Done Done false
13408 4485 Acked Write 2 Sent Done Done false
13409 4486 Acked Write 4 Sent Done Done false
13410 4487 Acked Read 6 Sent Done Done false
13411 4488 Acked Write 10 Sent Done Done false
13412 4489 Acked Write 1 Sent Done Done false
13413 4490 Acked Write 5 Sent Done Done false
13414 4491 Acked Write 3 Sent Done Done false
13415 4492 Acked Write 4 Sent Done Done false
13416 4493 Acked Read 5 Sent Done Done false
13417 4494 Acked Read 8 Sent Done Done false
13418 4495 Acked Read 3 Sent Done Done false
13419 4496 Acked Write 5 Sent Done Done false
13420 4497 Acked Read 5 Sent Done Done false
13421 4498 Acked Read 1 Sent Done Done false
13422 4499 Acked Read 3 Sent Done Done false
13423 4500 Acked Write 7 Sent Done Done false
13424 4501 Acked Write 10 Sent Done Done false
13425 4502 Acked Read 5 Sent Done Done false
13426 4503 Acked Read 2 Sent Done Done false
13427 4504 Acked Flush 0 Sent Done Done false
13428 4505 Acked Read 9 Sent Done Done false
13429 4506 Acked Write 6 Sent Done Done false
13430 4507 Acked Write 8 Sent Done Done false
13431 4508 Acked Write 2 Sent Done Done false
13432 4509 Acked Flush 0 Sent Done Done false
13433 4510 Acked Write 1 Sent Done Done false
13434 4511 Acked Read 10 Sent Done Done false
13435 4512 Acked Write 9 Sent Done Done false
13436 4513 Acked Read 1 Sent Done Done false
13437 4514 Acked Write 8 Sent Done Done false
13438 4515 Acked Read 8 Sent Done Done false
13439 4516 Acked Write 1 Sent Done Done false
13440 4517 Acked Read 4 Sent Done Done false
13441 4518 Acked Flush 0 Sent Done Done false
13442 4519 Acked Read 10 Sent Done Done false
13443 4520 Acked Read 7 Sent Done Done false
13444 4521 Acked Read 8 Sent Done Done false
13445 4522 Acked Write 2 Sent Done Done false
13446 4523 Acked Read 10 Sent Done Done false
13447 4524 Acked Write 10 Sent Done Done false
13448 4525 Acked Write 9 Sent Done Done false
13449 4526 Acked Write 3 Sent Done Done false
13450 4527 Acked Write 1 Sent Done Done false
13451 4528 Acked Write 9 Sent Done Done false
13452 4529 Acked Write 10 Sent Done Done false
13453 4530 Acked Write 9 Sent Done Done false
13454 4531 Acked Flush 0 Sent Done Done false
13455 4532 Acked Read 10 Sent Done Done false
13456 4533 Acked Write 8 Sent Done Done false
13457 4534 Acked Flush 0 Sent Done Done false
13458 4535 Acked Write 4 Sent Done Done false
13459 4536 Acked Write 9 Sent Done Done false
13460 4537 Acked Write 6 Sent Done Done false
13461 4538 Acked Write 10 Sent Done Done false
13462 4539 Acked Write 5 Sent Done Done false
13463 4540 Acked Write 7 Sent Done Done false
13464 4541 Acked Write 10 Sent Done Done false
13465 4542 Acked Read 9 Sent Done Done false
13466 4543 Acked Flush 0 Sent Done Done false
13467 4544 Acked Flush 0 Sent Done Done false
13468 4545 Acked Read 9 Sent Done Done false
13469 4546 Acked Read 9 Sent Done Done false
13470 4547 Acked Write 9 Sent Done Done false
13471 4548 Acked Read 2 Sent Done Done false
13472 4549 Acked Read 6 Sent Done Done false
13473 4550 Acked Write 3 Sent Done Done false
13474 4551 Acked Write 8 Sent Done Done false
13475 4552 Acked Read 4 Sent Done Done false
13476 4553 Acked Read 5 Sent Done Done false
13477 4554 Acked Read 7 Sent Done Done false
13478 4555 Acked Read 3 Sent Done Done false
13479 4556 Acked Read 6 Sent Done Done false
13480 4557 Acked Read 4 Sent Done Done false
13481 4558 Acked Write 8 Sent Done Done false
13482 4559 Acked Read 9 Sent Done Done false
13483 4560 Acked Write 3 Sent Done Done false
13484 4561 Acked Write 8 Sent Done Done false
13485 4562 Acked Write 5 Sent Done Done false
13486 4563 Acked Read 7 Sent Done Done false
13487 4564 Acked Read 9 Sent Done Done false
13488 4565 Acked Read 5 Sent Done Done false
13489 4566 Acked Write 9 Sent Done Done false
13490 4567 Acked Read 3 Sent Done Done false
13491 4568 Acked Write 8 Sent Done Done false
13492 4569 Acked Read 4 Sent Done Done false
13493 4570 Acked Write 6 Sent Done Done false
13494 4571 Acked Write 4 Sent Done Done false
13495 4572 Acked Write 5 Sent Done Done false
13496 4573 Acked Read 8 Sent Done Done false
13497 4574 Acked Flush 0 Sent Done Done false
13498 4575 Acked Read 2 Sent Done Done false
13499 4576 Acked Read 9 Sent Done Done false
13500 4577 Acked Write 2 Sent Done Done false
13501 4578 Acked Write 8 Sent Done Done false
13502 4579 Acked Read 4 Sent Done Done false
13503 4580 Acked Read 4 Sent Done Done false
13504 4581 Acked Read 3 Sent Done Done false
13505 4582 Acked Read 3 Sent Done Done false
13506 4583 Acked Read 6 Sent Done Done false
13507 4584 Acked Read 6 Sent Done Done false
13508 4585 Acked Read 6 Sent Done Done false
13509 4586 Acked Read 4 Sent Done Done false
13510 4587 Acked Read 10 Sent Done Done false
13511 4588 Acked Write 3 Sent Done Done false
13512 4589 Acked Read 6 Sent Done Done false
13513 4590 Acked Write 7 Sent Done Done false
13514 4591 Acked Write 9 Sent Done Done false
13515 4592 Acked Write 5 Sent Done Done false
13516 4593 Acked Read 3 Sent Done Done false
13517 4594 Acked Write 4 Sent Done Done false
13518 4595 Acked Read 6 Sent Done Done false
13519 4596 Acked Read 9 Sent Done Done false
13520 4597 Acked Flush 0 Sent Done Done false
13521 4598 Acked Flush 0 Sent Done Done false
13522 4599 Acked Read 10 Sent Done Done false
13523 4600 Acked Write 1 Sent Done Done false
13524 4601 Acked Write 6 Sent Done Done false
13525 4602 Acked Read 1 Sent Done Done false
13526 4603 Acked Read 10 Sent Done Done false
13527 4604 Acked Flush 0 Sent Done Done false
13528 4605 Acked Write 3 Sent Done Done false
13529 4606 Acked Write 10 Sent Done Done false
13530 4607 Acked Write 10 Sent Done Done false
13531 4608 Acked Write 6 Sent Done Done false
13532 4609 Acked Write 4 Sent Done Done false
13533 4610 Acked Write 6 Sent Done Done false
13534 4611 Acked Read 8 Sent Done Done false
13535 4612 Acked Read 6 Sent Done Done false
13536 4613 Acked Write 10 Sent Done Done false
13537 4614 Acked Read 8 Sent Done Done false
13538 4615 Acked Read 2 Sent Done Done false
13539 4616 Acked Read 8 Sent Done Done false
13540 4617 Acked Flush 0 Sent Done Done false
13541 4618 Acked Write 3 Sent Done Done false
13542 4619 Acked Flush 0 Sent Done Done false
13543 4620 Acked Read 5 Sent Done Done false
13544 4621 Acked Write 9 Sent Done Done false
13545 4622 Acked Write 1 Sent Done Done false
13546 STATES DS:0 DS:1 DS:2 TOTAL
13547 Sent 301 0 0 301
13548 Done 0 301 301 602
13549 Skipped 0 0 0 0
13550 Error 0 0 0 0
13551 Last Flush: Some(JobId(4321)) Some(JobId(4619)) Some(JobId(4619))
13552 Downstairs last five completed: 4321 4320 4319 4318 4317
13553 Upstairs last five completed: 4622 4621 4620 4619 4618
13554 CLIENT: Up:0 ds:301 act:2
135552025-01-29T05:13:39.204ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
135562025-01-29T05:13:40.206ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
135572025-01-29T05:13:41.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
135582025-01-29T05:13:41.707ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
13559 ----------------------------------------------------------------
13560 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13561 States: Offline Active Active
13562 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13563 4322 Acked Flush 0 Sent Done Done false
13564 4323 Acked Write 1 Sent Done Done false
13565 4324 Acked Read 5 Sent Done Done false
13566 4325 Acked Write 8 Sent Done Done false
13567 4326 Acked Write 3 Sent Done Done false
13568 4327 Acked Read 7 Sent Done Done false
13569 4328 Acked Write 5 Sent Done Done false
13570 4329 Acked Read 4 Sent Done Done false
13571 4330 Acked Write 8 Sent Done Done false
13572 4331 Acked Write 3 Sent Done Done false
13573 4332 Acked Write 3 Sent Done Done false
13574 4333 Acked Read 10 Sent Done Done false
13575 4334 Acked Flush 0 Sent Done Done false
13576 4335 Acked Read 4 Sent Done Done false
13577 4336 Acked Read 7 Sent Done Done false
13578 4337 Acked Read 9 Sent Done Done false
13579 4338 Acked Read 4 Sent Done Done false
13580 4339 Acked Read 2 Sent Done Done false
13581 4340 Acked Read 6 Sent Done Done false
13582 4341 Acked Write 1 Sent Done Done false
13583 4342 Acked Write 5 Sent Done Done false
13584 4343 Acked Read 10 Sent Done Done false
13585 4344 Acked Write 9 Sent Done Done false
13586 4345 Acked Read 1 Sent Done Done false
13587 4346 Acked Write 5 Sent Done Done false
13588 4347 Acked Flush 0 Sent Done Done false
13589 4348 Acked Read 7 Sent Done Done false
13590 4349 Acked Read 9 Sent Done Done false
13591 4350 Acked Write 1 Sent Done Done false
13592 4351 Acked Read 9 Sent Done Done false
13593 4352 Acked Read 4 Sent Done Done false
13594 4353 Acked Read 2 Sent Done Done false
13595 4354 Acked Flush 0 Sent Done Done false
13596 4355 Acked Write 8 Sent Done Done false
13597 4356 Acked Read 9 Sent Done Done false
13598 4357 Acked Read 2 Sent Done Done false
13599 4358 Acked Flush 0 Sent Done Done false
13600 4359 Acked Read 1 Sent Done Done false
13601 4360 Acked Write 9 Sent Done Done false
13602 4361 Acked Read 1 Sent Done Done false
13603 4362 Acked Read 1 Sent Done Done false
13604 4363 Acked Read 4 Sent Done Done false
13605 4364 Acked Read 9 Sent Done Done false
13606 4365 Acked Flush 0 Sent Done Done false
13607 4366 Acked Read 1 Sent Done Done false
13608 4367 Acked Flush 0 Sent Done Done false
13609 4368 Acked Write 2 Sent Done Done false
13610 4369 Acked Read 3 Sent Done Done false
13611 4370 Acked Write 3 Sent Done Done false
13612 4371 Acked Read 3 Sent Done Done false
13613 4372 Acked Read 5 Sent Done Done false
13614 4373 Acked Read 3 Sent Done Done false
13615 4374 Acked Read 8 Sent Done Done false
13616 4375 Acked Read 9 Sent Done Done false
13617 4376 Acked Flush 0 Sent Done Done false
13618 4377 Acked Write 2 Sent Done Done false
13619 4378 Acked Write 10 Sent Done Done false
13620 4379 Acked Read 10 Sent Done Done false
13621 4380 Acked Flush 0 Sent Done Done false
13622 4381 Acked Read 7 Sent Done Done false
13623 4382 Acked Read 8 Sent Done Done false
13624 4383 Acked Read 5 Sent Done Done false
13625 4384 Acked Write 6 Sent Done Done false
13626 4385 Acked Write 7 Sent Done Done false
13627 4386 Acked Read 7 Sent Done Done false
13628 4387 Acked Write 3 Sent Done Done false
13629 4388 Acked Read 3 Sent Done Done false
13630 4389 Acked Read 6 Sent Done Done false
13631 4390 Acked Read 6 Sent Done Done false
13632 4391 Acked Read 7 Sent Done Done false
13633 4392 Acked Read 8 Sent Done Done false
13634 4393 Acked Write 2 Sent Done Done false
13635 4394 Acked Read 9 Sent Done Done false
13636 4395 Acked Write 10 Sent Done Done false
13637 4396 Acked Write 3 Sent Done Done false
13638 4397 Acked Write 3 Sent Done Done false
13639 4398 Acked Write 2 Sent Done Done false
13640 4399 Acked Write 3 Sent Done Done false
13641 4400 Acked Write 8 Sent Done Done false
13642 4401 Acked Read 4 Sent Done Done false
13643 4402 Acked Read 10 Sent Done Done false
13644 4403 Acked Read 10 Sent Done Done false
13645 4404 Acked Read 7 Sent Done Done false
13646 4405 Acked Write 10 Sent Done Done false
13647 4406 Acked Read 2 Sent Done Done false
13648 4407 Acked Read 3 Sent Done Done false
13649 4408 Acked Read 9 Sent Done Done false
13650 4409 Acked Read 1 Sent Done Done false
13651 4410 Acked Flush 0 Sent Done Done false
13652 4411 Acked Read 1 Sent Done Done false
13653 4412 Acked Read 1 Sent Done Done false
13654 4413 Acked Read 6 Sent Done Done false
13655 4414 Acked Read 9 Sent Done Done false
13656 4415 Acked Read 1 Sent Done Done false
13657 4416 Acked Flush 0 Sent Done Done false
13658 4417 Acked Write 3 Sent Done Done false
13659 4418 Acked Write 2 Sent Done Done false
13660 4419 Acked Flush 0 Sent Done Done false
13661 4420 Acked Read 2 Sent Done Done false
13662 4421 Acked Read 5 Sent Done Done false
13663 4422 Acked Write 6 Sent Done Done false
13664 4423 Acked Flush 0 Sent Done Done false
13665 4424 Acked Write 7 Sent Done Done false
13666 4425 Acked Read 1 Sent Done Done false
13667 4426 Acked Read 5 Sent Done Done false
13668 4427 Acked Write 1 Sent Done Done false
13669 4428 Acked Read 3 Sent Done Done false
13670 4429 Acked Read 9 Sent Done Done false
13671 4430 Acked Write 9 Sent Done Done false
13672 4431 Acked Read 9 Sent Done Done false
13673 4432 Acked Read 4 Sent Done Done false
13674 4433 Acked Read 4 Sent Done Done false
13675 4434 Acked Write 4 Sent Done Done false
13676 4435 Acked Write 5 Sent Done Done false
13677 4436 Acked Read 6 Sent Done Done false
13678 4437 Acked Write 1 Sent Done Done false
13679 4438 Acked Read 8 Sent Done Done false
13680 4439 Acked Read 6 Sent Done Done false
13681 4440 Acked Read 7 Sent Done Done false
13682 4441 Acked Read 10 Sent Done Done false
13683 4442 Acked Read 3 Sent Done Done false
13684 4443 Acked Write 10 Sent Done Done false
13685 4444 Acked Read 1 Sent Done Done false
13686 4445 Acked Read 3 Sent Done Done false
13687 4446 Acked Read 1 Sent Done Done false
13688 4447 Acked Flush 0 Sent Done Done false
13689 4448 Acked Write 3 Sent Done Done false
13690 4449 Acked Write 7 Sent Done Done false
13691 4450 Acked Write 3 Sent Done Done false
13692 4451 Acked Read 7 Sent Done Done false
13693 4452 Acked Read 8 Sent Done Done false
13694 4453 Acked Write 8 Sent Done Done false
13695 4454 Acked Flush 0 Sent Done Done false
13696 4455 Acked Write 6 Sent Done Done false
13697 4456 Acked Read 5 Sent Done Done false
13698 4457 Acked Write 3 Sent Done Done false
13699 4458 Acked Read 5 Sent Done Done false
13700 4459 Acked Read 9 Sent Done Done false
13701 4460 Acked Read 5 Sent Done Done false
13702 4461 Acked Write 7 Sent Done Done false
13703 4462 Acked Read 3 Sent Done Done false
13704 4463 Acked Read 7 Sent Done Done false
13705 4464 Acked Read 8 Sent Done Done false
13706 4465 Acked Read 5 Sent Done Done false
13707 4466 Acked Write 6 Sent Done Done false
13708 4467 Acked Write 1 Sent Done Done false
13709 4468 Acked Read 7 Sent Done Done false
13710 4469 Acked Write 6 Sent Done Done false
13711 4470 Acked Read 7 Sent Done Done false
13712 4471 Acked Read 4 Sent Done Done false
13713 4472 Acked Flush 0 Sent Done Done false
13714 4473 Acked Write 2 Sent Done Done false
13715 4474 Acked Write 2 Sent Done Done false
13716 4475 Acked Read 3 Sent Done Done false
13717 4476 Acked Read 2 Sent Done Done false
13718 4477 Acked Write 6 Sent Done Done false
13719 4478 Acked Read 7 Sent Done Done false
13720 4479 Acked Flush 0 Sent Done Done false
13721 4480 Acked Read 1 Sent Done Done false
13722 4481 Acked Read 7 Sent Done Done false
13723 4482 Acked Read 10 Sent Done Done false
13724 4483 Acked Read 6 Sent Done Done false
13725 4484 Acked Read 3 Sent Done Done false
13726 4485 Acked Write 2 Sent Done Done false
13727 4486 Acked Write 4 Sent Done Done false
13728 4487 Acked Read 6 Sent Done Done false
13729 4488 Acked Write 10 Sent Done Done false
13730 4489 Acked Write 1 Sent Done Done false
13731 4490 Acked Write 5 Sent Done Done false
13732 4491 Acked Write 3 Sent Done Done false
13733 4492 Acked Write 4 Sent Done Done false
13734 4493 Acked Read 5 Sent Done Done false
13735 4494 Acked Read 8 Sent Done Done false
13736 4495 Acked Read 3 Sent Done Done false
13737 4496 Acked Write 5 Sent Done Done false
13738 4497 Acked Read 5 Sent Done Done false
13739 4498 Acked Read 1 Sent Done Done false
13740 4499 Acked Read 3 Sent Done Done false
13741 4500 Acked Write 7 Sent Done Done false
13742 4501 Acked Write 10 Sent Done Done false
13743 4502 Acked Read 5 Sent Done Done false
13744 4503 Acked Read 2 Sent Done Done false
13745 4504 Acked Flush 0 Sent Done Done false
13746 4505 Acked Read 9 Sent Done Done false
13747 4506 Acked Write 6 Sent Done Done false
13748 4507 Acked Write 8 Sent Done Done false
13749 4508 Acked Write 2 Sent Done Done false
13750 4509 Acked Flush 0 Sent Done Done false
13751 4510 Acked Write 1 Sent Done Done false
13752 4511 Acked Read 10 Sent Done Done false
13753 4512 Acked Write 9 Sent Done Done false
13754 4513 Acked Read 1 Sent Done Done false
13755 4514 Acked Write 8 Sent Done Done false
13756 4515 Acked Read 8 Sent Done Done false
13757 4516 Acked Write 1 Sent Done Done false
13758 4517 Acked Read 4 Sent Done Done false
13759 4518 Acked Flush 0 Sent Done Done false
13760 4519 Acked Read 10 Sent Done Done false
13761 4520 Acked Read 7 Sent Done Done false
13762 4521 Acked Read 8 Sent Done Done false
13763 4522 Acked Write 2 Sent Done Done false
13764 4523 Acked Read 10 Sent Done Done false
13765 4524 Acked Write 10 Sent Done Done false
13766 4525 Acked Write 9 Sent Done Done false
13767 4526 Acked Write 3 Sent Done Done false
13768 4527 Acked Write 1 Sent Done Done false
13769 4528 Acked Write 9 Sent Done Done false
13770 4529 Acked Write 10 Sent Done Done false
13771 4530 Acked Write 9 Sent Done Done false
13772 4531 Acked Flush 0 Sent Done Done false
13773 4532 Acked Read 10 Sent Done Done false
13774 4533 Acked Write 8 Sent Done Done false
13775 4534 Acked Flush 0 Sent Done Done false
13776 4535 Acked Write 4 Sent Done Done false
13777 4536 Acked Write 9 Sent Done Done false
13778 4537 Acked Write 6 Sent Done Done false
13779 4538 Acked Write 10 Sent Done Done false
13780 4539 Acked Write 5 Sent Done Done false
13781 4540 Acked Write 7 Sent Done Done false
13782 4541 Acked Write 10 Sent Done Done false
13783 4542 Acked Read 9 Sent Done Done false
13784 4543 Acked Flush 0 Sent Done Done false
13785 4544 Acked Flush 0 Sent Done Done false
13786 4545 Acked Read 9 Sent Done Done false
13787 4546 Acked Read 9 Sent Done Done false
13788 4547 Acked Write 9 Sent Done Done false
13789 4548 Acked Read 2 Sent Done Done false
13790 4549 Acked Read 6 Sent Done Done false
13791 4550 Acked Write 3 Sent Done Done false
13792 4551 Acked Write 8 Sent Done Done false
13793 4552 Acked Read 4 Sent Done Done false
13794 4553 Acked Read 5 Sent Done Done false
13795 4554 Acked Read 7 Sent Done Done false
13796 4555 Acked Read 3 Sent Done Done false
13797 4556 Acked Read 6 Sent Done Done false
13798 4557 Acked Read 4 Sent Done Done false
13799 4558 Acked Write 8 Sent Done Done false
13800 4559 Acked Read 9 Sent Done Done false
13801 4560 Acked Write 3 Sent Done Done false
13802 4561 Acked Write 8 Sent Done Done false
13803 4562 Acked Write 5 Sent Done Done false
13804 4563 Acked Read 7 Sent Done Done false
13805 4564 Acked Read 9 Sent Done Done false
13806 4565 Acked Read 5 Sent Done Done false
13807 4566 Acked Write 9 Sent Done Done false
13808 4567 Acked Read 3 Sent Done Done false
13809 4568 Acked Write 8 Sent Done Done false
13810 4569 Acked Read 4 Sent Done Done false
13811 4570 Acked Write 6 Sent Done Done false
13812 4571 Acked Write 4 Sent Done Done false
13813 4572 Acked Write 5 Sent Done Done false
13814 4573 Acked Read 8 Sent Done Done false
13815 4574 Acked Flush 0 Sent Done Done false
13816 4575 Acked Read 2 Sent Done Done false
13817 4576 Acked Read 9 Sent Done Done false
13818 4577 Acked Write 2 Sent Done Done false
13819 4578 Acked Write 8 Sent Done Done false
13820 4579 Acked Read 4 Sent Done Done false
13821 4580 Acked Read 4 Sent Done Done false
13822 4581 Acked Read 3 Sent Done Done false
13823 4582 Acked Read 3 Sent Done Done false
13824 4583 Acked Read 6 Sent Done Done false
13825 4584 Acked Read 6 Sent Done Done false
13826 4585 Acked Read 6 Sent Done Done false
13827 4586 Acked Read 4 Sent Done Done false
13828 4587 Acked Read 10 Sent Done Done false
13829 4588 Acked Write 3 Sent Done Done false
13830 4589 Acked Read 6 Sent Done Done false
13831 4590 Acked Write 7 Sent Done Done false
13832 4591 Acked Write 9 Sent Done Done false
13833 4592 Acked Write 5 Sent Done Done false
13834 4593 Acked Read 3 Sent Done Done false
13835 4594 Acked Write 4 Sent Done Done false
13836 4595 Acked Read 6 Sent Done Done false
13837 4596 Acked Read 9 Sent Done Done false
13838 4597 Acked Flush 0 Sent Done Done false
13839 4598 Acked Flush 0 Sent Done Done false
13840 4599 Acked Read 10 Sent Done Done false
13841 4600 Acked Write 1 Sent Done Done false
13842 4601 Acked Write 6 Sent Done Done false
13843 4602 Acked Read 1 Sent Done Done false
13844 4603 Acked Read 10 Sent Done Done false
13845 4604 Acked Flush 0 Sent Done Done false
13846 4605 Acked Write 3 Sent Done Done false
13847 4606 Acked Write 10 Sent Done Done false
13848 4607 Acked Write 10 Sent Done Done false
13849 4608 Acked Write 6 Sent Done Done false
13850 4609 Acked Write 4 Sent Done Done false
13851 4610 Acked Write 6 Sent Done Done false
13852 4611 Acked Read 8 Sent Done Done false
13853 4612 Acked Read 6 Sent Done Done false
13854 4613 Acked Write 10 Sent Done Done false
13855 4614 Acked Read 8 Sent Done Done false
13856 4615 Acked Read 2 Sent Done Done false
13857 4616 Acked Read 8 Sent Done Done false
13858 4617 Acked Flush 0 Sent Done Done false
13859 4618 Acked Write 3 Sent Done Done false
13860 4619 Acked Flush 0 Sent Done Done false
13861 4620 Acked Read 5 Sent Done Done false
13862 4621 Acked Write 9 Sent Done Done false
13863 4622 Acked Write 1 Sent Done Done false
13864 STATES DS:0 DS:1 DS:2 TOTAL
13865 Sent 301 0 0 301
13866 Done 0 301 301 602
13867 Skipped 0 0 0 0
13868 Error 0 0 0 0
13869 Last Flush: Some(JobId(4321)) Some(JobId(4619)) Some(JobId(4619))
13870 Downstairs last five completed: 4321 4320 4319 4318 4317
13871 Upstairs last five completed: 4622 4621 4620 4619 4618
13872 CLIENT: Up:0 ds:301 act:2
138732025-01-29T05:13:43.211ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138742025-01-29T05:13:44.213ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138752025-01-29T05:13:44.214ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138762025-01-29T05:13:44.214ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138772025-01-29T05:13:44.214ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138782025-01-29T05:13:44.214ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138792025-01-29T05:13:44.214ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
13880 [8810] got start action
13881 [8810] I am going to restart
13882 Starting downstairs at port 8810
13883 Make output file at "/tmp/dsc/downstairs-8810.txt"
13884 [8810][0] reports Starting
13885 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1214)
13886 [8810][0] reports Running
138872025-01-29T05:13:45.213ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
138882025-01-29T05:13:46.216ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
13889 ----------------------------------------------------------------
13890 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13891 States: Offline Active Active
13892 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13893 4322 Acked Flush 0 Sent Done Done false
13894 4323 Acked Write 1 Sent Done Done false
13895 4324 Acked Read 5 Sent Done Done false
13896 4325 Acked Write 8 Sent Done Done false
13897 4326 Acked Write 3 Sent Done Done false
13898 4327 Acked Read 7 Sent Done Done false
13899 4328 Acked Write 5 Sent Done Done false
13900 4329 Acked Read 4 Sent Done Done false
13901 4330 Acked Write 8 Sent Done Done false
13902 4331 Acked Write 3 Sent Done Done false
13903 4332 Acked Write 3 Sent Done Done false
13904 4333 Acked Read 10 Sent Done Done false
13905 4334 Acked Flush 0 Sent Done Done false
13906 4335 Acked Read 4 Sent Done Done false
13907 4336 Acked Read 7 Sent Done Done false
13908 4337 Acked Read 9 Sent Done Done false
13909 4338 Acked Read 4 Sent Done Done false
13910 4339 Acked Read 2 Sent Done Done false
13911 4340 Acked Read 6 Sent Done Done false
13912 4341 Acked Write 1 Sent Done Done false
13913 4342 Acked Write 5 Sent Done Done false
13914 4343 Acked Read 10 Sent Done Done false
13915 4344 Acked Write 9 Sent Done Done false
13916 4345 Acked Read 1 Sent Done Done false
13917 4346 Acked Write 5 Sent Done Done false
13918 4347 Acked Flush 0 Sent Done Done false
13919 4348 Acked Read 7 Sent Done Done false
13920 4349 Acked Read 9 Sent Done Done false
13921 4350 Acked Write 1 Sent Done Done false
13922 4351 Acked Read 9 Sent Done Done false
13923 4352 Acked Read 4 Sent Done Done false
13924 4353 Acked Read 2 Sent Done Done false
13925 4354 Acked Flush 0 Sent Done Done false
13926 4355 Acked Write 8 Sent Done Done false
13927 4356 Acked Read 9 Sent Done Done false
13928 4357 Acked Read 2 Sent Done Done false
13929 4358 Acked Flush 0 Sent Done Done false
13930 4359 Acked Read 1 Sent Done Done false
13931 4360 Acked Write 9 Sent Done Done false
13932 4361 Acked Read 1 Sent Done Done false
13933 4362 Acked Read 1 Sent Done Done false
13934 4363 Acked Read 4 Sent Done Done false
13935 4364 Acked Read 9 Sent Done Done false
13936 4365 Acked Flush 0 Sent Done Done false
13937 4366 Acked Read 1 Sent Done Done false
13938 4367 Acked Flush 0 Sent Done Done false
13939 4368 Acked Write 2 Sent Done Done false
13940 4369 Acked Read 3 Sent Done Done false
13941 4370 Acked Write 3 Sent Done Done false
13942 4371 Acked Read 3 Sent Done Done false
13943 4372 Acked Read 5 Sent Done Done false
13944 4373 Acked Read 3 Sent Done Done false
13945 4374 Acked Read 8 Sent Done Done false
13946 4375 Acked Read 9 Sent Done Done false
13947 4376 Acked Flush 0 Sent Done Done false
13948 4377 Acked Write 2 Sent Done Done false
13949 4378 Acked Write 10 Sent Done Done false
13950 4379 Acked Read 10 Sent Done Done false
13951 4380 Acked Flush 0 Sent Done Done false
13952 4381 Acked Read 7 Sent Done Done false
13953 4382 Acked Read 8 Sent Done Done false
13954 4383 Acked Read 5 Sent Done Done false
13955 4384 Acked Write 6 Sent Done Done false
13956 4385 Acked Write 7 Sent Done Done false
13957 4386 Acked Read 7 Sent Done Done false
13958 4387 Acked Write 3 Sent Done Done false
13959 4388 Acked Read 3 Sent Done Done false
13960 4389 Acked Read 6 Sent Done Done false
13961 4390 Acked Read 6 Sent Done Done false
13962 4391 Acked Read 7 Sent Done Done false
13963 4392 Acked Read 8 Sent Done Done false
13964 4393 Acked Write 2 Sent Done Done false
13965 4394 Acked Read 9 Sent Done Done false
13966 4395 Acked Write 10 Sent Done Done false
13967 4396 Acked Write 3 Sent Done Done false
13968 4397 Acked Write 3 Sent Done Done false
13969 4398 Acked Write 2 Sent Done Done false
13970 4399 Acked Write 3 Sent Done Done false
13971 4400 Acked Write 8 Sent Done Done false
13972 4401 Acked Read 4 Sent Done Done false
13973 4402 Acked Read 10 Sent Done Done false
13974 4403 Acked Read 10 Sent Done Done false
13975 4404 Acked Read 7 Sent Done Done false
13976 4405 Acked Write 10 Sent Done Done false
13977 4406 Acked Read 2 Sent Done Done false
13978 4407 Acked Read 3 Sent Done Done false
13979 4408 Acked Read 9 Sent Done Done false
13980 4409 Acked Read 1 Sent Done Done false
13981 4410 Acked Flush 0 Sent Done Done false
13982 4411 Acked Read 1 Sent Done Done false
13983 4412 Acked Read 1 Sent Done Done false
13984 4413 Acked Read 6 Sent Done Done false
13985 4414 Acked Read 9 Sent Done Done false
13986 4415 Acked Read 1 Sent Done Done false
13987 4416 Acked Flush 0 Sent Done Done false
13988 4417 Acked Write 3 Sent Done Done false
13989 4418 Acked Write 2 Sent Done Done false
13990 4419 Acked Flush 0 Sent Done Done false
13991 4420 Acked Read 2 Sent Done Done false
13992 4421 Acked Read 5 Sent Done Done false
13993 4422 Acked Write 6 Sent Done Done false
13994 4423 Acked Flush 0 Sent Done Done false
13995 4424 Acked Write 7 Sent Done Done false
13996 4425 Acked Read 1 Sent Done Done false
13997 4426 Acked Read 5 Sent Done Done false
13998 4427 Acked Write 1 Sent Done Done false
13999 4428 Acked Read 3 Sent Done Done false
14000 4429 Acked Read 9 Sent Done Done false
14001 4430 Acked Write 9 Sent Done Done false
14002 4431 Acked Read 9 Sent Done Done false
14003 4432 Acked Read 4 Sent Done Done false
14004 4433 Acked Read 4 Sent Done Done false
14005 4434 Acked Write 4 Sent Done Done false
14006 4435 Acked Write 5 Sent Done Done false
14007 4436 Acked Read 6 Sent Done Done false
14008 4437 Acked Write 1 Sent Done Done false
14009 4438 Acked Read 8 Sent Done Done false
14010 4439 Acked Read 6 Sent Done Done false
14011 4440 Acked Read 7 Sent Done Done false
14012 4441 Acked Read 10 Sent Done Done false
14013 4442 Acked Read 3 Sent Done Done false
14014 4443 Acked Write 10 Sent Done Done false
14015 4444 Acked Read 1 Sent Done Done false
14016 4445 Acked Read 3 Sent Done Done false
14017 4446 Acked Read 1 Sent Done Done false
14018 4447 Acked Flush 0 Sent Done Done false
14019 4448 Acked Write 3 Sent Done Done false
14020 4449 Acked Write 7 Sent Done Done false
14021 4450 Acked Write 3 Sent Done Done false
14022 4451 Acked Read 7 Sent Done Done false
14023 4452 Acked Read 8 Sent Done Done false
14024 4453 Acked Write 8 Sent Done Done false
14025 4454 Acked Flush 0 Sent Done Done false
14026 4455 Acked Write 6 Sent Done Done false
14027 4456 Acked Read 5 Sent Done Done false
14028 4457 Acked Write 3 Sent Done Done false
14029 4458 Acked Read 5 Sent Done Done false
14030 4459 Acked Read 9 Sent Done Done false
14031 4460 Acked Read 5 Sent Done Done false
14032 4461 Acked Write 7 Sent Done Done false
14033 4462 Acked Read 3 Sent Done Done false
14034 4463 Acked Read 7 Sent Done Done false
14035 4464 Acked Read 8 Sent Done Done false
14036 4465 Acked Read 5 Sent Done Done false
14037 4466 Acked Write 6 Sent Done Done false
14038 4467 Acked Write 1 Sent Done Done false
14039 4468 Acked Read 7 Sent Done Done false
14040 4469 Acked Write 6 Sent Done Done false
14041 4470 Acked Read 7 Sent Done Done false
14042 4471 Acked Read 4 Sent Done Done false
14043 4472 Acked Flush 0 Sent Done Done false
14044 4473 Acked Write 2 Sent Done Done false
14045 4474 Acked Write 2 Sent Done Done false
14046 4475 Acked Read 3 Sent Done Done false
14047 4476 Acked Read 2 Sent Done Done false
14048 4477 Acked Write 6 Sent Done Done false
14049 4478 Acked Read 7 Sent Done Done false
14050 4479 Acked Flush 0 Sent Done Done false
14051 4480 Acked Read 1 Sent Done Done false
14052 4481 Acked Read 7 Sent Done Done false
14053 4482 Acked Read 10 Sent Done Done false
14054 4483 Acked Read 6 Sent Done Done false
14055 4484 Acked Read 3 Sent Done Done false
14056 4485 Acked Write 2 Sent Done Done false
14057 4486 Acked Write 4 Sent Done Done false
14058 4487 Acked Read 6 Sent Done Done false
14059 4488 Acked Write 10 Sent Done Done false
14060 4489 Acked Write 1 Sent Done Done false
14061 4490 Acked Write 5 Sent Done Done false
14062 4491 Acked Write 3 Sent Done Done false
14063 4492 Acked Write 4 Sent Done Done false
14064 4493 Acked Read 5 Sent Done Done false
14065 4494 Acked Read 8 Sent Done Done false
14066 4495 Acked Read 3 Sent Done Done false
14067 4496 Acked Write 5 Sent Done Done false
14068 4497 Acked Read 5 Sent Done Done false
14069 4498 Acked Read 1 Sent Done Done false
14070 4499 Acked Read 3 Sent Done Done false
14071 4500 Acked Write 7 Sent Done Done false
14072 4501 Acked Write 10 Sent Done Done false
14073 4502 Acked Read 5 Sent Done Done false
14074 4503 Acked Read 2 Sent Done Done false
14075 4504 Acked Flush 0 Sent Done Done false
14076 4505 Acked Read 9 Sent Done Done false
14077 4506 Acked Write 6 Sent Done Done false
14078 4507 Acked Write 8 Sent Done Done false
14079 4508 Acked Write 2 Sent Done Done false
14080 4509 Acked Flush 0 Sent Done Done false
14081 4510 Acked Write 1 Sent Done Done false
14082 4511 Acked Read 10 Sent Done Done false
14083 4512 Acked Write 9 Sent Done Done false
14084 4513 Acked Read 1 Sent Done Done false
14085 4514 Acked Write 8 Sent Done Done false
14086 4515 Acked Read 8 Sent Done Done false
14087 4516 Acked Write 1 Sent Done Done false
14088 4517 Acked Read 4 Sent Done Done false
14089 4518 Acked Flush 0 Sent Done Done false
14090 4519 Acked Read 10 Sent Done Done false
14091 4520 Acked Read 7 Sent Done Done false
14092 4521 Acked Read 8 Sent Done Done false
14093 4522 Acked Write 2 Sent Done Done false
14094 4523 Acked Read 10 Sent Done Done false
14095 4524 Acked Write 10 Sent Done Done false
14096 4525 Acked Write 9 Sent Done Done false
14097 4526 Acked Write 3 Sent Done Done false
14098 4527 Acked Write 1 Sent Done Done false
14099 4528 Acked Write 9 Sent Done Done false
14100 4529 Acked Write 10 Sent Done Done false
14101 4530 Acked Write 9 Sent Done Done false
14102 4531 Acked Flush 0 Sent Done Done false
14103 4532 Acked Read 10 Sent Done Done false
14104 4533 Acked Write 8 Sent Done Done false
14105 4534 Acked Flush 0 Sent Done Done false
14106 4535 Acked Write 4 Sent Done Done false
14107 4536 Acked Write 9 Sent Done Done false
14108 4537 Acked Write 6 Sent Done Done false
14109 4538 Acked Write 10 Sent Done Done false
14110 4539 Acked Write 5 Sent Done Done false
14111 4540 Acked Write 7 Sent Done Done false
14112 4541 Acked Write 10 Sent Done Done false
14113 4542 Acked Read 9 Sent Done Done false
14114 4543 Acked Flush 0 Sent Done Done false
14115 4544 Acked Flush 0 Sent Done Done false
14116 4545 Acked Read 9 Sent Done Done false
14117 4546 Acked Read 9 Sent Done Done false
14118 4547 Acked Write 9 Sent Done Done false
14119 4548 Acked Read 2 Sent Done Done false
14120 4549 Acked Read 6 Sent Done Done false
14121 4550 Acked Write 3 Sent Done Done false
14122 4551 Acked Write 8 Sent Done Done false
14123 4552 Acked Read 4 Sent Done Done false
14124 4553 Acked Read 5 Sent Done Done false
14125 4554 Acked Read 7 Sent Done Done false
14126 4555 Acked Read 3 Sent Done Done false
14127 4556 Acked Read 6 Sent Done Done false
14128 4557 Acked Read 4 Sent Done Done false
14129 4558 Acked Write 8 Sent Done Done false
14130 4559 Acked Read 9 Sent Done Done false
14131 4560 Acked Write 3 Sent Done Done false
14132 4561 Acked Write 8 Sent Done Done false
14133 4562 Acked Write 5 Sent Done Done false
14134 4563 Acked Read 7 Sent Done Done false
14135 4564 Acked Read 9 Sent Done Done false
14136 4565 Acked Read 5 Sent Done Done false
14137 4566 Acked Write 9 Sent Done Done false
14138 4567 Acked Read 3 Sent Done Done false
14139 4568 Acked Write 8 Sent Done Done false
14140 4569 Acked Read 4 Sent Done Done false
14141 4570 Acked Write 6 Sent Done Done false
14142 4571 Acked Write 4 Sent Done Done false
14143 4572 Acked Write 5 Sent Done Done false
14144 4573 Acked Read 8 Sent Done Done false
14145 4574 Acked Flush 0 Sent Done Done false
14146 4575 Acked Read 2 Sent Done Done false
14147 4576 Acked Read 9 Sent Done Done false
14148 4577 Acked Write 2 Sent Done Done false
14149 4578 Acked Write 8 Sent Done Done false
14150 4579 Acked Read 4 Sent Done Done false
14151 4580 Acked Read 4 Sent Done Done false
14152 4581 Acked Read 3 Sent Done Done false
14153 4582 Acked Read 3 Sent Done Done false
14154 4583 Acked Read 6 Sent Done Done false
14155 4584 Acked Read 6 Sent Done Done false
14156 4585 Acked Read 6 Sent Done Done false
14157 4586 Acked Read 4 Sent Done Done false
14158 4587 Acked Read 10 Sent Done Done false
14159 4588 Acked Write 3 Sent Done Done false
14160 4589 Acked Read 6 Sent Done Done false
14161 4590 Acked Write 7 Sent Done Done false
14162 4591 Acked Write 9 Sent Done Done false
14163 4592 Acked Write 5 Sent Done Done false
14164 4593 Acked Read 3 Sent Done Done false
14165 4594 Acked Write 4 Sent Done Done false
14166 4595 Acked Read 6 Sent Done Done false
14167 4596 Acked Read 9 Sent Done Done false
14168 4597 Acked Flush 0 Sent Done Done false
14169 4598 Acked Flush 0 Sent Done Done false
14170 4599 Acked Read 10 Sent Done Done false
14171 4600 Acked Write 1 Sent Done Done false
14172 4601 Acked Write 6 Sent Done Done false
14173 4602 Acked Read 1 Sent Done Done false
14174 4603 Acked Read 10 Sent Done Done false
14175 4604 Acked Flush 0 Sent Done Done false
14176 4605 Acked Write 3 Sent Done Done false
14177 4606 Acked Write 10 Sent Done Done false
14178 4607 Acked Write 10 Sent Done Done false
14179 4608 Acked Write 6 Sent Done Done false
14180 4609 Acked Write 4 Sent Done Done false
14181 4610 Acked Write 6 Sent Done Done false
14182 4611 Acked Read 8 Sent Done Done false
14183 4612 Acked Read 6 Sent Done Done false
14184 4613 Acked Write 10 Sent Done Done false
14185 4614 Acked Read 8 Sent Done Done false
14186 4615 Acked Read 2 Sent Done Done false
14187 4616 Acked Read 8 Sent Done Done false
14188 4617 Acked Flush 0 Sent Done Done false
14189 4618 Acked Write 3 Sent Done Done false
14190 4619 Acked Flush 0 Sent Done Done false
14191 4620 Acked Read 5 Sent Done Done false
14192 4621 Acked Write 9 Sent Done Done false
14193 4622 Acked Write 1 Sent Done Done false
14194 STATES DS:0 DS:1 DS:2 TOTAL
14195 Sent 301 0 0 301
14196 Done 0 301 301 602
14197 Skipped 0 0 0 0
14198 Error 0 0 0 0
14199 Last Flush: Some(JobId(4321)) Some(JobId(4619)) Some(JobId(4619))
14200 Downstairs last five completed: 4321 4320 4319 4318 4317
14201 Upstairs last five completed: 4622 4621 4620 4619 4618
14202 CLIENT: Up:0 ds:301 act:2
142032025-01-29T05:13:47.465ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
142042025-01-29T05:13:48.219ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
142052025-01-29T05:13:49.221ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
142062025-01-29T05:13:50.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
14207 ----------------------------------------------------------------
14208 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14209 States: Offline Active Active
14210 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14211 4322 Acked Flush 0 Sent Done Done false
14212 4323 Acked Write 1 Sent Done Done false
14213 4324 Acked Read 5 Sent Done Done false
14214 4325 Acked Write 8 Sent Done Done false
14215 4326 Acked Write 3 Sent Done Done false
14216 4327 Acked Read 7 Sent Done Done false
14217 4328 Acked Write 5 Sent Done Done false
14218 4329 Acked Read 4 Sent Done Done false
14219 4330 Acked Write 8 Sent Done Done false
14220 4331 Acked Write 3 Sent Done Done false
14221 4332 Acked Write 3 Sent Done Done false
14222 4333 Acked Read 10 Sent Done Done false
14223 4334 Acked Flush 0 Sent Done Done false
14224 4335 Acked Read 4 Sent Done Done false
14225 4336 Acked Read 7 Sent Done Done false
14226 4337 Acked Read 9 Sent Done Done false
14227 4338 Acked Read 4 Sent Done Done false
14228 4339 Acked Read 2 Sent Done Done false
14229 4340 Acked Read 6 Sent Done Done false
14230 4341 Acked Write 1 Sent Done Done false
14231 4342 Acked Write 5 Sent Done Done false
14232 4343 Acked Read 10 Sent Done Done false
14233 4344 Acked Write 9 Sent Done Done false
14234 4345 Acked Read 1 Sent Done Done false
14235 4346 Acked Write 5 Sent Done Done false
14236 4347 Acked Flush 0 Sent Done Done false
14237 4348 Acked Read 7 Sent Done Done false
14238 4349 Acked Read 9 Sent Done Done false
14239 4350 Acked Write 1 Sent Done Done false
14240 4351 Acked Read 9 Sent Done Done false
14241 4352 Acked Read 4 Sent Done Done false
14242 4353 Acked Read 2 Sent Done Done false
14243 4354 Acked Flush 0 Sent Done Done false
14244 4355 Acked Write 8 Sent Done Done false
14245 4356 Acked Read 9 Sent Done Done false
14246 4357 Acked Read 2 Sent Done Done false
14247 4358 Acked Flush 0 Sent Done Done false
14248 4359 Acked Read 1 Sent Done Done false
14249 4360 Acked Write 9 Sent Done Done false
14250 4361 Acked Read 1 Sent Done Done false
14251 4362 Acked Read 1 Sent Done Done false
14252 4363 Acked Read 4 Sent Done Done false
14253 4364 Acked Read 9 Sent Done Done false
14254 4365 Acked Flush 0 Sent Done Done false
14255 4366 Acked Read 1 Sent Done Done false
14256 4367 Acked Flush 0 Sent Done Done false
14257 4368 Acked Write 2 Sent Done Done false
14258 4369 Acked Read 3 Sent Done Done false
14259 4370 Acked Write 3 Sent Done Done false
14260 4371 Acked Read 3 Sent Done Done false
14261 4372 Acked Read 5 Sent Done Done false
14262 4373 Acked Read 3 Sent Done Done false
14263 4374 Acked Read 8 Sent Done Done false
14264 4375 Acked Read 9 Sent Done Done false
14265 4376 Acked Flush 0 Sent Done Done false
14266 4377 Acked Write 2 Sent Done Done false
14267 4378 Acked Write 10 Sent Done Done false
14268 4379 Acked Read 10 Sent Done Done false
14269 4380 Acked Flush 0 Sent Done Done false
14270 4381 Acked Read 7 Sent Done Done false
14271 4382 Acked Read 8 Sent Done Done false
14272 4383 Acked Read 5 Sent Done Done false
14273 4384 Acked Write 6 Sent Done Done false
14274 4385 Acked Write 7 Sent Done Done false
14275 4386 Acked Read 7 Sent Done Done false
14276 4387 Acked Write 3 Sent Done Done false
14277 4388 Acked Read 3 Sent Done Done false
14278 4389 Acked Read 6 Sent Done Done false
14279 4390 Acked Read 6 Sent Done Done false
14280 4391 Acked Read 7 Sent Done Done false
14281 4392 Acked Read 8 Sent Done Done false
14282 4393 Acked Write 2 Sent Done Done false
14283 4394 Acked Read 9 Sent Done Done false
14284 4395 Acked Write 10 Sent Done Done false
14285 4396 Acked Write 3 Sent Done Done false
14286 4397 Acked Write 3 Sent Done Done false
14287 4398 Acked Write 2 Sent Done Done false
14288 4399 Acked Write 3 Sent Done Done false
14289 4400 Acked Write 8 Sent Done Done false
14290 4401 Acked Read 4 Sent Done Done false
14291 4402 Acked Read 10 Sent Done Done false
14292 4403 Acked Read 10 Sent Done Done false
14293 4404 Acked Read 7 Sent Done Done false
14294 4405 Acked Write 10 Sent Done Done false
14295 4406 Acked Read 2 Sent Done Done false
14296 4407 Acked Read 3 Sent Done Done false
14297 4408 Acked Read 9 Sent Done Done false
14298 4409 Acked Read 1 Sent Done Done false
14299 4410 Acked Flush 0 Sent Done Done false
14300 4411 Acked Read 1 Sent Done Done false
14301 4412 Acked Read 1 Sent Done Done false
14302 4413 Acked Read 6 Sent Done Done false
14303 4414 Acked Read 9 Sent Done Done false
14304 4415 Acked Read 1 Sent Done Done false
14305 4416 Acked Flush 0 Sent Done Done false
14306 4417 Acked Write 3 Sent Done Done false
14307 4418 Acked Write 2 Sent Done Done false
14308 4419 Acked Flush 0 Sent Done Done false
14309 4420 Acked Read 2 Sent Done Done false
14310 4421 Acked Read 5 Sent Done Done false
14311 4422 Acked Write 6 Sent Done Done false
14312 4423 Acked Flush 0 Sent Done Done false
14313 4424 Acked Write 7 Sent Done Done false
14314 4425 Acked Read 1 Sent Done Done false
14315 4426 Acked Read 5 Sent Done Done false
14316 4427 Acked Write 1 Sent Done Done false
14317 4428 Acked Read 3 Sent Done Done false
14318 4429 Acked Read 9 Sent Done Done false
14319 4430 Acked Write 9 Sent Done Done false
14320 4431 Acked Read 9 Sent Done Done false
14321 4432 Acked Read 4 Sent Done Done false
14322 4433 Acked Read 4 Sent Done Done false
14323 4434 Acked Write 4 Sent Done Done false
14324 4435 Acked Write 5 Sent Done Done false
14325 4436 Acked Read 6 Sent Done Done false
14326 4437 Acked Write 1 Sent Done Done false
14327 4438 Acked Read 8 Sent Done Done false
14328 4439 Acked Read 6 Sent Done Done false
14329 4440 Acked Read 7 Sent Done Done false
14330 4441 Acked Read 10 Sent Done Done false
14331 4442 Acked Read 3 Sent Done Done false
14332 4443 Acked Write 10 Sent Done Done false
14333 4444 Acked Read 1 Sent Done Done false
14334 4445 Acked Read 3 Sent Done Done false
14335 4446 Acked Read 1 Sent Done Done false
14336 4447 Acked Flush 0 Sent Done Done false
14337 4448 Acked Write 3 Sent Done Done false
14338 4449 Acked Write 7 Sent Done Done false
14339 4450 Acked Write 3 Sent Done Done false
14340 4451 Acked Read 7 Sent Done Done false
14341 4452 Acked Read 8 Sent Done Done false
14342 4453 Acked Write 8 Sent Done Done false
14343 4454 Acked Flush 0 Sent Done Done false
14344 4455 Acked Write 6 Sent Done Done false
14345 4456 Acked Read 5 Sent Done Done false
14346 4457 Acked Write 3 Sent Done Done false
14347 4458 Acked Read 5 Sent Done Done false
14348 4459 Acked Read 9 Sent Done Done false
14349 4460 Acked Read 5 Sent Done Done false
14350 4461 Acked Write 7 Sent Done Done false
14351 4462 Acked Read 3 Sent Done Done false
14352 4463 Acked Read 7 Sent Done Done false
14353 4464 Acked Read 8 Sent Done Done false
14354 4465 Acked Read 5 Sent Done Done false
14355 4466 Acked Write 6 Sent Done Done false
14356 4467 Acked Write 1 Sent Done Done false
14357 4468 Acked Read 7 Sent Done Done false
14358 4469 Acked Write 6 Sent Done Done false
14359 4470 Acked Read 7 Sent Done Done false
14360 4471 Acked Read 4 Sent Done Done false
14361 4472 Acked Flush 0 Sent Done Done false
14362 4473 Acked Write 2 Sent Done Done false
14363 4474 Acked Write 2 Sent Done Done false
14364 4475 Acked Read 3 Sent Done Done false
14365 4476 Acked Read 2 Sent Done Done false
14366 4477 Acked Write 6 Sent Done Done false
14367 4478 Acked Read 7 Sent Done Done false
14368 4479 Acked Flush 0 Sent Done Done false
14369 4480 Acked Read 1 Sent Done Done false
14370 4481 Acked Read 7 Sent Done Done false
14371 4482 Acked Read 10 Sent Done Done false
14372 4483 Acked Read 6 Sent Done Done false
14373 4484 Acked Read 3 Sent Done Done false
14374 4485 Acked Write 2 Sent Done Done false
14375 4486 Acked Write 4 Sent Done Done false
14376 4487 Acked Read 6 Sent Done Done false
14377 4488 Acked Write 10 Sent Done Done false
14378 4489 Acked Write 1 Sent Done Done false
14379 4490 Acked Write 5 Sent Done Done false
14380 4491 Acked Write 3 Sent Done Done false
14381 4492 Acked Write 4 Sent Done Done false
14382 4493 Acked Read 5 Sent Done Done false
14383 4494 Acked Read 8 Sent Done Done false
14384 4495 Acked Read 3 Sent Done Done false
14385 4496 Acked Write 5 Sent Done Done false
14386 4497 Acked Read 5 Sent Done Done false
14387 4498 Acked Read 1 Sent Done Done false
14388 4499 Acked Read 3 Sent Done Done false
14389 4500 Acked Write 7 Sent Done Done false
14390 4501 Acked Write 10 Sent Done Done false
14391 4502 Acked Read 5 Sent Done Done false
14392 4503 Acked Read 2 Sent Done Done false
14393 4504 Acked Flush 0 Sent Done Done false
14394 4505 Acked Read 9 Sent Done Done false
14395 4506 Acked Write 6 Sent Done Done false
14396 4507 Acked Write 8 Sent Done Done false
14397 4508 Acked Write 2 Sent Done Done false
14398 4509 Acked Flush 0 Sent Done Done false
14399 4510 Acked Write 1 Sent Done Done false
14400 4511 Acked Read 10 Sent Done Done false
14401 4512 Acked Write 9 Sent Done Done false
14402 4513 Acked Read 1 Sent Done Done false
14403 4514 Acked Write 8 Sent Done Done false
14404 4515 Acked Read 8 Sent Done Done false
14405 4516 Acked Write 1 Sent Done Done false
14406 4517 Acked Read 4 Sent Done Done false
14407 4518 Acked Flush 0 Sent Done Done false
14408 4519 Acked Read 10 Sent Done Done false
14409 4520 Acked Read 7 Sent Done Done false
14410 4521 Acked Read 8 Sent Done Done false
14411 4522 Acked Write 2 Sent Done Done false
14412 4523 Acked Read 10 Sent Done Done false
14413 4524 Acked Write 10 Sent Done Done false
14414 4525 Acked Write 9 Sent Done Done false
14415 4526 Acked Write 3 Sent Done Done false
14416 4527 Acked Write 1 Sent Done Done false
14417 4528 Acked Write 9 Sent Done Done false
14418 4529 Acked Write 10 Sent Done Done false
14419 4530 Acked Write 9 Sent Done Done false
14420 4531 Acked Flush 0 Sent Done Done false
14421 4532 Acked Read 10 Sent Done Done false
14422 4533 Acked Write 8 Sent Done Done false
14423 4534 Acked Flush 0 Sent Done Done false
14424 4535 Acked Write 4 Sent Done Done false
14425 4536 Acked Write 9 Sent Done Done false
14426 4537 Acked Write 6 Sent Done Done false
14427 4538 Acked Write 10 Sent Done Done false
14428 4539 Acked Write 5 Sent Done Done false
14429 4540 Acked Write 7 Sent Done Done false
14430 4541 Acked Write 10 Sent Done Done false
14431 4542 Acked Read 9 Sent Done Done false
14432 4543 Acked Flush 0 Sent Done Done false
14433 4544 Acked Flush 0 Sent Done Done false
14434 4545 Acked Read 9 Sent Done Done false
14435 4546 Acked Read 9 Sent Done Done false
14436 4547 Acked Write 9 Sent Done Done false
14437 4548 Acked Read 2 Sent Done Done false
14438 4549 Acked Read 6 Sent Done Done false
14439 4550 Acked Write 3 Sent Done Done false
14440 4551 Acked Write 8 Sent Done Done false
14441 4552 Acked Read 4 Sent Done Done false
14442 4553 Acked Read 5 Sent Done Done false
14443 4554 Acked Read 7 Sent Done Done false
14444 4555 Acked Read 3 Sent Done Done false
14445 4556 Acked Read 6 Sent Done Done false
14446 4557 Acked Read 4 Sent Done Done false
14447 4558 Acked Write 8 Sent Done Done false
14448 4559 Acked Read 9 Sent Done Done false
14449 4560 Acked Write 3 Sent Done Done false
14450 4561 Acked Write 8 Sent Done Done false
14451 4562 Acked Write 5 Sent Done Done false
14452 4563 Acked Read 7 Sent Done Done false
14453 4564 Acked Read 9 Sent Done Done false
14454 4565 Acked Read 5 Sent Done Done false
14455 4566 Acked Write 9 Sent Done Done false
14456 4567 Acked Read 3 Sent Done Done false
14457 4568 Acked Write 8 Sent Done Done false
14458 4569 Acked Read 4 Sent Done Done false
14459 4570 Acked Write 6 Sent Done Done false
14460 4571 Acked Write 4 Sent Done Done false
14461 4572 Acked Write 5 Sent Done Done false
14462 4573 Acked Read 8 Sent Done Done false
14463 4574 Acked Flush 0 Sent Done Done false
14464 4575 Acked Read 2 Sent Done Done false
14465 4576 Acked Read 9 Sent Done Done false
14466 4577 Acked Write 2 Sent Done Done false
14467 4578 Acked Write 8 Sent Done Done false
14468 4579 Acked Read 4 Sent Done Done false
14469 4580 Acked Read 4 Sent Done Done false
14470 4581 Acked Read 3 Sent Done Done false
14471 4582 Acked Read 3 Sent Done Done false
14472 4583 Acked Read 6 Sent Done Done false
14473 4584 Acked Read 6 Sent Done Done false
14474 4585 Acked Read 6 Sent Done Done false
14475 4586 Acked Read 4 Sent Done Done false
14476 4587 Acked Read 10 Sent Done Done false
14477 4588 Acked Write 3 Sent Done Done false
14478 4589 Acked Read 6 Sent Done Done false
14479 4590 Acked Write 7 Sent Done Done false
14480 4591 Acked Write 9 Sent Done Done false
14481 4592 Acked Write 5 Sent Done Done false
14482 4593 Acked Read 3 Sent Done Done false
14483 4594 Acked Write 4 Sent Done Done false
14484 4595 Acked Read 6 Sent Done Done false
14485 4596 Acked Read 9 Sent Done Done false
14486 4597 Acked Flush 0 Sent Done Done false
14487 4598 Acked Flush 0 Sent Done Done false
14488 4599 Acked Read 10 Sent Done Done false
14489 4600 Acked Write 1 Sent Done Done false
14490 4601 Acked Write 6 Sent Done Done false
14491 4602 Acked Read 1 Sent Done Done false
14492 4603 Acked Read 10 Sent Done Done false
14493 4604 Acked Flush 0 Sent Done Done false
14494 4605 Acked Write 3 Sent Done Done false
14495 4606 Acked Write 10 Sent Done Done false
14496 4607 Acked Write 10 Sent Done Done false
14497 4608 Acked Write 6 Sent Done Done false
14498 4609 Acked Write 4 Sent Done Done false
14499 4610 Acked Write 6 Sent Done Done false
14500 4611 Acked Read 8 Sent Done Done false
14501 4612 Acked Read 6 Sent Done Done false
14502 4613 Acked Write 10 Sent Done Done false
14503 4614 Acked Read 8 Sent Done Done false
14504 4615 Acked Read 2 Sent Done Done false
14505 4616 Acked Read 8 Sent Done Done false
14506 4617 Acked Flush 0 Sent Done Done false
14507 4618 Acked Write 3 Sent Done Done false
14508 4619 Acked Flush 0 Sent Done Done false
14509 4620 Acked Read 5 Sent Done Done false
14510 4621 Acked Write 9 Sent Done Done false
14511 4622 Acked Write 1 Sent Done Done false
14512 STATES DS:0 DS:1 DS:2 TOTAL
14513 Sent 301 0 0 301
14514 Done 0 301 301 602
14515 Skipped 0 0 0 0
14516 Error 0 0 0 0
14517 Last Flush: Some(JobId(4321)) Some(JobId(4619)) Some(JobId(4619))
14518 Downstairs last five completed: 4321 4320 4319 4318 4317
14519 Upstairs last five completed: 4622 4621 4620 4619 4618
14520 CLIENT: Up:0 ds:301 act:2
145212025-01-29T05:13:51.224ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145222025-01-29T05:13:52.466ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145232025-01-29T05:13:53.226ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145242025-01-29T05:13:54.215ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145252025-01-29T05:13:54.215ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51259) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145262025-01-29T05:13:54.216ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145272025-01-29T05:13:54.217ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145282025-01-29T05:13:54.217ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145292025-01-29T05:13:54.217ZINFOcrucible: send last flush ID to this DS: Some(JobId(4321)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145302025-01-29T05:13:54.217ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145312025-01-29T05:13:54.217ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4321)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145322025-01-29T05:13:54.217ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4321)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
14533 ----------------------------------------------------------------
14534 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
14535 States: Active Active Active
14536 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14537 4620 Acked Read 5 Done Done Done true
14538 4621 Acked Write 9 Done Done Done true
14539 4622 Acked Write 1 Done Done Done true
14540 STATES DS:0 DS:1 DS:2 TOTAL
14541 Sent 0 0 0 0
14542 Done 3 3 3 9
14543 Skipped 0 0 0 0
14544 Error 0 0 0 0
14545 Last Flush: Some(JobId(4619)) Some(JobId(4619)) Some(JobId(4619))
14546 Downstairs last five completed: 4619 4618 4617 4616 4615
14547 Upstairs last five completed: 4622 4621 4620 4619 4618
14548 CLIENT: Up:0 ds:3 act:3
14549 ----------------------------------------------------------------
14550 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14551 Downstairs last five completed: 4623 4622 4621 4620 4619
14552 Upstairs last five completed: 4623 4622 4621 4620 4619
14553 CLIENT: Up:0 ds:0 act:3
14554 Replay: All jobs finished
14555 Main task has work to do, go find it
14556 got dsc Stop(2)
14557 stop 2
14558 Jan 29 05:13:58.468 INFO [8830] Got stop action so:false kr:false
14559 request completed, latency_us: 381, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 63ec8fea-9345-4b60-9c2b-6af1100d0f5c, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
145602025-01-29T05:13:58.469ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145612025-01-29T05:13:58.469ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
145622025-01-29T05:13:58.469ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
14563 Jan 29 05:13:58.469 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/2, method:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:13:58.469527603Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
14564 GET, req_id: d5eba11d-4da2-449a-bbf8-1a576b9a7d70, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
14565 [8830] Exited with: ExitStatus(unix_wait_status(9))
14566 [8830][2] reports Exit
14567 Jan 29 05:14:02.471 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: cdc771d4-93bd-43bf-aa97-99c258e88cd3, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
14568 001/300 Read block 1198 len 20480
14569 002/300 Read block 1524 len 8192
14570 003/300 Read block 1402 len 12288
14571 004/300 Read block 2335 len 8192
14572 005/300 Read block 4538 len 24576
14573 006/300 Write block 4650 len 40960 data: 4 4 5 4 3 4 4 4 4 4
14574 007/300 Read block 159 len 20480
14575 008/300 Read block 3153 len 36864
14576 009/300 Read block 2986 len 40960
14577 010/300 Read block 549 len 8192
14578 011/300 Flush
14579 012/300 Write block 1287 len 20480 data: 6 6 5 3 2
14580 013/300 Read block 2450 len 20480
14581 014/300 Read block 2737 len 28672
14582 015/300 Flush
14583 016/300 Read block 64 len 28672
14584 017/300 Read block 3529 len 20480
14585 018/300 Read block 309 len 32768
14586 019/300 Write block 2913 len 20480 data: 4 4 4 4 4
14587 020/300 Write block 3502 len 4096 data: 4
14588 021/300 Read block 4832 len 20480
14589 022/300 Read block 4273 len 12288
14590 023/300 Write block 2371 len 4096 data: 4
14591 024/300 Write block 1987 len 24576 data: 4 4 4 4 5 5
14592 025/300 Read block 4328 len 36864
14593 026/300 Write block 4445 len 8192 data: 2 2
14594 027/300 Write block 3016 len 40960 data: 2 3 4 4 3 3 2 2 2 2
14595 028/300 Flush
14596 029/300 Read block 1684 len 16384
14597 030/300 Read block 3452 len 8192
14598 031/300 Read block 1587 len 40960
14599 032/300 Write block 6 len 20480 data: 3 3 3 3 2
14600 033/300 Read block 288 len 8192
14601 034/300 Read block 2425 len 36864
14602 035/300 Write block 1850 len 4096 data: 3
14603 036/300 Write block 2574 len 24576 data: 4 4 5 4 4 3
14604 037/300 Write block 952 len 36864 data: 7 7 6 5 4 4 3 2 3
14605 038/300 Read block 4948 len 20480
14606 039/300 Read block 317 len 12288
14607 040/300 Read block 4019 len 4096
14608 041/300 Read block 3211 len 20480
14609 042/300 Flush
14610 043/300 Write block 791 len 40960 data: 4 5 5 5 5 6 5 4 4 3
14611 044/300 Read block 4378 len 40960
14612 045/300 Write block 3691 len 20480 data: 2 2 2 2 2
14613 046/300 Write block 1707 len 32768 data: 3 3 3 3 4 4 5 4
14614 047/300 Read block 2740 len 28672
14615 048/300 Write block 4608 len 20480 data: 2 2 4 5 6
14616 049/300 Flush
14617 050/300 Read block 3600 len 28672
14618 051/300 Flush
14619 052/300 Flush
14620 053/300 Read block 801 len 28672
14621 054/300 Write block 4587 len 24576 data: 2 2 4 4 4 4
14622 055/300 Write block 3294 len 28672 data: 2 2 2 2 2 2 2
14623 056/300 Read block 3496 len 28672
14624 057/300 Read block 1480 len 16384
14625 058/300 Read block 3173 len 40960
14626 059/300 Read block 3174 len 8192
14627 060/300 Write block 2832 len 36864 data: 5 5 4 5 4 6 5 5 5
14628 061/300 Read block 3494 len 32768
14629 062/300 Write block 4953 len 36864 data: 4 4 5 4 6 6 6 4 4
14630 063/300 Read block 4116 len 16384
14631 064/300 Read block 4914 len 20480
14632 065/300 Read block 2212 len 28672
14633 066/300 Write block 4276 len 12288 data: 4 5 4
14634 067/300 Read block 3810 len 36864
14635 068/300 Flush
14636 069/300 Read block 96 len 8192
14637 070/300 Read block 836 len 36864
14638 071/300 Flush
14639 072/300 Write block 2327 len 20480 data: 2 2 2 2 3
14640 073/300 Write block 241 len 4096 data: 7
14641 074/300 Read block 3255 len 24576
14642 075/300 Read block 2248 len 24576
14643 076/300 Write block 2618 len 28672 data: 3 3 3 3 3 3 3
14644 077/300 Write block 1867 len 36864 data: 5 5 3 3 3 3 4 4 4
14645 078/300 Read block 2667 len 8192
14646 079/300 Write block 3013 len 28672 data: 2 2 2 3 4 5 5
14647 080/300 Read block 4766 len 40960
14648 081/300 Flush
14649 082/300 Read block 1056 len 36864
14650 083/300 Flush
14651 084/300 Read block 2419 len 4096
14652 085/300 Read block 4449 len 40960
14653 086/300 Read block 4137 len 8192
14654 087/300 Write block 1505 len 40960 data: 3 3 3 3 3 3 3 3 3 3
14655 088/300 Read block 4564 len 40960
14656 089/300 Read block 1645 len 20480
14657 090/300 Flush
14658 091/300 Write block 957 len 8192 data: 5 4
14659 092/300 Write block 2036 len 24576 data: 3 3 3 2 2 2
14660 093/300 Read block 2309 len 12288
14661 094/300 Write block 2941 len 32768 data: 3 3 3 4 4 7 7 7
14662 095/300 Write block 2484 len 12288 data: 3 3 5
14663 096/300 Read block 3436 len 40960
14664 097/300 Read block 2931 len 28672
14665 098/300 Read block 2593 len 16384
14666 099/300 Read block 999 len 40960
14667 100/300 Read block 521 len 4096
14668 101/300 Flush
14669 102/300 Write block 2433 len 32768 data: 3 4 4 4 4 6 4 4
14670 103/300 Read block 1098 len 12288
14671 104/300 Write block 4613 len 28672 data: 5 4 5 3 3 3 3
14672 105/300 Read block 1131 len 4096
14673 106/300 Write block 4475 len 8192 data: 4 5
14674 107/300 Read block 2719 len 16384
14675 108/300 Read block 2057 len 32768
14676 109/300 Flush
14677 110/300 Write block 4635 len 36864 data: 3 5 5 6 5 5 5 5 6
14678 111/300 Flush
14679 112/300 Write block 1506 len 28672 data: 4 4 4 4 4 4 4
14680 113/300 Write block 4921 len 12288 data: 3 3 3
14681 114/300 Write block 1164 len 12288 data: 3 3 3
14682 115/300 Write block 4563 len 40960 data: 2 2 2 2 3 3 4 4 3 2
14683 116/300 Write block 1522 len 8192 data: 3 3
14684 117/300 Write block 213 len 12288 data: 2 2 2
14685 118/300 Write block 1888 len 24576 data: 6 6 5 5 4 4
14686 119/300 Write block 4426 len 12288 data: 3 4 4
14687 120/300 Write block 1413 len 24576 data: 3 3 3 5 4 4
14688 121/300 Read block 1783 len 8192
14689 122/300 Write block 1795 len 12288 data: 5 4 4
14690 123/300 Write block 637 len 24576 data: 5 6 7 7 6 6
14691 124/300 Read block 6 len 20480
14692 125/300 Read block 3315 len 8192
14693 126/300 Write block 3011 len 28672 data: 3 3 3 3 3 4 5
14694 127/300 Read block 4462 len 28672
14695 128/300 Read block 4122 len 40960
14696 129/300 Read block 2913 len 32768
14697 130/300 Read block 1384 len 32768
14698 131/300 Read block 3892 len 28672
14699 132/300 Write block 3441 len 8192 data: 4 4
14700 133/300 Read block 20 len 20480
14701 134/300 Read block 3707 len 4096
14702 135/300 Read block 2270 len 36864
14703 136/300 Write block 3101 len 32768 data: 2 2 3 3 4 4 4 4
14704 137/300 Write block 3282 len 40960 data: 2 3 3 3 3 3 3 3 3 2
14705 138/300 Read block 3355 len 36864
14706 139/300 Read block 1195 len 40960
14707 140/300 Flush
14708 141/300 Read block 1786 len 8192
14709 142/300 Write block 3761 len 20480 data: 2 3 4 5 6
14710 143/300 Read block 3274 len 24576
14711 144/300 Read block 1102 len 20480
14712 145/300 Read block 2278 len 16384
14713 146/300 Write block 1884 len 4096 data: 7
14714 147/300 Read block 4349 len 24576
14715 148/300 Write block 3642 len 8192 data: 5 5
14716 149/300 Read block 3626 len 28672
14717 150/300 Read block 1077 len 40960
14718 151/300 Read block 3151 len 36864
14719 152/300 Write block 4886 len 36864 data: 7 6 6 6 6 4 3 3 3
14720 153/300 Read block 4573 len 32768
14721 154/300 Write block 4800 len 24576 data: 3 3 4 3 3 4
14722 155/300 Write block 3608 len 16384 data: 4 6 5 5
14723 156/300 Read block 2930 len 32768
14724 157/300 Flush
14725 158/300 Write block 1620 len 28672 data: 3 5 6 6 7 6 7
14726 159/300 Read block 1977 len 32768
14727 160/300 Read block 2200 len 36864
14728 161/300 Write block 4393 len 8192 data: 8 8
14729 162/300 Flush
14730 163/300 Read block 466 len 4096
14731 164/300 Read block 4839 len 40960
14732 165/300 Read block 702 len 40960
14733 166/300 Flush
14734 167/300 Read block 377 len 8192
14735 168/300 Read block 3709 len 24576
14736 169/300 Read block 3237 len 20480
14737 170/300 Read block 4143 len 12288
14738 171/300 Read block 682 len 28672
14739 172/300 Read block 1011 len 16384
14740 173/300 Write block 1466 len 12288 data: 2 3 3
14741 174/300 Read block 2970 len 8192
14742 175/300 Read block 4943 len 16384
14743 176/300 Write block 4729 len 16384 data: 2 3 2 2
14744 177/300 Read block 4482 len 20480
14745 178/300 Write block 1970 len 20480 data: 6 5 4 6 5
14746 179/300 Read block 155 len 20480
14747 180/300 Read block 1822 len 28672
14748 181/300 Read block 3447 len 24576
14749 182/300 Read block 3490 len 16384
14750 183/300 Read block 765 len 20480
14751 184/300 Write block 4544 len 28672 data: 4 3 2 4 5 6 5
14752 185/300 Read block 2346 len 24576
14753 186/300 Write block 993 len 24576 data: 2 2 2 3 3 3
14754 187/300 Read block 357 len 28672
14755 188/300 Read block 831 len 4096
14756 189/300 Write block 2882 len 36864 data: 4 3 3 4 4 4 4 4 4
14757 190/300 Write block 1409 len 32768 data: 2 3 3 3 4 4 4 6
14758 191/300 Write block 3936 len 28672 data: 6 6 6 6 6 6 7
14759 192/300 Flush
14760 193/300 Write block 2880 len 4096 data: 2
14761 194/300 Read block 1075 len 4096
14762 195/300 Write block 4604 len 12288 data: 8 6 4
14763 196/300 Write block 4895 len 4096 data: 4
14764 197/300 Read block 1179 len 20480
14765 198/300 Read block 865 len 40960
14766 199/300 Read block 4224 len 20480
14767 200/300 Write block 2309 len 28672 data: 4 5 5 4 4 5 5
14768 201/300 Read block 2341 len 8192
14769 202/300 Read block 1054 len 8192
14770 203/300 Flush
14771 204/300 Read block 2672 len 28672
14772 205/300 Read block 2356 len 20480
14773 206/300 Read block 1197 len 32768
14774 207/300 Write block 2954 len 20480 data: 5 4 3 3 3
14775 208/300 Read block 2359 len 32768
14776 209/300 Write block 3801 len 8192 data: 3 3
14777 210/300 Read block 950 len 28672
14778 211/300 Read block 2963 len 8192
14779 212/300 Read block 2795 len 16384
14780 213/300 Read block 4433 len 16384
14781 214/300 Write block 4196 len 32768 data: 3 4 5 5 6 6 6 6
14782 215/300 Read block 573 len 40960
14783 216/300 Read block 4253 len 12288
14784 217/300 Read block 4463 len 20480
14785 218/300 Write block 1096 len 24576 data: 3 3 2 3 4 3
14786 219/300 Read block 3989 len 40960
14787 220/300 Write block 124 len 20480 data: 8 7 6 7 4
14788 221/300 Read block 3649 len 28672
14789 222/300 Read block 61 len 28672
14790 223/300 Read block 4985 len 28672
14791 224/300 Write block 4141 len 8192 data: 6 4
14792 225/300 Write block 906 len 40960 data: 2 2 2 3 3 3 3 4 4 4
14793 226/300 Write block 307 len 36864 data: 3 3 3 3 3 4 3 4 4
14794 227/300 Write block 1212 len 8192 data: 2 3
14795 228/300 Flush
14796 229/300 Write block 4013 len 16384 data: 4 2 2 2
14797 230/300 Write block 516 len 12288 data: 5 5 4
14798 231/300 Read block 3728 len 4096
14799 232/300 Read block 1578 len 32768
14800 233/300 Read block 979 len 16384
14801 234/300 Write block 2971 len 12288 data: 3 4 3
14802 235/300 Read block 1614 len 28672
14803 236/300 Write block 4417 len 12288 data: 3 4 4
14804 237/300 Flush
14805 238/300 Read block 2071 len 12288
14806 239/300 Flush
14807 240/300 Read block 1547 len 28672
14808 241/300 Read block 4282 len 4096
14809 242/300 Write block 489 len 12288 data: 3 4 6
14810 243/300 Read block 3900 len 32768
14811 244/300 Read block 613 len 32768
14812 245/300 Write block 3885 len 8192 data: 2 2
14813 246/300 Read block 4542 len 16384
14814 247/300 Read block 2228 len 32768
14815 248/300 Write block 3551 len 8192 data: 4 3
14816 249/300 Write block 4353 len 24576 data: 3 3 2 4 4 4
14817 250/300 Read block 2837 len 36864
14818 251/300 Read block 4970 len 16384
14819 252/300 Write block 4745 len 20480 data: 5 4 4 4 3
14820 253/300 Read block 2083 len 16384
14821 254/300 Write block 2543 len 32768 data: 2 2 2 2 4 4 4 3
14822 255/300 Read block 2171 len 36864
14823 256/300 Read block 2055 len 36864
14824 257/300 Flush
14825 258/300 Read block 2706 len 20480
14826 259/300 Write block 463 len 4096 data: 2
14827 260/300 Write block 2508 len 24576 data: 3 2 2 2 2 2
14828 261/300 Write block 2460 len 16384 data: 3 2 3 3
14829 262/300 Read block 4574 len 36864
14830 263/300 Write block 4614 len 16384 data: 5 6 4 4
14831 264/300 Write block 1536 len 32768 data: 2 2 4 4 3 3 3 2
14832 265/300 Write block 558 len 8192 data: 4 4
14833 266/300 Write block 2198 len 36864 data: 3 5 4 4 3 3 4 4 4
14834 267/300 Write block 2911 len 36864 data: 3 4 5 5 5 5 5 4 4
14835 268/300 Read block 4053 len 24576
14836 269/300 Read block 4296 len 8192
14837 270/300 Read block 2430 len 24576
14838 271/300 Write block 2376 len 28672 data: 4 5 4 4 5 5 7
14839 272/300 Read block 4389 len 40960
14840 273/300 Read block 3779 len 20480
14841 274/300 Write block 2445 len 16384 data: 3 3 3 4
14842 275/300 Flush
14843 276/300 Write block 4998 len 8192 data: 3 3
14844 277/300 Write block 4748 len 16384 data: 5 4 4 4
14845 278/300 Read block 1265 len 8192
14846 279/300 Read block 3213 len 4096
14847 280/300 Flush
14848 281/300 Write block 2209 len 24576 data: 5 5 5 5 3 4
14849 282/300 Write block 4729 len 28672 data: 3 4 3 3 4 4 5
14850 283/300 Write block 4392 len 40960 data: 8 9 9 8 7 5 6 6 5 6
14851 284/300 Write block 471 len 28672 data: 4 3 4 4 4 4 4
14852 285/300 Read block 3645 len 8192
14853 286/300 Write block 4714 len 36864 data: 4 5 5 3 3 4 3 2 3
14854 287/300 Flush
14855 288/300 Write block 39 len 24576 data: 3 3 3 3 3 5
14856 289/300 Read block 4908 len 12288
14857 290/300 Write block 1330 len 28672 data: 3 3 3 3 3 3 3
14858 291/300 Write block 1519 len 12288 data: 4 3 4
14859 292/300 Write block 1719 len 20480 data: 2 3 3 3 3
14860 293/300 Read block 3469 len 16384
14861 294/300 Read block 1549 len 4096
14862 295/300 Read block 3264 len 16384
14863 296/300 Write block 178 len 20480 data: 4 3 3 3 3
14864 297/300 Write block 890 len 36864 data: 4 5 5 4 4 4 2 2 3
14865 298/300 Read block 2260 len 36864
14866 299/300 Read block 3451 len 12288
14867 300/300 Read block 3251 len 8192
14868 301/300 Read block 203 len 16384
14869 Main task has work to do, go find it
14870 got dsc Start(2)
14871 start 2
14872 Jan 29 05:14:02.722 INFO request completed, latency_us: 184, response_code: 204, uri: /start/cid/2, method: POST, req_id: 00fb440f-093c-4af8-9346-d536c5cfb578, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
14873 Replay: started 2, returned:Ok(())
14874 ----------------------------------------------------------------
14875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14876 States: Active Active Offline
14877 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14878 4624 Acked Read 5 Done Done Sent false
14879 4625 Acked Read 2 Done Done Sent false
14880 4626 Acked Read 3 Done Done Sent false
14881 4627 Acked Read 2 Done Done Sent false
14882 4628 Acked Read 6 Done Done Sent false
14883 4629 Acked Write 10 Done Done Sent false
14884 4630 Acked Read 5 Done Done Sent false
14885 4631 Acked Read 9 Done Done Sent false
14886 4632 Acked Read 10 Done Done Sent false
14887 4633 Acked Read 2 Done Done Sent false
14888 4634 Acked Flush 0 Done Done Sent false
14889 4635 Acked Write 5 Done Done Sent false
14890 4636 Acked Read 5 Done Done Sent false
14891 4637 Acked Read 7 Done Done Sent false
14892 4638 Acked Flush 0 Done Done Sent false
14893 4639 Acked Read 7 Done Done Sent false
14894 4640 Acked Read 5 Done Done Sent false
14895 4641 Acked Read 8 Done Done Sent false
14896 4642 Acked Write 5 Done Done Sent false
14897 4643 Acked Write 1 Done Done Sent false
14898 4644 Acked Read 5 Done Done Sent false
14899 4645 Acked Read 3 Done Done Sent false
14900 4646 Acked Write 1 Done Done Sent false
14901 4647 Acked Write 6 Done Done Sent false
14902 4648 Acked Read 9 Done Done Sent false
14903 4649 Acked Write 2 Done Done Sent false
14904 4650 Acked Write 10 Done Done Sent false
14905 4651 Acked Flush 0 Done Done Sent false
14906 4652 Acked Read 4 Done Done Sent false
14907 4653 Acked Read 2 Done Done Sent false
14908 4654 Acked Read 10 Done Done Sent false
14909 4655 Acked Write 5 Done Done Sent false
14910 4656 Acked Read 2 Done Done Sent false
14911 4657 Acked Read 9 Done Done Sent false
14912 4658 Acked Write 1 Done Done Sent false
14913 4659 Acked Write 6 Done Done Sent false
14914 4660 Acked Write 9 Done Done Sent false
14915 4661 Acked Read 5 Done Done Sent false
14916 4662 Acked Read 3 Done Done Sent false
14917 4663 Acked Read 1 Done Done Sent false
14918 4664 Acked Read 5 Done Done Sent false
14919 4665 Acked Flush 0 Done Done Sent false
14920 4666 Acked Write 10 Done Done Sent false
14921 4667 Acked Read 10 Done Done Sent false
14922 4668 Acked Write 5 Done Done Sent false
14923 4669 Acked Write 8 Done Done Sent false
14924 4670 Acked Read 7 Done Done Sent false
14925 4671 Acked Write 5 Done Done Sent false
14926 4672 Acked Flush 0 Done Done Sent false
14927 4673 Acked Read 7 Done Done Sent false
14928 4674 Acked Flush 0 Done Done Sent false
14929 4675 Acked Flush 0 Done Done Sent false
14930 4676 Acked Read 7 Done Done Sent false
14931 4677 Acked Write 6 Done Done Sent false
14932 4678 Acked Write 7 Done Done Sent false
14933 4679 Acked Read 7 Done Done Sent false
14934 4680 Acked Read 4 Done Done Sent false
14935 4681 Acked Read 10 Done Done Sent false
14936 4682 Acked Read 2 Done Done Sent false
14937 4683 Acked Write 9 Done Done Sent false
14938 4684 Acked Read 8 Done Done Sent false
14939 4685 Acked Write 9 Done Done Sent false
14940 4686 Acked Read 4 Done Done Sent false
14941 4687 Acked Read 5 Done Done Sent false
14942 4688 Acked Read 7 Done Done Sent false
14943 4689 Acked Write 3 Done Done Sent false
14944 4690 Acked Read 9 Done Done Sent false
14945 4691 Acked Flush 0 Done Done Sent false
14946 4692 Acked Read 2 Done Done Sent false
14947 4693 Acked Read 9 Done Done Sent false
14948 4694 Acked Flush 0 Done Done Sent false
14949 4695 Acked Write 5 Done Done Sent false
14950 4696 Acked Write 1 Done Done Sent false
14951 4697 Acked Read 6 Done Done Sent false
14952 4698 Acked Read 6 Done Done Sent false
14953 4699 Acked Write 7 Done Done Sent false
14954 4700 Acked Write 9 Done Done Sent false
14955 4701 Acked Read 2 Done Done Sent false
14956 4702 Acked Write 7 Done Done Sent false
14957 4703 Acked Read 10 Done Done Sent false
14958 4704 Acked Flush 0 Done Done Sent false
14959 4705 Acked Read 9 Done Done Sent false
14960 4706 Acked Flush 0 Done Done Sent false
14961 4707 Acked Read 1 Done Done Sent false
14962 4708 Acked Read 10 Done Done Sent false
14963 4709 Acked Read 2 Done Done Sent false
14964 4710 Acked Write 10 Done Done Sent false
14965 4711 Acked Read 10 Done Done Sent false
14966 4712 Acked Read 5 Done Done Sent false
14967 4713 Acked Flush 0 Done Done Sent false
14968 4714 Acked Write 2 Done Done Sent false
14969 4715 Acked Write 6 Done Done Sent false
14970 4716 Acked Read 3 Done Done Sent false
14971 4717 Acked Write 8 Done Done Sent false
14972 4718 Acked Write 3 Done Done Sent false
14973 4719 Acked Read 10 Done Done Sent false
14974 4720 Acked Read 7 Done Done Sent false
14975 4721 Acked Read 4 Done Done Sent false
14976 4722 Acked Read 10 Done Done Sent false
14977 4723 Acked Read 1 Done Done Sent false
14978 4724 Acked Flush 0 Done Done Sent false
14979 4725 Acked Write 8 Done Done Sent false
14980 4726 Acked Read 3 Done Done Sent false
14981 4727 Acked Write 7 Done Done Sent false
14982 4728 Acked Read 1 Done Done Sent false
14983 4729 Acked Write 2 Done Done Sent false
14984 4730 Acked Read 4 Done Done Sent false
14985 4731 Acked Read 8 Done Done Sent false
14986 4732 Acked Flush 0 Done Done Sent false
14987 4733 Acked Write 9 Done Done Sent false
14988 4734 Acked Flush 0 Done Done Sent false
14989 4735 Acked Write 7 Done Done Sent false
14990 4736 Acked Write 3 Done Done Sent false
14991 4737 Acked Write 3 Done Done Sent false
14992 4738 Acked Write 10 Done Done Sent false
14993 4739 Acked Write 2 Done Done Sent false
14994 4740 Acked Write 3 Done Done Sent false
14995 4741 Acked Write 6 Done Done Sent false
14996 4742 Acked Write 3 Done Done Sent false
14997 4743 Acked Write 6 Done Done Sent false
14998 4744 Acked Read 2 Done Done Sent false
14999 4745 Acked Write 3 Done Done Sent false
15000 4746 Acked Write 6 Done Done Sent false
15001 4747 Acked Read 5 Done Done Sent false
15002 4748 Acked Read 2 Done Done Sent false
15003 4749 Acked Write 7 Done Done Sent false
15004 4750 Acked Read 7 Done Done Sent false
15005 4751 Acked Read 10 Done Done Sent false
15006 4752 Acked Read 8 Done Done Sent false
15007 4753 Acked Read 8 Done Done Sent false
15008 4754 Acked Read 7 Done Done Sent false
15009 4755 Acked Write 2 Done Done Sent false
15010 4756 Acked Read 5 Done Done Sent false
15011 4757 Acked Read 1 Done Done Sent false
15012 4758 Acked Read 9 Done Done Sent false
15013 4759 Acked Write 8 Done Done Sent false
15014 4760 Acked Write 10 Done Done Sent false
15015 4761 Acked Read 9 Done Done Sent false
15016 4762 Acked Read 10 Done Done Sent false
15017 4763 Acked Flush 0 Done Done Sent false
15018 4764 Acked Read 2 Done Done Sent false
15019 4765 Acked Write 5 Done Done Sent false
15020 4766 Acked Read 6 Done Done Sent false
15021 4767 Acked Read 5 Done Done Sent false
15022 4768 Acked Read 4 Done Done Sent false
15023 4769 Acked Write 1 Done Done Sent false
15024 4770 Acked Read 6 Done Done Sent false
15025 4771 Acked Write 2 Done Done Sent false
15026 4772 Acked Read 7 Done Done Sent false
15027 4773 Acked Read 10 Done Done Sent false
15028 4774 Acked Read 9 Done Done Sent false
15029 4775 Acked Write 9 Done Done Sent false
15030 4776 Acked Read 8 Done Done Sent false
15031 4777 Acked Write 6 Done Done Sent false
15032 4778 Acked Write 4 Done Done Sent false
15033 4779 Acked Read 8 Done Done Sent false
15034 4780 Acked Flush 0 Done Done Sent false
15035 4781 Acked Write 7 Done Done Sent false
15036 4782 Acked Read 8 Done Done Sent false
15037 4783 Acked Read 9 Done Done Sent false
15038 4784 Acked Write 2 Done Done Sent false
15039 4785 Acked Flush 0 Done Done Sent false
15040 4786 Acked Read 1 Done Done Sent false
15041 4787 Acked Read 10 Done Done Sent false
15042 4788 Acked Read 10 Done Done Sent false
15043 4789 Acked Flush 0 Done Done Sent false
15044 4790 Acked Read 2 Done Done Sent false
15045 4791 Acked Read 6 Done Done Sent false
15046 4792 Acked Read 5 Done Done Sent false
15047 4793 Acked Read 3 Done Done Sent false
15048 4794 Acked Read 7 Done Done Sent false
15049 4795 Acked Read 4 Done Done Sent false
15050 4796 Acked Write 3 Done Done Sent false
15051 4797 Acked Read 2 Done Done Sent false
15052 4798 Acked Read 4 Done Done Sent false
15053 4799 Acked Write 4 Done Done Sent false
15054 4800 Acked Read 5 Done Done Sent false
15055 4801 Acked Write 5 Done Done Sent false
15056 4802 Acked Read 5 Done Done Sent false
15057 4803 Acked Read 7 Done Done Sent false
15058 4804 Acked Read 6 Done Done Sent false
15059 4805 Acked Read 4 Done Done Sent false
15060 4806 Acked Read 5 Done Done Sent false
15061 4807 Acked Write 7 Done Done Sent false
15062 4808 Acked Read 6 Done Done Sent false
15063 4809 Acked Write 6 Done Done Sent false
15064 4810 Acked Read 7 Done Done Sent false
15065 4811 Acked Read 1 Done Done Sent false
15066 4812 Acked Write 9 Done Done Sent false
15067 4813 Acked Write 8 Done Done Sent false
15068 4814 Acked Write 7 Done Done Sent false
15069 4815 Acked Flush 0 Done Done Sent false
15070 4816 Acked Write 1 Done Done Sent false
15071 4817 Acked Read 1 Done Done Sent false
15072 4818 Acked Write 3 Done Done Sent false
15073 4819 Acked Write 1 Done Done Sent false
15074 4820 Acked Read 5 Done Done Sent false
15075 4821 Acked Read 10 Done Done Sent false
15076 4822 Acked Read 5 Done Done Sent false
15077 4823 Acked Write 7 Done Done Sent false
15078 4824 Acked Read 2 Done Done Sent false
15079 4825 Acked Read 2 Done Done Sent false
15080 4826 Acked Flush 0 Done Done Sent false
15081 4827 Acked Read 7 Done Done Sent false
15082 4828 Acked Read 5 Done Done Sent false
15083 4829 Acked Read 8 Done Done Sent false
15084 4830 Acked Write 5 Done Done Sent false
15085 4831 Acked Read 8 Done Done Sent false
15086 4832 Acked Write 2 Done Done Sent false
15087 4833 Acked Read 7 Done Done Sent false
15088 4834 Acked Read 2 Done Done Sent false
15089 4835 Acked Read 4 Done Done Sent false
15090 4836 Acked Read 4 Done Done Sent false
15091 4837 Acked Write 8 Done Done Sent false
15092 4838 Acked Read 10 Done Done Sent false
15093 4839 Acked Read 3 Done Done Sent false
15094 4840 Acked Read 5 Done Done Sent false
15095 4841 Acked Write 6 Done Done Sent false
15096 4842 Acked Read 10 Done Done Sent false
15097 4843 Acked Write 5 Done Done Sent false
15098 4844 Acked Read 7 Done Done Sent false
15099 4845 Acked Read 7 Done Done Sent false
15100 4846 Acked Read 7 Done Done Sent false
15101 4847 Acked Write 2 Done Done Sent false
15102 4848 Acked Write 10 Done Done Sent false
15103 4849 Acked Write 9 Done Done Sent false
15104 4850 Acked Write 2 Done Done Sent false
15105 4851 Acked Flush 0 Done Done Sent false
15106 4852 Acked Write 4 Done Done Sent false
15107 4853 Acked Write 3 Done Done Sent false
15108 4854 Acked Read 1 Done Done Sent false
15109 4855 Acked Read 8 Done Done Sent false
15110 4856 Acked Read 4 Done Done Sent false
15111 4857 Acked Write 3 Done Done Sent false
15112 4858 Acked Read 7 Done Done Sent false
15113 4859 Acked Write 3 Done Done Sent false
15114 4860 Acked Flush 0 Done Done Sent false
15115 4861 Acked Read 3 Done Done Sent false
15116 4862 Acked Flush 0 Done Done Sent false
15117 4863 Acked Read 7 Done Done Sent false
15118 4864 Acked Read 1 Done Done Sent false
15119 4865 Acked Write 3 Done Done Sent false
15120 4866 Acked Read 8 Done Done Sent false
15121 4867 Acked Read 8 Done Done Sent false
15122 4868 Acked Write 2 Done Done Sent false
15123 4869 Acked Read 4 Done Done Sent false
15124 4870 Acked Read 8 Done Done Sent false
15125 4871 Acked Write 2 Done Done Sent false
15126 4872 Acked Write 6 Done Done Sent false
15127 4873 Acked Read 9 Done Done Sent false
15128 4874 Acked Read 4 Done Done Sent false
15129 4875 Acked Write 5 Done Done Sent false
15130 4876 Acked Read 4 Done Done Sent false
15131 4877 Acked Write 8 Done Done Sent false
15132 4878 Acked Read 9 Done Done Sent false
15133 4879 Acked Read 9 Done Done Sent false
15134 4880 Acked Flush 0 Done Done Sent false
15135 4881 Acked Read 5 Done Done Sent false
15136 4882 Acked Write 1 Done Done Sent false
15137 4883 Acked Write 6 Done Done Sent false
15138 4884 Acked Write 4 Done Done Sent false
15139 4885 Acked Read 9 Done Done Sent false
15140 4886 Acked Write 4 Done Done Sent false
15141 4887 Acked Write 8 Done Done Sent false
15142 4888 Acked Write 2 Done Done Sent false
15143 4889 Acked Write 9 Done Done Sent false
15144 4890 Acked Write 9 Done Done Sent false
15145 4891 Acked Read 6 Done Done Sent false
15146 4892 Acked Read 2 Done Done Sent false
15147 4893 Acked Read 6 Done Done Sent false
15148 4894 Acked Write 7 Done Done Sent false
15149 4895 Acked Read 10 Done Done Sent false
15150 4896 Acked Read 5 Done Done Sent false
15151 4897 Acked Write 4 Done Done Sent false
15152 4898 Acked Flush 0 Done Done Sent false
15153 4899 Acked Write 2 Done Done Sent false
15154 4900 Acked Write 4 Done Done Sent false
15155 4901 Acked Read 2 Done Done Sent false
15156 4902 Acked Read 1 Done Done Sent false
15157 4903 Acked Flush 0 Done Done Sent false
15158 4904 Acked Write 6 Done Done Sent false
15159 4905 Acked Write 7 Done Done Sent false
15160 4906 Acked Write 10 Done Done Sent false
15161 4907 Acked Write 7 Done Done Sent false
15162 4908 Acked Read 2 Done Done Sent false
15163 4909 Acked Write 9 Done Done Sent false
15164 4910 Acked Flush 0 Done Done Sent false
15165 4911 Acked Write 6 Done Done Sent false
15166 4912 Acked Read 3 Done Done Sent false
15167 4913 Acked Write 7 Done Done Sent false
15168 4914 Acked Write 3 Done Done Sent false
15169 4915 Acked Write 5 Done Done Sent false
15170 4916 Acked Read 4 Done Done Sent false
15171 4917 Acked Read 1 Done Done Sent false
15172 4918 Acked Read 4 Done Done Sent false
15173 4919 Acked Write 5 Done Done Sent false
15174 4920 Acked Write 9 Done Done Sent false
15175 4921 Acked Read 9 Done Done Sent false
15176 4922 Acked Read 3 Done Done Sent false
15177 4923 Acked Read 2 Done Done Sent false
15178 4924 Acked Read 4 Done Done Sent false
15179 STATES DS:0 DS:1 DS:2 TOTAL
15180 Sent 0 0 301 301
15181 Done 301 301 0 602
15182 Skipped 0 0 0 0
15183 Error 0 0 0 0
15184 Last Flush: Some(JobId(4910)) Some(JobId(4910)) Some(JobId(4623))
15185 Downstairs last five completed: 4623 4622 4621 4620 4619
15186 Upstairs last five completed: 4924 4923 4922 4921 4920
15187 CLIENT: Up:0 ds:301 act:2
151882025-01-29T05:14:03.244ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
151892025-01-29T05:14:04.720ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
151902025-01-29T05:14:05.248ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
151912025-01-29T05:14:06.249ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
15192 [8830] got start action
15193 [8830] I am going to restart
15194 Starting downstairs at port 8830
15195 Make output file at "/tmp/dsc/downstairs-8830.txt"
15196 [8830][2] reports Starting
15197 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
15198 [8830][2] reports Running
15199 ----------------------------------------------------------------
15200 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15201 States: Active Active Offline
15202 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15203 4624 Acked Read 5 Done Done Sent false
15204 4625 Acked Read 2 Done Done Sent false
15205 4626 Acked Read 3 Done Done Sent false
15206 4627 Acked Read 2 Done Done Sent false
15207 4628 Acked Read 6 Done Done Sent false
15208 4629 Acked Write 10 Done Done Sent false
15209 4630 Acked Read 5 Done Done Sent false
15210 4631 Acked Read 9 Done Done Sent false
15211 4632 Acked Read 10 Done Done Sent false
15212 4633 Acked Read 2 Done Done Sent false
15213 4634 Acked Flush 0 Done Done Sent false
15214 4635 Acked Write 5 Done Done Sent false
15215 4636 Acked Read 5 Done Done Sent false
15216 4637 Acked Read 7 Done Done Sent false
15217 4638 Acked Flush 0 Done Done Sent false
15218 4639 Acked Read 7 Done Done Sent false
15219 4640 Acked Read 5 Done Done Sent false
15220 4641 Acked Read 8 Done Done Sent false
15221 4642 Acked Write 5 Done Done Sent false
15222 4643 Acked Write 1 Done Done Sent false
15223 4644 Acked Read 5 Done Done Sent false
15224 4645 Acked Read 3 Done Done Sent false
15225 4646 Acked Write 1 Done Done Sent false
15226 4647 Acked Write 6 Done Done Sent false
15227 4648 Acked Read 9 Done Done Sent false
15228 4649 Acked Write 2 Done Done Sent false
15229 4650 Acked Write 10 Done Done Sent false
15230 4651 Acked Flush 0 Done Done Sent false
15231 4652 Acked Read 4 Done Done Sent false
15232 4653 Acked Read 2 Done Done Sent false
15233 4654 Acked Read 10 Done Done Sent false
15234 4655 Acked Write 5 Done Done Sent false
15235 4656 Acked Read 2 Done Done Sent false
15236 4657 Acked Read 9 Done Done Sent false
15237 4658 Acked Write 1 Done Done Sent false
15238 4659 Acked Write 6 Done Done Sent false
15239 4660 Acked Write 9 Done Done Sent false
15240 4661 Acked Read 5 Done Done Sent false
15241 4662 Acked Read 3 Done Done Sent false
15242 4663 Acked Read 1 Done Done Sent false
15243 4664 Acked Read 5 Done Done Sent false
15244 4665 Acked Flush 0 Done Done Sent false
15245 4666 Acked Write 10 Done Done Sent false
15246 4667 Acked Read 10 Done Done Sent false
15247 4668 Acked Write 5 Done Done Sent false
15248 4669 Acked Write 8 Done Done Sent false
15249 4670 Acked Read 7 Done Done Sent false
15250 4671 Acked Write 5 Done Done Sent false
15251 4672 Acked Flush 0 Done Done Sent false
15252 4673 Acked Read 7 Done Done Sent false
15253 4674 Acked Flush 0 Done Done Sent false
15254 4675 Acked Flush 0 Done Done Sent false
15255 4676 Acked Read 7 Done Done Sent false
15256 4677 Acked Write 6 Done Done Sent false
15257 4678 Acked Write 7 Done Done Sent false
15258 4679 Acked Read 7 Done Done Sent false
15259 4680 Acked Read 4 Done Done Sent false
15260 4681 Acked Read 10 Done Done Sent false
15261 4682 Acked Read 2 Done Done Sent false
15262 4683 Acked Write 9 Done Done Sent false
15263 4684 Acked Read 8 Done Done Sent false
15264 4685 Acked Write 9 Done Done Sent false
15265 4686 Acked Read 4 Done Done Sent false
15266 4687 Acked Read 5 Done Done Sent false
15267 4688 Acked Read 7 Done Done Sent false
15268 4689 Acked Write 3 Done Done Sent false
15269 4690 Acked Read 9 Done Done Sent false
15270 4691 Acked Flush 0 Done Done Sent false
15271 4692 Acked Read 2 Done Done Sent false
15272 4693 Acked Read 9 Done Done Sent false
15273 4694 Acked Flush 0 Done Done Sent false
15274 4695 Acked Write 5 Done Done Sent false
15275 4696 Acked Write 1 Done Done Sent false
15276 4697 Acked Read 6 Done Done Sent false
15277 4698 Acked Read 6 Done Done Sent false
15278 4699 Acked Write 7 Done Done Sent false
15279 4700 Acked Write 9 Done Done Sent false
15280 4701 Acked Read 2 Done Done Sent false
15281 4702 Acked Write 7 Done Done Sent false
15282 4703 Acked Read 10 Done Done Sent false
15283 4704 Acked Flush 0 Done Done Sent false
15284 4705 Acked Read 9 Done Done Sent false
15285 4706 Acked Flush 0 Done Done Sent false
15286 4707 Acked Read 1 Done Done Sent false
15287 4708 Acked Read 10 Done Done Sent false
15288 4709 Acked Read 2 Done Done Sent false
15289 4710 Acked Write 10 Done Done Sent false
15290 4711 Acked Read 10 Done Done Sent false
15291 4712 Acked Read 5 Done Done Sent false
15292 4713 Acked Flush 0 Done Done Sent false
15293 4714 Acked Write 2 Done Done Sent false
15294 4715 Acked Write 6 Done Done Sent false
15295 4716 Acked Read 3 Done Done Sent false
15296 4717 Acked Write 8 Done Done Sent false
15297 4718 Acked Write 3 Done Done Sent false
15298 4719 Acked Read 10 Done Done Sent false
15299 4720 Acked Read 7 Done Done Sent false
15300 4721 Acked Read 4 Done Done Sent false
15301 4722 Acked Read 10 Done Done Sent false
15302 4723 Acked Read 1 Done Done Sent false
15303 4724 Acked Flush 0 Done Done Sent false
15304 4725 Acked Write 8 Done Done Sent false
15305 4726 Acked Read 3 Done Done Sent false
15306 4727 Acked Write 7 Done Done Sent false
15307 4728 Acked Read 1 Done Done Sent false
15308 4729 Acked Write 2 Done Done Sent false
15309 4730 Acked Read 4 Done Done Sent false
15310 4731 Acked Read 8 Done Done Sent false
15311 4732 Acked Flush 0 Done Done Sent false
15312 4733 Acked Write 9 Done Done Sent false
15313 4734 Acked Flush 0 Done Done Sent false
15314 4735 Acked Write 7 Done Done Sent false
15315 4736 Acked Write 3 Done Done Sent false
15316 4737 Acked Write 3 Done Done Sent false
15317 4738 Acked Write 10 Done Done Sent false
15318 4739 Acked Write 2 Done Done Sent false
15319 4740 Acked Write 3 Done Done Sent false
15320 4741 Acked Write 6 Done Done Sent false
15321 4742 Acked Write 3 Done Done Sent false
15322 4743 Acked Write 6 Done Done Sent false
15323 4744 Acked Read 2 Done Done Sent false
15324 4745 Acked Write 3 Done Done Sent false
15325 4746 Acked Write 6 Done Done Sent false
15326 4747 Acked Read 5 Done Done Sent false
15327 4748 Acked Read 2 Done Done Sent false
15328 4749 Acked Write 7 Done Done Sent false
15329 4750 Acked Read 7 Done Done Sent false
15330 4751 Acked Read 10 Done Done Sent false
15331 4752 Acked Read 8 Done Done Sent false
15332 4753 Acked Read 8 Done Done Sent false
15333 4754 Acked Read 7 Done Done Sent false
15334 4755 Acked Write 2 Done Done Sent false
15335 4756 Acked Read 5 Done Done Sent false
15336 4757 Acked Read 1 Done Done Sent false
15337 4758 Acked Read 9 Done Done Sent false
15338 4759 Acked Write 8 Done Done Sent false
15339 4760 Acked Write 10 Done Done Sent false
15340 4761 Acked Read 9 Done Done Sent false
15341 4762 Acked Read 10 Done Done Sent false
15342 4763 Acked Flush 0 Done Done Sent false
15343 4764 Acked Read 2 Done Done Sent false
15344 4765 Acked Write 5 Done Done Sent false
15345 4766 Acked Read 6 Done Done Sent false
15346 4767 Acked Read 5 Done Done Sent false
15347 4768 Acked Read 4 Done Done Sent false
15348 4769 Acked Write 1 Done Done Sent false
15349 4770 Acked Read 6 Done Done Sent false
15350 4771 Acked Write 2 Done Done Sent false
15351 4772 Acked Read 7 Done Done Sent false
15352 4773 Acked Read 10 Done Done Sent false
15353 4774 Acked Read 9 Done Done Sent false
15354 4775 Acked Write 9 Done Done Sent false
15355 4776 Acked Read 8 Done Done Sent false
15356 4777 Acked Write 6 Done Done Sent false
15357 4778 Acked Write 4 Done Done Sent false
15358 4779 Acked Read 8 Done Done Sent false
15359 4780 Acked Flush 0 Done Done Sent false
15360 4781 Acked Write 7 Done Done Sent false
15361 4782 Acked Read 8 Done Done Sent false
15362 4783 Acked Read 9 Done Done Sent false
15363 4784 Acked Write 2 Done Done Sent false
15364 4785 Acked Flush 0 Done Done Sent false
15365 4786 Acked Read 1 Done Done Sent false
15366 4787 Acked Read 10 Done Done Sent false
15367 4788 Acked Read 10 Done Done Sent false
15368 4789 Acked Flush 0 Done Done Sent false
15369 4790 Acked Read 2 Done Done Sent false
15370 4791 Acked Read 6 Done Done Sent false
15371 4792 Acked Read 5 Done Done Sent false
15372 4793 Acked Read 3 Done Done Sent false
15373 4794 Acked Read 7 Done Done Sent false
15374 4795 Acked Read 4 Done Done Sent false
15375 4796 Acked Write 3 Done Done Sent false
15376 4797 Acked Read 2 Done Done Sent false
15377 4798 Acked Read 4 Done Done Sent false
15378 4799 Acked Write 4 Done Done Sent false
15379 4800 Acked Read 5 Done Done Sent false
15380 4801 Acked Write 5 Done Done Sent false
15381 4802 Acked Read 5 Done Done Sent false
15382 4803 Acked Read 7 Done Done Sent false
15383 4804 Acked Read 6 Done Done Sent false
15384 4805 Acked Read 4 Done Done Sent false
15385 4806 Acked Read 5 Done Done Sent false
15386 4807 Acked Write 7 Done Done Sent false
15387 4808 Acked Read 6 Done Done Sent false
15388 4809 Acked Write 6 Done Done Sent false
15389 4810 Acked Read 7 Done Done Sent false
15390 4811 Acked Read 1 Done Done Sent false
15391 4812 Acked Write 9 Done Done Sent false
15392 4813 Acked Write 8 Done Done Sent false
15393 4814 Acked Write 7 Done Done Sent false
15394 4815 Acked Flush 0 Done Done Sent false
15395 4816 Acked Write 1 Done Done Sent false
15396 4817 Acked Read 1 Done Done Sent false
15397 4818 Acked Write 3 Done Done Sent false
15398 4819 Acked Write 1 Done Done Sent false
15399 4820 Acked Read 5 Done Done Sent false
15400 4821 Acked Read 10 Done Done Sent false
15401 4822 Acked Read 5 Done Done Sent false
15402 4823 Acked Write 7 Done Done Sent false
15403 4824 Acked Read 2 Done Done Sent false
15404 4825 Acked Read 2 Done Done Sent false
15405 4826 Acked Flush 0 Done Done Sent false
15406 4827 Acked Read 7 Done Done Sent false
15407 4828 Acked Read 5 Done Done Sent false
15408 4829 Acked Read 8 Done Done Sent false
15409 4830 Acked Write 5 Done Done Sent false
15410 4831 Acked Read 8 Done Done Sent false
15411 4832 Acked Write 2 Done Done Sent false
15412 4833 Acked Read 7 Done Done Sent false
15413 4834 Acked Read 2 Done Done Sent false
15414 4835 Acked Read 4 Done Done Sent false
15415 4836 Acked Read 4 Done Done Sent false
15416 4837 Acked Write 8 Done Done Sent false
15417 4838 Acked Read 10 Done Done Sent false
15418 4839 Acked Read 3 Done Done Sent false
15419 4840 Acked Read 5 Done Done Sent false
15420 4841 Acked Write 6 Done Done Sent false
15421 4842 Acked Read 10 Done Done Sent false
15422 4843 Acked Write 5 Done Done Sent false
15423 4844 Acked Read 7 Done Done Sent false
15424 4845 Acked Read 7 Done Done Sent false
15425 4846 Acked Read 7 Done Done Sent false
15426 4847 Acked Write 2 Done Done Sent false
15427 4848 Acked Write 10 Done Done Sent false
15428 4849 Acked Write 9 Done Done Sent false
15429 4850 Acked Write 2 Done Done Sent false
15430 4851 Acked Flush 0 Done Done Sent false
15431 4852 Acked Write 4 Done Done Sent false
15432 4853 Acked Write 3 Done Done Sent false
15433 4854 Acked Read 1 Done Done Sent false
15434 4855 Acked Read 8 Done Done Sent false
15435 4856 Acked Read 4 Done Done Sent false
15436 4857 Acked Write 3 Done Done Sent false
15437 4858 Acked Read 7 Done Done Sent false
15438 4859 Acked Write 3 Done Done Sent false
15439 4860 Acked Flush 0 Done Done Sent false
15440 4861 Acked Read 3 Done Done Sent false
15441 4862 Acked Flush 0 Done Done Sent false
15442 4863 Acked Read 7 Done Done Sent false
15443 4864 Acked Read 1 Done Done Sent false
15444 4865 Acked Write 3 Done Done Sent false
15445 4866 Acked Read 8 Done Done Sent false
15446 4867 Acked Read 8 Done Done Sent false
15447 4868 Acked Write 2 Done Done Sent false
15448 4869 Acked Read 4 Done Done Sent false
15449 4870 Acked Read 8 Done Done Sent false
15450 4871 Acked Write 2 Done Done Sent false
15451 4872 Acked Write 6 Done Done Sent false
15452 4873 Acked Read 9 Done Done Sent false
15453 4874 Acked Read 4 Done Done Sent false
15454 4875 Acked Write 5 Done Done Sent false
15455 4876 Acked Read 4 Done Done Sent false
15456 4877 Acked Write 8 Done Done Sent false
15457 4878 Acked Read 9 Done Done Sent false
15458 4879 Acked Read 9 Done Done Sent false
15459 4880 Acked Flush 0 Done Done Sent false
15460 4881 Acked Read 5 Done Done Sent false
15461 4882 Acked Write 1 Done Done Sent false
15462 4883 Acked Write 6 Done Done Sent false
15463 4884 Acked Write 4 Done Done Sent false
15464 4885 Acked Read 9 Done Done Sent false
15465 4886 Acked Write 4 Done Done Sent false
15466 4887 Acked Write 8 Done Done Sent false
15467 4888 Acked Write 2 Done Done Sent false
15468 4889 Acked Write 9 Done Done Sent false
15469 4890 Acked Write 9 Done Done Sent false
15470 4891 Acked Read 6 Done Done Sent false
15471 4892 Acked Read 2 Done Done Sent false
15472 4893 Acked Read 6 Done Done Sent false
15473 4894 Acked Write 7 Done Done Sent false
15474 4895 Acked Read 10 Done Done Sent false
15475 4896 Acked Read 5 Done Done Sent false
15476 4897 Acked Write 4 Done Done Sent false
15477 4898 Acked Flush 0 Done Done Sent false
15478 4899 Acked Write 2 Done Done Sent false
15479 4900 Acked Write 4 Done Done Sent false
15480 4901 Acked Read 2 Done Done Sent false
15481 4902 Acked Read 1 Done Done Sent false
15482 4903 Acked Flush 0 Done Done Sent false
15483 4904 Acked Write 6 Done Done Sent false
15484 4905 Acked Write 7 Done Done Sent false
15485 4906 Acked Write 10 Done Done Sent false
15486 4907 Acked Write 7 Done Done Sent false
15487 4908 Acked Read 2 Done Done Sent false
15488 4909 Acked Write 9 Done Done Sent false
15489 4910 Acked Flush 0 Done Done Sent false
15490 4911 Acked Write 6 Done Done Sent false
15491 4912 Acked Read 3 Done Done Sent false
15492 4913 Acked Write 7 Done Done Sent false
15493 4914 Acked Write 3 Done Done Sent false
15494 4915 Acked Write 5 Done Done Sent false
15495 4916 Acked Read 4 Done Done Sent false
15496 4917 Acked Read 1 Done Done Sent false
15497 4918 Acked Read 4 Done Done Sent false
15498 4919 Acked Write 5 Done Done Sent false
15499 4920 Acked Write 9 Done Done Sent false
15500 4921 Acked Read 9 Done Done Sent false
15501 4922 Acked Read 3 Done Done Sent false
15502 4923 Acked Read 2 Done Done Sent false
15503 4924 Acked Read 4 Done Done Sent false
15504 STATES DS:0 DS:1 DS:2 TOTAL
15505 Sent 0 0 301 301
15506 Done 301 301 0 602
15507 Skipped 0 0 0 0
15508 Error 0 0 0 0
15509 Last Flush: Some(JobId(4910)) Some(JobId(4910)) Some(JobId(4623))
15510 Downstairs last five completed: 4623 4622 4621 4620 4619
15511 Upstairs last five completed: 4924 4923 4922 4921 4920
15512 CLIENT: Up:0 ds:301 act:2
155132025-01-29T05:14:07.470ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155142025-01-29T05:14:08.253ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155152025-01-29T05:14:08.470ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155162025-01-29T05:14:08.471ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43303) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155172025-01-29T05:14:08.472ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155182025-01-29T05:14:08.472ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155192025-01-29T05:14:08.472ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155202025-01-29T05:14:08.472ZINFOcrucible: send last flush ID to this DS: Some(JobId(4623)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155212025-01-29T05:14:08.472ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155222025-01-29T05:14:08.472ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4623)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155232025-01-29T05:14:08.473ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4623)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
15524 ----------------------------------------------------------------
15525 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15526 Downstairs last five completed: 4925 4924 4923 4922 4921
15527 Upstairs last five completed: 4925 4924 4923 4922 4921
15528 CLIENT: Up:0 ds:0 act:3
15529 Replay: All jobs finished
15530 Main task has work to do, go find it
15531 got dsc Stop(2)
15532 stop 2
15533 Jan 29 05:14:10.727 INFO [8830] Got stop action so:false kr:false
15534 request completed, latency_us: 365, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f503b53a-e5e6-43e8-a7a6-805fe36db734, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
155352025-01-29T05:14:10.728ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155362025-01-29T05:14:10.728ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155372025-01-29T05:14:10.728ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
155382025-01-29T05:14:10.728ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
15539 Jan 29 05:14:10.728 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/2, method: GET, req_id: ec40c919-e5de-420f-9d17-fde9d7bc3513, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
15540 [8830] Exited with: ExitStatus(unix_wait_status(9))
15541 [8830][2] reports Exit
15542 Jan 29 05:14:14.731 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3819ce2e-e736-4dd6-a7c1-75e02fb3b2f9, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
15543 001/300 Read block 435 len 28672
15544 002/300 Write block 1681 len 16384 data: 5 8 8 8
15545 003/300 Write block 2612 len 8192 data: 4 4
15546 004/300 Read block 3104 len 24576
15547 005/300 Read block 301 len 8192
15548 006/300 Read block 2320 len 24576
15549 007/300 Write block 4905 len 8192 data: 3 2
15550 008/300 Read block 933 len 40960
15551 009/300 Write block 4450 len 32768 data: 4 5 6 7 8 7 6 5
15552 010/300 Flush
15553 011/300 Read block 3693 len 36864
15554 012/300 Write block 219 len 8192 data: 4 4
15555 013/300 Write block 1775 len 24576 data: 7 7 6 6 6 5
15556 014/300 Write block 2089 len 24576 data: 3 3 3 4 4 4
15557 015/300 Flush
15558 016/300 Read block 4692 len 32768
15559 017/300 Read block 2001 len 32768
15560 018/300 Write block 1751 len 4096 data: 4
15561 019/300 Write block 2316 len 4096 data: 5
15562 020/300 Read block 2220 len 20480
15563 021/300 Write block 2650 len 4096 data: 2
15564 022/300 Read block 1445 len 20480
15565 023/300 Read block 3528 len 20480
15566 024/300 Read block 4964 len 16384
15567 025/300 Read block 1609 len 40960
15568 026/300 Read block 442 len 32768
15569 027/300 Write block 3881 len 24576 data: 3 3 3 3 3 3
15570 028/300 Write block 3512 len 24576 data: 3 4 5 4 4 3
15571 029/300 Read block 369 len 12288
15572 030/300 Read block 3750 len 40960
15573 031/300 Write block 3877 len 32768 data: 4 4 3 3 4 4 4 4
15574 032/300 Read block 2861 len 40960
15575 033/300 Write block 4183 len 16384 data: 5 4 3 3
15576 034/300 Read block 4904 len 24576
15577 035/300 Write block 1317 len 8192 data: 5 4
15578 036/300 Read block 109 len 12288
15579 037/300 Write block 857 len 24576 data: 3 2 2 2 4 5
15580 038/300 Read block 1632 len 40960
15581 039/300 Flush
15582 040/300 Read block 2094 len 36864
15583 041/300 Read block 3196 len 16384
15584 042/300 Read block 4063 len 40960
15585 043/300 Write block 2687 len 4096 data: 3
15586 044/300 Write block 1474 len 8192 data: 3 3
15587 045/300 Read block 346 len 36864
15588 046/300 Write block 1496 len 8192 data: 3 4
15589 047/300 Write block 2546 len 32768 data: 3 5 5 5 4 3 3 4
15590 048/300 Read block 4653 len 28672
15591 049/300 Write block 1842 len 4096 data: 5
15592 050/300 Read block 4629 len 36864
15593 051/300 Read block 1428 len 28672
15594 052/300 Read block 3826 len 8192
15595 053/300 Read block 3680 len 16384
15596 054/300 Read block 3902 len 40960
15597 055/300 Flush
15598 056/300 Write block 3084 len 28672 data: 3 3 2 2 2 3 4
15599 057/300 Read block 3089 len 8192
15600 058/300 Read block 1168 len 36864
15601 059/300 Write block 4117 len 8192 data: 4 4
15602 060/300 Write block 504 len 28672 data: 6 6 6 8 6 5 5
15603 061/300 Write block 2274 len 16384 data: 3 3 3 3
15604 062/300 Write block 4564 len 36864 data: 3 3 3 4 4 5 5 4 3
15605 063/300 Read block 3180 len 4096
15606 064/300 Write block 2140 len 16384 data: 7 9 8 6
15607 065/300 Write block 3492 len 40960 data: 2 2 3 3 3 3 4 4 4 4
15608 066/300 Write block 2456 len 40960 data: 6 6 5 3 4 3 4 4 2 3
15609 067/300 Write block 3032 len 32768 data: 2 2 2 2 2 3 5 4
15610 068/300 Write block 335 len 16384 data: 3 3 3 4
15611 069/300 Read block 1464 len 40960
15612 070/300 Write block 3332 len 28672 data: 3 3 3 3 3 3 3
15613 071/300 Read block 854 len 28672
15614 072/300 Read block 3255 len 40960
15615 073/300 Read block 56 len 4096
15616 074/300 Flush
15617 075/300 Read block 4553 len 32768
15618 076/300 Write block 412 len 24576 data: 3 2 3 3 3 3
15619 077/300 Flush
15620 078/300 Write block 2051 len 36864 data: 3 3 3 3 3 2 3 3 3
15621 079/300 Write block 1369 len 32768 data: 5 5 5 4 5 5 4 3
15622 080/300 Read block 1562 len 28672
15623 081/300 Write block 2275 len 16384 data: 4 4 4 3
15624 082/300 Write block 2595 len 40960 data: 2 2 2 2 2 3 4 4 4 4
15625 083/300 Write block 1855 len 32768 data: 2 3 3 3 4 4 4 5
15626 084/300 Read block 122 len 28672
15627 085/300 Write block 1233 len 24576 data: 3 3 3 3 2 2
15628 086/300 Read block 4028 len 16384
15629 087/300 Read block 3805 len 40960
15630 088/300 Flush
15631 089/300 Read block 2893 len 32768
15632 090/300 Write block 4394 len 16384 data: 10 9 8 6
15633 091/300 Write block 4360 len 28672 data: 4 3 4 4 4 4 4
15634 092/300 Write block 715 len 16384 data: 3 3 3 3
15635 093/300 Read block 4171 len 28672
15636 094/300 Write block 930 len 24576 data: 2 3 3 3 2 2
15637 095/300 Read block 2999 len 36864
15638 096/300 Write block 829 len 20480 data: 2 2 2 2 2
15639 097/300 Flush
15640 098/300 Read block 4797 len 32768
15641 099/300 Flush
15642 100/300 Read block 94 len 16384
15643 101/300 Read block 3146 len 8192
15644 102/300 Read block 3929 len 28672
15645 103/300 Read block 3080 len 12288
15646 104/300 Flush
15647 105/300 Read block 1292 len 32768
15648 106/300 Read block 3519 len 16384
15649 107/300 Read block 638 len 4096
15650 108/300 Write block 495 len 16384 data: 6 6 6 6
15651 109/300 Read block 323 len 40960
15652 110/300 Write block 1959 len 24576 data: 2 2 3 3 3 3
15653 111/300 Read block 4901 len 4096
15654 112/300 Write block 230 len 12288 data: 4 4 4
15655 113/300 Read block 748 len 40960
15656 114/300 Read block 2351 len 28672
15657 115/300 Write block 4288 len 16384 data: 4 2 3 3
15658 116/300 Read block 4169 len 8192
15659 117/300 Write block 4708 len 24576 data: 3 4 4 3 4 4
15660 118/300 Read block 2215 len 36864
15661 119/300 Read block 39 len 32768
15662 120/300 Write block 479 len 36864 data: 5 5 7 6 6 5 5 5 5
15663 121/300 Read block 4840 len 16384
15664 122/300 Write block 1936 len 4096 data: 5
15665 123/300 Flush
15666 124/300 Read block 1428 len 20480
15667 125/300 Read block 4979 len 28672
15668 126/300 Read block 4735 len 12288
15669 127/300 Write block 3040 len 12288 data: 4 4 3
15670 128/300 Flush
15671 129/300 Read block 4816 len 8192
15672 130/300 Read block 3336 len 24576
15673 131/300 Read block 1029 len 36864
15674 132/300 Read block 546 len 16384
15675 133/300 Write block 1868 len 24576 data: 6 4 4 4 4 5
15676 134/300 Read block 1715 len 4096
15677 135/300 Write block 3427 len 28672 data: 3 3 4 6 4 4 4
15678 136/300 Flush
15679 137/300 Read block 737 len 40960
15680 138/300 Write block 3301 len 4096 data: 2
15681 139/300 Write block 745 len 36864 data: 10 6 6 5 7 6 5 5 5
15682 140/300 Read block 3676 len 4096
15683 141/300 Read block 1489 len 24576
15684 142/300 Read block 3868 len 16384
15685 143/300 Write block 40 len 16384 data: 4 4 4 4
15686 144/300 Write block 3562 len 28672 data: 3 3 2 2 2 2 3
15687 145/300 Write block 3958 len 12288 data: 3 2 2
15688 146/300 Read block 3068 len 20480
15689 147/300 Write block 2297 len 20480 data: 5 3 3 3 3
15690 148/300 Flush
15691 149/300 Read block 3403 len 8192
15692 150/300 Read block 3255 len 36864
15693 151/300 Write block 4475 len 12288 data: 5 6 5
15694 152/300 Read block 2191 len 28672
15695 153/300 Read block 4058 len 16384
15696 154/300 Write block 2179 len 20480 data: 2 2 2 2 2
15697 155/300 Write block 325 len 8192 data: 3 3
15698 156/300 Read block 2391 len 8192
15699 157/300 Read block 3761 len 20480
15700 158/300 Write block 3865 len 12288 data: 3 3 3
15701 159/300 Read block 2752 len 12288
15702 160/300 Read block 3790 len 20480
15703 161/300 Flush
15704 162/300 Flush
15705 163/300 Read block 2600 len 36864
15706 164/300 Read block 4531 len 12288
15707 165/300 Read block 2263 len 12288
15708 166/300 Write block 2510 len 36864 data: 3 3 3 3 2 2 2 2 2
15709 167/300 Flush
15710 168/300 Read block 922 len 32768
15711 169/300 Read block 1938 len 12288
15712 170/300 Write block 3589 len 32768 data: 3 4 4 3 3 2 2 2
15713 171/300 Read block 3536 len 24576
15714 172/300 Write block 3280 len 20480 data: 3 3 3 4 4
15715 173/300 Read block 3672 len 12288
15716 174/300 Read block 1933 len 16384
15717 175/300 Read block 1135 len 4096
15718 176/300 Read block 1433 len 40960
15719 177/300 Read block 4129 len 12288
15720 178/300 Read block 1182 len 40960
15721 179/300 Read block 219 len 36864
15722 180/300 Write block 1788 len 12288 data: 3 3 4
15723 181/300 Write block 3773 len 32768 data: 4 4 4 5 5 4 4 4
15724 182/300 Read block 1470 len 16384
15725 183/300 Read block 1313 len 24576
15726 184/300 Write block 1028 len 32768 data: 4 5 4 4 4 5 4 4
15727 185/300 Write block 104 len 16384 data: 4 3 3 3
15728 186/300 Read block 862 len 12288
15729 187/300 Flush
15730 188/300 Write block 1968 len 16384 data: 5 6 7 6
15731 189/300 Read block 4092 len 24576
15732 190/300 Write block 4552 len 40960 data: 5 3 4 5 5 3 2 2 2 2
15733 191/300 Read block 1098 len 32768
15734 192/300 Read block 1930 len 40960
15735 193/300 Read block 1960 len 24576
15736 194/300 Write block 3608 len 28672 data: 5 7 6 6 5 5 5
15737 195/300 Write block 4384 len 32768 data: 4 4 5 6 5 4 4 7
15738 196/300 Read block 1080 len 28672
15739 197/300 Read block 4015 len 28672
15740 198/300 Read block 3138 len 8192
15741 199/300 Read block 2272 len 28672
15742 200/300 Read block 240 len 4096
15743 201/300 Read block 4393 len 8192
15744 202/300 Read block 80 len 28672
15745 203/300 Write block 3073 len 32768 data: 2 3 4 4 5 5 5 6
15746 204/300 Read block 2604 len 24576
15747 205/300 Write block 2635 len 20480 data: 4 4 5 6 6
15748 206/300 Write block 1533 len 20480 data: 4 3 2 3 3
15749 207/300 Write block 4528 len 40960 data: 5 5 5 5 5 6 6 5 3 4
15750 208/300 Read block 1259 len 24576
15751 209/300 Write block 2768 len 16384 data: 2 2 2 3
15752 210/300 Write block 2865 len 4096 data: 4
15753 211/300 Write block 4496 len 36864 data: 4 4 6 5 6 4 5 5 5
15754 212/300 Write block 933 len 12288 data: 4 3 3
15755 213/300 Write block 2916 len 16384 data: 6 6 5 5
15756 214/300 Read block 3958 len 16384
15757 215/300 Write block 701 len 40960 data: 3 3 3 3 4 4 4 4 4 3
15758 216/300 Write block 3960 len 4096 data: 3
15759 217/300 Write block 444 len 8192 data: 3 3
15760 218/300 Read block 1069 len 12288
15761 219/300 Write block 195 len 12288 data: 4 5 4
15762 220/300 Write block 2535 len 24576 data: 3 2 2 2 2 2
15763 221/300 Read block 1962 len 24576
15764 222/300 Read block 3806 len 4096
15765 223/300 Write block 4570 len 16384 data: 6 5 4 2
15766 224/300 Write block 2902 len 40960 data: 2 3 3 3 4 3 3 3 3 4
15767 225/300 Flush
15768 226/300 Write block 533 len 40960 data: 3 3 3 3 3 3 4 4 4 5
15769 227/300 Read block 1133 len 28672
15770 228/300 Read block 4504 len 16384
15771 229/300 Read block 2449 len 36864
15772 230/300 Write block 863 len 24576 data: 4 4 4 4 4 5
15773 231/300 Write block 3153 len 32768 data: 3 4 4 4 3 3 2 2
15774 232/300 Read block 2108 len 36864
15775 233/300 Flush
15776 234/300 Read block 324 len 32768
15777 235/300 Flush
15778 236/300 Read block 2209 len 12288
15779 237/300 Write block 1538 len 4096 data: 5
15780 238/300 Flush
15781 239/300 Flush
15782 240/300 Read block 2127 len 20480
15783 241/300 Write block 3039 len 8192 data: 5 5
15784 242/300 Write block 721 len 20480 data: 2 2 2 2 3
15785 243/300 Write block 1357 len 4096 data: 3
15786 244/300 Read block 2709 len 4096
15787 245/300 Read block 2592 len 28672
15788 246/300 Read block 2041 len 4096
15789 247/300 Read block 1749 len 32768
15790 248/300 Write block 4489 len 12288 data: 3 4 4
15791 249/300 Write block 884 len 20480 data: 6 5 4 4 3
15792 250/300 Read block 4315 len 8192
15793 251/300 Read block 451 len 12288
15794 252/300 Read block 1632 len 12288
15795 253/300 Write block 744 len 16384 data: 10 11 7 7
15796 254/300 Write block 2292 len 28672 data: 2 3 3 3 3 6 4
15797 255/300 Read block 3081 len 20480
15798 256/300 Flush
15799 257/300 Read block 2909 len 20480
15800 258/300 Read block 2313 len 8192
15801 259/300 Write block 4659 len 28672 data: 5 4 4 4 5 4 4
15802 260/300 Read block 1330 len 16384
15803 261/300 Write block 3460 len 24576 data: 5 5 5 5 3 3
15804 262/300 Write block 3458 len 24576 data: 3 4 6 6 6 6
15805 263/300 Write block 3934 len 24576 data: 3 3 7 7 7 7
15806 264/300 Read block 1045 len 40960
15807 265/300 Write block 4224 len 40960 data: 5 5 4 5 4 4 3 3 2 2
15808 266/300 Read block 2593 len 32768
15809 267/300 Write block 2717 len 8192 data: 5 5
15810 268/300 Read block 193 len 36864
15811 269/300 Read block 3100 len 36864
15812 270/300 Write block 3549 len 16384 data: 4 4 5 4
15813 271/300 Read block 3067 len 8192
15814 272/300 Flush
15815 273/300 Write block 4289 len 20480 data: 3 4 4 4 5
15816 274/300 Read block 2679 len 4096
15817 275/300 Read block 1831 len 16384
15818 276/300 Write block 1447 len 28672 data: 4 5 5 3 3 2 3
15819 277/300 Write block 4851 len 36864 data: 2 2 2 2 2 3 4 4 5
15820 278/300 Write block 791 len 36864 data: 5 6 6 6 6 7 6 5 5
15821 279/300 Read block 3662 len 32768
15822 280/300 Write block 1503 len 36864 data: 5 4 4 5 5 5 5 5 5
15823 281/300 Read block 2029 len 36864
15824 282/300 Write block 2515 len 36864 data: 3 3 3 3 2 2 3 4 5
15825 283/300 Read block 1106 len 4096
15826 284/300 Read block 2255 len 8192
15827 285/300 Write block 2341 len 36864 data: 5 5 3 3 3 4 4 3 3
15828 286/300 Write block 4269 len 8192 data: 2 2
15829 287/300 Flush
15830 288/300 Write block 2060 len 12288 data: 3 4 4
15831 289/300 Write block 3285 len 28672 data: 4 4 4 4 4 4 3
15832 290/300 Write block 812 len 36864 data: 6 6 6 6 5 5 5 5 4
15833 291/300 Write block 1416 len 16384 data: 7 5 5 3
15834 292/300 Read block 162 len 28672
15835 293/300 Read block 4335 len 16384
15836 294/300 Read block 4308 len 16384
15837 295/300 Write block 3777 len 24576 data: 6 5 5 5 2 2
15838 296/300 Write block 21 len 24576 data: 4 3 4 4 2 2
15839 297/300 Write block 1975 len 40960 data: 3 3 3 3 3 2 2 3 5 5
15840 298/300 Write block 732 len 24576 data: 5 4 4 4 6 6
15841 299/300 Write block 1791 len 12288 data: 3 5 4
15842 300/300 Read block 4190 len 28672
15843 301/300 Read block 1106 len 36864
15844 Main task has work to do, go find it
15845 got dsc Start(2)
15846 start 2
15847 Jan 29 05:14:14.974 INFO request completed, latency_us: 192, response_code: 204, uri: /start/cid/2, method: POST, req_id: b48740c6-7773-47ea-8369-47a8f9455f24, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
15848 Replay: started 2, returned:Ok(())
15849 ----------------------------------------------------------------
15850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15851 States: Active Active Offline
15852 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15853 4926 Acked Read 7 Done Done Sent false
15854 4927 Acked Write 4 Done Done Sent false
15855 4928 Acked Write 2 Done Done Sent false
15856 4929 Acked Read 6 Done Done Sent false
15857 4930 Acked Read 2 Done Done Sent false
15858 4931 Acked Read 6 Done Done Sent false
15859 4932 Acked Write 2 Done Done Sent false
15860 4933 Acked Read 10 Done Done Sent false
15861 4934 Acked Write 8 Done Done Sent false
15862 4935 Acked Flush 0 Done Done Sent false
15863 4936 Acked Read 9 Done Done Sent false
15864 4937 Acked Write 2 Done Done Sent false
15865 4938 Acked Write 6 Done Done Sent false
15866 4939 Acked Write 6 Done Done Sent false
15867 4940 Acked Flush 0 Done Done Sent false
15868 4941 Acked Read 8 Done Done Sent false
15869 4942 Acked Read 8 Done Done Sent false
15870 4943 Acked Write 1 Done Done Sent false
15871 4944 Acked Write 1 Done Done Sent false
15872 4945 Acked Read 5 Done Done Sent false
15873 4946 Acked Write 1 Done Done Sent false
15874 4947 Acked Read 5 Done Done Sent false
15875 4948 Acked Read 5 Done Done Sent false
15876 4949 Acked Read 4 Done Done Sent false
15877 4950 Acked Read 10 Done Done Sent false
15878 4951 Acked Read 8 Done Done Sent false
15879 4952 Acked Write 6 Done Done Sent false
15880 4953 Acked Write 6 Done Done Sent false
15881 4954 Acked Read 3 Done Done Sent false
15882 4955 Acked Read 10 Done Done Sent false
15883 4956 Acked Write 8 Done Done Sent false
15884 4957 Acked Read 10 Done Done Sent false
15885 4958 Acked Write 4 Done Done Sent false
15886 4959 Acked Read 6 Done Done Sent false
15887 4960 Acked Write 2 Done Done Sent false
15888 4961 Acked Read 3 Done Done Sent false
15889 4962 Acked Write 6 Done Done Sent false
15890 4963 Acked Read 10 Done Done Sent false
15891 4964 Acked Flush 0 Done Done Sent false
15892 4965 Acked Read 9 Done Done Sent false
15893 4966 Acked Read 4 Done Done Sent false
15894 4967 Acked Read 10 Done Done Sent false
15895 4968 Acked Write 1 Done Done Sent false
15896 4969 Acked Write 2 Done Done Sent false
15897 4970 Acked Read 9 Done Done Sent false
15898 4971 Acked Write 2 Done Done Sent false
15899 4972 Acked Write 8 Done Done Sent false
15900 4973 Acked Read 7 Done Done Sent false
15901 4974 Acked Write 1 Done Done Sent false
15902 4975 Acked Read 9 Done Done Sent false
15903 4976 Acked Read 7 Done Done Sent false
15904 4977 Acked Read 2 Done Done Sent false
15905 4978 Acked Read 4 Done Done Sent false
15906 4979 Acked Read 10 Done Done Sent false
15907 4980 Acked Flush 0 Done Done Sent false
15908 4981 Acked Write 7 Done Done Sent false
15909 4982 Acked Read 2 Done Done Sent false
15910 4983 Acked Read 9 Done Done Sent false
15911 4984 Acked Write 2 Done Done Sent false
15912 4985 Acked Write 7 Done Done Sent false
15913 4986 Acked Write 4 Done Done Sent false
15914 4987 Acked Write 9 Done Done Sent false
15915 4988 Acked Read 1 Done Done Sent false
15916 4989 Acked Write 4 Done Done Sent false
15917 4990 Acked Write 10 Done Done Sent false
15918 4991 Acked Write 10 Done Done Sent false
15919 4992 Acked Write 8 Done Done Sent false
15920 4993 Acked Write 4 Done Done Sent false
15921 4994 Acked Read 10 Done Done Sent false
15922 4995 Acked Write 7 Done Done Sent false
15923 4996 Acked Read 7 Done Done Sent false
15924 4997 Acked Read 10 Done Done Sent false
15925 4998 Acked Read 1 Done Done Sent false
15926 4999 Acked Flush 0 Done Done Sent false
15927 5000 Acked Read 8 Done Done Sent false
15928 5001 Acked Write 6 Done Done Sent false
15929 5002 Acked Flush 0 Done Done Sent false
15930 5003 Acked Write 9 Done Done Sent false
15931 5004 Acked Write 8 Done Done Sent false
15932 5005 Acked Read 7 Done Done Sent false
15933 5006 Acked Write 4 Done Done Sent false
15934 5007 Acked Write 10 Done Done Sent false
15935 5008 Acked Write 8 Done Done Sent false
15936 5009 Acked Read 7 Done Done Sent false
15937 5010 Acked Write 6 Done Done Sent false
15938 5011 Acked Read 4 Done Done Sent false
15939 5012 Acked Read 10 Done Done Sent false
15940 5013 Acked Flush 0 Done Done Sent false
15941 5014 Acked Read 8 Done Done Sent false
15942 5015 Acked Write 4 Done Done Sent false
15943 5016 Acked Write 7 Done Done Sent false
15944 5017 Acked Write 4 Done Done Sent false
15945 5018 Acked Read 7 Done Done Sent false
15946 5019 Acked Write 6 Done Done Sent false
15947 5020 Acked Read 9 Done Done Sent false
15948 5021 Acked Write 5 Done Done Sent false
15949 5022 Acked Flush 0 Done Done Sent false
15950 5023 Acked Read 8 Done Done Sent false
15951 5024 Acked Flush 0 Done Done Sent false
15952 5025 Acked Read 4 Done Done Sent false
15953 5026 Acked Read 2 Done Done Sent false
15954 5027 Acked Read 7 Done Done Sent false
15955 5028 Acked Read 3 Done Done Sent false
15956 5029 Acked Flush 0 Done Done Sent false
15957 5030 Acked Read 8 Done Done Sent false
15958 5031 Acked Read 4 Done Done Sent false
15959 5032 Acked Read 1 Done Done Sent false
15960 5033 Acked Write 4 Done Done Sent false
15961 5034 Acked Read 10 Done Done Sent false
15962 5035 Acked Write 6 Done Done Sent false
15963 5036 Acked Read 1 Done Done Sent false
15964 5037 Acked Write 3 Done Done Sent false
15965 5038 Acked Read 10 Done Done Sent false
15966 5039 Acked Read 7 Done Done Sent false
15967 5040 Acked Write 4 Done Done Sent false
15968 5041 Acked Read 2 Done Done Sent false
15969 5042 Acked Write 6 Done Done Sent false
15970 5043 Acked Read 9 Done Done Sent false
15971 5044 Acked Read 8 Done Done Sent false
15972 5045 Acked Write 9 Done Done Sent false
15973 5046 Acked Read 4 Done Done Sent false
15974 5047 Acked Write 1 Done Done Sent false
15975 5048 Acked Flush 0 Done Done Sent false
15976 5049 Acked Read 5 Done Done Sent false
15977 5050 Acked Read 7 Done Done Sent false
15978 5051 Acked Read 3 Done Done Sent false
15979 5052 Acked Write 3 Done Done Sent false
15980 5053 Acked Flush 0 Done Done Sent false
15981 5054 Acked Read 2 Done Done Sent false
15982 5055 Acked Read 6 Done Done Sent false
15983 5056 Acked Read 9 Done Done Sent false
15984 5057 Acked Read 4 Done Done Sent false
15985 5058 Acked Write 6 Done Done Sent false
15986 5059 Acked Read 1 Done Done Sent false
15987 5060 Acked Write 7 Done Done Sent false
15988 5061 Acked Flush 0 Done Done Sent false
15989 5062 Acked Read 10 Done Done Sent false
15990 5063 Acked Write 1 Done Done Sent false
15991 5064 Acked Write 9 Done Done Sent false
15992 5065 Acked Read 1 Done Done Sent false
15993 5066 Acked Read 6 Done Done Sent false
15994 5067 Acked Read 4 Done Done Sent false
15995 5068 Acked Write 4 Done Done Sent false
15996 5069 Acked Write 7 Done Done Sent false
15997 5070 Acked Write 3 Done Done Sent false
15998 5071 Acked Read 5 Done Done Sent false
15999 5072 Acked Write 5 Done Done Sent false
16000 5073 Acked Flush 0 Done Done Sent false
16001 5074 Acked Read 2 Done Done Sent false
16002 5075 Acked Read 9 Done Done Sent false
16003 5076 Acked Write 3 Done Done Sent false
16004 5077 Acked Read 7 Done Done Sent false
16005 5078 Acked Read 4 Done Done Sent false
16006 5079 Acked Write 5 Done Done Sent false
16007 5080 Acked Write 2 Done Done Sent false
16008 5081 Acked Read 2 Done Done Sent false
16009 5082 Acked Read 5 Done Done Sent false
16010 5083 Acked Write 3 Done Done Sent false
16011 5084 Acked Read 3 Done Done Sent false
16012 5085 Acked Read 5 Done Done Sent false
16013 5086 Acked Flush 0 Done Done Sent false
16014 5087 Acked Flush 0 Done Done Sent false
16015 5088 Acked Read 9 Done Done Sent false
16016 5089 Acked Read 3 Done Done Sent false
16017 5090 Acked Read 3 Done Done Sent false
16018 5091 Acked Write 9 Done Done Sent false
16019 5092 Acked Flush 0 Done Done Sent false
16020 5093 Acked Read 8 Done Done Sent false
16021 5094 Acked Read 3 Done Done Sent false
16022 5095 Acked Write 8 Done Done Sent false
16023 5096 Acked Read 6 Done Done Sent false
16024 5097 Acked Write 5 Done Done Sent false
16025 5098 Acked Read 3 Done Done Sent false
16026 5099 Acked Read 4 Done Done Sent false
16027 5100 Acked Read 1 Done Done Sent false
16028 5101 Acked Read 10 Done Done Sent false
16029 5102 Acked Read 3 Done Done Sent false
16030 5103 Acked Read 10 Done Done Sent false
16031 5104 Acked Read 9 Done Done Sent false
16032 5105 Acked Write 3 Done Done Sent false
16033 5106 Acked Write 8 Done Done Sent false
16034 5107 Acked Read 4 Done Done Sent false
16035 5108 Acked Read 6 Done Done Sent false
16036 5109 Acked Write 8 Done Done Sent false
16037 5110 Acked Write 4 Done Done Sent false
16038 5111 Acked Read 3 Done Done Sent false
16039 5112 Acked Flush 0 Done Done Sent false
16040 5113 Acked Write 4 Done Done Sent false
16041 5114 Acked Read 6 Done Done Sent false
16042 5115 Acked Write 10 Done Done Sent false
16043 5116 Acked Read 8 Done Done Sent false
16044 5117 Acked Read 10 Done Done Sent false
16045 5118 Acked Read 6 Done Done Sent false
16046 5119 Acked Write 7 Done Done Sent false
16047 5120 Acked Write 8 Done Done Sent false
16048 5121 Acked Read 7 Done Done Sent false
16049 5122 Acked Read 7 Done Done Sent false
16050 5123 Acked Read 2 Done Done Sent false
16051 5124 Acked Read 7 Done Done Sent false
16052 5125 Acked Read 1 Done Done Sent false
16053 5126 Acked Read 2 Done Done Sent false
16054 5127 Acked Read 7 Done Done Sent false
16055 5128 Acked Write 8 Done Done Sent false
16056 5129 Acked Read 6 Done Done Sent false
16057 5130 Acked Write 5 Done Done Sent false
16058 5131 Acked Write 5 Done Done Sent false
16059 5132 Acked Write 10 Done Done Sent false
16060 5133 Acked Read 6 Done Done Sent false
16061 5134 Acked Write 4 Done Done Sent false
16062 5135 Acked Write 1 Done Done Sent false
16063 5136 Acked Write 9 Done Done Sent false
16064 5137 Acked Write 3 Done Done Sent false
16065 5138 Acked Write 4 Done Done Sent false
16066 5139 Acked Read 4 Done Done Sent false
16067 5140 Acked Write 10 Done Done Sent false
16068 5141 Acked Write 1 Done Done Sent false
16069 5142 Acked Write 2 Done Done Sent false
16070 5143 Acked Read 3 Done Done Sent false
16071 5144 Acked Write 3 Done Done Sent false
16072 5145 Acked Write 6 Done Done Sent false
16073 5146 Acked Read 6 Done Done Sent false
16074 5147 Acked Read 1 Done Done Sent false
16075 5148 Acked Write 4 Done Done Sent false
16076 5149 Acked Write 10 Done Done Sent false
16077 5150 Acked Flush 0 Done Done Sent false
16078 5151 Acked Write 10 Done Done Sent false
16079 5152 Acked Read 7 Done Done Sent false
16080 5153 Acked Read 4 Done Done Sent false
16081 5154 Acked Read 9 Done Done Sent false
16082 5155 Acked Write 6 Done Done Sent false
16083 5156 Acked Write 8 Done Done Sent false
16084 5157 Acked Read 9 Done Done Sent false
16085 5158 Acked Flush 0 Done Done Sent false
16086 5159 Acked Read 8 Done Done Sent false
16087 5160 Acked Flush 0 Done Done Sent false
16088 5161 Acked Read 3 Done Done Sent false
16089 5162 Acked Write 1 Done Done Sent false
16090 5163 Acked Flush 0 Done Done Sent false
16091 5164 Acked Flush 0 Done Done Sent false
16092 5165 Acked Read 5 Done Done Sent false
16093 5166 Acked Write 2 Done Done Sent false
16094 5167 Acked Write 5 Done Done Sent false
16095 5168 Acked Write 1 Done Done Sent false
16096 5169 Acked Read 1 Done Done Sent false
16097 5170 Acked Read 7 Done Done Sent false
16098 5171 Acked Read 1 Done Done Sent false
16099 5172 Acked Read 8 Done Done Sent false
16100 5173 Acked Write 3 Done Done Sent false
16101 5174 Acked Write 5 Done Done Sent false
16102 5175 Acked Read 2 Done Done Sent false
16103 5176 Acked Read 3 Done Done Sent false
16104 5177 Acked Read 3 Done Done Sent false
16105 5178 Acked Write 4 Done Done Sent false
16106 5179 Acked Write 7 Done Done Sent false
16107 5180 Acked Read 5 Done Done Sent false
16108 5181 Acked Flush 0 Done Done Sent false
16109 5182 Acked Read 5 Done Done Sent false
16110 5183 Acked Read 2 Done Done Sent false
16111 5184 Acked Write 7 Done Done Sent false
16112 5185 Acked Read 4 Done Done Sent false
16113 5186 Acked Write 6 Done Done Sent false
16114 5187 Acked Write 6 Done Done Sent false
16115 5188 Acked Write 6 Done Done Sent false
16116 5189 Acked Read 10 Done Done Sent false
16117 5190 Acked Write 10 Done Done Sent false
16118 5191 Acked Read 8 Done Done Sent false
16119 5192 Acked Write 2 Done Done Sent false
16120 5193 Acked Read 9 Done Done Sent false
16121 5194 Acked Read 9 Done Done Sent false
16122 5195 Acked Write 4 Done Done Sent false
16123 5196 Acked Read 2 Done Done Sent false
16124 5197 Acked Flush 0 Done Done Sent false
16125 5198 Acked Write 5 Done Done Sent false
16126 5199 Acked Read 1 Done Done Sent false
16127 5200 Acked Read 4 Done Done Sent false
16128 5201 Acked Write 7 Done Done Sent false
16129 5202 Acked Write 9 Done Done Sent false
16130 5203 Acked Write 9 Done Done Sent false
16131 5204 Acked Read 8 Done Done Sent false
16132 5205 Acked Write 9 Done Done Sent false
16133 5206 Acked Read 9 Done Done Sent false
16134 5207 Acked Write 9 Done Done Sent false
16135 5208 Acked Read 1 Done Done Sent false
16136 5209 Acked Read 2 Done Done Sent false
16137 5210 Acked Write 9 Done Done Sent false
16138 5211 Acked Write 2 Done Done Sent false
16139 5212 Acked Flush 0 Done Done Sent false
16140 5213 Acked Write 3 Done Done Sent false
16141 5214 Acked Write 7 Done Done Sent false
16142 5215 Acked Write 9 Done Done Sent false
16143 5216 Acked Write 4 Done Done Sent false
16144 5217 Acked Read 7 Done Done Sent false
16145 5218 Acked Read 4 Done Done Sent false
16146 5219 Acked Read 4 Done Done Sent false
16147 5220 Acked Write 6 Done Done Sent false
16148 5221 Acked Write 6 Done Done Sent false
16149 5222 Acked Write 10 Done Done Sent false
16150 5223 Acked Write 6 Done Done Sent false
16151 5224 Acked Write 3 Done Done Sent false
16152 5225 Acked Read 7 Done Done Sent false
16153 5226 Acked Read 9 Done Done Sent false
16154 STATES DS:0 DS:1 DS:2 TOTAL
16155 Sent 0 0 301 301
16156 Done 301 301 0 602
16157 Skipped 0 0 0 0
16158 Error 0 0 0 0
16159 Last Flush: Some(JobId(5212)) Some(JobId(5212)) Some(JobId(4925))
16160 Downstairs last five completed: 4925 4924 4923 4922 4921
16161 Upstairs last five completed: 5226 5225 5224 5223 5222
16162 CLIENT: Up:0 ds:301 act:2
161632025-01-29T05:14:15.477ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
161642025-01-29T05:14:16.266ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
161652025-01-29T05:14:17.471ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
161662025-01-29T05:14:18.270ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
16167 [8830] got start action
16168 [8830] I am going to restart
16169 Starting downstairs at port 8830
16170 Make output file at "/tmp/dsc/downstairs-8830.txt"
16171 [8830][2] reports Starting
16172 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
16173 [8830][2] reports Running
16174 ----------------------------------------------------------------
16175 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16176 States: Active Active Offline
16177 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16178 4926 Acked Read 7 Done Done Sent false
16179 4927 Acked Write 4 Done Done Sent false
16180 4928 Acked Write 2 Done Done Sent false
16181 4929 Acked Read 6 Done Done Sent false
16182 4930 Acked Read 2 Done Done Sent false
16183 4931 Acked Read 6 Done Done Sent false
16184 4932 Acked Write 2 Done Done Sent false
16185 4933 Acked Read 10 Done Done Sent false
16186 4934 Acked Write 8 Done Done Sent false
16187 4935 Acked Flush 0 Done Done Sent false
16188 4936 Acked Read 9 Done Done Sent false
16189 4937 Acked Write 2 Done Done Sent false
16190 4938 Acked Write 6 Done Done Sent false
16191 4939 Acked Write 6 Done Done Sent false
16192 4940 Acked Flush 0 Done Done Sent false
16193 4941 Acked Read 8 Done Done Sent false
16194 4942 Acked Read 8 Done Done Sent false
16195 4943 Acked Write 1 Done Done Sent false
16196 4944 Acked Write 1 Done Done Sent false
16197 4945 Acked Read 5 Done Done Sent false
16198 4946 Acked Write 1 Done Done Sent false
16199 4947 Acked Read 5 Done Done Sent false
16200 4948 Acked Read 5 Done Done Sent false
16201 4949 Acked Read 4 Done Done Sent false
16202 4950 Acked Read 10 Done Done Sent false
16203 4951 Acked Read 8 Done Done Sent false
16204 4952 Acked Write 6 Done Done Sent false
16205 4953 Acked Write 6 Done Done Sent false
16206 4954 Acked Read 3 Done Done Sent false
16207 4955 Acked Read 10 Done Done Sent false
16208 4956 Acked Write 8 Done Done Sent false
16209 4957 Acked Read 10 Done Done Sent false
16210 4958 Acked Write 4 Done Done Sent false
16211 4959 Acked Read 6 Done Done Sent false
16212 4960 Acked Write 2 Done Done Sent false
16213 4961 Acked Read 3 Done Done Sent false
16214 4962 Acked Write 6 Done Done Sent false
16215 4963 Acked Read 10 Done Done Sent false
16216 4964 Acked Flush 0 Done Done Sent false
16217 4965 Acked Read 9 Done Done Sent false
16218 4966 Acked Read 4 Done Done Sent false
16219 4967 Acked Read 10 Done Done Sent false
16220 4968 Acked Write 1 Done Done Sent false
16221 4969 Acked Write 2 Done Done Sent false
16222 4970 Acked Read 9 Done Done Sent false
16223 4971 Acked Write 2 Done Done Sent false
16224 4972 Acked Write 8 Done Done Sent false
16225 4973 Acked Read 7 Done Done Sent false
16226 4974 Acked Write 1 Done Done Sent false
16227 4975 Acked Read 9 Done Done Sent false
16228 4976 Acked Read 7 Done Done Sent false
16229 4977 Acked Read 2 Done Done Sent false
16230 4978 Acked Read 4 Done Done Sent false
16231 4979 Acked Read 10 Done Done Sent false
16232 4980 Acked Flush 0 Done Done Sent false
16233 4981 Acked Write 7 Done Done Sent false
16234 4982 Acked Read 2 Done Done Sent false
16235 4983 Acked Read 9 Done Done Sent false
16236 4984 Acked Write 2 Done Done Sent false
16237 4985 Acked Write 7 Done Done Sent false
16238 4986 Acked Write 4 Done Done Sent false
16239 4987 Acked Write 9 Done Done Sent false
16240 4988 Acked Read 1 Done Done Sent false
16241 4989 Acked Write 4 Done Done Sent false
16242 4990 Acked Write 10 Done Done Sent false
16243 4991 Acked Write 10 Done Done Sent false
16244 4992 Acked Write 8 Done Done Sent false
16245 4993 Acked Write 4 Done Done Sent false
16246 4994 Acked Read 10 Done Done Sent false
16247 4995 Acked Write 7 Done Done Sent false
16248 4996 Acked Read 7 Done Done Sent false
16249 4997 Acked Read 10 Done Done Sent false
16250 4998 Acked Read 1 Done Done Sent false
16251 4999 Acked Flush 0 Done Done Sent false
16252 5000 Acked Read 8 Done Done Sent false
16253 5001 Acked Write 6 Done Done Sent false
16254 5002 Acked Flush 0 Done Done Sent false
16255 5003 Acked Write 9 Done Done Sent false
16256 5004 Acked Write 8 Done Done Sent false
16257 5005 Acked Read 7 Done Done Sent false
16258 5006 Acked Write 4 Done Done Sent false
16259 5007 Acked Write 10 Done Done Sent false
16260 5008 Acked Write 8 Done Done Sent false
16261 5009 Acked Read 7 Done Done Sent false
16262 5010 Acked Write 6 Done Done Sent false
16263 5011 Acked Read 4 Done Done Sent false
16264 5012 Acked Read 10 Done Done Sent false
16265 5013 Acked Flush 0 Done Done Sent false
16266 5014 Acked Read 8 Done Done Sent false
16267 5015 Acked Write 4 Done Done Sent false
16268 5016 Acked Write 7 Done Done Sent false
16269 5017 Acked Write 4 Done Done Sent false
16270 5018 Acked Read 7 Done Done Sent false
16271 5019 Acked Write 6 Done Done Sent false
16272 5020 Acked Read 9 Done Done Sent false
16273 5021 Acked Write 5 Done Done Sent false
16274 5022 Acked Flush 0 Done Done Sent false
16275 5023 Acked Read 8 Done Done Sent false
16276 5024 Acked Flush 0 Done Done Sent false
16277 5025 Acked Read 4 Done Done Sent false
16278 5026 Acked Read 2 Done Done Sent false
16279 5027 Acked Read 7 Done Done Sent false
16280 5028 Acked Read 3 Done Done Sent false
16281 5029 Acked Flush 0 Done Done Sent false
16282 5030 Acked Read 8 Done Done Sent false
16283 5031 Acked Read 4 Done Done Sent false
16284 5032 Acked Read 1 Done Done Sent false
16285 5033 Acked Write 4 Done Done Sent false
16286 5034 Acked Read 10 Done Done Sent false
16287 5035 Acked Write 6 Done Done Sent false
16288 5036 Acked Read 1 Done Done Sent false
16289 5037 Acked Write 3 Done Done Sent false
16290 5038 Acked Read 10 Done Done Sent false
16291 5039 Acked Read 7 Done Done Sent false
16292 5040 Acked Write 4 Done Done Sent false
16293 5041 Acked Read 2 Done Done Sent false
16294 5042 Acked Write 6 Done Done Sent false
16295 5043 Acked Read 9 Done Done Sent false
16296 5044 Acked Read 8 Done Done Sent false
16297 5045 Acked Write 9 Done Done Sent false
16298 5046 Acked Read 4 Done Done Sent false
16299 5047 Acked Write 1 Done Done Sent false
16300 5048 Acked Flush 0 Done Done Sent false
16301 5049 Acked Read 5 Done Done Sent false
16302 5050 Acked Read 7 Done Done Sent false
16303 5051 Acked Read 3 Done Done Sent false
16304 5052 Acked Write 3 Done Done Sent false
16305 5053 Acked Flush 0 Done Done Sent false
16306 5054 Acked Read 2 Done Done Sent false
16307 5055 Acked Read 6 Done Done Sent false
16308 5056 Acked Read 9 Done Done Sent false
16309 5057 Acked Read 4 Done Done Sent false
16310 5058 Acked Write 6 Done Done Sent false
16311 5059 Acked Read 1 Done Done Sent false
16312 5060 Acked Write 7 Done Done Sent false
16313 5061 Acked Flush 0 Done Done Sent false
16314 5062 Acked Read 10 Done Done Sent false
16315 5063 Acked Write 1 Done Done Sent false
16316 5064 Acked Write 9 Done Done Sent false
16317 5065 Acked Read 1 Done Done Sent false
16318 5066 Acked Read 6 Done Done Sent false
16319 5067 Acked Read 4 Done Done Sent false
16320 5068 Acked Write 4 Done Done Sent false
16321 5069 Acked Write 7 Done Done Sent false
16322 5070 Acked Write 3 Done Done Sent false
16323 5071 Acked Read 5 Done Done Sent false
16324 5072 Acked Write 5 Done Done Sent false
16325 5073 Acked Flush 0 Done Done Sent false
16326 5074 Acked Read 2 Done Done Sent false
16327 5075 Acked Read 9 Done Done Sent false
16328 5076 Acked Write 3 Done Done Sent false
16329 5077 Acked Read 7 Done Done Sent false
16330 5078 Acked Read 4 Done Done Sent false
16331 5079 Acked Write 5 Done Done Sent false
16332 5080 Acked Write 2 Done Done Sent false
16333 5081 Acked Read 2 Done Done Sent false
16334 5082 Acked Read 5 Done Done Sent false
16335 5083 Acked Write 3 Done Done Sent false
16336 5084 Acked Read 3 Done Done Sent false
16337 5085 Acked Read 5 Done Done Sent false
16338 5086 Acked Flush 0 Done Done Sent false
16339 5087 Acked Flush 0 Done Done Sent false
16340 5088 Acked Read 9 Done Done Sent false
16341 5089 Acked Read 3 Done Done Sent false
16342 5090 Acked Read 3 Done Done Sent false
16343 5091 Acked Write 9 Done Done Sent false
16344 5092 Acked Flush 0 Done Done Sent false
16345 5093 Acked Read 8 Done Done Sent false
16346 5094 Acked Read 3 Done Done Sent false
16347 5095 Acked Write 8 Done Done Sent false
16348 5096 Acked Read 6 Done Done Sent false
16349 5097 Acked Write 5 Done Done Sent false
16350 5098 Acked Read 3 Done Done Sent false
16351 5099 Acked Read 4 Done Done Sent false
16352 5100 Acked Read 1 Done Done Sent false
16353 5101 Acked Read 10 Done Done Sent false
16354 5102 Acked Read 3 Done Done Sent false
16355 5103 Acked Read 10 Done Done Sent false
16356 5104 Acked Read 9 Done Done Sent false
16357 5105 Acked Write 3 Done Done Sent false
16358 5106 Acked Write 8 Done Done Sent false
16359 5107 Acked Read 4 Done Done Sent false
16360 5108 Acked Read 6 Done Done Sent false
16361 5109 Acked Write 8 Done Done Sent false
16362 5110 Acked Write 4 Done Done Sent false
16363 5111 Acked Read 3 Done Done Sent false
16364 5112 Acked Flush 0 Done Done Sent false
16365 5113 Acked Write 4 Done Done Sent false
16366 5114 Acked Read 6 Done Done Sent false
16367 5115 Acked Write 10 Done Done Sent false
16368 5116 Acked Read 8 Done Done Sent false
16369 5117 Acked Read 10 Done Done Sent false
16370 5118 Acked Read 6 Done Done Sent false
16371 5119 Acked Write 7 Done Done Sent false
16372 5120 Acked Write 8 Done Done Sent false
16373 5121 Acked Read 7 Done Done Sent false
16374 5122 Acked Read 7 Done Done Sent false
16375 5123 Acked Read 2 Done Done Sent false
16376 5124 Acked Read 7 Done Done Sent false
16377 5125 Acked Read 1 Done Done Sent false
16378 5126 Acked Read 2 Done Done Sent false
16379 5127 Acked Read 7 Done Done Sent false
16380 5128 Acked Write 8 Done Done Sent false
16381 5129 Acked Read 6 Done Done Sent false
16382 5130 Acked Write 5 Done Done Sent false
16383 5131 Acked Write 5 Done Done Sent false
16384 5132 Acked Write 10 Done Done Sent false
16385 5133 Acked Read 6 Done Done Sent false
16386 5134 Acked Write 4 Done Done Sent false
16387 5135 Acked Write 1 Done Done Sent false
16388 5136 Acked Write 9 Done Done Sent false
16389 5137 Acked Write 3 Done Done Sent false
16390 5138 Acked Write 4 Done Done Sent false
16391 5139 Acked Read 4 Done Done Sent false
16392 5140 Acked Write 10 Done Done Sent false
16393 5141 Acked Write 1 Done Done Sent false
16394 5142 Acked Write 2 Done Done Sent false
16395 5143 Acked Read 3 Done Done Sent false
16396 5144 Acked Write 3 Done Done Sent false
16397 5145 Acked Write 6 Done Done Sent false
16398 5146 Acked Read 6 Done Done Sent false
16399 5147 Acked Read 1 Done Done Sent false
16400 5148 Acked Write 4 Done Done Sent false
16401 5149 Acked Write 10 Done Done Sent false
16402 5150 Acked Flush 0 Done Done Sent false
16403 5151 Acked Write 10 Done Done Sent false
16404 5152 Acked Read 7 Done Done Sent false
16405 5153 Acked Read 4 Done Done Sent false
16406 5154 Acked Read 9 Done Done Sent false
16407 5155 Acked Write 6 Done Done Sent false
16408 5156 Acked Write 8 Done Done Sent false
16409 5157 Acked Read 9 Done Done Sent false
16410 5158 Acked Flush 0 Done Done Sent false
16411 5159 Acked Read 8 Done Done Sent false
16412 5160 Acked Flush 0 Done Done Sent false
16413 5161 Acked Read 3 Done Done Sent false
16414 5162 Acked Write 1 Done Done Sent false
16415 5163 Acked Flush 0 Done Done Sent false
16416 5164 Acked Flush 0 Done Done Sent false
16417 5165 Acked Read 5 Done Done Sent false
16418 5166 Acked Write 2 Done Done Sent false
16419 5167 Acked Write 5 Done Done Sent false
16420 5168 Acked Write 1 Done Done Sent false
16421 5169 Acked Read 1 Done Done Sent false
16422 5170 Acked Read 7 Done Done Sent false
16423 5171 Acked Read 1 Done Done Sent false
16424 5172 Acked Read 8 Done Done Sent false
16425 5173 Acked Write 3 Done Done Sent false
16426 5174 Acked Write 5 Done Done Sent false
16427 5175 Acked Read 2 Done Done Sent false
16428 5176 Acked Read 3 Done Done Sent false
16429 5177 Acked Read 3 Done Done Sent false
16430 5178 Acked Write 4 Done Done Sent false
16431 5179 Acked Write 7 Done Done Sent false
16432 5180 Acked Read 5 Done Done Sent false
16433 5181 Acked Flush 0 Done Done Sent false
16434 5182 Acked Read 5 Done Done Sent false
16435 5183 Acked Read 2 Done Done Sent false
16436 5184 Acked Write 7 Done Done Sent false
16437 5185 Acked Read 4 Done Done Sent false
16438 5186 Acked Write 6 Done Done Sent false
16439 5187 Acked Write 6 Done Done Sent false
16440 5188 Acked Write 6 Done Done Sent false
16441 5189 Acked Read 10 Done Done Sent false
16442 5190 Acked Write 10 Done Done Sent false
16443 5191 Acked Read 8 Done Done Sent false
16444 5192 Acked Write 2 Done Done Sent false
16445 5193 Acked Read 9 Done Done Sent false
16446 5194 Acked Read 9 Done Done Sent false
16447 5195 Acked Write 4 Done Done Sent false
16448 5196 Acked Read 2 Done Done Sent false
16449 5197 Acked Flush 0 Done Done Sent false
16450 5198 Acked Write 5 Done Done Sent false
16451 5199 Acked Read 1 Done Done Sent false
16452 5200 Acked Read 4 Done Done Sent false
16453 5201 Acked Write 7 Done Done Sent false
16454 5202 Acked Write 9 Done Done Sent false
16455 5203 Acked Write 9 Done Done Sent false
16456 5204 Acked Read 8 Done Done Sent false
16457 5205 Acked Write 9 Done Done Sent false
16458 5206 Acked Read 9 Done Done Sent false
16459 5207 Acked Write 9 Done Done Sent false
16460 5208 Acked Read 1 Done Done Sent false
16461 5209 Acked Read 2 Done Done Sent false
16462 5210 Acked Write 9 Done Done Sent false
16463 5211 Acked Write 2 Done Done Sent false
16464 5212 Acked Flush 0 Done Done Sent false
16465 5213 Acked Write 3 Done Done Sent false
16466 5214 Acked Write 7 Done Done Sent false
16467 5215 Acked Write 9 Done Done Sent false
16468 5216 Acked Write 4 Done Done Sent false
16469 5217 Acked Read 7 Done Done Sent false
16470 5218 Acked Read 4 Done Done Sent false
16471 5219 Acked Read 4 Done Done Sent false
16472 5220 Acked Write 6 Done Done Sent false
16473 5221 Acked Write 6 Done Done Sent false
16474 5222 Acked Write 10 Done Done Sent false
16475 5223 Acked Write 6 Done Done Sent false
16476 5224 Acked Write 3 Done Done Sent false
16477 5225 Acked Read 7 Done Done Sent false
16478 5226 Acked Read 9 Done Done Sent false
16479 STATES DS:0 DS:1 DS:2 TOTAL
16480 Sent 0 0 301 301
16481 Done 301 301 0 602
16482 Skipped 0 0 0 0
16483 Error 0 0 0 0
16484 Last Flush: Some(JobId(5212)) Some(JobId(5212)) Some(JobId(4925))
16485 Downstairs last five completed: 4925 4924 4923 4922 4921
16486 Upstairs last five completed: 5226 5225 5224 5223 5222
16487 CLIENT: Up:0 ds:301 act:2
164882025-01-29T05:14:19.723ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164892025-01-29T05:14:20.273ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164902025-01-29T05:14:20.729ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164912025-01-29T05:14:20.730ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33882) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164922025-01-29T05:14:20.730ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164932025-01-29T05:14:20.731ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164942025-01-29T05:14:20.731ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164952025-01-29T05:14:20.731ZINFOcrucible: send last flush ID to this DS: Some(JobId(4925)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164962025-01-29T05:14:20.731ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164972025-01-29T05:14:20.731ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4925)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
164982025-01-29T05:14:20.731ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4925)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
16499 ----------------------------------------------------------------
16500 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16501 Downstairs last five completed: 5227 5226 5225 5224 5223
16502 Upstairs last five completed: 5227 5226 5225 5224 5223
16503 CLIENT: Up:0 ds:0 act:3
16504 Replay: All jobs finished
16505 Main task has work to do, go find it
16506 got dsc Stop(1)
16507 stop 1
16508 [8820] Got stop action so:false kr:false
16509 Jan 29 05:14:22.981 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: a601ae6a-0033-4f1b-a72d-70c8f9ccbcea, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
165102025-01-29T05:14:22.981ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
165112025-01-29T05:14:22.981ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
165122025-01-29T05:14:22.981ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
16513 Jan 29 05:14:22.981 INFO request completed, latency_us: 163, response_code: 200, uri: /state/cid/1, method: GET, req_id: b56524a1-bdf3-40c0-8023-5dce30edcc4a, remote_addr: 127.0.0.1:46995, local_addr{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:14:22.981910166Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
16514 : 127.0.0.1:9998
16515 [8820] Exited with: ExitStatus(unix_wait_status(9))
16516 [8820][1] reports Exit
16517 Jan 29 05:14:26.984 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/1, method: GET, req_id: 728ca7c4-378c-4afa-a08e-8c9f1a2bf27d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
16518 001/300 Read block 3408 len 4096
16519 002/300 Read block 4755 len 12288
16520 003/300 Read block 4227 len 12288
16521 004/300 Read block 2417 len 40960
16522 005/300 Read block 1151 len 32768
16523 006/300 Read block 499 len 28672
16524 007/300 Read block 2133 len 12288
16525 008/300 Write block 4050 len 40960 data: 3 2 2 3 2 2 2 2 3 3
16526 009/300 Write block 4616 len 40960 data: 5 5 4 4 3 3 3 3 3 3
16527 010/300 Write block 4637 len 16384 data: 6 7 6 6
16528 011/300 Write block 4481 len 32768 data: 4 3 3 2 3 3 3 3
16529 012/300 Read block 3925 len 16384
16530 013/300 Read block 3324 len 16384
16531 014/300 Write block 2016 len 36864 data: 4 3 3 3 5 3 3 3 3
16532 015/300 Read block 708 len 32768
16533 016/300 Write block 1824 len 36864 data: 3 2 2 3 3 3 3 2 2
16534 017/300 Read block 4628 len 32768
16535 018/300 Flush
16536 019/300 Write block 552 len 28672 data: 5 4 4 3 5 4 5
16537 020/300 Flush
16538 021/300 Read block 1750 len 40960
16539 022/300 Write block 1948 len 16384 data: 4 4 4 5
16540 023/300 Write block 4841 len 16384 data: 5 5 4 5
16541 024/300 Read block 3508 len 32768
16542 025/300 Read block 1063 len 32768
16543 026/300 Write block 3387 len 32768 data: 4 3 4 4 4 3 3 3
16544 027/300 Read block 3594 len 16384
16545 028/300 Read block 3580 len 16384
16546 029/300 Write block 3490 len 16384 data: 3 2 3 3
16547 030/300 Write block 4215 len 20480 data: 2 2 3 3 5
16548 031/300 Read block 2692 len 24576
16549 032/300 Read block 2388 len 4096
16550 033/300 Read block 1992 len 32768
16551 034/300 Read block 3218 len 8192
16552 035/300 Write block 3133 len 12288 data: 4 4 5
16553 036/300 Read block 3855 len 20480
16554 037/300 Read block 1386 len 24576
16555 038/300 Write block 3419 len 8192 data: 5 6
16556 039/300 Read block 72 len 24576
16557 040/300 Read block 237 len 16384
16558 041/300 Read block 1449 len 20480
16559 042/300 Read block 1279 len 12288
16560 043/300 Read block 3452 len 40960
16561 044/300 Write block 4419 len 28672 data: 5 3 3 3 3 3 3
16562 045/300 Read block 2825 len 40960
16563 046/300 Write block 3805 len 32768 data: 4 5 5 5 3 3 3 3
16564 047/300 Write block 4074 len 12288 data: 2 2 2
16565 048/300 Write block 4603 len 8192 data: 8 9
16566 049/300 Write block 3131 len 24576 data: 5 5 5 5 6 6
16567 050/300 Write block 4831 len 12288 data: 4 4 4
16568 051/300 Read block 4671 len 16384
16569 052/300 Write block 90 len 32768 data: 3 3 2 2 2 5 4 4
16570 053/300 Write block 4683 len 28672 data: 5 4 4 5 4 4 3
16571 054/300 Write block 1360 len 40960 data: 4 5 4 4 4 5 3 4 3 6
16572 055/300 Read block 1857 len 4096
16573 056/300 Read block 2849 len 12288
16574 057/300 Write block 1436 len 24576 data: 4 3 3 3 3 3
16575 058/300 Read block 169 len 32768
16576 059/300 Write block 3568 len 4096 data: 4
16577 060/300 Write block 3767 len 12288 data: 8 8 7
16578 061/300 Read block 4880 len 20480
16579 062/300 Read block 1811 len 28672
16580 063/300 Write block 1592 len 36864 data: 5 6 7 5 4 4 4 4 5
16581 064/300 Read block 1458 len 20480
16582 065/300 Read block 4984 len 24576
16583 066/300 Write block 4003 len 28672 data: 4 5 4 5 5 5 5
16584 067/300 Write block 3688 len 16384 data: 2 2 2 3
16585 068/300 Read block 4379 len 20480
16586 069/300 Write block 354 len 16384 data: 2 2 2 2
16587 070/300 Read block 1180 len 28672
16588 071/300 Flush
16589 072/300 Flush
16590 073/300 Write block 4072 len 28672 data: 2 2 3 3 3 2 3
16591 074/300 Flush
16592 075/300 Read block 1270 len 36864
16593 076/300 Write block 3343 len 40960 data: 2 2 3 3 3 4 4 3 5 4
16594 077/300 Write block 4460 len 40960 data: 4 4 4 3 3 3 3 2 2 3
16595 078/300 Write block 152 len 40960 data: 2 2 2 2 2 2 3 4 4 4
16596 079/300 Write block 1338 len 12288 data: 5 6 6
16597 080/300 Read block 1070 len 8192
16598 081/300 Write block 1551 len 16384 data: 4 4 4 5
16599 082/300 Read block 501 len 12288
16600 083/300 Read block 4689 len 24576
16601 084/300 Flush
16602 085/300 Read block 61 len 28672
16603 086/300 Write block 2702 len 8192 data: 4 4
16604 087/300 Write block 3531 len 16384 data: 4 4 4 4
16605 088/300 Read block 4003 len 28672
16606 089/300 Write block 1944 len 36864 data: 4 3 3 3 5 5 5 6 4
16607 090/300 Read block 2949 len 20480
16608 091/300 Write block 4043 len 36864 data: 4 4 4 6 6 4 3 4 3
16609 092/300 Read block 2237 len 20480
16610 093/300 Read block 1536 len 28672
16611 094/300 Read block 56 len 28672
16612 095/300 Write block 2012 len 16384 data: 5 4 5 5
16613 096/300 Write block 2741 len 36864 data: 4 5 4 2 3 4 5 6 5
16614 097/300 Write block 2325 len 4096 data: 3
16615 098/300 Read block 2880 len 28672
16616 099/300 Read block 1886 len 32768
16617 100/300 Read block 1345 len 36864
16618 101/300 Read block 2898 len 40960
16619 102/300 Write block 12 len 8192 data: 3 4
16620 103/300 Read block 1900 len 36864
16621 104/300 Read block 3475 len 40960
16622 105/300 Read block 2291 len 24576
16623 106/300 Read block 1326 len 4096
16624 107/300 Write block 1161 len 8192 data: 3 3
16625 108/300 Write block 175 len 20480 data: 4 3 3 5 4
16626 109/300 Write block 2632 len 12288 data: 3 3 2
16627 110/300 Flush
16628 111/300 Read block 1186 len 4096
16629 112/300 Write block 3229 len 28672 data: 5 6 7 7 7 6 7
16630 113/300 Write block 1293 len 12288 data: 2 2 2
16631 114/300 Read block 386 len 20480
16632 115/300 Read block 3997 len 36864
16633 116/300 Read block 2002 len 12288
16634 117/300 Read block 4988 len 24576
16635 118/300 Read block 1513 len 28672
16636 119/300 Write block 711 len 4096 data: 3
16637 120/300 Read block 909 len 20480
16638 121/300 Read block 727 len 12288
16639 122/300 Read block 3399 len 24576
16640 123/300 Read block 4155 len 20480
16641 124/300 Read block 2866 len 24576
16642 125/300 Read block 970 len 12288
16643 126/300 Read block 640 len 32768
16644 127/300 Write block 2270 len 16384 data: 3 3 3 3
16645 128/300 Flush
16646 129/300 Read block 1876 len 40960
16647 130/300 Read block 2283 len 36864
16648 131/300 Write block 4519 len 28672 data: 4 2 2 2 2 2 3
16649 132/300 Write block 3817 len 12288 data: 3 2 2
16650 133/300 Read block 3046 len 20480
16651 134/300 Write block 4056 len 16384 data: 3 3 4 4
16652 135/300 Read block 4436 len 24576
16653 136/300 Write block 2752 len 24576 data: 7 7 6 7 7 6
16654 137/300 Write block 875 len 24576 data: 3 3 4 4 4 6
16655 138/300 Flush
16656 139/300 Read block 3001 len 32768
16657 140/300 Write block 3682 len 28672 data: 4 3 4 4 3 3 3
16658 141/300 Write block 1105 len 32768 data: 2 2 3 4 4 4 4 4
16659 142/300 Read block 1985 len 16384
16660 143/300 Write block 4873 len 24576 data: 4 5 5 4 3 3
16661 144/300 Read block 2990 len 36864
16662 145/300 Read block 1508 len 8192
16663 146/300 Read block 4775 len 24576
16664 147/300 Read block 1502 len 20480
16665 148/300 Write block 3489 len 36864 data: 3 4 3 4 4 4 4 4 4
16666 149/300 Read block 2004 len 16384
16667 150/300 Write block 3344 len 28672 data: 3 4 4 4 5 5 4
16668 151/300 Write block 3479 len 32768 data: 4 4 5 5 5 4 3 4
16669 152/300 Read block 4638 len 20480
16670 153/300 Flush
16671 154/300 Read block 498 len 40960
16672 155/300 Read block 2600 len 4096
16673 156/300 Write block 2824 len 4096 data: 4
16674 157/300 Read block 728 len 12288
16675 158/300 Read block 3801 len 12288
16676 159/300 Read block 2426 len 16384
16677 160/300 Read block 2125 len 32768
16678 161/300 Write block 2986 len 24576 data: 5 5 4 4 4 4
16679 162/300 Read block 1721 len 24576
16680 163/300 Write block 3792 len 8192 data: 4 4
16681 164/300 Write block 749 len 12288 data: 8 7 6
16682 165/300 Write block 1342 len 4096 data: 5
16683 166/300 Read block 3616 len 8192
16684 167/300 Write block 3229 len 8192 data: 6 7
16685 168/300 Read block 2756 len 20480
16686 169/300 Write block 3445 len 4096 data: 4
16687 170/300 Write block 281 len 24576 data: 4 4 4 4 5 5
16688 171/300 Write block 325 len 36864 data: 4 4 3 3 3 3 4 4 4
16689 172/300 Read block 3375 len 32768
16690 173/300 Write block 1767 len 4096 data: 3
16691 174/300 Write block 4480 len 28672 data: 5 5 4 4 3 4 4
16692 175/300 Flush
16693 176/300 Read block 3262 len 28672
16694 177/300 Write block 35 len 28672 data: 3 3 3 4 4 5 5
16695 178/300 Write block 4946 len 8192 data: 2 3
16696 179/300 Flush
16697 180/300 Read block 3660 len 16384
16698 181/300 Write block 1422 len 40960 data: 4 4 4 4 4 4 4 3 2 2
16699 182/300 Write block 3301 len 12288 data: 3 2 2
16700 183/300 Read block 2508 len 28672
16701 184/300 Flush
16702 185/300 Write block 3382 len 12288 data: 8 7 8
16703 186/300 Write block 864 len 40960 data: 5 5 5 5 6 5 5 5 5 5
16704 187/300 Read block 574 len 12288
16705 188/300 Write block 1543 len 24576 data: 3 2 2 3 3 3
16706 189/300 Flush
16707 190/300 Read block 3440 len 4096
16708 191/300 Read block 4798 len 4096
16709 192/300 Write block 1089 len 24576 data: 2 3 3 4 4 3
16710 193/300 Read block 1653 len 28672
16711 194/300 Write block 3248 len 16384 data: 3 3 3 3
16712 195/300 Write block 4896 len 20480 data: 3 3 4 4 4
16713 196/300 Write block 2586 len 28672 data: 2 2 2 2 2 2 2
16714 197/300 Read block 4314 len 16384
16715 198/300 Write block 4087 len 24576 data: 3 3 3 3 3 3
16716 199/300 Write block 4370 len 36864 data: 4 4 4 4 4 3 3 3 3
16717 200/300 Write block 3490 len 12288 data: 5 4 5
16718 201/300 Write block 2012 len 8192 data: 6 5
16719 202/300 Read block 2726 len 36864
16720 203/300 Write block 3612 len 4096 data: 6
16721 204/300 Flush
16722 205/300 Write block 3198 len 32768 data: 4 4 4 4 4 4 4 3
16723 206/300 Flush
16724 207/300 Write block 1208 len 4096 data: 2
16725 208/300 Write block 854 len 12288 data: 4 4 3
16726 209/300 Write block 2842 len 4096 data: 4
16727 210/300 Write block 3797 len 36864 data: 3 4 2 2 4 4 3 3 5
16728 211/300 Write block 16 len 24576 data: 5 5 4 4 4 5
16729 212/300 Write block 478 len 36864 data: 5 6 6 8 7 7 6 6 6
16730 213/300 Write block 1254 len 8192 data: 3 4
16731 214/300 Read block 3295 len 36864
16732 215/300 Flush
16733 216/300 Write block 3275 len 24576 data: 3 3 3 3 3 4
16734 217/300 Read block 2534 len 32768
16735 218/300 Read block 4458 len 4096
16736 219/300 Read block 1703 len 16384
16737 220/300 Read block 823 len 24576
16738 221/300 Read block 2747 len 16384
16739 222/300 Flush
16740 223/300 Read block 126 len 20480
16741 224/300 Read block 3128 len 40960
16742 225/300 Write block 4754 len 36864 data: 3 3 3 4 4 3 3 4 4
16743 226/300 Read block 129 len 40960
16744 227/300 Read block 1048 len 20480
16745 228/300 Flush
16746 229/300 Read block 3438 len 8192
16747 230/300 Read block 4589 len 4096
16748 231/300 Read block 634 len 4096
16749 232/300 Write block 778 len 32768 data: 7 6 5 5 4 4 3 2
16750 233/300 Read block 383 len 12288
16751 234/300 Write block 870 len 20480 data: 6 6 6 6 4
16752 235/300 Read block 1390 len 8192
16753 236/300 Read block 4966 len 40960
16754 237/300 Read block 3824 len 28672
16755 238/300 Read block 510 len 40960
16756 239/300 Write block 1589 len 16384 data: 4 4 5 6
16757 240/300 Write block 478 len 8192 data: 6 7
16758 241/300 Read block 1929 len 4096
16759 242/300 Read block 3662 len 32768
16760 243/300 Write block 4305 len 20480 data: 3 3 2 3 3
16761 244/300 Read block 4843 len 4096
16762 245/300 Read block 815 len 4096
16763 246/300 Flush
16764 247/300 Write block 3310 len 28672 data: 2 2 2 2 2 2 2
16765 248/300 Read block 1033 len 28672
16766 249/300 Read block 592 len 20480
16767 250/300 Read block 3141 len 40960
16768 251/300 Flush
16769 252/300 Read block 1419 len 8192
16770 253/300 Write block 629 len 12288 data: 2 2 2
16771 254/300 Write block 4838 len 16384 data: 4 4 5 6
16772 255/300 Read block 2461 len 4096
16773 256/300 Read block 2459 len 12288
16774 257/300 Write block 3470 len 4096 data: 5
16775 258/300 Write block 2519 len 32768 data: 3 3 4 5 6 5 5 4
16776 259/300 Write block 724 len 20480 data: 3 4 3 3 3
16777 260/300 Read block 306 len 32768
16778 261/300 Write block 3164 len 4096 data: 2
16779 262/300 Write block 1225 len 40960 data: 4 3 2 2 2 2 2 2 4 4
16780 263/300 Read block 4705 len 32768
16781 264/300 Read block 2152 len 20480
16782 265/300 Read block 2615 len 8192
16783 266/300 Write block 49 len 16384 data: 5 5 6 5
16784 267/300 Read block 681 len 4096
16785 268/300 Read block 4453 len 28672
16786 269/300 Write block 4440 len 4096 data: 3
16787 270/300 Read block 1294 len 40960
16788 271/300 Write block 2731 len 40960 data: 4 4 5 6 6 6 6 6 6 4
16789 272/300 Read block 3916 len 12288
16790 273/300 Read block 2474 len 12288
16791 274/300 Write block 1124 len 20480 data: 5 5 5 5 4
16792 275/300 Read block 3038 len 20480
16793 276/300 Read block 2340 len 36864
16794 277/300 Flush
16795 278/300 Write block 1834 len 36864 data: 2 2 3 2 4 3 4 5 6
16796 279/300 Flush
16797 280/300 Write block 1544 len 36864 data: 3 3 4 4 4 3 3 5 5
16798 281/300 Write block 1718 len 8192 data: 3 3
16799 282/300 Write block 2262 len 28672 data: 5 5 4 3 4 3 3
16800 283/300 Write block 3134 len 32768 data: 6 7 7 6 6 4 5 5
16801 284/300 Write block 1073 len 36864 data: 3 3 2 3 3 3 3 3 3
16802 285/300 Read block 202 len 32768
16803 286/300 Read block 463 len 4096
16804 287/300 Write block 2965 len 32768 data: 5 5 4 3 3 3 4 5
16805 288/300 Write block 1230 len 24576 data: 3 3 3 5 5 4
16806 289/300 Read block 2516 len 4096
16807 290/300 Flush
16808 291/300 Write block 2002 len 12288 data: 6 6 5
16809 292/300 Read block 3802 len 40960
16810 293/300 Read block 1892 len 8192
16811 294/300 Write block 4177 len 40960 data: 3 3 3 3 4 5 6 5 4 4
16812 295/300 Write block 3495 len 28672 data: 5 5 5 5 5 5 5
16813 296/300 Flush
16814 297/300 Read block 2019 len 20480
16815 298/300 Read block 4893 len 8192
16816 299/300 Read block 4274 len 16384
16817 300/300 Read block 2773 len 32768
16818 301/300 Read block 2816 len 20480
16819 Main task has work to do, go find it
16820 got dsc Start(1)
16821 start 1
16822 Jan 29 05:14:27.224 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: d666b2cd-b2b5-465a-8bd0-87e1bfb02803, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
16823 Replay: started 1, returned:Ok(())
16824 ----------------------------------------------------------------
16825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16826 States: Active Offline Active
16827 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16828 5228 Acked Read 1 Done Sent Done false
16829 5229 Acked Read 3 Done Sent Done false
16830 5230 Acked Read 3 Done Sent Done false
16831 5231 Acked Read 10 Done Sent Done false
16832 5232 Acked Read 8 Done Sent Done false
16833 5233 Acked Read 7 Done Sent Done false
16834 5234 Acked Read 3 Done Sent Done false
16835 5235 Acked Write 10 Done Sent Done false
16836 5236 Acked Write 10 Done Sent Done false
16837 5237 Acked Write 4 Done Sent Done false
16838 5238 Acked Write 8 Done Sent Done false
16839 5239 Acked Read 4 Done Sent Done false
16840 5240 Acked Read 4 Done Sent Done false
16841 5241 Acked Write 9 Done Sent Done false
16842 5242 Acked Read 8 Done Sent Done false
16843 5243 Acked Write 9 Done Sent Done false
16844 5244 Acked Read 8 Done Sent Done false
16845 5245 Acked Flush 0 Done Sent Done false
16846 5246 Acked Write 7 Done Sent Done false
16847 5247 Acked Flush 0 Done Sent Done false
16848 5248 Acked Read 10 Done Sent Done false
16849 5249 Acked Write 4 Done Sent Done false
16850 5250 Acked Write 4 Done Sent Done false
16851 5251 Acked Read 8 Done Sent Done false
16852 5252 Acked Read 8 Done Sent Done false
16853 5253 Acked Write 8 Done Sent Done false
16854 5254 Acked Read 4 Done Sent Done false
16855 5255 Acked Read 4 Done Sent Done false
16856 5256 Acked Write 4 Done Sent Done false
16857 5257 Acked Write 5 Done Sent Done false
16858 5258 Acked Read 6 Done Sent Done false
16859 5259 Acked Read 1 Done Sent Done false
16860 5260 Acked Read 8 Done Sent Done false
16861 5261 Acked Read 2 Done Sent Done false
16862 5262 Acked Write 3 Done Sent Done false
16863 5263 Acked Read 5 Done Sent Done false
16864 5264 Acked Read 6 Done Sent Done false
16865 5265 Acked Write 2 Done Sent Done false
16866 5266 Acked Read 6 Done Sent Done false
16867 5267 Acked Read 4 Done Sent Done false
16868 5268 Acked Read 5 Done Sent Done false
16869 5269 Acked Read 3 Done Sent Done false
16870 5270 Acked Read 10 Done Sent Done false
16871 5271 Acked Write 7 Done Sent Done false
16872 5272 Acked Read 10 Done Sent Done false
16873 5273 Acked Write 8 Done Sent Done false
16874 5274 Acked Write 3 Done Sent Done false
16875 5275 Acked Write 2 Done Sent Done false
16876 5276 Acked Write 6 Done Sent Done false
16877 5277 Acked Write 3 Done Sent Done false
16878 5278 Acked Read 4 Done Sent Done false
16879 5279 Acked Write 8 Done Sent Done false
16880 5280 Acked Write 7 Done Sent Done false
16881 5281 Acked Write 10 Done Sent Done false
16882 5282 Acked Read 1 Done Sent Done false
16883 5283 Acked Read 3 Done Sent Done false
16884 5284 Acked Write 6 Done Sent Done false
16885 5285 Acked Read 8 Done Sent Done false
16886 5286 Acked Write 1 Done Sent Done false
16887 5287 Acked Write 3 Done Sent Done false
16888 5288 Acked Read 5 Done Sent Done false
16889 5289 Acked Read 7 Done Sent Done false
16890 5290 Acked Write 9 Done Sent Done false
16891 5291 Acked Read 5 Done Sent Done false
16892 5292 Acked Read 6 Done Sent Done false
16893 5293 Acked Write 7 Done Sent Done false
16894 5294 Acked Write 4 Done Sent Done false
16895 5295 Acked Read 5 Done Sent Done false
16896 5296 Acked Write 4 Done Sent Done false
16897 5297 Acked Read 7 Done Sent Done false
16898 5298 Acked Flush 0 Done Sent Done false
16899 5299 Acked Flush 0 Done Sent Done false
16900 5300 Acked Write 7 Done Sent Done false
16901 5301 Acked Flush 0 Done Sent Done false
16902 5302 Acked Read 9 Done Sent Done false
16903 5303 Acked Write 10 Done Sent Done false
16904 5304 Acked Write 10 Done Sent Done false
16905 5305 Acked Write 10 Done Sent Done false
16906 5306 Acked Write 3 Done Sent Done false
16907 5307 Acked Read 2 Done Sent Done false
16908 5308 Acked Write 4 Done Sent Done false
16909 5309 Acked Read 3 Done Sent Done false
16910 5310 Acked Read 6 Done Sent Done false
16911 5311 Acked Flush 0 Done Sent Done false
16912 5312 Acked Read 7 Done Sent Done false
16913 5313 Acked Write 2 Done Sent Done false
16914 5314 Acked Write 4 Done Sent Done false
16915 5315 Acked Read 7 Done Sent Done false
16916 5316 Acked Write 9 Done Sent Done false
16917 5317 Acked Read 5 Done Sent Done false
16918 5318 Acked Write 9 Done Sent Done false
16919 5319 Acked Read 5 Done Sent Done false
16920 5320 Acked Read 7 Done Sent Done false
16921 5321 Acked Read 7 Done Sent Done false
16922 5322 Acked Write 4 Done Sent Done false
16923 5323 Acked Write 9 Done Sent Done false
16924 5324 Acked Write 1 Done Sent Done false
16925 5325 Acked Read 7 Done Sent Done false
16926 5326 Acked Read 8 Done Sent Done false
16927 5327 Acked Read 9 Done Sent Done false
16928 5328 Acked Read 10 Done Sent Done false
16929 5329 Acked Write 2 Done Sent Done false
16930 5330 Acked Read 9 Done Sent Done false
16931 5331 Acked Read 10 Done Sent Done false
16932 5332 Acked Read 6 Done Sent Done false
16933 5333 Acked Read 1 Done Sent Done false
16934 5334 Acked Write 2 Done Sent Done false
16935 5335 Acked Write 5 Done Sent Done false
16936 5336 Acked Write 3 Done Sent Done false
16937 5337 Acked Flush 0 Done Sent Done false
16938 5338 Acked Read 1 Done Sent Done false
16939 5339 Acked Write 7 Done Sent Done false
16940 5340 Acked Write 3 Done Sent Done false
16941 5341 Acked Read 5 Done Sent Done false
16942 5342 Acked Read 9 Done Sent Done false
16943 5343 Acked Read 3 Done Sent Done false
16944 5344 Acked Read 6 Done Sent Done false
16945 5345 Acked Read 7 Done Sent Done false
16946 5346 Acked Write 1 Done Sent Done false
16947 5347 Acked Read 5 Done Sent Done false
16948 5348 Acked Read 3 Done Sent Done false
16949 5349 Acked Read 6 Done Sent Done false
16950 5350 Acked Read 5 Done Sent Done false
16951 5351 Acked Read 6 Done Sent Done false
16952 5352 Acked Read 3 Done Sent Done false
16953 5353 Acked Read 8 Done Sent Done false
16954 5354 Acked Write 4 Done Sent Done false
16955 5355 Acked Flush 0 Done Sent Done false
16956 5356 Acked Read 10 Done Sent Done false
16957 5357 Acked Read 9 Done Sent Done false
16958 5358 Acked Write 7 Done Sent Done false
16959 5359 Acked Write 3 Done Sent Done false
16960 5360 Acked Read 5 Done Sent Done false
16961 5361 Acked Write 4 Done Sent Done false
16962 5362 Acked Read 6 Done Sent Done false
16963 5363 Acked Write 6 Done Sent Done false
16964 5364 Acked Write 6 Done Sent Done false
16965 5365 Acked Flush 0 Done Sent Done false
16966 5366 Acked Read 8 Done Sent Done false
16967 5367 Acked Write 7 Done Sent Done false
16968 5368 Acked Write 8 Done Sent Done false
16969 5369 Acked Read 4 Done Sent Done false
16970 5370 Acked Write 6 Done Sent Done false
16971 5371 Acked Read 9 Done Sent Done false
16972 5372 Acked Read 2 Done Sent Done false
16973 5373 Acked Read 6 Done Sent Done false
16974 5374 Acked Read 5 Done Sent Done false
16975 5375 Acked Write 9 Done Sent Done false
16976 5376 Acked Read 4 Done Sent Done false
16977 5377 Acked Write 7 Done Sent Done false
16978 5378 Acked Write 8 Done Sent Done false
16979 5379 Acked Read 5 Done Sent Done false
16980 5380 Acked Flush 0 Done Sent Done false
16981 5381 Acked Read 10 Done Sent Done false
16982 5382 Acked Read 1 Done Sent Done false
16983 5383 Acked Write 1 Done Sent Done false
16984 5384 Acked Read 3 Done Sent Done false
16985 5385 Acked Read 3 Done Sent Done false
16986 5386 Acked Read 4 Done Sent Done false
16987 5387 Acked Read 8 Done Sent Done false
16988 5388 Acked Write 6 Done Sent Done false
16989 5389 Acked Read 6 Done Sent Done false
16990 5390 Acked Write 2 Done Sent Done false
16991 5391 Acked Write 3 Done Sent Done false
16992 5392 Acked Write 1 Done Sent Done false
16993 5393 Acked Read 2 Done Sent Done false
16994 5394 Acked Write 2 Done Sent Done false
16995 5395 Acked Read 5 Done Sent Done false
16996 5396 Acked Write 1 Done Sent Done false
16997 5397 Acked Write 6 Done Sent Done false
16998 5398 Acked Write 9 Done Sent Done false
16999 5399 Acked Read 8 Done Sent Done false
17000 5400 Acked Write 1 Done Sent Done false
17001 5401 Acked Write 7 Done Sent Done false
17002 5402 Acked Flush 0 Done Sent Done false
17003 5403 Acked Read 7 Done Sent Done false
17004 5404 Acked Write 7 Done Sent Done false
17005 5405 Acked Write 2 Done Sent Done false
17006 5406 Acked Flush 0 Done Sent Done false
17007 5407 Acked Read 4 Done Sent Done false
17008 5408 Acked Write 10 Done Sent Done false
17009 5409 Acked Write 3 Done Sent Done false
17010 5410 Acked Read 7 Done Sent Done false
17011 5411 Acked Flush 0 Done Sent Done false
17012 5412 Acked Write 3 Done Sent Done false
17013 5413 Acked Write 10 Done Sent Done false
17014 5414 Acked Read 3 Done Sent Done false
17015 5415 Acked Write 6 Done Sent Done false
17016 5416 Acked Flush 0 Done Sent Done false
17017 5417 Acked Read 1 Done Sent Done false
17018 5418 Acked Read 1 Done Sent Done false
17019 5419 Acked Write 6 Done Sent Done false
17020 5420 Acked Read 7 Done Sent Done false
17021 5421 Acked Write 4 Done Sent Done false
17022 5422 Acked Write 5 Done Sent Done false
17023 5423 Acked Write 7 Done Sent Done false
17024 5424 Acked Read 4 Done Sent Done false
17025 5425 Acked Write 6 Done Sent Done false
17026 5426 Acked Write 9 Done Sent Done false
17027 5427 Acked Write 3 Done Sent Done false
17028 5428 Acked Write 2 Done Sent Done false
17029 5429 Acked Read 9 Done Sent Done false
17030 5430 Acked Write 1 Done Sent Done false
17031 5431 Acked Flush 0 Done Sent Done false
17032 5432 Acked Write 8 Done Sent Done false
17033 5433 Acked Flush 0 Done Sent Done false
17034 5434 Acked Write 1 Done Sent Done false
17035 5435 Acked Write 3 Done Sent Done false
17036 5436 Acked Write 1 Done Sent Done false
17037 5437 Acked Write 9 Done Sent Done false
17038 5438 Acked Write 6 Done Sent Done false
17039 5439 Acked Write 9 Done Sent Done false
17040 5440 Acked Write 2 Done Sent Done false
17041 5441 Acked Read 9 Done Sent Done false
17042 5442 Acked Flush 0 Done Sent Done false
17043 5443 Acked Write 6 Done Sent Done false
17044 5444 Acked Read 8 Done Sent Done false
17045 5445 Acked Read 1 Done Sent Done false
17046 5446 Acked Read 4 Done Sent Done false
17047 5447 Acked Read 6 Done Sent Done false
17048 5448 Acked Read 4 Done Sent Done false
17049 5449 Acked Flush 0 Done Sent Done false
17050 5450 Acked Read 5 Done Sent Done false
17051 5451 Acked Read 10 Done Sent Done false
17052 5452 Acked Write 9 Done Sent Done false
17053 5453 Acked Read 10 Done Sent Done false
17054 5454 Acked Read 5 Done Sent Done false
17055 5455 Acked Flush 0 Done Sent Done false
17056 5456 Acked Read 2 Done Sent Done false
17057 5457 Acked Read 1 Done Sent Done false
17058 5458 Acked Read 1 Done Sent Done false
17059 5459 Acked Write 8 Done Sent Done false
17060 5460 Acked Read 3 Done Sent Done false
17061 5461 Acked Write 5 Done Sent Done false
17062 5462 Acked Read 2 Done Sent Done false
17063 5463 Acked Read 10 Done Sent Done false
17064 5464 Acked Read 7 Done Sent Done false
17065 5465 Acked Read 10 Done Sent Done false
17066 5466 Acked Write 4 Done Sent Done false
17067 5467 Acked Write 2 Done Sent Done false
17068 5468 Acked Read 1 Done Sent Done false
17069 5469 Acked Read 8 Done Sent Done false
17070 5470 Acked Write 5 Done Sent Done false
17071 5471 Acked Read 1 Done Sent Done false
17072 5472 Acked Read 1 Done Sent Done false
17073 5473 Acked Flush 0 Done Sent Done false
17074 5474 Acked Write 7 Done Sent Done false
17075 5475 Acked Read 7 Done Sent Done false
17076 5476 Acked Read 5 Done Sent Done false
17077 5477 Acked Read 10 Done Sent Done false
17078 5478 Acked Flush 0 Done Sent Done false
17079 5479 Acked Read 2 Done Sent Done false
17080 5480 Acked Write 3 Done Sent Done false
17081 5481 Acked Write 4 Done Sent Done false
17082 5482 Acked Read 1 Done Sent Done false
17083 5483 Acked Read 3 Done Sent Done false
17084 5484 Acked Write 1 Done Sent Done false
17085 5485 Acked Write 8 Done Sent Done false
17086 5486 Acked Write 5 Done Sent Done false
17087 5487 Acked Read 8 Done Sent Done false
17088 5488 Acked Write 1 Done Sent Done false
17089 5489 Acked Write 10 Done Sent Done false
17090 5490 Acked Read 8 Done Sent Done false
17091 5491 Acked Read 5 Done Sent Done false
17092 5492 Acked Read 2 Done Sent Done false
17093 5493 Acked Write 4 Done Sent Done false
17094 5494 Acked Read 1 Done Sent Done false
17095 5495 Acked Read 7 Done Sent Done false
17096 5496 Acked Write 1 Done Sent Done false
17097 5497 Acked Read 10 Done Sent Done false
17098 5498 Acked Write 10 Done Sent Done false
17099 5499 Acked Read 3 Done Sent Done false
17100 5500 Acked Read 3 Done Sent Done false
17101 5501 Acked Write 5 Done Sent Done false
17102 5502 Acked Read 5 Done Sent Done false
17103 5503 Acked Read 9 Done Sent Done false
17104 5504 Acked Flush 0 Done Sent Done false
17105 5505 Acked Write 9 Done Sent Done false
17106 5506 Acked Flush 0 Done Sent Done false
17107 5507 Acked Write 9 Done Sent Done false
17108 5508 Acked Write 2 Done Sent Done false
17109 5509 Acked Write 7 Done Sent Done false
17110 5510 Acked Write 8 Done Sent Done false
17111 5511 Acked Write 9 Done Sent Done false
17112 5512 Acked Read 8 Done Sent Done false
17113 5513 Acked Read 1 Done Sent Done false
17114 5514 Acked Write 8 Done Sent Done false
17115 5515 Acked Write 6 Done Sent Done false
17116 5516 Acked Read 1 Done Sent Done false
17117 5517 Acked Flush 0 Done Sent Done false
17118 5518 Acked Write 3 Done Sent Done false
17119 5519 Acked Read 10 Done Sent Done false
17120 5520 Acked Read 2 Done Sent Done false
17121 5521 Acked Write 10 Done Sent Done false
17122 5522 Acked Write 7 Done Sent Done false
17123 5523 Acked Flush 0 Done Sent Done false
17124 5524 Acked Read 5 Done Sent Done false
17125 5525 Acked Read 2 Done Sent Done false
17126 5526 Acked Read 4 Done Sent Done false
17127 5527 Acked Read 8 Done Sent Done false
17128 5528 Acked Read 5 Done Sent Done false
17129 STATES DS:0 DS:1 DS:2 TOTAL
17130 Sent 0 301 0 301
17131 Done 301 0 301 602
17132 Skipped 0 0 0 0
17133 Error 0 0 0 0
17134 Last Flush: Some(JobId(5523)) Some(JobId(5227)) Some(JobId(5523))
17135 Downstairs last five completed: 5227 5226 5225 5224 5223
17136 Upstairs last five completed: 5528 5527 5526 5525 5524
17137 CLIENT: Up:0 ds:301 act:2
171382025-01-29T05:14:27.727ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
171392025-01-29T05:14:28.291ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
171402025-01-29T05:14:29.724ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
171412025-01-29T05:14:30.292ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
17142 [8820] got start action
17143 [8820] I am going to restart
17144 Starting downstairs at port 8820
17145 Make output file at "/tmp/dsc/downstairs-8820.txt"
17146 [8820][1] reports Starting
17147 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
17148 [8820][1] reports Running
17149 ----------------------------------------------------------------
17150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17151 States: Active Offline Active
17152 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17153 5228 Acked Read 1 Done Sent Done false
17154 5229 Acked Read 3 Done Sent Done false
17155 5230 Acked Read 3 Done Sent Done false
17156 5231 Acked Read 10 Done Sent Done false
17157 5232 Acked Read 8 Done Sent Done false
17158 5233 Acked Read 7 Done Sent Done false
17159 5234 Acked Read 3 Done Sent Done false
17160 5235 Acked Write 10 Done Sent Done false
17161 5236 Acked Write 10 Done Sent Done false
17162 5237 Acked Write 4 Done Sent Done false
17163 5238 Acked Write 8 Done Sent Done false
17164 5239 Acked Read 4 Done Sent Done false
17165 5240 Acked Read 4 Done Sent Done false
17166 5241 Acked Write 9 Done Sent Done false
17167 5242 Acked Read 8 Done Sent Done false
17168 5243 Acked Write 9 Done Sent Done false
17169 5244 Acked Read 8 Done Sent Done false
17170 5245 Acked Flush 0 Done Sent Done false
17171 5246 Acked Write 7 Done Sent Done false
17172 5247 Acked Flush 0 Done Sent Done false
17173 5248 Acked Read 10 Done Sent Done false
17174 5249 Acked Write 4 Done Sent Done false
17175 5250 Acked Write 4 Done Sent Done false
17176 5251 Acked Read 8 Done Sent Done false
17177 5252 Acked Read 8 Done Sent Done false
17178 5253 Acked Write 8 Done Sent Done false
17179 5254 Acked Read 4 Done Sent Done false
17180 5255 Acked Read 4 Done Sent Done false
17181 5256 Acked Write 4 Done Sent Done false
17182 5257 Acked Write 5 Done Sent Done false
17183 5258 Acked Read 6 Done Sent Done false
17184 5259 Acked Read 1 Done Sent Done false
17185 5260 Acked Read 8 Done Sent Done false
17186 5261 Acked Read 2 Done Sent Done false
17187 5262 Acked Write 3 Done Sent Done false
17188 5263 Acked Read 5 Done Sent Done false
17189 5264 Acked Read 6 Done Sent Done false
17190 5265 Acked Write 2 Done Sent Done false
17191 5266 Acked Read 6 Done Sent Done false
17192 5267 Acked Read 4 Done Sent Done false
17193 5268 Acked Read 5 Done Sent Done false
17194 5269 Acked Read 3 Done Sent Done false
17195 5270 Acked Read 10 Done Sent Done false
17196 5271 Acked Write 7 Done Sent Done false
17197 5272 Acked Read 10 Done Sent Done false
17198 5273 Acked Write 8 Done Sent Done false
17199 5274 Acked Write 3 Done Sent Done false
17200 5275 Acked Write 2 Done Sent Done false
17201 5276 Acked Write 6 Done Sent Done false
17202 5277 Acked Write 3 Done Sent Done false
17203 5278 Acked Read 4 Done Sent Done false
17204 5279 Acked Write 8 Done Sent Done false
17205 5280 Acked Write 7 Done Sent Done false
17206 5281 Acked Write 10 Done Sent Done false
17207 5282 Acked Read 1 Done Sent Done false
17208 5283 Acked Read 3 Done Sent Done false
17209 5284 Acked Write 6 Done Sent Done false
17210 5285 Acked Read 8 Done Sent Done false
17211 5286 Acked Write 1 Done Sent Done false
17212 5287 Acked Write 3 Done Sent Done false
17213 5288 Acked Read 5 Done Sent Done false
17214 5289 Acked Read 7 Done Sent Done false
17215 5290 Acked Write 9 Done Sent Done false
17216 5291 Acked Read 5 Done Sent Done false
17217 5292 Acked Read 6 Done Sent Done false
17218 5293 Acked Write 7 Done Sent Done false
17219 5294 Acked Write 4 Done Sent Done false
17220 5295 Acked Read 5 Done Sent Done false
17221 5296 Acked Write 4 Done Sent Done false
17222 5297 Acked Read 7 Done Sent Done false
17223 5298 Acked Flush 0 Done Sent Done false
17224 5299 Acked Flush 0 Done Sent Done false
17225 5300 Acked Write 7 Done Sent Done false
17226 5301 Acked Flush 0 Done Sent Done false
17227 5302 Acked Read 9 Done Sent Done false
17228 5303 Acked Write 10 Done Sent Done false
17229 5304 Acked Write 10 Done Sent Done false
17230 5305 Acked Write 10 Done Sent Done false
17231 5306 Acked Write 3 Done Sent Done false
17232 5307 Acked Read 2 Done Sent Done false
17233 5308 Acked Write 4 Done Sent Done false
17234 5309 Acked Read 3 Done Sent Done false
17235 5310 Acked Read 6 Done Sent Done false
17236 5311 Acked Flush 0 Done Sent Done false
17237 5312 Acked Read 7 Done Sent Done false
17238 5313 Acked Write 2 Done Sent Done false
17239 5314 Acked Write 4 Done Sent Done false
17240 5315 Acked Read 7 Done Sent Done false
17241 5316 Acked Write 9 Done Sent Done false
17242 5317 Acked Read 5 Done Sent Done false
17243 5318 Acked Write 9 Done Sent Done false
17244 5319 Acked Read 5 Done Sent Done false
17245 5320 Acked Read 7 Done Sent Done false
17246 5321 Acked Read 7 Done Sent Done false
17247 5322 Acked Write 4 Done Sent Done false
17248 5323 Acked Write 9 Done Sent Done false
17249 5324 Acked Write 1 Done Sent Done false
17250 5325 Acked Read 7 Done Sent Done false
17251 5326 Acked Read 8 Done Sent Done false
17252 5327 Acked Read 9 Done Sent Done false
17253 5328 Acked Read 10 Done Sent Done false
17254 5329 Acked Write 2 Done Sent Done false
17255 5330 Acked Read 9 Done Sent Done false
17256 5331 Acked Read 10 Done Sent Done false
17257 5332 Acked Read 6 Done Sent Done false
17258 5333 Acked Read 1 Done Sent Done false
17259 5334 Acked Write 2 Done Sent Done false
17260 5335 Acked Write 5 Done Sent Done false
17261 5336 Acked Write 3 Done Sent Done false
17262 5337 Acked Flush 0 Done Sent Done false
17263 5338 Acked Read 1 Done Sent Done false
17264 5339 Acked Write 7 Done Sent Done false
17265 5340 Acked Write 3 Done Sent Done false
17266 5341 Acked Read 5 Done Sent Done false
17267 5342 Acked Read 9 Done Sent Done false
17268 5343 Acked Read 3 Done Sent Done false
17269 5344 Acked Read 6 Done Sent Done false
17270 5345 Acked Read 7 Done Sent Done false
17271 5346 Acked Write 1 Done Sent Done false
17272 5347 Acked Read 5 Done Sent Done false
17273 5348 Acked Read 3 Done Sent Done false
17274 5349 Acked Read 6 Done Sent Done false
17275 5350 Acked Read 5 Done Sent Done false
17276 5351 Acked Read 6 Done Sent Done false
17277 5352 Acked Read 3 Done Sent Done false
17278 5353 Acked Read 8 Done Sent Done false
17279 5354 Acked Write 4 Done Sent Done false
17280 5355 Acked Flush 0 Done Sent Done false
17281 5356 Acked Read 10 Done Sent Done false
17282 5357 Acked Read 9 Done Sent Done false
17283 5358 Acked Write 7 Done Sent Done false
17284 5359 Acked Write 3 Done Sent Done false
17285 5360 Acked Read 5 Done Sent Done false
17286 5361 Acked Write 4 Done Sent Done false
17287 5362 Acked Read 6 Done Sent Done false
17288 5363 Acked Write 6 Done Sent Done false
17289 5364 Acked Write 6 Done Sent Done false
17290 5365 Acked Flush 0 Done Sent Done false
17291 5366 Acked Read 8 Done Sent Done false
17292 5367 Acked Write 7 Done Sent Done false
17293 5368 Acked Write 8 Done Sent Done false
17294 5369 Acked Read 4 Done Sent Done false
17295 5370 Acked Write 6 Done Sent Done false
17296 5371 Acked Read 9 Done Sent Done false
17297 5372 Acked Read 2 Done Sent Done false
17298 5373 Acked Read 6 Done Sent Done false
17299 5374 Acked Read 5 Done Sent Done false
17300 5375 Acked Write 9 Done Sent Done false
17301 5376 Acked Read 4 Done Sent Done false
17302 5377 Acked Write 7 Done Sent Done false
17303 5378 Acked Write 8 Done Sent Done false
17304 5379 Acked Read 5 Done Sent Done false
17305 5380 Acked Flush 0 Done Sent Done false
17306 5381 Acked Read 10 Done Sent Done false
17307 5382 Acked Read 1 Done Sent Done false
17308 5383 Acked Write 1 Done Sent Done false
17309 5384 Acked Read 3 Done Sent Done false
17310 5385 Acked Read 3 Done Sent Done false
17311 5386 Acked Read 4 Done Sent Done false
17312 5387 Acked Read 8 Done Sent Done false
17313 5388 Acked Write 6 Done Sent Done false
17314 5389 Acked Read 6 Done Sent Done false
17315 5390 Acked Write 2 Done Sent Done false
17316 5391 Acked Write 3 Done Sent Done false
17317 5392 Acked Write 1 Done Sent Done false
17318 5393 Acked Read 2 Done Sent Done false
17319 5394 Acked Write 2 Done Sent Done false
17320 5395 Acked Read 5 Done Sent Done false
17321 5396 Acked Write 1 Done Sent Done false
17322 5397 Acked Write 6 Done Sent Done false
17323 5398 Acked Write 9 Done Sent Done false
17324 5399 Acked Read 8 Done Sent Done false
17325 5400 Acked Write 1 Done Sent Done false
17326 5401 Acked Write 7 Done Sent Done false
17327 5402 Acked Flush 0 Done Sent Done false
17328 5403 Acked Read 7 Done Sent Done false
17329 5404 Acked Write 7 Done Sent Done false
17330 5405 Acked Write 2 Done Sent Done false
17331 5406 Acked Flush 0 Done Sent Done false
17332 5407 Acked Read 4 Done Sent Done false
17333 5408 Acked Write 10 Done Sent Done false
17334 5409 Acked Write 3 Done Sent Done false
17335 5410 Acked Read 7 Done Sent Done false
17336 5411 Acked Flush 0 Done Sent Done false
17337 5412 Acked Write 3 Done Sent Done false
17338 5413 Acked Write 10 Done Sent Done false
17339 5414 Acked Read 3 Done Sent Done false
17340 5415 Acked Write 6 Done Sent Done false
17341 5416 Acked Flush 0 Done Sent Done false
17342 5417 Acked Read 1 Done Sent Done false
17343 5418 Acked Read 1 Done Sent Done false
17344 5419 Acked Write 6 Done Sent Done false
17345 5420 Acked Read 7 Done Sent Done false
17346 5421 Acked Write 4 Done Sent Done false
17347 5422 Acked Write 5 Done Sent Done false
17348 5423 Acked Write 7 Done Sent Done false
17349 5424 Acked Read 4 Done Sent Done false
17350 5425 Acked Write 6 Done Sent Done false
17351 5426 Acked Write 9 Done Sent Done false
17352 5427 Acked Write 3 Done Sent Done false
17353 5428 Acked Write 2 Done Sent Done false
17354 5429 Acked Read 9 Done Sent Done false
17355 5430 Acked Write 1 Done Sent Done false
17356 5431 Acked Flush 0 Done Sent Done false
17357 5432 Acked Write 8 Done Sent Done false
17358 5433 Acked Flush 0 Done Sent Done false
17359 5434 Acked Write 1 Done Sent Done false
17360 5435 Acked Write 3 Done Sent Done false
17361 5436 Acked Write 1 Done Sent Done false
17362 5437 Acked Write 9 Done Sent Done false
17363 5438 Acked Write 6 Done Sent Done false
17364 5439 Acked Write 9 Done Sent Done false
17365 5440 Acked Write 2 Done Sent Done false
17366 5441 Acked Read 9 Done Sent Done false
17367 5442 Acked Flush 0 Done Sent Done false
17368 5443 Acked Write 6 Done Sent Done false
17369 5444 Acked Read 8 Done Sent Done false
17370 5445 Acked Read 1 Done Sent Done false
17371 5446 Acked Read 4 Done Sent Done false
17372 5447 Acked Read 6 Done Sent Done false
17373 5448 Acked Read 4 Done Sent Done false
17374 5449 Acked Flush 0 Done Sent Done false
17375 5450 Acked Read 5 Done Sent Done false
17376 5451 Acked Read 10 Done Sent Done false
17377 5452 Acked Write 9 Done Sent Done false
17378 5453 Acked Read 10 Done Sent Done false
17379 5454 Acked Read 5 Done Sent Done false
17380 5455 Acked Flush 0 Done Sent Done false
17381 5456 Acked Read 2 Done Sent Done false
17382 5457 Acked Read 1 Done Sent Done false
17383 5458 Acked Read 1 Done Sent Done false
17384 5459 Acked Write 8 Done Sent Done false
17385 5460 Acked Read 3 Done Sent Done false
17386 5461 Acked Write 5 Done Sent Done false
17387 5462 Acked Read 2 Done Sent Done false
17388 5463 Acked Read 10 Done Sent Done false
17389 5464 Acked Read 7 Done Sent Done false
17390 5465 Acked Read 10 Done Sent Done false
17391 5466 Acked Write 4 Done Sent Done false
17392 5467 Acked Write 2 Done Sent Done false
17393 5468 Acked Read 1 Done Sent Done false
17394 5469 Acked Read 8 Done Sent Done false
17395 5470 Acked Write 5 Done Sent Done false
17396 5471 Acked Read 1 Done Sent Done false
17397 5472 Acked Read 1 Done Sent Done false
17398 5473 Acked Flush 0 Done Sent Done false
17399 5474 Acked Write 7 Done Sent Done false
17400 5475 Acked Read 7 Done Sent Done false
17401 5476 Acked Read 5 Done Sent Done false
17402 5477 Acked Read 10 Done Sent Done false
17403 5478 Acked Flush 0 Done Sent Done false
17404 5479 Acked Read 2 Done Sent Done false
17405 5480 Acked Write 3 Done Sent Done false
17406 5481 Acked Write 4 Done Sent Done false
17407 5482 Acked Read 1 Done Sent Done false
17408 5483 Acked Read 3 Done Sent Done false
17409 5484 Acked Write 1 Done Sent Done false
17410 5485 Acked Write 8 Done Sent Done false
17411 5486 Acked Write 5 Done Sent Done false
17412 5487 Acked Read 8 Done Sent Done false
17413 5488 Acked Write 1 Done Sent Done false
17414 5489 Acked Write 10 Done Sent Done false
17415 5490 Acked Read 8 Done Sent Done false
17416 5491 Acked Read 5 Done Sent Done false
17417 5492 Acked Read 2 Done Sent Done false
17418 5493 Acked Write 4 Done Sent Done false
17419 5494 Acked Read 1 Done Sent Done false
17420 5495 Acked Read 7 Done Sent Done false
17421 5496 Acked Write 1 Done Sent Done false
17422 5497 Acked Read 10 Done Sent Done false
17423 5498 Acked Write 10 Done Sent Done false
17424 5499 Acked Read 3 Done Sent Done false
17425 5500 Acked Read 3 Done Sent Done false
17426 5501 Acked Write 5 Done Sent Done false
17427 5502 Acked Read 5 Done Sent Done false
17428 5503 Acked Read 9 Done Sent Done false
17429 5504 Acked Flush 0 Done Sent Done false
17430 5505 Acked Write 9 Done Sent Done false
17431 5506 Acked Flush 0 Done Sent Done false
17432 5507 Acked Write 9 Done Sent Done false
17433 5508 Acked Write 2 Done Sent Done false
17434 5509 Acked Write 7 Done Sent Done false
17435 5510 Acked Write 8 Done Sent Done false
17436 5511 Acked Write 9 Done Sent Done false
17437 5512 Acked Read 8 Done Sent Done false
17438 5513 Acked Read 1 Done Sent Done false
17439 5514 Acked Write 8 Done Sent Done false
17440 5515 Acked Write 6 Done Sent Done false
17441 5516 Acked Read 1 Done Sent Done false
17442 5517 Acked Flush 0 Done Sent Done false
17443 5518 Acked Write 3 Done Sent Done false
17444 5519 Acked Read 10 Done Sent Done false
17445 5520 Acked Read 2 Done Sent Done false
17446 5521 Acked Write 10 Done Sent Done false
17447 5522 Acked Write 7 Done Sent Done false
17448 5523 Acked Flush 0 Done Sent Done false
17449 5524 Acked Read 5 Done Sent Done false
17450 5525 Acked Read 2 Done Sent Done false
17451 5526 Acked Read 4 Done Sent Done false
17452 5527 Acked Read 8 Done Sent Done false
17453 5528 Acked Read 5 Done Sent Done false
17454 STATES DS:0 DS:1 DS:2 TOTAL
17455 Sent 0 301 0 301
17456 Done 301 0 301 602
17457 Skipped 0 0 0 0
17458 Error 0 0 0 0
17459 Last Flush: Some(JobId(5523)) Some(JobId(5227)) Some(JobId(5523))
17460 Downstairs last five completed: 5227 5226 5225 5224 5223
17461 Upstairs last five completed: 5528 5527 5526 5525 5524
17462 CLIENT: Up:0 ds:301 act:2
174632025-01-29T05:14:31.730ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174642025-01-29T05:14:32.296ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174652025-01-29T05:14:32.796ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174662025-01-29T05:14:32.983ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174672025-01-29T05:14:32.983ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46873) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174682025-01-29T05:14:32.984ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174692025-01-29T05:14:32.985ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174702025-01-29T05:14:32.985ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174712025-01-29T05:14:32.985ZINFOcrucible: send last flush ID to this DS: Some(JobId(5227)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174722025-01-29T05:14:32.985ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174732025-01-29T05:14:32.985ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5227)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174742025-01-29T05:14:32.985ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5227)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
17475 ----------------------------------------------------------------
17476 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17477 Downstairs last five completed: 5529 5528 5527 5526 5525
17478 Upstairs last five completed: 5529 5528 5527 5526 5525
17479 CLIENT: Up:0 ds:0 act:3
17480 Replay: All jobs finished
17481 Main task has work to do, go find it
17482 got dsc Stop(0)
17483 stop 0
17484 [8810] Got stop action so:false kr:false
17485 Jan 29 05:14:35.231 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 20e598b1-fae2-4042-a769-5ee396fa65c1, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
174862025-01-29T05:14:35.231ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174872025-01-29T05:14:35.231ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174882025-01-29T05:14:35.231ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
174892025-01-29T05:14:35.231ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
17490 Jan 29 05:14:35.232 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4931cd11-45e0-4a91-822f-3205733052bd, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
17491 [8810] Exited with: ExitStatus(unix_wait_status(9))
17492 [8810][0] reports Exit
17493 Jan 29 05:14:39.234 INFO request completed, latency_us: 290, response_code: 200, uri: /state/cid/0, method: GET, req_id: b00bc5ee-e303-4e07-bf6a-ac47a3bc910f, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
17494 001/300 Write block 1584 len 8192 data: 4 4
17495 002/300 Write block 1169 len 36864 data: 5 5 6 7 6 5 5 5 3
17496 003/300 Write block 4769 len 36864 data: 4 6 6 6 6 6 6 6 6
17497 004/300 Read block 2545 len 40960
17498 005/300 Write block 2277 len 32768 data: 5 4 3 3 3 3 3 3
17499 006/300 Write block 2702 len 8192 data: 5 5
17500 007/300 Write block 3849 len 8192 data: 5 4
17501 008/300 Read block 612 len 4096
17502 009/300 Write block 2572 len 36864 data: 3 5 5 5 6 5 5 4 4
17503 010/300 Read block 4337 len 16384
17504 011/300 Read block 1698 len 4096
17505 012/300 Read block 750 len 12288
17506 013/300 Read block 4734 len 20480
17507 014/300 Read block 1097 len 40960
17508 015/300 Read block 4564 len 36864
17509 016/300 Read block 746 len 4096
17510 017/300 Write block 3070 len 32768 data: 2 2 2 3 4 5 5 6
17511 018/300 Read block 4606 len 40960
17512 019/300 Read block 3715 len 24576
17513 020/300 Read block 3052 len 20480
17514 021/300 Read block 1971 len 40960
17515 022/300 Read block 4641 len 40960
17516 023/300 Read block 2498 len 16384
17517 024/300 Write block 2716 len 20480 data: 4 6 6 4 3
17518 025/300 Flush
17519 026/300 Read block 3331 len 16384
17520 027/300 Write block 4234 len 12288 data: 3 3 3
17521 028/300 Read block 2375 len 36864
17522 029/300 Write block 2909 len 20480 data: 4 4 5 5 6
17523 030/300 Flush
17524 031/300 Flush
17525 032/300 Write block 2826 len 28672 data: 4 4 3 4 5 5 6
17526 033/300 Read block 1093 len 16384
17527 034/300 Flush
17528 035/300 Read block 2897 len 8192
17529 036/300 Write block 4140 len 4096 data: 6
17530 037/300 Read block 4468 len 20480
17531 038/300 Write block 4281 len 8192 data: 5 5
17532 039/300 Write block 251 len 8192 data: 4 2
17533 040/300 Write block 3126 len 16384 data: 4 4 4 5
17534 041/300 Read block 2202 len 16384
17535 042/300 Read block 49 len 28672
17536 043/300 Write block 4963 len 36864 data: 3 4 3 3 4 3 3 3 3
17537 044/300 Write block 543 len 4096 data: 5
17538 045/300 Read block 3250 len 40960
17539 046/300 Read block 2941 len 32768
17540 047/300 Flush
17541 048/300 Write block 3702 len 24576 data: 5 3 3 3 3 3
17542 049/300 Read block 4243 len 8192
17543 050/300 Read block 668 len 16384
17544 051/300 Write block 2173 len 8192 data: 4 3
17545 052/300 Write block 4623 len 12288 data: 4 4 4
17546 053/300 Write block 443 len 16384 data: 3 4 4 5
17547 054/300 Read block 4138 len 4096
17548 055/300 Read block 1578 len 32768
17549 056/300 Write block 4205 len 40960 data: 6 5 5 4 4 4 4 2 2 2
17550 057/300 Read block 4185 len 28672
17551 058/300 Write block 3385 len 20480 data: 5 4 5 4 5
17552 059/300 Read block 1593 len 28672
17553 060/300 Read block 3344 len 20480
17554 061/300 Write block 2252 len 24576 data: 3 3 2 2 3 3
17555 062/300 Write block 139 len 36864 data: 4 4 4 4 3 5 4 3 3
17556 063/300 Read block 1358 len 4096
17557 064/300 Read block 3878 len 28672
17558 065/300 Read block 1827 len 40960
17559 066/300 Read block 4075 len 32768
17560 067/300 Write block 476 len 32768 data: 5 5 7 8 7 9 8 8
17561 068/300 Write block 1906 len 32768 data: 4 5 5 5 5 6 4 4
17562 069/300 Read block 2553 len 24576
17563 070/300 Read block 1393 len 32768
17564 071/300 Flush
17565 072/300 Write block 14 len 28672 data: 5 5 6 6 5 5 5
17566 073/300 Read block 638 len 28672
17567 074/300 Read block 3182 len 36864
17568 075/300 Read block 4737 len 40960
17569 076/300 Read block 3269 len 40960
17570 077/300 Read block 132 len 24576
17571 078/300 Write block 2250 len 40960 data: 3 2 4 4 3 3 4 4 3 4
17572 079/300 Write block 2184 len 16384 data: 2 2 2 2
17573 080/300 Write block 554 len 28672 data: 5 4 6 5 6 5 3
17574 081/300 Read block 1502 len 36864
17575 082/300 Read block 3298 len 40960
17576 083/300 Read block 3263 len 24576
17577 084/300 Read block 3651 len 8192
17578 085/300 Read block 2134 len 4096
17579 086/300 Flush
17580 087/300 Write block 422 len 24576 data: 4 4 4 4 5 5
17581 088/300 Read block 4565 len 24576
17582 089/300 Flush
17583 090/300 Read block 1798 len 36864
17584 091/300 Write block 4691 len 28672 data: 3 4 4 4 4 4 5
17585 092/300 Flush
17586 093/300 Read block 4350 len 8192
17587 094/300 Write block 1882 len 36864 data: 7 6 8 6 6 6 7 7 6
17588 095/300 Write block 324 len 12288 data: 3 5 5
17589 096/300 Read block 2676 len 16384
17590 097/300 Read block 2852 len 4096
17591 098/300 Read block 174 len 36864
17592 099/300 Write block 4026 len 12288 data: 5 5 4
17593 100/300 Read block 4951 len 20480
17594 101/300 Write block 2327 len 4096 data: 3
17595 102/300 Read block 1265 len 20480
17596 103/300 Read block 241 len 24576
17597 104/300 Read block 347 len 12288
17598 105/300 Read block 2346 len 24576
17599 106/300 Read block 2822 len 32768
17600 107/300 Read block 3469 len 20480
17601 108/300 Read block 3683 len 8192
17602 109/300 Write block 627 len 16384 data: 2 2 3 3
17603 110/300 Read block 988 len 16384
17604 111/300 Write block 3512 len 20480 data: 4 5 6 5 5
17605 112/300 Read block 233 len 24576
17606 113/300 Read block 1580 len 40960
17607 114/300 Write block 1211 len 36864 data: 2 3 4 3 3 4 4 4 4
17608 115/300 Read block 4440 len 28672
17609 116/300 Write block 1947 len 24576 data: 4 6 6 6 7 5
17610 117/300 Write block 4296 len 24576 data: 5 4 4 4 3 3
17611 118/300 Read block 1441 len 36864
17612 119/300 Read block 2028 len 28672
17613 120/300 Flush
17614 121/300 Read block 3338 len 8192
17615 122/300 Write block 1565 len 32768 data: 3 3 4 4 3 3 3 2
17616 123/300 Write block 1638 len 36864 data: 4 4 2 3 3 3 2 2 2
17617 124/300 Read block 1883 len 40960
17618 125/300 Read block 619 len 8192
17619 126/300 Write block 3572 len 28672 data: 4 4 5 6 7 7 7
17620 127/300 Read block 2055 len 16384
17621 128/300 Read block 4614 len 32768
17622 129/300 Read block 533 len 8192
17623 130/300 Flush
17624 131/300 Read block 4354 len 40960
17625 132/300 Read block 4135 len 28672
17626 133/300 Write block 4778 len 4096 data: 6
17627 134/300 Read block 1824 len 36864
17628 135/300 Flush
17629 136/300 Write block 3030 len 28672 data: 2 2 3 3 3 3 3
17630 137/300 Read block 2510 len 16384
17631 138/300 Write block 1558 len 8192 data: 3 4
17632 139/300 Read block 1693 len 8192
17633 140/300 Write block 4139 len 12288 data: 6 7 7
17634 141/300 Write block 2067 len 24576 data: 4 5 3 2 3 3
17635 142/300 Write block 4044 len 8192 data: 5 5
17636 143/300 Flush
17637 144/300 Write block 1579 len 16384 data: 2 2 3 3
17638 145/300 Read block 3687 len 4096
17639 146/300 Write block 512 len 20480 data: 5 5 4 6 6
17640 147/300 Flush
17641 148/300 Write block 438 len 36864 data: 2 4 3 3 3 4 5 5 6
17642 149/300 Flush
17643 150/300 Write block 4880 len 4096 data: 3
17644 151/300 Write block 514 len 8192 data: 5 7
17645 152/300 Read block 516 len 24576
17646 153/300 Read block 1512 len 16384
17647 154/300 Read block 1703 len 16384
17648 155/300 Write block 348 len 32768 data: 5 4 5 4 4 3 3 3
17649 156/300 Read block 3349 len 28672
17650 157/300 Write block 3800 len 16384 data: 3 5 5 4
17651 158/300 Write block 66 len 16384 data: 3 3 4 4
17652 159/300 Write block 4490 len 12288 data: 5 5 4
17653 160/300 Read block 3185 len 28672
17654 161/300 Write block 778 len 20480 data: 8 7 6 6 5
17655 162/300 Read block 1100 len 8192
17656 163/300 Read block 3922 len 20480
17657 164/300 Read block 1457 len 12288
17658 165/300 Read block 3990 len 4096
17659 166/300 Read block 3762 len 4096
17660 167/300 Write block 2888 len 40960 data: 5 5 5 4 4 3 2 2 2 2
17661 168/300 Write block 1569 len 36864 data: 4 4 4 3 2 2 2 2 2
17662 169/300 Read block 2662 len 24576
17663 170/300 Write block 87 len 24576 data: 5 5 4 4 4 3
17664 171/300 Read block 250 len 12288
17665 172/300 Read block 3807 len 4096
17666 173/300 Write block 3327 len 36864 data: 3 3 3 2 3 4 4 4 4
17667 174/300 Read block 1833 len 12288
17668 175/300 Write block 1008 len 4096 data: 4
17669 176/300 Read block 1250 len 36864
17670 177/300 Read block 3664 len 32768
17671 178/300 Write block 3483 len 36864 data: 6 5 4 5 3 3 4 6 5
17672 179/300 Read block 74 len 20480
17673 180/300 Read block 4111 len 36864
17674 181/300 Read block 1443 len 24576
17675 182/300 Read block 2187 len 20480
17676 183/300 Read block 3535 len 40960
17677 184/300 Write block 3216 len 4096 data: 4
17678 185/300 Write block 745 len 4096 data: 12
17679 186/300 Write block 1070 len 24576 data: 3 3 3 4 4 3
17680 187/300 Read block 4878 len 28672
17681 188/300 Write block 2563 len 4096 data: 3
17682 189/300 Write block 3493 len 24576 data: 5 5 6 6 6 6
17683 190/300 Read block 455 len 40960
17684 191/300 Read block 1181 len 28672
17685 192/300 Write block 1973 len 16384 data: 7 6 4 4
17686 193/300 Read block 1920 len 40960
17687 194/300 Write block 2483 len 20480 data: 3 4 4 6 3
17688 195/300 Flush
17689 196/300 Write block 1508 len 4096 data: 6
17690 197/300 Write block 2444 len 8192 data: 5 4
17691 198/300 Read block 3081 len 28672
17692 199/300 Read block 4183 len 4096
17693 200/300 Write block 2886 len 28672 data: 5 5 6 6 6 5 5
17694 201/300 Read block 3356 len 4096
17695 202/300 Write block 4688 len 20480 data: 5 4 3 4 5
17696 203/300 Read block 4903 len 16384
17697 204/300 Write block 3986 len 20480 data: 3 4 4 5 5
17698 205/300 Read block 4133 len 28672
17699 206/300 Read block 4000 len 20480
17700 207/300 Flush
17701 208/300 Read block 1734 len 40960
17702 209/300 Write block 4447 len 24576 data: 2 2 4 5 6 7
17703 210/300 Read block 1346 len 28672
17704 211/300 Write block 2526 len 12288 data: 5 4 4
17705 212/300 Flush
17706 213/300 Read block 2127 len 20480
17707 214/300 Read block 3548 len 36864
17708 215/300 Read block 2885 len 24576
17709 216/300 Write block 2809 len 16384 data: 5 6 6 7
17710 217/300 Flush
17711 218/300 Write block 2869 len 40960 data: 4 3 3 2 2 2 2 2 2 2
17712 219/300 Read block 4165 len 40960
17713 220/300 Read block 797 len 16384
17714 221/300 Read block 4226 len 28672
17715 222/300 Read block 4232 len 36864
17716 223/300 Write block 1787 len 36864 data: 3 4 4 5 4 6 5 5 6
17717 224/300 Flush
17718 225/300 Flush
17719 226/300 Write block 1410 len 20480 data: 4 4 4 5 5
17720 227/300 Read block 2617 len 28672
17721 228/300 Read block 1121 len 24576
17722 229/300 Write block 1746 len 4096 data: 3
17723 230/300 Flush
17724 231/300 Read block 1607 len 20480
17725 232/300 Write block 1830 len 12288 data: 4 3 3
17726 233/300 Read block 2049 len 40960
17727 234/300 Read block 3888 len 16384
17728 235/300 Write block 3856 len 16384 data: 4 4 4 3
17729 236/300 Flush
17730 237/300 Read block 1891 len 32768
17731 238/300 Read block 631 len 8192
17732 239/300 Read block 1465 len 12288
17733 240/300 Write block 4665 len 16384 data: 5 4 3 3
17734 241/300 Write block 3667 len 16384 data: 2 2 2 2
17735 242/300 Read block 1418 len 28672
17736 243/300 Read block 3839 len 4096
17737 244/300 Write block 4939 len 20480 data: 2 2 2 2 2
17738 245/300 Write block 1238 len 40960 data: 3 2 2 4 5 4 4 4 3 4
17739 246/300 Read block 1217 len 36864
17740 247/300 Read block 2052 len 40960
17741 248/300 Write block 2512 len 4096 data: 4
17742 249/300 Write block 895 len 28672 data: 5 3 3 4 3 4 3
17743 250/300 Flush
17744 251/300 Read block 1765 len 32768
17745 252/300 Write block 1505 len 24576 data: 5 6 6 7 6 6
17746 253/300 Write block 1436 len 4096 data: 5
17747 254/300 Read block 4384 len 20480
17748 255/300 Write block 1893 len 24576 data: 5 3 3 3 3 3
17749 256/300 Read block 3768 len 40960
17750 257/300 Write block 333 len 32768 data: 5 4 4 4 4 5 4 4
17751 258/300 Write block 4042 len 20480 data: 5 5 6 6 7
17752 259/300 Read block 974 len 8192
17753 260/300 Read block 999 len 36864
17754 261/300 Write block 3692 len 24576 data: 3 3 3 3 2 2
17755 262/300 Read block 2229 len 8192
17756 263/300 Write block 2653 len 32768 data: 2 2 2 2 2 3 5 4
17757 264/300 Read block 376 len 40960
17758 265/300 Read block 3229 len 24576
17759 266/300 Read block 2538 len 28672
17760 267/300 Write block 1265 len 40960 data: 4 3 3 2 3 3 3 4 4 3
17761 268/300 Read block 2072 len 28672
17762 269/300 Write block 2788 len 8192 data: 3 3
17763 270/300 Write block 551 len 16384 data: 5 6 5 6
17764 271/300 Write block 4777 len 24576 data: 7 7 6 6 6 6
17765 272/300 Read block 608 len 4096
17766 273/300 Read block 565 len 40960
17767 274/300 Write block 498 len 32768 data: 7 5 5 3 6 6 7 7
17768 275/300 Write block 3063 len 28672 data: 3 3 3 2 2 2 2
17769 276/300 Write block 2847 len 28672 data: 2 2 2 2 2 3 3
17770 277/300 Read block 3178 len 40960
17771 278/300 Read block 2490 len 24576
17772 279/300 Write block 1142 len 28672 data: 2 3 2 2 2 2 3
17773 280/300 Read block 106 len 24576
17774 281/300 Read block 2291 len 36864
17775 282/300 Read block 4261 len 8192
17776 283/300 Write block 3615 len 24576 data: 4 3 3 2 3 3
17777 284/300 Read block 2033 len 28672
17778 285/300 Read block 3683 len 32768
17779 286/300 Write block 2037 len 36864 data: 4 4 3 3 3 2 2 2 2
17780 287/300 Write block 1570 len 4096 data: 5
17781 288/300 Write block 2005 len 8192 data: 7 7
17782 289/300 Write block 1117 len 36864 data: 3 3 4 3 3 4 5 6 6
17783 290/300 Write block 3245 len 32768 data: 2 3 3 4 4 4 4 3
17784 291/300 Flush
17785 292/300 Read block 1985 len 20480
17786 293/300 Write block 59 len 36864 data: 5 4 4 4 3 3 3 4 4
17787 294/300 Read block 429 len 36864
17788 295/300 Read block 2643 len 20480
17789 296/300 Flush
17790 297/300 Write block 1760 len 28672 data: 4 3 3 3 4 4 4
17791 298/300 Read block 952 len 40960
17792 299/300 Read block 3915 len 4096
17793 300/300 Write block 594 len 32768 data: 4 4 4 4 4 2 2 2
17794 301/300 Write block 3427 len 40960 data: 4 4 5 7 5 5 5 3 3 3
17795 Main task has work to do, go find it
17796 got dsc Start(0)
17797 start 0
17798 Jan 29 05:14:39.485 INFO request completed, latency_us: 222, response_code: 204, uri: /start/cid/0, method: POST, req_id: bc754795-5cb6-4156-8150-ac477c263ac8, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
17799 Replay: started 0, returned:Ok(())
17800 ----------------------------------------------------------------
17801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17802 States: Offline Active Active
17803 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17804 5530 Acked Write 2 Sent Done Done false
17805 5531 Acked Write 9 Sent Done Done false
17806 5532 Acked Write 9 Sent Done Done false
17807 5533 Acked Read 10 Sent Done Done false
17808 5534 Acked Write 8 Sent Done Done false
17809 5535 Acked Write 2 Sent Done Done false
17810 5536 Acked Write 2 Sent Done Done false
17811 5537 Acked Read 1 Sent Done Done false
17812 5538 Acked Write 9 Sent Done Done false
17813 5539 Acked Read 4 Sent Done Done false
17814 5540 Acked Read 1 Sent Done Done false
17815 5541 Acked Read 3 Sent Done Done false
17816 5542 Acked Read 5 Sent Done Done false
17817 5543 Acked Read 10 Sent Done Done false
17818 5544 Acked Read 9 Sent Done Done false
17819 5545 Acked Read 1 Sent Done Done false
17820 5546 Acked Write 8 Sent Done Done false
17821 5547 Acked Read 10 Sent Done Done false
17822 5548 Acked Read 6 Sent Done Done false
17823 5549 Acked Read 5 Sent Done Done false
17824 5550 Acked Read 10 Sent Done Done false
17825 5551 Acked Read 10 Sent Done Done false
17826 5552 Acked Read 4 Sent Done Done false
17827 5553 Acked Write 5 Sent Done Done false
17828 5554 Acked Flush 0 Sent Done Done false
17829 5555 Acked Read 4 Sent Done Done false
17830 5556 Acked Write 3 Sent Done Done false
17831 5557 Acked Read 9 Sent Done Done false
17832 5558 Acked Write 5 Sent Done Done false
17833 5559 Acked Flush 0 Sent Done Done false
17834 5560 Acked Flush 0 Sent Done Done false
17835 5561 Acked Write 7 Sent Done Done false
17836 5562 Acked Read 4 Sent Done Done false
17837 5563 Acked Flush 0 Sent Done Done false
17838 5564 Acked Read 2 Sent Done Done false
17839 5565 Acked Write 1 Sent Done Done false
17840 5566 Acked Read 5 Sent Done Done false
17841 5567 Acked Write 2 Sent Done Done false
17842 5568 Acked Write 2 Sent Done Done false
17843 5569 Acked Write 4 Sent Done Done false
17844 5570 Acked Read 4 Sent Done Done false
17845 5571 Acked Read 7 Sent Done Done false
17846 5572 Acked Write 9 Sent Done Done false
17847 5573 Acked Write 1 Sent Done Done false
17848 5574 Acked Read 10 Sent Done Done false
17849 5575 Acked Read 8 Sent Done Done false
17850 5576 Acked Flush 0 Sent Done Done false
17851 5577 Acked Write 6 Sent Done Done false
17852 5578 Acked Read 2 Sent Done Done false
17853 5579 Acked Read 4 Sent Done Done false
17854 5580 Acked Write 2 Sent Done Done false
17855 5581 Acked Write 3 Sent Done Done false
17856 5582 Acked Write 4 Sent Done Done false
17857 5583 Acked Read 1 Sent Done Done false
17858 5584 Acked Read 8 Sent Done Done false
17859 5585 Acked Write 10 Sent Done Done false
17860 5586 Acked Read 7 Sent Done Done false
17861 5587 Acked Write 5 Sent Done Done false
17862 5588 Acked Read 7 Sent Done Done false
17863 5589 Acked Read 5 Sent Done Done false
17864 5590 Acked Write 6 Sent Done Done false
17865 5591 Acked Write 9 Sent Done Done false
17866 5592 Acked Read 1 Sent Done Done false
17867 5593 Acked Read 7 Sent Done Done false
17868 5594 Acked Read 10 Sent Done Done false
17869 5595 Acked Read 8 Sent Done Done false
17870 5596 Acked Write 8 Sent Done Done false
17871 5597 Acked Write 8 Sent Done Done false
17872 5598 Acked Read 6 Sent Done Done false
17873 5599 Acked Read 8 Sent Done Done false
17874 5600 Acked Flush 0 Sent Done Done false
17875 5601 Acked Write 7 Sent Done Done false
17876 5602 Acked Read 7 Sent Done Done false
17877 5603 Acked Read 9 Sent Done Done false
17878 5604 Acked Read 10 Sent Done Done false
17879 5605 Acked Read 10 Sent Done Done false
17880 5606 Acked Read 6 Sent Done Done false
17881 5607 Acked Write 10 Sent Done Done false
17882 5608 Acked Write 4 Sent Done Done false
17883 5609 Acked Write 7 Sent Done Done false
17884 5610 Acked Read 9 Sent Done Done false
17885 5611 Acked Read 10 Sent Done Done false
17886 5612 Acked Read 6 Sent Done Done false
17887 5613 Acked Read 2 Sent Done Done false
17888 5614 Acked Read 1 Sent Done Done false
17889 5615 Acked Flush 0 Sent Done Done false
17890 5616 Acked Write 6 Sent Done Done false
17891 5617 Acked Read 6 Sent Done Done false
17892 5618 Acked Flush 0 Sent Done Done false
17893 5619 Acked Read 9 Sent Done Done false
17894 5620 Acked Write 7 Sent Done Done false
17895 5621 Acked Flush 0 Sent Done Done false
17896 5622 Acked Read 2 Sent Done Done false
17897 5623 Acked Write 9 Sent Done Done false
17898 5624 Acked Write 3 Sent Done Done false
17899 5625 Acked Read 4 Sent Done Done false
17900 5626 Acked Read 1 Sent Done Done false
17901 5627 Acked Read 9 Sent Done Done false
17902 5628 Acked Write 3 Sent Done Done false
17903 5629 Acked Read 5 Sent Done Done false
17904 5630 Acked Write 1 Sent Done Done false
17905 5631 Acked Read 5 Sent Done Done false
17906 5632 Acked Read 6 Sent Done Done false
17907 5633 Acked Read 3 Sent Done Done false
17908 5634 Acked Read 6 Sent Done Done false
17909 5635 Acked Read 8 Sent Done Done false
17910 5636 Acked Read 5 Sent Done Done false
17911 5637 Acked Read 2 Sent Done Done false
17912 5638 Acked Write 4 Sent Done Done false
17913 5639 Acked Read 4 Sent Done Done false
17914 5640 Acked Write 5 Sent Done Done false
17915 5641 Acked Read 6 Sent Done Done false
17916 5642 Acked Read 10 Sent Done Done false
17917 5643 Acked Write 9 Sent Done Done false
17918 5644 Acked Read 7 Sent Done Done false
17919 5645 Acked Write 6 Sent Done Done false
17920 5646 Acked Write 6 Sent Done Done false
17921 5647 Acked Read 9 Sent Done Done false
17922 5648 Acked Read 7 Sent Done Done false
17923 5649 Acked Flush 0 Sent Done Done false
17924 5650 Acked Read 2 Sent Done Done false
17925 5651 Acked Write 8 Sent Done Done false
17926 5652 Acked Write 9 Sent Done Done false
17927 5653 Acked Read 10 Sent Done Done false
17928 5654 Acked Read 2 Sent Done Done false
17929 5655 Acked Write 7 Sent Done Done false
17930 5656 Acked Read 4 Sent Done Done false
17931 5657 Acked Read 8 Sent Done Done false
17932 5658 Acked Read 2 Sent Done Done false
17933 5659 Acked Flush 0 Sent Done Done false
17934 5660 Acked Read 10 Sent Done Done false
17935 5661 Acked Read 7 Sent Done Done false
17936 5662 Acked Write 1 Sent Done Done false
17937 5663 Acked Read 9 Sent Done Done false
17938 5664 Acked Flush 0 Sent Done Done false
17939 5665 Acked Write 7 Sent Done Done false
17940 5666 Acked Read 4 Sent Done Done false
17941 5667 Acked Write 2 Sent Done Done false
17942 5668 Acked Read 2 Sent Done Done false
17943 5669 Acked Write 3 Sent Done Done false
17944 5670 Acked Write 6 Sent Done Done false
17945 5671 Acked Write 2 Sent Done Done false
17946 5672 Acked Flush 0 Sent Done Done false
17947 5673 Acked Write 4 Sent Done Done false
17948 5674 Acked Read 1 Sent Done Done false
17949 5675 Acked Write 5 Sent Done Done false
17950 5676 Acked Flush 0 Sent Done Done false
17951 5677 Acked Write 9 Sent Done Done false
17952 5678 Acked Flush 0 Sent Done Done false
17953 5679 Acked Write 1 Sent Done Done false
17954 5680 Acked Write 2 Sent Done Done false
17955 5681 Acked Read 6 Sent Done Done false
17956 5682 Acked Read 4 Sent Done Done false
17957 5683 Acked Read 4 Sent Done Done false
17958 5684 Acked Write 8 Sent Done Done false
17959 5685 Acked Read 7 Sent Done Done false
17960 5686 Acked Write 4 Sent Done Done false
17961 5687 Acked Write 4 Sent Done Done false
17962 5688 Acked Write 3 Sent Done Done false
17963 5689 Acked Read 7 Sent Done Done false
17964 5690 Acked Write 5 Sent Done Done false
17965 5691 Acked Read 2 Sent Done Done false
17966 5692 Acked Read 5 Sent Done Done false
17967 5693 Acked Read 3 Sent Done Done false
17968 5694 Acked Read 1 Sent Done Done false
17969 5695 Acked Read 1 Sent Done Done false
17970 5696 Acked Write 10 Sent Done Done false
17971 5697 Acked Write 9 Sent Done Done false
17972 5698 Acked Read 6 Sent Done Done false
17973 5699 Acked Write 6 Sent Done Done false
17974 5700 Acked Read 3 Sent Done Done false
17975 5701 Acked Read 1 Sent Done Done false
17976 5702 Acked Write 9 Sent Done Done false
17977 5703 Acked Read 3 Sent Done Done false
17978 5704 Acked Write 1 Sent Done Done false
17979 5705 Acked Read 9 Sent Done Done false
17980 5706 Acked Read 8 Sent Done Done false
17981 5707 Acked Write 9 Sent Done Done false
17982 5708 Acked Read 5 Sent Done Done false
17983 5709 Acked Read 9 Sent Done Done false
17984 5710 Acked Read 6 Sent Done Done false
17985 5711 Acked Read 5 Sent Done Done false
17986 5712 Acked Read 10 Sent Done Done false
17987 5713 Acked Write 1 Sent Done Done false
17988 5714 Acked Write 1 Sent Done Done false
17989 5715 Acked Write 6 Sent Done Done false
17990 5716 Acked Read 7 Sent Done Done false
17991 5717 Acked Write 1 Sent Done Done false
17992 5718 Acked Write 6 Sent Done Done false
17993 5719 Acked Read 10 Sent Done Done false
17994 5720 Acked Read 7 Sent Done Done false
17995 5721 Acked Write 4 Sent Done Done false
17996 5722 Acked Read 10 Sent Done Done false
17997 5723 Acked Write 5 Sent Done Done false
17998 5724 Acked Flush 0 Sent Done Done false
17999 5725 Acked Write 1 Sent Done Done false
18000 5726 Acked Write 2 Sent Done Done false
18001 5727 Acked Read 7 Sent Done Done false
18002 5728 Acked Read 1 Sent Done Done false
18003 5729 Acked Write 7 Sent Done Done false
18004 5730 Acked Read 1 Sent Done Done false
18005 5731 Acked Write 5 Sent Done Done false
18006 5732 Acked Read 4 Sent Done Done false
18007 5733 Acked Write 5 Sent Done Done false
18008 5734 Acked Read 7 Sent Done Done false
18009 5735 Acked Read 5 Sent Done Done false
18010 5736 Acked Flush 0 Sent Done Done false
18011 5737 Acked Read 10 Sent Done Done false
18012 5738 Acked Write 6 Sent Done Done false
18013 5739 Acked Read 7 Sent Done Done false
18014 5740 Acked Write 3 Sent Done Done false
18015 5741 Acked Flush 0 Sent Done Done false
18016 5742 Acked Read 5 Sent Done Done false
18017 5743 Acked Read 9 Sent Done Done false
18018 5744 Acked Read 6 Sent Done Done false
18019 5745 Acked Write 4 Sent Done Done false
18020 5746 Acked Flush 0 Sent Done Done false
18021 5747 Acked Write 10 Sent Done Done false
18022 5748 Acked Read 10 Sent Done Done false
18023 5749 Acked Read 4 Sent Done Done false
18024 5750 Acked Read 7 Sent Done Done false
18025 5751 Acked Read 9 Sent Done Done false
18026 5752 Acked Write 9 Sent Done Done false
18027 5753 Acked Flush 0 Sent Done Done false
18028 5754 Acked Flush 0 Sent Done Done false
18029 5755 Acked Write 5 Sent Done Done false
18030 5756 Acked Read 7 Sent Done Done false
18031 5757 Acked Read 6 Sent Done Done false
18032 5758 Acked Write 1 Sent Done Done false
18033 5759 Acked Flush 0 Sent Done Done false
18034 5760 Acked Read 5 Sent Done Done false
18035 5761 Acked Write 3 Sent Done Done false
18036 5762 Acked Read 10 Sent Done Done false
18037 5763 Acked Read 4 Sent Done Done false
18038 5764 Acked Write 4 Sent Done Done false
18039 5765 Acked Flush 0 Sent Done Done false
18040 5766 Acked Read 8 Sent Done Done false
18041 5767 Acked Read 2 Sent Done Done false
18042 5768 Acked Read 3 Sent Done Done false
18043 5769 Acked Write 4 Sent Done Done false
18044 5770 Acked Write 4 Sent Done Done false
18045 5771 Acked Read 7 Sent Done Done false
18046 5772 Acked Read 1 Sent Done Done false
18047 5773 Acked Write 5 Sent Done Done false
18048 5774 Acked Write 10 Sent Done Done false
18049 5775 Acked Read 9 Sent Done Done false
18050 5776 Acked Read 10 Sent Done Done false
18051 5777 Acked Write 1 Sent Done Done false
18052 5778 Acked Write 7 Sent Done Done false
18053 5779 Acked Flush 0 Sent Done Done false
18054 5780 Acked Read 8 Sent Done Done false
18055 5781 Acked Write 6 Sent Done Done false
18056 5782 Acked Write 1 Sent Done Done false
18057 5783 Acked Read 5 Sent Done Done false
18058 5784 Acked Write 6 Sent Done Done false
18059 5785 Acked Read 10 Sent Done Done false
18060 5786 Acked Write 8 Sent Done Done false
18061 5787 Acked Write 5 Sent Done Done false
18062 5788 Acked Read 2 Sent Done Done false
18063 5789 Acked Read 9 Sent Done Done false
18064 5790 Acked Write 6 Sent Done Done false
18065 5791 Acked Read 2 Sent Done Done false
18066 5792 Acked Write 8 Sent Done Done false
18067 5793 Acked Read 10 Sent Done Done false
18068 5794 Acked Read 6 Sent Done Done false
18069 5795 Acked Read 7 Sent Done Done false
18070 5796 Acked Write 10 Sent Done Done false
18071 5797 Acked Read 7 Sent Done Done false
18072 5798 Acked Write 2 Sent Done Done false
18073 5799 Acked Write 4 Sent Done Done false
18074 5800 Acked Write 6 Sent Done Done false
18075 5801 Acked Read 1 Sent Done Done false
18076 5802 Acked Read 10 Sent Done Done false
18077 5803 Acked Write 8 Sent Done Done false
18078 5804 Acked Write 7 Sent Done Done false
18079 5805 Acked Write 7 Sent Done Done false
18080 5806 Acked Read 10 Sent Done Done false
18081 5807 Acked Read 6 Sent Done Done false
18082 5808 Acked Write 7 Sent Done Done false
18083 5809 Acked Read 6 Sent Done Done false
18084 5810 Acked Read 9 Sent Done Done false
18085 5811 Acked Read 2 Sent Done Done false
18086 5812 Acked Write 6 Sent Done Done false
18087 5813 Acked Read 7 Sent Done Done false
18088 5814 Acked Read 8 Sent Done Done false
18089 5815 Acked Write 9 Sent Done Done false
18090 5816 Acked Write 1 Sent Done Done false
18091 5817 Acked Write 2 Sent Done Done false
18092 5818 Acked Write 9 Sent Done Done false
18093 5819 Acked Write 8 Sent Done Done false
18094 5820 Acked Flush 0 Sent Done Done false
18095 5821 Acked Read 5 Sent Done Done false
18096 5822 Acked Write 9 Sent Done Done false
18097 5823 Acked Read 9 Sent Done Done false
18098 5824 Acked Read 5 Sent Done Done false
18099 5825 Acked Flush 0 Sent Done Done false
18100 5826 Acked Write 7 Sent Done Done false
18101 5827 Acked Read 10 Sent Done Done false
18102 5828 Acked Read 1 Sent Done Done false
18103 5829 Acked Write 8 Sent Done Done false
18104 5830 Acked Write 10 Sent Done Sent false
18105 STATES DS:0 DS:1 DS:2 TOTAL
18106 Sent 301 0 1 302
18107 Done 0 301 300 601
18108 Skipped 0 0 0 0
18109 Error 0 0 0 0
18110 Last Flush: Some(JobId(5529)) Some(JobId(5825)) Some(JobId(5825))
18111 Downstairs last five completed: 5529 5528 5527 5526 5525
18112 Upstairs last five completed: 5830 5829 5828 5827 5826
18113 CLIENT: Up:0 ds:301 act:2
181142025-01-29T05:14:40.307ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
181152025-01-29T05:14:41.309ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
181162025-01-29T05:14:42.311ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
181172025-01-29T05:14:42.811ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
18118 [8810] got start action
18119 [8810] I am going to restart
18120 Starting downstairs at port 8810
18121 Make output file at "/tmp/dsc/downstairs-8810.txt"
18122 [8810][0] reports Starting
18123 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1218)
18124 [8810][0] reports Running
18125 {"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":----------------------------------------------------------------
18126 40 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18127 States: Offline Active Active
18128 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18129 5530 Acked Write 2 Sent Done Done false
18130 5531 Acked Write 9 Sent Done Done false
18131 5532 Acked Write 9 Sent Done Done false
18132 5533 Acked Read 10 Sent Done Done false
18133 5534 Acked Write 8 Sent Done Done false
18134 5535 Acked Write 2 Sent Done Done false
18135 ,"time 5536 Acked Write 2" Sent: Done" Done false
18136 2025-01-29T05:14:43.487913139Z" 5537 Acked Read 1, Sent" Donehostname" Done: false
18137 " 5538 Acked Write 9w-01JJR7DDKGFJDPMCDR9NFSE52V Sent" Done, Done" falsepid"
18138 : 5539 Acked Read 41202 Sent Done Done false
18139 ," 5540 Acked Read 1session_id" Sent: Done" Done false
18140 3b9f53a2-1179-49c9-b40f-5354d75015be" 5541 Acked Read 3 Sent} Done
18141 Done false
18142 5542 Acked Read 5 Sent Done Done false
18143 5543 Acked Read 10 Sent Done Done false
18144 5544 Acked Read 9 Sent Done Done false
18145 5545 Acked Read 1 Sent Done Done false
18146 5546 Acked Write 8 Sent Done Done false
18147 5547 Acked Read 10 Sent Done Done false
18148 5548 Acked Read 6 Sent Done Done false
18149 5549 Acked Read 5 Sent Done Done false
18150 5550 Acked Read 10 Sent Done Done false
18151 5551 Acked Read 10 Sent Done Done false
18152 5552 Acked Read 4 Sent Done Done false
18153 5553 Acked Write 5 Sent Done Done false
18154 5554 Acked Flush 0 Sent Done Done false
18155 5555 Acked Read 4 Sent Done Done false
18156 5556 Acked Write 3 Sent Done Done false
18157 5557 Acked Read 9 Sent Done Done false
18158 5558 Acked Write 5 Sent Done Done false
18159 5559 Acked Flush 0 Sent Done Done false
18160 5560 Acked Flush 0 Sent Done Done false
18161 5561 Acked Write 7 Sent Done Done false
18162 5562 Acked Read 4 Sent Done Done false
18163 5563 Acked Flush 0 Sent Done Done false
18164 5564 Acked Read 2 Sent Done Done false
18165 5565 Acked Write 1 Sent Done Done false
18166 5566 Acked Read 5 Sent Done Done false
18167 5567 Acked Write 2 Sent Done Done false
18168 5568 Acked Write 2 Sent Done Done false
18169 5569 Acked Write 4 Sent Done Done false
18170 5570 Acked Read 4 Sent Done Done false
18171 5571 Acked Read 7 Sent Done Done false
18172 5572 Acked Write 9 Sent Done Done false
18173 5573 Acked Write 1 Sent Done Done false
18174 5574 Acked Read 10 Sent Done Done false
18175 5575 Acked Read 8 Sent Done Done false
18176 5576 Acked Flush 0 Sent Done Done false
18177 5577 Acked Write 6 Sent Done Done false
18178 5578 Acked Read 2 Sent Done Done false
18179 5579 Acked Read 4 Sent Done Done false
18180 5580 Acked Write 2 Sent Done Done false
18181 5581 Acked Write 3 Sent Done Done false
18182 5582 Acked Write 4 Sent Done Done false
18183 5583 Acked Read 1 Sent Done Done false
18184 5584 Acked Read 8 Sent Done Done false
18185 5585 Acked Write 10 Sent Done Done false
18186 5586 Acked Read 7 Sent Done Done false
18187 5587 Acked Write 5 Sent Done Done false
18188 5588 Acked Read 7 Sent Done Done false
18189 5589 Acked Read 5 Sent Done Done false
18190 5590 Acked Write 6 Sent Done Done false
18191 5591 Acked Write 9 Sent Done Done false
18192 5592 Acked Read 1 Sent Done Done false
18193 5593 Acked Read 7 Sent Done Done false
18194 5594 Acked Read 10 Sent Done Done false
18195 5595 Acked Read 8 Sent Done Done false
18196 5596 Acked Write 8 Sent Done Done false
18197 5597 Acked Write 8 Sent Done Done false
18198 5598 Acked Read 6 Sent Done Done false
18199 5599 Acked Read 8 Sent Done Done false
18200 5600 Acked Flush 0 Sent Done Done false
18201 5601 Acked Write 7 Sent Done Done false
18202 5602 Acked Read 7 Sent Done Done false
18203 5603 Acked Read 9 Sent Done Done false
18204 5604 Acked Read 10 Sent Done Done false
18205 5605 Acked Read 10 Sent Done Done false
18206 5606 Acked Read 6 Sent Done Done false
18207 5607 Acked Write 10 Sent Done Done false
18208 5608 Acked Write 4 Sent Done Done false
18209 5609 Acked Write 7 Sent Done Done false
18210 5610 Acked Read 9 Sent Done Done false
18211 5611 Acked Read 10 Sent Done Done false
18212 5612 Acked Read 6 Sent Done Done false
18213 5613 Acked Read 2 Sent Done Done false
18214 5614 Acked Read 1 Sent Done Done false
18215 5615 Acked Flush 0 Sent Done Done false
18216 5616 Acked Write 6 Sent Done Done false
18217 5617 Acked Read 6 Sent Done Done false
18218 5618 Acked Flush 0 Sent Done Done false
18219 5619 Acked Read 9 Sent Done Done false
18220 5620 Acked Write 7 Sent Done Done false
18221 5621 Acked Flush 0 Sent Done Done false
18222 5622 Acked Read 2 Sent Done Done false
18223 5623 Acked Write 9 Sent Done Done false
18224 5624 Acked Write 3 Sent Done Done false
18225 5625 Acked Read 4 Sent Done Done false
18226 5626 Acked Read 1 Sent Done Done false
18227 5627 Acked Read 9 Sent Done Done false
18228 5628 Acked Write 3 Sent Done Done false
18229 5629 Acked Read 5 Sent Done Done false
18230 5630 Acked Write 1 Sent Done Done false
18231 5631 Acked Read 5 Sent Done Done false
18232 5632 Acked Read 6 Sent Done Done false
18233 5633 Acked Read 3 Sent Done Done false
18234 5634 Acked Read 6 Sent Done Done false
18235 5635 Acked Read 8 Sent Done Done false
18236 5636 Acked Read 5 Sent Done Done false
18237 5637 Acked Read 2 Sent Done Done false
18238 5638 Acked Write 4 Sent Done Done false
18239 5639 Acked Read 4 Sent Done Done false
18240 5640 Acked Write 5 Sent Done Done false
18241 5641 Acked Read 6 Sent Done Done false
18242 5642 Acked Read 10 Sent Done Done false
18243 5643 Acked Write 9 Sent Done Done false
18244 5644 Acked Read 7 Sent Done Done false
18245 5645 Acked Write 6 Sent Done Done false
18246 5646 Acked Write 6 Sent Done Done false
18247 5647 Acked Read 9 Sent Done Done false
18248 5648 Acked Read 7 Sent Done Done false
18249 5649 Acked Flush 0 Sent Done Done false
18250 5650 Acked Read 2 Sent Done Done false
18251 5651 Acked Write 8 Sent Done Done false
18252 5652 Acked Write 9 Sent Done Done false
18253 5653 Acked Read 10 Sent Done Done false
18254 5654 Acked Read 2 Sent Done Done false
18255 5655 Acked Write 7 Sent Done Done false
18256 5656 Acked Read 4 Sent Done Done false
18257 5657 Acked Read 8 Sent Done Done false
18258 5658 Acked Read 2 Sent Done Done false
18259 5659 Acked Flush 0 Sent Done Done false
18260 5660 Acked Read 10 Sent Done Done false
18261 5661 Acked Read 7 Sent Done Done false
18262 5662 Acked Write 1 Sent Done Done false
18263 5663 Acked Read 9 Sent Done Done false
18264 5664 Acked Flush 0 Sent Done Done false
18265 5665 Acked Write 7 Sent Done Done false
18266 5666 Acked Read 4 Sent Done Done false
18267 5667 Acked Write 2 Sent Done Done false
18268 5668 Acked Read 2 Sent Done Done false
18269 5669 Acked Write 3 Sent Done Done false
18270 5670 Acked Write 6 Sent Done Done false
18271 5671 Acked Write 2 Sent Done Done false
18272 5672 Acked Flush 0 Sent Done Done false
18273 5673 Acked Write 4 Sent Done Done false
18274 5674 Acked Read 1 Sent Done Done false
18275 5675 Acked Write 5 Sent Done Done false
18276 5676 Acked Flush 0 Sent Done Done false
18277 5677 Acked Write 9 Sent Done Done false
18278 5678 Acked Flush 0 Sent Done Done false
18279 5679 Acked Write 1 Sent Done Done false
18280 5680 Acked Write 2 Sent Done Done false
18281 5681 Acked Read 6 Sent Done Done false
18282 5682 Acked Read 4 Sent Done Done false
18283 5683 Acked Read 4 Sent Done Done false
18284 5684 Acked Write 8 Sent Done Done false
18285 5685 Acked Read 7 Sent Done Done false
18286 5686 Acked Write 4 Sent Done Done false
18287 5687 Acked Write 4 Sent Done Done false
18288 5688 Acked Write 3 Sent Done Done false
18289 5689 Acked Read 7 Sent Done Done false
18290 5690 Acked Write 5 Sent Done Done false
18291 5691 Acked Read 2 Sent Done Done false
18292 5692 Acked Read 5 Sent Done Done false
18293 5693 Acked Read 3 Sent Done Done false
18294 5694 Acked Read 1 Sent Done Done false
18295 5695 Acked Read 1 Sent Done Done false
18296 5696 Acked Write 10 Sent Done Done false
18297 5697 Acked Write 9 Sent Done Done false
18298 5698 Acked Read 6 Sent Done Done false
18299 5699 Acked Write 6 Sent Done Done false
18300 5700 Acked Read 3 Sent Done Done false
18301 5701 Acked Read 1 Sent Done Done false
18302 5702 Acked Write 9 Sent Done Done false
18303 5703 Acked Read 3 Sent Done Done false
18304 5704 Acked Write 1 Sent Done Done false
18305 5705 Acked Read 9 Sent Done Done false
18306 5706 Acked Read 8 Sent Done Done false
18307 5707 Acked Write 9 Sent Done Done false
18308 5708 Acked Read 5 Sent Done Done false
18309 5709 Acked Read 9 Sent Done Done false
18310 5710 Acked Read 6 Sent Done Done false
18311 5711 Acked Read 5 Sent Done Done false
18312 5712 Acked Read 10 Sent Done Done false
18313 5713 Acked Write 1 Sent Done Done false
18314 5714 Acked Write 1 Sent Done Done false
18315 5715 Acked Write 6 Sent Done Done false
18316 5716 Acked Read 7 Sent Done Done false
18317 5717 Acked Write 1 Sent Done Done false
18318 5718 Acked Write 6 Sent Done Done false
18319 5719 Acked Read 10 Sent Done Done false
18320 5720 Acked Read 7 Sent Done Done false
18321 5721 Acked Write 4 Sent Done Done false
18322 5722 Acked Read 10 Sent Done Done false
18323 5723 Acked Write 5 Sent Done Done false
18324 5724 Acked Flush 0 Sent Done Done false
18325 5725 Acked Write 1 Sent Done Done false
18326 5726 Acked Write 2 Sent Done Done false
18327 5727 Acked Read 7 Sent Done Done false
18328 5728 Acked Read 1 Sent Done Done false
18329 5729 Acked Write 7 Sent Done Done false
18330 5730 Acked Read 1 Sent Done Done false
18331 5731 Acked Write 5 Sent Done Done false
18332 5732 Acked Read 4 Sent Done Done false
18333 5733 Acked Write 5 Sent Done Done false
18334 5734 Acked Read 7 Sent Done Done false
18335 5735 Acked Read 5 Sent Done Done false
18336 5736 Acked Flush 0 Sent Done Done false
18337 5737 Acked Read 10 Sent Done Done false
18338 5738 Acked Write 6 Sent Done Done false
18339 5739 Acked Read 7 Sent Done Done false
18340 5740 Acked Write 3 Sent Done Done false
18341 5741 Acked Flush 0 Sent Done Done false
18342 5742 Acked Read 5 Sent Done Done false
18343 5743 Acked Read 9 Sent Done Done false
18344 5744 Acked Read 6 Sent Done Done false
18345 5745 Acked Write 4 Sent Done Done false
18346 5746 Acked Flush 0 Sent Done Done false
18347 5747 Acked Write 10 Sent Done Done false
18348 5748 Acked Read 10 Sent Done Done false
18349 5749 Acked Read 4 Sent Done Done false
18350 5750 Acked Read 7 Sent Done Done false
18351 5751 Acked Read 9 Sent Done Done false
18352 5752 Acked Write 9 Sent Done Done false
18353 5753 Acked Flush 0 Sent Done Done false
18354 5754 Acked Flush 0 Sent Done Done false
18355 5755 Acked Write 5 Sent Done Done false
18356 5756 Acked Read 7 Sent Done Done false
18357 5757 Acked Read 6 Sent Done Done false
18358 5758 Acked Write 1 Sent Done Done false
18359 5759 Acked Flush 0 Sent Done Done false
18360 5760 Acked Read 5 Sent Done Done false
18361 5761 Acked Write 3 Sent Done Done false
18362 5762 Acked Read 10 Sent Done Done false
18363 5763 Acked Read 4 Sent Done Done false
18364 5764 Acked Write 4 Sent Done Done false
18365 5765 Acked Flush 0 Sent Done Done false
18366 5766 Acked Read 8 Sent Done Done false
18367 5767 Acked Read 2 Sent Done Done false
18368 5768 Acked Read 3 Sent Done Done false
18369 5769 Acked Write 4 Sent Done Done false
18370 5770 Acked Write 4 Sent Done Done false
18371 5771 Acked Read 7 Sent Done Done false
18372 5772 Acked Read 1 Sent Done Done false
18373 5773 Acked Write 5 Sent Done Done false
18374 5774 Acked Write 10 Sent Done Done false
18375 5775 Acked Read 9 Sent Done Done false
18376 5776 Acked Read 10 Sent Done Done false
18377 5777 Acked Write 1 Sent Done Done false
18378 5778 Acked Write 7 Sent Done Done false
18379 5779 Acked Flush 0 Sent Done Done false
18380 5780 Acked Read 8 Sent Done Done false
18381 5781 Acked Write 6 Sent Done Done false
18382 5782 Acked Write 1 Sent Done Done false
18383 5783 Acked Read 5 Sent Done Done false
18384 5784 Acked Write 6 Sent Done Done false
18385 5785 Acked Read 10 Sent Done Done false
18386 5786 Acked Write 8 Sent Done Done false
18387 5787 Acked Write 5 Sent Done Done false
18388 5788 Acked Read 2 Sent Done Done false
18389 5789 Acked Read 9 Sent Done Done false
18390 5790 Acked Write 6 Sent Done Done false
18391 5791 Acked Read 2 Sent Done Done false
18392 5792 Acked Write 8 Sent Done Done false
18393 5793 Acked Read 10 Sent Done Done false
18394 5794 Acked Read 6 Sent Done Done false
18395 5795 Acked Read 7 Sent Done Done false
18396 5796 Acked Write 10 Sent Done Done false
18397 5797 Acked Read 7 Sent Done Done false
18398 5798 Acked Write 2 Sent Done Done false
18399 5799 Acked Write 4 Sent Done Done false
18400 5800 Acked Write 6 Sent Done Done false
18401 5801 Acked Read 1 Sent Done Done false
18402 5802 Acked Read 10 Sent Done Done false
18403 5803 Acked Write 8 Sent Done Done false
18404 5804 Acked Write 7 Sent Done Done false
18405 5805 Acked Write 7 Sent Done Done false
18406 5806 Acked Read 10 Sent Done Done false
18407 5807 Acked Read 6 Sent Done Done false
18408 5808 Acked Write 7 Sent Done Done false
18409 5809 Acked Read 6 Sent Done Done false
18410 5810 Acked Read 9 Sent Done Done false
18411 5811 Acked Read 2 Sent Done Done false
18412 5812 Acked Write 6 Sent Done Done false
18413 5813 Acked Read 7 Sent Done Done false
18414 5814 Acked Read 8 Sent Done Done false
18415 5815 Acked Write 9 Sent Done Done false
18416 5816 Acked Write 1 Sent Done Done false
18417 5817 Acked Write 2 Sent Done Done false
18418 5818 Acked Write 9 Sent Done Done false
18419 5819 Acked Write 8 Sent Done Done false
18420 5820 Acked Flush 0 Sent Done Done false
18421 5821 Acked Read 5 Sent Done Done false
18422 5822 Acked Write 9 Sent Done Done false
18423 5823 Acked Read 9 Sent Done Done false
18424 5824 Acked Read 5 Sent Done Done false
18425 5825 Acked Flush 0 Sent Done Done false
18426 5826 Acked Write 7 Sent Done Done false
18427 5827 Acked Read 10 Sent Done Done false
18428 5828 Acked Read 1 Sent Done Done false
18429 5829 Acked Write 8 Sent Done Done false
18430 5830 Acked Write 10 Sent Done Done false
18431 STATES DS:0 DS:1 DS:2 TOTAL
18432 Sent 301 0 0 301
18433 Done 0 301 301 602
18434 Skipped 0 0 0 0
18435 Error 0 0 0 0
18436 Last Flush: Some(JobId(5529)) Some(JobId(5825)) Some(JobId(5825))
18437 Downstairs last five completed: 5529 5528 5527 5526 5525
18438 Upstairs last five completed: 5830 5829 5828 5827 5826
18439 CLIENT: Up:0 ds:301 act:2
184402025-01-29T05:14:44.315ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184412025-01-29T05:14:45.233ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184422025-01-29T05:14:45.233ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49273) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184432025-01-29T05:14:45.234ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184442025-01-29T05:14:45.234ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184452025-01-29T05:14:45.234ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184462025-01-29T05:14:45.234ZINFOcrucible: send last flush ID to this DS: Some(JobId(5529)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184472025-01-29T05:14:45.234ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184482025-01-29T05:14:45.234ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5529)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184492025-01-29T05:14:45.235ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5529)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
18450 ----------------------------------------------------------------
18451 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18452 Downstairs last five completed: 5831 5830 5829 5828 5827
18453 Upstairs last five completed: 5831 5830 5829 5828 5827
18454 CLIENT: Up:0 ds:0 act:3
18455 Replay: All jobs finished
18456 Main task has work to do, go find it
18457 got dsc Stop(1)
18458 stop 1
18459 Jan 29 05:14:47.491 [8820] Got stop action so:false kr:false
18460 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/1, method: POST, req_id: bca92918-10fa-4cbd-9b95-4716168bb8c3, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
184612025-01-29T05:14:47.491ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184622025-01-29T05:14:47.492ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
184632025-01-29T05:14:47.492ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
18464 Jan 29 05:14:47.492 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8fd98e8d-d82c-4db4-a096-66662f3d9371, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
184652025-01-29T05:14:47.492ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
18466 [8820] Exited with: ExitStatus(unix_wait_status(9))
18467 [8820][1] reports Exit
18468 Jan 29 05:14:51.494 INFO request completed, latency_us: 242, response_code: 200, uri: /state/cid/1, method: GET, req_id: 348838f8-f8de-4841-b4e5-079cd711b65f, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
18469 001/300 Read block 4598 len 24576
18470 002/300 Read block 2669 len 32768
18471 003/300 Read block 3851 len 24576
18472 004/300 Flush
18473 005/300 Read block 565 len 12288
18474 006/300 Write block 2296 len 28672 data: 4 7 5 4 4 4 4
18475 007/300 Read block 2758 len 28672
18476 008/300 Read block 4995 len 16384
18477 009/300 Read block 2059 len 36864
18478 010/300 Read block 3420 len 12288
18479 011/300 Read block 3513 len 8192
18480 012/300 Write block 399 len 24576 data: 3 3 3 3 3 4
18481 013/300 Read block 1106 len 4096
18482 014/300 Read block 3420 len 12288
18483 015/300 Read block 1805 len 32768
18484 016/300 Read block 1879 len 16384
18485 017/300 Write block 925 len 8192 data: 2 2
18486 018/300 Write block 126 len 4096 data: 7
18487 019/300 Read block 1292 len 8192
18488 020/300 Read block 255 len 40960
18489 021/300 Write block 1292 len 16384 data: 2 3 3 3
18490 022/300 Read block 1367 len 8192
18491 023/300 Read block 3039 len 12288
18492 024/300 Write block 752 len 36864 data: 6 6 5 6 5 5 4 4 4
18493 025/300 Flush
18494 026/300 Write block 3260 len 36864 data: 3 2 2 2 2 2 2 3 2
18495 027/300 Write block 2727 len 36864 data: 3 3 3 3 5 5 6 7 7
18496 028/300 Flush
18497 029/300 Read block 661 len 24576
18498 030/300 Flush
18499 031/300 Read block 2720 len 16384
18500 032/300 Write block 1717 len 24576 data: 3 4 4 4 4 4
18501 033/300 Flush
18502 034/300 Read block 1140 len 20480
18503 035/300 Read block 1222 len 16384
18504 036/300 Read block 2713 len 36864
18505 037/300 Read block 4558 len 8192
18506 038/300 Write block 2790 len 16384 data: 4 3 3 5
18507 039/300 Write block 4558 len 40960 data: 3 3 3 3 2 3 4 4 4 5
18508 040/300 Read block 125 len 28672
18509 041/300 Read block 4314 len 8192
18510 042/300 Read block 1319 len 8192
18511 043/300 Read block 686 len 40960
18512 044/300 Write block 3098 len 20480 data: 3 2 2 3 3
18513 045/300 Flush
18514 046/300 Read block 835 len 24576
18515 047/300 Read block 4604 len 8192
18516 048/300 Write block 2369 len 24576 data: 4 4 5 4 4 4
18517 049/300 Read block 2471 len 4096
18518 050/300 Write block 3523 len 28672 data: 4 4 4 4 4 4 4
18519 051/300 Read block 1593 len 24576
18520 052/300 Read block 3058 len 4096
18521 053/300 Read block 3404 len 32768
18522 054/300 Write block 3925 len 36864 data: 2 2 2 3 2 2 2 2 3
18523 055/300 Read block 292 len 36864
18524 056/300 Write block 3024 len 4096 data: 3
18525 057/300 Write block 1881 len 4096 data: 6
18526 058/300 Read block 2662 len 24576
18527 059/300 Flush
18528 060/300 Read block 2925 len 16384
18529 061/300 Write block 3975 len 32768 data: 6 5 6 6 6 5 5 5
18530 062/300 Write block 1492 len 4096 data: 3
18531 063/300 Write block 3222 len 28672 data: 5 5 4 4 3 3 6
18532 064/300 Read block 3991 len 20480
18533 065/300 Read block 3493 len 8192
18534 066/300 Read block 1935 len 24576
18535 067/300 Write block 1570 len 28672 data: 6 5 4 3 3 3 3
18536 068/300 Write block 1915 len 36864 data: 3 2 2 3 3 3 4 4 4
18537 069/300 Read block 137 len 16384
18538 070/300 Read block 1271 len 28672
18539 071/300 Write block 4142 len 4096 data: 5
18540 072/300 Write block 2683 len 24576 data: 2 2 2 3 4 3
18541 073/300 Read block 4789 len 36864
18542 074/300 Write block 1306 len 24576 data: 3 3 2 3 3 3
18543 075/300 Read block 3061 len 40960
18544 076/300 Write block 3955 len 32768 data: 5 3 3 4 3 4 2 2
18545 077/300 Read block 1820 len 28672
18546 078/300 Write block 1333 len 20480 data: 4 4 4 4 4
18547 079/300 Read block 3841 len 32768
18548 080/300 Write block 1050 len 16384 data: 4 3 3 2
18549 081/300 Read block 2513 len 12288
18550 082/300 Read block 839 len 24576
18551 083/300 Read block 3960 len 4096
18552 084/300 Flush
18553 085/300 Read block 122 len 32768
18554 086/300 Read block 4318 len 16384
18555 087/300 Flush
18556 088/300 Read block 30 len 4096
18557 089/300 Write block 1087 len 28672 data: 4 3 3 4 4 5 5
18558 090/300 Read block 163 len 16384
18559 091/300 Read block 1249 len 12288
18560 092/300 Write block 560 len 36864 data: 4 2 2 2 2 2 2 2 2
18561 093/300 Write block 3234 len 28672 data: 7 8 8 8 3 3 3
18562 094/300 Write block 2170 len 12288 data: 4 5 5
18563 095/300 Read block 4864 len 36864
18564 096/300 Flush
18565 097/300 Write block 4173 len 32768 data: 5 5 4 6 4 4 4 4
18566 098/300 Write block 2758 len 20480 data: 7 8 8 7 7
18567 099/300 Read block 4638 len 12288
18568 100/300 Read block 834 len 24576
18569 101/300 Write block 4468 len 8192 data: 3 4
18570 102/300 Read block 4396 len 8192
18571 103/300 Flush
18572 104/300 Read block 3889 len 8192
18573 105/300 Write block 791 len 32768 data: 6 7 7 7 7 8 7 6
18574 106/300 Read block 1764 len 16384
18575 107/300 Write block 3144 len 24576 data: 4 4 4 5 5 4
18576 108/300 Write block 1835 len 28672 data: 3 4 3 5 4 5 6
18577 109/300 Read block 152 len 8192
18578 110/300 Write block 316 len 16384 data: 4 3 3 3
18579 111/300 Read block 1801 len 20480
18580 112/300 Write block 2599 len 24576 data: 3 4 5 5 5 5
18581 113/300 Read block 3293 len 16384
18582 114/300 Write block 4848 len 40960 data: 3 2 2 3 3 3 3 3 4 5
18583 115/300 Flush
18584 116/300 Read block 4841 len 24576
18585 117/300 Flush
18586 118/300 Write block 551 len 12288 data: 6 7 6
18587 119/300 Read block 2438 len 4096
18588 120/300 Write block 1138 len 16384 data: 3 2 2 2
18589 121/300 Read block 4751 len 16384
18590 122/300 Read block 513 len 20480
18591 123/300 Write block 194 len 20480 data: 5 5 6 5 4
18592 124/300 Read block 4482 len 24576
18593 125/300 Flush
18594 126/300 Read block 1645 len 24576
18595 127/300 Read block 11 len 8192
18596 128/300 Read block 4262 len 16384
18597 129/300 Write block 1768 len 24576 data: 3 4 4 4 3 5
18598 130/300 Write block 4493 len 16384 data: 4 3 3 5
18599 131/300 Write block 3257 len 20480 data: 2 2 3 4 3
18600 132/300 Write block 2110 len 12288 data: 5 6 6
18601 133/300 Write block 1429 len 16384 data: 4 3 3 4
18602 134/300 Read block 2044 len 16384
18603 135/300 Write block 2039 len 8192 data: 4 4
18604 136/300 Read block 1797 len 32768
18605 137/300 Write block 958 len 8192 data: 5 3
18606 138/300 Flush
18607 139/300 Write block 1620 len 20480 data: 4 6 7 7 8
18608 140/300 Flush
18609 141/300 Read block 4182 len 4096
18610 142/300 Write block 399 len 36864 data: 4 4 4 4 4 5 4 4 4
18611 143/300 Flush
18612 144/300 Flush
18613 145/300 Write block 4360 len 28672 data: 5 4 5 5 5 5 5
18614 146/300 Flush
18615 147/300 Write block 1659 len 40960 data: 3 3 3 2 2 2 2 2 2 2
18616 148/300 Write block 3889 len 28672 data: 3 3 3 3 3 3 3
18617 149/300 Flush
18618 150/300 Write block 2237 len 28672 data: 4 3 2 3 4 4 4
18619 151/300 Write block 2310 len 12288 data: 6 6 5
18620 152/300 Read block 1977 len 36864
18621 153/300 Write block 284 len 8192 data: 5 6
18622 154/300 Read block 1860 len 16384
18623 155/300 Write block 1171 len 40960 data: 7 8 7 6 6 6 4 3 3 2
18624 156/300 Read block 502 len 4096
18625 157/300 Read block 729 len 16384
18626 158/300 Read block 1334 len 4096
18627 159/300 Write block 2856 len 36864 data: 4 4 4 4 4 3 3 3 3
18628 160/300 Read block 2496 len 28672
18629 161/300 Read block 4017 len 32768
18630 162/300 Read block 950 len 4096
18631 163/300 Read block 2039 len 8192
18632 164/300 Read block 2693 len 36864
18633 165/300 Write block 1556 len 24576 data: 3 3 4 5 5 4
18634 166/300 Write block 3327 len 4096 data: 4
18635 167/300 Read block 256 len 8192
18636 168/300 Read block 1920 len 32768
18637 169/300 Read block 875 len 12288
18638 170/300 Flush
18639 171/300 Write block 1291 len 16384 data: 3 3 4 4
18640 172/300 Read block 601 len 28672
18641 173/300 Write block 2809 len 4096 data: 6
18642 174/300 Write block 2873 len 28672 data: 3 3 3 3 3 3 2
18643 175/300 Flush
18644 176/300 Read block 2062 len 20480
18645 177/300 Read block 3685 len 24576
18646 178/300 Read block 4897 len 16384
18647 179/300 Read block 3607 len 28672
18648 180/300 Write block 453 len 32768 data: 3 5 5 5 5 4 4 4
18649 181/300 Read block 3046 len 12288
18650 182/300 Read block 3323 len 28672
18651 183/300 Write block 4515 len 12288 data: 3 3 4
18652 184/300 Read block 2543 len 28672
18653 185/300 Write block 1361 len 40960 data: 6 5 5 5 6 4 5 4 7 6
18654 186/300 Read block 4345 len 16384
18655 187/300 Read block 4374 len 32768
18656 188/300 Flush
18657 189/300 Read block 4537 len 40960
18658 190/300 Write block 2069 len 16384 data: 4 3 4 4
18659 191/300 Read block 1137 len 40960
18660 192/300 Read block 2435 len 4096
18661 193/300 Read block 3479 len 4096
18662 194/300 Read block 4783 len 16384
18663 195/300 Read block 2858 len 20480
18664 196/300 Read block 571 len 28672
18665 197/300 Write block 1616 len 24576 data: 3 4 3 3 5 7
18666 198/300 Write block 664 len 16384 data: 5 4 4 4
18667 199/300 Write block 4320 len 40960 data: 3 3 3 4 4 4 4 4 4 6
18668 200/300 Flush
18669 201/300 Write block 1524 len 16384 data: 3 3 3 4
18670 202/300 Write block 1401 len 32768 data: 4 4 4 4 3 3 3 2
18671 203/300 Read block 3140 len 4096
18672 204/300 Read block 2184 len 28672
18673 205/300 Read block 2042 len 20480
18674 206/300 Read block 4513 len 20480
18675 207/300 Read block 2965 len 32768
18676 208/300 Read block 4825 len 16384
18677 209/300 Read block 2392 len 20480
18678 210/300 Read block 2470 len 12288
18679 211/300 Write block 649 len 16384 data: 4 4 4 4
18680 212/300 Write block 2452 len 20480 data: 5 5 6 7 7
18681 213/300 Read block 3489 len 40960
18682 214/300 Read block 4876 len 16384
18683 215/300 Write block 2923 len 24576 data: 4 4 5 5 5 5
18684 216/300 Read block 863 len 24576
18685 217/300 Flush
18686 218/300 Read block 1173 len 16384
18687 219/300 Write block 4277 len 36864 data: 6 5 4 5 6 6 5 5 3
18688 220/300 Write block 2075 len 20480 data: 2 2 2 2 3
18689 221/300 Read block 4652 len 24576
18690 222/300 Write block 2049 len 28672 data: 2 3 4 4 4 4 4
18691 223/300 Read block 534 len 16384
18692 224/300 Write block 3022 len 36864 data: 3 3 4 3 2 2 2 2 3
18693 225/300 Read block 842 len 36864
18694 226/300 Read block 1893 len 12288
18695 227/300 Read block 1476 len 8192
18696 228/300 Read block 4692 len 4096
18697 229/300 Write block 1841 len 24576 data: 7 7 5 4 4 4
18698 230/300 Write block 2457 len 8192 data: 7 6
18699 231/300 Read block 2173 len 8192
18700 232/300 Read block 4521 len 20480
18701 233/300 Read block 752 len 8192
18702 234/300 Write block 1052 len 20480 data: 4 3 2 2 2
18703 235/300 Read block 4601 len 12288
18704 236/300 Read block 3139 len 4096
18705 237/300 Write block 1494 len 20480 data: 2 3 4 5 4
18706 238/300 Write block 3013 len 20480 data: 4 4 4 5 6
18707 239/300 Read block 4047 len 16384
18708 240/300 Write block 2936 len 28672 data: 2 2 2 3 3 4 4
18709 241/300 Write block 3639 len 28672 data: 3 4 4 6 6 5 4
18710 242/300 Flush
18711 243/300 Write block 3930 len 40960 data: 3 3 3 4 4 4 8 8 8 8
18712 244/300 Flush
18713 245/300 Flush
18714 246/300 Write block 1038 len 8192 data: 2 3
18715 247/300 Read block 3940 len 40960
18716 248/300 Read block 221 len 28672
18717 249/300 Write block 2675 len 24576 data: 4 4 3 2 3 3
18718 250/300 Read block 4367 len 40960
18719 251/300 Flush
18720 252/300 Read block 3084 len 4096
18721 253/300 Read block 3409 len 36864
18722 254/300 Write block 3731 len 24576 data: 3 5 5 4 4 4
18723 255/300 Read block 4187 len 16384
18724 256/300 Read block 2588 len 32768
18725 257/300 Read block 1353 len 20480
18726 258/300 Flush
18727 259/300 Read block 4466 len 8192
18728 260/300 Read block 305 len 32768
18729 261/300 Flush
18730 262/300 Write block 2725 len 32768 data: 3 3 4 4 4 4 6 6
18731 263/300 Write block 686 len 36864 data: 2 3 4 5 5 5 5 4 4
18732 264/300 Read block 8 len 28672
18733 265/300 Read block 4565 len 40960
18734 266/300 Read block 4232 len 32768
18735 267/300 Write block 4303 len 32768 data: 3 3 4 4 3 4 4 3
18736 268/300 Write block 4050 len 12288 data: 5 4 3
18737 269/300 Write block 2632 len 40960 data: 4 4 3 5 5 6 7 7 4 4
18738 270/300 Read block 4585 len 20480
18739 271/300 Write block 2960 len 16384 data: 4 5 5 5
18740 272/300 Write block 4931 len 40960 data: 3 3 3 3 3 3 2 2 3 3
18741 273/300 Read block 3602 len 8192
18742 274/300 Write block 32 len 20480 data: 3 3 3 4 4
18743 275/300 Read block 2101 len 16384
18744 276/300 Flush
18745 277/300 Read block 33 len 20480
18746 278/300 Read block 1804 len 24576
18747 279/300 Write block 1715 len 24576 data: 4 3 4 5 5 5
18748 280/300 Read block 1762 len 8192
18749 281/300 Write block 1803 len 40960 data: 2 3 5 5 4 4 4 4 4 4
18750 282/300 Flush
18751 283/300 Read block 1844 len 20480
18752 284/300 Read block 4506 len 20480
18753 285/300 Write block 2248 len 8192 data: 3 2
18754 286/300 Read block 1546 len 16384
18755 287/300 Write block 907 len 32768 data: 3 3 4 4 4 4 5 5
18756 288/300 Read block 1173 len 40960
18757 289/300 Read block 253 len 12288
18758 290/300 Read block 1563 len 36864
18759 291/300 Flush
18760 292/300 Flush
18761 293/300 Write block 3573 len 28672 data: 5 6 7 8 8 8 5
18762 294/300 Write block 2801 len 4096 data: 3
18763 295/300 Read block 3340 len 36864
18764 296/300 Write block 2247 len 32768 data: 3 4 3 4 3 5 5 4
18765 297/300 Write block 1459 len 20480 data: 3 3 3 2 2
18766 298/300 Read block 253 len 4096
18767 299/300 Write block 4197 len 40960 data: 5 6 6 7 7 7 7 6 7 6
18768 300/300 Flush
18769 301/300 Flush
18770 Main task has work to do, go find it
18771 got dsc Start(1)
18772 start 1
18773 Jan 29 05:14:51.734 INFO request completed, latency_us: 181, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7b211da5-480e-4833-88e6-cd5808d2430b, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
18774 Replay: started 1, returned:Ok(())
18775 ----------------------------------------------------------------
18776 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18777 States: Active Offline Active
18778 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18779 5832 Acked Read 6 Done Sent Done false
18780 5833 Acked Read 8 Done Sent Done false
18781 5834 Acked Read 6 Done Sent Done false
18782 5835 Acked Flush 0 Done Sent Done false
18783 5836 Acked Read 3 Done Sent Done false
18784 5837 Acked Write 7 Done Sent Done false
18785 5838 Acked Read 7 Done Sent Done false
18786 5839 Acked Read 4 Done Sent Done false
18787 5840 Acked Read 9 Done Sent Done false
18788 5841 Acked Read 3 Done Sent Done false
18789 5842 Acked Read 2 Done Sent Done false
18790 5843 Acked Write 6 Done Sent Done false
18791 5844 Acked Read 1 Done Sent Done false
18792 5845 Acked Read 3 Done Sent Done false
18793 5846 Acked Read 8 Done Sent Done false
18794 5847 Acked Read 4 Done Sent Done false
18795 5848 Acked Write 2 Done Sent Done false
18796 5849 Acked Write 1 Done Sent Done false
18797 5850 Acked Read 2 Done Sent Done false
18798 5851 Acked Read 10 Done Sent Done false
18799 5852 Acked Write 4 Done Sent Done false
18800 5853 Acked Read 2 Done Sent Done false
18801 5854 Acked Read 3 Done Sent Done false
18802 5855 Acked Write 9 Done Sent Done false
18803 5856 Acked Flush 0 Done Sent Done false
18804 5857 Acked Write 9 Done Sent Done false
18805 5858 Acked Write 9 Done Sent Done false
18806 5859 Acked Flush 0 Done Sent Done false
18807 5860 Acked Read 6 Done Sent Done false
18808 5861 Acked Flush 0 Done Sent Done false
18809 5862 Acked Read 4 Done Sent Done false
18810 5863 Acked Write 6 Done Sent Done false
18811 5864 Acked Flush 0 Done Sent Done false
18812 5865 Acked Read 5 Done Sent Done false
18813 5866 Acked Read 4 Done Sent Done false
18814 5867 Acked Read 9 Done Sent Done false
18815 5868 Acked Read 2 Done Sent Done false
18816 5869 Acked Write 4 Done Sent Done false
18817 5870 Acked Write 10 Done Sent Done false
18818 5871 Acked Read 7 Done Sent Done false
18819 5872 Acked Read 2 Done Sent Done false
18820 5873 Acked Read 2 Done Sent Done false
18821 5874 Acked Read 10 Done Sent Done false
18822 5875 Acked Write 5 Done Sent Done false
18823 5876 Acked Flush 0 Done Sent Done false
18824 5877 Acked Read 6 Done Sent Done false
18825 5878 Acked Read 2 Done Sent Done false
18826 5879 Acked Write 6 Done Sent Done false
18827 5880 Acked Read 1 Done Sent Done false
18828 5881 Acked Write 7 Done Sent Done false
18829 5882 Acked Read 6 Done Sent Done false
18830 5883 Acked Read 1 Done Sent Done false
18831 5884 Acked Read 8 Done Sent Done false
18832 5885 Acked Write 9 Done Sent Done false
18833 5886 Acked Read 9 Done Sent Done false
18834 5887 Acked Write 1 Done Sent Done false
18835 5888 Acked Write 1 Done Sent Done false
18836 5889 Acked Read 6 Done Sent Done false
18837 5890 Acked Flush 0 Done Sent Done false
18838 5891 Acked Read 4 Done Sent Done false
18839 5892 Acked Write 8 Done Sent Done false
18840 5893 Acked Write 1 Done Sent Done false
18841 5894 Acked Write 7 Done Sent Done false
18842 5895 Acked Read 5 Done Sent Done false
18843 5896 Acked Read 2 Done Sent Done false
18844 5897 Acked Read 6 Done Sent Done false
18845 5898 Acked Write 7 Done Sent Done false
18846 5899 Acked Write 9 Done Sent Done false
18847 5900 Acked Read 4 Done Sent Done false
18848 5901 Acked Read 7 Done Sent Done false
18849 5902 Acked Write 1 Done Sent Done false
18850 5903 Acked Write 6 Done Sent Done false
18851 5904 Acked Read 9 Done Sent Done false
18852 5905 Acked Write 6 Done Sent Done false
18853 5906 Acked Read 10 Done Sent Done false
18854 5907 Acked Write 8 Done Sent Done false
18855 5908 Acked Read 7 Done Sent Done false
18856 5909 Acked Write 5 Done Sent Done false
18857 5910 Acked Read 8 Done Sent Done false
18858 5911 Acked Write 4 Done Sent Done false
18859 5912 Acked Read 3 Done Sent Done false
18860 5913 Acked Read 6 Done Sent Done false
18861 5914 Acked Read 1 Done Sent Done false
18862 5915 Acked Flush 0 Done Sent Done false
18863 5916 Acked Read 8 Done Sent Done false
18864 5917 Acked Read 4 Done Sent Done false
18865 5918 Acked Flush 0 Done Sent Done false
18866 5919 Acked Read 1 Done Sent Done false
18867 5920 Acked Write 7 Done Sent Done false
18868 5921 Acked Read 4 Done Sent Done false
18869 5922 Acked Read 3 Done Sent Done false
18870 5923 Acked Write 9 Done Sent Done false
18871 5924 Acked Write 7 Done Sent Done false
18872 5925 Acked Write 3 Done Sent Done false
18873 5926 Acked Read 9 Done Sent Done false
18874 5927 Acked Flush 0 Done Sent Done false
18875 5928 Acked Write 8 Done Sent Done false
18876 5929 Acked Write 5 Done Sent Done false
18877 5930 Acked Read 3 Done Sent Done false
18878 5931 Acked Read 6 Done Sent Done false
18879 5932 Acked Write 2 Done Sent Done false
18880 5933 Acked Read 2 Done Sent Done false
18881 5934 Acked Flush 0 Done Sent Done false
18882 5935 Acked Read 2 Done Sent Done false
18883 5936 Acked Write 8 Done Sent Done false
18884 5937 Acked Read 4 Done Sent Done false
18885 5938 Acked Write 6 Done Sent Done false
18886 5939 Acked Write 7 Done Sent Done false
18887 5940 Acked Read 2 Done Sent Done false
18888 5941 Acked Write 4 Done Sent Done false
18889 5942 Acked Read 5 Done Sent Done false
18890 5943 Acked Write 6 Done Sent Done false
18891 5944 Acked Read 4 Done Sent Done false
18892 5945 Acked Write 10 Done Sent Done false
18893 5946 Acked Flush 0 Done Sent Done false
18894 5947 Acked Read 6 Done Sent Done false
18895 5948 Acked Flush 0 Done Sent Done false
18896 5949 Acked Write 3 Done Sent Done false
18897 5950 Acked Read 1 Done Sent Done false
18898 5951 Acked Write 4 Done Sent Done false
18899 5952 Acked Read 4 Done Sent Done false
18900 5953 Acked Read 5 Done Sent Done false
18901 5954 Acked Write 5 Done Sent Done false
18902 5955 Acked Read 6 Done Sent Done false
18903 5956 Acked Flush 0 Done Sent Done false
18904 5957 Acked Read 6 Done Sent Done false
18905 5958 Acked Read 2 Done Sent Done false
18906 5959 Acked Read 4 Done Sent Done false
18907 5960 Acked Write 6 Done Sent Done false
18908 5961 Acked Write 4 Done Sent Done false
18909 5962 Acked Write 5 Done Sent Done false
18910 5963 Acked Write 3 Done Sent Done false
18911 5964 Acked Write 4 Done Sent Done false
18912 5965 Acked Read 4 Done Sent Done false
18913 5966 Acked Write 2 Done Sent Done false
18914 5967 Acked Read 8 Done Sent Done false
18915 5968 Acked Write 2 Done Sent Done false
18916 5969 Acked Flush 0 Done Sent Done false
18917 5970 Acked Write 5 Done Sent Done false
18918 5971 Acked Flush 0 Done Sent Done false
18919 5972 Acked Read 1 Done Sent Done false
18920 5973 Acked Write 9 Done Sent Done false
18921 5974 Acked Flush 0 Done Sent Done false
18922 5975 Acked Flush 0 Done Sent Done false
18923 5976 Acked Write 7 Done Sent Done false
18924 5977 Acked Flush 0 Done Sent Done false
18925 5978 Acked Write 10 Done Sent Done false
18926 5979 Acked Write 7 Done Sent Done false
18927 5980 Acked Flush 0 Done Sent Done false
18928 5981 Acked Write 7 Done Sent Done false
18929 5982 Acked Write 3 Done Sent Done false
18930 5983 Acked Read 9 Done Sent Done false
18931 5984 Acked Write 2 Done Sent Done false
18932 5985 Acked Read 4 Done Sent Done false
18933 5986 Acked Write 10 Done Sent Done false
18934 5987 Acked Read 1 Done Sent Done false
18935 5988 Acked Read 4 Done Sent Done false
18936 5989 Acked Read 1 Done Sent Done false
18937 5990 Acked Write 9 Done Sent Done false
18938 5991 Acked Read 7 Done Sent Done false
18939 5992 Acked Read 8 Done Sent Done false
18940 5993 Acked Read 1 Done Sent Done false
18941 5994 Acked Read 2 Done Sent Done false
18942 5995 Acked Read 9 Done Sent Done false
18943 5996 Acked Write 6 Done Sent Done false
18944 5997 Acked Write 1 Done Sent Done false
18945 5998 Acked Read 2 Done Sent Done false
18946 5999 Acked Read 8 Done Sent Done false
18947 6000 Acked Read 3 Done Sent Done false
18948 6001 Acked Flush 0 Done Sent Done false
18949 6002 Acked Write 4 Done Sent Done false
18950 6003 Acked Read 7 Done Sent Done false
18951 6004 Acked Write 1 Done Sent Done false
18952 6005 Acked Write 7 Done Sent Done false
18953 6006 Acked Flush 0 Done Sent Done false
18954 6007 Acked Read 5 Done Sent Done false
18955 6008 Acked Read 6 Done Sent Done false
18956 6009 Acked Read 4 Done Sent Done false
18957 6010 Acked Read 7 Done Sent Done false
18958 6011 Acked Write 8 Done Sent Done false
18959 6012 Acked Read 3 Done Sent Done false
18960 6013 Acked Read 7 Done Sent Done false
18961 6014 Acked Write 3 Done Sent Done false
18962 6015 Acked Read 7 Done Sent Done false
18963 6016 Acked Write 10 Done Sent Done false
18964 6017 Acked Read 4 Done Sent Done false
18965 6018 Acked Read 8 Done Sent Done false
18966 6019 Acked Flush 0 Done Sent Done false
18967 6020 Acked Read 10 Done Sent Done false
18968 6021 Acked Write 4 Done Sent Done false
18969 6022 Acked Read 10 Done Sent Done false
18970 6023 Acked Read 1 Done Sent Done false
18971 6024 Acked Read 1 Done Sent Done false
18972 6025 Acked Read 4 Done Sent Done false
18973 6026 Acked Read 5 Done Sent Done false
18974 6027 Acked Read 7 Done Sent Done false
18975 6028 Acked Write 6 Done Sent Done false
18976 6029 Acked Write 4 Done Sent Done false
18977 6030 Acked Write 10 Done Sent Done false
18978 6031 Acked Flush 0 Done Sent Done false
18979 6032 Acked Write 4 Done Sent Done false
18980 6033 Acked Write 8 Done Sent Done false
18981 6034 Acked Read 1 Done Sent Done false
18982 6035 Acked Read 7 Done Sent Done false
18983 6036 Acked Read 5 Done Sent Done false
18984 6037 Acked Read 5 Done Sent Done false
18985 6038 Acked Read 8 Done Sent Done false
18986 6039 Acked Read 4 Done Sent Done false
18987 6040 Acked Read 5 Done Sent Done false
18988 6041 Acked Read 3 Done Sent Done false
18989 6042 Acked Write 4 Done Sent Done false
18990 6043 Acked Write 5 Done Sent Done false
18991 6044 Acked Read 10 Done Sent Done false
18992 6045 Acked Read 4 Done Sent Done false
18993 6046 Acked Write 6 Done Sent Done false
18994 6047 Acked Read 6 Done Sent Done false
18995 6048 Acked Flush 0 Done Sent Done false
18996 6049 Acked Read 4 Done Sent Done false
18997 6050 Acked Write 9 Done Sent Done false
18998 6051 Acked Write 5 Done Sent Done false
18999 6052 Acked Read 6 Done Sent Done false
19000 6053 Acked Write 7 Done Sent Done false
19001 6054 Acked Read 4 Done Sent Done false
19002 6055 Acked Write 9 Done Sent Done false
19003 6056 Acked Read 9 Done Sent Done false
19004 6057 Acked Read 3 Done Sent Done false
19005 6058 Acked Read 2 Done Sent Done false
19006 6059 Acked Read 1 Done Sent Done false
19007 6060 Acked Write 6 Done Sent Done false
19008 6061 Acked Write 2 Done Sent Done false
19009 6062 Acked Read 2 Done Sent Done false
19010 6063 Acked Read 5 Done Sent Done false
19011 6064 Acked Read 2 Done Sent Done false
19012 6065 Acked Write 5 Done Sent Done false
19013 6066 Acked Read 3 Done Sent Done false
19014 6067 Acked Read 1 Done Sent Done false
19015 6068 Acked Write 5 Done Sent Done false
19016 6069 Acked Write 5 Done Sent Done false
19017 6070 Acked Read 4 Done Sent Done false
19018 6071 Acked Write 7 Done Sent Done false
19019 6072 Acked Write 7 Done Sent Done false
19020 6073 Acked Flush 0 Done Sent Done false
19021 6074 Acked Write 10 Done Sent Done false
19022 6075 Acked Flush 0 Done Sent Done false
19023 6076 Acked Flush 0 Done Sent Done false
19024 6077 Acked Write 2 Done Sent Done false
19025 6078 Acked Read 10 Done Sent Done false
19026 6079 Acked Read 7 Done Sent Done false
19027 6080 Acked Write 6 Done Sent Done false
19028 6081 Acked Read 10 Done Sent Done false
19029 6082 Acked Flush 0 Done Sent Done false
19030 6083 Acked Read 1 Done Sent Done false
19031 6084 Acked Read 9 Done Sent Done false
19032 6085 Acked Write 6 Done Sent Done false
19033 6086 Acked Read 4 Done Sent Done false
19034 6087 Acked Read 8 Done Sent Done false
19035 6088 Acked Read 5 Done Sent Done false
19036 6089 Acked Flush 0 Done Sent Done false
19037 6090 Acked Read 2 Done Sent Done false
19038 6091 Acked Read 8 Done Sent Done false
19039 6092 Acked Flush 0 Done Sent Done false
19040 6093 Acked Write 8 Done Sent Done false
19041 6094 Acked Write 9 Done Sent Done false
19042 6095 Acked Read 7 Done Sent Done false
19043 6096 Acked Read 10 Done Sent Done false
19044 6097 Acked Read 8 Done Sent Done false
19045 6098 Acked Write 8 Done Sent Done false
19046 6099 Acked Write 3 Done Sent Done false
19047 6100 Acked Write 10 Done Sent Done false
19048 6101 Acked Read 5 Done Sent Done false
19049 6102 Acked Write 4 Done Sent Done false
19050 6103 Acked Write 10 Done Sent Done false
19051 6104 Acked Read 2 Done Sent Done false
19052 6105 Acked Write 5 Done Sent Done false
19053 6106 Acked Read 4 Done Sent Done false
19054 6107 Acked Flush 0 Done Sent Done false
19055 6108 Acked Read 5 Done Sent Done false
19056 6109 Acked Read 6 Done Sent Done false
19057 6110 Acked Write 6 Done Sent Done false
19058 6111 Acked Read 2 Done Sent Done false
19059 6112 Acked Write 10 Done Sent Done false
19060 6113 Acked Flush 0 Done Sent Done false
19061 6114 Acked Read 5 Done Sent Done false
19062 6115 Acked Read 5 Done Sent Done false
19063 6116 Acked Write 2 Done Sent Done false
19064 6117 Acked Read 4 Done Sent Done false
19065 6118 Acked Write 8 Done Sent Done false
19066 6119 Acked Read 10 Done Sent Done false
19067 6120 Acked Read 3 Done Sent Done false
19068 6121 Acked Read 9 Done Sent Done false
19069 6122 Acked Flush 0 Done Sent Done false
19070 6123 Acked Flush 0 Done Sent Done false
19071 6124 Acked Write 7 Done Sent Done false
19072 6125 Acked Write 1 Done Sent Done false
19073 6126 Acked Read 9 Done Sent Done false
19074 6127 Acked Write 8 Done Sent Done false
19075 6128 Acked Write 5 Done Sent Done false
19076 6129 Acked Read 1 Done Sent Done false
19077 6130 Acked Write 10 Done Sent Done false
19078 6131 Acked Flush 0 Done Sent Done false
19079 6132 Acked Flush 0 Done Sent Done false
19080 STATES DS:0 DS:1 DS:2 TOTAL
19081 Sent 0 301 0 301
19082 Done 301 0 301 602
19083 Skipped 0 0 0 0
19084 Error 0 0 0 0
19085 Last Flush: Some(JobId(6132)) Some(JobId(5831)) Some(JobId(6132))
19086 Downstairs last five completed: 5831 5830 5829 5828 5827
19087 Upstairs last five completed: 6132 6131 6130 6129 6128
19088 CLIENT: Up:0 ds:301 act:2
190892025-01-29T05:14:52.329ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
190902025-01-29T05:14:53.331ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
19091 [8820] got start action
19092 [8820] I am going to restart
19093 Starting downstairs at port 8820
19094 Make output file at "/tmp/dsc/downstairs-8820.txt"
19095 [8820][1] reports Starting
19096 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1219)
19097 [8820][1] reports Running
190982025-01-29T05:14:54.333ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
19099 {"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":----------------------------------------------------------------
19100 40 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19101 States: Active Offline Active
19102 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19103 5832 Acked Read 6 Done Sent Done false
19104 5833 Acked Read 8 Done Sent Done false
19105 5834 Acked Read 6 Done Sent Done false
19106 5835 Acked Flush 0, Done" Senttime" Done: false"
19107 2025-01-29T05:14:55.737093708Z" 5836 Acked Read 3," Donehostname Sent" Done:" false
19108 w-01JJR7DDKGFJDPMCDR9NFSE52V"," 5837 Acked Write 7pid Done": Sent Done1202 false
19109 ," 5838 Acked Read 7session_id Done" Sent:" Done false
19110 3b9f53a2-1179-49c9-b40f-5354d75015be" 5839 Acked Read 4} Done
19111 Sent Done false
19112 5840 Acked Read 9 Done Sent Done false
19113 5841 Acked Read 3 Done Sent Done false
19114 5842 Acked Read 2 Done Sent Done false
19115 5843 Acked Write 6 Done Sent Done false
19116 5844 Acked Read 1 Done Sent Done false
19117 5845 Acked Read 3 Done Sent Done false
19118 5846 Acked Read 8 Done Sent Done false
19119 5847 Acked Read 4 Done Sent Done false
19120 5848 Acked Write 2 Done Sent Done false
19121 5849 Acked Write 1 Done Sent Done false
19122 5850 Acked Read 2 Done Sent Done false
19123 5851 Acked Read 10 Done Sent Done false
19124 5852 Acked Write 4 Done Sent Done false
19125 5853 Acked Read 2 Done Sent Done false
19126 5854 Acked Read 3 Done Sent Done false
19127 5855 Acked Write 9 Done Sent Done false
19128 5856 Acked Flush 0 Done Sent Done false
19129 5857 Acked Write 9 Done Sent Done false
19130 5858 Acked Write 9 Done Sent Done false
19131 5859 Acked Flush 0 Done Sent Done false
19132 5860 Acked Read 6 Done Sent Done false
19133 5861 Acked Flush 0 Done Sent Done false
19134 5862 Acked Read 4 Done Sent Done false
19135 5863 Acked Write 6 Done Sent Done false
19136 5864 Acked Flush 0 Done Sent Done false
19137 5865 Acked Read 5 Done Sent Done false
19138 5866 Acked Read 4 Done Sent Done false
19139 5867 Acked Read 9 Done Sent Done false
19140 5868 Acked Read 2 Done Sent Done false
19141 5869 Acked Write 4 Done Sent Done false
19142 5870 Acked Write 10 Done Sent Done false
19143 5871 Acked Read 7 Done Sent Done false
19144 5872 Acked Read 2 Done Sent Done false
19145 5873 Acked Read 2 Done Sent Done false
19146 5874 Acked Read 10 Done Sent Done false
19147 5875 Acked Write 5 Done Sent Done false
19148 5876 Acked Flush 0 Done Sent Done false
19149 5877 Acked Read 6 Done Sent Done false
19150 5878 Acked Read 2 Done Sent Done false
19151 5879 Acked Write 6 Done Sent Done false
19152 5880 Acked Read 1 Done Sent Done false
19153 5881 Acked Write 7 Done Sent Done false
19154 5882 Acked Read 6 Done Sent Done false
19155 5883 Acked Read 1 Done Sent Done false
19156 5884 Acked Read 8 Done Sent Done false
19157 5885 Acked Write 9 Done Sent Done false
19158 5886 Acked Read 9 Done Sent Done false
19159 5887 Acked Write 1 Done Sent Done false
19160 5888 Acked Write 1 Done Sent Done false
19161 5889 Acked Read 6 Done Sent Done false
19162 5890 Acked Flush 0 Done Sent Done false
19163 5891 Acked Read 4 Done Sent Done false
19164 5892 Acked Write 8 Done Sent Done false
19165 5893 Acked Write 1 Done Sent Done false
19166 5894 Acked Write 7 Done Sent Done false
19167 5895 Acked Read 5 Done Sent Done false
19168 5896 Acked Read 2 Done Sent Done false
19169 5897 Acked Read 6 Done Sent Done false
19170 5898 Acked Write 7 Done Sent Done false
19171 5899 Acked Write 9 Done Sent Done false
19172 5900 Acked Read 4 Done Sent Done false
19173 5901 Acked Read 7 Done Sent Done false
19174 5902 Acked Write 1 Done Sent Done false
19175 5903 Acked Write 6 Done Sent Done false
19176 5904 Acked Read 9 Done Sent Done false
19177 5905 Acked Write 6 Done Sent Done false
19178 5906 Acked Read 10 Done Sent Done false
19179 5907 Acked Write 8 Done Sent Done false
19180 5908 Acked Read 7 Done Sent Done false
19181 5909 Acked Write 5 Done Sent Done false
19182 5910 Acked Read 8 Done Sent Done false
19183 5911 Acked Write 4 Done Sent Done false
19184 5912 Acked Read 3 Done Sent Done false
19185 5913 Acked Read 6 Done Sent Done false
19186 5914 Acked Read 1 Done Sent Done false
19187 5915 Acked Flush 0 Done Sent Done false
19188 5916 Acked Read 8 Done Sent Done false
19189 5917 Acked Read 4 Done Sent Done false
19190 5918 Acked Flush 0 Done Sent Done false
19191 5919 Acked Read 1 Done Sent Done false
19192 5920 Acked Write 7 Done Sent Done false
19193 5921 Acked Read 4 Done Sent Done false
19194 5922 Acked Read 3 Done Sent Done false
19195 5923 Acked Write 9 Done Sent Done false
19196 5924 Acked Write 7 Done Sent Done false
19197 5925 Acked Write 3 Done Sent Done false
19198 5926 Acked Read 9 Done Sent Done false
19199 5927 Acked Flush 0 Done Sent Done false
19200 5928 Acked Write 8 Done Sent Done false
19201 5929 Acked Write 5 Done Sent Done false
19202 5930 Acked Read 3 Done Sent Done false
19203 5931 Acked Read 6 Done Sent Done false
19204 5932 Acked Write 2 Done Sent Done false
19205 5933 Acked Read 2 Done Sent Done false
19206 5934 Acked Flush 0 Done Sent Done false
19207 5935 Acked Read 2 Done Sent Done false
19208 5936 Acked Write 8 Done Sent Done false
19209 5937 Acked Read 4 Done Sent Done false
19210 5938 Acked Write 6 Done Sent Done false
19211 5939 Acked Write 7 Done Sent Done false
19212 5940 Acked Read 2 Done Sent Done false
19213 5941 Acked Write 4 Done Sent Done false
19214 5942 Acked Read 5 Done Sent Done false
19215 5943 Acked Write 6 Done Sent Done false
19216 5944 Acked Read 4 Done Sent Done false
19217 5945 Acked Write 10 Done Sent Done false
19218 5946 Acked Flush 0 Done Sent Done false
19219 5947 Acked Read 6 Done Sent Done false
19220 5948 Acked Flush 0 Done Sent Done false
19221 5949 Acked Write 3 Done Sent Done false
19222 5950 Acked Read 1 Done Sent Done false
19223 5951 Acked Write 4 Done Sent Done false
19224 5952 Acked Read 4 Done Sent Done false
19225 5953 Acked Read 5 Done Sent Done false
19226 5954 Acked Write 5 Done Sent Done false
19227 5955 Acked Read 6 Done Sent Done false
19228 5956 Acked Flush 0 Done Sent Done false
19229 5957 Acked Read 6 Done Sent Done false
19230 5958 Acked Read 2 Done Sent Done false
19231 5959 Acked Read 4 Done Sent Done false
19232 5960 Acked Write 6 Done Sent Done false
19233 5961 Acked Write 4 Done Sent Done false
19234 5962 Acked Write 5 Done Sent Done false
19235 5963 Acked Write 3 Done Sent Done false
19236 5964 Acked Write 4 Done Sent Done false
19237 5965 Acked Read 4 Done Sent Done false
19238 5966 Acked Write 2 Done Sent Done false
19239 5967 Acked Read 8 Done Sent Done false
19240 5968 Acked Write 2 Done Sent Done false
19241 5969 Acked Flush 0 Done Sent Done false
19242 5970 Acked Write 5 Done Sent Done false
19243 5971 Acked Flush 0 Done Sent Done false
19244 5972 Acked Read 1 Done Sent Done false
19245 5973 Acked Write 9 Done Sent Done false
19246 5974 Acked Flush 0 Done Sent Done false
19247 5975 Acked Flush 0 Done Sent Done false
19248 5976 Acked Write 7 Done Sent Done false
19249 5977 Acked Flush 0 Done Sent Done false
19250 5978 Acked Write 10 Done Sent Done false
19251 5979 Acked Write 7 Done Sent Done false
19252 5980 Acked Flush 0 Done Sent Done false
19253 5981 Acked Write 7 Done Sent Done false
19254 5982 Acked Write 3 Done Sent Done false
19255 5983 Acked Read 9 Done Sent Done false
19256 5984 Acked Write 2 Done Sent Done false
19257 5985 Acked Read 4 Done Sent Done false
19258 5986 Acked Write 10 Done Sent Done false
19259 5987 Acked Read 1 Done Sent Done false
19260 5988 Acked Read 4 Done Sent Done false
19261 5989 Acked Read 1 Done Sent Done false
19262 5990 Acked Write 9 Done Sent Done false
19263 5991 Acked Read 7 Done Sent Done false
19264 5992 Acked Read 8 Done Sent Done false
19265 5993 Acked Read 1 Done Sent Done false
19266 5994 Acked Read 2 Done Sent Done false
19267 5995 Acked Read 9 Done Sent Done false
19268 5996 Acked Write 6 Done Sent Done false
19269 5997 Acked Write 1 Done Sent Done false
19270 5998 Acked Read 2 Done Sent Done false
19271 5999 Acked Read 8 Done Sent Done false
19272 6000 Acked Read 3 Done Sent Done false
19273 6001 Acked Flush 0 Done Sent Done false
19274 6002 Acked Write 4 Done Sent Done false
19275 6003 Acked Read 7 Done Sent Done false
19276 6004 Acked Write 1 Done Sent Done false
19277 6005 Acked Write 7 Done Sent Done false
19278 6006 Acked Flush 0 Done Sent Done false
19279 6007 Acked Read 5 Done Sent Done false
19280 6008 Acked Read 6 Done Sent Done false
19281 6009 Acked Read 4 Done Sent Done false
19282 6010 Acked Read 7 Done Sent Done false
19283 6011 Acked Write 8 Done Sent Done false
19284 6012 Acked Read 3 Done Sent Done false
19285 6013 Acked Read 7 Done Sent Done false
19286 6014 Acked Write 3 Done Sent Done false
19287 6015 Acked Read 7 Done Sent Done false
19288 6016 Acked Write 10 Done Sent Done false
19289 6017 Acked Read 4 Done Sent Done false
19290 6018 Acked Read 8 Done Sent Done false
19291 6019 Acked Flush 0 Done Sent Done false
19292 6020 Acked Read 10 Done Sent Done false
19293 6021 Acked Write 4 Done Sent Done false
19294 6022 Acked Read 10 Done Sent Done false
19295 6023 Acked Read 1 Done Sent Done false
19296 6024 Acked Read 1 Done Sent Done false
19297 6025 Acked Read 4 Done Sent Done false
19298 6026 Acked Read 5 Done Sent Done false
19299 6027 Acked Read 7 Done Sent Done false
19300 6028 Acked Write 6 Done Sent Done false
19301 6029 Acked Write 4 Done Sent Done false
19302 6030 Acked Write 10 Done Sent Done false
19303 6031 Acked Flush 0 Done Sent Done false
19304 6032 Acked Write 4 Done Sent Done false
19305 6033 Acked Write 8 Done Sent Done false
19306 6034 Acked Read 1 Done Sent Done false
19307 6035 Acked Read 7 Done Sent Done false
19308 6036 Acked Read 5 Done Sent Done false
19309 6037 Acked Read 5 Done Sent Done false
19310 6038 Acked Read 8 Done Sent Done false
19311 6039 Acked Read 4 Done Sent Done false
19312 6040 Acked Read 5 Done Sent Done false
19313 6041 Acked Read 3 Done Sent Done false
19314 6042 Acked Write 4 Done Sent Done false
19315 6043 Acked Write 5 Done Sent Done false
19316 6044 Acked Read 10 Done Sent Done false
19317 6045 Acked Read 4 Done Sent Done false
19318 6046 Acked Write 6 Done Sent Done false
19319 6047 Acked Read 6 Done Sent Done false
19320 6048 Acked Flush 0 Done Sent Done false
19321 6049 Acked Read 4 Done Sent Done false
19322 6050 Acked Write 9 Done Sent Done false
19323 6051 Acked Write 5 Done Sent Done false
19324 6052 Acked Read 6 Done Sent Done false
19325 6053 Acked Write 7 Done Sent Done false
19326 6054 Acked Read 4 Done Sent Done false
19327 6055 Acked Write 9 Done Sent Done false
19328 6056 Acked Read 9 Done Sent Done false
19329 6057 Acked Read 3 Done Sent Done false
19330 6058 Acked Read 2 Done Sent Done false
19331 6059 Acked Read 1 Done Sent Done false
19332 6060 Acked Write 6 Done Sent Done false
19333 6061 Acked Write 2 Done Sent Done false
19334 6062 Acked Read 2 Done Sent Done false
19335 6063 Acked Read 5 Done Sent Done false
19336 6064 Acked Read 2 Done Sent Done false
19337 6065 Acked Write 5 Done Sent Done false
19338 6066 Acked Read 3 Done Sent Done false
19339 6067 Acked Read 1 Done Sent Done false
19340 6068 Acked Write 5 Done Sent Done false
19341 6069 Acked Write 5 Done Sent Done false
19342 6070 Acked Read 4 Done Sent Done false
19343 6071 Acked Write 7 Done Sent Done false
19344 6072 Acked Write 7 Done Sent Done false
19345 6073 Acked Flush 0 Done Sent Done false
19346 6074 Acked Write 10 Done Sent Done false
19347 6075 Acked Flush 0 Done Sent Done false
19348 6076 Acked Flush 0 Done Sent Done false
19349 6077 Acked Write 2 Done Sent Done false
19350 6078 Acked Read 10 Done Sent Done false
19351 6079 Acked Read 7 Done Sent Done false
19352 6080 Acked Write 6 Done Sent Done false
19353 6081 Acked Read 10 Done Sent Done false
19354 6082 Acked Flush 0 Done Sent Done false
19355 6083 Acked Read 1 Done Sent Done false
19356 6084 Acked Read 9 Done Sent Done false
19357 6085 Acked Write 6 Done Sent Done false
19358 6086 Acked Read 4 Done Sent Done false
19359 6087 Acked Read 8 Done Sent Done false
19360 6088 Acked Read 5 Done Sent Done false
19361 6089 Acked Flush 0 Done Sent Done false
19362 6090 Acked Read 2 Done Sent Done false
19363 6091 Acked Read 8 Done Sent Done false
19364 6092 Acked Flush 0 Done Sent Done false
19365 6093 Acked Write 8 Done Sent Done false
19366 6094 Acked Write 9 Done Sent Done false
19367 6095 Acked Read 7 Done Sent Done false
19368 6096 Acked Read 10 Done Sent Done false
19369 6097 Acked Read 8 Done Sent Done false
19370 6098 Acked Write 8 Done Sent Done false
19371 6099 Acked Write 3 Done Sent Done false
19372 6100 Acked Write 10 Done Sent Done false
19373 6101 Acked Read 5 Done Sent Done false
19374 6102 Acked Write 4 Done Sent Done false
19375 6103 Acked Write 10 Done Sent Done false
19376 6104 Acked Read 2 Done Sent Done false
19377 6105 Acked Write 5 Done Sent Done false
19378 6106 Acked Read 4 Done Sent Done false
19379 6107 Acked Flush 0 Done Sent Done false
19380 6108 Acked Read 5 Done Sent Done false
19381 6109 Acked Read 6 Done Sent Done false
19382 6110 Acked Write 6 Done Sent Done false
19383 6111 Acked Read 2 Done Sent Done false
19384 6112 Acked Write 10 Done Sent Done false
19385 6113 Acked Flush 0 Done Sent Done false
19386 6114 Acked Read 5 Done Sent Done false
19387 6115 Acked Read 5 Done Sent Done false
19388 6116 Acked Write 2 Done Sent Done false
19389 6117 Acked Read 4 Done Sent Done false
19390 6118 Acked Write 8 Done Sent Done false
19391 6119 Acked Read 10 Done Sent Done false
19392 6120 Acked Read 3 Done Sent Done false
19393 6121 Acked Read 9 Done Sent Done false
19394 6122 Acked Flush 0 Done Sent Done false
19395 6123 Acked Flush 0 Done Sent Done false
19396 6124 Acked Write 7 Done Sent Done false
19397 6125 Acked Write 1 Done Sent Done false
19398 6126 Acked Read 9 Done Sent Done false
19399 6127 Acked Write 8 Done Sent Done false
19400 6128 Acked Write 5 Done Sent Done false
19401 6129 Acked Read 1 Done Sent Done false
19402 6130 Acked Write 10 Done Sent Done false
19403 6131 Acked Flush 0 Done Sent Done false
19404 6132 Acked Flush 0 Done Sent Done false
19405 STATES DS:0 DS:1 DS:2 TOTAL
19406 Sent 0 301 0 301
19407 Done 301 0 301 602
19408 Skipped 0 0 0 0
19409 Error 0 0 0 0
19410 Last Flush: Some(JobId(6132)) Some(JobId(5831)) Some(JobId(6132))
19411 Downstairs last five completed: 5831 5830 5829 5828 5827
19412 Upstairs last five completed: 6132 6131 6130 6129 6128
19413 CLIENT: Up:0 ds:301 act:2
194142025-01-29T05:14:56.336ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194152025-01-29T05:14:57.338ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194162025-01-29T05:14:57.493ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194172025-01-29T05:14:57.493ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38406) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194182025-01-29T05:14:57.494ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194192025-01-29T05:14:57.495ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194202025-01-29T05:14:57.495ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194212025-01-29T05:14:57.495ZINFOcrucible: send last flush ID to this DS: Some(JobId(5831)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194222025-01-29T05:14:57.495ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194232025-01-29T05:14:57.495ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5831)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194242025-01-29T05:14:57.495ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5831)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
19425 ----------------------------------------------------------------
19426 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19427 Downstairs last five completed: 6132 6131 6130 6129 6128
19428 Upstairs last five completed: 6132 6131 6130 6129 6128
19429 CLIENT: Up:0 ds:0 act:3
19430 Replay: All jobs finished
19431 Main task has work to do, go find it
19432 got dsc Stop(1)
19433 stop 1
19434 [8820] Got stop action so:false kr:false
19435 Jan 29 05:14:59.740 INFO request completed, latency_us: 425, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1d159614-81e4-4fc5-b771-396eb05652a7, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
194362025-01-29T05:14:59.741ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194372025-01-29T05:14:59.741ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
194382025-01-29T05:14:59.741ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
19439 Jan 29 05:14:59.741 INFO request completed, latency_us: 127, response_code: 200, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:14:59.741805498Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
19440 uri: /state/cid/1, method: GET, req_id: 81d6f19d-48cc-426f-ac06-f2834f9b9df5, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
19441 [8820] Exited with: ExitStatus(unix_wait_status(9))
19442 [8820][1] reports Exit
19443 Jan 29 05:15:03.743 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/1, method: GET, req_id: c7720804-ba8a-4a7c-acc0-2630ab5bc01b, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
19444 001/300 Read block 1344 len 16384
19445 002/300 Flush
19446 003/300 Write block 2312 len 8192 data: 6 5
19447 004/300 Write block 187 len 12288 data: 2 4 4
19448 005/300 Read block 2612 len 24576
19449 006/300 Write block 831 len 16384 data: 3 3 3 3
19450 007/300 Read block 1296 len 12288
19451 008/300 Write block 940 len 36864 data: 2 2 2 2 2 4 5 5 5
19452 009/300 Read block 4529 len 16384
19453 010/300 Write block 4267 len 40960 data: 2 2 3 3 2 3 3 4 4 5
19454 011/300 Read block 1756 len 16384
19455 012/300 Read block 3790 len 24576
19456 013/300 Read block 1703 len 20480
19457 014/300 Read block 4508 len 40960
19458 015/300 Write block 1566 len 36864 data: 4 5 5 5 7 6 5 4 4
19459 016/300 Write block 2197 len 4096 data: 2
19460 017/300 Write block 4233 len 16384 data: 3 4 4 4
19461 018/300 Read block 3320 len 28672
19462 019/300 Read block 4446 len 32768
19463 020/300 Write block 836 len 24576 data: 3 3 4 4 4 4
19464 021/300 Write block 3681 len 28672 data: 4 5 4 5 5 4 4
19465 022/300 Read block 4224 len 4096
19466 023/300 Read block 4115 len 20480
19467 024/300 Flush
19468 025/300 Read block 353 len 28672
19469 026/300 Write block 3831 len 36864 data: 5 5 4 5 6 5 3 3 3
19470 027/300 Read block 323 len 36864
19471 028/300 Flush
19472 029/300 Flush
19473 030/300 Read block 1967 len 36864
19474 031/300 Write block 1223 len 20480 data: 3 4 5 4 3
19475 032/300 Write block 3210 len 16384 data: 3 2 2 2
19476 033/300 Write block 109 len 4096 data: 3
19477 034/300 Write block 3715 len 36864 data: 3 4 3 3 4 4 3 3 4
19478 035/300 Write block 4553 len 24576 data: 4 5 6 6 4 4
19479 036/300 Read block 4049 len 12288
19480 037/300 Write block 3771 len 8192 data: 5 5
19481 038/300 Write block 772 len 32768 data: 4 3 4 4 4 5 9 8
19482 039/300 Flush
19483 040/300 Read block 547 len 12288
19484 041/300 Read block 2872 len 32768
19485 042/300 Read block 4169 len 12288
19486 043/300 Write block 582 len 16384 data: 5 3 3 3
19487 044/300 Read block 3723 len 36864
19488 045/300 Read block 4030 len 28672
19489 046/300 Flush
19490 047/300 Read block 3578 len 32768
19491 048/300 Write block 2935 len 8192 data: 2 3
19492 049/300 Read block 822 len 24576
19493 050/300 Write block 2416 len 8192 data: 5 4
19494 051/300 Write block 1832 len 40960 data: 4 2 3 4 5 4 6 5 6 8
19495 052/300 Read block 4855 len 8192
19496 053/300 Write block 4370 len 8192 data: 5 5
19497 054/300 Write block 2288 len 36864 data: 3 3 3 3 3 4 4 4 5
19498 055/300 Write block 3915 len 8192 data: 2 2
19499 056/300 Read block 1967 len 16384
19500 057/300 Write block 3667 len 16384 data: 3 3 3 3
19501 058/300 Read block 4189 len 4096
19502 059/300 Read block 424 len 12288
19503 060/300 Read block 4806 len 20480
19504 061/300 Write block 1369 len 12288 data: 8 7 6
19505 062/300 Write block 2708 len 4096 data: 3
19506 063/300 Flush
19507 064/300 Read block 4527 len 20480
19508 065/300 Read block 4208 len 20480
19509 066/300 Read block 2394 len 40960
19510 067/300 Read block 2906 len 40960
19511 068/300 Read block 4434 len 24576
19512 069/300 Write block 4066 len 36864 data: 3 3 2 2 2 2 3 3 4
19513 070/300 Read block 1936 len 32768
19514 071/300 Read block 1323 len 8192
19515 072/300 Write block 1689 len 40960 data: 8 10 10 6 7 4 5 5 5 5
19516 073/300 Read block 2700 len 12288
19517 074/300 Read block 3501 len 36864
19518 075/300 Write block 2359 len 16384 data: 3 3 3 3
19519 076/300 Read block 3695 len 28672
19520 077/300 Write block 3711 len 28672 data: 3 3 3 3 4 5 4
19521 078/300 Write block 2607 len 24576 data: 5 6 6 5 5 5
19522 079/300 Write block 2419 len 8192 data: 3 3
19523 080/300 Write block 3876 len 20480 data: 4 5 5 4 4
19524 081/300 Read block 2115 len 12288
19525 082/300 Read block 2233 len 16384
19526 083/300 Write block 2546 len 4096 data: 4
19527 084/300 Read block 1536 len 12288
19528 085/300 Read block 882 len 40960
19529 086/300 Write block 135 len 20480 data: 4 4 5 4 5
19530 087/300 Read block 910 len 4096
19531 088/300 Read block 3977 len 4096
19532 089/300 Read block 174 len 12288
19533 090/300 Write block 590 len 24576 data: 5 4 4 4 5 5
19534 091/300 Flush
19535 092/300 Read block 2995 len 32768
19536 093/300 Write block 4073 len 24576 data: 4 5 4 4 3 4
19537 094/300 Write block 3360 len 24576 data: 3 3 3 3 4 2
19538 095/300 Write block 2868 len 8192 data: 4 5
19539 096/300 Write block 2423 len 12288 data: 4 4 6
19540 097/300 Read block 4923 len 12288
19541 098/300 Write block 2719 len 28672 data: 5 4 3 3 3 3 4
19542 099/300 Write block 3444 len 20480 data: 4 5 4 3 3
19543 100/300 Read block 2381 len 40960
19544 101/300 Write block 2888 len 36864 data: 7 7 7 6 6 4 3 3 3
19545 102/300 Read block 4954 len 40960
19546 103/300 Write block 2171 len 4096 data: 6
19547 104/300 Read block 3485 len 32768
19548 105/300 Write block 4786 len 4096 data: 3
19549 106/300 Read block 1333 len 32768
19550 107/300 Write block 1783 len 12288 data: 3 3 2
19551 108/300 Write block 3896 len 20480 data: 4 3 3 5 5
19552 109/300 Read block 999 len 36864
19553 110/300 Read block 2035 len 20480
19554 111/300 Write block 1553 len 4096 data: 5
19555 112/300 Write block 2915 len 40960 data: 6 7 7 6 6 4 4 2 5 5
19556 113/300 Write block 226 len 40960 data: 5 4 4 3 5 5 5 4 4 5
19557 114/300 Write block 1043 len 16384 data: 7 7 7 6
19558 115/300 Write block 885 len 16384 data: 6 5 5 4
19559 116/300 Read block 11 len 24576
19560 117/300 Read block 2340 len 28672
19561 118/300 Read block 494 len 16384
19562 119/300 Flush
19563 120/300 Write block 3959 len 24576 data: 4 5 3 3 2 2
19564 121/300 Read block 2607 len 16384
19565 122/300 Read block 3816 len 32768
19566 123/300 Flush
19567 124/300 Write block 1094 len 40960 data: 4 3 4 4 3 4 5 4 3 3
19568 125/300 Write block 2490 len 40960 data: 3 3 4 4 3 3 2 3 4 4
19569 126/300 Write block 957 len 40960 data: 6 6 4 4 4 4 4 2 5 5
19570 127/300 Read block 416 len 16384
19571 128/300 Write block 3161 len 32768 data: 2 2 2 3 2 3 3 3
19572 129/300 Read block 3346 len 36864
19573 130/300 Flush
19574 131/300 Write block 2290 len 16384 data: 4 4 4 5
19575 132/300 Read block 2547 len 8192
19576 133/300 Read block 3505 len 40960
19577 134/300 Read block 4211 len 24576
19578 135/300 Write block 2321 len 32768 data: 3 4 4 3 4 3 4 3
19579 136/300 Read block 3679 len 8192
19580 137/300 Write block 2712 len 36864 data: 3 4 4 3 5 7 7 6 5
19581 138/300 Write block 2528 len 40960 data: 5 4 3 3 4 3 3 4 3 3
19582 139/300 Write block 4018 len 8192 data: 2 2
19583 140/300 Read block 1276 len 28672
19584 141/300 Read block 2055 len 36864
19585 142/300 Read block 3475 len 40960
19586 143/300 Read block 191 len 40960
19587 144/300 Read block 1284 len 28672
19588 145/300 Read block 659 len 8192
19589 146/300 Read block 1288 len 12288
19590 147/300 Read block 4153 len 40960
19591 148/300 Read block 294 len 40960
19592 149/300 Read block 360 len 36864
19593 150/300 Read block 1099 len 24576
19594 151/300 Write block 525 len 8192 data: 2 2
19595 152/300 Read block 329 len 12288
19596 153/300 Write block 1174 len 36864 data: 7 7 7 5 4 4 3 2 2
19597 154/300 Write block 3454 len 40960 data: 2 2 2 2 4 5 7 7 7 7
19598 155/300 Read block 2422 len 36864
19599 156/300 Write block 3204 len 20480 data: 5 4 3 3 3
19600 157/300 Flush
19601 158/300 Write block 1750 len 4096 data: 4
19602 159/300 Read block 1131 len 32768
19603 160/300 Read block 1173 len 28672
19604 161/300 Flush
19605 162/300 Flush
19606 163/300 Read block 4271 len 4096
19607 164/300 Write block 4436 len 4096 data: 3
19608 165/300 Read block 4495 len 28672
19609 166/300 Read block 688 len 4096
19610 167/300 Flush
19611 168/300 Write block 4132 len 40960 data: 2 2 3 3 5 5 6 7 8 8
19612 169/300 Write block 1716 len 24576 data: 4 5 6 6 6 5
19613 170/300 Write block 506 len 28672 data: 7 9 7 6 6 5 6
19614 171/300 Read block 4710 len 24576
19615 172/300 Read block 1828 len 12288
19616 173/300 Write block 2517 len 32768 data: 4 4 4 4 5 6 7 6
19617 174/300 Flush
19618 175/300 Read block 902 len 12288
19619 176/300 Read block 2063 len 16384
19620 177/300 Read block 4566 len 20480
19621 178/300 Read block 4809 len 20480
19622 179/300 Read block 3680 len 40960
19623 180/300 Read block 477 len 24576
19624 181/300 Write block 1755 len 20480 data: 3 3 2 4 4
19625 182/300 Write block 2354 len 8192 data: 3 3
19626 183/300 Flush
19627 184/300 Write block 2097 len 16384 data: 4 4 3 3
19628 185/300 Write block 4855 len 4096 data: 4
19629 186/300 Read block 3077 len 36864
19630 187/300 Flush
19631 188/300 Read block 3918 len 32768
19632 189/300 Flush
19633 190/300 Read block 696 len 20480
19634 191/300 Read block 3285 len 28672
19635 192/300 Write block 1490 len 28672 data: 3 3 4 2 3 4 5
19636 193/300 Flush
19637 194/300 Read block 2709 len 32768
19638 195/300 Read block 376 len 12288
19639 196/300 Write block 4413 len 12288 data: 5 4 3
19640 197/300 Write block 4821 len 16384 data: 3 3 3 3
19641 198/300 Flush
19642 199/300 Write block 3462 len 40960 data: 8 8 4 4 3 4 4 4 6 4
19643 200/300 Read block 3521 len 24576
19644 201/300 Read block 3749 len 24576
19645 202/300 Read block 4495 len 40960
19646 203/300 Write block 1743 len 8192 data: 2 2
19647 204/300 Write block 558 len 36864 data: 7 6 5 3 3 3 3 3 3
19648 205/300 Read block 3608 len 12288
19649 206/300 Write block 3567 len 40960 data: 3 5 3 4 4 5 6 7 8 9
19650 207/300 Write block 2707 len 12288 data: 3 4 3
19651 208/300 Read block 1642 len 4096
19652 209/300 Read block 616 len 16384
19653 210/300 Write block 1102 len 32768 data: 4 4 3 3 3 4 5 5
19654 211/300 Read block 3768 len 24576
19655 212/300 Write block 4194 len 12288 data: 2 3 4
19656 213/300 Write block 659 len 40960 data: 5 5 5 5 5 6 5 5 5 4
19657 214/300 Write block 971 len 36864 data: 6 6 5 4 2 2 2 3 4
19658 215/300 Flush
19659 216/300 Read block 637 len 40960
19660 217/300 Flush
19661 218/300 Write block 893 len 28672 data: 5 5 6 4 4 5 4
19662 219/300 Read block 2668 len 28672
19663 220/300 Flush
19664 221/300 Write block 2193 len 8192 data: 2 2
19665 222/300 Read block 138 len 8192
19666 223/300 Read block 1076 len 20480
19667 224/300 Write block 4398 len 20480 data: 7 7 6 7 6
19668 225/300 Write block 4362 len 20480 data: 6 6 6 6 6
19669 226/300 Write block 2326 len 4096 data: 4
19670 227/300 Read block 2491 len 40960
19671 228/300 Read block 204 len 8192
19672 229/300 Read block 936 len 24576
19673 230/300 Read block 2967 len 28672
19674 231/300 Write block 1382 len 28672 data: 3 3 3 4 4 4 4
19675 232/300 Read block 4410 len 32768
19676 233/300 Write block 1043 len 20480 data: 8 8 8 7 5
19677 234/300 Flush
19678 235/300 Read block 4806 len 16384
19679 236/300 Read block 1756 len 4096
19680 237/300 Write block 1185 len 40960 data: 3 3 3 3 3 3 3 3 4 3
19681 238/300 Write block 1220 len 8192 data: 4 3
19682 239/300 Write block 4477 len 40960 data: 6 5 5 6 6 5 5 4 5 5
19683 240/300 Write block 2296 len 20480 data: 6 8 6 5 5
19684 241/300 Write block 2422 len 8192 data: 3 5
19685 242/300 Read block 3243 len 24576
19686 243/300 Read block 1250 len 32768
19687 244/300 Read block 321 len 40960
19688 245/300 Read block 684 len 16384
19689 246/300 Read block 4116 len 36864
19690 247/300 Write block 715 len 36864 data: 4 4 4 4 3 2 3 3 3
19691 248/300 Read block 4419 len 40960
19692 249/300 Write block 4061 len 12288 data: 3 3 3
19693 250/300 Read block 599 len 12288
19694 251/300 Write block 1826 len 4096 data: 3
19695 252/300 Flush
19696 253/300 Write block 96 len 32768 data: 5 5 4 4 4 4 3 3
19697 254/300 Read block 4642 len 40960
19698 255/300 Flush
19699 256/300 Read block 4801 len 36864
19700 257/300 Read block 2242 len 24576
19701 258/300 Write block 1171 len 24576 data: 8 9 8 8 8 8
19702 259/300 Write block 962 len 36864 data: 5 5 3 6 6 7 8 7 7
19703 260/300 Flush
19704 261/300 Flush
19705 262/300 Write block 240 len 28672 data: 7 8 7 6 6 6 5
19706 263/300 Write block 2955 len 20480 data: 5 4 4 4 3
19707 264/300 Write block 3149 len 8192 data: 5 3
19708 265/300 Write block 921 len 12288 data: 2 2 2
19709 266/300 Read block 1819 len 16384
19710 267/300 Read block 4380 len 16384
19711 268/300 Write block 1923 len 12288 data: 5 3 3
19712 269/300 Read block 1903 len 36864
19713 270/300 Write block 2169 len 24576 data: 2 5 7 6 5 4
19714 271/300 Write block 4003 len 16384 data: 5 6 5 6
19715 272/300 Read block 1597 len 36864
19716 273/300 Write block 2700 len 32768 data: 4 5 6 6 3 3 3 4
19717 274/300 Read block 3924 len 32768
19718 275/300 Read block 4432 len 12288
19719 276/300 Write block 1568 len 40960 data: 6 6 8 7 6 5 5 4 4 3
19720 277/300 Read block 4635 len 16384
19721 278/300 Flush
19722 279/300 Read block 703 len 4096
19723 280/300 Read block 2540 len 32768
19724 281/300 Write block 4692 len 36864 data: 6 5 5 5 5 6 5 4 4
19725 282/300 Write block 3839 len 12288 data: 4 2 4
19726 283/300 Read block 404 len 8192
19727 284/300 Write block 4298 len 16384 data: 5 5 4 4
19728 285/300 Write block 585 len 24576 data: 4 3 4 4 5 6
19729 286/300 Write block 480 len 32768 data: 8 10 9 9 7 7 7 6
19730 287/300 Write block 3642 len 36864 data: 7 7 6 5 4 4 4 5 5
19731 288/300 Read block 1086 len 24576
19732 289/300 Read block 413 len 36864
19733 290/300 Flush
19734 291/300 Read block 1971 len 32768
19735 292/300 Read block 584 len 28672
19736 293/300 Flush
19737 294/300 Read block 4561 len 8192
19738 295/300 Flush
19739 296/300 Read block 1652 len 12288
19740 297/300 Read block 4295 len 40960
19741 298/300 Write block 2024 len 16384 data: 4 3 4 4
19742 299/300 Write block 2226 len 32768 data: 2 2 2 3 4 4 4 4
19743 300/300 Read block 3446 len 24576
19744 301/300 Write block 3033 len 20480 data: 4 4 4 4 4
19745 Main task has work to do, go find it
19746 got dsc Start(1)
19747 start 1
19748 Jan 29 05:15:03.989 INFO request completed, latency_us: 185, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2242cd27-ce54-401e-ac33-99f3b3c5887d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
19749 Replay: started 1, returned:Ok(())
19750 ----------------------------------------------------------------
19751 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19752 States: Active Offline Active
19753 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19754 6133 Acked Read 4 Done Sent Done false
19755 6134 Acked Flush 0 Done Sent Done false
19756 6135 Acked Write 2 Done Sent Done false
19757 6136 Acked Write 3 Done Sent Done false
19758 6137 Acked Read 6 Done Sent Done false
19759 6138 Acked Write 4 Done Sent Done false
19760 6139 Acked Read 3 Done Sent Done false
19761 6140 Acked Write 9 Done Sent Done false
19762 6141 Acked Read 4 Done Sent Done false
19763 6142 Acked Write 10 Done Sent Done false
19764 6143 Acked Read 4 Done Sent Done false
19765 6144 Acked Read 6 Done Sent Done false
19766 6145 Acked Read 5 Done Sent Done false
19767 6146 Acked Read 10 Done Sent Done false
19768 6147 Acked Write 9 Done Sent Done false
19769 6148 Acked Write 1 Done Sent Done false
19770 6149 Acked Write 4 Done Sent Done false
19771 6150 Acked Read 7 Done Sent Done false
19772 6151 Acked Read 8 Done Sent Done false
19773 6152 Acked Write 6 Done Sent Done false
19774 6153 Acked Write 7 Done Sent Done false
19775 6154 Acked Read 1 Done Sent Done false
19776 6155 Acked Read 5 Done Sent Done false
19777 6156 Acked Flush 0 Done Sent Done false
19778 6157 Acked Read 7 Done Sent Done false
19779 6158 Acked Write 9 Done Sent Done false
19780 6159 Acked Read 9 Done Sent Done false
19781 6160 Acked Flush 0 Done Sent Done false
19782 6161 Acked Flush 0 Done Sent Done false
19783 6162 Acked Read 9 Done Sent Done false
19784 6163 Acked Write 5 Done Sent Done false
19785 6164 Acked Write 4 Done Sent Done false
19786 6165 Acked Write 1 Done Sent Done false
19787 6166 Acked Write 9 Done Sent Done false
19788 6167 Acked Write 6 Done Sent Done false
19789 6168 Acked Read 3 Done Sent Done false
19790 6169 Acked Write 2 Done Sent Done false
19791 6170 Acked Write 8 Done Sent Done false
19792 6171 Acked Flush 0 Done Sent Done false
19793 6172 Acked Read 3 Done Sent Done false
19794 6173 Acked Read 8 Done Sent Done false
19795 6174 Acked Read 3 Done Sent Done false
19796 6175 Acked Write 4 Done Sent Done false
19797 6176 Acked Read 9 Done Sent Done false
19798 6177 Acked Read 7 Done Sent Done false
19799 6178 Acked Flush 0 Done Sent Done false
19800 6179 Acked Read 8 Done Sent Done false
19801 6180 Acked Write 2 Done Sent Done false
19802 6181 Acked Read 6 Done Sent Done false
19803 6182 Acked Write 2 Done Sent Done false
19804 6183 Acked Write 10 Done Sent Done false
19805 6184 Acked Read 2 Done Sent Done false
19806 6185 Acked Write 2 Done Sent Done false
19807 6186 Acked Write 9 Done Sent Done false
19808 6187 Acked Write 2 Done Sent Done false
19809 6188 Acked Read 4 Done Sent Done false
19810 6189 Acked Write 4 Done Sent Done false
19811 6190 Acked Read 1 Done Sent Done false
19812 6191 Acked Read 3 Done Sent Done false
19813 6192 Acked Read 5 Done Sent Done false
19814 6193 Acked Write 3 Done Sent Done false
19815 6194 Acked Write 1 Done Sent Done false
19816 6195 Acked Flush 0 Done Sent Done false
19817 6196 Acked Read 5 Done Sent Done false
19818 6197 Acked Read 5 Done Sent Done false
19819 6198 Acked Read 10 Done Sent Done false
19820 6199 Acked Read 10 Done Sent Done false
19821 6200 Acked Read 6 Done Sent Done false
19822 6201 Acked Write 9 Done Sent Done false
19823 6202 Acked Read 8 Done Sent Done false
19824 6203 Acked Read 2 Done Sent Done false
19825 6204 Acked Write 10 Done Sent Done false
19826 6205 Acked Read 3 Done Sent Done false
19827 6206 Acked Read 9 Done Sent Done false
19828 6207 Acked Write 4 Done Sent Done false
19829 6208 Acked Read 7 Done Sent Done false
19830 6209 Acked Write 7 Done Sent Done false
19831 6210 Acked Write 6 Done Sent Done false
19832 6211 Acked Write 2 Done Sent Done false
19833 6212 Acked Write 5 Done Sent Done false
19834 6213 Acked Read 3 Done Sent Done false
19835 6214 Acked Read 4 Done Sent Done false
19836 6215 Acked Write 1 Done Sent Done false
19837 6216 Acked Read 3 Done Sent Done false
19838 6217 Acked Read 10 Done Sent Done false
19839 6218 Acked Write 5 Done Sent Done false
19840 6219 Acked Read 1 Done Sent Done false
19841 6220 Acked Read 1 Done Sent Done false
19842 6221 Acked Read 3 Done Sent Done false
19843 6222 Acked Write 6 Done Sent Done false
19844 6223 Acked Flush 0 Done Sent Done false
19845 6224 Acked Read 8 Done Sent Done false
19846 6225 Acked Write 6 Done Sent Done false
19847 6226 Acked Write 6 Done Sent Done false
19848 6227 Acked Write 2 Done Sent Done false
19849 6228 Acked Write 3 Done Sent Done false
19850 6229 Acked Read 3 Done Sent Done false
19851 6230 Acked Write 7 Done Sent Done false
19852 6231 Acked Write 5 Done Sent Done false
19853 6232 Acked Read 10 Done Sent Done false
19854 6233 Acked Write 9 Done Sent Done false
19855 6234 Acked Read 10 Done Sent Done false
19856 6235 Acked Write 1 Done Sent Done false
19857 6236 Acked Read 8 Done Sent Done false
19858 6237 Acked Write 1 Done Sent Done false
19859 6238 Acked Read 8 Done Sent Done false
19860 6239 Acked Write 3 Done Sent Done false
19861 6240 Acked Write 5 Done Sent Done false
19862 6241 Acked Read 9 Done Sent Done false
19863 6242 Acked Read 5 Done Sent Done false
19864 6243 Acked Write 1 Done Sent Done false
19865 6244 Acked Write 10 Done Sent Done false
19866 6245 Acked Write 10 Done Sent Done false
19867 6246 Acked Write 4 Done Sent Done false
19868 6247 Acked Write 4 Done Sent Done false
19869 6248 Acked Read 6 Done Sent Done false
19870 6249 Acked Read 7 Done Sent Done false
19871 6250 Acked Read 4 Done Sent Done false
19872 6251 Acked Flush 0 Done Sent Done false
19873 6252 Acked Write 6 Done Sent Done false
19874 6253 Acked Read 4 Done Sent Done false
19875 6254 Acked Read 8 Done Sent Done false
19876 6255 Acked Flush 0 Done Sent Done false
19877 6256 Acked Write 10 Done Sent Done false
19878 6257 Acked Write 10 Done Sent Done false
19879 6258 Acked Write 10 Done Sent Done false
19880 6259 Acked Read 4 Done Sent Done false
19881 6260 Acked Write 8 Done Sent Done false
19882 6261 Acked Read 9 Done Sent Done false
19883 6262 Acked Flush 0 Done Sent Done false
19884 6263 Acked Write 4 Done Sent Done false
19885 6264 Acked Read 2 Done Sent Done false
19886 6265 Acked Read 10 Done Sent Done false
19887 6266 Acked Read 6 Done Sent Done false
19888 6267 Acked Write 8 Done Sent Done false
19889 6268 Acked Read 2 Done Sent Done false
19890 6269 Acked Write 9 Done Sent Done false
19891 6270 Acked Write 10 Done Sent Done false
19892 6271 Acked Write 2 Done Sent Done false
19893 6272 Acked Read 7 Done Sent Done false
19894 6273 Acked Read 9 Done Sent Done false
19895 6274 Acked Read 10 Done Sent Done false
19896 6275 Acked Read 10 Done Sent Done false
19897 6276 Acked Read 7 Done Sent Done false
19898 6277 Acked Read 2 Done Sent Done false
19899 6278 Acked Read 3 Done Sent Done false
19900 6279 Acked Read 10 Done Sent Done false
19901 6280 Acked Read 10 Done Sent Done false
19902 6281 Acked Read 9 Done Sent Done false
19903 6282 Acked Read 6 Done Sent Done false
19904 6283 Acked Write 2 Done Sent Done false
19905 6284 Acked Read 3 Done Sent Done false
19906 6285 Acked Write 9 Done Sent Done false
19907 6286 Acked Write 10 Done Sent Done false
19908 6287 Acked Read 9 Done Sent Done false
19909 6288 Acked Write 5 Done Sent Done false
19910 6289 Acked Flush 0 Done Sent Done false
19911 6290 Acked Write 1 Done Sent Done false
19912 6291 Acked Read 8 Done Sent Done false
19913 6292 Acked Read 7 Done Sent Done false
19914 6293 Acked Flush 0 Done Sent Done false
19915 6294 Acked Flush 0 Done Sent Done false
19916 6295 Acked Read 1 Done Sent Done false
19917 6296 Acked Write 1 Done Sent Done false
19918 6297 Acked Read 7 Done Sent Done false
19919 6298 Acked Read 1 Done Sent Done false
19920 6299 Acked Flush 0 Done Sent Done false
19921 6300 Acked Write 10 Done Sent Done false
19922 6301 Acked Write 6 Done Sent Done false
19923 6302 Acked Write 7 Done Sent Done false
19924 6303 Acked Read 6 Done Sent Done false
19925 6304 Acked Read 3 Done Sent Done false
19926 6305 Acked Write 8 Done Sent Done false
19927 6306 Acked Flush 0 Done Sent Done false
19928 6307 Acked Read 3 Done Sent Done false
19929 6308 Acked Read 4 Done Sent Done false
19930 6309 Acked Read 5 Done Sent Done false
19931 6310 Acked Read 5 Done Sent Done false
19932 6311 Acked Read 10 Done Sent Done false
19933 6312 Acked Read 6 Done Sent Done false
19934 6313 Acked Write 5 Done Sent Done false
19935 6314 Acked Write 2 Done Sent Done false
19936 6315 Acked Flush 0 Done Sent Done false
19937 6316 Acked Write 4 Done Sent Done false
19938 6317 Acked Write 1 Done Sent Done false
19939 6318 Acked Read 9 Done Sent Done false
19940 6319 Acked Flush 0 Done Sent Done false
19941 6320 Acked Read 8 Done Sent Done false
19942 6321 Acked Flush 0 Done Sent Done false
19943 6322 Acked Read 5 Done Sent Done false
19944 6323 Acked Read 7 Done Sent Done false
19945 6324 Acked Write 7 Done Sent Done false
19946 6325 Acked Flush 0 Done Sent Done false
19947 6326 Acked Read 8 Done Sent Done false
19948 6327 Acked Read 3 Done Sent Done false
19949 6328 Acked Write 3 Done Sent Done false
19950 6329 Acked Write 4 Done Sent Done false
19951 6330 Acked Flush 0 Done Sent Done false
19952 6331 Acked Write 10 Done Sent Done false
19953 6332 Acked Read 6 Done Sent Done false
19954 6333 Acked Read 6 Done Sent Done false
19955 6334 Acked Read 10 Done Sent Done false
19956 6335 Acked Write 2 Done Sent Done false
19957 6336 Acked Write 9 Done Sent Done false
19958 6337 Acked Read 3 Done Sent Done false
19959 6338 Acked Write 10 Done Sent Done false
19960 6339 Acked Write 3 Done Sent Done false
19961 6340 Acked Read 1 Done Sent Done false
19962 6341 Acked Read 4 Done Sent Done false
19963 6342 Acked Write 8 Done Sent Done false
19964 6343 Acked Read 6 Done Sent Done false
19965 6344 Acked Write 3 Done Sent Done false
19966 6345 Acked Write 10 Done Sent Done false
19967 6346 Acked Write 9 Done Sent Done false
19968 6347 Acked Flush 0 Done Sent Done false
19969 6348 Acked Read 10 Done Sent Done false
19970 6349 Acked Flush 0 Done Sent Done false
19971 6350 Acked Write 7 Done Sent Done false
19972 6351 Acked Read 7 Done Sent Done false
19973 6352 Acked Flush 0 Done Sent Done false
19974 6353 Acked Write 2 Done Sent Done false
19975 6354 Acked Read 2 Done Sent Done false
19976 6355 Acked Read 5 Done Sent Done false
19977 6356 Acked Write 5 Done Sent Done false
19978 6357 Acked Write 5 Done Sent Done false
19979 6358 Acked Write 1 Done Sent Done false
19980 6359 Acked Read 10 Done Sent Done false
19981 6360 Acked Read 2 Done Sent Done false
19982 6361 Acked Read 6 Done Sent Done false
19983 6362 Acked Read 7 Done Sent Done false
19984 6363 Acked Write 7 Done Sent Done false
19985 6364 Acked Read 8 Done Sent Done false
19986 6365 Acked Write 5 Done Sent Done false
19987 6366 Acked Flush 0 Done Sent Done false
19988 6367 Acked Read 4 Done Sent Done false
19989 6368 Acked Read 1 Done Sent Done false
19990 6369 Acked Write 10 Done Sent Done false
19991 6370 Acked Write 2 Done Sent Done false
19992 6371 Acked Write 10 Done Sent Done false
19993 6372 Acked Write 5 Done Sent Done false
19994 6373 Acked Write 2 Done Sent Done false
19995 6374 Acked Read 6 Done Sent Done false
19996 6375 Acked Read 8 Done Sent Done false
19997 6376 Acked Read 10 Done Sent Done false
19998 6377 Acked Read 4 Done Sent Done false
19999 6378 Acked Read 9 Done Sent Done false
20000 6379 Acked Write 9 Done Sent Done false
20001 6380 Acked Read 10 Done Sent Done false
20002 6381 Acked Write 3 Done Sent Done false
20003 6382 Acked Read 3 Done Sent Done false
20004 6383 Acked Write 1 Done Sent Done false
20005 6384 Acked Flush 0 Done Sent Done false
20006 6385 Acked Write 8 Done Sent Done false
20007 6386 Acked Read 10 Done Sent Done false
20008 6387 Acked Flush 0 Done Sent Done false
20009 6388 Acked Read 9 Done Sent Done false
20010 6389 Acked Read 6 Done Sent Done false
20011 6390 Acked Write 6 Done Sent Done false
20012 6391 Acked Write 9 Done Sent Done false
20013 6392 Acked Flush 0 Done Sent Done false
20014 6393 Acked Flush 0 Done Sent Done false
20015 6394 Acked Write 7 Done Sent Done false
20016 6395 Acked Write 5 Done Sent Done false
20017 6396 Acked Write 2 Done Sent Done false
20018 6397 Acked Write 3 Done Sent Done false
20019 6398 Acked Read 4 Done Sent Done false
20020 6399 Acked Read 4 Done Sent Done false
20021 6400 Acked Write 3 Done Sent Done false
20022 6401 Acked Read 9 Done Sent Done false
20023 6402 Acked Write 6 Done Sent Done false
20024 6403 Acked Write 4 Done Sent Done false
20025 6404 Acked Read 9 Done Sent Done false
20026 6405 Acked Write 8 Done Sent Done false
20027 6406 Acked Read 8 Done Sent Done false
20028 6407 Acked Read 3 Done Sent Done false
20029 6408 Acked Write 10 Done Sent Done false
20030 6409 Acked Read 4 Done Sent Done false
20031 6410 Acked Flush 0 Done Sent Done false
20032 6411 Acked Read 1 Done Sent Done false
20033 6412 Acked Read 8 Done Sent Done false
20034 6413 Acked Write 9 Done Sent Done false
20035 6414 Acked Write 3 Done Sent Done false
20036 6415 Acked Read 2 Done Sent Done false
20037 6416 Acked Write 4 Done Sent Done false
20038 6417 Acked Write 6 Done Sent Done false
20039 6418 Acked Write 8 Done Sent Done false
20040 6419 Acked Write 9 Done Sent Done false
20041 6420 Acked Read 6 Done Sent Done false
20042 6421 Acked Read 9 Done Sent Done false
20043 6422 Acked Flush 0 Done Sent Done false
20044 6423 Acked Read 8 Done Sent Done false
20045 6424 Acked Read 7 Done Sent Done false
20046 6425 Acked Flush 0 Done Sent Done false
20047 6426 Acked Read 2 Done Sent Done false
20048 6427 Acked Flush 0 Done Sent Done false
20049 6428 Acked Read 3 Done Sent Done false
20050 6429 Acked Read 10 Done Sent Done false
20051 6430 Acked Write 4 Done Sent Done false
20052 6431 Acked Write 8 Done Sent Done false
20053 6432 Acked Read 6 Done Sent Done false
20054 6433 Acked Write 5 Done Sent Done false
20055 STATES DS:0 DS:1 DS:2 TOTAL
20056 Sent 0 301 0 301
20057 Done 301 0 301 602
20058 Skipped 0 0 0 0
20059 Error 0 0 0 0
20060 Last Flush: Some(JobId(6427)) Some(JobId(6132)) Some(JobId(6427))
20061 Downstairs last five completed: 6132 6131 6130 6129 6128
20062 Upstairs last five completed: 6433 6432 6431 6430 6429
20063 CLIENT: Up:0 ds:301 act:2
200642025-01-29T05:15:04.491ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
200652025-01-29T05:15:05.738ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
200662025-01-29T05:15:06.351ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
200672025-01-29T05:15:07.352ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
20068 ----------------------------------------------------------------
20069 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20070 States: Active Offline Active
20071 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20072 6133 Acked Read 4 Done Sent Done false
20073 6134 Acked Flush 0 Done Sent Done false
20074 6135 Acked Write 2 Done Sent Done false
20075 6136 Acked Write 3 Done Sent Done false
20076 6137 Acked Read 6 Done Sent Done false
20077 6138 Acked Write 4 Done Sent Done false
20078 6139 Acked Read 3 Done Sent Done false
20079 6140 Acked Write 9 Done Sent Done false
20080 6141 Acked Read 4 Done Sent Done false
20081 6142 Acked Write 10 Done Sent Done false
20082 6143 Acked Read 4 Done Sent Done false
20083 6144 Acked Read 6 Done Sent Done false
20084 6145 Acked Read 5 Done Sent Done false
20085 6146 Acked Read 10 Done Sent Done false
20086 6147 Acked Write 9 Done Sent Done false
20087 6148 Acked Write 1 Done Sent Done false
20088 6149 Acked Write 4 Done Sent Done false
20089 6150 Acked Read 7 Done Sent Done false
20090 6151 Acked Read 8 Done Sent Done false
20091 6152 Acked Write 6 Done Sent Done false
20092 6153 Acked Write 7 Done Sent Done false
20093 6154 Acked Read 1 Done Sent Done false
20094 6155 Acked Read 5 Done Sent Done false
20095 6156 Acked Flush 0 Done Sent Done false
20096 6157 Acked Read 7 Done Sent Done false
20097 6158 Acked Write 9 Done Sent Done false
20098 6159 Acked Read 9 Done Sent Done false
20099 6160 Acked Flush 0 Done Sent Done false
20100 6161 Acked Flush 0 Done Sent Done false
20101 6162 Acked Read 9 Done Sent Done false
20102 6163 Acked Write 5 Done Sent Done false
20103 6164 Acked Write 4 Done Sent Done false
20104 6165 Acked Write 1 Done Sent Done false
20105 6166 Acked Write 9 Done Sent Done false
20106 6167 Acked Write 6 Done Sent Done false
20107 6168 Acked Read 3 Done Sent Done false
20108 6169 Acked Write 2 Done Sent Done false
20109 6170 Acked Write 8 Done Sent Done false
20110 6171 Acked Flush 0 Done Sent Done false
20111 6172 Acked Read 3 Done Sent Done false
20112 6173 Acked Read 8 Done Sent Done false
20113 6174 Acked Read 3 Done Sent Done false
20114 6175 Acked Write 4 Done Sent Done false
20115 6176 Acked Read 9 Done Sent Done false
20116 6177 Acked Read 7 Done Sent Done false
20117 6178 Acked Flush 0 Done Sent Done false
20118 6179 Acked Read 8 Done Sent Done false
20119 6180 Acked Write 2 Done Sent Done false
20120 6181 Acked Read 6 Done Sent Done false
20121 6182 Acked Write 2 Done Sent Done false
20122 6183 Acked Write 10 Done Sent Done false
20123 6184 Acked Read 2 Done Sent Done false
20124 6185 Acked Write 2 Done Sent Done false
20125 6186 Acked Write 9 Done Sent Done false
20126 6187 Acked Write 2 Done Sent Done false
20127 6188 Acked Read 4 Done Sent Done false
20128 6189 Acked Write 4 Done Sent Done false
20129 6190 Acked Read 1 Done Sent Done false
20130 6191 Acked Read 3 Done Sent Done false
20131 6192 Acked Read 5 Done Sent Done false
20132 6193 Acked Write 3 Done Sent Done false
20133 6194 Acked Write 1 Done Sent Done false
20134 6195 Acked Flush 0 Done Sent Done false
20135 6196 Acked Read 5 Done Sent Done false
20136 6197 Acked Read 5 Done Sent Done false
20137 6198 Acked Read 10 Done Sent Done false
20138 6199 Acked Read 10 Done Sent Done false
20139 6200 Acked Read 6 Done Sent Done false
20140 6201 Acked Write 9 Done Sent Done false
20141 6202 Acked Read 8 Done Sent Done false
20142 6203 Acked Read 2 Done Sent Done false
20143 6204 Acked Write 10 Done Sent Done false
20144 6205 Acked Read 3 Done Sent Done false
20145 6206 Acked Read 9 Done Sent Done false
20146 6207 Acked Write 4 Done Sent Done false
20147 6208 Acked Read 7 Done Sent Done false
20148 6209 Acked Write 7 Done Sent Done false
20149 6210 Acked Write 6 Done Sent Done false
20150 6211 Acked Write 2 Done Sent Done false
20151 6212 Acked Write 5 Done Sent Done false
20152 6213 Acked Read 3 Done Sent Done false
20153 6214 Acked Read 4 Done Sent Done false
20154 6215 Acked Write 1 Done Sent Done false
20155 6216 Acked Read 3 Done Sent Done false
20156 6217 Acked Read 10 Done Sent Done false
20157 6218 Acked Write 5 Done Sent Done false
20158 6219 Acked Read 1 Done Sent Done false
20159 6220 Acked Read 1 Done Sent Done false
20160 6221 Acked Read 3 Done Sent Done false
20161 6222 Acked Write 6 Done Sent Done false
20162 6223 Acked Flush 0 Done Sent Done false
20163 6224 Acked Read 8 Done Sent Done false
20164 6225 Acked Write 6 Done Sent Done false
20165 6226 Acked Write 6 Done Sent Done false
20166 6227 Acked Write 2 Done Sent Done false
20167 6228 Acked Write 3 Done Sent Done false
20168 6229 Acked Read 3 Done Sent Done false
20169 6230 Acked Write 7 Done Sent Done false
20170 6231 Acked Write 5 Done Sent Done false
20171 6232 Acked Read 10 Done Sent Done false
20172 6233 Acked Write 9 Done Sent Done false
20173 6234 Acked Read 10 Done Sent Done false
20174 6235 Acked Write 1 Done Sent Done false
20175 6236 Acked Read 8 Done Sent Done false
20176 6237 Acked Write 1 Done Sent Done false
20177 6238 Acked Read 8 Done Sent Done false
20178 6239 Acked Write 3 Done Sent Done false
20179 6240 Acked Write 5 Done Sent Done false
20180 6241 Acked Read 9 Done Sent Done false
20181 6242 Acked Read 5 Done Sent Done false
20182 6243 Acked Write 1 Done Sent Done false
20183 6244 Acked Write 10 Done Sent Done false
20184 6245 Acked Write 10 Done Sent Done false
20185 6246 Acked Write 4 Done Sent Done false
20186 6247 Acked Write 4 Done Sent Done false
20187 6248 Acked Read 6 Done Sent Done false
20188 6249 Acked Read 7 Done Sent Done false
20189 6250 Acked Read 4 Done Sent Done false
20190 6251 Acked Flush 0 Done Sent Done false
20191 6252 Acked Write 6 Done Sent Done false
20192 6253 Acked Read 4 Done Sent Done false
20193 6254 Acked Read 8 Done Sent Done false
20194 6255 Acked Flush 0 Done Sent Done false
20195 6256 Acked Write 10 Done Sent Done false
20196 6257 Acked Write 10 Done Sent Done false
20197 6258 Acked Write 10 Done Sent Done false
20198 6259 Acked Read 4 Done Sent Done false
20199 6260 Acked Write 8 Done Sent Done false
20200 6261 Acked Read 9 Done Sent Done false
20201 6262 Acked Flush 0 Done Sent Done false
20202 6263 Acked Write 4 Done Sent Done false
20203 6264 Acked Read 2 Done Sent Done false
20204 6265 Acked Read 10 Done Sent Done false
20205 6266 Acked Read 6 Done Sent Done false
20206 6267 Acked Write 8 Done Sent Done false
20207 6268 Acked Read 2 Done Sent Done false
20208 6269 Acked Write 9 Done Sent Done false
20209 6270 Acked Write 10 Done Sent Done false
20210 6271 Acked Write 2 Done Sent Done false
20211 6272 Acked Read 7 Done Sent Done false
20212 6273 Acked Read 9 Done Sent Done false
20213 6274 Acked Read 10 Done Sent Done false
20214 6275 Acked Read 10 Done Sent Done false
20215 6276 Acked Read 7 Done Sent Done false
20216 6277 Acked Read 2 Done Sent Done false
20217 6278 Acked Read 3 Done Sent Done false
20218 6279 Acked Read 10 Done Sent Done false
20219 6280 Acked Read 10 Done Sent Done false
20220 6281 Acked Read 9 Done Sent Done false
20221 6282 Acked Read 6 Done Sent Done false
20222 6283 Acked Write 2 Done Sent Done false
20223 6284 Acked Read 3 Done Sent Done false
20224 6285 Acked Write 9 Done Sent Done false
20225 6286 Acked Write 10 Done Sent Done false
20226 6287 Acked Read 9 Done Sent Done false
20227 6288 Acked Write 5 Done Sent Done false
20228 6289 Acked Flush 0 Done Sent Done false
20229 6290 Acked Write 1 Done Sent Done false
20230 6291 Acked Read 8 Done Sent Done false
20231 6292 Acked Read 7 Done Sent Done false
20232 6293 Acked Flush 0 Done Sent Done false
20233 6294 Acked Flush 0 Done Sent Done false
20234 6295 Acked Read 1 Done Sent Done false
20235 6296 Acked Write 1 Done Sent Done false
20236 6297 Acked Read 7 Done Sent Done false
20237 6298 Acked Read 1 Done Sent Done false
20238 6299 Acked Flush 0 Done Sent Done false
20239 6300 Acked Write 10 Done Sent Done false
20240 6301 Acked Write 6 Done Sent Done false
20241 6302 Acked Write 7 Done Sent Done false
20242 6303 Acked Read 6 Done Sent Done false
20243 6304 Acked Read 3 Done Sent Done false
20244 6305 Acked Write 8 Done Sent Done false
20245 6306 Acked Flush 0 Done Sent Done false
20246 6307 Acked Read 3 Done Sent Done false
20247 6308 Acked Read 4 Done Sent Done false
20248 6309 Acked Read 5 Done Sent Done false
20249 6310 Acked Read 5 Done Sent Done false
20250 6311 Acked Read 10 Done Sent Done false
20251 6312 Acked Read 6 Done Sent Done false
20252 6313 Acked Write 5 Done Sent Done false
20253 6314 Acked Write 2 Done Sent Done false
20254 6315 Acked Flush 0 Done Sent Done false
20255 6316 Acked Write 4 Done Sent Done false
20256 6317 Acked Write 1 Done Sent Done false
20257 6318 Acked Read 9 Done Sent Done false
20258 6319 Acked Flush 0 Done Sent Done false
20259 6320 Acked Read 8 Done Sent Done false
20260 6321 Acked Flush 0 Done Sent Done false
20261 6322 Acked Read 5 Done Sent Done false
20262 6323 Acked Read 7 Done Sent Done false
20263 6324 Acked Write 7 Done Sent Done false
20264 6325 Acked Flush 0 Done Sent Done false
20265 6326 Acked Read 8 Done Sent Done false
20266 6327 Acked Read 3 Done Sent Done false
20267 6328 Acked Write 3 Done Sent Done false
20268 6329 Acked Write 4 Done Sent Done false
20269 6330 Acked Flush 0 Done Sent Done false
20270 6331 Acked Write 10 Done Sent Done false
20271 6332 Acked Read 6 Done Sent Done false
20272 6333 Acked Read 6 Done Sent Done false
20273 6334 Acked Read 10 Done Sent Done false
20274 6335 Acked Write 2 Done Sent Done false
20275 6336 Acked Write 9 Done Sent Done false
20276 6337 Acked Read 3 Done Sent Done false
20277 6338 Acked Write 10 Done Sent Done false
20278 6339 Acked Write 3 Done Sent Done false
20279 6340 Acked Read 1 Done Sent Done false
20280 6341 Acked Read 4 Done Sent Done false
20281 6342 Acked Write 8 Done Sent Done false
20282 6343 Acked Read 6 Done Sent Done false
20283 6344 Acked Write 3 Done Sent Done false
20284 6345 Acked Write 10 Done Sent Done false
20285 6346 Acked Write 9 Done Sent Done false
20286 6347 Acked Flush 0 Done Sent Done false
20287 6348 Acked Read 10 Done Sent Done false
20288 6349 Acked Flush 0 Done Sent Done false
20289 6350 Acked Write 7 Done Sent Done false
20290 6351 Acked Read 7 Done Sent Done false
20291 6352 Acked Flush 0 Done Sent Done false
20292 6353 Acked Write 2 Done Sent Done false
20293 6354 Acked Read 2 Done Sent Done false
20294 6355 Acked Read 5 Done Sent Done false
20295 6356 Acked Write 5 Done Sent Done false
20296 6357 Acked Write 5 Done Sent Done false
20297 6358 Acked Write 1 Done Sent Done false
20298 6359 Acked Read 10 Done Sent Done false
20299 6360 Acked Read 2 Done Sent Done false
20300 6361 Acked Read 6 Done Sent Done false
20301 6362 Acked Read 7 Done Sent Done false
20302 6363 Acked Write 7 Done Sent Done false
20303 6364 Acked Read 8 Done Sent Done false
20304 6365 Acked Write 5 Done Sent Done false
20305 6366 Acked Flush 0 Done Sent Done false
20306 6367 Acked Read 4 Done Sent Done false
20307 6368 Acked Read 1 Done Sent Done false
20308 6369 Acked Write 10 Done Sent Done false
20309 6370 Acked Write 2 Done Sent Done false
20310 6371 Acked Write 10 Done Sent Done false
20311 6372 Acked Write 5 Done Sent Done false
20312 6373 Acked Write 2 Done Sent Done false
20313 6374 Acked Read 6 Done Sent Done false
20314 6375 Acked Read 8 Done Sent Done false
20315 6376 Acked Read 10 Done Sent Done false
20316 6377 Acked Read 4 Done Sent Done false
20317 6378 Acked Read 9 Done Sent Done false
20318 6379 Acked Write 9 Done Sent Done false
20319 6380 Acked Read 10 Done Sent Done false
20320 6381 Acked Write 3 Done Sent Done false
20321 6382 Acked Read 3 Done Sent Done false
20322 6383 Acked Write 1 Done Sent Done false
20323 6384 Acked Flush 0 Done Sent Done false
20324 6385 Acked Write 8 Done Sent Done false
20325 6386 Acked Read 10 Done Sent Done false
20326 6387 Acked Flush 0 Done Sent Done false
20327 6388 Acked Read 9 Done Sent Done false
20328 6389 Acked Read 6 Done Sent Done false
20329 6390 Acked Write 6 Done Sent Done false
20330 6391 Acked Write 9 Done Sent Done false
20331 6392 Acked Flush 0 Done Sent Done false
20332 6393 Acked Flush 0 Done Sent Done false
20333 6394 Acked Write 7 Done Sent Done false
20334 6395 Acked Write 5 Done Sent Done false
20335 6396 Acked Write 2 Done Sent Done false
20336 6397 Acked Write 3 Done Sent Done false
20337 6398 Acked Read 4 Done Sent Done false
20338 6399 Acked Read 4 Done Sent Done false
20339 6400 Acked Write 3 Done Sent Done false
20340 6401 Acked Read 9 Done Sent Done false
20341 6402 Acked Write 6 Done Sent Done false
20342 6403 Acked Write 4 Done Sent Done false
20343 6404 Acked Read 9 Done Sent Done false
20344 6405 Acked Write 8 Done Sent Done false
20345 6406 Acked Read 8 Done Sent Done false
20346 6407 Acked Read 3 Done Sent Done false
20347 6408 Acked Write 10 Done Sent Done false
20348 6409 Acked Read 4 Done Sent Done false
20349 6410 Acked Flush 0 Done Sent Done false
20350 6411 Acked Read 1 Done Sent Done false
20351 6412 Acked Read 8 Done Sent Done false
20352 6413 Acked Write 9 Done Sent Done false
20353 6414 Acked Write 3 Done Sent Done false
20354 6415 Acked Read 2 Done Sent Done false
20355 6416 Acked Write 4 Done Sent Done false
20356 6417 Acked Write 6 Done Sent Done false
20357 6418 Acked Write 8 Done Sent Done false
20358 6419 Acked Write 9 Done Sent Done false
20359 6420 Acked Read 6 Done Sent Done false
20360 6421 Acked Read 9 Done Sent Done false
20361 6422 Acked Flush 0 Done Sent Done false
20362 6423 Acked Read 8 Done Sent Done false
20363 6424 Acked Read 7 Done Sent Done false
20364 6425 Acked Flush 0 Done Sent Done false
20365 6426 Acked Read 2 Done Sent Done false
20366 6427 Acked Flush 0 Done Sent Done false
20367 6428 Acked Read 3 Done Sent Done false
20368 6429 Acked Read 10 Done Sent Done false
20369 6430 Acked Write 4 Done Sent Done false
20370 6431 Acked Write 8 Done Sent Done false
20371 6432 Acked Read 6 Done Sent Done false
20372 6433 Acked Write 5 Done Sent Done false
20373 STATES DS:0 DS:1 DS:2 TOTAL
20374 Sent 0 301 0 301
20375 Done 301 0 301 602
20376 Skipped 0 0 0 0
20377 Error 0 0 0 0
20378 Last Flush: Some(JobId(6427)) Some(JobId(6132)) Some(JobId(6427))
20379 Downstairs last five completed: 6132 6131 6130 6129 6128
20380 Upstairs last five completed: 6433 6432 6431 6430 6429
20381 CLIENT: Up:0 ds:301 act:2
203822025-01-29T05:15:08.493ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203832025-01-29T05:15:09.355ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203842025-01-29T05:15:09.743ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203852025-01-29T05:15:09.743ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203862025-01-29T05:15:09.743ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203872025-01-29T05:15:09.743ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203882025-01-29T05:15:09.743ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203892025-01-29T05:15:10.739ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
203902025-01-29T05:15:11.359ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
20391 [8820] got start action
20392 [8820] I am going to restart
20393 Starting downstairs at port 8820
20394 Make output file at "/tmp/dsc/downstairs-8820.txt"
20395 [8820][1] reports Starting
20396 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1220)
20397 [8820][1] reports Running
203982025-01-29T05:15:11.859ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
20399 ----------------------------------------------------------------
20400 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20401 States: Active Offline Active
20402 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20403 6133 Acked Read 4 Done Sent Done false
20404 6134 Acked Flush 0 Done Sent Done false
20405 6135 Acked Write 2 Done Sent Done false
20406 6136 Acked Write 3 Done Sent Done false
20407 6137 Acked Read 6 Done Sent Done false
20408 6138 Acked Write 4 Done Sent Done false
20409 6139 Acked Read 3 Done Sent Done false
20410 6140 Acked Write 9 Done Sent Done false
20411 6141 Acked Read 4 Done Sent Done false
20412 6142 Acked Write 10 Done Sent Done false
20413 6143 Acked Read 4 Done Sent Done false
20414 6144 Acked Read 6 Done Sent Done false
20415 6145 Acked Read 5 Done Sent Done false
20416 6146 Acked Read 10 Done Sent Done false
20417 6147 Acked Write 9 Done Sent Done false
20418 6148 Acked Write 1 Done Sent Done false
20419 6149 Acked Write 4 Done Sent Done false
20420 6150 Acked Read 7 Done Sent Done false
20421 6151 Acked Read 8 Done Sent Done false
20422 6152 Acked Write 6 Done Sent Done false
20423 6153 Acked Write 7 Done Sent Done false
20424 6154 Acked Read 1 Done Sent Done false
20425 6155 Acked Read 5 Done Sent Done false
20426 6156 Acked Flush 0 Done Sent Done false
20427 6157 Acked Read 7 Done Sent Done false
20428 6158 Acked Write 9 Done Sent Done false
20429 6159 Acked Read 9 Done Sent Done false
20430 6160 Acked Flush 0 Done Sent Done false
20431 6161 Acked Flush 0 Done Sent Done false
20432 6162 Acked Read 9 Done Sent Done false
20433 6163 Acked Write 5 Done Sent Done false
20434 6164 Acked Write 4 Done Sent Done false
20435 6165 Acked Write 1 Done Sent Done false
20436 6166 Acked Write 9 Done Sent Done false
20437 6167 Acked Write 6 Done Sent Done false
20438 6168 Acked Read 3 Done Sent Done false
20439 6169 Acked Write 2 Done Sent Done false
20440 6170 Acked Write 8 Done Sent Done false
20441 6171 Acked Flush 0 Done Sent Done false
20442 6172 Acked Read 3 Done Sent Done false
20443 6173 Acked Read 8 Done Sent Done false
20444 6174 Acked Read 3 Done Sent Done false
20445 6175 Acked Write 4 Done Sent Done false
20446 6176 Acked Read 9 Done Sent Done false
20447 6177 Acked Read 7 Done Sent Done false
20448 6178 Acked Flush 0 Done Sent Done false
20449 6179 Acked Read 8 Done Sent Done false
20450 6180 Acked Write 2 Done Sent Done false
20451 6181 Acked Read 6 Done Sent Done false
20452 6182 Acked Write 2 Done Sent Done false
20453 6183 Acked Write 10 Done Sent Done false
20454 6184 Acked Read 2 Done Sent Done false
20455 6185 Acked Write 2 Done Sent Done false
20456 6186 Acked Write 9 Done Sent Done false
20457 6187 Acked Write 2 Done Sent Done false
20458 6188 Acked Read 4 Done Sent Done false
20459 6189 Acked Write 4 Done Sent Done false
20460 6190 Acked Read 1 Done Sent Done false
20461 6191 Acked Read 3 Done Sent Done false
20462 6192 Acked Read 5 Done Sent Done false
20463 6193 Acked Write 3 Done Sent Done false
20464 6194 Acked Write 1 Done Sent Done false
20465 6195 Acked Flush 0 Done Sent Done false
20466 6196 Acked Read 5 Done Sent Done false
20467 6197 Acked Read 5 Done Sent Done false
20468 6198 Acked Read 10 Done Sent Done false
20469 6199 Acked Read 10 Done Sent Done false
20470 6200 Acked Read 6 Done Sent Done false
20471 6201 Acked Write 9 Done Sent Done false
20472 6202 Acked Read 8 Done Sent Done false
20473 6203 Acked Read 2 Done Sent Done false
20474 6204 Acked Write 10 Done Sent Done false
20475 6205 Acked Read 3 Done Sent Done false
20476 6206 Acked Read 9 Done Sent Done false
20477 6207 Acked Write 4 Done Sent Done false
20478 6208 Acked Read 7 Done Sent Done false
20479 6209 Acked Write 7 Done Sent Done false
20480 6210 Acked Write 6 Done Sent Done false
20481 6211 Acked Write 2 Done Sent Done false
20482 6212 Acked Write 5 Done Sent Done false
20483 6213 Acked Read 3 Done Sent Done false
20484 6214 Acked Read 4 Done Sent Done false
20485 6215 Acked Write 1 Done Sent Done false
20486 6216 Acked Read 3 Done Sent Done false
20487 6217 Acked Read 10 Done Sent Done false
20488 6218 Acked Write 5 Done Sent Done false
20489 6219 Acked Read 1 Done Sent Done false
20490 6220 Acked Read 1 Done Sent Done false
20491 6221 Acked Read 3 Done Sent Done false
20492 6222 Acked Write 6 Done Sent Done false
20493 6223 Acked Flush 0 Done Sent Done false
20494 6224 Acked Read 8 Done Sent Done false
20495 6225 Acked Write 6 Done Sent Done false
20496 6226 Acked Write 6 Done Sent Done false
20497 6227 Acked Write 2 Done Sent Done false
20498 6228 Acked Write 3 Done Sent Done false
20499 6229 Acked Read 3 Done Sent Done false
20500 6230 Acked Write 7 Done Sent Done false
20501 6231 Acked Write 5 Done Sent Done false
20502 6232 Acked Read 10 Done Sent Done false
20503 6233 Acked Write 9 Done Sent Done false
20504 6234 Acked Read 10 Done Sent Done false
20505 6235 Acked Write 1 Done Sent Done false
20506 6236 Acked Read 8 Done Sent Done false
20507 6237 Acked Write 1 Done Sent Done false
20508 6238 Acked Read 8 Done Sent Done false
20509 6239 Acked Write 3 Done Sent Done false
20510 6240 Acked Write 5 Done Sent Done false
20511 6241 Acked Read 9 Done Sent Done false
20512 6242 Acked Read 5 Done Sent Done false
20513 6243 Acked Write 1 Done Sent Done false
20514 6244 Acked Write 10 Done Sent Done false
20515 6245 Acked Write 10 Done Sent Done false
20516 6246 Acked Write 4 Done Sent Done false
20517 6247 Acked Write 4 Done Sent Done false
20518 6248 Acked Read 6 Done Sent Done false
20519 6249 Acked Read 7 Done Sent Done false
20520 6250 Acked Read 4 Done Sent Done false
20521 6251 Acked Flush 0 Done Sent Done false
20522 6252 Acked Write 6 Done Sent Done false
20523 6253 Acked Read 4 Done Sent Done false
20524 6254 Acked Read 8 Done Sent Done false
20525 6255 Acked Flush 0 Done Sent Done false
20526 6256 Acked Write 10 Done Sent Done false
20527 6257 Acked Write 10 Done Sent Done false
20528 6258 Acked Write 10 Done Sent Done false
20529 6259 Acked Read 4 Done Sent Done false
20530 6260 Acked Write 8 Done Sent Done false
20531 6261 Acked Read 9 Done Sent Done false
20532 6262 Acked Flush 0 Done Sent Done false
20533 6263 Acked Write 4 Done Sent Done false
20534 6264 Acked Read 2 Done Sent Done false
20535 6265 Acked Read 10 Done Sent Done false
20536 6266 Acked Read 6 Done Sent Done false
20537 6267 Acked Write 8 Done Sent Done false
20538 6268 Acked Read 2 Done Sent Done false
20539 6269 Acked Write 9 Done Sent Done false
20540 6270 Acked Write 10 Done Sent Done false
20541 6271 Acked Write 2 Done Sent Done false
20542 6272 Acked Read 7 Done Sent Done false
20543 6273 Acked Read 9 Done Sent Done false
20544 6274 Acked Read 10 Done Sent Done false
20545 6275 Acked Read 10 Done Sent Done false
20546 6276 Acked Read 7 Done Sent Done false
20547 6277 Acked Read 2 Done Sent Done false
20548 6278 Acked Read 3 Done Sent Done false
20549 6279 Acked Read 10 Done Sent Done false
20550 6280 Acked Read 10 Done Sent Done false
20551 6281 Acked Read 9 Done Sent Done false
20552 6282 Acked Read 6 Done Sent Done false
20553 6283 Acked Write 2 Done Sent Done false
20554 6284 Acked Read 3 Done Sent Done false
20555 6285 Acked Write 9 Done Sent Done false
20556 6286 Acked Write 10 Done Sent Done false
20557 6287 Acked Read 9 Done Sent Done false
20558 6288 Acked Write 5 Done Sent Done false
20559 6289 Acked Flush 0 Done Sent Done false
20560 6290 Acked Write 1 Done Sent Done false
20561 6291 Acked Read 8 Done Sent Done false
20562 6292 Acked Read 7 Done Sent Done false
20563 6293 Acked Flush 0 Done Sent Done false
20564 6294 Acked Flush 0 Done Sent Done false
20565 6295 Acked Read 1 Done Sent Done false
20566 6296 Acked Write 1 Done Sent Done false
20567 6297 Acked Read 7 Done Sent Done false
20568 6298 Acked Read 1 Done Sent Done false
20569 6299 Acked Flush 0 Done Sent Done false
20570 6300 Acked Write 10 Done Sent Done false
20571 6301 Acked Write 6 Done Sent Done false
20572 6302 Acked Write 7 Done Sent Done false
20573 6303 Acked Read 6 Done Sent Done false
20574 6304 Acked Read 3 Done Sent Done false
20575 6305 Acked Write 8 Done Sent Done false
20576 6306 Acked Flush 0 Done Sent Done false
20577 6307 Acked Read 3 Done Sent Done false
20578 6308 Acked Read 4 Done Sent Done false
20579 6309 Acked Read 5 Done Sent Done false
20580 6310 Acked Read 5 Done Sent Done false
20581 6311 Acked Read 10 Done Sent Done false
20582 6312 Acked Read 6 Done Sent Done false
20583 6313 Acked Write 5 Done Sent Done false
20584 6314 Acked Write 2 Done Sent Done false
20585 6315 Acked Flush 0 Done Sent Done false
20586 6316 Acked Write 4 Done Sent Done false
20587 6317 Acked Write 1 Done Sent Done false
20588 6318 Acked Read 9 Done Sent Done false
20589 6319 Acked Flush 0 Done Sent Done false
20590 6320 Acked Read 8 Done Sent Done false
20591 6321 Acked Flush 0 Done Sent Done false
20592 6322 Acked Read 5 Done Sent Done false
20593 6323 Acked Read 7 Done Sent Done false
20594 6324 Acked Write 7 Done Sent Done false
20595 6325 Acked Flush 0 Done Sent Done false
20596 6326 Acked Read 8 Done Sent Done false
20597 6327 Acked Read 3 Done Sent Done false
20598 6328 Acked Write 3 Done Sent Done false
20599 6329 Acked Write 4 Done Sent Done false
20600 6330 Acked Flush 0 Done Sent Done false
20601 6331 Acked Write 10 Done Sent Done false
20602 6332 Acked Read 6 Done Sent Done false
20603 6333 Acked Read 6 Done Sent Done false
20604 6334 Acked Read 10 Done Sent Done false
20605 6335 Acked Write 2 Done Sent Done false
20606 6336 Acked Write 9 Done Sent Done false
20607 6337 Acked Read 3 Done Sent Done false
20608 6338 Acked Write 10 Done Sent Done false
20609 6339 Acked Write 3 Done Sent Done false
20610 6340 Acked Read 1 Done Sent Done false
20611 6341 Acked Read 4 Done Sent Done false
20612 6342 Acked Write 8 Done Sent Done false
20613 6343 Acked Read 6 Done Sent Done false
20614 6344 Acked Write 3 Done Sent Done false
20615 6345 Acked Write 10 Done Sent Done false
20616 6346 Acked Write 9 Done Sent Done false
20617 6347 Acked Flush 0 Done Sent Done false
20618 6348 Acked Read 10 Done Sent Done false
20619 6349 Acked Flush 0 Done Sent Done false
20620 6350 Acked Write 7 Done Sent Done false
20621 6351 Acked Read 7 Done Sent Done false
20622 6352 Acked Flush 0 Done Sent Done false
20623 6353 Acked Write 2 Done Sent Done false
20624 6354 Acked Read 2 Done Sent Done false
20625 6355 Acked Read 5 Done Sent Done false
20626 6356 Acked Write 5 Done Sent Done false
20627 6357 Acked Write 5 Done Sent Done false
20628 6358 Acked Write 1 Done Sent Done false
20629 6359 Acked Read 10 Done Sent Done false
20630 6360 Acked Read 2 Done Sent Done false
20631 6361 Acked Read 6 Done Sent Done false
20632 6362 Acked Read 7 Done Sent Done false
20633 6363 Acked Write 7 Done Sent Done false
20634 6364 Acked Read 8 Done Sent Done false
20635 6365 Acked Write 5 Done Sent Done false
20636 6366 Acked Flush 0 Done Sent Done false
20637 6367 Acked Read 4 Done Sent Done false
20638 6368 Acked Read 1 Done Sent Done false
20639 6369 Acked Write 10 Done Sent Done false
20640 6370 Acked Write 2 Done Sent Done false
20641 6371 Acked Write 10 Done Sent Done false
20642 6372 Acked Write 5 Done Sent Done false
20643 6373 Acked Write 2 Done Sent Done false
20644 6374 Acked Read 6 Done Sent Done false
20645 6375 Acked Read 8 Done Sent Done false
20646 6376 Acked Read 10 Done Sent Done false
20647 6377 Acked Read 4 Done Sent Done false
20648 6378 Acked Read 9 Done Sent Done false
20649 6379 Acked Write 9 Done Sent Done false
20650 6380 Acked Read 10 Done Sent Done false
20651 6381 Acked Write 3 Done Sent Done false
20652 6382 Acked Read 3 Done Sent Done false
20653 6383 Acked Write 1 Done Sent Done false
20654 6384 Acked Flush 0 Done Sent Done false
20655 6385 Acked Write 8 Done Sent Done false
20656 6386 Acked Read 10 Done Sent Done false
20657 6387 Acked Flush 0 Done Sent Done false
20658 6388 Acked Read 9 Done Sent Done false
20659 6389 Acked Read 6 Done Sent Done false
20660 6390 Acked Write 6 Done Sent Done false
20661 6391 Acked Write 9 Done Sent Done false
20662 6392 Acked Flush 0 Done Sent Done false
20663 6393 Acked Flush 0 Done Sent Done false
20664 6394 Acked Write 7 Done Sent Done false
20665 6395 Acked Write 5 Done Sent Done false
20666 6396 Acked Write 2 Done Sent Done false
20667 6397 Acked Write 3 Done Sent Done false
20668 6398 Acked Read 4 Done Sent Done false
20669 6399 Acked Read 4 Done Sent Done false
20670 6400 Acked Write 3 Done Sent Done false
20671 6401 Acked Read 9 Done Sent Done false
20672 6402 Acked Write 6 Done Sent Done false
20673 6403 Acked Write 4 Done Sent Done false
20674 6404 Acked Read 9 Done Sent Done false
20675 6405 Acked Write 8 Done Sent Done false
20676 6406 Acked Read 8 Done Sent Done false
20677 6407 Acked Read 3 Done Sent Done false
20678 6408 Acked Write 10 Done Sent Done false
20679 6409 Acked Read 4 Done Sent Done false
20680 6410 Acked Flush 0 Done Sent Done false
20681 6411 Acked Read 1 Done Sent Done false
20682 6412 Acked Read 8 Done Sent Done false
20683 6413 Acked Write 9 Done Sent Done false
20684 6414 Acked Write 3 Done Sent Done false
20685 6415 Acked Read 2 Done Sent Done false
20686 6416 Acked Write 4 Done Sent Done false
20687 6417 Acked Write 6 Done Sent Done false
20688 6418 Acked Write 8 Done Sent Done false
20689 6419 Acked Write 9 Done Sent Done false
20690 6420 Acked Read 6 Done Sent Done false
20691 6421 Acked Read 9 Done Sent Done false
20692 6422 Acked Flush 0 Done Sent Done false
20693 6423 Acked Read 8 Done Sent Done false
20694 6424 Acked Read 7 Done Sent Done false
20695 6425 Acked Flush 0 Done Sent Done false
20696 6426 Acked Read 2 Done Sent Done false
20697 6427 Acked Flush 0 Done Sent Done false
20698 6428 Acked Read 3 Done Sent Done false
20699 6429 Acked Read 10 Done Sent Done false
20700 6430 Acked Write 4 Done Sent Done false
20701 6431 Acked Write 8 Done Sent Done false
20702 6432 Acked Read 6 Done Sent Done false
20703 6433 Acked Write 5 Done Sent Done false
20704 STATES DS:0 DS:1 DS:2 TOTAL
20705 Sent 0 301 0 301
20706 Done 301 0 301 602
20707 Skipped 0 0 0 0
20708 Error 0 0 0 0
20709 Last Flush: Some(JobId(6427)) Some(JobId(6132)) Some(JobId(6427))
20710 Downstairs last five completed: 6132 6131 6130 6129 6128
20711 Upstairs last five completed: 6433 6432 6431 6430 6429
20712 CLIENT: Up:0 ds:301 act:2
207132025-01-29T05:15:12.495ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
207142025-01-29T05:15:13.363ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
207152025-01-29T05:15:14.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
207162025-01-29T05:15:15.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
20717 ----------------------------------------------------------------
20718 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20719 States: Active Offline Active
20720 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20721 6133 Acked Read 4 Done Sent Done false
20722 6134 Acked Flush 0 Done Sent Done false
20723 6135 Acked Write 2 Done Sent Done false
20724 6136 Acked Write 3 Done Sent Done false
20725 6137 Acked Read 6 Done Sent Done false
20726 6138 Acked Write 4 Done Sent Done false
20727 6139 Acked Read 3 Done Sent Done false
20728 6140 Acked Write 9 Done Sent Done false
20729 6141 Acked Read 4 Done Sent Done false
20730 6142 Acked Write 10 Done Sent Done false
20731 6143 Acked Read 4 Done Sent Done false
20732 6144 Acked Read 6 Done Sent Done false
20733 6145 Acked Read 5 Done Sent Done false
20734 6146 Acked Read 10 Done Sent Done false
20735 6147 Acked Write 9 Done Sent Done false
20736 6148 Acked Write 1 Done Sent Done false
20737 6149 Acked Write 4 Done Sent Done false
20738 6150 Acked Read 7 Done Sent Done false
20739 6151 Acked Read 8 Done Sent Done false
20740 6152 Acked Write 6 Done Sent Done false
20741 6153 Acked Write 7 Done Sent Done false
20742 6154 Acked Read 1 Done Sent Done false
20743 6155 Acked Read 5 Done Sent Done false
20744 6156 Acked Flush 0 Done Sent Done false
20745 6157 Acked Read 7 Done Sent Done false
20746 6158 Acked Write 9 Done Sent Done false
20747 6159 Acked Read 9 Done Sent Done false
20748 6160 Acked Flush 0 Done Sent Done false
20749 6161 Acked Flush 0 Done Sent Done false
20750 6162 Acked Read 9 Done Sent Done false
20751 6163 Acked Write 5 Done Sent Done false
20752 6164 Acked Write 4 Done Sent Done false
20753 6165 Acked Write 1 Done Sent Done false
20754 6166 Acked Write 9 Done Sent Done false
20755 6167 Acked Write 6 Done Sent Done false
20756 6168 Acked Read 3 Done Sent Done false
20757 6169 Acked Write 2 Done Sent Done false
20758 6170 Acked Write 8 Done Sent Done false
20759 6171 Acked Flush 0 Done Sent Done false
20760 6172 Acked Read 3 Done Sent Done false
20761 6173 Acked Read 8 Done Sent Done false
20762 6174 Acked Read 3 Done Sent Done false
20763 6175 Acked Write 4 Done Sent Done false
20764 6176 Acked Read 9 Done Sent Done false
20765 6177 Acked Read 7 Done Sent Done false
20766 6178 Acked Flush 0 Done Sent Done false
20767 6179 Acked Read 8 Done Sent Done false
20768 6180 Acked Write 2 Done Sent Done false
20769 6181 Acked Read 6 Done Sent Done false
20770 6182 Acked Write 2 Done Sent Done false
20771 6183 Acked Write 10 Done Sent Done false
20772 6184 Acked Read 2 Done Sent Done false
20773 6185 Acked Write 2 Done Sent Done false
20774 6186 Acked Write 9 Done Sent Done false
20775 6187 Acked Write 2 Done Sent Done false
20776 6188 Acked Read 4 Done Sent Done false
20777 6189 Acked Write 4 Done Sent Done false
20778 6190 Acked Read 1 Done Sent Done false
20779 6191 Acked Read 3 Done Sent Done false
20780 6192 Acked Read 5 Done Sent Done false
20781 6193 Acked Write 3 Done Sent Done false
20782 6194 Acked Write 1 Done Sent Done false
20783 6195 Acked Flush 0 Done Sent Done false
20784 6196 Acked Read 5 Done Sent Done false
20785 6197 Acked Read 5 Done Sent Done false
20786 6198 Acked Read 10 Done Sent Done false
20787 6199 Acked Read 10 Done Sent Done false
20788 6200 Acked Read 6 Done Sent Done false
20789 6201 Acked Write 9 Done Sent Done false
20790 6202 Acked Read 8 Done Sent Done false
20791 6203 Acked Read 2 Done Sent Done false
20792 6204 Acked Write 10 Done Sent Done false
20793 6205 Acked Read 3 Done Sent Done false
20794 6206 Acked Read 9 Done Sent Done false
20795 6207 Acked Write 4 Done Sent Done false
20796 6208 Acked Read 7 Done Sent Done false
20797 6209 Acked Write 7 Done Sent Done false
20798 6210 Acked Write 6 Done Sent Done false
20799 6211 Acked Write 2 Done Sent Done false
20800 6212 Acked Write 5 Done Sent Done false
20801 6213 Acked Read 3 Done Sent Done false
20802 6214 Acked Read 4 Done Sent Done false
20803 6215 Acked Write 1 Done Sent Done false
20804 6216 Acked Read 3 Done Sent Done false
20805 6217 Acked Read 10 Done Sent Done false
20806 6218 Acked Write 5 Done Sent Done false
20807 6219 Acked Read 1 Done Sent Done false
20808 6220 Acked Read 1 Done Sent Done false
20809 6221 Acked Read 3 Done Sent Done false
20810 6222 Acked Write 6 Done Sent Done false
20811 6223 Acked Flush 0 Done Sent Done false
20812 6224 Acked Read 8 Done Sent Done false
20813 6225 Acked Write 6 Done Sent Done false
20814 6226 Acked Write 6 Done Sent Done false
20815 6227 Acked Write 2 Done Sent Done false
20816 6228 Acked Write 3 Done Sent Done false
20817 6229 Acked Read 3 Done Sent Done false
20818 6230 Acked Write 7 Done Sent Done false
20819 6231 Acked Write 5 Done Sent Done false
20820 6232 Acked Read 10 Done Sent Done false
20821 6233 Acked Write 9 Done Sent Done false
20822 6234 Acked Read 10 Done Sent Done false
20823 6235 Acked Write 1 Done Sent Done false
20824 6236 Acked Read 8 Done Sent Done false
20825 6237 Acked Write 1 Done Sent Done false
20826 6238 Acked Read 8 Done Sent Done false
20827 6239 Acked Write 3 Done Sent Done false
20828 6240 Acked Write 5 Done Sent Done false
20829 6241 Acked Read 9 Done Sent Done false
20830 6242 Acked Read 5 Done Sent Done false
20831 6243 Acked Write 1 Done Sent Done false
20832 6244 Acked Write 10 Done Sent Done false
20833 6245 Acked Write 10 Done Sent Done false
20834 6246 Acked Write 4 Done Sent Done false
20835 6247 Acked Write 4 Done Sent Done false
20836 6248 Acked Read 6 Done Sent Done false
20837 6249 Acked Read 7 Done Sent Done false
20838 6250 Acked Read 4 Done Sent Done false
20839 6251 Acked Flush 0 Done Sent Done false
20840 6252 Acked Write 6 Done Sent Done false
20841 6253 Acked Read 4 Done Sent Done false
20842 6254 Acked Read 8 Done Sent Done false
20843 6255 Acked Flush 0 Done Sent Done false
20844 6256 Acked Write 10 Done Sent Done false
20845 6257 Acked Write 10 Done Sent Done false
20846 6258 Acked Write 10 Done Sent Done false
20847 6259 Acked Read 4 Done Sent Done false
20848 6260 Acked Write 8 Done Sent Done false
20849 6261 Acked Read 9 Done Sent Done false
20850 6262 Acked Flush 0 Done Sent Done false
20851 6263 Acked Write 4 Done Sent Done false
20852 6264 Acked Read 2 Done Sent Done false
20853 6265 Acked Read 10 Done Sent Done false
20854 6266 Acked Read 6 Done Sent Done false
20855 6267 Acked Write 8 Done Sent Done false
20856 6268 Acked Read 2 Done Sent Done false
20857 6269 Acked Write 9 Done Sent Done false
20858 6270 Acked Write 10 Done Sent Done false
20859 6271 Acked Write 2 Done Sent Done false
20860 6272 Acked Read 7 Done Sent Done false
20861 6273 Acked Read 9 Done Sent Done false
20862 6274 Acked Read 10 Done Sent Done false
20863 6275 Acked Read 10 Done Sent Done false
20864 6276 Acked Read 7 Done Sent Done false
20865 6277 Acked Read 2 Done Sent Done false
20866 6278 Acked Read 3 Done Sent Done false
20867 6279 Acked Read 10 Done Sent Done false
20868 6280 Acked Read 10 Done Sent Done false
20869 6281 Acked Read 9 Done Sent Done false
20870 6282 Acked Read 6 Done Sent Done false
20871 6283 Acked Write 2 Done Sent Done false
20872 6284 Acked Read 3 Done Sent Done false
20873 6285 Acked Write 9 Done Sent Done false
20874 6286 Acked Write 10 Done Sent Done false
20875 6287 Acked Read 9 Done Sent Done false
20876 6288 Acked Write 5 Done Sent Done false
20877 6289 Acked Flush 0 Done Sent Done false
20878 6290 Acked Write 1 Done Sent Done false
20879 6291 Acked Read 8 Done Sent Done false
20880 6292 Acked Read 7 Done Sent Done false
20881 6293 Acked Flush 0 Done Sent Done false
20882 6294 Acked Flush 0 Done Sent Done false
20883 6295 Acked Read 1 Done Sent Done false
20884 6296 Acked Write 1 Done Sent Done false
20885 6297 Acked Read 7 Done Sent Done false
20886 6298 Acked Read 1 Done Sent Done false
20887 6299 Acked Flush 0 Done Sent Done false
20888 6300 Acked Write 10 Done Sent Done false
20889 6301 Acked Write 6 Done Sent Done false
20890 6302 Acked Write 7 Done Sent Done false
20891 6303 Acked Read 6 Done Sent Done false
20892 6304 Acked Read 3 Done Sent Done false
20893 6305 Acked Write 8 Done Sent Done false
20894 6306 Acked Flush 0 Done Sent Done false
20895 6307 Acked Read 3 Done Sent Done false
20896 6308 Acked Read 4 Done Sent Done false
20897 6309 Acked Read 5 Done Sent Done false
20898 6310 Acked Read 5 Done Sent Done false
20899 6311 Acked Read 10 Done Sent Done false
20900 6312 Acked Read 6 Done Sent Done false
20901 6313 Acked Write 5 Done Sent Done false
20902 6314 Acked Write 2 Done Sent Done false
20903 6315 Acked Flush 0 Done Sent Done false
20904 6316 Acked Write 4 Done Sent Done false
20905 6317 Acked Write 1 Done Sent Done false
20906 6318 Acked Read 9 Done Sent Done false
20907 6319 Acked Flush 0 Done Sent Done false
20908 6320 Acked Read 8 Done Sent Done false
20909 6321 Acked Flush 0 Done Sent Done false
20910 6322 Acked Read 5 Done Sent Done false
20911 6323 Acked Read 7 Done Sent Done false
20912 6324 Acked Write 7 Done Sent Done false
20913 6325 Acked Flush 0 Done Sent Done false
20914 6326 Acked Read 8 Done Sent Done false
20915 6327 Acked Read 3 Done Sent Done false
20916 6328 Acked Write 3 Done Sent Done false
20917 6329 Acked Write 4 Done Sent Done false
20918 6330 Acked Flush 0 Done Sent Done false
20919 6331 Acked Write 10 Done Sent Done false
20920 6332 Acked Read 6 Done Sent Done false
20921 6333 Acked Read 6 Done Sent Done false
20922 6334 Acked Read 10 Done Sent Done false
20923 6335 Acked Write 2 Done Sent Done false
20924 6336 Acked Write 9 Done Sent Done false
20925 6337 Acked Read 3 Done Sent Done false
20926 6338 Acked Write 10 Done Sent Done false
20927 6339 Acked Write 3 Done Sent Done false
20928 6340 Acked Read 1 Done Sent Done false
20929 6341 Acked Read 4 Done Sent Done false
20930 6342 Acked Write 8 Done Sent Done false
20931 6343 Acked Read 6 Done Sent Done false
20932 6344 Acked Write 3 Done Sent Done false
20933 6345 Acked Write 10 Done Sent Done false
20934 6346 Acked Write 9 Done Sent Done false
20935 6347 Acked Flush 0 Done Sent Done false
20936 6348 Acked Read 10 Done Sent Done false
20937 6349 Acked Flush 0 Done Sent Done false
20938 6350 Acked Write 7 Done Sent Done false
20939 6351 Acked Read 7 Done Sent Done false
20940 6352 Acked Flush 0 Done Sent Done false
20941 6353 Acked Write 2 Done Sent Done false
20942 6354 Acked Read 2 Done Sent Done false
20943 6355 Acked Read 5 Done Sent Done false
20944 6356 Acked Write 5 Done Sent Done false
20945 6357 Acked Write 5 Done Sent Done false
20946 6358 Acked Write 1 Done Sent Done false
20947 6359 Acked Read 10 Done Sent Done false
20948 6360 Acked Read 2 Done Sent Done false
20949 6361 Acked Read 6 Done Sent Done false
20950 6362 Acked Read 7 Done Sent Done false
20951 6363 Acked Write 7 Done Sent Done false
20952 6364 Acked Read 8 Done Sent Done false
20953 6365 Acked Write 5 Done Sent Done false
20954 6366 Acked Flush 0 Done Sent Done false
20955 6367 Acked Read 4 Done Sent Done false
20956 6368 Acked Read 1 Done Sent Done false
20957 6369 Acked Write 10 Done Sent Done false
20958 6370 Acked Write 2 Done Sent Done false
20959 6371 Acked Write 10 Done Sent Done false
20960 6372 Acked Write 5 Done Sent Done false
20961 6373 Acked Write 2 Done Sent Done false
20962 6374 Acked Read 6 Done Sent Done false
20963 6375 Acked Read 8 Done Sent Done false
20964 6376 Acked Read 10 Done Sent Done false
20965 6377 Acked Read 4 Done Sent Done false
20966 6378 Acked Read 9 Done Sent Done false
20967 6379 Acked Write 9 Done Sent Done false
20968 6380 Acked Read 10 Done Sent Done false
20969 6381 Acked Write 3 Done Sent Done false
20970 6382 Acked Read 3 Done Sent Done false
20971 6383 Acked Write 1 Done Sent Done false
20972 6384 Acked Flush 0 Done Sent Done false
20973 6385 Acked Write 8 Done Sent Done false
20974 6386 Acked Read 10 Done Sent Done false
20975 6387 Acked Flush 0 Done Sent Done false
20976 6388 Acked Read 9 Done Sent Done false
20977 6389 Acked Read 6 Done Sent Done false
20978 6390 Acked Write 6 Done Sent Done false
20979 6391 Acked Write 9 Done Sent Done false
20980 6392 Acked Flush 0 Done Sent Done false
20981 6393 Acked Flush 0 Done Sent Done false
20982 6394 Acked Write 7 Done Sent Done false
20983 6395 Acked Write 5 Done Sent Done false
20984 6396 Acked Write 2 Done Sent Done false
20985 6397 Acked Write 3 Done Sent Done false
20986 6398 Acked Read 4 Done Sent Done false
20987 6399 Acked Read 4 Done Sent Done false
20988 6400 Acked Write 3 Done Sent Done false
20989 6401 Acked Read 9 Done Sent Done false
20990 6402 Acked Write 6 Done Sent Done false
20991 6403 Acked Write 4 Done Sent Done false
20992 6404 Acked Read 9 Done Sent Done false
20993 6405 Acked Write 8 Done Sent Done false
20994 6406 Acked Read 8 Done Sent Done false
20995 6407 Acked Read 3 Done Sent Done false
20996 6408 Acked Write 10 Done Sent Done false
20997 6409 Acked Read 4 Done Sent Done false
20998 6410 Acked Flush 0 Done Sent Done false
20999 6411 Acked Read 1 Done Sent Done false
21000 6412 Acked Read 8 Done Sent Done false
21001 6413 Acked Write 9 Done Sent Done false
21002 6414 Acked Write 3 Done Sent Done false
21003 6415 Acked Read 2 Done Sent Done false
21004 6416 Acked Write 4 Done Sent Done false
21005 6417 Acked Write 6 Done Sent Done false
21006 6418 Acked Write 8 Done Sent Done false
21007 6419 Acked Write 9 Done Sent Done false
21008 6420 Acked Read 6 Done Sent Done false
21009 6421 Acked Read 9 Done Sent Done false
21010 6422 Acked Flush 0 Done Sent Done false
21011 6423 Acked Read 8 Done Sent Done false
21012 6424 Acked Read 7 Done Sent Done false
21013 6425 Acked Flush 0 Done Sent Done false
21014 6426 Acked Read 2 Done Sent Done false
21015 6427 Acked Flush 0 Done Sent Done false
21016 6428 Acked Read 3 Done Sent Done false
21017 6429 Acked Read 10 Done Sent Done false
21018 6430 Acked Write 4 Done Sent Done false
21019 6431 Acked Write 8 Done Sent Done false
21020 6432 Acked Read 6 Done Sent Done false
21021 6433 Acked Write 5 Done Sent Done false
21022 STATES DS:0 DS:1 DS:2 TOTAL
21023 Sent 0 301 0 301
21024 Done 301 0 301 602
21025 Skipped 0 0 0 0
21026 Error 0 0 0 0
21027 Last Flush: Some(JobId(6427)) Some(JobId(6132)) Some(JobId(6427))
21028 Downstairs last five completed: 6132 6131 6130 6129 6128
21029 Upstairs last five completed: 6433 6432 6431 6430 6429
21030 CLIENT: Up:0 ds:301 act:2
210312025-01-29T05:15:16.498ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210322025-01-29T05:15:17.371ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210332025-01-29T05:15:18.373ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210342025-01-29T05:15:19.374ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210352025-01-29T05:15:19.745ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210362025-01-29T05:15:19.745ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41751) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210372025-01-29T05:15:19.746ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210382025-01-29T05:15:19.746ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210392025-01-29T05:15:19.746ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210402025-01-29T05:15:19.746ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210412025-01-29T05:15:19.746ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210422025-01-29T05:15:19.746ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
210432025-01-29T05:15:19.747ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21044 ----------------------------------------------------------------
21045 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:6
21046 States: Active Active Active
21047 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21048 6428 Acked Read 3 Done Done Done true
21049 6429 Acked Read 10 Done Done Done true
21050 6430 Acked Write 4 Done Done Done true
21051 6431 Acked Write 8 Done Done Done true
21052 6432 Acked Read 6 Done Done Done true
21053 6433 Acked Write 5 Done Done Done true
21054 STATES DS:0 DS:1 DS:2 TOTAL
21055 Sent 0 0 0 0
21056 Done 6 6 6 18
21057 Skipped 0 0 0 0
21058 Error 0 0 0 0
21059 Last Flush: Some(JobId(6427)) Some(JobId(6427)) Some(JobId(6427))
21060 Downstairs last five completed: 6427 6426 6425 6424 6423
21061 Upstairs last five completed: 6433 6432 6431 6430 6429
21062 CLIENT: Up:0 ds:6 act:3
21063 ----------------------------------------------------------------
21064 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21065 Downstairs last five completed: 6434 6433 6432 6431 6430
21066 Upstairs last five completed: 6434 6433 6432 6431 6430
21067 CLIENT: Up:0 ds:0 act:3
21068 Replay: All jobs finished
21069 Main task has work to do, go find it
21070 got dsc Stop(2)
21071 stop 2
21072 Jan 29 05:15:24.000 INFO request completed, [8830] Got stop action so:false kr:false
21073 latency_us: 310, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ad53b80f-d175-4a06-b09e-dbf098b2f78d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
21074 Jan 29 05:15:24.001{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:15:24.001286815Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
21075 INFO request completed, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:15:24.001369499Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
21076 latency_us: 201, response_code: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:15:24.001397137Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
21077 200, uri: /state/cid/2, method: GET, req_id: 5153d880-382e-4a4e-8179-4f4141782f6d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
210782025-01-29T05:15:24.001ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21079 [8830] Exited with: ExitStatus(unix_wait_status(9))
21080 [8830][2] reports Exit
21081 Jan 29 05:15:28.003 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: b468c50f-6081-4db8-b659-c2217c13027d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
21082 001/300 Flush
21083 002/300 Read block 4900 len 16384
21084 003/300 Write block 2686 len 36864 data: 4 5 4 3 2 2 3 3 3
21085 004/300 Write block 823 len 16384 data: 3 3 3 3
21086 005/300 Read block 3272 len 4096
21087 006/300 Write block 4189 len 36864 data: 4 4 4 2 3 3 4 5 6
21088 007/300 Flush
21089 008/300 Read block 4863 len 36864
21090 009/300 Read block 3944 len 20480
21091 010/300 Flush
21092 011/300 Write block 829 len 12288 data: 3 3 4
21093 012/300 Read block 395 len 32768
21094 013/300 Write block 883 len 24576 data: 5 7 7 6 6 5
21095 014/300 Write block 4839 len 8192 data: 5 6
21096 015/300 Read block 3738 len 36864
21097 016/300 Write block 1810 len 4096 data: 5
21098 017/300 Write block 2019 len 24576 data: 4 6 4 4 4 5
21099 018/300 Write block 1313 len 32768 data: 3 3 4 5 6 5 3 3
21100 019/300 Read block 3408 len 28672
21101 020/300 Read block 4515 len 36864
21102 021/300 Read block 3759 len 28672
21103 022/300 Flush
21104 023/300 Read block 446 len 32768
21105 024/300 Flush
21106 025/300 Read block 3050 len 36864
21107 026/300 Read block 4555 len 4096
21108 027/300 Write block 4601 len 36864 data: 7 7 9 10 7 5 4 3 3
21109 028/300 Write block 3692 len 20480 data: 4 4 4 4 3
21110 029/300 Read block 1505 len 36864
21111 030/300 Read block 3024 len 36864
21112 031/300 Read block 3050 len 8192
21113 032/300 Write block 3801 len 36864 data: 6 6 5 4 6 6 6 6 4
21114 033/300 Write block 3939 len 24576 data: 9 7 7 8 5 6
21115 034/300 Read block 4022 len 20480
21116 035/300 Write block 4097 len 24576 data: 3 4 4 5 6 7
21117 036/300 Read block 1785 len 40960
21118 037/300 Write block 4002 len 16384 data: 5 6 7 6
21119 038/300 Write block 3593 len 16384 data: 4 3 3 3
21120 039/300 Read block 3906 len 4096
21121 040/300 Write block 190 len 12288 data: 4 4 5
21122 041/300 Read block 397 len 40960
21123 042/300 Read block 1101 len 40960
21124 043/300 Write block 3233 len 4096 data: 8
21125 044/300 Read block 2055 len 36864
21126 045/300 Write block 3891 len 20480 data: 4 4 4 4 4
21127 046/300 Read block 2631 len 36864
21128 047/300 Read block 3514 len 20480
21129 048/300 Write block 4958 len 8192 data: 7 7
21130 049/300 Read block 735 len 28672
21131 050/300 Write block 930 len 8192 data: 3 4
21132 051/300 Write block 2630 len 4096 data: 3
21133 052/300 Read block 1751 len 20480
21134 053/300 Write block 3102 len 20480 data: 4 4 4 5 5
21135 054/300 Write block 2133 len 28672 data: 2 2 3 4 3 4 5
21136 055/300 Write block 3873 len 40960 data: 3 4 5 5 6 6 5 5 5 5
21137 056/300 Read block 2440 len 8192
21138 057/300 Read block 4822 len 28672
21139 058/300 Write block 4238 len 16384 data: 4 4 4 3
21140 059/300 Write block 3695 len 28672 data: 5 4 3 2 4 4 4
21141 060/300 Read block 2515 len 28672
21142 061/300 Write block 2319 len 8192 data: 4 4
21143 062/300 Write block 1578 len 40960 data: 2 3 3 4 4 4 5 5 5 4
21144 063/300 Write block 1842 len 32768 data: 8 6 5 5 5 4 3 2
21145 064/300 Read block 381 len 20480
21146 065/300 Flush
21147 066/300 Flush
21148 067/300 Read block 2814 len 20480
21149 068/300 Read block 110 len 32768
21150 069/300 Write block 2100 len 36864 data: 4 2 3 5 4 4 4 5 5
21151 070/300 Write block 2803 len 20480 data: 3 3 4 3 3
21152 071/300 Write block 4060 len 8192 data: 3 4
21153 072/300 Read block 1317 len 36864
21154 073/300 Read block 3007 len 32768
21155 074/300 Write block 4069 len 28672 data: 3 3 3 4 5 6 5
21156 075/300 Write block 1726 len 24576 data: 4 4 3 3 3 3
21157 076/300 Write block 2818 len 36864 data: 4 4 4 4 5 4 5 4 5
21158 077/300 Read block 342 len 20480
21159 078/300 Read block 4034 len 4096
21160 079/300 Read block 2117 len 28672
21161 080/300 Read block 1148 len 20480
21162 081/300 Read block 815 len 12288
21163 082/300 Flush
21164 083/300 Write block 4743 len 8192 data: 6 6
21165 084/300 Write block 4927 len 20480 data: 2 3 4 4 4
21166 085/300 Flush
21167 086/300 Read block 4565 len 36864
21168 087/300 Write block 4786 len 4096 data: 4
21169 088/300 Read block 1502 len 40960
21170 089/300 Write block 648 len 12288 data: 4 5 5
21171 090/300 Write block 1004 len 12288 data: 4 4 2
21172 091/300 Write block 2203 len 24576 data: 4 5 5 5 4 5
21173 092/300 Write block 2051 len 16384 data: 5 5 5 5
21174 093/300 Flush
21175 094/300 Read block 4179 len 12288
21176 095/300 Read block 236 len 28672
21177 096/300 Flush
21178 097/300 Flush
21179 098/300 Read block 2997 len 40960
21180 099/300 Write block 2739 len 16384 data: 7 5 5 6
21181 100/300 Write block 482 len 4096 data: 10
21182 101/300 Write block 928 len 20480 data: 2 2 4 5 4
21183 102/300 Write block 773 len 12288 data: 4 5 5
21184 103/300 Read block 3027 len 32768
21185 104/300 Read block 1555 len 36864
21186 105/300 Read block 3097 len 20480
21187 106/300 Write block 1915 len 36864 data: 4 3 3 4 4 4 5 5 6
21188 107/300 Write block 2839 len 8192 data: 6 6
21189 108/300 Write block 1120 len 36864 data: 4 4 5 6 7 7 6 6 5
21190 109/300 Read block 3993 len 24576
21191 110/300 Read block 2165 len 24576
21192 111/300 Read block 2177 len 36864
21193 112/300 Write block 59 len 12288 data: 6 5 5
21194 113/300 Write block 3214 len 40960 data: 2 2 5 4 4 6 7 5 6 6
21195 114/300 Read block 293 len 36864
21196 115/300 Read block 4718 len 20480
21197 116/300 Read block 1201 len 12288
21198 117/300 Read block 106 len 28672
21199 118/300 Read block 3498 len 20480
21200 119/300 Read block 2252 len 4096
21201 120/300 Write block 1936 len 28672 data: 6 7 6 7 8 6 5
21202 121/300 Read block 3584 len 36864
21203 122/300 Read block 2646 len 16384
21204 123/300 Read block 4842 len 40960
21205 124/300 Read block 2934 len 40960
21206 125/300 Read block 3766 len 32768
21207 126/300 Write block 872 len 24576 data: 7 7 5 4 4 5
21208 127/300 Write block 1514 len 12288 data: 4 3 4
21209 128/300 Read block 958 len 20480
21210 129/300 Write block 3872 len 20480 data: 3 4 5 6 6
21211 130/300 Write block 1447 len 4096 data: 5
21212 131/300 Read block 3129 len 40960
21213 132/300 Read block 580 len 12288
21214 133/300 Read block 4799 len 20480
21215 134/300 Write block 2608 len 16384 data: 7 7 6 6
21216 135/300 Write block 764 len 16384 data: 3 4 5 4
21217 136/300 Read block 2146 len 12288
21218 137/300 Write block 3057 len 40960 data: 3 3 3 3 3 3 4 4 4 3
21219 138/300 Read block 4147 len 28672
21220 139/300 Write block 3613 len 12288 data: 6 6 5
21221 140/300 Read block 826 len 12288
21222 141/300 Read block 4902 len 32768
21223 142/300 Write block 3966 len 32768 data: 2 2 2 3 5 5 5 5
21224 143/300 Write block 590 len 24576 data: 7 5 5 5 6 6
21225 144/300 Write block 866 len 24576 data: 6 6 7 6 7 7
21226 145/300 Write block 2217 len 20480 data: 3 3 4 4 4
21227 146/300 Read block 4205 len 24576
21228 147/300 Write block 3531 len 20480 data: 5 5 5 5 4
21229 148/300 Read block 1802 len 8192
21230 149/300 Write block 1497 len 32768 data: 6 5 4 4 6 6 6 5
21231 150/300 Read block 2097 len 12288
21232 151/300 Flush
21233 152/300 Write block 205 len 16384 data: 2 2 2 2
21234 153/300 Write block 2727 len 40960 data: 5 5 5 5 7 7 7 8 8 7
21235 154/300 Write block 2020 len 40960 data: 7 5 5 5 6 4 5 5 4 5
21236 155/300 Read block 2329 len 12288
21237 156/300 Read block 1778 len 32768
21238 157/300 Read block 3634 len 20480
21239 158/300 Read block 3829 len 24576
21240 159/300 Read block 1985 len 20480
21241 160/300 Read block 832 len 4096
21242 161/300 Read block 276 len 40960
21243 162/300 Flush
21244 163/300 Read block 1350 len 40960
21245 164/300 Read block 2310 len 12288
21246 165/300 Write block 1791 len 36864 data: 5 7 6 6 7 5 5 4 3
21247 166/300 Read block 4683 len 32768
21248 167/300 Write block 35 len 12288 data: 5 5 4
21249 168/300 Write block 496 len 28672 data: 7 7 8 6 6 4 7
21250 169/300 Write block 2617 len 40960 data: 3 4 4 4 4 4 4 4 3 3
21251 170/300 Write block 634 len 16384 data: 4 5 5 6
21252 171/300 Read block 103 len 32768
21253 172/300 Write block 1229 len 28672 data: 3 4 4 4 6 6 5
21254 173/300 Read block 1655 len 8192
21255 174/300 Read block 4651 len 24576
21256 175/300 Read block 4271 len 32768
21257 176/300 Flush
21258 177/300 Write block 4987 len 36864 data: 5 5 6 6 6 7 5 5 4
21259 178/300 Write block 4105 len 36864 data: 4 4 2 2 2 4 3 3 3
21260 179/300 Write block 2431 len 20480 data: 3 3 4 5 5
21261 180/300 Write block 4762 len 40960 data: 5 4 3 3 4 4 5 5 7 7
21262 181/300 Read block 3151 len 40960
21263 182/300 Read block 1876 len 8192
21264 183/300 Read block 61 len 32768
21265 184/300 Flush
21266 185/300 Write block 1711 len 12288 data: 5 5 6
21267 186/300 Read block 3247 len 36864
21268 187/300 Read block 279 len 4096
21269 188/300 Write block 3191 len 24576 data: 2 2 2 3 4 4
21270 189/300 Write block 4245 len 20480 data: 2 2 2 2 3
21271 190/300 Flush
21272 191/300 Flush
21273 192/300 Read block 2528 len 32768
21274 193/300 Read block 4756 len 32768
21275 194/300 Write block 1424 len 40960 data: 5 5 5 5 5 5 4 4 5 4
21276 195/300 Read block 3413 len 12288
21277 196/300 Read block 1956 len 32768
21278 197/300 Write block 3029 len 4096 data: 3
21279 198/300 Write block 3385 len 32768 data: 6 5 6 5 6 5 5 4
21280 199/300 Read block 2606 len 4096
21281 200/300 Read block 2121 len 12288
21282 201/300 Read block 1561 len 24576
21283 202/300 Read block 4335 len 32768
21284 203/300 Read block 4511 len 8192
21285 204/300 Read block 3751 len 4096
21286 205/300 Read block 4661 len 8192
21287 206/300 Write block 3258 len 16384 data: 3 4 5 4
21288 207/300 Read block 352 len 32768
21289 208/300 Read block 2410 len 12288
21290 209/300 Flush
21291 210/300 Read block 1332 len 12288
21292 211/300 Read block 1843 len 24576
21293 212/300 Flush
21294 213/300 Read block 3265 len 4096
21295 214/300 Write block 189 len 20480 data: 5 5 5 6 5
21296 215/300 Read block 1045 len 20480
21297 216/300 Read block 901 len 36864
21298 217/300 Read block 971 len 16384
21299 218/300 Write block 3002 len 12288 data: 6 6 5
21300 219/300 Write block 197 len 28672 data: 6 5 3 3 3 3 2
21301 220/300 Write block 1292 len 4096 data: 4
21302 221/300 Write block 1327 len 8192 data: 4 5
21303 222/300 Read block 2462 len 40960
21304 223/300 Write block 98 len 4096 data: 5
21305 224/300 Write block 3102 len 12288 data: 5 5 5
21306 225/300 Read block 4221 len 20480
21307 226/300 Read block 4828 len 28672
21308 227/300 Write block 2280 len 24576 data: 4 4 4 4 4 2
21309 228/300 Read block 2093 len 12288
21310 229/300 Flush
21311 230/300 Write block 106 len 8192 data: 4 4
21312 231/300 Read block 637 len 20480
21313 232/300 Write block 2559 len 20480 data: 4 3 3 3 4
21314 233/300 Write block 1520 len 36864 data: 4 5 4 4 4 4 4 5 4
21315 234/300 Read block 409 len 20480
21316 235/300 Write block 1066 len 40960 data: 2 3 3 3 4 4 4 5 5 4
21317 236/300 Write block 1159 len 32768 data: 2 3 4 4 3 4 4 4
21318 237/300 Read block 207 len 20480
21319 238/300 Flush
21320 239/300 Read block 1277 len 4096
21321 240/300 Read block 1786 len 28672
21322 241/300 Flush
21323 242/300 Read block 3184 len 40960
21324 243/300 Write block 4237 len 12288 data: 3 5 5
21325 244/300 Write block 1056 len 28672 data: 3 2 2 3 3 3 3
21326 245/300 Read block 4223 len 20480
21327 246/300 Read block 1846 len 4096
21328 247/300 Read block 2551 len 12288
21329 248/300 Write block 490 len 20480 data: 5 7 7 7 7
21330 249/300 Read block 3346 len 36864
21331 250/300 Read block 4422 len 24576
21332 251/300 Read block 1544 len 8192
21333 252/300 Read block 2586 len 36864
21334 253/300 Read block 2521 len 40960
21335 254/300 Read block 1635 len 4096
21336 255/300 Flush
21337 256/300 Write block 1544 len 24576 data: 4 4 5 5 5 4
21338 257/300 Write block 3522 len 16384 data: 5 5 5 5
21339 258/300 Read block 697 len 12288
21340 259/300 Flush
21341 260/300 Read block 4465 len 40960
21342 261/300 Read block 4668 len 12288
21343 262/300 Write block 4155 len 12288 data: 3 3 2
21344 263/300 Write block 1760 len 32768 data: 5 4 4 4 5 5 5 4
21345 264/300 Read block 3298 len 28672
21346 265/300 Flush
21347 266/300 Write block 3883 len 20480 data: 5 5 4 4 2
21348 267/300 Write block 3601 len 12288 data: 2 2 2
21349 268/300 Write block 1900 len 4096 data: 2
21350 269/300 Write block 2378 len 40960 data: 5 5 6 6 8 5 5 4 3 2
21351 270/300 Write block 1903 len 24576 data: 3 3 3 5 6 6
21352 271/300 Flush
21353 272/300 Read block 2108 len 16384
21354 273/300 Write block 826 len 32768 data: 4 3 2 4 4 5 4 4
21355 274/300 Read block 2284 len 32768
21356 275/300 Read block 3099 len 12288
21357 276/300 Write block 3788 len 28672 data: 4 4 4 4 5 5 3
21358 277/300 Read block 3199 len 16384
21359 278/300 Write block 236 len 36864 data: 5 7 8 8 8 9 8 7 7
21360 279/300 Write block 4513 len 24576 data: 2 2 4 4 5 4
21361 280/300 Write block 2512 len 8192 data: 5 4
21362 281/300 Flush
21363 282/300 Read block 1674 len 4096
21364 283/300 Flush
21365 284/300 Flush
21366 285/300 Read block 3100 len 8192
21367 286/300 Flush
21368 287/300 Read block 3123 len 12288
21369 288/300 Flush
21370 289/300 Flush
21371 290/300 Read block 2591 len 12288
21372 291/300 Read block 1724 len 20480
21373 292/300 Write block 1385 len 16384 data: 5 5 5 5
21374 293/300 Read block 2255 len 40960
21375 294/300 Read block 3184 len 8192
21376 295/300 Read block 12 len 40960
21377 296/300 Read block 1692 len 24576
21378 297/300 Read block 474 len 36864
21379 298/300 Read block 2910 len 32768
21380 299/300 Read block 3961 len 20480
21381 300/300 Read block 1282 len 36864
21382 301/300 Write block 2078 len 28672 data: 3 4 3 3 3 4 5
21383 Main task has work to do, go find it
21384 got dsc Start(2)
21385 start 2
21386 Jan 29 05:15:28.247 INFO request completed, latency_us: 237, response_code: 204, uri: /start/cid/2, method: POST, req_id: 4d25cf11-f24b-4cb3-8d99-11302843ff56, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
21387 Replay: started 2, returned:Ok(())
21388 ----------------------------------------------------------------
21389 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21390 States: Active Active Offline
21391 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21392 6435 Acked Flush 0 Done Done Sent false
21393 6436 Acked Read 4 Done Done Sent false
21394 6437 Acked Write 9 Done Done Sent false
21395 6438 Acked Write 4 Done Done Sent false
21396 6439 Acked Read 1 Done Done Sent false
21397 6440 Acked Write 9 Done Done Sent false
21398 6441 Acked Flush 0 Done Done Sent false
21399 6442 Acked Read 9 Done Done Sent false
21400 6443 Acked Read 5 Done Done Sent false
21401 6444 Acked Flush 0 Done Done Sent false
21402 6445 Acked Write 3 Done Done Sent false
21403 6446 Acked Read 8 Done Done Sent false
21404 6447 Acked Write 6 Done Done Sent false
21405 6448 Acked Write 2 Done Done Sent false
21406 6449 Acked Read 9 Done Done Sent false
21407 6450 Acked Write 1 Done Done Sent false
21408 6451 Acked Write 6 Done Done Sent false
21409 6452 Acked Write 8 Done Done Sent false
21410 6453 Acked Read 7 Done Done Sent false
21411 6454 Acked Read 9 Done Done Sent false
21412 6455 Acked Read 7 Done Done Sent false
21413 6456 Acked Flush 0 Done Done Sent false
21414 6457 Acked Read 8 Done Done Sent false
21415 6458 Acked Flush 0 Done Done Sent false
21416 6459 Acked Read 9 Done Done Sent false
21417 6460 Acked Read 1 Done Done Sent false
21418 6461 Acked Write 9 Done Done Sent false
21419 6462 Acked Write 5 Done Done Sent false
21420 6463 Acked Read 9 Done Done Sent false
21421 6464 Acked Read 9 Done Done Sent false
21422 6465 Acked Read 2 Done Done Sent false
21423 6466 Acked Write 9 Done Done Sent false
21424 6467 Acked Write 6 Done Done Sent false
21425 6468 Acked Read 5 Done Done Sent false
21426 6469 Acked Write 6 Done Done Sent false
21427 6470 Acked Read 10 Done Done Sent false
21428 6471 Acked Write 4 Done Done Sent false
21429 6472 Acked Write 4 Done Done Sent false
21430 6473 Acked Read 1 Done Done Sent false
21431 6474 Acked Write 3 Done Done Sent false
21432 6475 Acked Read 10 Done Done Sent false
21433 6476 Acked Read 10 Done Done Sent false
21434 6477 Acked Write 1 Done Done Sent false
21435 6478 Acked Read 9 Done Done Sent false
21436 6479 Acked Write 5 Done Done Sent false
21437 6480 Acked Read 9 Done Done Sent false
21438 6481 Acked Read 5 Done Done Sent false
21439 6482 Acked Write 2 Done Done Sent false
21440 6483 Acked Read 7 Done Done Sent false
21441 6484 Acked Write 2 Done Done Sent false
21442 6485 Acked Write 1 Done Done Sent false
21443 6486 Acked Read 5 Done Done Sent false
21444 6487 Acked Write 5 Done Done Sent false
21445 6488 Acked Write 7 Done Done Sent false
21446 6489 Acked Write 10 Done Done Sent false
21447 6490 Acked Read 2 Done Done Sent false
21448 6491 Acked Read 7 Done Done Sent false
21449 6492 Acked Write 4 Done Done Sent false
21450 6493 Acked Write 7 Done Done Sent false
21451 6494 Acked Read 7 Done Done Sent false
21452 6495 Acked Write 2 Done Done Sent false
21453 6496 Acked Write 10 Done Done Sent false
21454 6497 Acked Write 8 Done Done Sent false
21455 6498 Acked Read 5 Done Done Sent false
21456 6499 Acked Flush 0 Done Done Sent false
21457 6500 Acked Flush 0 Done Done Sent false
21458 6501 Acked Read 5 Done Done Sent false
21459 6502 Acked Read 8 Done Done Sent false
21460 6503 Acked Write 9 Done Done Sent false
21461 6504 Acked Write 5 Done Done Sent false
21462 6505 Acked Write 2 Done Done Sent false
21463 6506 Acked Read 9 Done Done Sent false
21464 6507 Acked Read 8 Done Done Sent false
21465 6508 Acked Write 7 Done Done Sent false
21466 6509 Acked Write 6 Done Done Sent false
21467 6510 Acked Write 9 Done Done Sent false
21468 6511 Acked Read 5 Done Done Sent false
21469 6512 Acked Read 1 Done Done Sent false
21470 6513 Acked Read 7 Done Done Sent false
21471 6514 Acked Read 5 Done Done Sent false
21472 6515 Acked Read 3 Done Done Sent false
21473 6516 Acked Flush 0 Done Done Sent false
21474 6517 Acked Write 2 Done Done Sent false
21475 6518 Acked Write 5 Done Done Sent false
21476 6519 Acked Flush 0 Done Done Sent false
21477 6520 Acked Read 9 Done Done Sent false
21478 6521 Acked Write 1 Done Done Sent false
21479 6522 Acked Read 10 Done Done Sent false
21480 6523 Acked Write 3 Done Done Sent false
21481 6524 Acked Write 3 Done Done Sent false
21482 6525 Acked Write 6 Done Done Sent false
21483 6526 Acked Write 4 Done Done Sent false
21484 6527 Acked Flush 0 Done Done Sent false
21485 6528 Acked Read 3 Done Done Sent false
21486 6529 Acked Read 7 Done Done Sent false
21487 6530 Acked Flush 0 Done Done Sent false
21488 6531 Acked Flush 0 Done Done Sent false
21489 6532 Acked Read 10 Done Done Sent false
21490 6533 Acked Write 4 Done Done Sent false
21491 6534 Acked Write 1 Done Done Sent false
21492 6535 Acked Write 5 Done Done Sent false
21493 6536 Acked Write 3 Done Done Sent false
21494 6537 Acked Read 8 Done Done Sent false
21495 6538 Acked Read 9 Done Done Sent false
21496 6539 Acked Read 5 Done Done Sent false
21497 6540 Acked Write 9 Done Done Sent false
21498 6541 Acked Write 2 Done Done Sent false
21499 6542 Acked Write 9 Done Done Sent false
21500 6543 Acked Read 6 Done Done Sent false
21501 6544 Acked Read 6 Done Done Sent false
21502 6545 Acked Read 9 Done Done Sent false
21503 6546 Acked Write 3 Done Done Sent false
21504 6547 Acked Write 10 Done Done Sent false
21505 6548 Acked Read 9 Done Done Sent false
21506 6549 Acked Read 5 Done Done Sent false
21507 6550 Acked Read 3 Done Done Sent false
21508 6551 Acked Read 7 Done Done Sent false
21509 6552 Acked Read 5 Done Done Sent false
21510 6553 Acked Read 1 Done Done Sent false
21511 6554 Acked Write 7 Done Done Sent false
21512 6555 Acked Read 9 Done Done Sent false
21513 6556 Acked Read 4 Done Done Sent false
21514 6557 Acked Read 10 Done Done Sent false
21515 6558 Acked Read 10 Done Done Sent false
21516 6559 Acked Read 8 Done Done Sent false
21517 6560 Acked Write 6 Done Done Sent false
21518 6561 Acked Write 3 Done Done Sent false
21519 6562 Acked Read 5 Done Done Sent false
21520 6563 Acked Write 5 Done Done Sent false
21521 6564 Acked Write 1 Done Done Sent false
21522 6565 Acked Read 10 Done Done Sent false
21523 6566 Acked Read 3 Done Done Sent false
21524 6567 Acked Read 5 Done Done Sent false
21525 6568 Acked Write 4 Done Done Sent false
21526 6569 Acked Write 4 Done Done Sent false
21527 6570 Acked Read 3 Done Done Sent false
21528 6571 Acked Write 10 Done Done Sent false
21529 6572 Acked Read 7 Done Done Sent false
21530 6573 Acked Write 3 Done Done Sent false
21531 6574 Acked Read 3 Done Done Sent false
21532 6575 Acked Read 8 Done Done Sent false
21533 6576 Acked Write 8 Done Done Sent false
21534 6577 Acked Write 6 Done Done Sent false
21535 6578 Acked Write 6 Done Done Sent false
21536 6579 Acked Write 5 Done Done Sent false
21537 6580 Acked Read 6 Done Done Sent false
21538 6581 Acked Write 5 Done Done Sent false
21539 6582 Acked Read 2 Done Done Sent false
21540 6583 Acked Write 8 Done Done Sent false
21541 6584 Acked Read 3 Done Done Sent false
21542 6585 Acked Flush 0 Done Done Sent false
21543 6586 Acked Write 4 Done Done Sent false
21544 6587 Acked Write 10 Done Done Sent false
21545 6588 Acked Write 10 Done Done Sent false
21546 6589 Acked Read 3 Done Done Sent false
21547 6590 Acked Read 8 Done Done Sent false
21548 6591 Acked Read 5 Done Done Sent false
21549 6592 Acked Read 6 Done Done Sent false
21550 6593 Acked Read 5 Done Done Sent false
21551 6594 Acked Read 1 Done Done Sent false
21552 6595 Acked Read 10 Done Done Sent false
21553 6596 Acked Flush 0 Done Done Sent false
21554 6597 Acked Read 10 Done Done Sent false
21555 6598 Acked Read 3 Done Done Sent false
21556 6599 Acked Write 9 Done Done Sent false
21557 6600 Acked Read 8 Done Done Sent false
21558 6601 Acked Write 3 Done Done Sent false
21559 6602 Acked Write 7 Done Done Sent false
21560 6603 Acked Write 10 Done Done Sent false
21561 6604 Acked Write 4 Done Done Sent false
21562 6605 Acked Read 8 Done Done Sent false
21563 6606 Acked Write 7 Done Done Sent false
21564 6607 Acked Read 2 Done Done Sent false
21565 6608 Acked Read 6 Done Done Sent false
21566 6609 Acked Read 8 Done Done Sent false
21567 6610 Acked Flush 0 Done Done Sent false
21568 6611 Acked Write 9 Done Done Sent false
21569 6612 Acked Write 9 Done Done Sent false
21570 6613 Acked Write 5 Done Done Sent false
21571 6614 Acked Write 10 Done Done Sent false
21572 6615 Acked Read 10 Done Done Sent false
21573 6616 Acked Read 2 Done Done Sent false
21574 6617 Acked Read 8 Done Done Sent false
21575 6618 Acked Flush 0 Done Done Sent false
21576 6619 Acked Write 3 Done Done Sent false
21577 6620 Acked Read 9 Done Done Sent false
21578 6621 Acked Read 1 Done Done Sent false
21579 6622 Acked Write 6 Done Done Sent false
21580 6623 Acked Write 5 Done Done Sent false
21581 6624 Acked Flush 0 Done Done Sent false
21582 6625 Acked Flush 0 Done Done Sent false
21583 6626 Acked Read 8 Done Done Sent false
21584 6627 Acked Read 8 Done Done Sent false
21585 6628 Acked Write 10 Done Done Sent false
21586 6629 Acked Read 3 Done Done Sent false
21587 6630 Acked Read 8 Done Done Sent false
21588 6631 Acked Write 1 Done Done Sent false
21589 6632 Acked Write 8 Done Done Sent false
21590 6633 Acked Read 1 Done Done Sent false
21591 6634 Acked Read 3 Done Done Sent false
21592 6635 Acked Read 6 Done Done Sent false
21593 6636 Acked Read 8 Done Done Sent false
21594 6637 Acked Read 2 Done Done Sent false
21595 6638 Acked Read 1 Done Done Sent false
21596 6639 Acked Read 2 Done Done Sent false
21597 6640 Acked Write 4 Done Done Sent false
21598 6641 Acked Read 8 Done Done Sent false
21599 6642 Acked Read 3 Done Done Sent false
21600 6643 Acked Flush 0 Done Done Sent false
21601 6644 Acked Read 3 Done Done Sent false
21602 6645 Acked Read 6 Done Done Sent false
21603 6646 Acked Flush 0 Done Done Sent false
21604 6647 Acked Read 1 Done Done Sent false
21605 6648 Acked Write 5 Done Done Sent false
21606 6649 Acked Read 5 Done Done Sent false
21607 6650 Acked Read 9 Done Done Sent false
21608 6651 Acked Read 4 Done Done Sent false
21609 6652 Acked Write 3 Done Done Sent false
21610 6653 Acked Write 7 Done Done Sent false
21611 6654 Acked Write 1 Done Done Sent false
21612 6655 Acked Write 2 Done Done Sent false
21613 6656 Acked Read 10 Done Done Sent false
21614 6657 Acked Write 1 Done Done Sent false
21615 6658 Acked Write 3 Done Done Sent false
21616 6659 Acked Read 5 Done Done Sent false
21617 6660 Acked Read 7 Done Done Sent false
21618 6661 Acked Write 6 Done Done Sent false
21619 6662 Acked Read 3 Done Done Sent false
21620 6663 Acked Flush 0 Done Done Sent false
21621 6664 Acked Write 2 Done Done Sent false
21622 6665 Acked Read 5 Done Done Sent false
21623 6666 Acked Write 5 Done Done Sent false
21624 6667 Acked Write 9 Done Done Sent false
21625 6668 Acked Read 5 Done Done Sent false
21626 6669 Acked Write 10 Done Done Sent false
21627 6670 Acked Write 8 Done Done Sent false
21628 6671 Acked Read 5 Done Done Sent false
21629 6672 Acked Flush 0 Done Done Sent false
21630 6673 Acked Read 1 Done Done Sent false
21631 6674 Acked Read 7 Done Done Sent false
21632 6675 Acked Flush 0 Done Done Sent false
21633 6676 Acked Read 10 Done Done Sent false
21634 6677 Acked Write 3 Done Done Sent false
21635 6678 Acked Write 7 Done Done Sent false
21636 6679 Acked Read 5 Done Done Sent false
21637 6680 Acked Read 1 Done Done Sent false
21638 6681 Acked Read 3 Done Done Sent false
21639 6682 Acked Write 5 Done Done Sent false
21640 6683 Acked Read 9 Done Done Sent false
21641 6684 Acked Read 6 Done Done Sent false
21642 6685 Acked Read 2 Done Done Sent false
21643 6686 Acked Read 9 Done Done Sent false
21644 6687 Acked Read 10 Done Done Sent false
21645 6688 Acked Read 1 Done Done Sent false
21646 6689 Acked Flush 0 Done Done Sent false
21647 6690 Acked Write 6 Done Done Sent false
21648 6691 Acked Write 4 Done Done Sent false
21649 6692 Acked Read 3 Done Done Sent false
21650 6693 Acked Flush 0 Done Done Sent false
21651 6694 Acked Read 10 Done Done Sent false
21652 6695 Acked Read 3 Done Done Sent false
21653 6696 Acked Write 3 Done Done Sent false
21654 6697 Acked Write 8 Done Done Sent false
21655 6698 Acked Read 7 Done Done Sent false
21656 6699 Acked Flush 0 Done Done Sent false
21657 6700 Acked Write 5 Done Done Sent false
21658 6701 Acked Write 3 Done Done Sent false
21659 6702 Acked Write 1 Done Done Sent false
21660 6703 Acked Write 10 Done Done Sent false
21661 6704 Acked Write 6 Done Done Sent false
21662 6705 Acked Flush 0 Done Done Sent false
21663 6706 Acked Read 4 Done Done Sent false
21664 6707 Acked Write 8 Done Done Sent false
21665 6708 Acked Read 8 Done Done Sent false
21666 6709 Acked Read 3 Done Done Sent false
21667 6710 Acked Write 7 Done Done Sent false
21668 6711 Acked Read 4 Done Done Sent false
21669 6712 Acked Write 9 Done Done Sent false
21670 6713 Acked Write 6 Done Done Sent false
21671 6714 Acked Write 2 Done Done Sent false
21672 6715 Acked Flush 0 Done Done Sent false
21673 6716 Acked Read 1 Done Done Sent false
21674 6717 Acked Flush 0 Done Done Sent false
21675 6718 Acked Flush 0 Done Done Sent false
21676 6719 Acked Read 2 Done Done Sent false
21677 6720 Acked Flush 0 Done Done Sent false
21678 6721 Acked Read 3 Done Done Sent false
21679 6722 Acked Flush 0 Done Done Sent false
21680 6723 Acked Flush 0 Done Done Sent false
21681 6724 Acked Read 3 Done Done Sent false
21682 6725 Acked Read 5 Done Done Sent false
21683 6726 Acked Write 4 Done Done Sent false
21684 6727 Acked Read 10 Done Done Sent false
21685 6728 Acked Read 2 Done Done Sent false
21686 6729 Acked Read 10 Done Done Sent false
21687 6730 Acked Read 6 Done Done Sent false
21688 6731 Acked Read 9 Done Done Sent false
21689 6732 Acked Read 8 Done Done Sent false
21690 6733 Acked Read 5 Done Done Sent false
21691 6734 Acked Read 9 Done Done Sent false
21692 6735 Acked Write 7 Done Done Sent false
21693 STATES DS:0 DS:1 DS:2 TOTAL
21694 Sent 0 0 301 301
21695 Done 301 301 0 602
21696 Skipped 0 0 0 0
21697 Error 0 0 0 0
21698 Last Flush: Some(JobId(6723)) Some(JobId(6723)) Some(JobId(6434))
21699 Downstairs last five completed: 6434 6433 6432 6431 6430
21700 Upstairs last five completed: 6735 6734 6733 6732 6731
21701 CLIENT: Up:0 ds:301 act:2
217022025-01-29T05:15:28.748ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
217032025-01-29T05:15:29.390ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21704 [8830] got start action
21705 [8830] I am going to restart
21706 Starting downstairs at port 8830
21707 Make output file at "/tmp/dsc/downstairs-8830.txt"
21708 [8830][2] reports Starting
21709 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
21710 [8830][2] reports Running
217112025-01-29T05:15:30.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
217122025-01-29T05:15:31.393ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
21713 ----------------------------------------------------------------
21714 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21715 States: Active Active Offline
21716 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21717 6435 Acked Flush 0 Done Done Sent false
21718 6436 Acked Read 4 Done Done Sent false
21719 6437 Acked Write 9 Done Done Sent false
21720 6438 Acked Write 4 Done Done Sent false
21721 6439 Acked Read 1 Done Done Sent false
21722 6440 Acked Write 9 Done Done Sent false
21723 6441 Acked Flush 0 Done Done Sent false
21724 6442 Acked Read 9 Done Done Sent false
21725 6443 Acked Read 5 Done Done Sent false
21726 6444 Acked Flush 0 Done Done Sent false
21727 6445 Acked Write 3 Done Done Sent false
21728 6446 Acked Read 8 Done Done Sent false
21729 6447 Acked Write 6 Done Done Sent false
21730 6448 Acked Write 2 Done Done Sent false
21731 6449 Acked Read 9 Done Done Sent false
21732 6450 Acked Write 1 Done Done Sent false
21733 6451 Acked Write 6 Done Done Sent false
21734 6452 Acked Write 8 Done Done Sent false
21735 6453 Acked Read 7 Done Done Sent false
21736 6454 Acked Read 9 Done Done Sent false
21737 6455 Acked Read 7 Done Done Sent false
21738 6456 Acked Flush 0 Done Done Sent false
21739 6457 Acked Read 8 Done Done Sent false
21740 6458 Acked Flush 0 Done Done Sent false
21741 6459 Acked Read 9 Done Done Sent false
21742 6460 Acked Read 1 Done Done Sent false
21743 6461 Acked Write 9 Done Done Sent false
21744 6462 Acked Write 5 Done Done Sent false
21745 6463 Acked Read 9 Done Done Sent false
21746 6464 Acked Read 9 Done Done Sent false
21747 6465 Acked Read 2 Done Done Sent false
21748 6466 Acked Write 9 Done Done Sent false
21749 6467 Acked Write 6 Done Done Sent false
21750 6468 Acked Read 5 Done Done Sent false
21751 6469 Acked Write 6 Done Done Sent false
21752 6470 Acked Read 10 Done Done Sent false
21753 6471 Acked Write 4 Done Done Sent false
21754 6472 Acked Write 4 Done Done Sent false
21755 6473 Acked Read 1 Done Done Sent false
21756 6474 Acked Write 3 Done Done Sent false
21757 6475 Acked Read 10 Done Done Sent false
21758 6476 Acked Read 10 Done Done Sent false
21759 6477 Acked Write 1 Done Done Sent false
21760 6478 Acked Read 9 Done Done Sent false
21761 6479 Acked Write 5 Done Done Sent false
21762 6480 Acked Read 9 Done Done Sent false
21763 6481 Acked Read 5 Done Done Sent false
21764 6482 Acked Write 2 Done Done Sent false
21765 6483 Acked Read 7 Done Done Sent false
21766 6484 Acked Write 2 Done Done Sent false
21767 6485 Acked Write 1 Done Done Sent false
21768 6486 Acked Read 5 Done Done Sent false
21769 6487 Acked Write 5 Done Done Sent false
21770 6488 Acked Write 7 Done Done Sent false
21771 6489 Acked Write 10 Done Done Sent false
21772 6490 Acked Read 2 Done Done Sent false
21773 6491 Acked Read 7 Done Done Sent false
21774 6492 Acked Write 4 Done Done Sent false
21775 6493 Acked Write 7 Done Done Sent false
21776 6494 Acked Read 7 Done Done Sent false
21777 6495 Acked Write 2 Done Done Sent false
21778 6496 Acked Write 10 Done Done Sent false
21779 6497 Acked Write 8 Done Done Sent false
21780 6498 Acked Read 5 Done Done Sent false
21781 6499 Acked Flush 0 Done Done Sent false
21782 6500 Acked Flush 0 Done Done Sent false
21783 6501 Acked Read 5 Done Done Sent false
21784 6502 Acked Read 8 Done Done Sent false
21785 6503 Acked Write 9 Done Done Sent false
21786 6504 Acked Write 5 Done Done Sent false
21787 6505 Acked Write 2 Done Done Sent false
21788 6506 Acked Read 9 Done Done Sent false
21789 6507 Acked Read 8 Done Done Sent false
21790 6508 Acked Write 7 Done Done Sent false
21791 6509 Acked Write 6 Done Done Sent false
21792 6510 Acked Write 9 Done Done Sent false
21793 6511 Acked Read 5 Done Done Sent false
21794 6512 Acked Read 1 Done Done Sent false
21795 6513 Acked Read 7 Done Done Sent false
21796 6514 Acked Read 5 Done Done Sent false
21797 6515 Acked Read 3 Done Done Sent false
21798 6516 Acked Flush 0 Done Done Sent false
21799 6517 Acked Write 2 Done Done Sent false
21800 6518 Acked Write 5 Done Done Sent false
21801 6519 Acked Flush 0 Done Done Sent false
21802 6520 Acked Read 9 Done Done Sent false
21803 6521 Acked Write 1 Done Done Sent false
21804 6522 Acked Read 10 Done Done Sent false
21805 6523 Acked Write 3 Done Done Sent false
21806 6524 Acked Write 3 Done Done Sent false
21807 6525 Acked Write 6 Done Done Sent false
21808 6526 Acked Write 4 Done Done Sent false
21809 6527 Acked Flush 0 Done Done Sent false
21810 6528 Acked Read 3 Done Done Sent false
21811 6529 Acked Read 7 Done Done Sent false
21812 6530 Acked Flush 0 Done Done Sent false
21813 6531 Acked Flush 0 Done Done Sent false
21814 6532 Acked Read 10 Done Done Sent false
21815 6533 Acked Write 4 Done Done Sent false
21816 6534 Acked Write 1 Done Done Sent false
21817 6535 Acked Write 5 Done Done Sent false
21818 6536 Acked Write 3 Done Done Sent false
21819 6537 Acked Read 8 Done Done Sent false
21820 6538 Acked Read 9 Done Done Sent false
21821 6539 Acked Read 5 Done Done Sent false
21822 6540 Acked Write 9 Done Done Sent false
21823 6541 Acked Write 2 Done Done Sent false
21824 6542 Acked Write 9 Done Done Sent false
21825 6543 Acked Read 6 Done Done Sent false
21826 6544 Acked Read 6 Done Done Sent false
21827 6545 Acked Read 9 Done Done Sent false
21828 6546 Acked Write 3 Done Done Sent false
21829 6547 Acked Write 10 Done Done Sent false
21830 6548 Acked Read 9 Done Done Sent false
21831 6549 Acked Read 5 Done Done Sent false
21832 6550 Acked Read 3 Done Done Sent false
21833 6551 Acked Read 7 Done Done Sent false
21834 6552 Acked Read 5 Done Done Sent false
21835 6553 Acked Read 1 Done Done Sent false
21836 6554 Acked Write 7 Done Done Sent false
21837 6555 Acked Read 9 Done Done Sent false
21838 6556 Acked Read 4 Done Done Sent false
21839 6557 Acked Read 10 Done Done Sent false
21840 6558 Acked Read 10 Done Done Sent false
21841 6559 Acked Read 8 Done Done Sent false
21842 6560 Acked Write 6 Done Done Sent false
21843 6561 Acked Write 3 Done Done Sent false
21844 6562 Acked Read 5 Done Done Sent false
21845 6563 Acked Write 5 Done Done Sent false
21846 6564 Acked Write 1 Done Done Sent false
21847 6565 Acked Read 10 Done Done Sent false
21848 6566 Acked Read 3 Done Done Sent false
21849 6567 Acked Read 5 Done Done Sent false
21850 6568 Acked Write 4 Done Done Sent false
21851 6569 Acked Write 4 Done Done Sent false
21852 6570 Acked Read 3 Done Done Sent false
21853 6571 Acked Write 10 Done Done Sent false
21854 6572 Acked Read 7 Done Done Sent false
21855 6573 Acked Write 3 Done Done Sent false
21856 6574 Acked Read 3 Done Done Sent false
21857 6575 Acked Read 8 Done Done Sent false
21858 6576 Acked Write 8 Done Done Sent false
21859 6577 Acked Write 6 Done Done Sent false
21860 6578 Acked Write 6 Done Done Sent false
21861 6579 Acked Write 5 Done Done Sent false
21862 6580 Acked Read 6 Done Done Sent false
21863 6581 Acked Write 5 Done Done Sent false
21864 6582 Acked Read 2 Done Done Sent false
21865 6583 Acked Write 8 Done Done Sent false
21866 6584 Acked Read 3 Done Done Sent false
21867 6585 Acked Flush 0 Done Done Sent false
21868 6586 Acked Write 4 Done Done Sent false
21869 6587 Acked Write 10 Done Done Sent false
21870 6588 Acked Write 10 Done Done Sent false
21871 6589 Acked Read 3 Done Done Sent false
21872 6590 Acked Read 8 Done Done Sent false
21873 6591 Acked Read 5 Done Done Sent false
21874 6592 Acked Read 6 Done Done Sent false
21875 6593 Acked Read 5 Done Done Sent false
21876 6594 Acked Read 1 Done Done Sent false
21877 6595 Acked Read 10 Done Done Sent false
21878 6596 Acked Flush 0 Done Done Sent false
21879 6597 Acked Read 10 Done Done Sent false
21880 6598 Acked Read 3 Done Done Sent false
21881 6599 Acked Write 9 Done Done Sent false
21882 6600 Acked Read 8 Done Done Sent false
21883 6601 Acked Write 3 Done Done Sent false
21884 6602 Acked Write 7 Done Done Sent false
21885 6603 Acked Write 10 Done Done Sent false
21886 6604 Acked Write 4 Done Done Sent false
21887 6605 Acked Read 8 Done Done Sent false
21888 6606 Acked Write 7 Done Done Sent false
21889 6607 Acked Read 2 Done Done Sent false
21890 6608 Acked Read 6 Done Done Sent false
21891 6609 Acked Read 8 Done Done Sent false
21892 6610 Acked Flush 0 Done Done Sent false
21893 6611 Acked Write 9 Done Done Sent false
21894 6612 Acked Write 9 Done Done Sent false
21895 6613 Acked Write 5 Done Done Sent false
21896 6614 Acked Write 10 Done Done Sent false
21897 6615 Acked Read 10 Done Done Sent false
21898 6616 Acked Read 2 Done Done Sent false
21899 6617 Acked Read 8 Done Done Sent false
21900 6618 Acked Flush 0 Done Done Sent false
21901 6619 Acked Write 3 Done Done Sent false
21902 6620 Acked Read 9 Done Done Sent false
21903 6621 Acked Read 1 Done Done Sent false
21904 6622 Acked Write 6 Done Done Sent false
21905 6623 Acked Write 5 Done Done Sent false
21906 6624 Acked Flush 0 Done Done Sent false
21907 6625 Acked Flush 0 Done Done Sent false
21908 6626 Acked Read 8 Done Done Sent false
21909 6627 Acked Read 8 Done Done Sent false
21910 6628 Acked Write 10 Done Done Sent false
21911 6629 Acked Read 3 Done Done Sent false
21912 6630 Acked Read 8 Done Done Sent false
21913 6631 Acked Write 1 Done Done Sent false
21914 6632 Acked Write 8 Done Done Sent false
21915 6633 Acked Read 1 Done Done Sent false
21916 6634 Acked Read 3 Done Done Sent false
21917 6635 Acked Read 6 Done Done Sent false
21918 6636 Acked Read 8 Done Done Sent false
21919 6637 Acked Read 2 Done Done Sent false
21920 6638 Acked Read 1 Done Done Sent false
21921 6639 Acked Read 2 Done Done Sent false
21922 6640 Acked Write 4 Done Done Sent false
21923 6641 Acked Read 8 Done Done Sent false
21924 6642 Acked Read 3 Done Done Sent false
21925 6643 Acked Flush 0 Done Done Sent false
21926 6644 Acked Read 3 Done Done Sent false
21927 6645 Acked Read 6 Done Done Sent false
21928 6646 Acked Flush 0 Done Done Sent false
21929 6647 Acked Read 1 Done Done Sent false
21930 6648 Acked Write 5 Done Done Sent false
21931 6649 Acked Read 5 Done Done Sent false
21932 6650 Acked Read 9 Done Done Sent false
21933 6651 Acked Read 4 Done Done Sent false
21934 6652 Acked Write 3 Done Done Sent false
21935 6653 Acked Write 7 Done Done Sent false
21936 6654 Acked Write 1 Done Done Sent false
21937 6655 Acked Write 2 Done Done Sent false
21938 6656 Acked Read 10 Done Done Sent false
21939 6657 Acked Write 1 Done Done Sent false
21940 6658 Acked Write 3 Done Done Sent false
21941 6659 Acked Read 5 Done Done Sent false
21942 6660 Acked Read 7 Done Done Sent false
21943 6661 Acked Write 6 Done Done Sent false
21944 6662 Acked Read 3 Done Done Sent false
21945 6663 Acked Flush 0 Done Done Sent false
21946 6664 Acked Write 2 Done Done Sent false
21947 6665 Acked Read 5 Done Done Sent false
21948 6666 Acked Write 5 Done Done Sent false
21949 6667 Acked Write 9 Done Done Sent false
21950 6668 Acked Read 5 Done Done Sent false
21951 6669 Acked Write 10 Done Done Sent false
21952 6670 Acked Write 8 Done Done Sent false
21953 6671 Acked Read 5 Done Done Sent false
21954 6672 Acked Flush 0 Done Done Sent false
21955 6673 Acked Read 1 Done Done Sent false
21956 6674 Acked Read 7 Done Done Sent false
21957 6675 Acked Flush 0 Done Done Sent false
21958 6676 Acked Read 10 Done Done Sent false
21959 6677 Acked Write 3 Done Done Sent false
21960 6678 Acked Write 7 Done Done Sent false
21961 6679 Acked Read 5 Done Done Sent false
21962 6680 Acked Read 1 Done Done Sent false
21963 6681 Acked Read 3 Done Done Sent false
21964 6682 Acked Write 5 Done Done Sent false
21965 6683 Acked Read 9 Done Done Sent false
21966 6684 Acked Read 6 Done Done Sent false
21967 6685 Acked Read 2 Done Done Sent false
21968 6686 Acked Read 9 Done Done Sent false
21969 6687 Acked Read 10 Done Done Sent false
21970 6688 Acked Read 1 Done Done Sent false
21971 6689 Acked Flush 0 Done Done Sent false
21972 6690 Acked Write 6 Done Done Sent false
21973 6691 Acked Write 4 Done Done Sent false
21974 6692 Acked Read 3 Done Done Sent false
21975 6693 Acked Flush 0 Done Done Sent false
21976 6694 Acked Read 10 Done Done Sent false
21977 6695 Acked Read 3 Done Done Sent false
21978 6696 Acked Write 3 Done Done Sent false
21979 6697 Acked Write 8 Done Done Sent false
21980 6698 Acked Read 7 Done Done Sent false
21981 6699 Acked Flush 0 Done Done Sent false
21982 6700 Acked Write 5 Done Done Sent false
21983 6701 Acked Write 3 Done Done Sent false
21984 6702 Acked Write 1 Done Done Sent false
21985 6703 Acked Write 10 Done Done Sent false
21986 6704 Acked Write 6 Done Done Sent false
21987 6705 Acked Flush 0 Done Done Sent false
21988 6706 Acked Read 4 Done Done Sent false
21989 6707 Acked Write 8 Done Done Sent false
21990 6708 Acked Read 8 Done Done Sent false
21991 6709 Acked Read 3 Done Done Sent false
21992 6710 Acked Write 7 Done Done Sent false
21993 6711 Acked Read 4 Done Done Sent false
21994 6712 Acked Write 9 Done Done Sent false
21995 6713 Acked Write 6 Done Done Sent false
21996 6714 Acked Write 2 Done Done Sent false
21997 6715 Acked Flush 0 Done Done Sent false
21998 6716 Acked Read 1 Done Done Sent false
21999 6717 Acked Flush 0 Done Done Sent false
22000 6718 Acked Flush 0 Done Done Sent false
22001 6719 Acked Read 2 Done Done Sent false
22002 6720 Acked Flush 0 Done Done Sent false
22003 6721 Acked Read 3 Done Done Sent false
22004 6722 Acked Flush 0 Done Done Sent false
22005 6723 Acked Flush 0 Done Done Sent false
22006 6724 Acked Read 3 Done Done Sent false
22007 6725 Acked Read 5 Done Done Sent false
22008 6726 Acked Write 4 Done Done Sent false
22009 6727 Acked Read 10 Done Done Sent false
22010 6728 Acked Read 2 Done Done Sent false
22011 6729 Acked Read 10 Done Done Sent false
22012 6730 Acked Read 6 Done Done Sent false
22013 6731 Acked Read 9 Done Done Sent false
22014 6732 Acked Read 8 Done Done Sent false
22015 6733 Acked Read 5 Done Done Sent false
22016 6734 Acked Read 9 Done Done Sent false
22017 6735 Acked Write 7 Done Done Sent false
22018 STATES DS:0 DS:1 DS:2 TOTAL
22019 Sent 0 0 301 301
22020 Done 301 301 0 602
22021 Skipped 0 0 0 0
22022 Error 0 0 0 0
22023 Last Flush: Some(JobId(6723)) Some(JobId(6723)) Some(JobId(6434))
22024 Downstairs last five completed: 6434 6433 6432 6431 6430
22025 Upstairs last five completed: 6735 6734 6733 6732 6731
22026 CLIENT: Up:0 ds:301 act:2
220272025-01-29T05:15:32.751ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220282025-01-29T05:15:33.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220292025-01-29T05:15:34.002ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220302025-01-29T05:15:34.003ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41301) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220312025-01-29T05:15:34.004ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220322025-01-29T05:15:34.004ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220332025-01-29T05:15:34.004ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220342025-01-29T05:15:34.004ZINFOcrucible: send last flush ID to this DS: Some(JobId(6434)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220352025-01-29T05:15:34.004ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220362025-01-29T05:15:34.004ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6434)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220372025-01-29T05:15:34.004ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6434)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
22038 ----------------------------------------------------------------
22039 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22040 Downstairs last five completed: 6736 6735 6734 6733 6732
22041 Upstairs last five completed: 6736 6735 6734 6733 6732
22042 CLIENT: Up:0 ds:0 act:3
22043 Replay: All jobs finished
22044 Main task has work to do, go find it
22045 got dsc Stop(2)
22046 stop 2
22047 Jan 29 05:15:36.252 INFO [8830] Got stop action so:false kr:false
22048 request completed, latency_us: 344, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8916ad83-7eb8-4f58-acbd-3eafdf2acedb, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
22049 Jan 29 05:15:36.253 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/2, method: GET, req_id: aa2500f7-b976-4579-bc7b-dcec3cb0d6f8, remote_addr: 127.0.0.1{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:15:36.253407004Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
22050 :46995, local_addr: 127.0.0.1:9998
220512025-01-29T05:15:36.253ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220522025-01-29T05:15:36.253ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
220532025-01-29T05:15:36.253ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
22054 [8830] Exited with: ExitStatus(unix_wait_status(9))
22055 [8830][2] reports Exit
22056 Jan 29 05:15:40.256 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5b4f32c0-ad16-4a79-9ba7-f28f7b88322e, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
22057 001/300 Read block 4117 len 4096
22058 002/300 Write block 420 len 24576 data: 4 4 5 5 5 5
22059 003/300 Read block 271 len 20480
22060 004/300 Write block 3763 len 24576 data: 5 6 7 7 9 9
22061 005/300 Read block 1786 len 20480
22062 006/300 Read block 558 len 24576
22063 007/300 Write block 101 len 32768 data: 5 4 4 5 4 5 5 3
22064 008/300 Write block 4566 len 4096 data: 5
22065 009/300 Write block 2930 len 28672 data: 5 5 5 3 2 3 4
22066 010/300 Read block 1433 len 16384
22067 011/300 Read block 623 len 32768
22068 012/300 Flush
22069 013/300 Write block 2061 len 12288 data: 5 5 3
22070 014/300 Read block 1188 len 40960
22071 015/300 Read block 2063 len 40960
22072 016/300 Read block 2225 len 20480
22073 017/300 Read block 2305 len 12288
22074 018/300 Flush
22075 019/300 Read block 1370 len 8192
22076 020/300 Write block 650 len 8192 data: 6 5
22077 021/300 Write block 629 len 40960 data: 4 4 3 2 2 5 6 6 7 7
22078 022/300 Read block 3166 len 8192
22079 023/300 Flush
22080 024/300 Write block 916 len 8192 data: 3 3
22081 025/300 Read block 4496 len 28672
22082 026/300 Read block 4753 len 20480
22083 027/300 Read block 1128 len 16384
22084 028/300 Write block 2100 len 8192 data: 5 3
22085 029/300 Write block 209 len 4096 data: 2
22086 030/300 Read block 2903 len 32768
22087 031/300 Write block 1766 len 32768 data: 6 5 4 5 5 5 4 6
22088 032/300 Write block 2556 len 40960 data: 3 4 4 5 4 4 4 5 2 2
22089 033/300 Flush
22090 034/300 Read block 1034 len 32768
22091 035/300 Read block 3365 len 36864
22092 036/300 Write block 332 len 8192 data: 5 6
22093 037/300 Write block 3983 len 4096 data: 4
22094 038/300 Write block 806 len 16384 data: 3 4 5 5
22095 039/300 Read block 4003 len 28672
22096 040/300 Write block 152 len 12288 data: 3 3 3
22097 041/300 Flush
22098 042/300 Read block 1618 len 20480
22099 043/300 Read block 11 len 32768
22100 044/300 Read block 2686 len 4096
22101 045/300 Read block 2458 len 4096
22102 046/300 Read block 4800 len 12288
22103 047/300 Read block 823 len 36864
22104 048/300 Write block 531 len 4096 data: 3
22105 049/300 Write block 3499 len 28672 data: 6 6 6 5 5 5 5
22106 050/300 Write block 1732 len 20480 data: 3 2 2 2 2
22107 051/300 Flush
22108 052/300 Write block 4581 len 24576 data: 3 3 3 4 5 3
22109 053/300 Read block 4123 len 4096
22110 054/300 Write block 452 len 16384 data: 3 4 6 6
22111 055/300 Read block 1324 len 28672
22112 056/300 Read block 4454 len 28672
22113 057/300 Read block 3669 len 20480
22114 058/300 Read block 2923 len 8192
22115 059/300 Read block 1965 len 4096
22116 060/300 Read block 3929 len 28672
22117 061/300 Flush
22118 062/300 Read block 2697 len 12288
22119 063/300 Write block 816 len 20480 data: 6 6 6 6 5
22120 064/300 Read block 3180 len 36864
22121 065/300 Read block 329 len 36864
22122 066/300 Read block 2717 len 8192
22123 067/300 Read block 1753 len 36864
22124 068/300 Read block 1258 len 16384
22125 069/300 Flush
22126 070/300 Read block 4758 len 24576
22127 071/300 Write block 3320 len 20480 data: 3 3 4 4 4
22128 072/300 Read block 1894 len 8192
22129 073/300 Read block 3178 len 16384
22130 074/300 Read block 2407 len 8192
22131 075/300 Read block 1340 len 4096
22132 076/300 Read block 4063 len 12288
22133 077/300 Write block 1898 len 36864 data: 4 3 3 2 3 4 4 4 6
22134 078/300 Read block 2782 len 24576
22135 079/300 Read block 896 len 24576
22136 080/300 Read block 2046 len 12288
22137 081/300 Read block 625 len 8192
22138 082/300 Read block 2715 len 36864
22139 083/300 Flush
22140 084/300 Read block 4433 len 32768
22141 085/300 Read block 1938 len 8192
22142 086/300 Write block 4904 len 20480 data: 3 4 3 2 2
22143 087/300 Read block 1052 len 12288
22144 088/300 Write block 3168 len 32768 data: 4 4 4 4 4 5 4 4
22145 089/300 Write block 2786 len 32768 data: 4 4 4 4 5 4 4 6
22146 090/300 Read block 67 len 28672
22147 091/300 Flush
22148 092/300 Read block 1911 len 12288
22149 093/300 Write block 4439 len 32768 data: 4 4 3 2 2 2 3 3
22150 094/300 Read block 3501 len 24576
22151 095/300 Write block 4656 len 32768 data: 5 5 5 6 5 5 5 6
22152 096/300 Read block 965 len 28672
22153 097/300 Write block 1435 len 28672 data: 4 6 4 4 4 4 4
22154 098/300 Read block 2150 len 12288
22155 099/300 Write block 2452 len 36864 data: 6 6 7 8 8 8 7 4 5
22156 100/300 Write block 3906 len 8192 data: 4 4
22157 101/300 Read block 2216 len 40960
22158 102/300 Write block 1782 len 36864 data: 3 4 4 3 2 4 5 5 6
22159 103/300 Write block 2792 len 4096 data: 5
22160 104/300 Write block 3215 len 28672 data: 3 6 5 5 7 8 6
22161 105/300 Write block 3983 len 16384 data: 5 3 3 4
22162 106/300 Write block 2960 len 20480 data: 5 6 6 6 5
22163 107/300 Read block 519 len 12288
22164 108/300 Write block 3964 len 32768 data: 3 2 3 3 3 4 6 6
22165 109/300 Read block 2166 len 36864
22166 110/300 Write block 4289 len 28672 data: 4 5 5 5 6 6 5
22167 111/300 Write block 2120 len 32768 data: 3 3 3 3 3 3 3 2
22168 112/300 Write block 1982 len 24576 data: 4 6 6 5 5 5
22169 113/300 Write block 1701 len 12288 data: 2 2 3
22170 114/300 Write block 3639 len 20480 data: 4 5 5 8 8
22171 115/300 Flush
22172 116/300 Read block 886 len 40960
22173 117/300 Write block 3423 len 28672 data: 6 4 3 3 5 5 6
22174 118/300 Write block 4763 len 16384 data: 5 4 4 5
22175 119/300 Read block 892 len 36864
22176 120/300 Read block 2639 len 28672
22177 121/300 Flush
22178 122/300 Write block 167 len 8192 data: 3 3
22179 123/300 Write block 3804 len 32768 data: 5 7 7 7 7 5 4 4
22180 124/300 Write block 1703 len 28672 data: 4 4 3 3 4 4 4
22181 125/300 Write block 732 len 32768 data: 6 5 5 5 7 7 6 6
22182 126/300 Read block 2002 len 24576
22183 127/300 Read block 1727 len 4096
22184 128/300 Read block 4449 len 16384
22185 129/300 Read block 41 len 16384
22186 130/300 Read block 4802 len 12288
22187 131/300 Write block 2863 len 28672 data: 4 4 5 4 4 5 6
22188 132/300 Write block 1774 len 24576 data: 6 8 8 7 7 7
22189 133/300 Write block 682 len 16384 data: 2 2 2 2
22190 134/300 Read block 874 len 32768
22191 135/300 Write block 1686 len 32768 data: 5 7 8 9 11 11 7 8
22192 136/300 Write block 2425 len 8192 data: 7 6
22193 137/300 Read block 1386 len 24576
22194 138/300 Read block 313 len 12288
22195 139/300 Read block 324 len 8192
22196 140/300 Read block 4040 len 8192
22197 141/300 Read block 1548 len 4096
22198 142/300 Write block 898 len 24576 data: 6 5 5 4 3 3
22199 143/300 Read block 2940 len 40960
22200 144/300 Read block 4490 len 8192
22201 145/300 Read block 4662 len 36864
22202 146/300 Read block 3739 len 32768
22203 147/300 Write block 170 len 32768 data: 7 7 6 6 4 5 4 4
22204 148/300 Read block 2000 len 24576
22205 149/300 Write block 127 len 16384 data: 8 5 3 3
22206 150/300 Read block 4635 len 36864
22207 151/300 Read block 4924 len 32768
22208 152/300 Read block 4168 len 8192
22209 153/300 Read block 2898 len 16384
22210 154/300 Read block 612 len 40960
22211 155/300 Read block 161 len 20480
22212 156/300 Read block 3321 len 8192
22213 157/300 Read block 1800 len 20480
22214 158/300 Read block 2043 len 8192
22215 159/300 Read block 389 len 36864
22216 160/300 Write block 1819 len 28672 data: 3 3 3 3 4 4 3
22217 161/300 Flush
22218 162/300 Read block 1874 len 4096
22219 163/300 Write block 3569 len 40960 data: 4 5 5 6 7 8 9 10 9 9
22220 164/300 Read block 4686 len 36864
22221 165/300 Read block 2611 len 20480
22222 166/300 Read block 932 len 28672
22223 167/300 Read block 979 len 28672
22224 168/300 Write block 4438 len 8192 data: 3 5
22225 169/300 Write block 2738 len 32768 data: 7 8 6 6 7 5 3 4
22226 170/300 Read block 1634 len 32768
22227 171/300 Read block 3751 len 16384
22228 172/300 Read block 2365 len 12288
22229 173/300 Flush
22230 174/300 Write block 4777 len 24576 data: 8 8 7 7 7 7
22231 175/300 Write block 542 len 12288 data: 6 6 5
22232 176/300 Read block 3892 len 16384
22233 177/300 Read block 4340 len 36864
22234 178/300 Read block 4258 len 32768
22235 179/300 Write block 3913 len 40960 data: 3 2 3 3 2 2 2 2 2 2
22236 180/300 Write block 3664 len 4096 data: 2
22237 181/300 Write block 2665 len 20480 data: 5 4 3 4 6
22238 182/300 Flush
22239 183/300 Write block 2518 len 24576 data: 5 5 5 6 7 8
22240 184/300 Write block 3722 len 36864 data: 4 5 4 3 4 3 4 4 4
22241 185/300 Write block 4401 len 24576 data: 8 7 6 6 4 4
22242 186/300 Write block 837 len 8192 data: 4 5
22243 187/300 Flush
22244 188/300 Write block 2896 len 4096 data: 4
22245 189/300 Read block 4226 len 36864
22246 190/300 Read block 645 len 4096
22247 191/300 Flush
22248 192/300 Read block 2401 len 12288
22249 193/300 Read block 3428 len 24576
22250 194/300 Flush
22251 195/300 Write block 3355 len 8192 data: 3 3
22252 196/300 Read block 1186 len 36864
22253 197/300 Write block 4640 len 28672 data: 7 6 6 7 6 5 4
22254 198/300 Read block 1565 len 24576
22255 199/300 Write block 3263 len 4096 data: 3
22256 200/300 Read block 2949 len 16384
22257 201/300 Read block 4409 len 24576
22258 202/300 Flush
22259 203/300 Write block 3938 len 36864 data: 9 10 8 8 9 6 7 6 5
22260 204/300 Read block 1624 len 12288
22261 205/300 Write block 2144 len 24576 data: 6 5 4 6 6 5
22262 206/300 Read block 4836 len 16384
22263 207/300 Read block 1679 len 4096
22264 208/300 Flush
22265 209/300 Write block 3741 len 32768 data: 3 3 3 3 2 2 2 2
22266 210/300 Write block 1394 len 24576 data: 2 2 2 3 4 4
22267 211/300 Read block 2282 len 28672
22268 212/300 Write block 3817 len 24576 data: 4 3 3 3 4 4
22269 213/300 Read block 4602 len 28672
22270 214/300 Read block 1069 len 32768
22271 215/300 Write block 4163 len 36864 data: 2 2 3 3 4 4 5 5 5
22272 216/300 Read block 969 len 20480
22273 217/300 Write block 1823 len 4096 data: 5
22274 218/300 Write block 1412 len 36864 data: 5 6 6 5 8 6 6 4 4
22275 219/300 Write block 1114 len 4096 data: 3
22276 220/300 Read block 4406 len 40960
22277 221/300 Write block 4652 len 20480 data: 6 5 4 5 6
22278 222/300 Flush
22279 223/300 Read block 1857 len 28672
22280 224/300 Write block 417 len 16384 data: 4 3 4 5
22281 225/300 Read block 2310 len 28672
22282 226/300 Flush
22283 227/300 Write block 1093 len 12288 data: 6 5 4
22284 228/300 Read block 791 len 16384
22285 229/300 Write block 3103 len 24576 data: 6 6 6 6 5 5
22286 230/300 Write block 3884 len 36864 data: 6 5 5 3 3 4 4 5 5
22287 231/300 Write block 577 len 40960 data: 7 5 4 4 4 6 4 4 5 4
22288 232/300 Write block 4349 len 8192 data: 2 4
22289 233/300 Read block 2195 len 4096
22290 234/300 Read block 2160 len 16384
22291 235/300 Read block 1670 len 4096
22292 236/300 Read block 4267 len 32768
22293 237/300 Write block 4695 len 24576 data: 6 6 7 6 5 5
22294 238/300 Write block 3731 len 28672 data: 4 6 6 5 5 5 3
22295 239/300 Write block 3983 len 36864 data: 6 4 4 5 5 5 6 6 5
22296 240/300 Write block 4534 len 28672 data: 7 6 4 5 4 5 4
22297 241/300 Read block 3897 len 32768
22298 242/300 Write block 311 len 12288 data: 4 5 4
22299 243/300 Write block 158 len 4096 data: 4
22300 244/300 Write block 229 len 28672 data: 4 6 6 6 5 5 6
22301 245/300 Write block 4905 len 24576 data: 5 4 3 3 2 2
22302 246/300 Read block 3079 len 8192
22303 247/300 Read block 1305 len 24576
22304 248/300 Write block 4641 len 32768 data: 7 7 8 7 6 5 5 5
22305 249/300 Write block 2589 len 40960 data: 3 3 3 3 2 2 3 3 3 3
22306 250/300 Read block 4086 len 24576
22307 251/300 Read block 322 len 4096
22308 252/300 Write block 3495 len 36864 data: 7 7 7 7 7 7 7 6 6
22309 253/300 Read block 1980 len 28672
22310 254/300 Write block 4161 len 12288 data: 2 2 3
22311 255/300 Read block 3744 len 16384
22312 256/300 Read block 4623 len 20480
22313 257/300 Write block 4424 len 8192 data: 4 4
22314 258/300 Read block 4498 len 12288
22315 259/300 Write block 2223 len 40960 data: 3 2 2 3 3 3 4 5 5 5
22316 260/300 Write block 4932 len 16384 data: 4 4 4 4
22317 261/300 Flush
22318 262/300 Write block 1571 len 24576 data: 8 7 6 6 5 5
22319 263/300 Flush
22320 264/300 Read block 3302 len 40960
22321 265/300 Read block 411 len 16384
22322 266/300 Write block 2261 len 8192 data: 5 6
22323 267/300 Write block 1544 len 4096 data: 5
22324 268/300 Read block 2569 len 28672
22325 269/300 Read block 4495 len 28672
22326 270/300 Read block 3341 len 24576
22327 271/300 Read block 2012 len 16384
22328 272/300 Read block 3468 len 20480
22329 273/300 Read block 1827 len 4096
22330 274/300 Read block 3192 len 4096
22331 275/300 Write block 1727 len 24576 data: 5 4 4 4 4 4
22332 276/300 Write block 418 len 28672 data: 4 5 6 5 6 6 6
22333 277/300 Read block 3362 len 28672
22334 278/300 Write block 4991 len 24576 data: 7 8 6 6 5 2
22335 279/300 Write block 1169 len 8192 data: 6 6
22336 280/300 Read block 674 len 16384
22337 281/300 Write block 2166 len 40960 data: 3 3 2 3 6 8 7 6 5 3
22338 282/300 Write block 1168 len 16384 data: 5 7 7 9
22339 283/300 Write block 4989 len 28672 data: 7 7 8 9 7 7 6
22340 284/300 Write block 2230 len 8192 data: 6 6
22341 285/300 Read block 304 len 4096
22342 286/300 Read block 4672 len 12288
22343 287/300 Write block 3346 len 36864 data: 5 5 6 6 5 6 5 3 3
22344 288/300 Write block 1835 len 4096 data: 5
22345 289/300 Write block 2844 len 40960 data: 2 2 2 3 3 3 3 3 4 4
22346 290/300 Write block 1847 len 16384 data: 5 4 3 4
22347 291/300 Write block 182 len 36864 data: 4 3 3 3 3 3 5 6 6
22348 292/300 Write block 312 len 8192 data: 6 5
22349 293/300 Read block 1238 len 20480
22350 294/300 Write block 728 len 36864 data: 4 3 3 4 7 6 6 6 8
22351 295/300 Write block 3540 len 32768 data: 2 3 3 3 5 4 4 4
22352 296/300 Read block 2079 len 24576
22353 297/300 Write block 468 len 32768 data: 6 5 5 5 4 5 5 5
22354 298/300 Read block 373 len 28672
22355 299/300 Write block 3105 len 28672 data: 7 7 6 6 4 6 5
22356 300/300 Read block 1601 len 36864
22357 301/300 Write block 3874 len 16384 data: 6 7 7 7
22358 Main task has work to do, go find it
22359 got dsc Start(2)
22360 start 2
22361 Jan 29 05:15:40.492 INFO request completed, latency_us: 284, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3cf9384b-f250-4087-a2f7-c4c366792f8a, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
22362 Replay: started 2, returned:Ok(())
22363 ----------------------------------------------------------------
22364 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22365 States: Active Active Offline
22366 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22367 6737 Acked Read 1 Done Done Sent false
22368 6738 Acked Write 6 Done Done Sent false
22369 6739 Acked Read 5 Done Done Sent false
22370 6740 Acked Write 6 Done Done Sent false
22371 6741 Acked Read 5 Done Done Sent false
22372 6742 Acked Read 6 Done Done Sent false
22373 6743 Acked Write 8 Done Done Sent false
22374 6744 Acked Write 1 Done Done Sent false
22375 6745 Acked Write 7 Done Done Sent false
22376 6746 Acked Read 4 Done Done Sent false
22377 6747 Acked Read 8 Done Done Sent false
22378 6748 Acked Flush 0 Done Done Sent false
22379 6749 Acked Write 3 Done Done Sent false
22380 6750 Acked Read 10 Done Done Sent false
22381 6751 Acked Read 10 Done Done Sent false
22382 6752 Acked Read 5 Done Done Sent false
22383 6753 Acked Read 3 Done Done Sent false
22384 6754 Acked Flush 0 Done Done Sent false
22385 6755 Acked Read 2 Done Done Sent false
22386 6756 Acked Write 2 Done Done Sent false
22387 6757 Acked Write 10 Done Done Sent false
22388 6758 Acked Read 2 Done Done Sent false
22389 6759 Acked Flush 0 Done Done Sent false
22390 6760 Acked Write 2 Done Done Sent false
22391 6761 Acked Read 7 Done Done Sent false
22392 6762 Acked Read 5 Done Done Sent false
22393 6763 Acked Read 4 Done Done Sent false
22394 6764 Acked Write 2 Done Done Sent false
22395 6765 Acked Write 1 Done Done Sent false
22396 6766 Acked Read 8 Done Done Sent false
22397 6767 Acked Write 8 Done Done Sent false
22398 6768 Acked Write 10 Done Done Sent false
22399 6769 Acked Flush 0 Done Done Sent false
22400 6770 Acked Read 8 Done Done Sent false
22401 6771 Acked Read 9 Done Done Sent false
22402 6772 Acked Write 2 Done Done Sent false
22403 6773 Acked Write 1 Done Done Sent false
22404 6774 Acked Write 4 Done Done Sent false
22405 6775 Acked Read 7 Done Done Sent false
22406 6776 Acked Write 3 Done Done Sent false
22407 6777 Acked Flush 0 Done Done Sent false
22408 6778 Acked Read 5 Done Done Sent false
22409 6779 Acked Read 8 Done Done Sent false
22410 6780 Acked Read 1 Done Done Sent false
22411 6781 Acked Read 1 Done Done Sent false
22412 6782 Acked Read 3 Done Done Sent false
22413 6783 Acked Read 9 Done Done Sent false
22414 6784 Acked Write 1 Done Done Sent false
22415 6785 Acked Write 7 Done Done Sent false
22416 6786 Acked Write 5 Done Done Sent false
22417 6787 Acked Flush 0 Done Done Sent false
22418 6788 Acked Write 6 Done Done Sent false
22419 6789 Acked Read 1 Done Done Sent false
22420 6790 Acked Write 4 Done Done Sent false
22421 6791 Acked Read 7 Done Done Sent false
22422 6792 Acked Read 7 Done Done Sent false
22423 6793 Acked Read 5 Done Done Sent false
22424 6794 Acked Read 2 Done Done Sent false
22425 6795 Acked Read 1 Done Done Sent false
22426 6796 Acked Read 7 Done Done Sent false
22427 6797 Acked Flush 0 Done Done Sent false
22428 6798 Acked Read 3 Done Done Sent false
22429 6799 Acked Write 5 Done Done Sent false
22430 6800 Acked Read 9 Done Done Sent false
22431 6801 Acked Read 9 Done Done Sent false
22432 6802 Acked Read 2 Done Done Sent false
22433 6803 Acked Read 9 Done Done Sent false
22434 6804 Acked Read 4 Done Done Sent false
22435 6805 Acked Flush 0 Done Done Sent false
22436 6806 Acked Read 6 Done Done Sent false
22437 6807 Acked Write 5 Done Done Sent false
22438 6808 Acked Read 2 Done Done Sent false
22439 6809 Acked Read 4 Done Done Sent false
22440 6810 Acked Read 2 Done Done Sent false
22441 6811 Acked Read 1 Done Done Sent false
22442 6812 Acked Read 3 Done Done Sent false
22443 6813 Acked Write 9 Done Done Sent false
22444 6814 Acked Read 6 Done Done Sent false
22445 6815 Acked Read 6 Done Done Sent false
22446 6816 Acked Read 3 Done Done Sent false
22447 6817 Acked Read 2 Done Done Sent false
22448 6818 Acked Read 9 Done Done Sent false
22449 6819 Acked Flush 0 Done Done Sent false
22450 6820 Acked Read 8 Done Done Sent false
22451 6821 Acked Read 2 Done Done Sent false
22452 6822 Acked Write 5 Done Done Sent false
22453 6823 Acked Read 3 Done Done Sent false
22454 6824 Acked Write 8 Done Done Sent false
22455 6825 Acked Write 8 Done Done Sent false
22456 6826 Acked Read 7 Done Done Sent false
22457 6827 Acked Flush 0 Done Done Sent false
22458 6828 Acked Read 3 Done Done Sent false
22459 6829 Acked Write 8 Done Done Sent false
22460 6830 Acked Read 6 Done Done Sent false
22461 6831 Acked Write 8 Done Done Sent false
22462 6832 Acked Read 7 Done Done Sent false
22463 6833 Acked Write 7 Done Done Sent false
22464 6834 Acked Read 3 Done Done Sent false
22465 6835 Acked Write 9 Done Done Sent false
22466 6836 Acked Write 2 Done Done Sent false
22467 6837 Acked Read 10 Done Done Sent false
22468 6838 Acked Write 9 Done Done Sent false
22469 6839 Acked Write 1 Done Done Sent false
22470 6840 Acked Write 7 Done Done Sent false
22471 6841 Acked Write 4 Done Done Sent false
22472 6842 Acked Write 5 Done Done Sent false
22473 6843 Acked Read 3 Done Done Sent false
22474 6844 Acked Write 8 Done Done Sent false
22475 6845 Acked Read 9 Done Done Sent false
22476 6846 Acked Write 7 Done Done Sent false
22477 6847 Acked Write 8 Done Done Sent false
22478 6848 Acked Write 6 Done Done Sent false
22479 6849 Acked Write 3 Done Done Sent false
22480 6850 Acked Write 5 Done Done Sent false
22481 6851 Acked Flush 0 Done Done Sent false
22482 6852 Acked Read 10 Done Done Sent false
22483 6853 Acked Write 7 Done Done Sent false
22484 6854 Acked Write 4 Done Done Sent false
22485 6855 Acked Read 9 Done Done Sent false
22486 6856 Acked Read 7 Done Done Sent false
22487 6857 Acked Flush 0 Done Done Sent false
22488 6858 Acked Write 2 Done Done Sent false
22489 6859 Acked Write 8 Done Done Sent false
22490 6860 Acked Write 7 Done Done Sent false
22491 6861 Acked Write 8 Done Done Sent false
22492 6862 Acked Read 6 Done Done Sent false
22493 6863 Acked Read 1 Done Done Sent false
22494 6864 Acked Read 4 Done Done Sent false
22495 6865 Acked Read 4 Done Done Sent false
22496 6866 Acked Read 3 Done Done Sent false
22497 6867 Acked Write 7 Done Done Sent false
22498 6868 Acked Write 6 Done Done Sent false
22499 6869 Acked Write 4 Done Done Sent false
22500 6870 Acked Read 8 Done Done Sent false
22501 6871 Acked Write 8 Done Done Sent false
22502 6872 Acked Write 2 Done Done Sent false
22503 6873 Acked Read 6 Done Done Sent false
22504 6874 Acked Read 3 Done Done Sent false
22505 6875 Acked Read 2 Done Done Sent false
22506 6876 Acked Read 2 Done Done Sent false
22507 6877 Acked Read 1 Done Done Sent false
22508 6878 Acked Write 6 Done Done Sent false
22509 6879 Acked Read 10 Done Done Sent false
22510 6880 Acked Read 2 Done Done Sent false
22511 6881 Acked Read 9 Done Done Sent false
22512 6882 Acked Read 8 Done Done Sent false
22513 6883 Acked Write 8 Done Done Sent false
22514 6884 Acked Read 6 Done Done Sent false
22515 6885 Acked Write 4 Done Done Sent false
22516 6886 Acked Read 9 Done Done Sent false
22517 6887 Acked Read 8 Done Done Sent false
22518 6888 Acked Read 2 Done Done Sent false
22519 6889 Acked Read 4 Done Done Sent false
22520 6890 Acked Read 10 Done Done Sent false
22521 6891 Acked Read 5 Done Done Sent false
22522 6892 Acked Read 2 Done Done Sent false
22523 6893 Acked Read 5 Done Done Sent false
22524 6894 Acked Read 2 Done Done Sent false
22525 6895 Acked Read 9 Done Done Sent false
22526 6896 Acked Write 7 Done Done Sent false
22527 6897 Acked Flush 0 Done Done Sent false
22528 6898 Acked Read 1 Done Done Sent false
22529 6899 Acked Write 10 Done Done Sent false
22530 6900 Acked Read 9 Done Done Sent false
22531 6901 Acked Read 5 Done Done Sent false
22532 6902 Acked Read 7 Done Done Sent false
22533 6903 Acked Read 7 Done Done Sent false
22534 6904 Acked Write 2 Done Done Sent false
22535 6905 Acked Write 8 Done Done Sent false
22536 6906 Acked Read 8 Done Done Sent false
22537 6907 Acked Read 4 Done Done Sent false
22538 6908 Acked Read 3 Done Done Sent false
22539 6909 Acked Flush 0 Done Done Sent false
22540 6910 Acked Write 6 Done Done Sent false
22541 6911 Acked Write 3 Done Done Sent false
22542 6912 Acked Read 4 Done Done Sent false
22543 6913 Acked Read 9 Done Done Sent false
22544 6914 Acked Read 8 Done Done Sent false
22545 6915 Acked Write 10 Done Done Sent false
22546 6916 Acked Write 1 Done Done Sent false
22547 6917 Acked Write 5 Done Done Sent false
22548 6918 Acked Flush 0 Done Done Sent false
22549 6919 Acked Write 6 Done Done Sent false
22550 6920 Acked Write 9 Done Done Sent false
22551 6921 Acked Write 6 Done Done Sent false
22552 6922 Acked Write 2 Done Done Sent false
22553 6923 Acked Flush 0 Done Done Sent false
22554 6924 Acked Write 1 Done Done Sent false
22555 6925 Acked Read 9 Done Done Sent false
22556 6926 Acked Read 1 Done Done Sent false
22557 6927 Acked Flush 0 Done Done Sent false
22558 6928 Acked Read 3 Done Done Sent false
22559 6929 Acked Read 6 Done Done Sent false
22560 6930 Acked Flush 0 Done Done Sent false
22561 6931 Acked Write 2 Done Done Sent false
22562 6932 Acked Read 9 Done Done Sent false
22563 6933 Acked Write 7 Done Done Sent false
22564 6934 Acked Read 6 Done Done Sent false
22565 6935 Acked Write 1 Done Done Sent false
22566 6936 Acked Read 4 Done Done Sent false
22567 6937 Acked Read 6 Done Done Sent false
22568 6938 Acked Flush 0 Done Done Sent false
22569 6939 Acked Write 9 Done Done Sent false
22570 6940 Acked Read 3 Done Done Sent false
22571 6941 Acked Write 6 Done Done Sent false
22572 6942 Acked Read 4 Done Done Sent false
22573 6943 Acked Read 1 Done Done Sent false
22574 6944 Acked Flush 0 Done Done Sent false
22575 6945 Acked Write 8 Done Done Sent false
22576 6946 Acked Write 6 Done Done Sent false
22577 6947 Acked Read 7 Done Done Sent false
22578 6948 Acked Write 6 Done Done Sent false
22579 6949 Acked Read 7 Done Done Sent false
22580 6950 Acked Read 8 Done Done Sent false
22581 6951 Acked Write 9 Done Done Sent false
22582 6952 Acked Read 5 Done Done Sent false
22583 6953 Acked Write 1 Done Done Sent false
22584 6954 Acked Write 9 Done Done Sent false
22585 6955 Acked Write 1 Done Done Sent false
22586 6956 Acked Read 10 Done Done Sent false
22587 6957 Acked Write 5 Done Done Sent false
22588 6958 Acked Flush 0 Done Done Sent false
22589 6959 Acked Read 7 Done Done Sent false
22590 6960 Acked Write 4 Done Done Sent false
22591 6961 Acked Read 7 Done Done Sent false
22592 6962 Acked Flush 0 Done Done Sent false
22593 6963 Acked Write 3 Done Done Sent false
22594 6964 Acked Read 4 Done Done Sent false
22595 6965 Acked Write 6 Done Done Sent false
22596 6966 Acked Write 9 Done Done Sent false
22597 6967 Acked Write 10 Done Done Sent false
22598 6968 Acked Write 2 Done Done Sent false
22599 6969 Acked Read 1 Done Done Sent false
22600 6970 Acked Read 4 Done Done Sent false
22601 6971 Acked Read 1 Done Done Sent false
22602 6972 Acked Read 8 Done Done Sent false
22603 6973 Acked Write 6 Done Done Sent false
22604 6974 Acked Write 7 Done Done Sent false
22605 6975 Acked Write 9 Done Done Sent false
22606 6976 Acked Write 7 Done Done Sent false
22607 6977 Acked Read 8 Done Done Sent false
22608 6978 Acked Write 3 Done Done Sent false
22609 6979 Acked Write 1 Done Done Sent false
22610 6980 Acked Write 7 Done Done Sent false
22611 6981 Acked Write 6 Done Done Sent false
22612 6982 Acked Read 2 Done Done Sent false
22613 6983 Acked Read 6 Done Done Sent false
22614 6984 Acked Write 8 Done Done Sent false
22615 6985 Acked Write 10 Done Done Sent false
22616 6986 Acked Read 6 Done Done Sent false
22617 6987 Acked Read 1 Done Done Sent false
22618 6988 Acked Write 9 Done Done Sent false
22619 6989 Acked Read 7 Done Done Sent false
22620 6990 Acked Write 3 Done Done Sent false
22621 6991 Acked Read 4 Done Done Sent false
22622 6992 Acked Read 5 Done Done Sent false
22623 6993 Acked Write 2 Done Done Sent false
22624 6994 Acked Read 3 Done Done Sent false
22625 6995 Acked Write 10 Done Done Sent false
22626 6996 Acked Write 4 Done Done Sent false
22627 6997 Acked Flush 0 Done Done Sent false
22628 6998 Acked Write 6 Done Done Sent false
22629 6999 Acked Flush 0 Done Done Sent false
22630 7000 Acked Read 10 Done Done Sent false
22631 7001 Acked Read 4 Done Done Sent false
22632 7002 Acked Write 2 Done Done Sent false
22633 7003 Acked Write 1 Done Done Sent false
22634 7004 Acked Read 7 Done Done Sent false
22635 7005 Acked Read 7 Done Done Sent false
22636 7006 Acked Read 6 Done Done Sent false
22637 7007 Acked Read 4 Done Done Sent false
22638 7008 Acked Read 5 Done Done Sent false
22639 7009 Acked Read 1 Done Done Sent false
22640 7010 Acked Read 1 Done Done Sent false
22641 7011 Acked Write 6 Done Done Sent false
22642 7012 Acked Write 7 Done Done Sent false
22643 7013 Acked Read 7 Done Done Sent false
22644 7014 Acked Write 6 Done Done Sent false
22645 7015 Acked Write 2 Done Done Sent false
22646 7016 Acked Read 4 Done Done Sent false
22647 7017 Acked Write 10 Done Done Sent false
22648 7018 Acked Write 4 Done Done Sent false
22649 7019 Acked Write 7 Done Done Sent false
22650 7020 Acked Write 2 Done Done Sent false
22651 7021 Acked Read 1 Done Done Sent false
22652 7022 Acked Read 3 Done Done Sent false
22653 7023 Acked Write 9 Done Done Sent false
22654 7024 Acked Write 1 Done Done Sent false
22655 7025 Acked Write 10 Done Done Sent false
22656 7026 Acked Write 4 Done Done Sent false
22657 7027 Acked Write 9 Done Done Sent false
22658 7028 Acked Write 2 Done Done Sent false
22659 7029 Acked Read 5 Done Done Sent false
22660 7030 Acked Write 9 Done Done Sent false
22661 7031 Acked Write 8 Done Done Sent false
22662 7032 Acked Read 6 Done Done Sent false
22663 7033 Acked Write 8 Done Done Sent false
22664 7034 Acked Read 7 Done Done Sent false
22665 7035 Acked Write 7 Done Done Sent false
22666 7036 Acked Read 9 Done Done Sent false
22667 7037 Acked Write 4 Done Done Sent false
22668 STATES DS:0 DS:1 DS:2 TOTAL
22669 Sent 0 0 301 301
22670 Done 301 301 0 602
22671 Skipped 0 0 0 0
22672 Error 0 0 0 0
22673 Last Flush: Some(JobId(6999)) Some(JobId(6999)) Some(JobId(6736))
22674 Downstairs last five completed: 6736 6735 6734 6733 6732
22675 Upstairs last five completed: 7037 7036 7035 7034 7033
22676 CLIENT: Up:0 ds:301 act:2
226772025-01-29T05:15:41.408ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
226782025-01-29T05:15:42.411ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
226792025-01-29T05:15:43.413ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
22680 [8830] got start action
22681 [8830] I am going to restart
22682 Starting downstairs at port 8830
22683 Make output file at "/tmp/dsc/downstairs-8830.txt"
22684 [8830][2] reports Starting
22685 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
22686 [8830][2] reports Running
226872025-01-29T05:15:44.415ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
22688 ----------------------------------------------------------------
22689 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22690 States: Active Active Offline
22691 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22692 6737 Acked Read 1 Done Done Sent false
22693 6738 Acked Write 6 Done Done Sent false
22694 6739 Acked Read 5 Done Done Sent false
22695 6740 Acked Write 6 Done Done Sent false
22696 6741 Acked Read 5 Done Done Sent false
22697 6742 Acked Read 6 Done Done Sent false
22698 6743 Acked Write 8 Done Done Sent false
22699 6744 Acked Write 1 Done Done Sent false
22700 6745 Acked Write 7 Done Done Sent false
22701 6746 Acked Read 4 Done Done Sent false
22702 6747 Acked Read 8 Done Done Sent false
22703 6748 Acked Flush 0 Done Done Sent false
22704 6749 Acked Write 3 Done Done Sent false
22705 6750 Acked Read 10 Done Done Sent false
22706 6751 Acked Read 10 Done Done Sent false
22707 6752 Acked Read 5 Done Done Sent false
22708 6753 Acked Read 3 Done Done Sent false
22709 6754 Acked Flush 0 Done Done Sent false
22710 6755 Acked Read 2 Done Done Sent false
22711 6756 Acked Write 2 Done Done Sent false
22712 6757 Acked Write 10 Done Done Sent false
22713 6758 Acked Read 2 Done Done Sent false
22714 6759 Acked Flush 0 Done Done Sent false
22715 6760 Acked Write 2 Done Done Sent false
22716 6761 Acked Read 7 Done Done Sent false
22717 6762 Acked Read 5 Done Done Sent false
22718 6763 Acked Read 4 Done Done Sent false
22719 6764 Acked Write 2 Done Done Sent false
22720 6765 Acked Write 1 Done Done Sent false
22721 6766 Acked Read 8 Done Done Sent false
22722 6767 Acked Write 8 Done Done Sent false
22723 6768 Acked Write 10 Done Done Sent false
22724 6769 Acked Flush 0 Done Done Sent false
22725 6770 Acked Read 8 Done Done Sent false
22726 6771 Acked Read 9 Done Done Sent false
22727 6772 Acked Write 2 Done Done Sent false
22728 6773 Acked Write 1 Done Done Sent false
22729 6774 Acked Write 4 Done Done Sent false
22730 6775 Acked Read 7 Done Done Sent false
22731 6776 Acked Write 3 Done Done Sent false
22732 6777 Acked Flush 0 Done Done Sent false
22733 6778 Acked Read 5 Done Done Sent false
22734 6779 Acked Read 8 Done Done Sent false
22735 6780 Acked Read 1 Done Done Sent false
22736 6781 Acked Read 1 Done Done Sent false
22737 6782 Acked Read 3 Done Done Sent false
22738 6783 Acked Read 9 Done Done Sent false
22739 6784 Acked Write 1 Done Done Sent false
22740 6785 Acked Write 7 Done Done Sent false
22741 6786 Acked Write 5 Done Done Sent false
22742 6787 Acked Flush 0 Done Done Sent false
22743 6788 Acked Write 6 Done Done Sent false
22744 6789 Acked Read 1 Done Done Sent false
22745 6790 Acked Write 4 Done Done Sent false
22746 6791 Acked Read 7 Done Done Sent false
22747 6792 Acked Read 7 Done Done Sent false
22748 6793 Acked Read 5 Done Done Sent false
22749 6794 Acked Read 2 Done Done Sent false
22750 6795 Acked Read 1 Done Done Sent false
22751 6796 Acked Read 7 Done Done Sent false
22752 6797 Acked Flush 0 Done Done Sent false
22753 6798 Acked Read 3 Done Done Sent false
22754 6799 Acked Write 5 Done Done Sent false
22755 6800 Acked Read 9 Done Done Sent false
22756 6801 Acked Read 9 Done Done Sent false
22757 6802 Acked Read 2 Done Done Sent false
22758 6803 Acked Read 9 Done Done Sent false
22759 6804 Acked Read 4 Done Done Sent false
22760 6805 Acked Flush 0 Done Done Sent false
22761 6806 Acked Read 6 Done Done Sent false
22762 6807 Acked Write 5 Done Done Sent false
22763 6808 Acked Read 2 Done Done Sent false
22764 6809 Acked Read 4 Done Done Sent false
22765 6810 Acked Read 2 Done Done Sent false
22766 6811 Acked Read 1 Done Done Sent false
22767 6812 Acked Read 3 Done Done Sent false
22768 6813 Acked Write 9 Done Done Sent false
22769 6814 Acked Read 6 Done Done Sent false
22770 6815 Acked Read 6 Done Done Sent false
22771 6816 Acked Read 3 Done Done Sent false
22772 6817 Acked Read 2 Done Done Sent false
22773 6818 Acked Read 9 Done Done Sent false
22774 6819 Acked Flush 0 Done Done Sent false
22775 6820 Acked Read 8 Done Done Sent false
22776 6821 Acked Read 2 Done Done Sent false
22777 6822 Acked Write 5 Done Done Sent false
22778 6823 Acked Read 3 Done Done Sent false
22779 6824 Acked Write 8 Done Done Sent false
22780 6825 Acked Write 8 Done Done Sent false
22781 6826 Acked Read 7 Done Done Sent false
22782 6827 Acked Flush 0 Done Done Sent false
22783 6828 Acked Read 3 Done Done Sent false
22784 6829 Acked Write 8 Done Done Sent false
22785 6830 Acked Read 6 Done Done Sent false
22786 6831 Acked Write 8 Done Done Sent false
22787 6832 Acked Read 7 Done Done Sent false
22788 6833 Acked Write 7 Done Done Sent false
22789 6834 Acked Read 3 Done Done Sent false
22790 6835 Acked Write 9 Done Done Sent false
22791 6836 Acked Write 2 Done Done Sent false
22792 6837 Acked Read 10 Done Done Sent false
22793 6838 Acked Write 9 Done Done Sent false
22794 6839 Acked Write 1 Done Done Sent false
22795 6840 Acked Write 7 Done Done Sent false
22796 6841 Acked Write 4 Done Done Sent false
22797 6842 Acked Write 5 Done Done Sent false
22798 6843 Acked Read 3 Done Done Sent false
22799 6844 Acked Write 8 Done Done Sent false
22800 6845 Acked Read 9 Done Done Sent false
22801 6846 Acked Write 7 Done Done Sent false
22802 6847 Acked Write 8 Done Done Sent false
22803 6848 Acked Write 6 Done Done Sent false
22804 6849 Acked Write 3 Done Done Sent false
22805 6850 Acked Write 5 Done Done Sent false
22806 6851 Acked Flush 0 Done Done Sent false
22807 6852 Acked Read 10 Done Done Sent false
22808 6853 Acked Write 7 Done Done Sent false
22809 6854 Acked Write 4 Done Done Sent false
22810 6855 Acked Read 9 Done Done Sent false
22811 6856 Acked Read 7 Done Done Sent false
22812 6857 Acked Flush 0 Done Done Sent false
22813 6858 Acked Write 2 Done Done Sent false
22814 6859 Acked Write 8 Done Done Sent false
22815 6860 Acked Write 7 Done Done Sent false
22816 6861 Acked Write 8 Done Done Sent false
22817 6862 Acked Read 6 Done Done Sent false
22818 6863 Acked Read 1 Done Done Sent false
22819 6864 Acked Read 4 Done Done Sent false
22820 6865 Acked Read 4 Done Done Sent false
22821 6866 Acked Read 3 Done Done Sent false
22822 6867 Acked Write 7 Done Done Sent false
22823 6868 Acked Write 6 Done Done Sent false
22824 6869 Acked Write 4 Done Done Sent false
22825 6870 Acked Read 8 Done Done Sent false
22826 6871 Acked Write 8 Done Done Sent false
22827 6872 Acked Write 2 Done Done Sent false
22828 6873 Acked Read 6 Done Done Sent false
22829 6874 Acked Read 3 Done Done Sent false
22830 6875 Acked Read 2 Done Done Sent false
22831 6876 Acked Read 2 Done Done Sent false
22832 6877 Acked Read 1 Done Done Sent false
22833 6878 Acked Write 6 Done Done Sent false
22834 6879 Acked Read 10 Done Done Sent false
22835 6880 Acked Read 2 Done Done Sent false
22836 6881 Acked Read 9 Done Done Sent false
22837 6882 Acked Read 8 Done Done Sent false
22838 6883 Acked Write 8 Done Done Sent false
22839 6884 Acked Read 6 Done Done Sent false
22840 6885 Acked Write 4 Done Done Sent false
22841 6886 Acked Read 9 Done Done Sent false
22842 6887 Acked Read 8 Done Done Sent false
22843 6888 Acked Read 2 Done Done Sent false
22844 6889 Acked Read 4 Done Done Sent false
22845 6890 Acked Read 10 Done Done Sent false
22846 6891 Acked Read 5 Done Done Sent false
22847 6892 Acked Read 2 Done Done Sent false
22848 6893 Acked Read 5 Done Done Sent false
22849 6894 Acked Read 2 Done Done Sent false
22850 6895 Acked Read 9 Done Done Sent false
22851 6896 Acked Write 7 Done Done Sent false
22852 6897 Acked Flush 0 Done Done Sent false
22853 6898 Acked Read 1 Done Done Sent false
22854 6899 Acked Write 10 Done Done Sent false
22855 6900 Acked Read 9 Done Done Sent false
22856 6901 Acked Read 5 Done Done Sent false
22857 6902 Acked Read 7 Done Done Sent false
22858 6903 Acked Read 7 Done Done Sent false
22859 6904 Acked Write 2 Done Done Sent false
22860 6905 Acked Write 8 Done Done Sent false
22861 6906 Acked Read 8 Done Done Sent false
22862 6907 Acked Read 4 Done Done Sent false
22863 6908 Acked Read 3 Done Done Sent false
22864 6909 Acked Flush 0 Done Done Sent false
22865 6910 Acked Write 6 Done Done Sent false
22866 6911 Acked Write 3 Done Done Sent false
22867 6912 Acked Read 4 Done Done Sent false
22868 6913 Acked Read 9 Done Done Sent false
22869 6914 Acked Read 8 Done Done Sent false
22870 6915 Acked Write 10 Done Done Sent false
22871 6916 Acked Write 1 Done Done Sent false
22872 6917 Acked Write 5 Done Done Sent false
22873 6918 Acked Flush 0 Done Done Sent false
22874 6919 Acked Write 6 Done Done Sent false
22875 6920 Acked Write 9 Done Done Sent false
22876 6921 Acked Write 6 Done Done Sent false
22877 6922 Acked Write 2 Done Done Sent false
22878 6923 Acked Flush 0 Done Done Sent false
22879 6924 Acked Write 1 Done Done Sent false
22880 6925 Acked Read 9 Done Done Sent false
22881 6926 Acked Read 1 Done Done Sent false
22882 6927 Acked Flush 0 Done Done Sent false
22883 6928 Acked Read 3 Done Done Sent false
22884 6929 Acked Read 6 Done Done Sent false
22885 6930 Acked Flush 0 Done Done Sent false
22886 6931 Acked Write 2 Done Done Sent false
22887 6932 Acked Read 9 Done Done Sent false
22888 6933 Acked Write 7 Done Done Sent false
22889 6934 Acked Read 6 Done Done Sent false
22890 6935 Acked Write 1 Done Done Sent false
22891 6936 Acked Read 4 Done Done Sent false
22892 6937 Acked Read 6 Done Done Sent false
22893 6938 Acked Flush 0 Done Done Sent false
22894 6939 Acked Write 9 Done Done Sent false
22895 6940 Acked Read 3 Done Done Sent false
22896 6941 Acked Write 6 Done Done Sent false
22897 6942 Acked Read 4 Done Done Sent false
22898 6943 Acked Read 1 Done Done Sent false
22899 6944 Acked Flush 0 Done Done Sent false
22900 6945 Acked Write 8 Done Done Sent false
22901 6946 Acked Write 6 Done Done Sent false
22902 6947 Acked Read 7 Done Done Sent false
22903 6948 Acked Write 6 Done Done Sent false
22904 6949 Acked Read 7 Done Done Sent false
22905 6950 Acked Read 8 Done Done Sent false
22906 6951 Acked Write 9 Done Done Sent false
22907 6952 Acked Read 5 Done Done Sent false
22908 6953 Acked Write 1 Done Done Sent false
22909 6954 Acked Write 9 Done Done Sent false
22910 6955 Acked Write 1 Done Done Sent false
22911 6956 Acked Read 10 Done Done Sent false
22912 6957 Acked Write 5 Done Done Sent false
22913 6958 Acked Flush 0 Done Done Sent false
22914 6959 Acked Read 7 Done Done Sent false
22915 6960 Acked Write 4 Done Done Sent false
22916 6961 Acked Read 7 Done Done Sent false
22917 6962 Acked Flush 0 Done Done Sent false
22918 6963 Acked Write 3 Done Done Sent false
22919 6964 Acked Read 4 Done Done Sent false
22920 6965 Acked Write 6 Done Done Sent false
22921 6966 Acked Write 9 Done Done Sent false
22922 6967 Acked Write 10 Done Done Sent false
22923 6968 Acked Write 2 Done Done Sent false
22924 6969 Acked Read 1 Done Done Sent false
22925 6970 Acked Read 4 Done Done Sent false
22926 6971 Acked Read 1 Done Done Sent false
22927 6972 Acked Read 8 Done Done Sent false
22928 6973 Acked Write 6 Done Done Sent false
22929 6974 Acked Write 7 Done Done Sent false
22930 6975 Acked Write 9 Done Done Sent false
22931 6976 Acked Write 7 Done Done Sent false
22932 6977 Acked Read 8 Done Done Sent false
22933 6978 Acked Write 3 Done Done Sent false
22934 6979 Acked Write 1 Done Done Sent false
22935 6980 Acked Write 7 Done Done Sent false
22936 6981 Acked Write 6 Done Done Sent false
22937 6982 Acked Read 2 Done Done Sent false
22938 6983 Acked Read 6 Done Done Sent false
22939 6984 Acked Write 8 Done Done Sent false
22940 6985 Acked Write 10 Done Done Sent false
22941 6986 Acked Read 6 Done Done Sent false
22942 6987 Acked Read 1 Done Done Sent false
22943 6988 Acked Write 9 Done Done Sent false
22944 6989 Acked Read 7 Done Done Sent false
22945 6990 Acked Write 3 Done Done Sent false
22946 6991 Acked Read 4 Done Done Sent false
22947 6992 Acked Read 5 Done Done Sent false
22948 6993 Acked Write 2 Done Done Sent false
22949 6994 Acked Read 3 Done Done Sent false
22950 6995 Acked Write 10 Done Done Sent false
22951 6996 Acked Write 4 Done Done Sent false
22952 6997 Acked Flush 0 Done Done Sent false
22953 6998 Acked Write 6 Done Done Sent false
22954 6999 Acked Flush 0 Done Done Sent false
22955 7000 Acked Read 10 Done Done Sent false
22956 7001 Acked Read 4 Done Done Sent false
22957 7002 Acked Write 2 Done Done Sent false
22958 7003 Acked Write 1 Done Done Sent false
22959 7004 Acked Read 7 Done Done Sent false
22960 7005 Acked Read 7 Done Done Sent false
22961 7006 Acked Read 6 Done Done Sent false
22962 7007 Acked Read 4 Done Done Sent false
22963 7008 Acked Read 5 Done Done Sent false
22964 7009 Acked Read 1 Done Done Sent false
22965 7010 Acked Read 1 Done Done Sent false
22966 7011 Acked Write 6 Done Done Sent false
22967 7012 Acked Write 7 Done Done Sent false
22968 7013 Acked Read 7 Done Done Sent false
22969 7014 Acked Write 6 Done Done Sent false
22970 7015 Acked Write 2 Done Done Sent false
22971 7016 Acked Read 4 Done Done Sent false
22972 7017 Acked Write 10 Done Done Sent false
22973 7018 Acked Write 4 Done Done Sent false
22974 7019 Acked Write 7 Done Done Sent false
22975 7020 Acked Write 2 Done Done Sent false
22976 7021 Acked Read 1 Done Done Sent false
22977 7022 Acked Read 3 Done Done Sent false
22978 7023 Acked Write 9 Done Done Sent false
22979 7024 Acked Write 1 Done Done Sent false
22980 7025 Acked Write 10 Done Done Sent false
22981 7026 Acked Write 4 Done Done Sent false
22982 7027 Acked Write 9 Done Done Sent false
22983 7028 Acked Write 2 Done Done Sent false
22984 7029 Acked Read 5 Done Done Sent false
22985 7030 Acked Write 9 Done Done Sent false
22986 7031 Acked Write 8 Done Done Sent false
22987 7032 Acked Read 6 Done Done Sent false
22988 7033 Acked Write 8 Done Done Sent false
22989 7034 Acked Read 7 Done Done Sent false
22990 7035 Acked Write 7 Done Done Sent false
22991 7036 Acked Read 9 Done Done Sent false
22992 7037 Acked Write 4 Done Done Sent false
22993 STATES DS:0 DS:1 DS:2 TOTAL
22994 Sent 0 0 301 301
22995 Done 301 301 0 602
22996 Skipped 0 0 0 0
22997 Error 0 0 0 0
22998 Last Flush: Some(JobId(6999)) Some(JobId(6999)) Some(JobId(6736))
22999 Downstairs last five completed: 6736 6735 6734 6733 6732
23000 Upstairs last five completed: 7037 7036 7035 7034 7033
23001 CLIENT: Up:0 ds:301 act:2
230022025-01-29T05:15:45.746ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230032025-01-29T05:15:46.254ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230042025-01-29T05:15:46.255ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56094) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230052025-01-29T05:15:46.256ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230062025-01-29T05:15:46.256ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230072025-01-29T05:15:46.256ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230082025-01-29T05:15:46.256ZINFOcrucible: send last flush ID to this DS: Some(JobId(6736)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230092025-01-29T05:15:46.256ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230102025-01-29T05:15:46.256ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6736)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230112025-01-29T05:15:46.257ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6736)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
23012 ----------------------------------------------------------------
23013 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23014 Downstairs last five completed: 7038 7037 7036 7035 7034
23015 Upstairs last five completed: 7038 7037 7036 7035 7034
23016 CLIENT: Up:0 ds:0 act:3
23017 Replay: All jobs finished
23018 Main task has work to do, go find it
23019 got dsc Stop(1)
23020 stop 1
23021 Jan 29 05:15:48.499 INFO request completed, latency_us[8820] Got stop action so:false kr:false
23022 : 381, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 300e7ef7-f1da-42a1-86fa-f2602464c306, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
230232025-01-29T05:15:48.499ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230242025-01-29T05:15:48.499ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
230252025-01-29T05:15:48.499ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
23026 Jan 29 05:15:48.499 INFO request completed, latency_us: 146, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1f41feb5-e621-4b75-bf52-305ce6a21aad, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:15:48.499880465Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
23027 
23028 [8820] Exited with: ExitStatus(unix_wait_status(9))
23029 [8820][1] reports Exit
23030 Jan 29 05:15:52.502 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/1, method: GET, req_id: 33d20067-1dd7-4dd2-842a-405d94cd61cc, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
23031 001/300 Write block 1502 len 20480 data: 7 7 6 6 7
23032 002/300 Write block 2927 len 32768 data: 6 6 5 6 6 6 4 3
23033 003/300 Write block 3427 len 12288 data: 6 6 7
23034 004/300 Write block 898 len 12288 data: 7 6 6
23035 005/300 Write block 3800 len 4096 data: 4
23036 006/300 Read block 4589 len 28672
23037 007/300 Write block 3896 len 16384 data: 5 4 4 6
23038 008/300 Read block 3910 len 24576
23039 009/300 Read block 3132 len 40960
23040 010/300 Flush
23041 011/300 Flush
23042 012/300 Write block 3480 len 40960 data: 5 6 6 7 6 5 6 4 4 5
23043 013/300 Write block 4899 len 8192 data: 5 5
23044 014/300 Write block 2648 len 28672 data: 2 2 3 2 2 3 3
23045 015/300 Read block 2913 len 24576
23046 016/300 Write block 2602 len 28672 data: 6 6 6 4 4 6 8
23047 017/300 Read block 2383 len 32768
23048 018/300 Read block 4580 len 32768
23049 019/300 Read block 3446 len 16384
23050 020/300 Write block 2007 len 8192 data: 5 7
23051 021/300 Write block 1990 len 4096 data: 5
23052 022/300 Read block 3934 len 8192
23053 023/300 Write block 1047 len 16384 data: 6 4 4 5
23054 024/300 Write block 4896 len 12288 data: 4 4 5
23055 025/300 Write block 427 len 32768 data: 6 4 4 3 3 2 2 2
23056 026/300 Read block 2023 len 8192
23057 027/300 Read block 3920 len 24576
23058 028/300 Flush
23059 029/300 Read block 1813 len 4096
23060 030/300 Read block 4528 len 24576
23061 031/300 Read block 954 len 8192
23062 032/300 Write block 1012 len 28672 data: 5 5 3 3 3 3 2
23063 033/300 Flush
23064 034/300 Write block 3529 len 4096 data: 5
23065 035/300 Write block 2552 len 36864 data: 4 5 4 4 4 5 5 6 5
23066 036/300 Write block 2193 len 16384 data: 3 3 2 2
23067 037/300 Write block 1157 len 12288 data: 3 2 3
23068 038/300 Write block 1354 len 4096 data: 3
23069 039/300 Read block 4726 len 40960
23070 040/300 Write block 1071 len 28672 data: 5 5 6 6 5 4 4
23071 041/300 Write block 4004 len 16384 data: 8 7 7 6
23072 042/300 Read block 2856 len 16384
23073 043/300 Read block 741 len 8192
23074 044/300 Read block 1058 len 16384
23075 045/300 Read block 2861 len 40960
23076 046/300 Write block 1689 len 12288 data: 10 12 12
23077 047/300 Read block 4294 len 40960
23078 048/300 Write block 2228 len 28672 data: 4 5 7 7 6 5 4
23079 049/300 Write block 4092 len 36864 data: 4 2 3 4 4 4 5 5 6
23080 050/300 Write block 1683 len 20480 data: 9 9 8 6 8
23081 051/300 Read block 1102 len 24576
23082 052/300 Read block 2669 len 12288
23083 053/300 Write block 3946 len 20480 data: 6 6 6 5 4
23084 054/300 Write block 2842 len 24576 data: 5 4 3 3 3 4
23085 055/300 Read block 4261 len 4096
23086 056/300 Write block 4894 len 36864 data: 4 5 5 5 6 6 6 3 4
23087 057/300 Write block 2850 len 8192 data: 4 4
23088 058/300 Read block 4538 len 36864
23089 059/300 Read block 4640 len 40960
23090 060/300 Write block 2331 len 12288 data: 4 3 4
23091 061/300 Read block 4062 len 12288
23092 062/300 Read block 2150 len 28672
23093 063/300 Read block 2131 len 28672
23094 064/300 Read block 4128 len 16384
23095 065/300 Write block 2702 len 16384 data: 7 7 4 4
23096 066/300 Read block 3298 len 36864
23097 067/300 Read block 885 len 8192
23098 068/300 Write block 6 len 24576 data: 4 4 4 4 3 2
23099 069/300 Read block 3372 len 24576
23100 070/300 Write block 3719 len 32768 data: 5 5 4 5 6 5 4 5
23101 071/300 Write block 1991 len 40960 data: 6 6 3 2 2 3 3 4 5 5
23102 072/300 Read block 4493 len 28672
23103 073/300 Write block 3678 len 24576 data: 4 4 4 5 6 5
23104 074/300 Flush
23105 075/300 Read block 625 len 28672
23106 076/300 Write block 1018 len 16384 data: 3 3 3 3
23107 077/300 Write block 1561 len 4096 data: 5
23108 078/300 Read block 2318 len 8192
23109 079/300 Read block 1117 len 24576
23110 080/300 Write block 4747 len 24576 data: 5 6 5 5 5 3
23111 081/300 Write block 456 len 20480 data: 6 6 5 5 5
23112 082/300 Write block 2581 len 32768 data: 2 2 2 2 2 3 3 3
23113 083/300 Write block 2657 len 28672 data: 3 4 6 5 4 4 6
23114 084/300 Read block 1483 len 20480
23115 085/300 Flush
23116 086/300 Write block 487 len 40960 data: 7 3 4 6 8 8 8 8 7 8
23117 087/300 Read block 56 len 4096
23118 088/300 Write block 4527 len 36864 data: 4 6 6 6 6 6 7 8 7
23119 089/300 Write block 1191 len 24576 data: 4 4 5 4 3 3
23120 090/300 Read block 2479 len 12288
23121 091/300 Read block 3810 len 20480
23122 092/300 Read block 4851 len 8192
23123 093/300 Write block 1401 len 40960 data: 5 5 5 5 4 4 4 3 3 5
23124 094/300 Write block 2237 len 40960 data: 5 4 3 4 5 5 5 4 4 4
23125 095/300 Read block 1726 len 16384
23126 096/300 Flush
23127 097/300 Write block 1901 len 36864 data: 3 4 5 5 5 7 7 7 6
23128 098/300 Read block 4713 len 20480
23129 099/300 Read block 4298 len 12288
23130 100/300 Flush
23131 101/300 Read block 862 len 32768
23132 102/300 Read block 2637 len 24576
23133 103/300 Read block 179 len 28672
23134 104/300 Read block 2362 len 8192
23135 105/300 Read block 1723 len 40960
23136 106/300 Write block 3744 len 20480 data: 4 3 3 3 3
23137 107/300 Read block 3329 len 24576
23138 108/300 Flush
23139 109/300 Read block 2782 len 16384
23140 110/300 Flush
23141 111/300 Read block 459 len 20480
23142 112/300 Read block 4401 len 16384
23143 113/300 Write block 1056 len 16384 data: 4 3 3 4
23144 114/300 Write block 3497 len 20480 data: 8 8 8 8 8
23145 115/300 Write block 3587 len 4096 data: 2
23146 116/300 Write block 2894 len 36864 data: 4 4 5 3 2 2 2 2 3
23147 117/300 Write block 1823 len 28672 data: 6 5 4 4 4 4 4
23148 118/300 Read block 1370 len 32768
23149 119/300 Read block 3152 len 24576
23150 120/300 Read block 1030 len 20480
23151 121/300 Flush
23152 122/300 Read block 214 len 28672
23153 123/300 Flush
23154 124/300 Write block 320 len 4096 data: 3
23155 125/300 Read block 1022 len 40960
23156 126/300 Write block 3337 len 40960 data: 4 4 3 2 2 2 3 4 5 6
23157 127/300 Flush
23158 128/300 Read block 4502 len 28672
23159 129/300 Flush
23160 130/300 Read block 307 len 40960
23161 131/300 Write block 2587 len 28672 data: 4 4 4 4 4 4 3
23162 132/300 Read block 159 len 16384
23163 133/300 Read block 2863 len 40960
23164 134/300 Read block 2883 len 36864
23165 135/300 Write block 4930 len 12288 data: 5 5 5
23166 136/300 Flush
23167 137/300 Read block 3268 len 36864
23168 138/300 Read block 209 len 32768
23169 139/300 Flush
23170 140/300 Read block 201 len 24576
23171 141/300 Write block 3466 len 36864 data: 4 5 5 5 7 5 3 3 4
23172 142/300 Read block 3095 len 20480
23173 143/300 Flush
23174 144/300 Read block 416 len 40960
23175 145/300 Write block 4159 len 12288 data: 2 2 3
23176 146/300 Write block 3760 len 28672 data: 2 3 4 6 7 8 8
23177 147/300 Write block 3515 len 36864 data: 6 6 4 3 4 5 5 6 6
23178 148/300 Read block 3715 len 40960
23179 149/300 Write block 1243 len 12288 data: 5 5 5
23180 150/300 Flush
23181 151/300 Write block 1610 len 28672 data: 4 4 4 4 4 4 4
23182 152/300 Flush
23183 153/300 Read block 3922 len 24576
23184 154/300 Flush
23185 155/300 Write block 1074 len 40960 data: 7 6 5 5 4 4 4 4 3 5
23186 156/300 Write block 1351 len 8192 data: 2 2
23187 157/300 Write block 586 len 12288 data: 5 5 5
23188 158/300 Write block 1974 len 4096 data: 7
23189 159/300 Read block 1406 len 16384
23190 160/300 Write block 764 len 28672 data: 4 5 6 5 3 2 3
23191 161/300 Read block 96 len 8192
23192 162/300 Read block 4673 len 4096
23193 163/300 Write block 1195 len 28672 data: 4 4 3 3 3 3 3
23194 164/300 Write block 4960 len 40960 data: 5 5 4 4 5 4 4 5 4 4
23195 165/300 Read block 2201 len 28672
23196 166/300 Flush
23197 167/300 Read block 4304 len 24576
23198 168/300 Write block 720 len 40960 data: 3 4 4 4 4 5 4 4 5 4
23199 169/300 Write block 3516 len 28672 data: 7 5 4 5 6 6 7
23200 170/300 Read block 711 len 12288
23201 171/300 Read block 1721 len 36864
23202 172/300 Flush
23203 173/300 Write block 4846 len 32768 data: 4 3 4 3 3 4 4 4
23204 174/300 Write block 2342 len 20480 data: 6 4 4 4 5
23205 175/300 Read block 222 len 8192
23206 176/300 Read block 2565 len 24576
23207 177/300 Write block 4206 len 12288 data: 7 6 5
23208 178/300 Read block 2830 len 28672
23209 179/300 Write block 874 len 32768 data: 6 5 5 6 5 5 7 5
23210 180/300 Read block 4408 len 16384
23211 181/300 Write block 1739 len 8192 data: 2 2
23212 182/300 Flush
23213 183/300 Write block 3749 len 8192 data: 5 5
23214 184/300 Flush
23215 185/300 Read block 1679 len 28672
23216 186/300 Read block 1090 len 16384
23217 187/300 Read block 3464 len 16384
23218 188/300 Read block 940 len 28672
23219 189/300 Read block 2773 len 20480
23220 190/300 Write block 282 len 12288 data: 5 5 6
23221 191/300 Read block 3211 len 24576
23222 192/300 Write block 165 len 28672 data: 4 4 4 4 6 8 8
23223 193/300 Read block 1607 len 28672
23224 194/300 Read block 3433 len 28672
23225 195/300 Write block 4330 len 12288 data: 6 6 3
23226 196/300 Write block 1460 len 28672 data: 4 4 3 3 2 2 3
23227 197/300 Write block 31 len 8192 data: 3 4
23228 198/300 Write block 4809 len 20480 data: 4 6 5 5 5
23229 199/300 Write block 4111 len 8192 data: 4 4
23230 200/300 Write block 4915 len 28672 data: 2 3 3 3 3 3 4
23231 201/300 Write block 337 len 32768 data: 5 6 5 5 3 3 4 5
23232 202/300 Read block 1190 len 8192
23233 203/300 Read block 1457 len 8192
23234 204/300 Read block 1340 len 20480
23235 205/300 Flush
23236 206/300 Read block 3334 len 16384
23237 207/300 Write block 3062 len 4096 data: 4
23238 208/300 Read block 1871 len 8192
23239 209/300 Read block 1408 len 16384
23240 210/300 Write block 3669 len 32768 data: 4 4 4 4 4 4 2 3
23241 211/300 Read block 4190 len 24576
23242 212/300 Flush
23243 213/300 Flush
23244 214/300 Write block 3868 len 36864 data: 4 4 4 3 4 5 7 8 8
23245 215/300 Read block 3134 len 24576
23246 216/300 Read block 3641 len 36864
23247 217/300 Read block 2800 len 8192
23248 218/300 Write block 3476 len 32768 data: 4 3 4 5 6 7 7 8
23249 219/300 Read block 2915 len 32768
23250 220/300 Read block 3984 len 12288
23251 221/300 Read block 4261 len 16384
23252 222/300 Write block 3576 len 40960 data: 11 10 10 6 5 5 4 2 2 2
23253 223/300 Read block 3409 len 40960
23254 224/300 Write block 1851 len 24576 data: 3 3 3 3 3 4
23255 225/300 Read block 4365 len 32768
23256 226/300 Read block 2111 len 24576
23257 227/300 Read block 2747 len 28672
23258 228/300 Read block 416 len 24576
23259 229/300 Read block 4821 len 40960
23260 230/300 Read block 2164 len 40960
23261 231/300 Read block 312 len 12288
23262 232/300 Read block 1147 len 16384
23263 233/300 Write block 2085 len 12288 data: 5 5 4
23264 234/300 Write block 2110 len 32768 data: 6 7 7 6 5 4 4 3
23265 235/300 Write block 447 len 16384 data: 5 4 3 3
23266 236/300 Write block 1686 len 16384 data: 7 9 9 11
23267 237/300 Write block 495 len 36864 data: 8 9 8 9 7 7 5 8 7
23268 238/300 Read block 882 len 12288
23269 239/300 Read block 12 len 12288
23270 240/300 Write block 1352 len 16384 data: 3 3 4 3
23271 241/300 Write block 1624 len 40960 data: 9 7 8 7 7 6 5 5 6 5
23272 242/300 Read block 3358 len 32768
23273 243/300 Flush
23274 244/300 Read block 787 len 32768
23275 245/300 Flush
23276 246/300 Write block 1934 len 32768 data: 3 4 7 8 7 8 9 7
23277 247/300 Write block 1198 len 24576 data: 4 4 4 4 2 2
23278 248/300 Write block 4901 len 40960 data: 4 5 3 4 6 5 4 4 3 3
23279 249/300 Read block 1820 len 16384
23280 250/300 Read block 3298 len 32768
23281 251/300 Flush
23282 252/300 Read block 2978 len 4096
23283 253/300 Write block 345 len 20480 data: 4 4 5 6 5
23284 254/300 Write block 4009 len 12288 data: 6 5 5
23285 255/300 Read block 4081 len 12288
23286 256/300 Read block 3493 len 8192
23287 257/300 Flush
23288 258/300 Read block 1864 len 36864
23289 259/300 Read block 367 len 36864
23290 260/300 Read block 565 len 8192
23291 261/300 Write block 1011 len 36864 data: 5 6 6 4 4 4 4 4 4
23292 262/300 Flush
23293 263/300 Read block 633 len 40960
23294 264/300 Read block 1898 len 16384
23295 265/300 Write block 2985 len 20480 data: 6 6 6 5 5
23296 266/300 Read block 2038 len 8192
23297 267/300 Flush
23298 268/300 Read block 3328 len 16384
23299 269/300 Read block 2496 len 36864
23300 270/300 Write block 1437 len 24576 data: 5 5 5 5 5 4
23301 271/300 Write block 619 len 24576 data: 3 3 3 3 3 3
23302 272/300 Write block 1958 len 20480 data: 2 3 3 4 4
23303 273/300 Write block 3535 len 4096 data: 5
23304 274/300 Write block 1139 len 24576 data: 3 3 3 3 4 3
23305 275/300 Write block 3815 len 24576 data: 3 3 5 4 4 4
23306 276/300 Write block 899 len 16384 data: 7 7 5 4
23307 277/300 Read block 2578 len 4096
23308 278/300 Read block 3859 len 28672
23309 279/300 Flush
23310 280/300 Write block 4018 len 8192 data: 3 3
23311 281/300 Write block 2969 len 36864 data: 4 4 5 6 4 3 4 8 7
23312 282/300 Read block 1801 len 28672
23313 283/300 Read block 627 len 20480
23314 284/300 Write block 2486 len 28672 data: 7 4 3 4 4 4 5
23315 285/300 Flush
23316 286/300 Flush
23317 287/300 Write block 2409 len 16384 data: 5 4 5 6
23318 288/300 Read block 3991 len 12288
23319 289/300 Read block 2886 len 40960
23320 290/300 Read block 4143 len 20480
23321 291/300 Write block 1362 len 4096 data: 6
23322 292/300 Flush
23323 293/300 Read block 4960 len 24576
23324 294/300 Read block 661 len 20480
23325 295/300 Write block 4634 len 36864 data: 3 4 6 7 8 7 8 8 8
23326 296/300 Read block 2756 len 8192
23327 297/300 Read block 467 len 24576
23328 298/300 Read block 4795 len 24576
23329 299/300 Write block 4308 len 8192 data: 5 5
23330 300/300 Write block 4267 len 8192 data: 3 3
23331 301/300 Read block 1938 len 20480
23332 Main task has work to do, go find it
23333 got dsc Start(1)
23334 start 1
23335 Jan 29 05:15:52.739 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/1, method: POST, req_id: adad00a4-ed26-44fc-b666-8481f1567bb3, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
23336 Replay: started 1, returned:Ok(())
23337 ----------------------------------------------------------------
23338 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23339 States: Active Offline Active
23340 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23341 7039 Acked Write 5 Done Sent Done false
23342 7040 Acked Write 8 Done Sent Done false
23343 7041 Acked Write 3 Done Sent Done false
23344 7042 Acked Write 3 Done Sent Done false
23345 7043 Acked Write 1 Done Sent Done false
23346 7044 Acked Read 7 Done Sent Done false
23347 7045 Acked Write 4 Done Sent Done false
23348 7046 Acked Read 6 Done Sent Done false
23349 7047 Acked Read 10 Done Sent Done false
23350 7048 Acked Flush 0 Done Sent Done false
23351 7049 Acked Flush 0 Done Sent Done false
23352 7050 Acked Write 10 Done Sent Done false
23353 7051 Acked Write 2 Done Sent Done false
23354 7052 Acked Write 7 Done Sent Done false
23355 7053 Acked Read 6 Done Sent Done false
23356 7054 Acked Write 7 Done Sent Done false
23357 7055 Acked Read 8 Done Sent Done false
23358 7056 Acked Read 8 Done Sent Done false
23359 7057 Acked Read 4 Done Sent Done false
23360 7058 Acked Write 2 Done Sent Done false
23361 7059 Acked Write 1 Done Sent Done false
23362 7060 Acked Read 2 Done Sent Done false
23363 7061 Acked Write 4 Done Sent Done false
23364 7062 Acked Write 3 Done Sent Done false
23365 7063 Acked Write 8 Done Sent Done false
23366 7064 Acked Read 2 Done Sent Done false
23367 7065 Acked Read 6 Done Sent Done false
23368 7066 Acked Flush 0 Done Sent Done false
23369 7067 Acked Read 1 Done Sent Done false
23370 7068 Acked Read 6 Done Sent Done false
23371 7069 Acked Read 2 Done Sent Done false
23372 7070 Acked Write 7 Done Sent Done false
23373 7071 Acked Flush 0 Done Sent Done false
23374 7072 Acked Write 1 Done Sent Done false
23375 7073 Acked Write 9 Done Sent Done false
23376 7074 Acked Write 4 Done Sent Done false
23377 7075 Acked Write 3 Done Sent Done false
23378 7076 Acked Write 1 Done Sent Done false
23379 7077 Acked Read 10 Done Sent Done false
23380 7078 Acked Write 7 Done Sent Done false
23381 7079 Acked Write 4 Done Sent Done false
23382 7080 Acked Read 4 Done Sent Done false
23383 7081 Acked Read 2 Done Sent Done false
23384 7082 Acked Read 4 Done Sent Done false
23385 7083 Acked Read 10 Done Sent Done false
23386 7084 Acked Write 3 Done Sent Done false
23387 7085 Acked Read 10 Done Sent Done false
23388 7086 Acked Write 7 Done Sent Done false
23389 7087 Acked Write 9 Done Sent Done false
23390 7088 Acked Write 5 Done Sent Done false
23391 7089 Acked Read 6 Done Sent Done false
23392 7090 Acked Read 3 Done Sent Done false
23393 7091 Acked Write 5 Done Sent Done false
23394 7092 Acked Write 6 Done Sent Done false
23395 7093 Acked Read 1 Done Sent Done false
23396 7094 Acked Write 9 Done Sent Done false
23397 7095 Acked Write 2 Done Sent Done false
23398 7096 Acked Read 9 Done Sent Done false
23399 7097 Acked Read 10 Done Sent Done false
23400 7098 Acked Write 3 Done Sent Done false
23401 7099 Acked Read 3 Done Sent Done false
23402 7100 Acked Read 7 Done Sent Done false
23403 7101 Acked Read 7 Done Sent Done false
23404 7102 Acked Read 4 Done Sent Done false
23405 7103 Acked Write 4 Done Sent Done false
23406 7104 Acked Read 9 Done Sent Done false
23407 7105 Acked Read 2 Done Sent Done false
23408 7106 Acked Write 6 Done Sent Done false
23409 7107 Acked Read 6 Done Sent Done false
23410 7108 Acked Write 8 Done Sent Done false
23411 7109 Acked Write 10 Done Sent Done false
23412 7110 Acked Read 7 Done Sent Done false
23413 7111 Acked Write 6 Done Sent Done false
23414 7112 Acked Flush 0 Done Sent Done false
23415 7113 Acked Read 7 Done Sent Done false
23416 7114 Acked Write 4 Done Sent Done false
23417 7115 Acked Write 1 Done Sent Done false
23418 7116 Acked Read 2 Done Sent Done false
23419 7117 Acked Read 6 Done Sent Done false
23420 7118 Acked Write 6 Done Sent Done false
23421 7119 Acked Write 5 Done Sent Done false
23422 7120 Acked Write 8 Done Sent Done false
23423 7121 Acked Write 7 Done Sent Done false
23424 7122 Acked Read 5 Done Sent Done false
23425 7123 Acked Flush 0 Done Sent Done false
23426 7124 Acked Write 10 Done Sent Done false
23427 7125 Acked Read 1 Done Sent Done false
23428 7126 Acked Write 9 Done Sent Done false
23429 7127 Acked Write 6 Done Sent Done false
23430 7128 Acked Read 3 Done Sent Done false
23431 7129 Acked Read 5 Done Sent Done false
23432 7130 Acked Read 2 Done Sent Done false
23433 7131 Acked Write 10 Done Sent Done false
23434 7132 Acked Write 10 Done Sent Done false
23435 7133 Acked Read 4 Done Sent Done false
23436 7134 Acked Flush 0 Done Sent Done false
23437 7135 Acked Write 9 Done Sent Done false
23438 7136 Acked Read 5 Done Sent Done false
23439 7137 Acked Read 3 Done Sent Done false
23440 7138 Acked Flush 0 Done Sent Done false
23441 7139 Acked Read 8 Done Sent Done false
23442 7140 Acked Read 6 Done Sent Done false
23443 7141 Acked Read 7 Done Sent Done false
23444 7142 Acked Read 2 Done Sent Done false
23445 7143 Acked Read 10 Done Sent Done false
23446 7144 Acked Write 5 Done Sent Done false
23447 7145 Acked Read 6 Done Sent Done false
23448 7146 Acked Flush 0 Done Sent Done false
23449 7147 Acked Read 4 Done Sent Done false
23450 7148 Acked Flush 0 Done Sent Done false
23451 7149 Acked Read 5 Done Sent Done false
23452 7150 Acked Read 4 Done Sent Done false
23453 7151 Acked Write 4 Done Sent Done false
23454 7152 Acked Write 5 Done Sent Done false
23455 7153 Acked Write 1 Done Sent Done false
23456 7154 Acked Write 9 Done Sent Done false
23457 7155 Acked Write 7 Done Sent Done false
23458 7156 Acked Read 8 Done Sent Done false
23459 7157 Acked Read 6 Done Sent Done false
23460 7158 Acked Read 5 Done Sent Done false
23461 7159 Acked Flush 0 Done Sent Done false
23462 7160 Acked Read 7 Done Sent Done false
23463 7161 Acked Flush 0 Done Sent Done false
23464 7162 Acked Write 1 Done Sent Done false
23465 7163 Acked Read 10 Done Sent Done false
23466 7164 Acked Write 10 Done Sent Done false
23467 7165 Acked Flush 0 Done Sent Done false
23468 7166 Acked Read 7 Done Sent Done false
23469 7167 Acked Flush 0 Done Sent Done false
23470 7168 Acked Read 10 Done Sent Done false
23471 7169 Acked Write 7 Done Sent Done false
23472 7170 Acked Read 4 Done Sent Done false
23473 7171 Acked Read 10 Done Sent Done false
23474 7172 Acked Read 9 Done Sent Done false
23475 7173 Acked Write 3 Done Sent Done false
23476 7174 Acked Flush 0 Done Sent Done false
23477 7175 Acked Read 9 Done Sent Done false
23478 7176 Acked Read 8 Done Sent Done false
23479 7177 Acked Flush 0 Done Sent Done false
23480 7178 Acked Read 6 Done Sent Done false
23481 7179 Acked Write 9 Done Sent Done false
23482 7180 Acked Read 5 Done Sent Done false
23483 7181 Acked Flush 0 Done Sent Done false
23484 7182 Acked Read 10 Done Sent Done false
23485 7183 Acked Write 3 Done Sent Done false
23486 7184 Acked Write 7 Done Sent Done false
23487 7185 Acked Write 9 Done Sent Done false
23488 7186 Acked Read 10 Done Sent Done false
23489 7187 Acked Write 3 Done Sent Done false
23490 7188 Acked Flush 0 Done Sent Done false
23491 7189 Acked Write 7 Done Sent Done false
23492 7190 Acked Flush 0 Done Sent Done false
23493 7191 Acked Read 6 Done Sent Done false
23494 7192 Acked Flush 0 Done Sent Done false
23495 7193 Acked Write 10 Done Sent Done false
23496 7194 Acked Write 2 Done Sent Done false
23497 7195 Acked Write 3 Done Sent Done false
23498 7196 Acked Write 1 Done Sent Done false
23499 7197 Acked Read 4 Done Sent Done false
23500 7198 Acked Write 7 Done Sent Done false
23501 7199 Acked Read 2 Done Sent Done false
23502 7200 Acked Read 1 Done Sent Done false
23503 7201 Acked Write 7 Done Sent Done false
23504 7202 Acked Write 10 Done Sent Done false
23505 7203 Acked Read 7 Done Sent Done false
23506 7204 Acked Flush 0 Done Sent Done false
23507 7205 Acked Read 6 Done Sent Done false
23508 7206 Acked Write 10 Done Sent Done false
23509 7207 Acked Write 7 Done Sent Done false
23510 7208 Acked Read 3 Done Sent Done false
23511 7209 Acked Read 9 Done Sent Done false
23512 7210 Acked Flush 0 Done Sent Done false
23513 7211 Acked Write 8 Done Sent Done false
23514 7212 Acked Write 5 Done Sent Done false
23515 7213 Acked Read 2 Done Sent Done false
23516 7214 Acked Read 6 Done Sent Done false
23517 7215 Acked Write 3 Done Sent Done false
23518 7216 Acked Read 7 Done Sent Done false
23519 7217 Acked Write 8 Done Sent Done false
23520 7218 Acked Read 4 Done Sent Done false
23521 7219 Acked Write 2 Done Sent Done false
23522 7220 Acked Flush 0 Done Sent Done false
23523 7221 Acked Write 2 Done Sent Done false
23524 7222 Acked Flush 0 Done Sent Done false
23525 7223 Acked Read 7 Done Sent Done false
23526 7224 Acked Read 4 Done Sent Done false
23527 7225 Acked Read 4 Done Sent Done false
23528 7226 Acked Read 7 Done Sent Done false
23529 7227 Acked Read 5 Done Sent Done false
23530 7228 Acked Write 3 Done Sent Done false
23531 7229 Acked Read 6 Done Sent Done false
23532 7230 Acked Write 7 Done Sent Done false
23533 7231 Acked Read 7 Done Sent Done false
23534 7232 Acked Read 7 Done Sent Done false
23535 7233 Acked Write 3 Done Sent Done false
23536 7234 Acked Write 7 Done Sent Done false
23537 7235 Acked Write 2 Done Sent Done false
23538 7236 Acked Write 5 Done Sent Done false
23539 7237 Acked Write 2 Done Sent Done false
23540 7238 Acked Write 7 Done Sent Done false
23541 7239 Acked Write 8 Done Sent Done false
23542 7240 Acked Read 2 Done Sent Done false
23543 7241 Acked Read 2 Done Sent Done false
23544 7242 Acked Read 5 Done Sent Done false
23545 7243 Acked Flush 0 Done Sent Done false
23546 7244 Acked Read 4 Done Sent Done false
23547 7245 Acked Write 1 Done Sent Done false
23548 7246 Acked Read 2 Done Sent Done false
23549 7247 Acked Read 4 Done Sent Done false
23550 7248 Acked Write 8 Done Sent Done false
23551 7249 Acked Read 6 Done Sent Done false
23552 7250 Acked Flush 0 Done Sent Done false
23553 7251 Acked Flush 0 Done Sent Done false
23554 7252 Acked Write 9 Done Sent Done false
23555 7253 Acked Read 6 Done Sent Done false
23556 7254 Acked Read 9 Done Sent Done false
23557 7255 Acked Read 2 Done Sent Done false
23558 7256 Acked Write 8 Done Sent Done false
23559 7257 Acked Read 8 Done Sent Done false
23560 7258 Acked Read 3 Done Sent Done false
23561 7259 Acked Read 4 Done Sent Done false
23562 7260 Acked Write 10 Done Sent Done false
23563 7261 Acked Read 10 Done Sent Done false
23564 7262 Acked Write 6 Done Sent Done false
23565 7263 Acked Read 8 Done Sent Done false
23566 7264 Acked Read 6 Done Sent Done false
23567 7265 Acked Read 7 Done Sent Done false
23568 7266 Acked Read 6 Done Sent Done false
23569 7267 Acked Read 10 Done Sent Done false
23570 7268 Acked Read 10 Done Sent Done false
23571 7269 Acked Read 3 Done Sent Done false
23572 7270 Acked Read 4 Done Sent Done false
23573 7271 Acked Write 3 Done Sent Done false
23574 7272 Acked Write 8 Done Sent Done false
23575 7273 Acked Write 4 Done Sent Done false
23576 7274 Acked Write 4 Done Sent Done false
23577 7275 Acked Write 9 Done Sent Done false
23578 7276 Acked Read 3 Done Sent Done false
23579 7277 Acked Read 3 Done Sent Done false
23580 7278 Acked Write 4 Done Sent Done false
23581 7279 Acked Write 10 Done Sent Done false
23582 7280 Acked Read 8 Done Sent Done false
23583 7281 Acked Flush 0 Done Sent Done false
23584 7282 Acked Read 8 Done Sent Done false
23585 7283 Acked Flush 0 Done Sent Done false
23586 7284 Acked Write 8 Done Sent Done false
23587 7285 Acked Write 6 Done Sent Done false
23588 7286 Acked Write 10 Done Sent Done false
23589 7287 Acked Read 4 Done Sent Done false
23590 7288 Acked Read 8 Done Sent Done false
23591 7289 Acked Flush 0 Done Sent Done false
23592 7290 Acked Read 1 Done Sent Done false
23593 7291 Acked Write 5 Done Sent Done false
23594 7292 Acked Write 3 Done Sent Done false
23595 7293 Acked Read 3 Done Sent Done false
23596 7294 Acked Read 2 Done Sent Done false
23597 7295 Acked Flush 0 Done Sent Done false
23598 7296 Acked Read 9 Done Sent Done false
23599 7297 Acked Read 9 Done Sent Done false
23600 7298 Acked Read 2 Done Sent Done false
23601 7299 Acked Write 9 Done Sent Done false
23602 7300 Acked Flush 0 Done Sent Done false
23603 7301 Acked Read 10 Done Sent Done false
23604 7302 Acked Read 4 Done Sent Done false
23605 7303 Acked Write 5 Done Sent Done false
23606 7304 Acked Read 2 Done Sent Done false
23607 7305 Acked Flush 0 Done Sent Done false
23608 7306 Acked Read 4 Done Sent Done false
23609 7307 Acked Read 9 Done Sent Done false
23610 7308 Acked Write 6 Done Sent Done false
23611 7309 Acked Write 6 Done Sent Done false
23612 7310 Acked Write 5 Done Sent Done false
23613 7311 Acked Write 1 Done Sent Done false
23614 7312 Acked Write 6 Done Sent Done false
23615 7313 Acked Write 6 Done Sent Done false
23616 7314 Acked Write 4 Done Sent Done false
23617 7315 Acked Read 1 Done Sent Done false
23618 7316 Acked Read 7 Done Sent Done false
23619 7317 Acked Flush 0 Done Sent Done false
23620 7318 Acked Write 2 Done Sent Done false
23621 7319 Acked Write 9 Done Sent Done false
23622 7320 Acked Read 7 Done Sent Done false
23623 7321 Acked Read 5 Done Sent Done false
23624 7322 Acked Write 7 Done Sent Done false
23625 7323 Acked Flush 0 Done Sent Done false
23626 7324 Acked Flush 0 Done Sent Done false
23627 7325 Acked Write 4 Done Sent Done false
23628 7326 Acked Read 3 Done Sent Done false
23629 7327 Acked Read 10 Done Sent Done false
23630 7328 Acked Read 5 Done Sent Done false
23631 7329 Acked Write 1 Done Sent Done false
23632 7330 Acked Flush 0 Done Sent Done false
23633 7331 Acked Read 6 Done Sent Done false
23634 7332 Acked Read 5 Done Sent Done false
23635 7333 Acked Write 9 Done Sent Done false
23636 7334 Acked Read 2 Done Sent Done false
23637 7335 Acked Read 6 Done Sent Done false
23638 7336 Acked Read 6 Done Sent Done false
23639 7337 Acked Write 2 Done Sent Done false
23640 7338 Acked Write 2 Done Sent Done false
23641 7339 Acked Read 5 Done Sent Done false
23642 STATES DS:0 DS:1 DS:2 TOTAL
23643 Sent 0 301 0 301
23644 Done 301 0 301 602
23645 Skipped 0 0 0 0
23646 Error 0 0 0 0
23647 Last Flush: Some(JobId(7330)) Some(JobId(7038)) Some(JobId(7330))
23648 Downstairs last five completed: 7038 7037 7036 7035 7034
23649 Upstairs last five completed: 7339 7338 7337 7336 7335
23650 CLIENT: Up:0 ds:301 act:2
236512025-01-29T05:15:53.430ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
236522025-01-29T05:15:54.432ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
236532025-01-29T05:15:55.747ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
23654 [8820] got start action
23655 [8820] I am going to restart
23656 Starting downstairs at port 8820
23657 Make output file at "/tmp/dsc/downstairs-8820.txt"
23658 [8820][1] reports Starting
23659 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
23660 [8820][1] reports Running
23661 ----------------------------------------------------------------
23662 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23663 States: Active Offline Active
23664 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23665 7039 Acked Write 5 Done Sent Done false
23666 7040 Acked Write 8 Done Sent Done false
23667 7041 Acked Write 3 Done Sent Done false
23668 7042 Acked Write 3 Done Sent Done false
23669 7043 Acked Write 1 Done Sent Done false
23670 7044 Acked Read 7 Done Sent Done false
23671 7045 Acked Write 4 Done Sent Done false
23672 7046 Acked Read 6 Done Sent Done false
23673 7047 Acked Read 10 Done Sent Done false
23674 7048 Acked Flush 0 Done Sent Done false
23675 7049 Acked Flush 0 Done Sent Done false
23676 7050 Acked Write 10 Done Sent Done false
23677 7051 Acked Write 2 Done Sent Done false
23678 7052 Acked Write 7 Done Sent Done false
23679 7053 Acked Read 6 Done Sent Done false
23680 7054 Acked Write 7 Done Sent Done false
23681 7055 Acked Read 8 Done Sent Done false
23682 7056 Acked Read 8 Done Sent Done false
23683 7057 Acked Read 4 Done Sent Done false
23684 7058 Acked Write 2 Done Sent Done false
23685 7059 Acked Write 1 Done Sent Done false
23686 7060 Acked Read 2 Done Sent Done false
23687 7061 Acked Write 4 Done Sent Done false
23688 7062 Acked Write 3 Done Sent Done false
23689 7063 Acked Write 8 Done Sent Done false
23690 7064 Acked Read 2 Done Sent Done false
23691 7065 Acked Read 6 Done Sent Done false
23692 7066 Acked Flush 0 Done Sent Done false
23693 7067 Acked Read 1 Done Sent Done false
23694 7068 Acked Read 6 Done Sent Done false
23695 7069 Acked Read 2 Done Sent Done false
23696 7070 Acked Write 7 Done Sent Done false
23697 7071 Acked Flush 0 Done Sent Done false
23698 7072 Acked Write 1 Done Sent Done false
23699 7073 Acked Write 9 Done Sent Done false
23700 7074 Acked Write 4 Done Sent Done false
23701 7075 Acked Write 3 Done Sent Done false
23702 7076 Acked Write 1 Done Sent Done false
23703 7077 Acked Read 10 Done Sent Done false
23704 7078 Acked Write 7 Done Sent Done false
23705 7079 Acked Write 4 Done Sent Done false
23706 7080 Acked Read 4 Done Sent Done false
23707 7081 Acked Read 2 Done Sent Done false
23708 7082 Acked Read 4 Done Sent Done false
23709 7083 Acked Read 10 Done Sent Done false
23710 7084 Acked Write 3 Done Sent Done false
23711 7085 Acked Read 10 Done Sent Done false
23712 7086 Acked Write 7 Done Sent Done false
23713 7087 Acked Write 9 Done Sent Done false
23714 7088 Acked Write 5 Done Sent Done false
23715 7089 Acked Read 6 Done Sent Done false
23716 7090 Acked Read 3 Done Sent Done false
23717 7091 Acked Write 5 Done Sent Done false
23718 7092 Acked Write 6 Done Sent Done false
23719 7093 Acked Read 1 Done Sent Done false
23720 7094 Acked Write 9 Done Sent Done false
23721 7095 Acked Write 2 Done Sent Done false
23722 7096 Acked Read 9 Done Sent Done false
23723 7097 Acked Read 10 Done Sent Done false
23724 7098 Acked Write 3 Done Sent Done false
23725 7099 Acked Read 3 Done Sent Done false
23726 7100 Acked Read 7 Done Sent Done false
23727 7101 Acked Read 7 Done Sent Done false
23728 7102 Acked Read 4 Done Sent Done false
23729 7103 Acked Write 4 Done Sent Done false
23730 7104 Acked Read 9 Done Sent Done false
23731 7105 Acked Read 2 Done Sent Done false
23732 7106 Acked Write 6 Done Sent Done false
23733 7107 Acked Read 6 Done Sent Done false
23734 7108 Acked Write 8 Done Sent Done false
23735 7109 Acked Write 10 Done Sent Done false
23736 7110 Acked Read 7 Done Sent Done false
23737 7111 Acked Write 6 Done Sent Done false
23738 7112 Acked Flush 0 Done Sent Done false
23739 7113 Acked Read 7 Done Sent Done false
23740 7114 Acked Write 4 Done Sent Done false
23741 7115 Acked Write 1 Done Sent Done false
23742 7116 Acked Read 2 Done Sent Done false
23743 7117 Acked Read 6 Done Sent Done false
23744 7118 Acked Write 6 Done Sent Done false
23745 7119 Acked Write 5 Done Sent Done false
23746 7120 Acked Write 8 Done Sent Done false
23747 7121 Acked Write 7 Done Sent Done false
23748 7122 Acked Read 5 Done Sent Done false
23749 7123 Acked Flush 0 Done Sent Done false
23750 7124 Acked Write 10 Done Sent Done false
23751 7125 Acked Read 1 Done Sent Done false
23752 7126 Acked Write 9 Done Sent Done false
23753 7127 Acked Write 6 Done Sent Done false
23754 7128 Acked Read 3 Done Sent Done false
23755 7129 Acked Read 5 Done Sent Done false
23756 7130 Acked Read 2 Done Sent Done false
23757 7131 Acked Write 10 Done Sent Done false
23758 7132 Acked Write 10 Done Sent Done false
23759 7133 Acked Read 4 Done Sent Done false
23760 7134 Acked Flush 0 Done Sent Done false
23761 7135 Acked Write 9 Done Sent Done false
23762 7136 Acked Read 5 Done Sent Done false
23763 7137 Acked Read 3 Done Sent Done false
23764 7138 Acked Flush 0 Done Sent Done false
23765 7139 Acked Read 8 Done Sent Done false
23766 7140 Acked Read 6 Done Sent Done false
23767 7141 Acked Read 7 Done Sent Done false
23768 7142 Acked Read 2 Done Sent Done false
23769 7143 Acked Read 10 Done Sent Done false
23770 7144 Acked Write 5 Done Sent Done false
23771 7145 Acked Read 6 Done Sent Done false
23772 7146 Acked Flush 0 Done Sent Done false
23773 7147 Acked Read 4 Done Sent Done false
23774 7148 Acked Flush 0 Done Sent Done false
23775 7149 Acked Read 5 Done Sent Done false
23776 7150 Acked Read 4 Done Sent Done false
23777 7151 Acked Write 4 Done Sent Done false
23778 7152 Acked Write 5 Done Sent Done false
23779 7153 Acked Write 1 Done Sent Done false
23780 7154 Acked Write 9 Done Sent Done false
23781 7155 Acked Write 7 Done Sent Done false
23782 7156 Acked Read 8 Done Sent Done false
23783 7157 Acked Read 6 Done Sent Done false
23784 7158 Acked Read 5 Done Sent Done false
23785 7159 Acked Flush 0 Done Sent Done false
23786 7160 Acked Read 7 Done Sent Done false
23787 7161 Acked Flush 0 Done Sent Done false
23788 7162 Acked Write 1 Done Sent Done false
23789 7163 Acked Read 10 Done Sent Done false
23790 7164 Acked Write 10 Done Sent Done false
23791 7165 Acked Flush 0 Done Sent Done false
23792 7166 Acked Read 7 Done Sent Done false
23793 7167 Acked Flush 0 Done Sent Done false
23794 7168 Acked Read 10 Done Sent Done false
23795 7169 Acked Write 7 Done Sent Done false
23796 7170 Acked Read 4 Done Sent Done false
23797 7171 Acked Read 10 Done Sent Done false
23798 7172 Acked Read 9 Done Sent Done false
23799 7173 Acked Write 3 Done Sent Done false
23800 7174 Acked Flush 0 Done Sent Done false
23801 7175 Acked Read 9 Done Sent Done false
23802 7176 Acked Read 8 Done Sent Done false
23803 7177 Acked Flush 0 Done Sent Done false
23804 7178 Acked Read 6 Done Sent Done false
23805 7179 Acked Write 9 Done Sent Done false
23806 7180 Acked Read 5 Done Sent Done false
23807 7181 Acked Flush 0 Done Sent Done false
23808 7182 Acked Read 10 Done Sent Done false
23809 7183 Acked Write 3 Done Sent Done false
23810 7184 Acked Write 7 Done Sent Done false
23811 7185 Acked Write 9 Done Sent Done false
23812 7186 Acked Read 10 Done Sent Done false
23813 7187 Acked Write 3 Done Sent Done false
23814 7188 Acked Flush 0 Done Sent Done false
23815 7189 Acked Write 7 Done Sent Done false
23816 7190 Acked Flush 0 Done Sent Done false
23817 7191 Acked Read 6 Done Sent Done false
23818 7192 Acked Flush 0 Done Sent Done false
23819 7193 Acked Write 10 Done Sent Done false
23820 7194 Acked Write 2 Done Sent Done false
23821 7195 Acked Write 3 Done Sent Done false
23822 7196 Acked Write 1 Done Sent Done false
23823 7197 Acked Read 4 Done Sent Done false
23824 7198 Acked Write 7 Done Sent Done false
23825 7199 Acked Read 2 Done Sent Done false
23826 7200 Acked Read 1 Done Sent Done false
23827 7201 Acked Write 7 Done Sent Done false
23828 7202 Acked Write 10 Done Sent Done false
23829 7203 Acked Read 7 Done Sent Done false
23830 7204 Acked Flush 0 Done Sent Done false
23831 7205 Acked Read 6 Done Sent Done false
23832 7206 Acked Write 10 Done Sent Done false
23833 7207 Acked Write 7 Done Sent Done false
23834 7208 Acked Read 3 Done Sent Done false
23835 7209 Acked Read 9 Done Sent Done false
23836 7210 Acked Flush 0 Done Sent Done false
23837 7211 Acked Write 8 Done Sent Done false
23838 7212 Acked Write 5 Done Sent Done false
23839 7213 Acked Read 2 Done Sent Done false
23840 7214 Acked Read 6 Done Sent Done false
23841 7215 Acked Write 3 Done Sent Done false
23842 7216 Acked Read 7 Done Sent Done false
23843 7217 Acked Write 8 Done Sent Done false
23844 7218 Acked Read 4 Done Sent Done false
23845 7219 Acked Write 2 Done Sent Done false
23846 7220 Acked Flush 0 Done Sent Done false
23847 7221 Acked Write 2 Done Sent Done false
23848 7222 Acked Flush 0 Done Sent Done false
23849 7223 Acked Read 7 Done Sent Done false
23850 7224 Acked Read 4 Done Sent Done false
23851 7225 Acked Read 4 Done Sent Done false
23852 7226 Acked Read 7 Done Sent Done false
23853 7227 Acked Read 5 Done Sent Done false
23854 7228 Acked Write 3 Done Sent Done false
23855 7229 Acked Read 6 Done Sent Done false
23856 7230 Acked Write 7 Done Sent Done false
23857 7231 Acked Read 7 Done Sent Done false
23858 7232 Acked Read 7 Done Sent Done false
23859 7233 Acked Write 3 Done Sent Done false
23860 7234 Acked Write 7 Done Sent Done false
23861 7235 Acked Write 2 Done Sent Done false
23862 7236 Acked Write 5 Done Sent Done false
23863 7237 Acked Write 2 Done Sent Done false
23864 7238 Acked Write 7 Done Sent Done false
23865 7239 Acked Write 8 Done Sent Done false
23866 7240 Acked Read 2 Done Sent Done false
23867 7241 Acked Read 2 Done Sent Done false
23868 7242 Acked Read 5 Done Sent Done false
23869 7243 Acked Flush 0 Done Sent Done false
23870 7244 Acked Read 4 Done Sent Done false
23871 7245 Acked Write 1 Done Sent Done false
23872 7246 Acked Read 2 Done Sent Done false
23873 7247 Acked Read 4 Done Sent Done false
23874 7248 Acked Write 8 Done Sent Done false
23875 7249 Acked Read 6 Done Sent Done false
23876 7250 Acked Flush 0 Done Sent Done false
23877 7251 Acked Flush 0 Done Sent Done false
23878 7252 Acked Write 9 Done Sent Done false
23879 7253 Acked Read 6 Done Sent Done false
23880 7254 Acked Read 9 Done Sent Done false
23881 7255 Acked Read 2 Done Sent Done false
23882 7256 Acked Write 8 Done Sent Done false
23883 7257 Acked Read 8 Done Sent Done false
23884 7258 Acked Read 3 Done Sent Done false
23885 7259 Acked Read 4 Done Sent Done false
23886 7260 Acked Write 10 Done Sent Done false
23887 7261 Acked Read 10 Done Sent Done false
23888 7262 Acked Write 6 Done Sent Done false
23889 7263 Acked Read 8 Done Sent Done false
23890 7264 Acked Read 6 Done Sent Done false
23891 7265 Acked Read 7 Done Sent Done false
23892 7266 Acked Read 6 Done Sent Done false
23893 7267 Acked Read 10 Done Sent Done false
23894 7268 Acked Read 10 Done Sent Done false
23895 7269 Acked Read 3 Done Sent Done false
23896 7270 Acked Read 4 Done Sent Done false
23897 7271 Acked Write 3 Done Sent Done false
23898 7272 Acked Write 8 Done Sent Done false
23899 7273 Acked Write 4 Done Sent Done false
23900 7274 Acked Write 4 Done Sent Done false
23901 7275 Acked Write 9 Done Sent Done false
23902 7276 Acked Read 3 Done Sent Done false
23903 7277 Acked Read 3 Done Sent Done false
23904 7278 Acked Write 4 Done Sent Done false
23905 7279 Acked Write 10 Done Sent Done false
23906 7280 Acked Read 8 Done Sent Done false
23907 7281 Acked Flush 0 Done Sent Done false
23908 7282 Acked Read 8 Done Sent Done false
23909 7283 Acked Flush 0 Done Sent Done false
23910 7284 Acked Write 8 Done Sent Done false
23911 7285 Acked Write 6 Done Sent Done false
23912 7286 Acked Write 10 Done Sent Done false
23913 7287 Acked Read 4 Done Sent Done false
23914 7288 Acked Read 8 Done Sent Done false
23915 7289 Acked Flush 0 Done Sent Done false
23916 7290 Acked Read 1 Done Sent Done false
23917 7291 Acked Write 5 Done Sent Done false
23918 7292 Acked Write 3 Done Sent Done false
23919 7293 Acked Read 3 Done Sent Done false
23920 7294 Acked Read 2 Done Sent Done false
23921 7295 Acked Flush 0 Done Sent Done false
23922 7296 Acked Read 9 Done Sent Done false
23923 7297 Acked Read 9 Done Sent Done false
23924 7298 Acked Read 2 Done Sent Done false
23925 7299 Acked Write 9 Done Sent Done false
23926 7300 Acked Flush 0 Done Sent Done false
23927 7301 Acked Read 10 Done Sent Done false
23928 7302 Acked Read 4 Done Sent Done false
23929 7303 Acked Write 5 Done Sent Done false
23930 7304 Acked Read 2 Done Sent Done false
23931 7305 Acked Flush 0 Done Sent Done false
23932 7306 Acked Read 4 Done Sent Done false
23933 7307 Acked Read 9 Done Sent Done false
23934 7308 Acked Write 6 Done Sent Done false
23935 7309 Acked Write 6 Done Sent Done false
23936 7310 Acked Write 5 Done Sent Done false
23937 7311 Acked Write 1 Done Sent Done false
23938 7312 Acked Write 6 Done Sent Done false
23939 7313 Acked Write 6 Done Sent Done false
23940 7314 Acked Write 4 Done Sent Done false
23941 7315 Acked Read 1 Done Sent Done false
23942 7316 Acked Read 7 Done Sent Done false
23943 7317 Acked Flush 0 Done Sent Done false
23944 7318 Acked Write 2 Done Sent Done false
23945 7319 Acked Write 9 Done Sent Done false
23946 7320 Acked Read 7 Done Sent Done false
23947 7321 Acked Read 5 Done Sent Done false
23948 7322 Acked Write 7 Done Sent Done false
23949 7323 Acked Flush 0 Done Sent Done false
23950 7324 Acked Flush 0 Done Sent Done false
23951 7325 Acked Write 4 Done Sent Done false
23952 7326 Acked Read 3 Done Sent Done false
23953 7327 Acked Read 10 Done Sent Done false
23954 7328 Acked Read 5 Done Sent Done false
23955 7329 Acked Write 1 Done Sent Done false
23956 7330 Acked Flush 0 Done Sent Done false
23957 7331 Acked Read 6 Done Sent Done false
23958 7332 Acked Read 5 Done Sent Done false
23959 7333 Acked Write 9 Done Sent Done false
23960 7334 Acked Read 2 Done Sent Done false
23961 7335 Acked Read 6 Done Sent Done false
23962 7336 Acked Read 6 Done Sent Done false
23963 7337 Acked Write 2 Done Sent Done false
23964 7338 Acked Write 2 Done Sent Done false
23965 7339 Acked Read 5 Done Sent Done false
23966 STATES DS:0 DS:1 DS:2 TOTAL
23967 Sent 0 301 0 301
23968 Done 301 0 301 602
23969 Skipped 0 0 0 0
23970 Error 0 0 0 0
23971 Last Flush: Some(JobId(7330)) Some(JobId(7038)) Some(JobId(7330))
23972 Downstairs last five completed: 7038 7037 7036 7035 7034
23973 Upstairs last five completed: 7339 7338 7337 7336 7335
23974 CLIENT: Up:0 ds:301 act:2
239752025-01-29T05:15:57.438ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239762025-01-29T05:15:58.440ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239772025-01-29T05:15:58.501ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239782025-01-29T05:15:58.501ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59587) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239792025-01-29T05:15:58.503ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239802025-01-29T05:15:58.503ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239812025-01-29T05:15:58.503ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239822025-01-29T05:15:58.503ZINFOcrucible: send last flush ID to this DS: Some(JobId(7038)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239832025-01-29T05:15:58.503ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239842025-01-29T05:15:58.503ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7038)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239852025-01-29T05:15:58.503ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7038)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
23986 ----------------------------------------------------------------
23987 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23988 Downstairs last five completed: 7340 7339 7338 7337 7336
23989 Upstairs last five completed: 7340 7339 7338 7337 7336
23990 CLIENT: Up:0 ds:0 act:3
23991 Replay: All jobs finished
23992 Main task has work to do, go find it
23993 got dsc Stop(1)
23994 stop 1
23995 Jan 29 05:16:00.744 INFO request completed, latency_us[8820] Got stop action so:false kr:false
23996 : 361, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f089a020-7743-40c1-bb3b-bc4680842cc5, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
239972025-01-29T05:16:00.744ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239982025-01-29T05:16:00.744ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
239992025-01-29T05:16:00.744ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
24000 Jan 29 05:16:00.744 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/1, method: GET, req_id: 55c55546-1c8f-457f-8ef4-3de50b110c71, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
240012025-01-29T05:16:00.745ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
24002 [8820] Exited with: ExitStatus(unix_wait_status(9))
24003 [8820][1] reports Exit
24004 Jan 29 05:16:04.746 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4a95a192-738f-4b17-b91e-e11008f79116, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
24005 001/300 Read block 3931 len 8192
24006 002/300 Write block 4815 len 8192 data: 4 4
24007 003/300 Read block 167 len 16384
24008 004/300 Read block 470 len 40960
24009 005/300 Read block 4315 len 12288
24010 006/300 Read block 3267 len 36864
24011 007/300 Write block 3629 len 24576 data: 4 4 4 4 3 2
24012 008/300 Read block 1955 len 40960
24013 009/300 Read block 2940 len 20480
24014 010/300 Read block 3032 len 32768
24015 011/300 Read block 442 len 8192
24016 012/300 Write block 365 len 28672 data: 3 5 5 5 5 4 3
24017 013/300 Read block 998 len 24576
24018 014/300 Read block 2158 len 4096
24019 015/300 Write block 686 len 28672 data: 3 4 5 6 6 6 6
24020 016/300 Write block 4727 len 40960 data: 2 2 4 5 4 4 5 5 6 6
24021 017/300 Read block 4544 len 12288
24022 018/300 Read block 4570 len 16384
24023 019/300 Read block 2803 len 24576
24024 020/300 Write block 1612 len 24576 data: 5 5 5 5 5 5
24025 021/300 Read block 3935 len 36864
24026 022/300 Write block 3800 len 24576 data: 5 7 7 6 6 8
24027 023/300 Write block 4749 len 4096 data: 6
24028 024/300 Write block 2900 len 36864 data: 3 3 4 4 4 4 5 4 4
24029 025/300 Read block 3261 len 16384
24030 026/300 Read block 4781 len 8192
24031 027/300 Write block 2685 len 40960 data: 3 5 6 5 4 3 3 4 4 4
24032 028/300 Read block 3476 len 8192
24033 029/300 Write block 3524 len 24576 data: 6 6 5 5 5 6
24034 030/300 Read block 393 len 12288
24035 031/300 Read block 633 len 36864
24036 032/300 Read block 1093 len 20480
24037 033/300 Write block 3208 len 4096 data: 4
24038 034/300 Read block 4192 len 32768
24039 035/300 Flush
24040 036/300 Write block 1316 len 16384 data: 6 7 6 4
24041 037/300 Read block 3211 len 20480
24042 038/300 Flush
24043 039/300 Write block 1675 len 40960 data: 5 4 3 3 4 5 6 9 10 10
24044 040/300 Write block 4961 len 20480 data: 6 5 5 6 5
24045 041/300 Flush
24046 042/300 Read block 1492 len 20480
24047 043/300 Read block 3643 len 20480
24048 044/300 Read block 3909 len 4096
24049 045/300 Read block 3615 len 32768
24050 046/300 Write block 4140 len 28672 data: 9 9 6 4 5 5 4
24051 047/300 Read block 2629 len 4096
24052 048/300 Read block 1214 len 16384
24053 049/300 Read block 3743 len 28672
24054 050/300 Read block 1724 len 40960
24055 051/300 Write block 3819 len 36864 data: 5 5 5 5 4 4 4 4 5
24056 052/300 Read block 4634 len 12288
24057 053/300 Write block 604 len 36864 data: 2 2 2 2 3 3 3 3 3
24058 054/300 Write block 1813 len 36864 data: 4 3 2 2 3 3 4 4 4
24059 055/300 Flush
24060 056/300 Read block 3381 len 16384
24061 057/300 Write block 575 len 24576 data: 5 5 8 6 5 5
24062 058/300 Read block 1192 len 20480
24063 059/300 Read block 4354 len 8192
24064 060/300 Write block 835 len 20480 data: 3 4 5 6 5
24065 061/300 Write block 4188 len 28672 data: 4 5 5 5 3 4 4
24066 062/300 Write block 1877 len 32768 data: 3 3 3 4 7 8 7 9
24067 063/300 Flush
24068 064/300 Write block 4423 len 20480 data: 4 5 5 4 5
24069 065/300 Write block 3219 len 20480 data: 8 9 7 7 7
24070 066/300 Write block 2548 len 12288 data: 6 6 5
24071 067/300 Read block 4428 len 16384
24072 068/300 Read block 2073 len 8192
24073 069/300 Read block 3482 len 24576
24074 070/300 Write block 1662 len 40960 data: 3 3 3 3 3 3 3 2 2 2
24075 071/300 Write block 2749 len 16384 data: 6 5 7 8
24076 072/300 Write block 3446 len 16384 data: 5 4 4 3
24077 073/300 Read block 3268 len 40960
24078 074/300 Write block 2949 len 4096 data: 5
24079 075/300 Write block 4487 len 4096 data: 4
24080 076/300 Read block 1864 len 16384
24081 077/300 Read block 990 len 8192
24082 078/300 Read block 1043 len 20480
24083 079/300 Read block 3393 len 20480
24084 080/300 Read block 3087 len 8192
24085 081/300 Read block 3742 len 24576
24086 082/300 Write block 3174 len 4096 data: 5
24087 083/300 Write block 2395 len 20480 data: 2 2 2 2 2
24088 084/300 Read block 2626 len 12288
24089 085/300 Flush
24090 086/300 Write block 343 len 20480 data: 5 6 5 5 6
24091 087/300 Write block 1475 len 32768 data: 4 3 2 2 3 2 2 2
24092 088/300 Read block 3064 len 36864
24093 089/300 Read block 1523 len 24576
24094 090/300 Flush
24095 091/300 Read block 1834 len 28672
24096 092/300 Read block 41 len 36864
24097 093/300 Read block 1769 len 4096
24098 094/300 Read block 3877 len 40960
24099 095/300 Flush
24100 096/300 Flush
24101 097/300 Write block 4642 len 20480 data: 9 9 8 7 6
24102 098/300 Read block 4284 len 40960
24103 099/300 Write block 1851 len 16384 data: 4 4 4 4
24104 100/300 Read block 3024 len 24576
24105 101/300 Flush
24106 102/300 Flush
24107 103/300 Read block 1509 len 16384
24108 104/300 Write block 351 len 16384 data: 5 5 4 4
24109 105/300 Write block 2862 len 36864 data: 4 5 5 6 5 5 6 7 4
24110 106/300 Read block 1534 len 8192
24111 107/300 Read block 1491 len 8192
24112 108/300 Read block 1590 len 4096
24113 109/300 Read block 3690 len 20480
24114 110/300 Flush
24115 111/300 Flush
24116 112/300 Write block 1212 len 12288 data: 4 5 4
24117 113/300 Write block 825 len 8192 data: 4 5
24118 114/300 Read block 1133 len 12288
24119 115/300 Write block 2281 len 8192 data: 5 5
24120 116/300 Read block 3249 len 36864
24121 117/300 Read block 1209 len 12288
24122 118/300 Read block 905 len 32768
24123 119/300 Write block 3745 len 32768 data: 4 4 4 4 6 6 4 6
24124 120/300 Write block 1058 len 24576 data: 4 5 4 4 4 3
24125 121/300 Read block 3752 len 36864
24126 122/300 Read block 2305 len 40960
24127 123/300 Read block 997 len 28672
24128 124/300 Read block 4852 len 16384
24129 125/300 Read block 1607 len 4096
24130 126/300 Read block 4093 len 16384
24131 127/300 Write block 2975 len 8192 data: 5 9
24132 128/300 Write block 3441 len 32768 data: 5 5 4 5 6 6 5 5
24133 129/300 Read block 238 len 36864
24134 130/300 Write block 1616 len 24576 data: 6 6 4 4 6 8
24135 131/300 Read block 1088 len 28672
24136 132/300 Flush
24137 133/300 Read block 1167 len 16384
24138 134/300 Read block 4176 len 28672
24139 135/300 Read block 1251 len 40960
24140 136/300 Write block 3905 len 16384 data: 6 5 5 4
24141 137/300 Flush
24142 138/300 Read block 2824 len 24576
24143 139/300 Read block 269 len 40960
24144 140/300 Write block 4116 len 36864 data: 3 5 5 4 5 4 4 5 6
24145 141/300 Read block 2794 len 12288
24146 142/300 Write block 4990 len 40960 data: 8 9 10 8 8 7 3 3 4 4
24147 143/300 Write block 3254 len 32768 data: 3 2 2 3 4 5 6 5
24148 144/300 Write block 805 len 40960 data: 3 4 5 6 6 4 5 7 7 7
24149 145/300 Write block 2253 len 16384 data: 6 5 4 5
24150 146/300 Write block 997 len 16384 data: 4 4 3 4
24151 147/300 Read block 335 len 8192
24152 148/300 Write block 4129 len 40960 data: 3 3 2 3 3 4 4 6 6 7
24153 149/300 Flush
24154 150/300 Write block 1971 len 16384 data: 7 5 8 8
24155 151/300 Read block 3385 len 32768
24156 152/300 Read block 3019 len 12288
24157 153/300 Write block 4848 len 40960 data: 5 4 4 5 5 5 4 5 5 6
24158 154/300 Read block 3739 len 32768
24159 155/300 Read block 4428 len 36864
24160 156/300 Read block 3387 len 36864
24161 157/300 Write block 1789 len 28672 data: 6 7 6 8 7 7 8
24162 158/300 Flush
24163 159/300 Write block 4745 len 24576 data: 6 5 6 7 7 6
24164 160/300 Read block 1553 len 4096
24165 161/300 Write block 3800 len 20480 data: 6 8 8 7 7
24166 162/300 Read block 1078 len 16384
24167 163/300 Flush
24168 164/300 Read block 4023 len 12288
24169 165/300 Read block 4220 len 8192
24170 166/300 Flush
24171 167/300 Write block 2136 len 40960 data: 5 4 5 6 8 10 9 7 7 6
24172 168/300 Write block 1685 len 20480 data: 9 8 10 10 12
24173 169/300 Write block 1438 len 28672 data: 6 6 6 6 5 5 4
24174 170/300 Write block 191 len 8192 data: 6 7
24175 171/300 Write block 487 len 8192 data: 8 4
24176 172/300 Read block 236 len 28672
24177 173/300 Read block 1036 len 40960
24178 174/300 Read block 2912 len 28672
24179 175/300 Read block 4412 len 36864
24180 176/300 Write block 4415 len 8192 data: 4 3
24181 177/300 Read block 4201 len 24576
24182 178/300 Write block 4319 len 16384 data: 2 4 4 4
24183 179/300 Read block 2502 len 16384
24184 180/300 Read block 702 len 24576
24185 181/300 Write block 4565 len 36864 data: 5 6 6 5 6 7 6 5 3
24186 182/300 Write block 3953 len 28672 data: 3 5 6 4 4 5 5
24187 183/300 Write block 4505 len 28672 data: 5 4 4 3 3 3 3
24188 184/300 Write block 3481 len 28672 data: 8 8 9 7 6 7 5
24189 185/300 Read block 2424 len 28672
24190 186/300 Write block 3212 len 4096 data: 3
24191 187/300 Write block 3356 len 32768 data: 4 3 3 3 4 4 4 4
24192 188/300 Write block 4600 len 12288 data: 8 8 8
24193 189/300 Write block 1338 len 16384 data: 6 7 7 6
24194 190/300 Write block 2935 len 36864 data: 4 5 3 3 4 4 5 5 4
24195 191/300 Read block 758 len 36864
24196 192/300 Write block 1831 len 8192 data: 4 5
24197 193/300 Read block 3798 len 16384
24198 194/300 Write block 4716 len 20480 data: 6 4 4 5 4
24199 195/300 Write block 2185 len 28672 data: 3 3 3 2 2 2 2
24200 196/300 Read block 731 len 20480
24201 197/300 Flush
24202 198/300 Write block 816 len 4096 data: 7
24203 199/300 Read block 2334 len 8192
24204 200/300 Write block 4346 len 32768 data: 2 2 2 3 5 3 3 4
24205 201/300 Write block 2854 len 4096 data: 3
24206 202/300 Read block 2567 len 8192
24207 203/300 Write block 3748 len 20480 data: 5 7 7 5 7
24208 204/300 Flush
24209 205/300 Write block 4307 len 8192 data: 4 6
24210 206/300 Read block 224 len 28672
24211 207/300 Read block 1640 len 12288
24212 208/300 Read block 4563 len 32768
24213 209/300 Read block 1951 len 20480
24214 210/300 Read block 516 len 8192
24215 211/300 Write block 3238 len 8192 data: 4 4
24216 212/300 Read block 1226 len 28672
24217 213/300 Write block 170 len 32768 data: 9 9 7 7 5 6 5 5
24218 214/300 Write block 2210 len 4096 data: 6
24219 215/300 Write block 1649 len 16384 data: 3 3 4 4
24220 216/300 Flush
24221 217/300 Write block 4295 len 16384 data: 6 6 5 6
24222 218/300 Write block 2665 len 4096 data: 6
24223 219/300 Write block 2115 len 40960 data: 5 5 4 2 3 4 4 4 4 4
24224 220/300 Read block 3975 len 16384
24225 221/300 Read block 1755 len 32768
24226 222/300 Read block 2774 len 36864
24227 223/300 Write block 3903 len 16384 data: 5 7 7 6
24228 224/300 Read block 3831 len 36864
24229 225/300 Read block 4265 len 24576
24230 226/300 Read block 721 len 12288
24231 227/300 Flush
24232 228/300 Write block 4246 len 12288 data: 3 3 3
24233 229/300 Write block 4557 len 20480 data: 5 5 4 4 4
24234 230/300 Write block 4526 len 36864 data: 4 5 7 7 7 7 7 8 9
24235 231/300 Read block 223 len 24576
24236 232/300 Read block 1886 len 32768
24237 233/300 Write block 3543 len 12288 data: 4 6 5
24238 234/300 Read block 3598 len 40960
24239 235/300 Read block 661 len 4096
24240 236/300 Flush
24241 237/300 Read block 696 len 28672
24242 238/300 Flush
24243 239/300 Read block 315 len 8192
24244 240/300 Write block 4044 len 36864 data: 7 7 8 7 5 4 6 5 4
24245 241/300 Write block 2162 len 4096 data: 3
24246 242/300 Read block 3246 len 32768
24247 243/300 Write block 4411 len 4096 data: 5
24248 244/300 Write block 3099 len 20480 data: 3 3 4 6 7
24249 245/300 Write block 2460 len 24576 data: 6 4 5 5 3 4
24250 246/300 Write block 630 len 4096 data: 5
24251 247/300 Write block 237 len 12288 data: 8 9 9
24252 248/300 Read block 1940 len 12288
24253 249/300 Read block 807 len 24576
24254 250/300 Flush
24255 251/300 Write block 391 len 32768 data: 4 3 4 4 5 5 5 3
24256 252/300 Write block 967 len 36864 data: 8 9 8 8 7 7 6 5 3
24257 253/300 Flush
24258 254/300 Read block 725 len 32768
24259 255/300 Read block 1687 len 12288
24260 256/300 Read block 3674 len 16384
24261 257/300 Read block 4154 len 20480
24262 258/300 Read block 4744 len 32768
24263 259/300 Read block 2993 len 32768
24264 260/300 Read block 3358 len 8192
24265 261/300 Read block 2585 len 8192
24266 262/300 Write block 2277 len 20480 data: 6 5 4 5 6
24267 263/300 Read block 1810 len 4096
24268 264/300 Write block 245 len 36864 data: 7 6 4 3 3 3 5 3 3
24269 265/300 Read block 2660 len 20480
24270 266/300 Write block 2672 len 4096 data: 5
24271 267/300 Read block 3530 len 20480
24272 268/300 Read block 3959 len 24576
24273 269/300 Write block 721 len 16384 data: 5 5 5 5
24274 270/300 Read block 1003 len 8192
24275 271/300 Read block 1332 len 20480
24276 272/300 Write block 3795 len 36864 data: 3 3 4 5 3 7 9 9 8
24277 273/300 Write block 3973 len 28672 data: 6 6 7 6 7 7 7
24278 274/300 Write block 23 len 32768 data: 5 5 3 3 3 3 4 3
24279 275/300 Read block 1123 len 36864
24280 276/300 Write block 3194 len 20480 data: 4 5 5 4 5
24281 277/300 Write block 4709 len 28672 data: 5 5 4 5 5 5 6
24282 278/300 Write block 3032 len 20480 data: 4 5 5 5 5
24283 279/300 Write block 2423 len 36864 data: 6 5 8 7 5 4 3 3 4
24284 280/300 Read block 48 len 16384
24285 281/300 Read block 34 len 16384
24286 282/300 Read block 4005 len 8192
24287 283/300 Write block 762 len 32768 data: 3 3 5 6 7 6 4 3
24288 284/300 Read block 4070 len 36864
24289 285/300 Write block 2619 len 24576 data: 5 5 5 5 5 5
24290 286/300 Write block 1872 len 12288 data: 5 6 5
24291 287/300 Read block 4820 len 12288
24292 288/300 Read block 3781 len 32768
24293 289/300 Flush
24294 290/300 Read block 849 len 20480
24295 291/300 Read block 2178 len 20480
24296 292/300 Flush
24297 293/300 Read block 2326 len 8192
24298 294/300 Write block 3717 len 40960 data: 5 4 6 6 5 6 7 6 5 6
24299 295/300 Read block 3850 len 40960
24300 296/300 Read block 181 len 4096
24301 297/300 Flush
24302 298/300 Flush
24303 299/300 Read block 4812 len 12288
24304 300/300 Read block 4225 len 8192
24305 301/300 Read block 1589 len 36864
24306 Main task has work to do, go find it
24307 got dsc Start(1)
24308 start 1
24309 Jan 29 05:16:04.984 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/1, method: POST, req_id: b4cef645-96aa-4780-a2e0-20c6d3dd357c, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
24310 Replay: started 1, returned:Ok(())
24311 ----------------------------------------------------------------
24312 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24313 States: Active Offline Active
24314 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24315 7341 Acked Read 2 Done Sent Done false
24316 7342 Acked Write 2 Done Sent Done false
24317 7343 Acked Read 4 Done Sent Done false
24318 7344 Acked Read 10 Done Sent Done false
24319 7345 Acked Read 3 Done Sent Done false
24320 7346 Acked Read 9 Done Sent Done false
24321 7347 Acked Write 6 Done Sent Done false
24322 7348 Acked Read 10 Done Sent Done false
24323 7349 Acked Read 5 Done Sent Done false
24324 7350 Acked Read 8 Done Sent Done false
24325 7351 Acked Read 2 Done Sent Done false
24326 7352 Acked Write 7 Done Sent Done false
24327 7353 Acked Read 6 Done Sent Done false
24328 7354 Acked Read 1 Done Sent Done false
24329 7355 Acked Write 7 Done Sent Done false
24330 7356 Acked Write 10 Done Sent Done false
24331 7357 Acked Read 3 Done Sent Done false
24332 7358 Acked Read 4 Done Sent Done false
24333 7359 Acked Read 6 Done Sent Done false
24334 7360 Acked Write 6 Done Sent Done false
24335 7361 Acked Read 9 Done Sent Done false
24336 7362 Acked Write 6 Done Sent Done false
24337 7363 Acked Write 1 Done Sent Done false
24338 7364 Acked Write 9 Done Sent Done false
24339 7365 Acked Read 4 Done Sent Done false
24340 7366 Acked Read 2 Done Sent Done false
24341 7367 Acked Write 10 Done Sent Done false
24342 7368 Acked Read 2 Done Sent Done false
24343 7369 Acked Write 6 Done Sent Done false
24344 7370 Acked Read 3 Done Sent Done false
24345 7371 Acked Read 9 Done Sent Done false
24346 7372 Acked Read 5 Done Sent Done false
24347 7373 Acked Write 1 Done Sent Done false
24348 7374 Acked Read 8 Done Sent Done false
24349 7375 Acked Flush 0 Done Sent Done false
24350 7376 Acked Write 4 Done Sent Done false
24351 7377 Acked Read 5 Done Sent Done false
24352 7378 Acked Flush 0 Done Sent Done false
24353 7379 Acked Write 10 Done Sent Done false
24354 7380 Acked Write 5 Done Sent Done false
24355 7381 Acked Flush 0 Done Sent Done false
24356 7382 Acked Read 5 Done Sent Done false
24357 7383 Acked Read 5 Done Sent Done false
24358 7384 Acked Read 1 Done Sent Done false
24359 7385 Acked Read 8 Done Sent Done false
24360 7386 Acked Write 7 Done Sent Done false
24361 7387 Acked Read 1 Done Sent Done false
24362 7388 Acked Read 4 Done Sent Done false
24363 7389 Acked Read 7 Done Sent Done false
24364 7390 Acked Read 10 Done Sent Done false
24365 7391 Acked Write 9 Done Sent Done false
24366 7392 Acked Read 3 Done Sent Done false
24367 7393 Acked Write 9 Done Sent Done false
24368 7394 Acked Write 9 Done Sent Done false
24369 7395 Acked Flush 0 Done Sent Done false
24370 7396 Acked Read 4 Done Sent Done false
24371 7397 Acked Write 6 Done Sent Done false
24372 7398 Acked Read 5 Done Sent Done false
24373 7399 Acked Read 2 Done Sent Done false
24374 7400 Acked Write 5 Done Sent Done false
24375 7401 Acked Write 7 Done Sent Done false
24376 7402 Acked Write 8 Done Sent Done false
24377 7403 Acked Flush 0 Done Sent Done false
24378 7404 Acked Write 5 Done Sent Done false
24379 7405 Acked Write 5 Done Sent Done false
24380 7406 Acked Write 3 Done Sent Done false
24381 7407 Acked Read 4 Done Sent Done false
24382 7408 Acked Read 2 Done Sent Done false
24383 7409 Acked Read 6 Done Sent Done false
24384 7410 Acked Write 10 Done Sent Done false
24385 7411 Acked Write 4 Done Sent Done false
24386 7412 Acked Write 4 Done Sent Done false
24387 7413 Acked Read 10 Done Sent Done false
24388 7414 Acked Write 1 Done Sent Done false
24389 7415 Acked Write 1 Done Sent Done false
24390 7416 Acked Read 4 Done Sent Done false
24391 7417 Acked Read 2 Done Sent Done false
24392 7418 Acked Read 5 Done Sent Done false
24393 7419 Acked Read 5 Done Sent Done false
24394 7420 Acked Read 2 Done Sent Done false
24395 7421 Acked Read 6 Done Sent Done false
24396 7422 Acked Write 1 Done Sent Done false
24397 7423 Acked Write 5 Done Sent Done false
24398 7424 Acked Read 3 Done Sent Done false
24399 7425 Acked Flush 0 Done Sent Done false
24400 7426 Acked Write 5 Done Sent Done false
24401 7427 Acked Write 8 Done Sent Done false
24402 7428 Acked Read 9 Done Sent Done false
24403 7429 Acked Read 6 Done Sent Done false
24404 7430 Acked Flush 0 Done Sent Done false
24405 7431 Acked Read 7 Done Sent Done false
24406 7432 Acked Read 9 Done Sent Done false
24407 7433 Acked Read 1 Done Sent Done false
24408 7434 Acked Read 10 Done Sent Done false
24409 7435 Acked Flush 0 Done Sent Done false
24410 7436 Acked Flush 0 Done Sent Done false
24411 7437 Acked Write 5 Done Sent Done false
24412 7438 Acked Read 10 Done Sent Done false
24413 7439 Acked Write 4 Done Sent Done false
24414 7440 Acked Read 6 Done Sent Done false
24415 7441 Acked Flush 0 Done Sent Done false
24416 7442 Acked Flush 0 Done Sent Done false
24417 7443 Acked Read 4 Done Sent Done false
24418 7444 Acked Write 4 Done Sent Done false
24419 7445 Acked Write 9 Done Sent Done false
24420 7446 Acked Read 2 Done Sent Done false
24421 7447 Acked Read 2 Done Sent Done false
24422 7448 Acked Read 1 Done Sent Done false
24423 7449 Acked Read 5 Done Sent Done false
24424 7450 Acked Flush 0 Done Sent Done false
24425 7451 Acked Flush 0 Done Sent Done false
24426 7452 Acked Write 3 Done Sent Done false
24427 7453 Acked Write 2 Done Sent Done false
24428 7454 Acked Read 3 Done Sent Done false
24429 7455 Acked Write 2 Done Sent Done false
24430 7456 Acked Read 9 Done Sent Done false
24431 7457 Acked Read 3 Done Sent Done false
24432 7458 Acked Read 8 Done Sent Done false
24433 7459 Acked Write 8 Done Sent Done false
24434 7460 Acked Write 6 Done Sent Done false
24435 7461 Acked Read 9 Done Sent Done false
24436 7462 Acked Read 10 Done Sent Done false
24437 7463 Acked Read 7 Done Sent Done false
24438 7464 Acked Read 4 Done Sent Done false
24439 7465 Acked Read 1 Done Sent Done false
24440 7466 Acked Read 4 Done Sent Done false
24441 7467 Acked Write 2 Done Sent Done false
24442 7468 Acked Write 8 Done Sent Done false
24443 7469 Acked Read 9 Done Sent Done false
24444 7470 Acked Write 6 Done Sent Done false
24445 7471 Acked Read 7 Done Sent Done false
24446 7472 Acked Flush 0 Done Sent Done false
24447 7473 Acked Read 4 Done Sent Done false
24448 7474 Acked Read 7 Done Sent Done false
24449 7475 Acked Read 10 Done Sent Done false
24450 7476 Acked Write 4 Done Sent Done false
24451 7477 Acked Flush 0 Done Sent Done false
24452 7478 Acked Read 6 Done Sent Done false
24453 7479 Acked Read 10 Done Sent Done false
24454 7480 Acked Write 9 Done Sent Done false
24455 7481 Acked Read 3 Done Sent Done false
24456 7482 Acked Write 10 Done Sent Done false
24457 7483 Acked Write 8 Done Sent Done false
24458 7484 Acked Write 10 Done Sent Done false
24459 7485 Acked Write 4 Done Sent Done false
24460 7486 Acked Write 4 Done Sent Done false
24461 7487 Acked Read 2 Done Sent Done false
24462 7488 Acked Write 10 Done Sent Done false
24463 7489 Acked Flush 0 Done Sent Done false
24464 7490 Acked Write 4 Done Sent Done false
24465 7491 Acked Read 8 Done Sent Done false
24466 7492 Acked Read 3 Done Sent Done false
24467 7493 Acked Write 10 Done Sent Done false
24468 7494 Acked Read 8 Done Sent Done false
24469 7495 Acked Read 9 Done Sent Done false
24470 7496 Acked Read 9 Done Sent Done false
24471 7497 Acked Write 7 Done Sent Done false
24472 7498 Acked Flush 0 Done Sent Done false
24473 7499 Acked Write 6 Done Sent Done false
24474 7500 Acked Read 1 Done Sent Done false
24475 7501 Acked Write 5 Done Sent Done false
24476 7502 Acked Read 4 Done Sent Done false
24477 7503 Acked Flush 0 Done Sent Done false
24478 7504 Acked Read 3 Done Sent Done false
24479 7505 Acked Read 2 Done Sent Done false
24480 7506 Acked Flush 0 Done Sent Done false
24481 7507 Acked Write 10 Done Sent Done false
24482 7508 Acked Write 5 Done Sent Done false
24483 7509 Acked Write 7 Done Sent Done false
24484 7510 Acked Write 2 Done Sent Done false
24485 7511 Acked Write 2 Done Sent Done false
24486 7512 Acked Read 7 Done Sent Done false
24487 7513 Acked Read 10 Done Sent Done false
24488 7514 Acked Read 7 Done Sent Done false
24489 7515 Acked Read 9 Done Sent Done false
24490 7516 Acked Write 2 Done Sent Done false
24491 7517 Acked Read 6 Done Sent Done false
24492 7518 Acked Write 4 Done Sent Done false
24493 7519 Acked Read 4 Done Sent Done false
24494 7520 Acked Read 6 Done Sent Done false
24495 7521 Acked Write 9 Done Sent Done false
24496 7522 Acked Write 7 Done Sent Done false
24497 7523 Acked Write 7 Done Sent Done false
24498 7524 Acked Write 7 Done Sent Done false
24499 7525 Acked Read 7 Done Sent Done false
24500 7526 Acked Write 1 Done Sent Done false
24501 7527 Acked Write 8 Done Sent Done false
24502 7528 Acked Write 3 Done Sent Done false
24503 7529 Acked Write 4 Done Sent Done false
24504 7530 Acked Write 9 Done Sent Done false
24505 7531 Acked Read 9 Done Sent Done false
24506 7532 Acked Write 2 Done Sent Done false
24507 7533 Acked Read 4 Done Sent Done false
24508 7534 Acked Write 5 Done Sent Done false
24509 7535 Acked Write 7 Done Sent Done false
24510 7536 Acked Read 5 Done Sent Done false
24511 7537 Acked Flush 0 Done Sent Done false
24512 7538 Acked Write 1 Done Sent Done false
24513 7539 Acked Read 2 Done Sent Done false
24514 7540 Acked Write 8 Done Sent Done false
24515 7541 Acked Write 1 Done Sent Done false
24516 7542 Acked Read 2 Done Sent Done false
24517 7543 Acked Write 5 Done Sent Done false
24518 7544 Acked Flush 0 Done Sent Done false
24519 7545 Acked Write 2 Done Sent Done false
24520 7546 Acked Read 7 Done Sent Done false
24521 7547 Acked Read 3 Done Sent Done false
24522 7548 Acked Read 8 Done Sent Done false
24523 7549 Acked Read 5 Done Sent Done false
24524 7550 Acked Read 2 Done Sent Done false
24525 7551 Acked Write 2 Done Sent Done false
24526 7552 Acked Read 7 Done Sent Done false
24527 7553 Acked Write 8 Done Sent Done false
24528 7554 Acked Write 1 Done Sent Done false
24529 7555 Acked Write 4 Done Sent Done false
24530 7556 Acked Flush 0 Done Sent Done false
24531 7557 Acked Write 4 Done Sent Done false
24532 7558 Acked Write 1 Done Sent Done false
24533 7559 Acked Write 10 Done Sent Done false
24534 7560 Acked Read 4 Done Sent Done false
24535 7561 Acked Read 8 Done Sent Done false
24536 7562 Acked Read 9 Done Sent Done false
24537 7563 Acked Write 4 Done Sent Done false
24538 7564 Acked Read 9 Done Sent Done false
24539 7565 Acked Read 6 Done Sent Done false
24540 7566 Acked Read 3 Done Sent Done false
24541 7567 Acked Flush 0 Done Sent Done false
24542 7568 Acked Write 3 Done Sent Done false
24543 7569 Acked Write 5 Done Sent Done false
24544 7570 Acked Write 9 Done Sent Done false
24545 7571 Acked Read 6 Done Sent Done false
24546 7572 Acked Read 8 Done Sent Done false
24547 7573 Acked Write 3 Done Sent Done false
24548 7574 Acked Read 10 Done Sent Done false
24549 7575 Acked Read 1 Done Sent Done false
24550 7576 Acked Flush 0 Done Sent Done false
24551 7577 Acked Read 7 Done Sent Done false
24552 7578 Acked Flush 0 Done Sent Done false
24553 7579 Acked Read 2 Done Sent Done false
24554 7580 Acked Write 9 Done Sent Done false
24555 7581 Acked Write 1 Done Sent Done false
24556 7582 Acked Read 8 Done Sent Done false
24557 7583 Acked Write 1 Done Sent Done false
24558 7584 Acked Write 5 Done Sent Done false
24559 7585 Acked Write 6 Done Sent Done false
24560 7586 Acked Write 1 Done Sent Done false
24561 7587 Acked Write 3 Done Sent Done false
24562 7588 Acked Read 3 Done Sent Done false
24563 7589 Acked Read 6 Done Sent Done false
24564 7590 Acked Flush 0 Done Sent Done false
24565 7591 Acked Write 8 Done Sent Done false
24566 7592 Acked Write 9 Done Sent Done false
24567 7593 Acked Flush 0 Done Sent Done false
24568 7594 Acked Read 8 Done Sent Done false
24569 7595 Acked Read 3 Done Sent Done false
24570 7596 Acked Read 4 Done Sent Done false
24571 7597 Acked Read 5 Done Sent Done false
24572 7598 Acked Read 8 Done Sent Done false
24573 7599 Acked Read 8 Done Sent Done false
24574 7600 Acked Read 2 Done Sent Done false
24575 7601 Acked Read 2 Done Sent Done false
24576 7602 Acked Write 5 Done Sent Done false
24577 7603 Acked Read 1 Done Sent Done false
24578 7604 Acked Write 9 Done Sent Done false
24579 7605 Acked Read 5 Done Sent Done false
24580 7606 Acked Write 1 Done Sent Done false
24581 7607 Acked Read 5 Done Sent Done false
24582 7608 Acked Read 6 Done Sent Done false
24583 7609 Acked Write 4 Done Sent Done false
24584 7610 Acked Read 2 Done Sent Done false
24585 7611 Acked Read 5 Done Sent Done false
24586 7612 Acked Write 9 Done Sent Done false
24587 7613 Acked Write 7 Done Sent Done false
24588 7614 Acked Write 8 Done Sent Done false
24589 7615 Acked Read 9 Done Sent Done false
24590 7616 Acked Write 5 Done Sent Done false
24591 7617 Acked Write 7 Done Sent Done false
24592 7618 Acked Write 5 Done Sent Done false
24593 7619 Acked Write 9 Done Sent Done false
24594 7620 Acked Read 4 Done Sent Done false
24595 7621 Acked Read 4 Done Sent Done false
24596 7622 Acked Read 2 Done Sent Done false
24597 7623 Acked Write 8 Done Sent Done false
24598 7624 Acked Read 9 Done Sent Done false
24599 7625 Acked Write 6 Done Sent Done false
24600 7626 Acked Write 3 Done Sent Done false
24601 7627 Acked Read 3 Done Sent Done false
24602 7628 Acked Read 8 Done Sent Done false
24603 7629 Acked Flush 0 Done Sent Done false
24604 7630 Acked Read 5 Done Sent Done false
24605 7631 Acked Read 5 Done Sent Done false
24606 7632 Acked Flush 0 Done Sent Done false
24607 7633 Acked Read 2 Done Sent Done false
24608 7634 Acked Write 10 Done Sent Done false
24609 7635 Acked Read 10 Done Sent Done false
24610 7636 Acked Read 1 Done Sent Done false
24611 7637 Acked Flush 0 Done Sent Done false
24612 7638 Acked Flush 0 Done Sent Done false
24613 7639 Acked Read 3 Done Sent Done false
24614 7640 Acked Read 2 Done Sent Done false
24615 7641 Acked Read 9 Done Sent Done false
24616 STATES DS:0 DS:1 DS:2 TOTAL
24617 Sent 0 301 0 301
24618 Done 301 0 301 602
24619 Skipped 0 0 0 0
24620 Error 0 0 0 0
24621 Last Flush: Some(JobId(7638)) Some(JobId(7340)) Some(JobId(7638))
24622 Downstairs last five completed: 7340 7339 7338 7337 7336
24623 Upstairs last five completed: 7641 7640 7639 7638 7637
24624 CLIENT: Up:0 ds:301 act:2
246252025-01-29T05:16:05.751ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
246262025-01-29T05:16:06.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
246272025-01-29T05:16:07.457ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
246282025-01-29T05:16:08.459ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
24629 ----------------------------------------------------------------
24630 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24631 States: Active Offline Active
24632 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24633 7341 Acked Read 2 Done Sent Done false
24634 7342 Acked Write 2 Done Sent Done false
24635 7343 Acked Read 4 Done Sent Done false
24636 7344 Acked Read 10 Done Sent Done false
24637 7345 Acked Read 3 Done Sent Done false
24638 7346 Acked Read 9 Done Sent Done false
24639 7347 Acked Write 6 Done Sent Done false
24640 7348 Acked Read 10 Done Sent Done false
24641 7349 Acked Read 5 Done Sent Done false
24642 7350 Acked Read 8 Done Sent Done false
24643 7351 Acked Read 2 Done Sent Done false
24644 7352 Acked Write 7 Done Sent Done false
24645 7353 Acked Read 6 Done Sent Done false
24646 7354 Acked Read 1 Done Sent Done false
24647 7355 Acked Write 7 Done Sent Done false
24648 7356 Acked Write 10 Done Sent Done false
24649 7357 Acked Read 3 Done Sent Done false
24650 7358 Acked Read 4 Done Sent Done false
24651 7359 Acked Read 6 Done Sent Done false
24652 7360 Acked Write 6 Done Sent Done false
24653 7361 Acked Read 9 Done Sent Done false
24654 7362 Acked Write 6 Done Sent Done false
24655 7363 Acked Write 1 Done Sent Done false
24656 7364 Acked Write 9 Done Sent Done false
24657 7365 Acked Read 4 Done Sent Done false
24658 7366 Acked Read 2 Done Sent Done false
24659 7367 Acked Write 10 Done Sent Done false
24660 7368 Acked Read 2 Done Sent Done false
24661 7369 Acked Write 6 Done Sent Done false
24662 7370 Acked Read 3 Done Sent Done false
24663 7371 Acked Read 9 Done Sent Done false
24664 7372 Acked Read 5 Done Sent Done false
24665 7373 Acked Write 1 Done Sent Done false
24666 7374 Acked Read 8 Done Sent Done false
24667 7375 Acked Flush 0 Done Sent Done false
24668 7376 Acked Write 4 Done Sent Done false
24669 7377 Acked Read 5 Done Sent Done false
24670 7378 Acked Flush 0 Done Sent Done false
24671 7379 Acked Write 10 Done Sent Done false
24672 7380 Acked Write 5 Done Sent Done false
24673 7381 Acked Flush 0 Done Sent Done false
24674 7382 Acked Read 5 Done Sent Done false
24675 7383 Acked Read 5 Done Sent Done false
24676 7384 Acked Read 1 Done Sent Done false
24677 7385 Acked Read 8 Done Sent Done false
24678 7386 Acked Write 7 Done Sent Done false
24679 7387 Acked Read 1 Done Sent Done false
24680 7388 Acked Read 4 Done Sent Done false
24681 7389 Acked Read 7 Done Sent Done false
24682 7390 Acked Read 10 Done Sent Done false
24683 7391 Acked Write 9 Done Sent Done false
24684 7392 Acked Read 3 Done Sent Done false
24685 7393 Acked Write 9 Done Sent Done false
24686 7394 Acked Write 9 Done Sent Done false
24687 7395 Acked Flush 0 Done Sent Done false
24688 7396 Acked Read 4 Done Sent Done false
24689 7397 Acked Write 6 Done Sent Done false
24690 7398 Acked Read 5 Done Sent Done false
24691 7399 Acked Read 2 Done Sent Done false
24692 7400 Acked Write 5 Done Sent Done false
24693 7401 Acked Write 7 Done Sent Done false
24694 7402 Acked Write 8 Done Sent Done false
24695 7403 Acked Flush 0 Done Sent Done false
24696 7404 Acked Write 5 Done Sent Done false
24697 7405 Acked Write 5 Done Sent Done false
24698 7406 Acked Write 3 Done Sent Done false
24699 7407 Acked Read 4 Done Sent Done false
24700 7408 Acked Read 2 Done Sent Done false
24701 7409 Acked Read 6 Done Sent Done false
24702 7410 Acked Write 10 Done Sent Done false
24703 7411 Acked Write 4 Done Sent Done false
24704 7412 Acked Write 4 Done Sent Done false
24705 7413 Acked Read 10 Done Sent Done false
24706 7414 Acked Write 1 Done Sent Done false
24707 7415 Acked Write 1 Done Sent Done false
24708 7416 Acked Read 4 Done Sent Done false
24709 7417 Acked Read 2 Done Sent Done false
24710 7418 Acked Read 5 Done Sent Done false
24711 7419 Acked Read 5 Done Sent Done false
24712 7420 Acked Read 2 Done Sent Done false
24713 7421 Acked Read 6 Done Sent Done false
24714 7422 Acked Write 1 Done Sent Done false
24715 7423 Acked Write 5 Done Sent Done false
24716 7424 Acked Read 3 Done Sent Done false
24717 7425 Acked Flush 0 Done Sent Done false
24718 7426 Acked Write 5 Done Sent Done false
24719 7427 Acked Write 8 Done Sent Done false
24720 7428 Acked Read 9 Done Sent Done false
24721 7429 Acked Read 6 Done Sent Done false
24722 7430 Acked Flush 0 Done Sent Done false
24723 7431 Acked Read 7 Done Sent Done false
24724 7432 Acked Read 9 Done Sent Done false
24725 7433 Acked Read 1 Done Sent Done false
24726 7434 Acked Read 10 Done Sent Done false
24727 7435 Acked Flush 0 Done Sent Done false
24728 7436 Acked Flush 0 Done Sent Done false
24729 7437 Acked Write 5 Done Sent Done false
24730 7438 Acked Read 10 Done Sent Done false
24731 7439 Acked Write 4 Done Sent Done false
24732 7440 Acked Read 6 Done Sent Done false
24733 7441 Acked Flush 0 Done Sent Done false
24734 7442 Acked Flush 0 Done Sent Done false
24735 7443 Acked Read 4 Done Sent Done false
24736 7444 Acked Write 4 Done Sent Done false
24737 7445 Acked Write 9 Done Sent Done false
24738 7446 Acked Read 2 Done Sent Done false
24739 7447 Acked Read 2 Done Sent Done false
24740 7448 Acked Read 1 Done Sent Done false
24741 7449 Acked Read 5 Done Sent Done false
24742 7450 Acked Flush 0 Done Sent Done false
24743 7451 Acked Flush 0 Done Sent Done false
24744 7452 Acked Write 3 Done Sent Done false
24745 7453 Acked Write 2 Done Sent Done false
24746 7454 Acked Read 3 Done Sent Done false
24747 7455 Acked Write 2 Done Sent Done false
24748 7456 Acked Read 9 Done Sent Done false
24749 7457 Acked Read 3 Done Sent Done false
24750 7458 Acked Read 8 Done Sent Done false
24751 7459 Acked Write 8 Done Sent Done false
24752 7460 Acked Write 6 Done Sent Done false
24753 7461 Acked Read 9 Done Sent Done false
24754 7462 Acked Read 10 Done Sent Done false
24755 7463 Acked Read 7 Done Sent Done false
24756 7464 Acked Read 4 Done Sent Done false
24757 7465 Acked Read 1 Done Sent Done false
24758 7466 Acked Read 4 Done Sent Done false
24759 7467 Acked Write 2 Done Sent Done false
24760 7468 Acked Write 8 Done Sent Done false
24761 7469 Acked Read 9 Done Sent Done false
24762 7470 Acked Write 6 Done Sent Done false
24763 7471 Acked Read 7 Done Sent Done false
24764 7472 Acked Flush 0 Done Sent Done false
24765 7473 Acked Read 4 Done Sent Done false
24766 7474 Acked Read 7 Done Sent Done false
24767 7475 Acked Read 10 Done Sent Done false
24768 7476 Acked Write 4 Done Sent Done false
24769 7477 Acked Flush 0 Done Sent Done false
24770 7478 Acked Read 6 Done Sent Done false
24771 7479 Acked Read 10 Done Sent Done false
24772 7480 Acked Write 9 Done Sent Done false
24773 7481 Acked Read 3 Done Sent Done false
24774 7482 Acked Write 10 Done Sent Done false
24775 7483 Acked Write 8 Done Sent Done false
24776 7484 Acked Write 10 Done Sent Done false
24777 7485 Acked Write 4 Done Sent Done false
24778 7486 Acked Write 4 Done Sent Done false
24779 7487 Acked Read 2 Done Sent Done false
24780 7488 Acked Write 10 Done Sent Done false
24781 7489 Acked Flush 0 Done Sent Done false
24782 7490 Acked Write 4 Done Sent Done false
24783 7491 Acked Read 8 Done Sent Done false
24784 7492 Acked Read 3 Done Sent Done false
24785 7493 Acked Write 10 Done Sent Done false
24786 7494 Acked Read 8 Done Sent Done false
24787 7495 Acked Read 9 Done Sent Done false
24788 7496 Acked Read 9 Done Sent Done false
24789 7497 Acked Write 7 Done Sent Done false
24790 7498 Acked Flush 0 Done Sent Done false
24791 7499 Acked Write 6 Done Sent Done false
24792 7500 Acked Read 1 Done Sent Done false
24793 7501 Acked Write 5 Done Sent Done false
24794 7502 Acked Read 4 Done Sent Done false
24795 7503 Acked Flush 0 Done Sent Done false
24796 7504 Acked Read 3 Done Sent Done false
24797 7505 Acked Read 2 Done Sent Done false
24798 7506 Acked Flush 0 Done Sent Done false
24799 7507 Acked Write 10 Done Sent Done false
24800 7508 Acked Write 5 Done Sent Done false
24801 7509 Acked Write 7 Done Sent Done false
24802 7510 Acked Write 2 Done Sent Done false
24803 7511 Acked Write 2 Done Sent Done false
24804 7512 Acked Read 7 Done Sent Done false
24805 7513 Acked Read 10 Done Sent Done false
24806 7514 Acked Read 7 Done Sent Done false
24807 7515 Acked Read 9 Done Sent Done false
24808 7516 Acked Write 2 Done Sent Done false
24809 7517 Acked Read 6 Done Sent Done false
24810 7518 Acked Write 4 Done Sent Done false
24811 7519 Acked Read 4 Done Sent Done false
24812 7520 Acked Read 6 Done Sent Done false
24813 7521 Acked Write 9 Done Sent Done false
24814 7522 Acked Write 7 Done Sent Done false
24815 7523 Acked Write 7 Done Sent Done false
24816 7524 Acked Write 7 Done Sent Done false
24817 7525 Acked Read 7 Done Sent Done false
24818 7526 Acked Write 1 Done Sent Done false
24819 7527 Acked Write 8 Done Sent Done false
24820 7528 Acked Write 3 Done Sent Done false
24821 7529 Acked Write 4 Done Sent Done false
24822 7530 Acked Write 9 Done Sent Done false
24823 7531 Acked Read 9 Done Sent Done false
24824 7532 Acked Write 2 Done Sent Done false
24825 7533 Acked Read 4 Done Sent Done false
24826 7534 Acked Write 5 Done Sent Done false
24827 7535 Acked Write 7 Done Sent Done false
24828 7536 Acked Read 5 Done Sent Done false
24829 7537 Acked Flush 0 Done Sent Done false
24830 7538 Acked Write 1 Done Sent Done false
24831 7539 Acked Read 2 Done Sent Done false
24832 7540 Acked Write 8 Done Sent Done false
24833 7541 Acked Write 1 Done Sent Done false
24834 7542 Acked Read 2 Done Sent Done false
24835 7543 Acked Write 5 Done Sent Done false
24836 7544 Acked Flush 0 Done Sent Done false
24837 7545 Acked Write 2 Done Sent Done false
24838 7546 Acked Read 7 Done Sent Done false
24839 7547 Acked Read 3 Done Sent Done false
24840 7548 Acked Read 8 Done Sent Done false
24841 7549 Acked Read 5 Done Sent Done false
24842 7550 Acked Read 2 Done Sent Done false
24843 7551 Acked Write 2 Done Sent Done false
24844 7552 Acked Read 7 Done Sent Done false
24845 7553 Acked Write 8 Done Sent Done false
24846 7554 Acked Write 1 Done Sent Done false
24847 7555 Acked Write 4 Done Sent Done false
24848 7556 Acked Flush 0 Done Sent Done false
24849 7557 Acked Write 4 Done Sent Done false
24850 7558 Acked Write 1 Done Sent Done false
24851 7559 Acked Write 10 Done Sent Done false
24852 7560 Acked Read 4 Done Sent Done false
24853 7561 Acked Read 8 Done Sent Done false
24854 7562 Acked Read 9 Done Sent Done false
24855 7563 Acked Write 4 Done Sent Done false
24856 7564 Acked Read 9 Done Sent Done false
24857 7565 Acked Read 6 Done Sent Done false
24858 7566 Acked Read 3 Done Sent Done false
24859 7567 Acked Flush 0 Done Sent Done false
24860 7568 Acked Write 3 Done Sent Done false
24861 7569 Acked Write 5 Done Sent Done false
24862 7570 Acked Write 9 Done Sent Done false
24863 7571 Acked Read 6 Done Sent Done false
24864 7572 Acked Read 8 Done Sent Done false
24865 7573 Acked Write 3 Done Sent Done false
24866 7574 Acked Read 10 Done Sent Done false
24867 7575 Acked Read 1 Done Sent Done false
24868 7576 Acked Flush 0 Done Sent Done false
24869 7577 Acked Read 7 Done Sent Done false
24870 7578 Acked Flush 0 Done Sent Done false
24871 7579 Acked Read 2 Done Sent Done false
24872 7580 Acked Write 9 Done Sent Done false
24873 7581 Acked Write 1 Done Sent Done false
24874 7582 Acked Read 8 Done Sent Done false
24875 7583 Acked Write 1 Done Sent Done false
24876 7584 Acked Write 5 Done Sent Done false
24877 7585 Acked Write 6 Done Sent Done false
24878 7586 Acked Write 1 Done Sent Done false
24879 7587 Acked Write 3 Done Sent Done false
24880 7588 Acked Read 3 Done Sent Done false
24881 7589 Acked Read 6 Done Sent Done false
24882 7590 Acked Flush 0 Done Sent Done false
24883 7591 Acked Write 8 Done Sent Done false
24884 7592 Acked Write 9 Done Sent Done false
24885 7593 Acked Flush 0 Done Sent Done false
24886 7594 Acked Read 8 Done Sent Done false
24887 7595 Acked Read 3 Done Sent Done false
24888 7596 Acked Read 4 Done Sent Done false
24889 7597 Acked Read 5 Done Sent Done false
24890 7598 Acked Read 8 Done Sent Done false
24891 7599 Acked Read 8 Done Sent Done false
24892 7600 Acked Read 2 Done Sent Done false
24893 7601 Acked Read 2 Done Sent Done false
24894 7602 Acked Write 5 Done Sent Done false
24895 7603 Acked Read 1 Done Sent Done false
24896 7604 Acked Write 9 Done Sent Done false
24897 7605 Acked Read 5 Done Sent Done false
24898 7606 Acked Write 1 Done Sent Done false
24899 7607 Acked Read 5 Done Sent Done false
24900 7608 Acked Read 6 Done Sent Done false
24901 7609 Acked Write 4 Done Sent Done false
24902 7610 Acked Read 2 Done Sent Done false
24903 7611 Acked Read 5 Done Sent Done false
24904 7612 Acked Write 9 Done Sent Done false
24905 7613 Acked Write 7 Done Sent Done false
24906 7614 Acked Write 8 Done Sent Done false
24907 7615 Acked Read 9 Done Sent Done false
24908 7616 Acked Write 5 Done Sent Done false
24909 7617 Acked Write 7 Done Sent Done false
24910 7618 Acked Write 5 Done Sent Done false
24911 7619 Acked Write 9 Done Sent Done false
24912 7620 Acked Read 4 Done Sent Done false
24913 7621 Acked Read 4 Done Sent Done false
24914 7622 Acked Read 2 Done Sent Done false
24915 7623 Acked Write 8 Done Sent Done false
24916 7624 Acked Read 9 Done Sent Done false
24917 7625 Acked Write 6 Done Sent Done false
24918 7626 Acked Write 3 Done Sent Done false
24919 7627 Acked Read 3 Done Sent Done false
24920 7628 Acked Read 8 Done Sent Done false
24921 7629 Acked Flush 0 Done Sent Done false
24922 7630 Acked Read 5 Done Sent Done false
24923 7631 Acked Read 5 Done Sent Done false
24924 7632 Acked Flush 0 Done Sent Done false
24925 7633 Acked Read 2 Done Sent Done false
24926 7634 Acked Write 10 Done Sent Done false
24927 7635 Acked Read 10 Done Sent Done false
24928 7636 Acked Read 1 Done Sent Done false
24929 7637 Acked Flush 0 Done Sent Done false
24930 7638 Acked Flush 0 Done Sent Done false
24931 7639 Acked Read 3 Done Sent Done false
24932 7640 Acked Read 2 Done Sent Done false
24933 7641 Acked Read 9 Done Sent Done false
24934 STATES DS:0 DS:1 DS:2 TOTAL
24935 Sent 0 301 0 301
24936 Done 301 0 301 602
24937 Skipped 0 0 0 0
24938 Error 0 0 0 0
24939 Last Flush: Some(JobId(7638)) Some(JobId(7340)) Some(JobId(7638))
24940 Downstairs last five completed: 7340 7339 7338 7337 7336
24941 Upstairs last five completed: 7641 7640 7639 7638 7637
24942 CLIENT: Up:0 ds:301 act:2
249432025-01-29T05:16:09.490ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249442025-01-29T05:16:10.746ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249452025-01-29T05:16:10.746ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249462025-01-29T05:16:10.746ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249472025-01-29T05:16:10.746ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249482025-01-29T05:16:10.746ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
24949 [8820] got start action
24950 [8820] I am going to restart
24951 Starting downstairs at port 8820
24952 Make output file at "/tmp/dsc/downstairs-8820.txt"
24953 [8820][1] reports Starting
24954 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1224)
24955 [8820][1] reports Running
249562025-01-29T05:16:11.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
249572025-01-29T05:16:12.465ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
24958 ----------------------------------------------------------------
24959 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24960 States: Active Offline Active
24961 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24962 7341 Acked Read 2 Done Sent Done false
24963 7342 Acked Write 2 Done Sent Done false
24964 7343 Acked Read 4 Done Sent Done false
24965 7344 Acked Read 10 Done Sent Done false
24966 7345 Acked Read 3 Done Sent Done false
24967 7346 Acked Read 9 Done Sent Done false
24968 7347 Acked Write 6 Done Sent Done false
24969 7348 Acked Read 10 Done Sent Done false
24970 7349 Acked Read 5 Done Sent Done false
24971 7350 Acked Read 8 Done Sent Done false
24972 7351 Acked Read 2 Done Sent Done false
24973 7352 Acked Write 7 Done Sent Done false
24974 7353 Acked Read 6 Done Sent Done false
24975 7354 Acked Read 1 Done Sent Done false
24976 7355 Acked Write 7 Done Sent Done false
24977 7356 Acked Write 10 Done Sent Done false
24978 7357 Acked Read 3 Done Sent Done false
24979 7358 Acked Read 4 Done Sent Done false
24980 7359 Acked Read 6 Done Sent Done false
24981 7360 Acked Write 6 Done Sent Done false
24982 7361 Acked Read 9 Done Sent Done false
24983 7362 Acked Write 6 Done Sent Done false
24984 7363 Acked Write 1 Done Sent Done false
24985 7364 Acked Write 9 Done Sent Done false
24986 7365 Acked Read 4 Done Sent Done false
24987 7366 Acked Read 2 Done Sent Done false
24988 7367 Acked Write 10 Done Sent Done false
24989 7368 Acked Read 2 Done Sent Done false
24990 7369 Acked Write 6 Done Sent Done false
24991 7370 Acked Read 3 Done Sent Done false
24992 7371 Acked Read 9 Done Sent Done false
24993 7372 Acked Read 5 Done Sent Done false
24994 7373 Acked Write 1 Done Sent Done false
24995 7374 Acked Read 8 Done Sent Done false
24996 7375 Acked Flush 0 Done Sent Done false
24997 7376 Acked Write 4 Done Sent Done false
24998 7377 Acked Read 5 Done Sent Done false
24999 7378 Acked Flush 0 Done Sent Done false
25000 7379 Acked Write 10 Done Sent Done false
25001 7380 Acked Write 5 Done Sent Done false
25002 7381 Acked Flush 0 Done Sent Done false
25003 7382 Acked Read 5 Done Sent Done false
25004 7383 Acked Read 5 Done Sent Done false
25005 7384 Acked Read 1 Done Sent Done false
25006 7385 Acked Read 8 Done Sent Done false
25007 7386 Acked Write 7 Done Sent Done false
25008 7387 Acked Read 1 Done Sent Done false
25009 7388 Acked Read 4 Done Sent Done false
25010 7389 Acked Read 7 Done Sent Done false
25011 7390 Acked Read 10 Done Sent Done false
25012 7391 Acked Write 9 Done Sent Done false
25013 7392 Acked Read 3 Done Sent Done false
25014 7393 Acked Write 9 Done Sent Done false
25015 7394 Acked Write 9 Done Sent Done false
25016 7395 Acked Flush 0 Done Sent Done false
25017 7396 Acked Read 4 Done Sent Done false
25018 7397 Acked Write 6 Done Sent Done false
25019 7398 Acked Read 5 Done Sent Done false
25020 7399 Acked Read 2 Done Sent Done false
25021 7400 Acked Write 5 Done Sent Done false
25022 7401 Acked Write 7 Done Sent Done false
25023 7402 Acked Write 8 Done Sent Done false
25024 7403 Acked Flush 0 Done Sent Done false
25025 7404 Acked Write 5 Done Sent Done false
25026 7405 Acked Write 5 Done Sent Done false
25027 7406 Acked Write 3 Done Sent Done false
25028 7407 Acked Read 4 Done Sent Done false
25029 7408 Acked Read 2 Done Sent Done false
25030 7409 Acked Read 6 Done Sent Done false
25031 7410 Acked Write 10 Done Sent Done false
25032 7411 Acked Write 4 Done Sent Done false
25033 7412 Acked Write 4 Done Sent Done false
25034 7413 Acked Read 10 Done Sent Done false
25035 7414 Acked Write 1 Done Sent Done false
25036 7415 Acked Write 1 Done Sent Done false
25037 7416 Acked Read 4 Done Sent Done false
25038 7417 Acked Read 2 Done Sent Done false
25039 7418 Acked Read 5 Done Sent Done false
25040 7419 Acked Read 5 Done Sent Done false
25041 7420 Acked Read 2 Done Sent Done false
25042 7421 Acked Read 6 Done Sent Done false
25043 7422 Acked Write 1 Done Sent Done false
25044 7423 Acked Write 5 Done Sent Done false
25045 7424 Acked Read 3 Done Sent Done false
25046 7425 Acked Flush 0 Done Sent Done false
25047 7426 Acked Write 5 Done Sent Done false
25048 7427 Acked Write 8 Done Sent Done false
25049 7428 Acked Read 9 Done Sent Done false
25050 7429 Acked Read 6 Done Sent Done false
25051 7430 Acked Flush 0 Done Sent Done false
25052 7431 Acked Read 7 Done Sent Done false
25053 7432 Acked Read 9 Done Sent Done false
25054 7433 Acked Read 1 Done Sent Done false
25055 7434 Acked Read 10 Done Sent Done false
25056 7435 Acked Flush 0 Done Sent Done false
25057 7436 Acked Flush 0 Done Sent Done false
25058 7437 Acked Write 5 Done Sent Done false
25059 7438 Acked Read 10 Done Sent Done false
25060 7439 Acked Write 4 Done Sent Done false
25061 7440 Acked Read 6 Done Sent Done false
25062 7441 Acked Flush 0 Done Sent Done false
25063 7442 Acked Flush 0 Done Sent Done false
25064 7443 Acked Read 4 Done Sent Done false
25065 7444 Acked Write 4 Done Sent Done false
25066 7445 Acked Write 9 Done Sent Done false
25067 7446 Acked Read 2 Done Sent Done false
25068 7447 Acked Read 2 Done Sent Done false
25069 7448 Acked Read 1 Done Sent Done false
25070 7449 Acked Read 5 Done Sent Done false
25071 7450 Acked Flush 0 Done Sent Done false
25072 7451 Acked Flush 0 Done Sent Done false
25073 7452 Acked Write 3 Done Sent Done false
25074 7453 Acked Write 2 Done Sent Done false
25075 7454 Acked Read 3 Done Sent Done false
25076 7455 Acked Write 2 Done Sent Done false
25077 7456 Acked Read 9 Done Sent Done false
25078 7457 Acked Read 3 Done Sent Done false
25079 7458 Acked Read 8 Done Sent Done false
25080 7459 Acked Write 8 Done Sent Done false
25081 7460 Acked Write 6 Done Sent Done false
25082 7461 Acked Read 9 Done Sent Done false
25083 7462 Acked Read 10 Done Sent Done false
25084 7463 Acked Read 7 Done Sent Done false
25085 7464 Acked Read 4 Done Sent Done false
25086 7465 Acked Read 1 Done Sent Done false
25087 7466 Acked Read 4 Done Sent Done false
25088 7467 Acked Write 2 Done Sent Done false
25089 7468 Acked Write 8 Done Sent Done false
25090 7469 Acked Read 9 Done Sent Done false
25091 7470 Acked Write 6 Done Sent Done false
25092 7471 Acked Read 7 Done Sent Done false
25093 7472 Acked Flush 0 Done Sent Done false
25094 7473 Acked Read 4 Done Sent Done false
25095 7474 Acked Read 7 Done Sent Done false
25096 7475 Acked Read 10 Done Sent Done false
25097 7476 Acked Write 4 Done Sent Done false
25098 7477 Acked Flush 0 Done Sent Done false
25099 7478 Acked Read 6 Done Sent Done false
25100 7479 Acked Read 10 Done Sent Done false
25101 7480 Acked Write 9 Done Sent Done false
25102 7481 Acked Read 3 Done Sent Done false
25103 7482 Acked Write 10 Done Sent Done false
25104 7483 Acked Write 8 Done Sent Done false
25105 7484 Acked Write 10 Done Sent Done false
25106 7485 Acked Write 4 Done Sent Done false
25107 7486 Acked Write 4 Done Sent Done false
25108 7487 Acked Read 2 Done Sent Done false
25109 7488 Acked Write 10 Done Sent Done false
25110 7489 Acked Flush 0 Done Sent Done false
25111 7490 Acked Write 4 Done Sent Done false
25112 7491 Acked Read 8 Done Sent Done false
25113 7492 Acked Read 3 Done Sent Done false
25114 7493 Acked Write 10 Done Sent Done false
25115 7494 Acked Read 8 Done Sent Done false
25116 7495 Acked Read 9 Done Sent Done false
25117 7496 Acked Read 9 Done Sent Done false
25118 7497 Acked Write 7 Done Sent Done false
25119 7498 Acked Flush 0 Done Sent Done false
25120 7499 Acked Write 6 Done Sent Done false
25121 7500 Acked Read 1 Done Sent Done false
25122 7501 Acked Write 5 Done Sent Done false
25123 7502 Acked Read 4 Done Sent Done false
25124 7503 Acked Flush 0 Done Sent Done false
25125 7504 Acked Read 3 Done Sent Done false
25126 7505 Acked Read 2 Done Sent Done false
25127 7506 Acked Flush 0 Done Sent Done false
25128 7507 Acked Write 10 Done Sent Done false
25129 7508 Acked Write 5 Done Sent Done false
25130 7509 Acked Write 7 Done Sent Done false
25131 7510 Acked Write 2 Done Sent Done false
25132 7511 Acked Write 2 Done Sent Done false
25133 7512 Acked Read 7 Done Sent Done false
25134 7513 Acked Read 10 Done Sent Done false
25135 7514 Acked Read 7 Done Sent Done false
25136 7515 Acked Read 9 Done Sent Done false
25137 7516 Acked Write 2 Done Sent Done false
25138 7517 Acked Read 6 Done Sent Done false
25139 7518 Acked Write 4 Done Sent Done false
25140 7519 Acked Read 4 Done Sent Done false
25141 7520 Acked Read 6 Done Sent Done false
25142 7521 Acked Write 9 Done Sent Done false
25143 7522 Acked Write 7 Done Sent Done false
25144 7523 Acked Write 7 Done Sent Done false
25145 7524 Acked Write 7 Done Sent Done false
25146 7525 Acked Read 7 Done Sent Done false
25147 7526 Acked Write 1 Done Sent Done false
25148 7527 Acked Write 8 Done Sent Done false
25149 7528 Acked Write 3 Done Sent Done false
25150 7529 Acked Write 4 Done Sent Done false
25151 7530 Acked Write 9 Done Sent Done false
25152 7531 Acked Read 9 Done Sent Done false
25153 7532 Acked Write 2 Done Sent Done false
25154 7533 Acked Read 4 Done Sent Done false
25155 7534 Acked Write 5 Done Sent Done false
25156 7535 Acked Write 7 Done Sent Done false
25157 7536 Acked Read 5 Done Sent Done false
25158 7537 Acked Flush 0 Done Sent Done false
25159 7538 Acked Write 1 Done Sent Done false
25160 7539 Acked Read 2 Done Sent Done false
25161 7540 Acked Write 8 Done Sent Done false
25162 7541 Acked Write 1 Done Sent Done false
25163 7542 Acked Read 2 Done Sent Done false
25164 7543 Acked Write 5 Done Sent Done false
25165 7544 Acked Flush 0 Done Sent Done false
25166 7545 Acked Write 2 Done Sent Done false
25167 7546 Acked Read 7 Done Sent Done false
25168 7547 Acked Read 3 Done Sent Done false
25169 7548 Acked Read 8 Done Sent Done false
25170 7549 Acked Read 5 Done Sent Done false
25171 7550 Acked Read 2 Done Sent Done false
25172 7551 Acked Write 2 Done Sent Done false
25173 7552 Acked Read 7 Done Sent Done false
25174 7553 Acked Write 8 Done Sent Done false
25175 7554 Acked Write 1 Done Sent Done false
25176 7555 Acked Write 4 Done Sent Done false
25177 7556 Acked Flush 0 Done Sent Done false
25178 7557 Acked Write 4 Done Sent Done false
25179 7558 Acked Write 1 Done Sent Done false
25180 7559 Acked Write 10 Done Sent Done false
25181 7560 Acked Read 4 Done Sent Done false
25182 7561 Acked Read 8 Done Sent Done false
25183 7562 Acked Read 9 Done Sent Done false
25184 7563 Acked Write 4 Done Sent Done false
25185 7564 Acked Read 9 Done Sent Done false
25186 7565 Acked Read 6 Done Sent Done false
25187 7566 Acked Read 3 Done Sent Done false
25188 7567 Acked Flush 0 Done Sent Done false
25189 7568 Acked Write 3 Done Sent Done false
25190 7569 Acked Write 5 Done Sent Done false
25191 7570 Acked Write 9 Done Sent Done false
25192 7571 Acked Read 6 Done Sent Done false
25193 7572 Acked Read 8 Done Sent Done false
25194 7573 Acked Write 3 Done Sent Done false
25195 7574 Acked Read 10 Done Sent Done false
25196 7575 Acked Read 1 Done Sent Done false
25197 7576 Acked Flush 0 Done Sent Done false
25198 7577 Acked Read 7 Done Sent Done false
25199 7578 Acked Flush 0 Done Sent Done false
25200 7579 Acked Read 2 Done Sent Done false
25201 7580 Acked Write 9 Done Sent Done false
25202 7581 Acked Write 1 Done Sent Done false
25203 7582 Acked Read 8 Done Sent Done false
25204 7583 Acked Write 1 Done Sent Done false
25205 7584 Acked Write 5 Done Sent Done false
25206 7585 Acked Write 6 Done Sent Done false
25207 7586 Acked Write 1 Done Sent Done false
25208 7587 Acked Write 3 Done Sent Done false
25209 7588 Acked Read 3 Done Sent Done false
25210 7589 Acked Read 6 Done Sent Done false
25211 7590 Acked Flush 0 Done Sent Done false
25212 7591 Acked Write 8 Done Sent Done false
25213 7592 Acked Write 9 Done Sent Done false
25214 7593 Acked Flush 0 Done Sent Done false
25215 7594 Acked Read 8 Done Sent Done false
25216 7595 Acked Read 3 Done Sent Done false
25217 7596 Acked Read 4 Done Sent Done false
25218 7597 Acked Read 5 Done Sent Done false
25219 7598 Acked Read 8 Done Sent Done false
25220 7599 Acked Read 8 Done Sent Done false
25221 7600 Acked Read 2 Done Sent Done false
25222 7601 Acked Read 2 Done Sent Done false
25223 7602 Acked Write 5 Done Sent Done false
25224 7603 Acked Read 1 Done Sent Done false
25225 7604 Acked Write 9 Done Sent Done false
25226 7605 Acked Read 5 Done Sent Done false
25227 7606 Acked Write 1 Done Sent Done false
25228 7607 Acked Read 5 Done Sent Done false
25229 7608 Acked Read 6 Done Sent Done false
25230 7609 Acked Write 4 Done Sent Done false
25231 7610 Acked Read 2 Done Sent Done false
25232 7611 Acked Read 5 Done Sent Done false
25233 7612 Acked Write 9 Done Sent Done false
25234 7613 Acked Write 7 Done Sent Done false
25235 7614 Acked Write 8 Done Sent Done false
25236 7615 Acked Read 9 Done Sent Done false
25237 7616 Acked Write 5 Done Sent Done false
25238 7617 Acked Write 7 Done Sent Done false
25239 7618 Acked Write 5 Done Sent Done false
25240 7619 Acked Write 9 Done Sent Done false
25241 7620 Acked Read 4 Done Sent Done false
25242 7621 Acked Read 4 Done Sent Done false
25243 7622 Acked Read 2 Done Sent Done false
25244 7623 Acked Write 8 Done Sent Done false
25245 7624 Acked Read 9 Done Sent Done false
25246 7625 Acked Write 6 Done Sent Done false
25247 7626 Acked Write 3 Done Sent Done false
25248 7627 Acked Read 3 Done Sent Done false
25249 7628 Acked Read 8 Done Sent Done false
25250 7629 Acked Flush 0 Done Sent Done false
25251 7630 Acked Read 5 Done Sent Done false
25252 7631 Acked Read 5 Done Sent Done false
25253 7632 Acked Flush 0 Done Sent Done false
25254 7633 Acked Read 2 Done Sent Done false
25255 7634 Acked Write 10 Done Sent Done false
25256 7635 Acked Read 10 Done Sent Done false
25257 7636 Acked Read 1 Done Sent Done false
25258 7637 Acked Flush 0 Done Sent Done false
25259 7638 Acked Flush 0 Done Sent Done false
25260 7639 Acked Read 3 Done Sent Done false
25261 7640 Acked Read 2 Done Sent Done false
25262 7641 Acked Read 9 Done Sent Done false
25263 STATES DS:0 DS:1 DS:2 TOTAL
25264 Sent 0 301 0 301
25265 Done 301 0 301 602
25266 Skipped 0 0 0 0
25267 Error 0 0 0 0
25268 Last Flush: Some(JobId(7638)) Some(JobId(7340)) Some(JobId(7638))
25269 Downstairs last five completed: 7340 7339 7338 7337 7336
25270 Upstairs last five completed: 7641 7640 7639 7638 7637
25271 CLIENT: Up:0 ds:301 act:2
252722025-01-29T05:16:13.492ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
252732025-01-29T05:16:14.470ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
252742025-01-29T05:16:15.752ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
252752025-01-29T05:16:16.764ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
25276 ----------------------------------------------------------------
25277 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25278 States: Active Offline Active
25279 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25280 7341 Acked Read 2 Done Sent Done false
25281 7342 Acked Write 2 Done Sent Done false
25282 7343 Acked Read 4 Done Sent Done false
25283 7344 Acked Read 10 Done Sent Done false
25284 7345 Acked Read 3 Done Sent Done false
25285 7346 Acked Read 9 Done Sent Done false
25286 7347 Acked Write 6 Done Sent Done false
25287 7348 Acked Read 10 Done Sent Done false
25288 7349 Acked Read 5 Done Sent Done false
25289 7350 Acked Read 8 Done Sent Done false
25290 7351 Acked Read 2 Done Sent Done false
25291 7352 Acked Write 7 Done Sent Done false
25292 7353 Acked Read 6 Done Sent Done false
25293 7354 Acked Read 1 Done Sent Done false
25294 7355 Acked Write 7 Done Sent Done false
25295 7356 Acked Write 10 Done Sent Done false
25296 7357 Acked Read 3 Done Sent Done false
25297 7358 Acked Read 4 Done Sent Done false
25298 7359 Acked Read 6 Done Sent Done false
25299 7360 Acked Write 6 Done Sent Done false
25300 7361 Acked Read 9 Done Sent Done false
25301 7362 Acked Write 6 Done Sent Done false
25302 7363 Acked Write 1 Done Sent Done false
25303 7364 Acked Write 9 Done Sent Done false
25304 7365 Acked Read 4 Done Sent Done false
25305 7366 Acked Read 2 Done Sent Done false
25306 7367 Acked Write 10 Done Sent Done false
25307 7368 Acked Read 2 Done Sent Done false
25308 7369 Acked Write 6 Done Sent Done false
25309 7370 Acked Read 3 Done Sent Done false
25310 7371 Acked Read 9 Done Sent Done false
25311 7372 Acked Read 5 Done Sent Done false
25312 7373 Acked Write 1 Done Sent Done false
25313 7374 Acked Read 8 Done Sent Done false
25314 7375 Acked Flush 0 Done Sent Done false
25315 7376 Acked Write 4 Done Sent Done false
25316 7377 Acked Read 5 Done Sent Done false
25317 7378 Acked Flush 0 Done Sent Done false
25318 7379 Acked Write 10 Done Sent Done false
25319 7380 Acked Write 5 Done Sent Done false
25320 7381 Acked Flush 0 Done Sent Done false
25321 7382 Acked Read 5 Done Sent Done false
25322 7383 Acked Read 5 Done Sent Done false
25323 7384 Acked Read 1 Done Sent Done false
25324 7385 Acked Read 8 Done Sent Done false
25325 7386 Acked Write 7 Done Sent Done false
25326 7387 Acked Read 1 Done Sent Done false
25327 7388 Acked Read 4 Done Sent Done false
25328 7389 Acked Read 7 Done Sent Done false
25329 7390 Acked Read 10 Done Sent Done false
25330 7391 Acked Write 9 Done Sent Done false
25331 7392 Acked Read 3 Done Sent Done false
25332 7393 Acked Write 9 Done Sent Done false
25333 7394 Acked Write 9 Done Sent Done false
25334 7395 Acked Flush 0 Done Sent Done false
25335 7396 Acked Read 4 Done Sent Done false
25336 7397 Acked Write 6 Done Sent Done false
25337 7398 Acked Read 5 Done Sent Done false
25338 7399 Acked Read 2 Done Sent Done false
25339 7400 Acked Write 5 Done Sent Done false
25340 7401 Acked Write 7 Done Sent Done false
25341 7402 Acked Write 8 Done Sent Done false
25342 7403 Acked Flush 0 Done Sent Done false
25343 7404 Acked Write 5 Done Sent Done false
25344 7405 Acked Write 5 Done Sent Done false
25345 7406 Acked Write 3 Done Sent Done false
25346 7407 Acked Read 4 Done Sent Done false
25347 7408 Acked Read 2 Done Sent Done false
25348 7409 Acked Read 6 Done Sent Done false
25349 7410 Acked Write 10 Done Sent Done false
25350 7411 Acked Write 4 Done Sent Done false
25351 7412 Acked Write 4 Done Sent Done false
25352 7413 Acked Read 10 Done Sent Done false
25353 7414 Acked Write 1 Done Sent Done false
25354 7415 Acked Write 1 Done Sent Done false
25355 7416 Acked Read 4 Done Sent Done false
25356 7417 Acked Read 2 Done Sent Done false
25357 7418 Acked Read 5 Done Sent Done false
25358 7419 Acked Read 5 Done Sent Done false
25359 7420 Acked Read 2 Done Sent Done false
25360 7421 Acked Read 6 Done Sent Done false
25361 7422 Acked Write 1 Done Sent Done false
25362 7423 Acked Write 5 Done Sent Done false
25363 7424 Acked Read 3 Done Sent Done false
25364 7425 Acked Flush 0 Done Sent Done false
25365 7426 Acked Write 5 Done Sent Done false
25366 7427 Acked Write 8 Done Sent Done false
25367 7428 Acked Read 9 Done Sent Done false
25368 7429 Acked Read 6 Done Sent Done false
25369 7430 Acked Flush 0 Done Sent Done false
25370 7431 Acked Read 7 Done Sent Done false
25371 7432 Acked Read 9 Done Sent Done false
25372 7433 Acked Read 1 Done Sent Done false
25373 7434 Acked Read 10 Done Sent Done false
25374 7435 Acked Flush 0 Done Sent Done false
25375 7436 Acked Flush 0 Done Sent Done false
25376 7437 Acked Write 5 Done Sent Done false
25377 7438 Acked Read 10 Done Sent Done false
25378 7439 Acked Write 4 Done Sent Done false
25379 7440 Acked Read 6 Done Sent Done false
25380 7441 Acked Flush 0 Done Sent Done false
25381 7442 Acked Flush 0 Done Sent Done false
25382 7443 Acked Read 4 Done Sent Done false
25383 7444 Acked Write 4 Done Sent Done false
25384 7445 Acked Write 9 Done Sent Done false
25385 7446 Acked Read 2 Done Sent Done false
25386 7447 Acked Read 2 Done Sent Done false
25387 7448 Acked Read 1 Done Sent Done false
25388 7449 Acked Read 5 Done Sent Done false
25389 7450 Acked Flush 0 Done Sent Done false
25390 7451 Acked Flush 0 Done Sent Done false
25391 7452 Acked Write 3 Done Sent Done false
25392 7453 Acked Write 2 Done Sent Done false
25393 7454 Acked Read 3 Done Sent Done false
25394 7455 Acked Write 2 Done Sent Done false
25395 7456 Acked Read 9 Done Sent Done false
25396 7457 Acked Read 3 Done Sent Done false
25397 7458 Acked Read 8 Done Sent Done false
25398 7459 Acked Write 8 Done Sent Done false
25399 7460 Acked Write 6 Done Sent Done false
25400 7461 Acked Read 9 Done Sent Done false
25401 7462 Acked Read 10 Done Sent Done false
25402 7463 Acked Read 7 Done Sent Done false
25403 7464 Acked Read 4 Done Sent Done false
25404 7465 Acked Read 1 Done Sent Done false
25405 7466 Acked Read 4 Done Sent Done false
25406 7467 Acked Write 2 Done Sent Done false
25407 7468 Acked Write 8 Done Sent Done false
25408 7469 Acked Read 9 Done Sent Done false
25409 7470 Acked Write 6 Done Sent Done false
25410 7471 Acked Read 7 Done Sent Done false
25411 7472 Acked Flush 0 Done Sent Done false
25412 7473 Acked Read 4 Done Sent Done false
25413 7474 Acked Read 7 Done Sent Done false
25414 7475 Acked Read 10 Done Sent Done false
25415 7476 Acked Write 4 Done Sent Done false
25416 7477 Acked Flush 0 Done Sent Done false
25417 7478 Acked Read 6 Done Sent Done false
25418 7479 Acked Read 10 Done Sent Done false
25419 7480 Acked Write 9 Done Sent Done false
25420 7481 Acked Read 3 Done Sent Done false
25421 7482 Acked Write 10 Done Sent Done false
25422 7483 Acked Write 8 Done Sent Done false
25423 7484 Acked Write 10 Done Sent Done false
25424 7485 Acked Write 4 Done Sent Done false
25425 7486 Acked Write 4 Done Sent Done false
25426 7487 Acked Read 2 Done Sent Done false
25427 7488 Acked Write 10 Done Sent Done false
25428 7489 Acked Flush 0 Done Sent Done false
25429 7490 Acked Write 4 Done Sent Done false
25430 7491 Acked Read 8 Done Sent Done false
25431 7492 Acked Read 3 Done Sent Done false
25432 7493 Acked Write 10 Done Sent Done false
25433 7494 Acked Read 8 Done Sent Done false
25434 7495 Acked Read 9 Done Sent Done false
25435 7496 Acked Read 9 Done Sent Done false
25436 7497 Acked Write 7 Done Sent Done false
25437 7498 Acked Flush 0 Done Sent Done false
25438 7499 Acked Write 6 Done Sent Done false
25439 7500 Acked Read 1 Done Sent Done false
25440 7501 Acked Write 5 Done Sent Done false
25441 7502 Acked Read 4 Done Sent Done false
25442 7503 Acked Flush 0 Done Sent Done false
25443 7504 Acked Read 3 Done Sent Done false
25444 7505 Acked Read 2 Done Sent Done false
25445 7506 Acked Flush 0 Done Sent Done false
25446 7507 Acked Write 10 Done Sent Done false
25447 7508 Acked Write 5 Done Sent Done false
25448 7509 Acked Write 7 Done Sent Done false
25449 7510 Acked Write 2 Done Sent Done false
25450 7511 Acked Write 2 Done Sent Done false
25451 7512 Acked Read 7 Done Sent Done false
25452 7513 Acked Read 10 Done Sent Done false
25453 7514 Acked Read 7 Done Sent Done false
25454 7515 Acked Read 9 Done Sent Done false
25455 7516 Acked Write 2 Done Sent Done false
25456 7517 Acked Read 6 Done Sent Done false
25457 7518 Acked Write 4 Done Sent Done false
25458 7519 Acked Read 4 Done Sent Done false
25459 7520 Acked Read 6 Done Sent Done false
25460 7521 Acked Write 9 Done Sent Done false
25461 7522 Acked Write 7 Done Sent Done false
25462 7523 Acked Write 7 Done Sent Done false
25463 7524 Acked Write 7 Done Sent Done false
25464 7525 Acked Read 7 Done Sent Done false
25465 7526 Acked Write 1 Done Sent Done false
25466 7527 Acked Write 8 Done Sent Done false
25467 7528 Acked Write 3 Done Sent Done false
25468 7529 Acked Write 4 Done Sent Done false
25469 7530 Acked Write 9 Done Sent Done false
25470 7531 Acked Read 9 Done Sent Done false
25471 7532 Acked Write 2 Done Sent Done false
25472 7533 Acked Read 4 Done Sent Done false
25473 7534 Acked Write 5 Done Sent Done false
25474 7535 Acked Write 7 Done Sent Done false
25475 7536 Acked Read 5 Done Sent Done false
25476 7537 Acked Flush 0 Done Sent Done false
25477 7538 Acked Write 1 Done Sent Done false
25478 7539 Acked Read 2 Done Sent Done false
25479 7540 Acked Write 8 Done Sent Done false
25480 7541 Acked Write 1 Done Sent Done false
25481 7542 Acked Read 2 Done Sent Done false
25482 7543 Acked Write 5 Done Sent Done false
25483 7544 Acked Flush 0 Done Sent Done false
25484 7545 Acked Write 2 Done Sent Done false
25485 7546 Acked Read 7 Done Sent Done false
25486 7547 Acked Read 3 Done Sent Done false
25487 7548 Acked Read 8 Done Sent Done false
25488 7549 Acked Read 5 Done Sent Done false
25489 7550 Acked Read 2 Done Sent Done false
25490 7551 Acked Write 2 Done Sent Done false
25491 7552 Acked Read 7 Done Sent Done false
25492 7553 Acked Write 8 Done Sent Done false
25493 7554 Acked Write 1 Done Sent Done false
25494 7555 Acked Write 4 Done Sent Done false
25495 7556 Acked Flush 0 Done Sent Done false
25496 7557 Acked Write 4 Done Sent Done false
25497 7558 Acked Write 1 Done Sent Done false
25498 7559 Acked Write 10 Done Sent Done false
25499 7560 Acked Read 4 Done Sent Done false
25500 7561 Acked Read 8 Done Sent Done false
25501 7562 Acked Read 9 Done Sent Done false
25502 7563 Acked Write 4 Done Sent Done false
25503 7564 Acked Read 9 Done Sent Done false
25504 7565 Acked Read 6 Done Sent Done false
25505 7566 Acked Read 3 Done Sent Done false
25506 7567 Acked Flush 0 Done Sent Done false
25507 7568 Acked Write 3 Done Sent Done false
25508 7569 Acked Write 5 Done Sent Done false
25509 7570 Acked Write 9 Done Sent Done false
25510 7571 Acked Read 6 Done Sent Done false
25511 7572 Acked Read 8 Done Sent Done false
25512 7573 Acked Write 3 Done Sent Done false
25513 7574 Acked Read 10 Done Sent Done false
25514 7575 Acked Read 1 Done Sent Done false
25515 7576 Acked Flush 0 Done Sent Done false
25516 7577 Acked Read 7 Done Sent Done false
25517 7578 Acked Flush 0 Done Sent Done false
25518 7579 Acked Read 2 Done Sent Done false
25519 7580 Acked Write 9 Done Sent Done false
25520 7581 Acked Write 1 Done Sent Done false
25521 7582 Acked Read 8 Done Sent Done false
25522 7583 Acked Write 1 Done Sent Done false
25523 7584 Acked Write 5 Done Sent Done false
25524 7585 Acked Write 6 Done Sent Done false
25525 7586 Acked Write 1 Done Sent Done false
25526 7587 Acked Write 3 Done Sent Done false
25527 7588 Acked Read 3 Done Sent Done false
25528 7589 Acked Read 6 Done Sent Done false
25529 7590 Acked Flush 0 Done Sent Done false
25530 7591 Acked Write 8 Done Sent Done false
25531 7592 Acked Write 9 Done Sent Done false
25532 7593 Acked Flush 0 Done Sent Done false
25533 7594 Acked Read 8 Done Sent Done false
25534 7595 Acked Read 3 Done Sent Done false
25535 7596 Acked Read 4 Done Sent Done false
25536 7597 Acked Read 5 Done Sent Done false
25537 7598 Acked Read 8 Done Sent Done false
25538 7599 Acked Read 8 Done Sent Done false
25539 7600 Acked Read 2 Done Sent Done false
25540 7601 Acked Read 2 Done Sent Done false
25541 7602 Acked Write 5 Done Sent Done false
25542 7603 Acked Read 1 Done Sent Done false
25543 7604 Acked Write 9 Done Sent Done false
25544 7605 Acked Read 5 Done Sent Done false
25545 7606 Acked Write 1 Done Sent Done false
25546 7607 Acked Read 5 Done Sent Done false
25547 7608 Acked Read 6 Done Sent Done false
25548 7609 Acked Write 4 Done Sent Done false
25549 7610 Acked Read 2 Done Sent Done false
25550 7611 Acked Read 5 Done Sent Done false
25551 7612 Acked Write 9 Done Sent Done false
25552 7613 Acked Write 7 Done Sent Done false
25553 7614 Acked Write 8 Done Sent Done false
25554 7615 Acked Read 9 Done Sent Done false
25555 7616 Acked Write 5 Done Sent Done false
25556 7617 Acked Write 7 Done Sent Done false
25557 7618 Acked Write 5 Done Sent Done false
25558 7619 Acked Write 9 Done Sent Done false
25559 7620 Acked Read 4 Done Sent Done false
25560 7621 Acked Read 4 Done Sent Done false
25561 7622 Acked Read 2 Done Sent Done false
25562 7623 Acked Write 8 Done Sent Done false
25563 7624 Acked Read 9 Done Sent Done false
25564 7625 Acked Write 6 Done Sent Done false
25565 7626 Acked Write 3 Done Sent Done false
25566 7627 Acked Read 3 Done Sent Done false
25567 7628 Acked Read 8 Done Sent Done false
25568 7629 Acked Flush 0 Done Sent Done false
25569 7630 Acked Read 5 Done Sent Done false
25570 7631 Acked Read 5 Done Sent Done false
25571 7632 Acked Flush 0 Done Sent Done false
25572 7633 Acked Read 2 Done Sent Done false
25573 7634 Acked Write 10 Done Sent Done false
25574 7635 Acked Read 10 Done Sent Done false
25575 7636 Acked Read 1 Done Sent Done false
25576 7637 Acked Flush 0 Done Sent Done false
25577 7638 Acked Flush 0 Done Sent Done false
25578 7639 Acked Read 3 Done Sent Done false
25579 7640 Acked Read 2 Done Sent Done false
25580 7641 Acked Read 9 Done Sent Done false
25581 STATES DS:0 DS:1 DS:2 TOTAL
25582 Sent 0 301 0 301
25583 Done 301 0 301 602
25584 Skipped 0 0 0 0
25585 Error 0 0 0 0
25586 Last Flush: Some(JobId(7638)) Some(JobId(7340)) Some(JobId(7638))
25587 Downstairs last five completed: 7340 7339 7338 7337 7336
25588 Upstairs last five completed: 7641 7640 7639 7638 7637
25589 CLIENT: Up:0 ds:301 act:2
255902025-01-29T05:16:17.495ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255912025-01-29T05:16:18.475ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255922025-01-29T05:16:19.477ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255932025-01-29T05:16:20.747ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255942025-01-29T05:16:20.747ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38889) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255952025-01-29T05:16:20.748ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255962025-01-29T05:16:20.749ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255972025-01-29T05:16:20.749ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255982025-01-29T05:16:20.749ZINFOcrucible: send last flush ID to this DS: Some(JobId(7340)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
255992025-01-29T05:16:20.749ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
256002025-01-29T05:16:20.749ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7340)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
256012025-01-29T05:16:20.749ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7340)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
25602 ----------------------------------------------------------------
25603 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
25604 States: Active Active Active
25605 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25606 7639 Acked Read 3 Done Done Done true
25607 7640 Acked Read 2 Done Done Done true
25608 7641 Acked Read 9 Done Done Done true
25609 STATES DS:0 DS:1 DS:2 TOTAL
25610 Sent 0 0 0 0
25611 Done 3 3 3 9
25612 Skipped 0 0 0 0
25613 Error 0 0 0 0
25614 Last Flush: Some(JobId(7638)) Some(JobId(7638)) Some(JobId(7638))
25615 Downstairs last five completed: 7638 7637 7636 7635 7634
25616 Upstairs last five completed: 7641 7640 7639 7638 7637
25617 CLIENT: Up:0 ds:3 act:3
25618 ----------------------------------------------------------------
25619 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25620 Downstairs last five completed: 7642 7641 7640 7639 7638
25621 Upstairs last five completed: 7642 7641 7640 7639 7638
25622 CLIENT: Up:0 ds:0 act:3
25623 Replay: All jobs finished
25624 Main task has work to do, go find it
25625 got dsc Stop(2)
25626 stop 2
25627 Jan 29 05:16:24.997 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 84a6ba32-d570-451d-980e-4f053fa7ac79, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
25628 [8830] Got stop action so:false kr:false
25629 Jan 29 05:16:24.998 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3408a89f-65d1-4aaa-aa25-daea17b35823, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
256302025-01-29T05:16:24.998ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
256312025-01-29T05:16:24.998ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
256322025-01-29T05:16:24.998ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
256332025-01-29T05:16:24.998ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
25634 [8830] Exited with: ExitStatus(unix_wait_status(9))
25635 [8830][2] reports Exit
25636 Jan 29 05:16:28.999 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4bba4d78-363d-4b51-939c-471364548ac3, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
25637 001/300 Read block 4382 len 36864
25638 002/300 Read block 595 len 20480
25639 003/300 Write block 605 len 4096 data: 3
25640 004/300 Flush
25641 005/300 Write block 1169 len 28672 data: 8 8 10 10 9 9 9
25642 006/300 Read block 2758 len 28672
25643 007/300 Write block 4235 len 36864 data: 5 5 4 6 6 5 4 2 2
25644 008/300 Write block 668 len 32768 data: 5 3 4 4 4 4 4 4
25645 009/300 Write block 1123 len 28672 data: 7 8 8 7 7 6 4
25646 010/300 Write block 1264 len 32768 data: 4 5 4 4 3 4 4 4
25647 011/300 Write block 3653 len 16384 data: 3 3 3 4
25648 012/300 Read block 4429 len 40960
25649 013/300 Read block 2336 len 12288
25650 014/300 Write block 612 len 12288 data: 4 3 3
25651 015/300 Flush
25652 016/300 Read block 4947 len 16384
25653 017/300 Read block 4731 len 16384
25654 018/300 Read block 3121 len 20480
25655 019/300 Write block 3564 len 28672 data: 3 3 3 4 6 5 6
25656 020/300 Write block 4932 len 20480 data: 6 5 5 5 4
25657 021/300 Flush
25658 022/300 Read block 4565 len 36864
25659 023/300 Read block 810 len 4096
25660 024/300 Read block 970 len 16384
25661 025/300 Read block 18 len 8192
25662 026/300 Read block 779 len 20480
25663 027/300 Read block 3254 len 28672
25664 028/300 Read block 2126 len 36864
25665 029/300 Read block 1257 len 8192
25666 030/300 Read block 4828 len 20480
25667 031/300 Write block 2729 len 40960 data: 6 6 8 8 8 9 9 8 7 8
25668 032/300 Write block 2695 len 28672 data: 5 5 5 5 5 5 6
25669 033/300 Write block 1040 len 12288 data: 4 6 7
25670 034/300 Write block 826 len 24576 data: 6 4 3 5 5 6
25671 035/300 Flush
25672 036/300 Flush
25673 037/300 Write block 4162 len 20480 data: 3 4 3 4 4
25674 038/300 Write block 3025 len 36864 data: 4 3 3 3 4 4 3 5 6
25675 039/300 Read block 3814 len 32768
25676 040/300 Read block 1760 len 36864
25677 041/300 Read block 4992 len 32768
25678 042/300 Write block 4448 len 40960 data: 3 5 6 7 8 8 9 8 7 6
25679 043/300 Read block 1837 len 12288
25680 044/300 Write block 1513 len 32768 data: 4 5 4 5 4 4 5 5
25681 045/300 Flush
25682 046/300 Read block 92 len 20480
25683 047/300 Flush
25684 048/300 Read block 1278 len 4096
25685 049/300 Read block 4981 len 8192
25686 050/300 Flush
25687 051/300 Read block 2662 len 40960
25688 052/300 Write block 1061 len 36864 data: 5 5 4 2 2 3 4 4 4
25689 053/300 Read block 2862 len 8192
25690 054/300 Write block 1936 len 8192 data: 8 9
25691 055/300 Read block 2148 len 28672
25692 056/300 Write block 862 len 12288 data: 6 5 6
25693 057/300 Write block 4200 len 28672 data: 8 8 8 8 7 8 8
25694 058/300 Flush
25695 059/300 Read block 3909 len 24576
25696 060/300 Write block 3727 len 36864 data: 4 5 5 5 5 7 7 6 6
25697 061/300 Read block 318 len 24576
25698 062/300 Write block 1706 len 4096 data: 4
25699 063/300 Read block 1897 len 40960
25700 064/300 Flush
25701 065/300 Write block 3417 len 28672 data: 4 5 6 7 6 6 7
25702 066/300 Read block 2494 len 36864
25703 067/300 Flush
25704 068/300 Write block 2041 len 36864 data: 4 3 3 3 3 2 2 2 3
25705 069/300 Write block 3344 len 36864 data: 5 6 7 6 7 7 6 7 6
25706 070/300 Flush
25707 071/300 Flush
25708 072/300 Flush
25709 073/300 Read block 2368 len 8192
25710 074/300 Read block 4245 len 16384
25711 075/300 Read block 3534 len 40960
25712 076/300 Write block 2996 len 20480 data: 3 3 4 5 5
25713 077/300 Read block 3025 len 24576
25714 078/300 Read block 863 len 20480
25715 079/300 Read block 4030 len 8192
25716 080/300 Read block 2128 len 8192
25717 081/300 Write block 941 len 28672 data: 3 3 3 3 5 6 6
25718 082/300 Read block 3820 len 28672
25719 083/300 Write block 4450 len 32768 data: 7 8 9 9 10 9 8 7
25720 084/300 Read block 726 len 32768
25721 085/300 Read block 3459 len 8192
25722 086/300 Read block 4805 len 32768
25723 087/300 Write block 925 len 8192 data: 3 3
25724 088/300 Read block 247 len 4096
25725 089/300 Read block 1678 len 36864
25726 090/300 Read block 3943 len 20480
25727 091/300 Write block 4877 len 12288 data: 4 4 3
25728 092/300 Read block 3670 len 40960
25729 093/300 Read block 1288 len 12288
25730 094/300 Write block 4305 len 40960 data: 5 5 5 7 6 4 3 3 5 5
25731 095/300 Flush
25732 096/300 Read block 1435 len 28672
25733 097/300 Read block 4382 len 8192
25734 098/300 Write block 3241 len 4096 data: 3
25735 099/300 Read block 1910 len 20480
25736 100/300 Read block 3211 len 40960
25737 101/300 Read block 1837 len 28672
25738 102/300 Write block 225 len 16384 data: 4 6 5 5
25739 103/300 Write block 2875 len 8192 data: 4 4
25740 104/300 Read block 1599 len 8192
25741 105/300 Read block 144 len 12288
25742 106/300 Read block 2970 len 40960
25743 107/300 Write block 3257 len 12288 data: 4 5 6
25744 108/300 Write block 1696 len 40960 data: 6 6 6 5 3 3 3 5 5 4
25745 109/300 Read block 2376 len 20480
25746 110/300 Write block 569 len 28672 data: 3 3 3 4 5 5 6
25747 111/300 Write block 771 len 24576 data: 3 5 5 6 6 5
25748 112/300 Read block 3986 len 24576
25749 113/300 Read block 1031 len 32768
25750 114/300 Read block 3214 len 8192
25751 115/300 Read block 4704 len 40960
25752 116/300 Write block 23 len 16384 data: 6 6 4 4
25753 117/300 Write block 2416 len 4096 data: 6
25754 118/300 Write block 1015 len 24576 data: 5 5 5 5 5 4
25755 119/300 Read block 605 len 20480
25756 120/300 Read block 2695 len 4096
25757 121/300 Read block 87 len 40960
25758 122/300 Read block 4708 len 36864
25759 123/300 Read block 1491 len 40960
25760 124/300 Write block 4929 len 20480 data: 5 6 6 7 6
25761 125/300 Write block 3919 len 4096 data: 3
25762 126/300 Read block 1452 len 28672
25763 127/300 Read block 4971 len 12288
25764 128/300 Read block 542 len 4096
25765 129/300 Read block 3516 len 28672
25766 130/300 Read block 2858 len 32768
25767 131/300 Write block 1792 len 24576 data: 9 8 8 9 6 6
25768 132/300 Flush
25769 133/300 Read block 4752 len 8192
25770 134/300 Read block 3437 len 16384
25771 135/300 Flush
25772 136/300 Write block 108 len 8192 data: 4 4
25773 137/300 Write block 4236 len 4096 data: 6
25774 138/300 Flush
25775 139/300 Read block 2523 len 16384
25776 140/300 Write block 1393 len 4096 data: 2
25777 141/300 Write block 4214 len 8192 data: 3 3
25778 142/300 Write block 3112 len 32768 data: 5 4 4 5 4 4 6 6
25779 143/300 Read block 3528 len 40960
25780 144/300 Flush
25781 145/300 Read block 3796 len 28672
25782 146/300 Flush
25783 147/300 Read block 1071 len 32768
25784 148/300 Read block 2023 len 4096
25785 149/300 Read block 3654 len 32768
25786 150/300 Read block 4918 len 32768
25787 151/300 Read block 3960 len 12288
25788 152/300 Read block 2058 len 20480
25789 153/300 Read block 3219 len 4096
25790 154/300 Read block 4086 len 36864
25791 155/300 Read block 1263 len 20480
25792 156/300 Write block 1214 len 8192 data: 5 4
25793 157/300 Read block 459 len 24576
25794 158/300 Read block 1505 len 32768
25795 159/300 Flush
25796 160/300 Read block 3212 len 8192
25797 161/300 Write block 219 len 40960 data: 5 5 4 4 3 4 5 7 6 6
25798 162/300 Flush
25799 163/300 Write block 1933 len 8192 data: 3 4
25800 164/300 Read block 1026 len 16384
25801 165/300 Write block 4527 len 40960 data: 6 8 8 8 8 8 9 10 8 5
25802 166/300 Write block 3312 len 24576 data: 3 3 3 3 3 3
25803 167/300 Read block 925 len 28672
25804 168/300 Write block 3997 len 4096 data: 4
25805 169/300 Read block 3817 len 8192
25806 170/300 Read block 3377 len 12288
25807 171/300 Write block 2502 len 40960 data: 5 5 4 5 3 3 4 3 4 4
25808 172/300 Write block 274 len 4096 data: 3
25809 173/300 Read block 3050 len 20480
25810 174/300 Read block 4467 len 36864
25811 175/300 Read block 1489 len 28672
25812 176/300 Read block 3699 len 20480
25813 177/300 Write block 2005 len 32768 data: 8 8 6 8 6 5 6 7
25814 178/300 Flush
25815 179/300 Flush
25816 180/300 Flush
25817 181/300 Read block 1648 len 20480
25818 182/300 Read block 4923 len 12288
25819 183/300 Read block 4314 len 8192
25820 184/300 Flush
25821 185/300 Write block 2476 len 32768 data: 5 5 4 3 3 4 4 4
25822 186/300 Read block 2713 len 36864
25823 187/300 Write block 2549 len 28672 data: 7 6 4 5 6 5 5
25824 188/300 Read block 4942 len 32768
25825 189/300 Flush
25826 190/300 Flush
25827 191/300 Write block 2580 len 20480 data: 5 3 3 3 3
25828 192/300 Write block 1348 len 32768 data: 3 2 2 3 4 4 5 4
25829 193/300 Write block 3923 len 40960 data: 2 2 3 3 3 4 3 4 4 4
25830 194/300 Read block 2306 len 24576
25831 195/300 Read block 3618 len 40960
25832 196/300 Read block 2437 len 16384
25833 197/300 Write block 3047 len 16384 data: 3 3 3 3
25834 198/300 Read block 2292 len 36864
25835 199/300 Read block 3407 len 12288
25836 200/300 Flush
25837 201/300 Write block 2645 len 16384 data: 3 3 3 3
25838 202/300 Write block 2577 len 4096 data: 6
25839 203/300 Read block 1976 len 24576
25840 204/300 Write block 3635 len 20480 data: 3 3 3 3 5
25841 205/300 Flush
25842 206/300 Read block 643 len 28672
25843 207/300 Read block 1995 len 28672
25844 208/300 Write block 583 len 36864 data: 5 5 6 6 6 6 6 8 6
25845 209/300 Write block 968 len 40960 data: 10 9 9 8 8 7 6 4 3 3
25846 210/300 Read block 631 len 28672
25847 211/300 Flush
25848 212/300 Read block 973 len 12288
25849 213/300 Read block 625 len 36864
25850 214/300 Flush
25851 215/300 Read block 2957 len 36864
25852 216/300 Read block 2489 len 28672
25853 217/300 Read block 4506 len 8192
25854 218/300 Read block 2524 len 32768
25855 219/300 Flush
25856 220/300 Write block 2544 len 36864 data: 3 3 5 6 7 8 7 5 6
25857 221/300 Write block 271 len 8192 data: 3 3
25858 222/300 Read block 1767 len 8192
25859 223/300 Write block 4750 len 12288 data: 7 6 4
25860 224/300 Write block 1787 len 28672 data: 5 6 7 8 7 10 9
25861 225/300 Write block 4401 len 4096 data: 9
25862 226/300 Flush
25863 227/300 Write block 824 len 4096 data: 4
25864 228/300 Write block 3127 len 12288 data: 5 5 6
25865 229/300 Read block 409 len 40960
25866 230/300 Read block 2975 len 20480
25867 231/300 Flush
25868 232/300 Read block 4232 len 36864
25869 233/300 Write block 304 len 16384 data: 3 2 2 4
25870 234/300 Write block 4303 len 12288 data: 4 4 6
25871 235/300 Read block 4468 len 4096
25872 236/300 Read block 1459 len 12288
25873 237/300 Read block 2210 len 40960
25874 238/300 Write block 738 len 24576 data: 7 7 7 6 7 7
25875 239/300 Write block 224 len 8192 data: 5 6
25876 240/300 Flush
25877 241/300 Write block 2395 len 28672 data: 3 3 3 3 3 3 2
25878 242/300 Write block 4245 len 12288 data: 3 4 4
25879 243/300 Write block 2317 len 32768 data: 5 4 5 5 4 5 5 4
25880 244/300 Write block 4937 len 32768 data: 3 3 4 4 3 3 3 2
25881 245/300 Flush
25882 246/300 Read block 3731 len 8192
25883 247/300 Read block 3431 len 8192
25884 248/300 Read block 2036 len 4096
25885 249/300 Flush
25886 250/300 Read block 1169 len 16384
25887 251/300 Write block 1045 len 16384 data: 9 8 7 5
25888 252/300 Flush
25889 253/300 Flush
25890 254/300 Flush
25891 255/300 Read block 2642 len 20480
25892 256/300 Write block 2742 len 36864 data: 8 6 4 5 5 6 7 7 6
25893 257/300 Read block 3246 len 8192
25894 258/300 Write block 727 len 16384 data: 5 6 5 4
25895 259/300 Read block 4290 len 12288
25896 260/300 Read block 2517 len 8192
25897 261/300 Read block 4668 len 36864
25898 262/300 Write block 4038 len 16384 data: 3 3 3 4
25899 263/300 Flush
25900 264/300 Read block 1131 len 32768
25901 265/300 Write block 2526 len 28672 data: 6 5 6 5 4 4 5
25902 266/300 Read block 722 len 8192
25903 267/300 Read block 3903 len 24576
25904 268/300 Write block 825 len 8192 data: 5 7
25905 269/300 Read block 4240 len 4096
25906 270/300 Read block 4267 len 12288
25907 271/300 Read block 4266 len 40960
25908 272/300 Read block 502 len 28672
25909 273/300 Write block 2419 len 28672 data: 4 4 3 4 7 6 9
25910 274/300 Write block 2436 len 20480 data: 5 5 7 5 5
25911 275/300 Read block 737 len 40960
25912 276/300 Read block 2535 len 28672
25913 277/300 Write block 1545 len 28672 data: 5 6 6 6 5 4 6
25914 278/300 Read block 1134 len 20480
25915 279/300 Read block 1259 len 12288
25916 280/300 Read block 2681 len 24576
25917 281/300 Write block 3927 len 24576 data: 4 5 4 5 5 5
25918 282/300 Write block 1917 len 28672 data: 4 5 5 5 6 6 7
25919 283/300 Flush
25920 284/300 Write block 2591 len 40960 data: 5 5 4 3 4 4 4 4 4 5
25921 285/300 Write block 4716 len 24576 data: 7 5 5 6 5 3
25922 286/300 Read block 4677 len 8192
25923 287/300 Read block 2207 len 36864
25924 288/300 Write block 3067 len 24576 data: 3 3 3 3 3 3
25925 289/300 Write block 1309 len 28672 data: 4 4 4 3 4 4 5
25926 290/300 Read block 2810 len 36864
25927 291/300 Write block 1365 len 16384 data: 7 5 6 5
25928 292/300 Read block 3134 len 24576
25929 293/300 Flush
25930 294/300 Read block 4669 len 40960
25931 295/300 Read block 4693 len 20480
25932 296/300 Write block 4021 len 24576 data: 2 2 3 3 5 6
25933 297/300 Write block 3646 len 24576 data: 5 5 5 6 6 4
25934 298/300 Flush
25935 299/300 Read block 3947 len 40960
25936 300/300 Write block 4089 len 16384 data: 4 4 4 5
25937 301/300 Write block 3787 len 20480 data: 3 5 5 5 5
25938 Main task has work to do, go find it
25939 got dsc Start(2)
25940 start 2
25941 Jan 29 05:16:29.243 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/2, method: POST, req_id: 342ac360-02f2-4f23-9cf3-8dd115c3ea89, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
25942 Replay: started 2, returned:Ok(())
25943 ----------------------------------------------------------------
25944 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25945 States: Active Active Offline
25946 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25947 7643 Acked Read 9 Done Done Sent false
25948 7644 Acked Read 5 Done Done Sent false
25949 7645 Acked Write 1 Done Done Sent false
25950 7646 Acked Flush 0 Done Done Sent false
25951 7647 Acked Write 7 Done Done Sent false
25952 7648 Acked Read 7 Done Done Sent false
25953 7649 Acked Write 9 Done Done Sent false
25954 7650 Acked Write 8 Done Done Sent false
25955 7651 Acked Write 7 Done Done Sent false
25956 7652 Acked Write 8 Done Done Sent false
25957 7653 Acked Write 4 Done Done Sent false
25958 7654 Acked Read 10 Done Done Sent false
25959 7655 Acked Read 3 Done Done Sent false
25960 7656 Acked Write 3 Done Done Sent false
25961 7657 Acked Flush 0 Done Done Sent false
25962 7658 Acked Read 4 Done Done Sent false
25963 7659 Acked Read 4 Done Done Sent false
25964 7660 Acked Read 5 Done Done Sent false
25965 7661 Acked Write 7 Done Done Sent false
25966 7662 Acked Write 5 Done Done Sent false
25967 7663 Acked Flush 0 Done Done Sent false
25968 7664 Acked Read 9 Done Done Sent false
25969 7665 Acked Read 1 Done Done Sent false
25970 7666 Acked Read 4 Done Done Sent false
25971 7667 Acked Read 2 Done Done Sent false
25972 7668 Acked Read 5 Done Done Sent false
25973 7669 Acked Read 7 Done Done Sent false
25974 7670 Acked Read 9 Done Done Sent false
25975 7671 Acked Read 2 Done Done Sent false
25976 7672 Acked Read 5 Done Done Sent false
25977 7673 Acked Write 10 Done Done Sent false
25978 7674 Acked Write 7 Done Done Sent false
25979 7675 Acked Write 3 Done Done Sent false
25980 7676 Acked Write 6 Done Done Sent false
25981 7677 Acked Flush 0 Done Done Sent false
25982 7678 Acked Flush 0 Done Done Sent false
25983 7679 Acked Write 5 Done Done Sent false
25984 7680 Acked Write 9 Done Done Sent false
25985 7681 Acked Read 8 Done Done Sent false
25986 7682 Acked Read 9 Done Done Sent false
25987 7683 Acked Read 8 Done Done Sent false
25988 7684 Acked Write 10 Done Done Sent false
25989 7685 Acked Read 3 Done Done Sent false
25990 7686 Acked Write 8 Done Done Sent false
25991 7687 Acked Flush 0 Done Done Sent false
25992 7688 Acked Read 5 Done Done Sent false
25993 7689 Acked Flush 0 Done Done Sent false
25994 7690 Acked Read 1 Done Done Sent false
25995 7691 Acked Read 2 Done Done Sent false
25996 7692 Acked Flush 0 Done Done Sent false
25997 7693 Acked Read 10 Done Done Sent false
25998 7694 Acked Write 9 Done Done Sent false
25999 7695 Acked Read 2 Done Done Sent false
26000 7696 Acked Write 2 Done Done Sent false
26001 7697 Acked Read 7 Done Done Sent false
26002 7698 Acked Write 3 Done Done Sent false
26003 7699 Acked Write 7 Done Done Sent false
26004 7700 Acked Flush 0 Done Done Sent false
26005 7701 Acked Read 6 Done Done Sent false
26006 7702 Acked Write 9 Done Done Sent false
26007 7703 Acked Read 6 Done Done Sent false
26008 7704 Acked Write 1 Done Done Sent false
26009 7705 Acked Read 10 Done Done Sent false
26010 7706 Acked Flush 0 Done Done Sent false
26011 7707 Acked Write 7 Done Done Sent false
26012 7708 Acked Read 9 Done Done Sent false
26013 7709 Acked Flush 0 Done Done Sent false
26014 7710 Acked Write 9 Done Done Sent false
26015 7711 Acked Write 9 Done Done Sent false
26016 7712 Acked Flush 0 Done Done Sent false
26017 7713 Acked Flush 0 Done Done Sent false
26018 7714 Acked Flush 0 Done Done Sent false
26019 7715 Acked Read 2 Done Done Sent false
26020 7716 Acked Read 4 Done Done Sent false
26021 7717 Acked Read 10 Done Done Sent false
26022 7718 Acked Write 5 Done Done Sent false
26023 7719 Acked Read 6 Done Done Sent false
26024 7720 Acked Read 5 Done Done Sent false
26025 7721 Acked Read 2 Done Done Sent false
26026 7722 Acked Read 2 Done Done Sent false
26027 7723 Acked Write 7 Done Done Sent false
26028 7724 Acked Read 7 Done Done Sent false
26029 7725 Acked Write 8 Done Done Sent false
26030 7726 Acked Read 8 Done Done Sent false
26031 7727 Acked Read 2 Done Done Sent false
26032 7728 Acked Read 8 Done Done Sent false
26033 7729 Acked Write 2 Done Done Sent false
26034 7730 Acked Read 1 Done Done Sent false
26035 7731 Acked Read 9 Done Done Sent false
26036 7732 Acked Read 5 Done Done Sent false
26037 7733 Acked Write 3 Done Done Sent false
26038 7734 Acked Read 10 Done Done Sent false
26039 7735 Acked Read 3 Done Done Sent false
26040 7736 Acked Write 10 Done Done Sent false
26041 7737 Acked Flush 0 Done Done Sent false
26042 7738 Acked Read 7 Done Done Sent false
26043 7739 Acked Read 2 Done Done Sent false
26044 7740 Acked Write 1 Done Done Sent false
26045 7741 Acked Read 5 Done Done Sent false
26046 7742 Acked Read 10 Done Done Sent false
26047 7743 Acked Read 7 Done Done Sent false
26048 7744 Acked Write 4 Done Done Sent false
26049 7745 Acked Write 2 Done Done Sent false
26050 7746 Acked Read 2 Done Done Sent false
26051 7747 Acked Read 3 Done Done Sent false
26052 7748 Acked Read 10 Done Done Sent false
26053 7749 Acked Write 3 Done Done Sent false
26054 7750 Acked Write 10 Done Done Sent false
26055 7751 Acked Read 5 Done Done Sent false
26056 7752 Acked Write 7 Done Done Sent false
26057 7753 Acked Write 6 Done Done Sent false
26058 7754 Acked Read 6 Done Done Sent false
26059 7755 Acked Read 8 Done Done Sent false
26060 7756 Acked Read 2 Done Done Sent false
26061 7757 Acked Read 10 Done Done Sent false
26062 7758 Acked Write 4 Done Done Sent false
26063 7759 Acked Write 1 Done Done Sent false
26064 7760 Acked Write 6 Done Done Sent false
26065 7761 Acked Read 5 Done Done Sent false
26066 7762 Acked Read 1 Done Done Sent false
26067 7763 Acked Read 10 Done Done Sent false
26068 7764 Acked Read 9 Done Done Sent false
26069 7765 Acked Read 10 Done Done Sent false
26070 7766 Acked Write 5 Done Done Sent false
26071 7767 Acked Write 1 Done Done Sent false
26072 7768 Acked Read 7 Done Done Sent false
26073 7769 Acked Read 3 Done Done Sent false
26074 7770 Acked Read 1 Done Done Sent false
26075 7771 Acked Read 7 Done Done Sent false
26076 7772 Acked Read 8 Done Done Sent false
26077 7773 Acked Write 6 Done Done Sent false
26078 7774 Acked Flush 0 Done Done Sent false
26079 7775 Acked Read 2 Done Done Sent false
26080 7776 Acked Read 4 Done Done Sent false
26081 7777 Acked Flush 0 Done Done Sent false
26082 7778 Acked Write 2 Done Done Sent false
26083 7779 Acked Write 1 Done Done Sent false
26084 7780 Acked Flush 0 Done Done Sent false
26085 7781 Acked Read 4 Done Done Sent false
26086 7782 Acked Write 1 Done Done Sent false
26087 7783 Acked Write 2 Done Done Sent false
26088 7784 Acked Write 8 Done Done Sent false
26089 7785 Acked Read 10 Done Done Sent false
26090 7786 Acked Flush 0 Done Done Sent false
26091 7787 Acked Read 7 Done Done Sent false
26092 7788 Acked Flush 0 Done Done Sent false
26093 7789 Acked Read 8 Done Done Sent false
26094 7790 Acked Read 1 Done Done Sent false
26095 7791 Acked Read 8 Done Done Sent false
26096 7792 Acked Read 8 Done Done Sent false
26097 7793 Acked Read 3 Done Done Sent false
26098 7794 Acked Read 5 Done Done Sent false
26099 7795 Acked Read 1 Done Done Sent false
26100 7796 Acked Read 9 Done Done Sent false
26101 7797 Acked Read 5 Done Done Sent false
26102 7798 Acked Write 2 Done Done Sent false
26103 7799 Acked Read 6 Done Done Sent false
26104 7800 Acked Read 8 Done Done Sent false
26105 7801 Acked Flush 0 Done Done Sent false
26106 7802 Acked Read 2 Done Done Sent false
26107 7803 Acked Write 10 Done Done Sent false
26108 7804 Acked Flush 0 Done Done Sent false
26109 7805 Acked Write 2 Done Done Sent false
26110 7806 Acked Read 4 Done Done Sent false
26111 7807 Acked Write 10 Done Done Sent false
26112 7808 Acked Write 6 Done Done Sent false
26113 7809 Acked Read 7 Done Done Sent false
26114 7810 Acked Write 1 Done Done Sent false
26115 7811 Acked Read 2 Done Done Sent false
26116 7812 Acked Read 3 Done Done Sent false
26117 7813 Acked Write 10 Done Done Sent false
26118 7814 Acked Write 1 Done Done Sent false
26119 7815 Acked Read 5 Done Done Sent false
26120 7816 Acked Read 9 Done Done Sent false
26121 7817 Acked Read 7 Done Done Sent false
26122 7818 Acked Read 5 Done Done Sent false
26123 7819 Acked Write 8 Done Done Sent false
26124 7820 Acked Flush 0 Done Done Sent false
26125 7821 Acked Flush 0 Done Done Sent false
26126 7822 Acked Flush 0 Done Done Sent false
26127 7823 Acked Read 5 Done Done Sent false
26128 7824 Acked Read 3 Done Done Sent false
26129 7825 Acked Read 2 Done Done Sent false
26130 7826 Acked Flush 0 Done Done Sent false
26131 7827 Acked Write 8 Done Done Sent false
26132 7828 Acked Read 9 Done Done Sent false
26133 7829 Acked Write 7 Done Done Sent false
26134 7830 Acked Read 8 Done Done Sent false
26135 7831 Acked Flush 0 Done Done Sent false
26136 7832 Acked Flush 0 Done Done Sent false
26137 7833 Acked Write 5 Done Done Sent false
26138 7834 Acked Write 8 Done Done Sent false
26139 7835 Acked Write 10 Done Done Sent false
26140 7836 Acked Read 6 Done Done Sent false
26141 7837 Acked Read 10 Done Done Sent false
26142 7838 Acked Read 4 Done Done Sent false
26143 7839 Acked Write 4 Done Done Sent false
26144 7840 Acked Read 9 Done Done Sent false
26145 7841 Acked Read 3 Done Done Sent false
26146 7842 Acked Flush 0 Done Done Sent false
26147 7843 Acked Write 4 Done Done Sent false
26148 7844 Acked Write 1 Done Done Sent false
26149 7845 Acked Read 6 Done Done Sent false
26150 7846 Acked Write 5 Done Done Sent false
26151 7847 Acked Flush 0 Done Done Sent false
26152 7848 Acked Read 7 Done Done Sent false
26153 7849 Acked Read 7 Done Done Sent false
26154 7850 Acked Write 9 Done Done Sent false
26155 7851 Acked Write 10 Done Done Sent false
26156 7852 Acked Read 7 Done Done Sent false
26157 7853 Acked Flush 0 Done Done Sent false
26158 7854 Acked Read 3 Done Done Sent false
26159 7855 Acked Read 9 Done Done Sent false
26160 7856 Acked Flush 0 Done Done Sent false
26161 7857 Acked Read 9 Done Done Sent false
26162 7858 Acked Read 7 Done Done Sent false
26163 7859 Acked Read 2 Done Done Sent false
26164 7860 Acked Read 8 Done Done Sent false
26165 7861 Acked Flush 0 Done Done Sent false
26166 7862 Acked Write 9 Done Done Sent false
26167 7863 Acked Write 2 Done Done Sent false
26168 7864 Acked Read 2 Done Done Sent false
26169 7865 Acked Write 3 Done Done Sent false
26170 7866 Acked Write 7 Done Done Sent false
26171 7867 Acked Write 1 Done Done Sent false
26172 7868 Acked Flush 0 Done Done Sent false
26173 7869 Acked Write 1 Done Done Sent false
26174 7870 Acked Write 3 Done Done Sent false
26175 7871 Acked Read 10 Done Done Sent false
26176 7872 Acked Read 5 Done Done Sent false
26177 7873 Acked Flush 0 Done Done Sent false
26178 7874 Acked Read 9 Done Done Sent false
26179 7875 Acked Write 4 Done Done Sent false
26180 7876 Acked Write 3 Done Done Sent false
26181 7877 Acked Read 1 Done Done Sent false
26182 7878 Acked Read 3 Done Done Sent false
26183 7879 Acked Read 10 Done Done Sent false
26184 7880 Acked Write 6 Done Done Sent false
26185 7881 Acked Write 2 Done Done Sent false
26186 7882 Acked Flush 0 Done Done Sent false
26187 7883 Acked Write 7 Done Done Sent false
26188 7884 Acked Write 3 Done Done Sent false
26189 7885 Acked Write 8 Done Done Sent false
26190 7886 Acked Write 8 Done Done Sent false
26191 7887 Acked Flush 0 Done Done Sent false
26192 7888 Acked Read 2 Done Done Sent false
26193 7889 Acked Read 2 Done Done Sent false
26194 7890 Acked Read 1 Done Done Sent false
26195 7891 Acked Flush 0 Done Done Sent false
26196 7892 Acked Read 4 Done Done Sent false
26197 7893 Acked Write 4 Done Done Sent false
26198 7894 Acked Flush 0 Done Done Sent false
26199 7895 Acked Flush 0 Done Done Sent false
26200 7896 Acked Flush 0 Done Done Sent false
26201 7897 Acked Read 5 Done Done Sent false
26202 7898 Acked Write 9 Done Done Sent false
26203 7899 Acked Read 2 Done Done Sent false
26204 7900 Acked Write 4 Done Done Sent false
26205 7901 Acked Read 3 Done Done Sent false
26206 7902 Acked Read 2 Done Done Sent false
26207 7903 Acked Read 9 Done Done Sent false
26208 7904 Acked Write 4 Done Done Sent false
26209 7905 Acked Flush 0 Done Done Sent false
26210 7906 Acked Read 8 Done Done Sent false
26211 7907 Acked Write 7 Done Done Sent false
26212 7908 Acked Read 2 Done Done Sent false
26213 7909 Acked Read 6 Done Done Sent false
26214 7910 Acked Write 2 Done Done Sent false
26215 7911 Acked Read 1 Done Done Sent false
26216 7912 Acked Read 3 Done Done Sent false
26217 7913 Acked Read 10 Done Done Sent false
26218 7914 Acked Read 7 Done Done Sent false
26219 7915 Acked Write 7 Done Done Sent false
26220 7916 Acked Write 5 Done Done Sent false
26221 7917 Acked Read 10 Done Done Sent false
26222 7918 Acked Read 7 Done Done Sent false
26223 7919 Acked Write 7 Done Done Sent false
26224 7920 Acked Read 5 Done Done Sent false
26225 7921 Acked Read 3 Done Done Sent false
26226 7922 Acked Read 6 Done Done Sent false
26227 7923 Acked Write 6 Done Done Sent false
26228 7924 Acked Write 7 Done Done Sent false
26229 7925 Acked Flush 0 Done Done Sent false
26230 7926 Acked Write 10 Done Done Sent false
26231 7927 Acked Write 6 Done Done Sent false
26232 7928 Acked Read 2 Done Done Sent false
26233 7929 Acked Read 9 Done Done Sent false
26234 7930 Acked Write 6 Done Done Sent false
26235 7931 Acked Write 7 Done Done Sent false
26236 7932 Acked Read 9 Done Done Sent false
26237 7933 Acked Write 4 Done Done Sent false
26238 7934 Acked Read 6 Done Done Sent false
26239 7935 Acked Flush 0 Done Done Sent false
26240 7936 Acked Read 10 Done Done Sent false
26241 7937 Acked Read 5 Done Done Sent false
26242 7938 Acked Write 6 Done Done Sent false
26243 7939 Acked Write 6 Done Done Sent false
26244 7940 Acked Flush 0 Done Done Sent false
26245 7941 Acked Read 10 Done Done Sent false
26246 7942 Acked Write 4 Done Done Sent false
26247 7943 Acked Write 5 Done Done Sent false
26248 STATES DS:0 DS:1 DS:2 TOTAL
26249 Sent 0 0 301 301
26250 Done 301 301 0 602
26251 Skipped 0 0 0 0
26252 Error 0 0 0 0
26253 Last Flush: Some(JobId(7940)) Some(JobId(7940)) Some(JobId(7642))
26254 Downstairs last five completed: 7642 7641 7640 7639 7638
26255 Upstairs last five completed: 7943 7942 7941 7940 7939
26256 CLIENT: Up:0 ds:301 act:2
262572025-01-29T05:16:29.746ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
26258 [8830] got start action
26259 [8830] I am going to restart
26260 Starting downstairs at port 8830
26261 Make output file at "/tmp/dsc/downstairs-8830.txt"
26262 [8830][2] reports Starting
26263 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
26264 [8830][2] reports Running
262652025-01-29T05:16:31.498ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
262662025-01-29T05:16:31.998ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
262672025-01-29T05:16:32.500ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
26268 ----------------------------------------------------------------
26269 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26270 States: Active Active Offline
26271 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26272 7643 Acked Read 9 Done Done Sent false
26273 7644 Acked Read 5 Done Done Sent false
26274 7645 Acked Write 1 Done Done Sent false
26275 7646 Acked Flush 0 Done Done Sent false
26276 7647 Acked Write 7 Done Done Sent false
26277 7648 Acked Read 7 Done Done Sent false
26278 7649 Acked Write 9 Done Done Sent false
26279 7650 Acked Write 8 Done Done Sent false
26280 7651 Acked Write 7 Done Done Sent false
26281 7652 Acked Write 8 Done Done Sent false
26282 7653 Acked Write 4 Done Done Sent false
26283 7654 Acked Read 10 Done Done Sent false
26284 7655 Acked Read 3 Done Done Sent false
26285 7656 Acked Write 3 Done Done Sent false
26286 7657 Acked Flush 0 Done Done Sent false
26287 7658 Acked Read 4 Done Done Sent false
26288 7659 Acked Read 4 Done Done Sent false
26289 7660 Acked Read 5 Done Done Sent false
26290 7661 Acked Write 7 Done Done Sent false
26291 7662 Acked Write 5 Done Done Sent false
26292 7663 Acked Flush 0 Done Done Sent false
26293 7664 Acked Read 9 Done Done Sent false
26294 7665 Acked Read 1 Done Done Sent false
26295 7666 Acked Read 4 Done Done Sent false
26296 7667 Acked Read 2 Done Done Sent false
26297 7668 Acked Read 5 Done Done Sent false
26298 7669 Acked Read 7 Done Done Sent false
26299 7670 Acked Read 9 Done Done Sent false
26300 7671 Acked Read 2 Done Done Sent false
26301 7672 Acked Read 5 Done Done Sent false
26302 7673 Acked Write 10 Done Done Sent false
26303 7674 Acked Write 7 Done Done Sent false
26304 7675 Acked Write 3 Done Done Sent false
26305 7676 Acked Write 6 Done Done Sent false
26306 7677 Acked Flush 0 Done Done Sent false
26307 7678 Acked Flush 0 Done Done Sent false
26308 7679 Acked Write 5 Done Done Sent false
26309 7680 Acked Write 9 Done Done Sent false
26310 7681 Acked Read 8 Done Done Sent false
26311 7682 Acked Read 9 Done Done Sent false
26312 7683 Acked Read 8 Done Done Sent false
26313 7684 Acked Write 10 Done Done Sent false
26314 7685 Acked Read 3 Done Done Sent false
26315 7686 Acked Write 8 Done Done Sent false
26316 7687 Acked Flush 0 Done Done Sent false
26317 7688 Acked Read 5 Done Done Sent false
26318 7689 Acked Flush 0 Done Done Sent false
26319 7690 Acked Read 1 Done Done Sent false
26320 7691 Acked Read 2 Done Done Sent false
26321 7692 Acked Flush 0 Done Done Sent false
26322 7693 Acked Read 10 Done Done Sent false
26323 7694 Acked Write 9 Done Done Sent false
26324 7695 Acked Read 2 Done Done Sent false
26325 7696 Acked Write 2 Done Done Sent false
26326 7697 Acked Read 7 Done Done Sent false
26327 7698 Acked Write 3 Done Done Sent false
26328 7699 Acked Write 7 Done Done Sent false
26329 7700 Acked Flush 0 Done Done Sent false
26330 7701 Acked Read 6 Done Done Sent false
26331 7702 Acked Write 9 Done Done Sent false
26332 7703 Acked Read 6 Done Done Sent false
26333 7704 Acked Write 1 Done Done Sent false
26334 7705 Acked Read 10 Done Done Sent false
26335 7706 Acked Flush 0 Done Done Sent false
26336 7707 Acked Write 7 Done Done Sent false
26337 7708 Acked Read 9 Done Done Sent false
26338 7709 Acked Flush 0 Done Done Sent false
26339 7710 Acked Write 9 Done Done Sent false
26340 7711 Acked Write 9 Done Done Sent false
26341 7712 Acked Flush 0 Done Done Sent false
26342 7713 Acked Flush 0 Done Done Sent false
26343 7714 Acked Flush 0 Done Done Sent false
26344 7715 Acked Read 2 Done Done Sent false
26345 7716 Acked Read 4 Done Done Sent false
26346 7717 Acked Read 10 Done Done Sent false
26347 7718 Acked Write 5 Done Done Sent false
26348 7719 Acked Read 6 Done Done Sent false
26349 7720 Acked Read 5 Done Done Sent false
26350 7721 Acked Read 2 Done Done Sent false
26351 7722 Acked Read 2 Done Done Sent false
26352 7723 Acked Write 7 Done Done Sent false
26353 7724 Acked Read 7 Done Done Sent false
26354 7725 Acked Write 8 Done Done Sent false
26355 7726 Acked Read 8 Done Done Sent false
26356 7727 Acked Read 2 Done Done Sent false
26357 7728 Acked Read 8 Done Done Sent false
26358 7729 Acked Write 2 Done Done Sent false
26359 7730 Acked Read 1 Done Done Sent false
26360 7731 Acked Read 9 Done Done Sent false
26361 7732 Acked Read 5 Done Done Sent false
26362 7733 Acked Write 3 Done Done Sent false
26363 7734 Acked Read 10 Done Done Sent false
26364 7735 Acked Read 3 Done Done Sent false
26365 7736 Acked Write 10 Done Done Sent false
26366 7737 Acked Flush 0 Done Done Sent false
26367 7738 Acked Read 7 Done Done Sent false
26368 7739 Acked Read 2 Done Done Sent false
26369 7740 Acked Write 1 Done Done Sent false
26370 7741 Acked Read 5 Done Done Sent false
26371 7742 Acked Read 10 Done Done Sent false
26372 7743 Acked Read 7 Done Done Sent false
26373 7744 Acked Write 4 Done Done Sent false
26374 7745 Acked Write 2 Done Done Sent false
26375 7746 Acked Read 2 Done Done Sent false
26376 7747 Acked Read 3 Done Done Sent false
26377 7748 Acked Read 10 Done Done Sent false
26378 7749 Acked Write 3 Done Done Sent false
26379 7750 Acked Write 10 Done Done Sent false
26380 7751 Acked Read 5 Done Done Sent false
26381 7752 Acked Write 7 Done Done Sent false
26382 7753 Acked Write 6 Done Done Sent false
26383 7754 Acked Read 6 Done Done Sent false
26384 7755 Acked Read 8 Done Done Sent false
26385 7756 Acked Read 2 Done Done Sent false
26386 7757 Acked Read 10 Done Done Sent false
26387 7758 Acked Write 4 Done Done Sent false
26388 7759 Acked Write 1 Done Done Sent false
26389 7760 Acked Write 6 Done Done Sent false
26390 7761 Acked Read 5 Done Done Sent false
26391 7762 Acked Read 1 Done Done Sent false
26392 7763 Acked Read 10 Done Done Sent false
26393 7764 Acked Read 9 Done Done Sent false
26394 7765 Acked Read 10 Done Done Sent false
26395 7766 Acked Write 5 Done Done Sent false
26396 7767 Acked Write 1 Done Done Sent false
26397 7768 Acked Read 7 Done Done Sent false
26398 7769 Acked Read 3 Done Done Sent false
26399 7770 Acked Read 1 Done Done Sent false
26400 7771 Acked Read 7 Done Done Sent false
26401 7772 Acked Read 8 Done Done Sent false
26402 7773 Acked Write 6 Done Done Sent false
26403 7774 Acked Flush 0 Done Done Sent false
26404 7775 Acked Read 2 Done Done Sent false
26405 7776 Acked Read 4 Done Done Sent false
26406 7777 Acked Flush 0 Done Done Sent false
26407 7778 Acked Write 2 Done Done Sent false
26408 7779 Acked Write 1 Done Done Sent false
26409 7780 Acked Flush 0 Done Done Sent false
26410 7781 Acked Read 4 Done Done Sent false
26411 7782 Acked Write 1 Done Done Sent false
26412 7783 Acked Write 2 Done Done Sent false
26413 7784 Acked Write 8 Done Done Sent false
26414 7785 Acked Read 10 Done Done Sent false
26415 7786 Acked Flush 0 Done Done Sent false
26416 7787 Acked Read 7 Done Done Sent false
26417 7788 Acked Flush 0 Done Done Sent false
26418 7789 Acked Read 8 Done Done Sent false
26419 7790 Acked Read 1 Done Done Sent false
26420 7791 Acked Read 8 Done Done Sent false
26421 7792 Acked Read 8 Done Done Sent false
26422 7793 Acked Read 3 Done Done Sent false
26423 7794 Acked Read 5 Done Done Sent false
26424 7795 Acked Read 1 Done Done Sent false
26425 7796 Acked Read 9 Done Done Sent false
26426 7797 Acked Read 5 Done Done Sent false
26427 7798 Acked Write 2 Done Done Sent false
26428 7799 Acked Read 6 Done Done Sent false
26429 7800 Acked Read 8 Done Done Sent false
26430 7801 Acked Flush 0 Done Done Sent false
26431 7802 Acked Read 2 Done Done Sent false
26432 7803 Acked Write 10 Done Done Sent false
26433 7804 Acked Flush 0 Done Done Sent false
26434 7805 Acked Write 2 Done Done Sent false
26435 7806 Acked Read 4 Done Done Sent false
26436 7807 Acked Write 10 Done Done Sent false
26437 7808 Acked Write 6 Done Done Sent false
26438 7809 Acked Read 7 Done Done Sent false
26439 7810 Acked Write 1 Done Done Sent false
26440 7811 Acked Read 2 Done Done Sent false
26441 7812 Acked Read 3 Done Done Sent false
26442 7813 Acked Write 10 Done Done Sent false
26443 7814 Acked Write 1 Done Done Sent false
26444 7815 Acked Read 5 Done Done Sent false
26445 7816 Acked Read 9 Done Done Sent false
26446 7817 Acked Read 7 Done Done Sent false
26447 7818 Acked Read 5 Done Done Sent false
26448 7819 Acked Write 8 Done Done Sent false
26449 7820 Acked Flush 0 Done Done Sent false
26450 7821 Acked Flush 0 Done Done Sent false
26451 7822 Acked Flush 0 Done Done Sent false
26452 7823 Acked Read 5 Done Done Sent false
26453 7824 Acked Read 3 Done Done Sent false
26454 7825 Acked Read 2 Done Done Sent false
26455 7826 Acked Flush 0 Done Done Sent false
26456 7827 Acked Write 8 Done Done Sent false
26457 7828 Acked Read 9 Done Done Sent false
26458 7829 Acked Write 7 Done Done Sent false
26459 7830 Acked Read 8 Done Done Sent false
26460 7831 Acked Flush 0 Done Done Sent false
26461 7832 Acked Flush 0 Done Done Sent false
26462 7833 Acked Write 5 Done Done Sent false
26463 7834 Acked Write 8 Done Done Sent false
26464 7835 Acked Write 10 Done Done Sent false
26465 7836 Acked Read 6 Done Done Sent false
26466 7837 Acked Read 10 Done Done Sent false
26467 7838 Acked Read 4 Done Done Sent false
26468 7839 Acked Write 4 Done Done Sent false
26469 7840 Acked Read 9 Done Done Sent false
26470 7841 Acked Read 3 Done Done Sent false
26471 7842 Acked Flush 0 Done Done Sent false
26472 7843 Acked Write 4 Done Done Sent false
26473 7844 Acked Write 1 Done Done Sent false
26474 7845 Acked Read 6 Done Done Sent false
26475 7846 Acked Write 5 Done Done Sent false
26476 7847 Acked Flush 0 Done Done Sent false
26477 7848 Acked Read 7 Done Done Sent false
26478 7849 Acked Read 7 Done Done Sent false
26479 7850 Acked Write 9 Done Done Sent false
26480 7851 Acked Write 10 Done Done Sent false
26481 7852 Acked Read 7 Done Done Sent false
26482 7853 Acked Flush 0 Done Done Sent false
26483 7854 Acked Read 3 Done Done Sent false
26484 7855 Acked Read 9 Done Done Sent false
26485 7856 Acked Flush 0 Done Done Sent false
26486 7857 Acked Read 9 Done Done Sent false
26487 7858 Acked Read 7 Done Done Sent false
26488 7859 Acked Read 2 Done Done Sent false
26489 7860 Acked Read 8 Done Done Sent false
26490 7861 Acked Flush 0 Done Done Sent false
26491 7862 Acked Write 9 Done Done Sent false
26492 7863 Acked Write 2 Done Done Sent false
26493 7864 Acked Read 2 Done Done Sent false
26494 7865 Acked Write 3 Done Done Sent false
26495 7866 Acked Write 7 Done Done Sent false
26496 7867 Acked Write 1 Done Done Sent false
26497 7868 Acked Flush 0 Done Done Sent false
26498 7869 Acked Write 1 Done Done Sent false
26499 7870 Acked Write 3 Done Done Sent false
26500 7871 Acked Read 10 Done Done Sent false
26501 7872 Acked Read 5 Done Done Sent false
26502 7873 Acked Flush 0 Done Done Sent false
26503 7874 Acked Read 9 Done Done Sent false
26504 7875 Acked Write 4 Done Done Sent false
26505 7876 Acked Write 3 Done Done Sent false
26506 7877 Acked Read 1 Done Done Sent false
26507 7878 Acked Read 3 Done Done Sent false
26508 7879 Acked Read 10 Done Done Sent false
26509 7880 Acked Write 6 Done Done Sent false
26510 7881 Acked Write 2 Done Done Sent false
26511 7882 Acked Flush 0 Done Done Sent false
26512 7883 Acked Write 7 Done Done Sent false
26513 7884 Acked Write 3 Done Done Sent false
26514 7885 Acked Write 8 Done Done Sent false
26515 7886 Acked Write 8 Done Done Sent false
26516 7887 Acked Flush 0 Done Done Sent false
26517 7888 Acked Read 2 Done Done Sent false
26518 7889 Acked Read 2 Done Done Sent false
26519 7890 Acked Read 1 Done Done Sent false
26520 7891 Acked Flush 0 Done Done Sent false
26521 7892 Acked Read 4 Done Done Sent false
26522 7893 Acked Write 4 Done Done Sent false
26523 7894 Acked Flush 0 Done Done Sent false
26524 7895 Acked Flush 0 Done Done Sent false
26525 7896 Acked Flush 0 Done Done Sent false
26526 7897 Acked Read 5 Done Done Sent false
26527 7898 Acked Write 9 Done Done Sent false
26528 7899 Acked Read 2 Done Done Sent false
26529 7900 Acked Write 4 Done Done Sent false
26530 7901 Acked Read 3 Done Done Sent false
26531 7902 Acked Read 2 Done Done Sent false
26532 7903 Acked Read 9 Done Done Sent false
26533 7904 Acked Write 4 Done Done Sent false
26534 7905 Acked Flush 0 Done Done Sent false
26535 7906 Acked Read 8 Done Done Sent false
26536 7907 Acked Write 7 Done Done Sent false
26537 7908 Acked Read 2 Done Done Sent false
26538 7909 Acked Read 6 Done Done Sent false
26539 7910 Acked Write 2 Done Done Sent false
26540 7911 Acked Read 1 Done Done Sent false
26541 7912 Acked Read 3 Done Done Sent false
26542 7913 Acked Read 10 Done Done Sent false
26543 7914 Acked Read 7 Done Done Sent false
26544 7915 Acked Write 7 Done Done Sent false
26545 7916 Acked Write 5 Done Done Sent false
26546 7917 Acked Read 10 Done Done Sent false
26547 7918 Acked Read 7 Done Done Sent false
26548 7919 Acked Write 7 Done Done Sent false
26549 7920 Acked Read 5 Done Done Sent false
26550 7921 Acked Read 3 Done Done Sent false
26551 7922 Acked Read 6 Done Done Sent false
26552 7923 Acked Write 6 Done Done Sent false
26553 7924 Acked Write 7 Done Done Sent false
26554 7925 Acked Flush 0 Done Done Sent false
26555 7926 Acked Write 10 Done Done Sent false
26556 7927 Acked Write 6 Done Done Sent false
26557 7928 Acked Read 2 Done Done Sent false
26558 7929 Acked Read 9 Done Done Sent false
26559 7930 Acked Write 6 Done Done Sent false
26560 7931 Acked Write 7 Done Done Sent false
26561 7932 Acked Read 9 Done Done Sent false
26562 7933 Acked Write 4 Done Done Sent false
26563 7934 Acked Read 6 Done Done Sent false
26564 7935 Acked Flush 0 Done Done Sent false
26565 7936 Acked Read 10 Done Done Sent false
26566 7937 Acked Read 5 Done Done Sent false
26567 7938 Acked Write 6 Done Done Sent false
26568 7939 Acked Write 6 Done Done Sent false
26569 7940 Acked Flush 0 Done Done Sent false
26570 7941 Acked Read 10 Done Done Sent false
26571 7942 Acked Write 4 Done Done Sent false
26572 7943 Acked Write 5 Done Done Sent false
26573 STATES DS:0 DS:1 DS:2 TOTAL
26574 Sent 0 0 301 301
26575 Done 301 301 0 602
26576 Skipped 0 0 0 0
26577 Error 0 0 0 0
26578 Last Flush: Some(JobId(7940)) Some(JobId(7940)) Some(JobId(7642))
26579 Downstairs last five completed: 7642 7641 7640 7639 7638
26580 Upstairs last five completed: 7943 7942 7941 7940 7939
26581 CLIENT: Up:0 ds:301 act:2
265822025-01-29T05:16:33.748ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265832025-01-29T05:16:34.502ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265842025-01-29T05:16:34.999ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265852025-01-29T05:16:34.999ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38154) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265862025-01-29T05:16:35.000ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265872025-01-29T05:16:35.000ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265882025-01-29T05:16:35.000ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265892025-01-29T05:16:35.000ZINFOcrucible: send last flush ID to this DS: Some(JobId(7642)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265902025-01-29T05:16:35.001ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265912025-01-29T05:16:35.001ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7642)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
265922025-01-29T05:16:35.001ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7642)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
26593 ----------------------------------------------------------------
26594 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26595 Downstairs last five completed: 7944 7943 7942 7941 7940
26596 Upstairs last five completed: 7944 7943 7942 7941 7940
26597 CLIENT: Up:0 ds:0 act:3
26598 Replay: All jobs finished
26599 Main task has work to do, go find it
26600 got dsc Stop(0)
26601 stop 0
26602 [8810] Got stop action so:false kr:false
26603 Jan 29 05:16:37.249 INFO request completed, latency_us: 356, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 16bf5aa5-d9fd-4a1c-8efb-c9792e85a3c7, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
266042025-01-29T05:16:37.250ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
266052025-01-29T05:16:37.250ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
266062025-01-29T05:16:37.250ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
26607 Jan 29 05:16:37.250 INFO request completed, latency_us: 175, response_code: 200, uri: /state/cid/0, method: GET, req_id: ca28cb10-1ecb-4d94-9803-44c19c86bae5, remote_addr: 127.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:16:37.250218396Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
26608 0.1:46995, local_addr: 127.0.0.1:9998
26609 [8810] Exited with: ExitStatus(unix_wait_status(9))
26610 [8810][0] reports Exit
26611 Jan 29 05:16:41.252 INFO request completed, latency_us: 328, response_code: 200, uri: /state/cid/0, method: GET, req_id: 09d01b91-bd8d-4651-86eb-1240205ab177, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
26612 001/300 Read block 384 len 20480
26613 002/300 Read block 2102 len 4096
26614 003/300 Write block 3806 len 16384 data: 8 8 8 6
26615 004/300 Write block 36 len 24576 data: 6 5 5 5 6 6
26616 005/300 Write block 3028 len 24576 data: 4 5 5 4 6 7
26617 006/300 Write block 4457 len 32768 data: 8 4 4 5 5 5 4 4
26618 007/300 Read block 2933 len 36864
26619 008/300 Write block 3393 len 20480 data: 4 4 3 4 4
26620 009/300 Read block 3016 len 28672
26621 010/300 Flush
26622 011/300 Write block 2647 len 16384 data: 4 4 3 4
26623 012/300 Flush
26624 013/300 Flush
26625 014/300 Flush
26626 015/300 Write block 1402 len 40960 data: 6 6 6 5 5 5 4 4 6 5
26627 016/300 Write block 588 len 20480 data: 7 7 9 7 6
26628 017/300 Read block 443 len 24576
26629 018/300 Write block 1154 len 16384 data: 3 3 3 4
26630 019/300 Write block 4502 len 36864 data: 6 6 6 6 5 5 4 4 4
26631 020/300 Read block 4757 len 8192
26632 021/300 Read block 930 len 40960
26633 022/300 Read block 4492 len 16384
26634 023/300 Flush
26635 024/300 Write block 4126 len 32768 data: 5 4 4 4 4 3 4 4
26636 025/300 Read block 4881 len 28672
26637 026/300 Write block 2107 len 16384 data: 6 6 5 7
26638 027/300 Read block 2699 len 32768
26639 028/300 Read block 3457 len 24576
26640 029/300 Read block 835 len 20480
26641 030/300 Write block 592 len 20480 data: 7 6 7 7 5
26642 031/300 Read block 452 len 12288
26643 032/300 Read block 3479 len 40960
26644 033/300 Read block 1828 len 12288
26645 034/300 Read block 4795 len 4096
26646 035/300 Write block 3641 len 32768 data: 6 9 9 7 6 6 6 6
26647 036/300 Read block 685 len 32768
26648 037/300 Read block 4731 len 20480
26649 038/300 Read block 4049 len 24576
26650 039/300 Read block 2548 len 32768
26651 040/300 Read block 2486 len 4096
26652 041/300 Read block 2564 len 24576
26653 042/300 Write block 1712 len 4096 data: 6
26654 043/300 Read block 3169 len 20480
26655 044/300 Read block 4890 len 16384
26656 045/300 Read block 1662 len 12288
26657 046/300 Read block 2209 len 24576
26658 047/300 Write block 27 len 12288 data: 4 4 5
26659 048/300 Write block 2426 len 28672 data: 8 6 5 4 4 5 4
26660 049/300 Write block 3053 len 28672 data: 3 3 2 3 4 4 4
26661 050/300 Read block 3713 len 40960
26662 051/300 Read block 718 len 4096
26663 052/300 Write block 3847 len 4096 data: 4
26664 053/300 Write block 3271 len 12288 data: 3 3 3
26665 054/300 Write block 579 len 20480 data: 6 6 5 7 6
26666 055/300 Write block 3606 len 36864 data: 4 3 6 8 7 7 7 7 7
26667 056/300 Write block 2178 len 40960 data: 2 3 3 3 3 3 3 4 4 4
26668 057/300 Read block 4567 len 8192
26669 058/300 Write block 1078 len 40960 data: 5 5 5 5 4 6 3 3 3 5
26670 059/300 Write block 4594 len 32768 data: 4 3 3 5 5 7 9 9
26671 060/300 Read block 4847 len 20480
26672 061/300 Read block 4974 len 40960
26673 062/300 Write block 4216 len 24576 data: 3 4 4 6 6 5
26674 063/300 Read block 2335 len 20480
26675 064/300 Flush
26676 065/300 Read block 1283 len 12288
26677 066/300 Write block 448 len 12288 data: 5 4 4
26678 067/300 Write block 2355 len 8192 data: 4 4
26679 068/300 Read block 3349 len 4096
26680 069/300 Read block 3117 len 40960
26681 070/300 Write block 4337 len 8192 data: 3 3
26682 071/300 Read block 1295 len 4096
26683 072/300 Write block 4988 len 4096 data: 6
26684 073/300 Read block 2886 len 12288
26685 074/300 Read block 2031 len 12288
26686 075/300 Read block 1492 len 40960
26687 076/300 Write block 194 len 8192 data: 6 6
26688 077/300 Flush
26689 078/300 Flush
26690 079/300 Write block 245 len 24576 data: 8 7 5 4 4 4
26691 080/300 Read block 2774 len 16384
26692 081/300 Write block 2125 len 24576 data: 4 4 3 2 2 2
26693 082/300 Read block 1218 len 36864
26694 083/300 Read block 2660 len 32768
26695 084/300 Write block 3699 len 20480 data: 5 5 5 6 4
26696 085/300 Read block 4490 len 28672
26697 086/300 Read block 3390 len 16384
26698 087/300 Write block 4111 len 16384 data: 5 5 4 3
26699 088/300 Read block 1970 len 24576
26700 089/300 Write block 979 len 16384 data: 5 4 3 3
26701 090/300 Read block 4405 len 12288
26702 091/300 Read block 1610 len 16384
26703 092/300 Read block 2150 len 40960
26704 093/300 Flush
26705 094/300 Read block 1491 len 8192
26706 095/300 Read block 4153 len 36864
26707 096/300 Read block 4320 len 32768
26708 097/300 Read block 2203 len 40960
26709 098/300 Read block 639 len 28672
26710 099/300 Write block 4289 len 12288 data: 5 6 6
26711 100/300 Write block 215 len 20480 data: 3 4 5 4 6
26712 101/300 Write block 3981 len 12288 data: 6 6 7
26713 102/300 Read block 1296 len 24576
26714 103/300 Read block 3100 len 36864
26715 104/300 Read block 2551 len 32768
26716 105/300 Read block 3037 len 12288
26717 106/300 Flush
26718 107/300 Write block 238 len 12288 data: 10 10 9
26719 108/300 Read block 3112 len 28672
26720 109/300 Write block 3369 len 12288 data: 4 5 5
26721 110/300 Read block 558 len 32768
26722 111/300 Read block 2239 len 32768
26723 112/300 Read block 1263 len 32768
26724 113/300 Write block 2488 len 24576 data: 4 5 5 5 6 5
26725 114/300 Read block 1859 len 32768
26726 115/300 Read block 73 len 20480
26727 116/300 Read block 4645 len 36864
26728 117/300 Write block 1206 len 8192 data: 2 2
26729 118/300 Write block 3357 len 24576 data: 4 4 4 5 5 5
26730 119/300 Write block 3575 len 20480 data: 10 12 11 11 7
26731 120/300 Read block 1040 len 28672
26732 121/300 Read block 2430 len 8192
26733 122/300 Write block 2888 len 4096 data: 8
26734 123/300 Write block 2166 len 4096 data: 4
26735 124/300 Read block 348 len 36864
26736 125/300 Read block 1584 len 20480
26737 126/300 Flush
26738 127/300 Read block 2322 len 40960
26739 128/300 Write block 3035 len 8192 data: 6 6
26740 129/300 Read block 1817 len 4096
26741 130/300 Write block 211 len 24576 data: 2 2 3 3 4 5
26742 131/300 Read block 4075 len 20480
26743 132/300 Write block 4161 len 8192 data: 4 4
26744 133/300 Read block 1355 len 40960
26745 134/300 Write block 1374 len 28672 data: 6 5 4 3 3 2 2
26746 135/300 Read block 2584 len 4096
26747 136/300 Write block 4111 len 28672 data: 6 6 5 4 3 4 6
26748 137/300 Write block 157 len 4096 data: 3
26749 138/300 Flush
26750 139/300 Read block 1165 len 40960
26751 140/300 Read block 4903 len 20480
26752 141/300 Read block 2188 len 28672
26753 142/300 Read block 2832 len 8192
26754 143/300 Write block 2790 len 8192 data: 6 5
26755 144/300 Write block 4070 len 40960 data: 4 4 5 6 7 6 5 4 5 2
26756 145/300 Write block 61 len 24576 data: 6 5 4 4 4 5
26757 146/300 Write block 2395 len 8192 data: 4 4
26758 147/300 Read block 1069 len 32768
26759 148/300 Write block 4005 len 40960 data: 8 8 7 6 7 6 6 5 5 3
26760 149/300 Write block 1304 len 32768 data: 5 4 4 4 3 5 5 5
26761 150/300 Write block 1750 len 4096 data: 5
26762 151/300 Read block 4325 len 12288
26763 152/300 Write block 1529 len 40960 data: 4 4 4 5 5 4 3 4 4 6
26764 153/300 Flush
26765 154/300 Read block 4959 len 12288
26766 155/300 Read block 1799 len 32768
26767 156/300 Read block 1542 len 24576
26768 157/300 Read block 4382 len 12288
26769 158/300 Read block 3489 len 28672
26770 159/300 Write block 4724 len 40960 data: 4 3 2 3 3 5 6 5 5 6
26771 160/300 Write block 2114 len 20480 data: 6 6 6 5 3
26772 161/300 Write block 3804 len 16384 data: 8 9 9 9
26773 162/300 Read block 2065 len 28672
26774 163/300 Read block 3394 len 8192
26775 164/300 Read block 248 len 16384
26776 165/300 Read block 1587 len 4096
26777 166/300 Write block 3827 len 36864 data: 6 4 4 5 6 6 5 6 7
26778 167/300 Read block 3851 len 36864
26779 168/300 Write block 889 len 24576 data: 2 5 6 6 6 6
26780 169/300 Read block 2379 len 40960
26781 170/300 Read block 3897 len 16384
26782 171/300 Read block 4376 len 8192
26783 172/300 Read block 2005 len 24576
26784 173/300 Flush
26785 174/300 Flush
26786 175/300 Write block 4980 len 24576 data: 4 4 3 3 4 3
26787 176/300 Write block 3526 len 28672 data: 6 6 6 7 3 6 6
26788 177/300 Write block 3682 len 8192 data: 7 6
26789 178/300 Flush
26790 179/300 Write block 2937 len 12288 data: 4 4 5
26791 180/300 Read block 74 len 40960
26792 181/300 Read block 4567 len 8192
26793 182/300 Write block 4311 len 20480 data: 4 4 6 6 4
26794 183/300 Flush
26795 184/300 Read block 4082 len 40960
26796 185/300 Read block 3502 len 28672
26797 186/300 Write block 1675 len 16384 data: 6 5 4 4
26798 187/300 Write block 4351 len 12288 data: 4 4 5
26799 188/300 Write block 2981 len 16384 data: 4 4 5 6
26800 189/300 Write block 4109 len 28672 data: 3 5 7 7 6 5 4
26801 190/300 Read block 3693 len 28672
26802 191/300 Read block 1613 len 20480
26803 192/300 Flush
26804 193/300 Write block 4048 len 36864 data: 6 5 7 6 5 4 3 3 4
26805 194/300 Flush
26806 195/300 Read block 2334 len 20480
26807 196/300 Read block 4650 len 12288
26808 197/300 Flush
26809 198/300 Read block 4164 len 8192
26810 199/300 Flush
26811 200/300 Read block 2162 len 16384
26812 201/300 Read block 999 len 36864
26813 202/300 Read block 1966 len 40960
26814 203/300 Read block 1858 len 16384
26815 204/300 Write block 2031 len 36864 data: 5 5 5 6 5 4 5 5 5
26816 205/300 Write block 4021 len 28672 data: 3 3 4 4 6 7 6
26817 206/300 Write block 1130 len 32768 data: 4 3 3 3 3 3 3 3
26818 207/300 Write block 2747 len 4096 data: 7
26819 208/300 Write block 783 len 4096 data: 5
26820 209/300 Read block 2459 len 28672
26821 210/300 Write block 3202 len 8192 data: 5 5
26822 211/300 Write block 4751 len 32768 data: 7 5 3 4 4 4 5 5
26823 212/300 Read block 3729 len 20480
26824 213/300 Read block 1932 len 16384
26825 214/300 Write block 3794 len 4096 data: 4
26826 215/300 Read block 2028 len 36864
26827 216/300 Read block 3341 len 24576
26828 217/300 Read block 912 len 8192
26829 218/300 Read block 3200 len 40960
26830 219/300 Read block 2412 len 28672
26831 220/300 Read block 4889 len 16384
26832 221/300 Read block 3128 len 24576
26833 222/300 Write block 817 len 36864 data: 7 7 7 6 4 3 4 5 6
26834 223/300 Read block 2654 len 28672
26835 224/300 Write block 4619 len 16384 data: 5 4 4 4
26836 225/300 Write block 511 len 12288 data: 6 7 6
26837 226/300 Read block 1486 len 20480
26838 227/300 Write block 4016 len 40960 data: 3 2 4 4 2 4 4 5 5 7
26839 228/300 Read block 3399 len 8192
26840 229/300 Write block 4779 len 4096 data: 8
26841 230/300 Read block 173 len 36864
26842 231/300 Flush
26843 232/300 Read block 3136 len 24576
26844 233/300 Read block 2177 len 16384
26845 234/300 Read block 2642 len 20480
26846 235/300 Read block 4085 len 28672
26847 236/300 Read block 3397 len 20480
26848 237/300 Write block 1083 len 28672 data: 7 4 4 4 6 4 4
26849 238/300 Read block 4391 len 4096
26850 239/300 Read block 4434 len 24576
26851 240/300 Read block 3518 len 36864
26852 241/300 Write block 1932 len 4096 data: 3
26853 242/300 Write block 1134 len 40960 data: 4 4 4 4 4 4 4 4 4 5
26854 243/300 Read block 3522 len 28672
26855 244/300 Write block 3843 len 24576 data: 4 4 4 4 5 4
26856 245/300 Write block 4033 len 16384 data: 4 4 3 3
26857 246/300 Read block 804 len 20480
26858 247/300 Read block 3264 len 8192
26859 248/300 Write block 1090 len 40960 data: 5 5 6 7 6 5 5 5 4 5
26860 249/300 Read block 1465 len 24576
26861 250/300 Write block 1470 len 40960 data: 3 3 3 3 4 5 4 3 3 4
26862 251/300 Write block 346 len 4096 data: 6
26863 252/300 Read block 3791 len 4096
26864 253/300 Read block 1703 len 16384
26865 254/300 Write block 2261 len 28672 data: 6 7 6 5 4 5 4
26866 255/300 Read block 1536 len 16384
26867 256/300 Read block 1460 len 20480
26868 257/300 Write block 4220 len 36864 data: 7 6 5 5 6 6 5 6 5
26869 258/300 Write block 4647 len 20480 data: 6 6 4 5 5
26870 259/300 Write block 2478 len 24576 data: 5 4 4 5 5 5
26871 260/300 Write block 6 len 8192 data: 5 5
26872 261/300 Write block 1270 len 28672 data: 5 5 5 5 4 2 2
26873 262/300 Read block 1047 len 40960
26874 263/300 Read block 2791 len 24576
26875 264/300 Write block 604 len 20480 data: 3 4 3 3 4
26876 265/300 Read block 1562 len 40960
26877 266/300 Read block 3840 len 24576
26878 267/300 Read block 814 len 32768
26879 268/300 Write block 992 len 40960 data: 2 3 3 3 4 5 5 4 5 4
26880 269/300 Write block 2815 len 36864 data: 5 4 3 5 5 5 5 6 5
26881 270/300 Read block 1372 len 24576
26882 271/300 Read block 904 len 16384
26883 272/300 Write block 4921 len 12288 data: 5 4 4
26884 273/300 Write block 1005 len 28672 data: 5 3 2 5 5 5 6
26885 274/300 Read block 2253 len 28672
26886 275/300 Write block 716 len 40960 data: 5 5 5 4 4 6 6 6 6 6
26887 276/300 Flush
26888 277/300 Read block 4181 len 16384
26889 278/300 Read block 3413 len 28672
26890 279/300 Read block 4485 len 24576
26891 280/300 Read block 3831 len 40960
26892 281/300 Read block 4684 len 28672
26893 282/300 Write block 2187 len 16384 data: 5 3 3 3
26894 283/300 Read block 3618 len 16384
26895 284/300 Write block 1379 len 12288 data: 3 3 3
26896 285/300 Read block 1722 len 16384
26897 286/300 Write block 1420 len 8192 data: 5 5
26898 287/300 Read block 1967 len 16384
26899 288/300 Flush
26900 289/300 Flush
26901 290/300 Flush
26902 291/300 Write block 3780 len 40960 data: 6 3 3 2 2 2 3 4 6 6
26903 292/300 Flush
26904 293/300 Write block 3902 len 16384 data: 5 6 8 8
26905 294/300 Write block 1275 len 8192 data: 3 3
26906 295/300 Write block 611 len 36864 data: 4 5 4 4 3 3 3 2 4
26907 296/300 Flush
26908 297/300 Write block 569 len 36864 data: 4 4 4 5 6 6 7 6 9
26909 298/300 Write block 4917 len 8192 data: 4 4
26910 299/300 Flush
26911 300/300 Read block 559 len 32768
26912 301/300 Write block 2290 len 4096 data: 5
26913 Main task has work to do, go find it
26914 got dsc Start(0)
26915 start 0
26916 Jan 29 05:16:41.496 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/0, method: POST, req_id: f31fa1cf-7b5e-46e4-a4c1-13c06215ce9b, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
26917 Replay: started 0, returned:Ok(())
26918 ----------------------------------------------------------------
26919 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26920 States: Offline Active Active
26921 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26922 7945 Acked Read 5 Sent Done Done false
26923 7946 Acked Read 1 Sent Done Done false
26924 7947 Acked Write 4 Sent Done Done false
26925 7948 Acked Write 6 Sent Done Done false
26926 7949 Acked Write 6 Sent Done Done false
26927 7950 Acked Write 8 Sent Done Done false
26928 7951 Acked Read 9 Sent Done Done false
26929 7952 Acked Write 5 Sent Done Done false
26930 7953 Acked Read 7 Sent Done Done false
26931 7954 Acked Flush 0 Sent Done Done false
26932 7955 Acked Write 4 Sent Done Done false
26933 7956 Acked Flush 0 Sent Done Done false
26934 7957 Acked Flush 0 Sent Done Done false
26935 7958 Acked Flush 0 Sent Done Done false
26936 7959 Acked Write 10 Sent Done Done false
26937 7960 Acked Write 5 Sent Done Done false
26938 7961 Acked Read 6 Sent Done Done false
26939 7962 Acked Write 4 Sent Done Done false
26940 7963 Acked Write 9 Sent Done Done false
26941 7964 Acked Read 2 Sent Done Done false
26942 7965 Acked Read 10 Sent Done Done false
26943 7966 Acked Read 4 Sent Done Done false
26944 7967 Acked Flush 0 Sent Done Done false
26945 7968 Acked Write 8 Sent Done Done false
26946 7969 Acked Read 7 Sent Done Done false
26947 7970 Acked Write 4 Sent Done Done false
26948 7971 Acked Read 8 Sent Done Done false
26949 7972 Acked Read 6 Sent Done Done false
26950 7973 Acked Read 5 Sent Done Done false
26951 7974 Acked Write 5 Sent Done Done false
26952 7975 Acked Read 3 Sent Done Done false
26953 7976 Acked Read 10 Sent Done Done false
26954 7977 Acked Read 3 Sent Done Done false
26955 7978 Acked Read 1 Sent Done Done false
26956 7979 Acked Write 8 Sent Done Done false
26957 7980 Acked Read 8 Sent Done Done false
26958 7981 Acked Read 5 Sent Done Done false
26959 7982 Acked Read 6 Sent Done Done false
26960 7983 Acked Read 8 Sent Done Done false
26961 7984 Acked Read 1 Sent Done Done false
26962 7985 Acked Read 6 Sent Done Done false
26963 7986 Acked Write 1 Sent Done Done false
26964 7987 Acked Read 5 Sent Done Done false
26965 7988 Acked Read 4 Sent Done Done false
26966 7989 Acked Read 3 Sent Done Done false
26967 7990 Acked Read 6 Sent Done Done false
26968 7991 Acked Write 3 Sent Done Done false
26969 7992 Acked Write 7 Sent Done Done false
26970 7993 Acked Write 7 Sent Done Done false
26971 7994 Acked Read 10 Sent Done Done false
26972 7995 Acked Read 1 Sent Done Done false
26973 7996 Acked Write 1 Sent Done Done false
26974 7997 Acked Write 3 Sent Done Done false
26975 7998 Acked Write 5 Sent Done Done false
26976 7999 Acked Write 9 Sent Done Done false
26977 8000 Acked Write 10 Sent Done Done false
26978 8001 Acked Read 2 Sent Done Done false
26979 8002 Acked Write 10 Sent Done Done false
26980 8003 Acked Write 8 Sent Done Done false
26981 8004 Acked Read 5 Sent Done Done false
26982 8005 Acked Read 10 Sent Done Done false
26983 8006 Acked Write 6 Sent Done Done false
26984 8007 Acked Read 5 Sent Done Done false
26985 8008 Acked Flush 0 Sent Done Done false
26986 8009 Acked Read 3 Sent Done Done false
26987 8010 Acked Write 3 Sent Done Done false
26988 8011 Acked Write 2 Sent Done Done false
26989 8012 Acked Read 1 Sent Done Done false
26990 8013 Acked Read 10 Sent Done Done false
26991 8014 Acked Write 2 Sent Done Done false
26992 8015 Acked Read 1 Sent Done Done false
26993 8016 Acked Write 1 Sent Done Done false
26994 8017 Acked Read 3 Sent Done Done false
26995 8018 Acked Read 3 Sent Done Done false
26996 8019 Acked Read 10 Sent Done Done false
26997 8020 Acked Write 2 Sent Done Done false
26998 8021 Acked Flush 0 Sent Done Done false
26999 8022 Acked Flush 0 Sent Done Done false
27000 8023 Acked Write 6 Sent Done Done false
27001 8024 Acked Read 4 Sent Done Done false
27002 8025 Acked Write 6 Sent Done Done false
27003 8026 Acked Read 9 Sent Done Done false
27004 8027 Acked Read 8 Sent Done Done false
27005 8028 Acked Write 5 Sent Done Done false
27006 8029 Acked Read 7 Sent Done Done false
27007 8030 Acked Read 4 Sent Done Done false
27008 8031 Acked Write 4 Sent Done Done false
27009 8032 Acked Read 6 Sent Done Done false
27010 8033 Acked Write 4 Sent Done Done false
27011 8034 Acked Read 3 Sent Done Done false
27012 8035 Acked Read 4 Sent Done Done false
27013 8036 Acked Read 10 Sent Done Done false
27014 8037 Acked Flush 0 Sent Done Done false
27015 8038 Acked Read 2 Sent Done Done false
27016 8039 Acked Read 9 Sent Done Done false
27017 8040 Acked Read 8 Sent Done Done false
27018 8041 Acked Read 10 Sent Done Done false
27019 8042 Acked Read 7 Sent Done Done false
27020 8043 Acked Write 3 Sent Done Done false
27021 8044 Acked Write 5 Sent Done Done false
27022 8045 Acked Write 3 Sent Done Done false
27023 8046 Acked Read 6 Sent Done Done false
27024 8047 Acked Read 9 Sent Done Done false
27025 8048 Acked Read 8 Sent Done Done false
27026 8049 Acked Read 3 Sent Done Done false
27027 8050 Acked Flush 0 Sent Done Done false
27028 8051 Acked Write 3 Sent Done Done false
27029 8052 Acked Read 7 Sent Done Done false
27030 8053 Acked Write 3 Sent Done Done false
27031 8054 Acked Read 8 Sent Done Done false
27032 8055 Acked Read 8 Sent Done Done false
27033 8056 Acked Read 8 Sent Done Done false
27034 8057 Acked Write 6 Sent Done Done false
27035 8058 Acked Read 8 Sent Done Done false
27036 8059 Acked Read 5 Sent Done Done false
27037 8060 Acked Read 9 Sent Done Done false
27038 8061 Acked Write 2 Sent Done Done false
27039 8062 Acked Write 6 Sent Done Done false
27040 8063 Acked Write 5 Sent Done Done false
27041 8064 Acked Read 7 Sent Done Done false
27042 8065 Acked Read 2 Sent Done Done false
27043 8066 Acked Write 1 Sent Done Done false
27044 8067 Acked Write 1 Sent Done Done false
27045 8068 Acked Read 9 Sent Done Done false
27046 8069 Acked Read 5 Sent Done Done false
27047 8070 Acked Flush 0 Sent Done Done false
27048 8071 Acked Read 10 Sent Done Done false
27049 8072 Acked Write 2 Sent Done Done false
27050 8073 Acked Read 1 Sent Done Done false
27051 8074 Acked Write 6 Sent Done Done false
27052 8075 Acked Read 5 Sent Done Done false
27053 8076 Acked Write 2 Sent Done Done false
27054 8077 Acked Read 10 Sent Done Done false
27055 8078 Acked Write 7 Sent Done Done false
27056 8079 Acked Read 1 Sent Done Done false
27057 8080 Acked Write 7 Sent Done Done false
27058 8081 Acked Write 1 Sent Done Done false
27059 8082 Acked Flush 0 Sent Done Done false
27060 8083 Acked Read 10 Sent Done Done false
27061 8084 Acked Read 5 Sent Done Done false
27062 8085 Acked Read 7 Sent Done Done false
27063 8086 Acked Read 2 Sent Done Done false
27064 8087 Acked Write 2 Sent Done Done false
27065 8088 Acked Write 10 Sent Done Done false
27066 8089 Acked Write 6 Sent Done Done false
27067 8090 Acked Write 2 Sent Done Done false
27068 8091 Acked Read 8 Sent Done Done false
27069 8092 Acked Write 10 Sent Done Done false
27070 8093 Acked Write 8 Sent Done Done false
27071 8094 Acked Write 1 Sent Done Done false
27072 8095 Acked Read 3 Sent Done Done false
27073 8096 Acked Write 10 Sent Done Done false
27074 8097 Acked Flush 0 Sent Done Done false
27075 8098 Acked Read 3 Sent Done Done false
27076 8099 Acked Read 8 Sent Done Done false
27077 8100 Acked Read 6 Sent Done Done false
27078 8101 Acked Read 3 Sent Done Done false
27079 8102 Acked Read 7 Sent Done Done false
27080 8103 Acked Write 10 Sent Done Done false
27081 8104 Acked Write 5 Sent Done Done false
27082 8105 Acked Write 4 Sent Done Done false
27083 8106 Acked Read 7 Sent Done Done false
27084 8107 Acked Read 2 Sent Done Done false
27085 8108 Acked Read 4 Sent Done Done false
27086 8109 Acked Read 1 Sent Done Done false
27087 8110 Acked Write 9 Sent Done Done false
27088 8111 Acked Read 9 Sent Done Done false
27089 8112 Acked Write 6 Sent Done Done false
27090 8113 Acked Read 10 Sent Done Done false
27091 8114 Acked Read 4 Sent Done Done false
27092 8115 Acked Read 2 Sent Done Done false
27093 8116 Acked Read 6 Sent Done Done false
27094 8117 Acked Flush 0 Sent Done Done false
27095 8118 Acked Flush 0 Sent Done Done false
27096 8119 Acked Write 6 Sent Done Done false
27097 8120 Acked Write 7 Sent Done Done false
27098 8121 Acked Write 2 Sent Done Done false
27099 8122 Acked Flush 0 Sent Done Done false
27100 8123 Acked Write 3 Sent Done Done false
27101 8124 Acked Read 10 Sent Done Done false
27102 8125 Acked Read 2 Sent Done Done false
27103 8126 Acked Write 5 Sent Done Done false
27104 8127 Acked Flush 0 Sent Done Done false
27105 8128 Acked Read 10 Sent Done Done false
27106 8129 Acked Read 7 Sent Done Done false
27107 8130 Acked Write 4 Sent Done Done false
27108 8131 Acked Write 3 Sent Done Done false
27109 8132 Acked Write 4 Sent Done Done false
27110 8133 Acked Write 7 Sent Done Done false
27111 8134 Acked Read 7 Sent Done Done false
27112 8135 Acked Read 5 Sent Done Done false
27113 8136 Acked Flush 0 Sent Done Done false
27114 8137 Acked Write 9 Sent Done Done false
27115 8138 Acked Flush 0 Sent Done Done false
27116 8139 Acked Read 5 Sent Done Done false
27117 8140 Acked Read 3 Sent Done Done false
27118 8141 Acked Flush 0 Sent Done Done false
27119 8142 Acked Read 2 Sent Done Done false
27120 8143 Acked Flush 0 Sent Done Done false
27121 8144 Acked Read 4 Sent Done Done false
27122 8145 Acked Read 9 Sent Done Done false
27123 8146 Acked Read 10 Sent Done Done false
27124 8147 Acked Read 4 Sent Done Done false
27125 8148 Acked Write 9 Sent Done Done false
27126 8149 Acked Write 7 Sent Done Done false
27127 8150 Acked Write 8 Sent Done Done false
27128 8151 Acked Write 1 Sent Done Done false
27129 8152 Acked Write 1 Sent Done Done false
27130 8153 Acked Read 7 Sent Done Done false
27131 8154 Acked Write 2 Sent Done Done false
27132 8155 Acked Write 8 Sent Done Done false
27133 8156 Acked Read 5 Sent Done Done false
27134 8157 Acked Read 4 Sent Done Done false
27135 8158 Acked Write 1 Sent Done Done false
27136 8159 Acked Read 9 Sent Done Done false
27137 8160 Acked Read 6 Sent Done Done false
27138 8161 Acked Read 2 Sent Done Done false
27139 8162 Acked Read 10 Sent Done Done false
27140 8163 Acked Read 7 Sent Done Done false
27141 8164 Acked Read 4 Sent Done Done false
27142 8165 Acked Read 6 Sent Done Done false
27143 8166 Acked Write 9 Sent Done Done false
27144 8167 Acked Read 7 Sent Done Done false
27145 8168 Acked Write 4 Sent Done Done false
27146 8169 Acked Write 3 Sent Done Done false
27147 8170 Acked Read 5 Sent Done Done false
27148 8171 Acked Write 10 Sent Done Done false
27149 8172 Acked Read 2 Sent Done Done false
27150 8173 Acked Write 1 Sent Done Done false
27151 8174 Acked Read 9 Sent Done Done false
27152 8175 Acked Flush 0 Sent Done Done false
27153 8176 Acked Read 6 Sent Done Done false
27154 8177 Acked Read 4 Sent Done Done false
27155 8178 Acked Read 5 Sent Done Done false
27156 8179 Acked Read 7 Sent Done Done false
27157 8180 Acked Read 5 Sent Done Done false
27158 8181 Acked Write 7 Sent Done Done false
27159 8182 Acked Read 1 Sent Done Done false
27160 8183 Acked Read 6 Sent Done Done false
27161 8184 Acked Read 9 Sent Done Done false
27162 8185 Acked Write 1 Sent Done Done false
27163 8186 Acked Write 10 Sent Done Done false
27164 8187 Acked Read 7 Sent Done Done false
27165 8188 Acked Write 6 Sent Done Done false
27166 8189 Acked Write 4 Sent Done Done false
27167 8190 Acked Read 5 Sent Done Done false
27168 8191 Acked Read 2 Sent Done Done false
27169 8192 Acked Write 10 Sent Done Done false
27170 8193 Acked Read 6 Sent Done Done false
27171 8194 Acked Write 10 Sent Done Done false
27172 8195 Acked Write 1 Sent Done Done false
27173 8196 Acked Read 1 Sent Done Done false
27174 8197 Acked Read 4 Sent Done Done false
27175 8198 Acked Write 7 Sent Done Done false
27176 8199 Acked Read 4 Sent Done Done false
27177 8200 Acked Read 5 Sent Done Done false
27178 8201 Acked Write 9 Sent Done Done false
27179 8202 Acked Write 5 Sent Done Done false
27180 8203 Acked Write 6 Sent Done Done false
27181 8204 Acked Write 2 Sent Done Done false
27182 8205 Acked Write 7 Sent Done Done false
27183 8206 Acked Read 10 Sent Done Done false
27184 8207 Acked Read 6 Sent Done Done false
27185 8208 Acked Write 5 Sent Done Done false
27186 8209 Acked Read 10 Sent Done Done false
27187 8210 Acked Read 6 Sent Done Done false
27188 8211 Acked Read 8 Sent Done Done false
27189 8212 Acked Write 10 Sent Done Done false
27190 8213 Acked Write 9 Sent Done Done false
27191 8214 Acked Read 6 Sent Done Done false
27192 8215 Acked Read 4 Sent Done Done false
27193 8216 Acked Write 3 Sent Done Done false
27194 8217 Acked Write 7 Sent Done Done false
27195 8218 Acked Read 7 Sent Done Done false
27196 8219 Acked Write 10 Sent Done Done false
27197 8220 Acked Flush 0 Sent Done Done false
27198 8221 Acked Read 4 Sent Done Done false
27199 8222 Acked Read 7 Sent Done Done false
27200 8223 Acked Read 6 Sent Done Done false
27201 8224 Acked Read 10 Sent Done Done false
27202 8225 Acked Read 7 Sent Done Done false
27203 8226 Acked Write 4 Sent Done Done false
27204 8227 Acked Read 4 Sent Done Done false
27205 8228 Acked Write 3 Sent Done Done false
27206 8229 Acked Read 4 Sent Done Done false
27207 8230 Acked Write 2 Sent Done Done false
27208 8231 Acked Read 4 Sent Done Done false
27209 8232 Acked Flush 0 Sent Done Done false
27210 8233 Acked Flush 0 Sent Done Done false
27211 8234 Acked Flush 0 Sent Done Done false
27212 8235 Acked Write 10 Sent Done Done false
27213 8236 Acked Flush 0 Sent Done Done false
27214 8237 Acked Write 4 Sent Done Done false
27215 8238 Acked Write 2 Sent Done Done false
27216 8239 Acked Write 9 Sent Done Done false
27217 8240 Acked Flush 0 Sent Done Done false
27218 8241 Acked Write 9 Sent Done Done false
27219 8242 Acked Write 2 Sent Done Done false
27220 8243 Acked Flush 0 Sent Done Done false
27221 8244 Acked Read 8 Sent Done Done false
27222 8245 Acked Write 1 Sent Done Done false
27223 STATES DS:0 DS:1 DS:2 TOTAL
27224 Sent 301 0 0 301
27225 Done 0 301 301 602
27226 Skipped 0 0 0 0
27227 Error 0 0 0 0
27228 Last Flush: Some(JobId(7944)) Some(JobId(8243)) Some(JobId(8243))
27229 Downstairs last five completed: 7944 7943 7942 7941 7940
27230 Upstairs last five completed: 8245 8244 8243 8242 8241
27231 CLIENT: Up:0 ds:301 act:2
272322025-01-29T05:16:41.998ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
272332025-01-29T05:16:42.517ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
272342025-01-29T05:16:43.519ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
272352025-01-29T05:16:44.519ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
27236 ----------------------------------------------------------------
27237 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27238 States: Offline Active Active
27239 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27240 7945 Acked Read 5 Sent Done Done false
27241 7946 Acked Read 1 Sent Done Done false
27242 7947 Acked Write 4 Sent Done Done false
27243 7948 Acked Write 6 Sent Done Done false
27244 7949 Acked Write 6 Sent Done Done false
27245 7950 Acked Write 8 Sent Done Done false
27246 7951 Acked Read 9 Sent Done Done false
27247 7952 Acked Write 5 Sent Done Done false
27248 7953 Acked Read 7 Sent Done Done false
27249 7954 Acked Flush 0 Sent Done Done false
27250 7955 Acked Write 4 Sent Done Done false
27251 7956 Acked Flush 0 Sent Done Done false
27252 7957 Acked Flush 0 Sent Done Done false
27253 7958 Acked Flush 0 Sent Done Done false
27254 7959 Acked Write 10 Sent Done Done false
27255 7960 Acked Write 5 Sent Done Done false
27256 7961 Acked Read 6 Sent Done Done false
27257 7962 Acked Write 4 Sent Done Done false
27258 7963 Acked Write 9 Sent Done Done false
27259 7964 Acked Read 2 Sent Done Done false
27260 7965 Acked Read 10 Sent Done Done false
27261 7966 Acked Read 4 Sent Done Done false
27262 7967 Acked Flush 0 Sent Done Done false
27263 7968 Acked Write 8 Sent Done Done false
27264 7969 Acked Read 7 Sent Done Done false
27265 7970 Acked Write 4 Sent Done Done false
27266 7971 Acked Read 8 Sent Done Done false
27267 7972 Acked Read 6 Sent Done Done false
27268 7973 Acked Read 5 Sent Done Done false
27269 7974 Acked Write 5 Sent Done Done false
27270 7975 Acked Read 3 Sent Done Done false
27271 7976 Acked Read 10 Sent Done Done false
27272 7977 Acked Read 3 Sent Done Done false
27273 7978 Acked Read 1 Sent Done Done false
27274 7979 Acked Write 8 Sent Done Done false
27275 7980 Acked Read 8 Sent Done Done false
27276 7981 Acked Read 5 Sent Done Done false
27277 7982 Acked Read 6 Sent Done Done false
27278 7983 Acked Read 8 Sent Done Done false
27279 7984 Acked Read 1 Sent Done Done false
27280 7985 Acked Read 6 Sent Done Done false
27281 7986 Acked Write 1 Sent Done Done false
27282 7987 Acked Read 5 Sent Done Done false
27283 7988 Acked Read 4 Sent Done Done false
27284 7989 Acked Read 3 Sent Done Done false
27285 7990 Acked Read 6 Sent Done Done false
27286 7991 Acked Write 3 Sent Done Done false
27287 7992 Acked Write 7 Sent Done Done false
27288 7993 Acked Write 7 Sent Done Done false
27289 7994 Acked Read 10 Sent Done Done false
27290 7995 Acked Read 1 Sent Done Done false
27291 7996 Acked Write 1 Sent Done Done false
27292 7997 Acked Write 3 Sent Done Done false
27293 7998 Acked Write 5 Sent Done Done false
27294 7999 Acked Write 9 Sent Done Done false
27295 8000 Acked Write 10 Sent Done Done false
27296 8001 Acked Read 2 Sent Done Done false
27297 8002 Acked Write 10 Sent Done Done false
27298 8003 Acked Write 8 Sent Done Done false
27299 8004 Acked Read 5 Sent Done Done false
27300 8005 Acked Read 10 Sent Done Done false
27301 8006 Acked Write 6 Sent Done Done false
27302 8007 Acked Read 5 Sent Done Done false
27303 8008 Acked Flush 0 Sent Done Done false
27304 8009 Acked Read 3 Sent Done Done false
27305 8010 Acked Write 3 Sent Done Done false
27306 8011 Acked Write 2 Sent Done Done false
27307 8012 Acked Read 1 Sent Done Done false
27308 8013 Acked Read 10 Sent Done Done false
27309 8014 Acked Write 2 Sent Done Done false
27310 8015 Acked Read 1 Sent Done Done false
27311 8016 Acked Write 1 Sent Done Done false
27312 8017 Acked Read 3 Sent Done Done false
27313 8018 Acked Read 3 Sent Done Done false
27314 8019 Acked Read 10 Sent Done Done false
27315 8020 Acked Write 2 Sent Done Done false
27316 8021 Acked Flush 0 Sent Done Done false
27317 8022 Acked Flush 0 Sent Done Done false
27318 8023 Acked Write 6 Sent Done Done false
27319 8024 Acked Read 4 Sent Done Done false
27320 8025 Acked Write 6 Sent Done Done false
27321 8026 Acked Read 9 Sent Done Done false
27322 8027 Acked Read 8 Sent Done Done false
27323 8028 Acked Write 5 Sent Done Done false
27324 8029 Acked Read 7 Sent Done Done false
27325 8030 Acked Read 4 Sent Done Done false
27326 8031 Acked Write 4 Sent Done Done false
27327 8032 Acked Read 6 Sent Done Done false
27328 8033 Acked Write 4 Sent Done Done false
27329 8034 Acked Read 3 Sent Done Done false
27330 8035 Acked Read 4 Sent Done Done false
27331 8036 Acked Read 10 Sent Done Done false
27332 8037 Acked Flush 0 Sent Done Done false
27333 8038 Acked Read 2 Sent Done Done false
27334 8039 Acked Read 9 Sent Done Done false
27335 8040 Acked Read 8 Sent Done Done false
27336 8041 Acked Read 10 Sent Done Done false
27337 8042 Acked Read 7 Sent Done Done false
27338 8043 Acked Write 3 Sent Done Done false
27339 8044 Acked Write 5 Sent Done Done false
27340 8045 Acked Write 3 Sent Done Done false
27341 8046 Acked Read 6 Sent Done Done false
27342 8047 Acked Read 9 Sent Done Done false
27343 8048 Acked Read 8 Sent Done Done false
27344 8049 Acked Read 3 Sent Done Done false
27345 8050 Acked Flush 0 Sent Done Done false
27346 8051 Acked Write 3 Sent Done Done false
27347 8052 Acked Read 7 Sent Done Done false
27348 8053 Acked Write 3 Sent Done Done false
27349 8054 Acked Read 8 Sent Done Done false
27350 8055 Acked Read 8 Sent Done Done false
27351 8056 Acked Read 8 Sent Done Done false
27352 8057 Acked Write 6 Sent Done Done false
27353 8058 Acked Read 8 Sent Done Done false
27354 8059 Acked Read 5 Sent Done Done false
27355 8060 Acked Read 9 Sent Done Done false
27356 8061 Acked Write 2 Sent Done Done false
27357 8062 Acked Write 6 Sent Done Done false
27358 8063 Acked Write 5 Sent Done Done false
27359 8064 Acked Read 7 Sent Done Done false
27360 8065 Acked Read 2 Sent Done Done false
27361 8066 Acked Write 1 Sent Done Done false
27362 8067 Acked Write 1 Sent Done Done false
27363 8068 Acked Read 9 Sent Done Done false
27364 8069 Acked Read 5 Sent Done Done false
27365 8070 Acked Flush 0 Sent Done Done false
27366 8071 Acked Read 10 Sent Done Done false
27367 8072 Acked Write 2 Sent Done Done false
27368 8073 Acked Read 1 Sent Done Done false
27369 8074 Acked Write 6 Sent Done Done false
27370 8075 Acked Read 5 Sent Done Done false
27371 8076 Acked Write 2 Sent Done Done false
27372 8077 Acked Read 10 Sent Done Done false
27373 8078 Acked Write 7 Sent Done Done false
27374 8079 Acked Read 1 Sent Done Done false
27375 8080 Acked Write 7 Sent Done Done false
27376 8081 Acked Write 1 Sent Done Done false
27377 8082 Acked Flush 0 Sent Done Done false
27378 8083 Acked Read 10 Sent Done Done false
27379 8084 Acked Read 5 Sent Done Done false
27380 8085 Acked Read 7 Sent Done Done false
27381 8086 Acked Read 2 Sent Done Done false
27382 8087 Acked Write 2 Sent Done Done false
27383 8088 Acked Write 10 Sent Done Done false
27384 8089 Acked Write 6 Sent Done Done false
27385 8090 Acked Write 2 Sent Done Done false
27386 8091 Acked Read 8 Sent Done Done false
27387 8092 Acked Write 10 Sent Done Done false
27388 8093 Acked Write 8 Sent Done Done false
27389 8094 Acked Write 1 Sent Done Done false
27390 8095 Acked Read 3 Sent Done Done false
27391 8096 Acked Write 10 Sent Done Done false
27392 8097 Acked Flush 0 Sent Done Done false
27393 8098 Acked Read 3 Sent Done Done false
27394 8099 Acked Read 8 Sent Done Done false
27395 8100 Acked Read 6 Sent Done Done false
27396 8101 Acked Read 3 Sent Done Done false
27397 8102 Acked Read 7 Sent Done Done false
27398 8103 Acked Write 10 Sent Done Done false
27399 8104 Acked Write 5 Sent Done Done false
27400 8105 Acked Write 4 Sent Done Done false
27401 8106 Acked Read 7 Sent Done Done false
27402 8107 Acked Read 2 Sent Done Done false
27403 8108 Acked Read 4 Sent Done Done false
27404 8109 Acked Read 1 Sent Done Done false
27405 8110 Acked Write 9 Sent Done Done false
27406 8111 Acked Read 9 Sent Done Done false
27407 8112 Acked Write 6 Sent Done Done false
27408 8113 Acked Read 10 Sent Done Done false
27409 8114 Acked Read 4 Sent Done Done false
27410 8115 Acked Read 2 Sent Done Done false
27411 8116 Acked Read 6 Sent Done Done false
27412 8117 Acked Flush 0 Sent Done Done false
27413 8118 Acked Flush 0 Sent Done Done false
27414 8119 Acked Write 6 Sent Done Done false
27415 8120 Acked Write 7 Sent Done Done false
27416 8121 Acked Write 2 Sent Done Done false
27417 8122 Acked Flush 0 Sent Done Done false
27418 8123 Acked Write 3 Sent Done Done false
27419 8124 Acked Read 10 Sent Done Done false
27420 8125 Acked Read 2 Sent Done Done false
27421 8126 Acked Write 5 Sent Done Done false
27422 8127 Acked Flush 0 Sent Done Done false
27423 8128 Acked Read 10 Sent Done Done false
27424 8129 Acked Read 7 Sent Done Done false
27425 8130 Acked Write 4 Sent Done Done false
27426 8131 Acked Write 3 Sent Done Done false
27427 8132 Acked Write 4 Sent Done Done false
27428 8133 Acked Write 7 Sent Done Done false
27429 8134 Acked Read 7 Sent Done Done false
27430 8135 Acked Read 5 Sent Done Done false
27431 8136 Acked Flush 0 Sent Done Done false
27432 8137 Acked Write 9 Sent Done Done false
27433 8138 Acked Flush 0 Sent Done Done false
27434 8139 Acked Read 5 Sent Done Done false
27435 8140 Acked Read 3 Sent Done Done false
27436 8141 Acked Flush 0 Sent Done Done false
27437 8142 Acked Read 2 Sent Done Done false
27438 8143 Acked Flush 0 Sent Done Done false
27439 8144 Acked Read 4 Sent Done Done false
27440 8145 Acked Read 9 Sent Done Done false
27441 8146 Acked Read 10 Sent Done Done false
27442 8147 Acked Read 4 Sent Done Done false
27443 8148 Acked Write 9 Sent Done Done false
27444 8149 Acked Write 7 Sent Done Done false
27445 8150 Acked Write 8 Sent Done Done false
27446 8151 Acked Write 1 Sent Done Done false
27447 8152 Acked Write 1 Sent Done Done false
27448 8153 Acked Read 7 Sent Done Done false
27449 8154 Acked Write 2 Sent Done Done false
27450 8155 Acked Write 8 Sent Done Done false
27451 8156 Acked Read 5 Sent Done Done false
27452 8157 Acked Read 4 Sent Done Done false
27453 8158 Acked Write 1 Sent Done Done false
27454 8159 Acked Read 9 Sent Done Done false
27455 8160 Acked Read 6 Sent Done Done false
27456 8161 Acked Read 2 Sent Done Done false
27457 8162 Acked Read 10 Sent Done Done false
27458 8163 Acked Read 7 Sent Done Done false
27459 8164 Acked Read 4 Sent Done Done false
27460 8165 Acked Read 6 Sent Done Done false
27461 8166 Acked Write 9 Sent Done Done false
27462 8167 Acked Read 7 Sent Done Done false
27463 8168 Acked Write 4 Sent Done Done false
27464 8169 Acked Write 3 Sent Done Done false
27465 8170 Acked Read 5 Sent Done Done false
27466 8171 Acked Write 10 Sent Done Done false
27467 8172 Acked Read 2 Sent Done Done false
27468 8173 Acked Write 1 Sent Done Done false
27469 8174 Acked Read 9 Sent Done Done false
27470 8175 Acked Flush 0 Sent Done Done false
27471 8176 Acked Read 6 Sent Done Done false
27472 8177 Acked Read 4 Sent Done Done false
27473 8178 Acked Read 5 Sent Done Done false
27474 8179 Acked Read 7 Sent Done Done false
27475 8180 Acked Read 5 Sent Done Done false
27476 8181 Acked Write 7 Sent Done Done false
27477 8182 Acked Read 1 Sent Done Done false
27478 8183 Acked Read 6 Sent Done Done false
27479 8184 Acked Read 9 Sent Done Done false
27480 8185 Acked Write 1 Sent Done Done false
27481 8186 Acked Write 10 Sent Done Done false
27482 8187 Acked Read 7 Sent Done Done false
27483 8188 Acked Write 6 Sent Done Done false
27484 8189 Acked Write 4 Sent Done Done false
27485 8190 Acked Read 5 Sent Done Done false
27486 8191 Acked Read 2 Sent Done Done false
27487 8192 Acked Write 10 Sent Done Done false
27488 8193 Acked Read 6 Sent Done Done false
27489 8194 Acked Write 10 Sent Done Done false
27490 8195 Acked Write 1 Sent Done Done false
27491 8196 Acked Read 1 Sent Done Done false
27492 8197 Acked Read 4 Sent Done Done false
27493 8198 Acked Write 7 Sent Done Done false
27494 8199 Acked Read 4 Sent Done Done false
27495 8200 Acked Read 5 Sent Done Done false
27496 8201 Acked Write 9 Sent Done Done false
27497 8202 Acked Write 5 Sent Done Done false
27498 8203 Acked Write 6 Sent Done Done false
27499 8204 Acked Write 2 Sent Done Done false
27500 8205 Acked Write 7 Sent Done Done false
27501 8206 Acked Read 10 Sent Done Done false
27502 8207 Acked Read 6 Sent Done Done false
27503 8208 Acked Write 5 Sent Done Done false
27504 8209 Acked Read 10 Sent Done Done false
27505 8210 Acked Read 6 Sent Done Done false
27506 8211 Acked Read 8 Sent Done Done false
27507 8212 Acked Write 10 Sent Done Done false
27508 8213 Acked Write 9 Sent Done Done false
27509 8214 Acked Read 6 Sent Done Done false
27510 8215 Acked Read 4 Sent Done Done false
27511 8216 Acked Write 3 Sent Done Done false
27512 8217 Acked Write 7 Sent Done Done false
27513 8218 Acked Read 7 Sent Done Done false
27514 8219 Acked Write 10 Sent Done Done false
27515 8220 Acked Flush 0 Sent Done Done false
27516 8221 Acked Read 4 Sent Done Done false
27517 8222 Acked Read 7 Sent Done Done false
27518 8223 Acked Read 6 Sent Done Done false
27519 8224 Acked Read 10 Sent Done Done false
27520 8225 Acked Read 7 Sent Done Done false
27521 8226 Acked Write 4 Sent Done Done false
27522 8227 Acked Read 4 Sent Done Done false
27523 8228 Acked Write 3 Sent Done Done false
27524 8229 Acked Read 4 Sent Done Done false
27525 8230 Acked Write 2 Sent Done Done false
27526 8231 Acked Read 4 Sent Done Done false
27527 8232 Acked Flush 0 Sent Done Done false
27528 8233 Acked Flush 0 Sent Done Done false
27529 8234 Acked Flush 0 Sent Done Done false
27530 8235 Acked Write 10 Sent Done Done false
27531 8236 Acked Flush 0 Sent Done Done false
27532 8237 Acked Write 4 Sent Done Done false
27533 8238 Acked Write 2 Sent Done Done false
27534 8239 Acked Write 9 Sent Done Done false
27535 8240 Acked Flush 0 Sent Done Done false
27536 8241 Acked Write 9 Sent Done Done false
27537 8242 Acked Write 2 Sent Done Done false
27538 8243 Acked Flush 0 Sent Done Done false
27539 8244 Acked Read 8 Sent Done Done false
27540 8245 Acked Write 1 Sent Done Done false
27541 STATES DS:0 DS:1 DS:2 TOTAL
27542 Sent 301 0 0 301
27543 Done 0 301 301 602
27544 Skipped 0 0 0 0
27545 Error 0 0 0 0
27546 Last Flush: Some(JobId(7944)) Some(JobId(8243)) Some(JobId(8243))
27547 Downstairs last five completed: 7944 7943 7942 7941 7940
27548 Upstairs last five completed: 8245 8244 8243 8242 8241
27549 CLIENT: Up:0 ds:301 act:2
275502025-01-29T05:16:46.524ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275512025-01-29T05:16:47.251ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275522025-01-29T05:16:47.251ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275532025-01-29T05:16:47.251ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275542025-01-29T05:16:47.252ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275552025-01-29T05:16:47.252ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275562025-01-29T05:16:47.753ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
275572025-01-29T05:16:48.527ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
27558 [8810] got start action
27559 [8810] I am going to restart
27560 Starting downstairs at port 8810
27561 Make output file at "/tmp/dsc/downstairs-8810.txt"
27562 [8810][0] reports Starting
27563 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1226)
27564 [8810][0] reports Running
27565 ----------------------------------------------------------------
27566 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27567 States: Offline Active Active
27568 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27569 7945 Acked Read 5 Sent Done Done false
27570 7946 Acked Read 1 Sent Done Done false
27571 7947 Acked Write 4 Sent Done Done false
27572 7948 Acked Write 6 Sent Done Done false
27573 7949 Acked Write 6 Sent Done Done false
27574 7950 Acked Write 8 Sent Done Done false
27575 7951 Acked Read 9 Sent Done Done false
27576 7952 Acked Write 5 Sent Done Done false
27577 7953 Acked Read 7 Sent Done Done false
27578 7954 Acked Flush 0 Sent Done Done false
27579 7955 Acked Write 4 Sent Done Done false
27580 7956 Acked Flush 0 Sent Done Done false
27581 7957 Acked Flush 0 Sent Done Done false
27582 7958 Acked Flush 0 Sent Done Done false
27583 7959 Acked Write 10 Sent Done Done false
27584 7960 Acked Write 5 Sent Done Done false
27585 7961 Acked Read 6 Sent Done Done false
27586 7962 Acked Write 4 Sent Done Done false
27587 7963 Acked Write 9 Sent Done Done false
27588 7964 Acked Read 2 Sent Done Done false
27589 7965 Acked Read 10 Sent Done Done false
27590 7966 Acked Read 4 Sent Done Done false
27591 7967 Acked Flush 0 Sent Done Done false
27592 7968 Acked Write 8 Sent Done Done false
27593 7969 Acked Read 7 Sent Done Done false
27594 7970 Acked Write 4 Sent Done Done false
27595 7971 Acked Read 8 Sent Done Done false
27596 7972 Acked Read 6 Sent Done Done false
27597 7973 Acked Read 5 Sent Done Done false
27598 7974 Acked Write 5 Sent Done Done false
27599 7975 Acked Read 3 Sent Done Done false
27600 7976 Acked Read 10 Sent Done Done false
27601 7977 Acked Read 3 Sent Done Done false
27602 7978 Acked Read 1 Sent Done Done false
27603 7979 Acked Write 8 Sent Done Done false
27604 7980 Acked Read 8 Sent Done Done false
27605 7981 Acked Read 5 Sent Done Done false
27606 7982 Acked Read 6 Sent Done Done false
27607 7983 Acked Read 8 Sent Done Done false
27608 7984 Acked Read 1 Sent Done Done false
27609 7985 Acked Read 6 Sent Done Done false
27610 7986 Acked Write 1 Sent Done Done false
27611 7987 Acked Read 5 Sent Done Done false
27612 7988 Acked Read 4 Sent Done Done false
27613 7989 Acked Read 3 Sent Done Done false
27614 7990 Acked Read 6 Sent Done Done false
27615 7991 Acked Write 3 Sent Done Done false
27616 7992 Acked Write 7 Sent Done Done false
27617 7993 Acked Write 7 Sent Done Done false
27618 7994 Acked Read 10 Sent Done Done false
27619 7995 Acked Read 1 Sent Done Done false
27620 7996 Acked Write 1 Sent Done Done false
27621 7997 Acked Write 3 Sent Done Done false
27622 7998 Acked Write 5 Sent Done Done false
27623 7999 Acked Write 9 Sent Done Done false
27624 8000 Acked Write 10 Sent Done Done false
27625 8001 Acked Read 2 Sent Done Done false
27626 8002 Acked Write 10 Sent Done Done false
27627 8003 Acked Write 8 Sent Done Done false
27628 8004 Acked Read 5 Sent Done Done false
27629 8005 Acked Read 10 Sent Done Done false
27630 8006 Acked Write 6 Sent Done Done false
27631 8007 Acked Read 5 Sent Done Done false
27632 8008 Acked Flush 0 Sent Done Done false
27633 8009 Acked Read 3 Sent Done Done false
27634 8010 Acked Write 3 Sent Done Done false
27635 8011 Acked Write 2 Sent Done Done false
27636 8012 Acked Read 1 Sent Done Done false
27637 8013 Acked Read 10 Sent Done Done false
27638 8014 Acked Write 2 Sent Done Done false
27639 8015 Acked Read 1 Sent Done Done false
27640 8016 Acked Write 1 Sent Done Done false
27641 8017 Acked Read 3 Sent Done Done false
27642 8018 Acked Read 3 Sent Done Done false
27643 8019 Acked Read 10 Sent Done Done false
27644 8020 Acked Write 2 Sent Done Done false
27645 8021 Acked Flush 0 Sent Done Done false
27646 8022 Acked Flush 0 Sent Done Done false
27647 8023 Acked Write 6 Sent Done Done false
27648 8024 Acked Read 4 Sent Done Done false
27649 8025 Acked Write 6 Sent Done Done false
27650 8026 Acked Read 9 Sent Done Done false
27651 8027 Acked Read 8 Sent Done Done false
27652 8028 Acked Write 5 Sent Done Done false
27653 8029 Acked Read 7 Sent Done Done false
27654 8030 Acked Read 4 Sent Done Done false
27655 8031 Acked Write 4 Sent Done Done false
27656 8032 Acked Read 6 Sent Done Done false
27657 8033 Acked Write 4 Sent Done Done false
27658 8034 Acked Read 3 Sent Done Done false
27659 8035 Acked Read 4 Sent Done Done false
27660 8036 Acked Read 10 Sent Done Done false
27661 8037 Acked Flush 0 Sent Done Done false
27662 8038 Acked Read 2 Sent Done Done false
27663 8039 Acked Read 9 Sent Done Done false
27664 8040 Acked Read 8 Sent Done Done false
27665 8041 Acked Read 10 Sent Done Done false
27666 8042 Acked Read 7 Sent Done Done false
27667 8043 Acked Write 3 Sent Done Done false
27668 8044 Acked Write 5 Sent Done Done false
27669 8045 Acked Write 3 Sent Done Done false
27670 8046 Acked Read 6 Sent Done Done false
27671 8047 Acked Read 9 Sent Done Done false
27672 8048 Acked Read 8 Sent Done Done false
27673 8049 Acked Read 3 Sent Done Done false
27674 8050 Acked Flush 0 Sent Done Done false
27675 8051 Acked Write 3 Sent Done Done false
27676 8052 Acked Read 7 Sent Done Done false
27677 8053 Acked Write 3 Sent Done Done false
27678 8054 Acked Read 8 Sent Done Done false
27679 8055 Acked Read 8 Sent Done Done false
27680 8056 Acked Read 8 Sent Done Done false
27681 8057 Acked Write 6 Sent Done Done false
27682 8058 Acked Read 8 Sent Done Done false
27683 8059 Acked Read 5 Sent Done Done false
27684 8060 Acked Read 9 Sent Done Done false
27685 8061 Acked Write 2 Sent Done Done false
27686 8062 Acked Write 6 Sent Done Done false
27687 8063 Acked Write 5 Sent Done Done false
27688 8064 Acked Read 7 Sent Done Done false
27689 8065 Acked Read 2 Sent Done Done false
27690 8066 Acked Write 1 Sent Done Done false
27691 8067 Acked Write 1 Sent Done Done false
27692 8068 Acked Read 9 Sent Done Done false
27693 8069 Acked Read 5 Sent Done Done false
27694 8070 Acked Flush 0 Sent Done Done false
27695 8071 Acked Read 10 Sent Done Done false
27696 8072 Acked Write 2 Sent Done Done false
27697 8073 Acked Read 1 Sent Done Done false
27698 8074 Acked Write 6 Sent Done Done false
27699 8075 Acked Read 5 Sent Done Done false
27700 8076 Acked Write 2 Sent Done Done false
27701 8077 Acked Read 10 Sent Done Done false
27702 8078 Acked Write 7 Sent Done Done false
27703 8079 Acked Read 1 Sent Done Done false
27704 8080 Acked Write 7 Sent Done Done false
27705 8081 Acked Write 1 Sent Done Done false
27706 8082 Acked Flush 0 Sent Done Done false
27707 8083 Acked Read 10 Sent Done Done false
27708 8084 Acked Read 5 Sent Done Done false
27709 8085 Acked Read 7 Sent Done Done false
27710 8086 Acked Read 2 Sent Done Done false
27711 8087 Acked Write 2 Sent Done Done false
27712 8088 Acked Write 10 Sent Done Done false
27713 8089 Acked Write 6 Sent Done Done false
27714 8090 Acked Write 2 Sent Done Done false
27715 8091 Acked Read 8 Sent Done Done false
27716 8092 Acked Write 10 Sent Done Done false
27717 8093 Acked Write 8 Sent Done Done false
27718 8094 Acked Write 1 Sent Done Done false
27719 8095 Acked Read 3 Sent Done Done false
27720 8096 Acked Write 10 Sent Done Done false
27721 8097 Acked Flush 0 Sent Done Done false
27722 8098 Acked Read 3 Sent Done Done false
27723 8099 Acked Read 8 Sent Done Done false
27724 8100 Acked Read 6 Sent Done Done false
27725 8101 Acked Read 3 Sent Done Done false
27726 8102 Acked Read 7 Sent Done Done false
27727 8103 Acked Write 10 Sent Done Done false
27728 8104 Acked Write 5 Sent Done Done false
27729 8105 Acked Write 4 Sent Done Done false
27730 8106 Acked Read 7 Sent Done Done false
27731 8107 Acked Read 2 Sent Done Done false
27732 8108 Acked Read 4 Sent Done Done false
27733 8109 Acked Read 1 Sent Done Done false
27734 8110 Acked Write 9 Sent Done Done false
27735 8111 Acked Read 9 Sent Done Done false
27736 8112 Acked Write 6 Sent Done Done false
27737 8113 Acked Read 10 Sent Done Done false
27738 8114 Acked Read 4 Sent Done Done false
27739 8115 Acked Read 2 Sent Done Done false
27740 8116 Acked Read 6 Sent Done Done false
27741 8117 Acked Flush 0 Sent Done Done false
27742 8118 Acked Flush 0 Sent Done Done false
27743 8119 Acked Write 6 Sent Done Done false
27744 8120 Acked Write 7 Sent Done Done false
27745 8121 Acked Write 2 Sent Done Done false
27746 8122 Acked Flush 0 Sent Done Done false
27747 8123 Acked Write 3 Sent Done Done false
27748 8124 Acked Read 10 Sent Done Done false
27749 8125 Acked Read 2 Sent Done Done false
27750 8126 Acked Write 5 Sent Done Done false
27751 8127 Acked Flush 0 Sent Done Done false
27752 8128 Acked Read 10 Sent Done Done false
27753 8129 Acked Read 7 Sent Done Done false
27754 8130 Acked Write 4 Sent Done Done false
27755 8131 Acked Write 3 Sent Done Done false
27756 8132 Acked Write 4 Sent Done Done false
27757 8133 Acked Write 7 Sent Done Done false
27758 8134 Acked Read 7 Sent Done Done false
27759 8135 Acked Read 5 Sent Done Done false
27760 8136 Acked Flush 0 Sent Done Done false
27761 8137 Acked Write 9 Sent Done Done false
27762 8138 Acked Flush 0 Sent Done Done false
27763 8139 Acked Read 5 Sent Done Done false
27764 8140 Acked Read 3 Sent Done Done false
27765 8141 Acked Flush 0 Sent Done Done false
27766 8142 Acked Read 2 Sent Done Done false
27767 8143 Acked Flush 0 Sent Done Done false
27768 8144 Acked Read 4 Sent Done Done false
27769 8145 Acked Read 9 Sent Done Done false
27770 8146 Acked Read 10 Sent Done Done false
27771 8147 Acked Read 4 Sent Done Done false
27772 8148 Acked Write 9 Sent Done Done false
27773 8149 Acked Write 7 Sent Done Done false
27774 8150 Acked Write 8 Sent Done Done false
27775 8151 Acked Write 1 Sent Done Done false
27776 8152 Acked Write 1 Sent Done Done false
27777 8153 Acked Read 7 Sent Done Done false
27778 8154 Acked Write 2 Sent Done Done false
27779 8155 Acked Write 8 Sent Done Done false
27780 8156 Acked Read 5 Sent Done Done false
27781 8157 Acked Read 4 Sent Done Done false
27782 8158 Acked Write 1 Sent Done Done false
27783 8159 Acked Read 9 Sent Done Done false
27784 8160 Acked Read 6 Sent Done Done false
27785 8161 Acked Read 2 Sent Done Done false
27786 8162 Acked Read 10 Sent Done Done false
27787 8163 Acked Read 7 Sent Done Done false
27788 8164 Acked Read 4 Sent Done Done false
27789 8165 Acked Read 6 Sent Done Done false
27790 8166 Acked Write 9 Sent Done Done false
27791 8167 Acked Read 7 Sent Done Done false
27792 8168 Acked Write 4 Sent Done Done false
27793 8169 Acked Write 3 Sent Done Done false
27794 8170 Acked Read 5 Sent Done Done false
27795 8171 Acked Write 10 Sent Done Done false
27796 8172 Acked Read 2 Sent Done Done false
27797 8173 Acked Write 1 Sent Done Done false
27798 8174 Acked Read 9 Sent Done Done false
27799 8175 Acked Flush 0 Sent Done Done false
27800 8176 Acked Read 6 Sent Done Done false
27801 8177 Acked Read 4 Sent Done Done false
27802 8178 Acked Read 5 Sent Done Done false
27803 8179 Acked Read 7 Sent Done Done false
27804 8180 Acked Read 5 Sent Done Done false
27805 8181 Acked Write 7 Sent Done Done false
27806 8182 Acked Read 1 Sent Done Done false
27807 8183 Acked Read 6 Sent Done Done false
27808 8184 Acked Read 9 Sent Done Done false
27809 8185 Acked Write 1 Sent Done Done false
27810 8186 Acked Write 10 Sent Done Done false
27811 8187 Acked Read 7 Sent Done Done false
27812 8188 Acked Write 6 Sent Done Done false
27813 8189 Acked Write 4 Sent Done Done false
27814 8190 Acked Read 5 Sent Done Done false
27815 8191 Acked Read 2 Sent Done Done false
27816 8192 Acked Write 10 Sent Done Done false
27817 8193 Acked Read 6 Sent Done Done false
27818 8194 Acked Write 10 Sent Done Done false
27819 8195 Acked Write 1 Sent Done Done false
27820 8196 Acked Read 1 Sent Done Done false
27821 8197 Acked Read 4 Sent Done Done false
27822 8198 Acked Write 7 Sent Done Done false
27823 8199 Acked Read 4 Sent Done Done false
27824 8200 Acked Read 5 Sent Done Done false
27825 8201 Acked Write 9 Sent Done Done false
27826 8202 Acked Write 5 Sent Done Done false
27827 8203 Acked Write 6 Sent Done Done false
27828 8204 Acked Write 2 Sent Done Done false
27829 8205 Acked Write 7 Sent Done Done false
27830 8206 Acked Read 10 Sent Done Done false
27831 8207 Acked Read 6 Sent Done Done false
27832 8208 Acked Write 5 Sent Done Done false
27833 8209 Acked Read 10 Sent Done Done false
27834 8210 Acked Read 6 Sent Done Done false
27835 8211 Acked Read 8 Sent Done Done false
27836 8212 Acked Write 10 Sent Done Done false
27837 8213 Acked Write 9 Sent Done Done false
27838 8214 Acked Read 6 Sent Done Done false
27839 8215 Acked Read 4 Sent Done Done false
27840 8216 Acked Write 3 Sent Done Done false
27841 8217 Acked Write 7 Sent Done Done false
27842 8218 Acked Read 7 Sent Done Done false
27843 8219 Acked Write 10 Sent Done Done false
27844 8220 Acked Flush 0 Sent Done Done false
27845 8221 Acked Read 4 Sent Done Done false
27846 8222 Acked Read 7 Sent Done Done false
27847 8223 Acked Read 6 Sent Done Done false
27848 8224 Acked Read 10 Sent Done Done false
27849 8225 Acked Read 7 Sent Done Done false
27850 8226 Acked Write 4 Sent Done Done false
27851 8227 Acked Read 4 Sent Done Done false
27852 8228 Acked Write 3 Sent Done Done false
27853 8229 Acked Read 4 Sent Done Done false
27854 8230 Acked Write 2 Sent Done Done false
27855 8231 Acked Read 4 Sent Done Done false
27856 8232 Acked Flush 0 Sent Done Done false
27857 8233 Acked Flush 0 Sent Done Done false
27858 8234 Acked Flush 0 Sent Done Done false
27859 8235 Acked Write 10 Sent Done Done false
27860 8236 Acked Flush 0 Sent Done Done false
27861 8237 Acked Write 4 Sent Done Done false
27862 8238 Acked Write 2 Sent Done Done false
27863 8239 Acked Write 9 Sent Done Done false
27864 8240 Acked Flush 0 Sent Done Done false
27865 8241 Acked Write 9 Sent Done Done false
27866 8242 Acked Write 2 Sent Done Done false
27867 8243 Acked Flush 0 Sent Done Done false
27868 8244 Acked Read 8 Sent Done Done false
27869 8245 Acked Write 1 Sent Done Done false
27870 STATES DS:0 DS:1 DS:2 TOTAL
27871 Sent 301 0 0 301
27872 Done 0 301 301 602
27873 Skipped 0 0 0 0
27874 Error 0 0 0 0
27875 Last Flush: Some(JobId(7944)) Some(JobId(8243)) Some(JobId(8243))
27876 Downstairs last five completed: 7944 7943 7942 7941 7940
27877 Upstairs last five completed: 8245 8244 8243 8242 8241
27878 CLIENT: Up:0 ds:301 act:2
278792025-01-29T05:16:50.003ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
278802025-01-29T05:16:50.529ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
278812025-01-29T05:16:51.531ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
278822025-01-29T05:16:52.533ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
27883 ----------------------------------------------------------------
27884 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27885 States: Offline Active Active
27886 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27887 7945 Acked Read 5 Sent Done Done false
27888 7946 Acked Read 1 Sent Done Done false
27889 7947 Acked Write 4 Sent Done Done false
27890 7948 Acked Write 6 Sent Done Done false
27891 7949 Acked Write 6 Sent Done Done false
27892 7950 Acked Write 8 Sent Done Done false
27893 7951 Acked Read 9 Sent Done Done false
27894 7952 Acked Write 5 Sent Done Done false
27895 7953 Acked Read 7 Sent Done Done false
27896 7954 Acked Flush 0 Sent Done Done false
27897 7955 Acked Write 4 Sent Done Done false
27898 7956 Acked Flush 0 Sent Done Done false
27899 7957 Acked Flush 0 Sent Done Done false
27900 7958 Acked Flush 0 Sent Done Done false
27901 7959 Acked Write 10 Sent Done Done false
27902 7960 Acked Write 5 Sent Done Done false
27903 7961 Acked Read 6 Sent Done Done false
27904 7962 Acked Write 4 Sent Done Done false
27905 7963 Acked Write 9 Sent Done Done false
27906 7964 Acked Read 2 Sent Done Done false
27907 7965 Acked Read 10 Sent Done Done false
27908 7966 Acked Read 4 Sent Done Done false
27909 7967 Acked Flush 0 Sent Done Done false
27910 7968 Acked Write 8 Sent Done Done false
27911 7969 Acked Read 7 Sent Done Done false
27912 7970 Acked Write 4 Sent Done Done false
27913 7971 Acked Read 8 Sent Done Done false
27914 7972 Acked Read 6 Sent Done Done false
27915 7973 Acked Read 5 Sent Done Done false
27916 7974 Acked Write 5 Sent Done Done false
27917 7975 Acked Read 3 Sent Done Done false
27918 7976 Acked Read 10 Sent Done Done false
27919 7977 Acked Read 3 Sent Done Done false
27920 7978 Acked Read 1 Sent Done Done false
27921 7979 Acked Write 8 Sent Done Done false
27922 7980 Acked Read 8 Sent Done Done false
27923 7981 Acked Read 5 Sent Done Done false
27924 7982 Acked Read 6 Sent Done Done false
27925 7983 Acked Read 8 Sent Done Done false
27926 7984 Acked Read 1 Sent Done Done false
27927 7985 Acked Read 6 Sent Done Done false
27928 7986 Acked Write 1 Sent Done Done false
27929 7987 Acked Read 5 Sent Done Done false
27930 7988 Acked Read 4 Sent Done Done false
27931 7989 Acked Read 3 Sent Done Done false
27932 7990 Acked Read 6 Sent Done Done false
27933 7991 Acked Write 3 Sent Done Done false
27934 7992 Acked Write 7 Sent Done Done false
27935 7993 Acked Write 7 Sent Done Done false
27936 7994 Acked Read 10 Sent Done Done false
27937 7995 Acked Read 1 Sent Done Done false
27938 7996 Acked Write 1 Sent Done Done false
27939 7997 Acked Write 3 Sent Done Done false
27940 7998 Acked Write 5 Sent Done Done false
27941 7999 Acked Write 9 Sent Done Done false
27942 8000 Acked Write 10 Sent Done Done false
27943 8001 Acked Read 2 Sent Done Done false
27944 8002 Acked Write 10 Sent Done Done false
27945 8003 Acked Write 8 Sent Done Done false
27946 8004 Acked Read 5 Sent Done Done false
27947 8005 Acked Read 10 Sent Done Done false
27948 8006 Acked Write 6 Sent Done Done false
27949 8007 Acked Read 5 Sent Done Done false
27950 8008 Acked Flush 0 Sent Done Done false
27951 8009 Acked Read 3 Sent Done Done false
27952 8010 Acked Write 3 Sent Done Done false
27953 8011 Acked Write 2 Sent Done Done false
27954 8012 Acked Read 1 Sent Done Done false
27955 8013 Acked Read 10 Sent Done Done false
27956 8014 Acked Write 2 Sent Done Done false
27957 8015 Acked Read 1 Sent Done Done false
27958 8016 Acked Write 1 Sent Done Done false
27959 8017 Acked Read 3 Sent Done Done false
27960 8018 Acked Read 3 Sent Done Done false
27961 8019 Acked Read 10 Sent Done Done false
27962 8020 Acked Write 2 Sent Done Done false
27963 8021 Acked Flush 0 Sent Done Done false
27964 8022 Acked Flush 0 Sent Done Done false
27965 8023 Acked Write 6 Sent Done Done false
27966 8024 Acked Read 4 Sent Done Done false
27967 8025 Acked Write 6 Sent Done Done false
27968 8026 Acked Read 9 Sent Done Done false
27969 8027 Acked Read 8 Sent Done Done false
27970 8028 Acked Write 5 Sent Done Done false
27971 8029 Acked Read 7 Sent Done Done false
27972 8030 Acked Read 4 Sent Done Done false
27973 8031 Acked Write 4 Sent Done Done false
27974 8032 Acked Read 6 Sent Done Done false
27975 8033 Acked Write 4 Sent Done Done false
27976 8034 Acked Read 3 Sent Done Done false
27977 8035 Acked Read 4 Sent Done Done false
27978 8036 Acked Read 10 Sent Done Done false
27979 8037 Acked Flush 0 Sent Done Done false
27980 8038 Acked Read 2 Sent Done Done false
27981 8039 Acked Read 9 Sent Done Done false
27982 8040 Acked Read 8 Sent Done Done false
27983 8041 Acked Read 10 Sent Done Done false
27984 8042 Acked Read 7 Sent Done Done false
27985 8043 Acked Write 3 Sent Done Done false
27986 8044 Acked Write 5 Sent Done Done false
27987 8045 Acked Write 3 Sent Done Done false
27988 8046 Acked Read 6 Sent Done Done false
27989 8047 Acked Read 9 Sent Done Done false
27990 8048 Acked Read 8 Sent Done Done false
27991 8049 Acked Read 3 Sent Done Done false
27992 8050 Acked Flush 0 Sent Done Done false
27993 8051 Acked Write 3 Sent Done Done false
27994 8052 Acked Read 7 Sent Done Done false
27995 8053 Acked Write 3 Sent Done Done false
27996 8054 Acked Read 8 Sent Done Done false
27997 8055 Acked Read 8 Sent Done Done false
27998 8056 Acked Read 8 Sent Done Done false
27999 8057 Acked Write 6 Sent Done Done false
28000 8058 Acked Read 8 Sent Done Done false
28001 8059 Acked Read 5 Sent Done Done false
28002 8060 Acked Read 9 Sent Done Done false
28003 8061 Acked Write 2 Sent Done Done false
28004 8062 Acked Write 6 Sent Done Done false
28005 8063 Acked Write 5 Sent Done Done false
28006 8064 Acked Read 7 Sent Done Done false
28007 8065 Acked Read 2 Sent Done Done false
28008 8066 Acked Write 1 Sent Done Done false
28009 8067 Acked Write 1 Sent Done Done false
28010 8068 Acked Read 9 Sent Done Done false
28011 8069 Acked Read 5 Sent Done Done false
28012 8070 Acked Flush 0 Sent Done Done false
28013 8071 Acked Read 10 Sent Done Done false
28014 8072 Acked Write 2 Sent Done Done false
28015 8073 Acked Read 1 Sent Done Done false
28016 8074 Acked Write 6 Sent Done Done false
28017 8075 Acked Read 5 Sent Done Done false
28018 8076 Acked Write 2 Sent Done Done false
28019 8077 Acked Read 10 Sent Done Done false
28020 8078 Acked Write 7 Sent Done Done false
28021 8079 Acked Read 1 Sent Done Done false
28022 8080 Acked Write 7 Sent Done Done false
28023 8081 Acked Write 1 Sent Done Done false
28024 8082 Acked Flush 0 Sent Done Done false
28025 8083 Acked Read 10 Sent Done Done false
28026 8084 Acked Read 5 Sent Done Done false
28027 8085 Acked Read 7 Sent Done Done false
28028 8086 Acked Read 2 Sent Done Done false
28029 8087 Acked Write 2 Sent Done Done false
28030 8088 Acked Write 10 Sent Done Done false
28031 8089 Acked Write 6 Sent Done Done false
28032 8090 Acked Write 2 Sent Done Done false
28033 8091 Acked Read 8 Sent Done Done false
28034 8092 Acked Write 10 Sent Done Done false
28035 8093 Acked Write 8 Sent Done Done false
28036 8094 Acked Write 1 Sent Done Done false
28037 8095 Acked Read 3 Sent Done Done false
28038 8096 Acked Write 10 Sent Done Done false
28039 8097 Acked Flush 0 Sent Done Done false
28040 8098 Acked Read 3 Sent Done Done false
28041 8099 Acked Read 8 Sent Done Done false
28042 8100 Acked Read 6 Sent Done Done false
28043 8101 Acked Read 3 Sent Done Done false
28044 8102 Acked Read 7 Sent Done Done false
28045 8103 Acked Write 10 Sent Done Done false
28046 8104 Acked Write 5 Sent Done Done false
28047 8105 Acked Write 4 Sent Done Done false
28048 8106 Acked Read 7 Sent Done Done false
28049 8107 Acked Read 2 Sent Done Done false
28050 8108 Acked Read 4 Sent Done Done false
28051 8109 Acked Read 1 Sent Done Done false
28052 8110 Acked Write 9 Sent Done Done false
28053 8111 Acked Read 9 Sent Done Done false
28054 8112 Acked Write 6 Sent Done Done false
28055 8113 Acked Read 10 Sent Done Done false
28056 8114 Acked Read 4 Sent Done Done false
28057 8115 Acked Read 2 Sent Done Done false
28058 8116 Acked Read 6 Sent Done Done false
28059 8117 Acked Flush 0 Sent Done Done false
28060 8118 Acked Flush 0 Sent Done Done false
28061 8119 Acked Write 6 Sent Done Done false
28062 8120 Acked Write 7 Sent Done Done false
28063 8121 Acked Write 2 Sent Done Done false
28064 8122 Acked Flush 0 Sent Done Done false
28065 8123 Acked Write 3 Sent Done Done false
28066 8124 Acked Read 10 Sent Done Done false
28067 8125 Acked Read 2 Sent Done Done false
28068 8126 Acked Write 5 Sent Done Done false
28069 8127 Acked Flush 0 Sent Done Done false
28070 8128 Acked Read 10 Sent Done Done false
28071 8129 Acked Read 7 Sent Done Done false
28072 8130 Acked Write 4 Sent Done Done false
28073 8131 Acked Write 3 Sent Done Done false
28074 8132 Acked Write 4 Sent Done Done false
28075 8133 Acked Write 7 Sent Done Done false
28076 8134 Acked Read 7 Sent Done Done false
28077 8135 Acked Read 5 Sent Done Done false
28078 8136 Acked Flush 0 Sent Done Done false
28079 8137 Acked Write 9 Sent Done Done false
28080 8138 Acked Flush 0 Sent Done Done false
28081 8139 Acked Read 5 Sent Done Done false
28082 8140 Acked Read 3 Sent Done Done false
28083 8141 Acked Flush 0 Sent Done Done false
28084 8142 Acked Read 2 Sent Done Done false
28085 8143 Acked Flush 0 Sent Done Done false
28086 8144 Acked Read 4 Sent Done Done false
28087 8145 Acked Read 9 Sent Done Done false
28088 8146 Acked Read 10 Sent Done Done false
28089 8147 Acked Read 4 Sent Done Done false
28090 8148 Acked Write 9 Sent Done Done false
28091 8149 Acked Write 7 Sent Done Done false
28092 8150 Acked Write 8 Sent Done Done false
28093 8151 Acked Write 1 Sent Done Done false
28094 8152 Acked Write 1 Sent Done Done false
28095 8153 Acked Read 7 Sent Done Done false
28096 8154 Acked Write 2 Sent Done Done false
28097 8155 Acked Write 8 Sent Done Done false
28098 8156 Acked Read 5 Sent Done Done false
28099 8157 Acked Read 4 Sent Done Done false
28100 8158 Acked Write 1 Sent Done Done false
28101 8159 Acked Read 9 Sent Done Done false
28102 8160 Acked Read 6 Sent Done Done false
28103 8161 Acked Read 2 Sent Done Done false
28104 8162 Acked Read 10 Sent Done Done false
28105 8163 Acked Read 7 Sent Done Done false
28106 8164 Acked Read 4 Sent Done Done false
28107 8165 Acked Read 6 Sent Done Done false
28108 8166 Acked Write 9 Sent Done Done false
28109 8167 Acked Read 7 Sent Done Done false
28110 8168 Acked Write 4 Sent Done Done false
28111 8169 Acked Write 3 Sent Done Done false
28112 8170 Acked Read 5 Sent Done Done false
28113 8171 Acked Write 10 Sent Done Done false
28114 8172 Acked Read 2 Sent Done Done false
28115 8173 Acked Write 1 Sent Done Done false
28116 8174 Acked Read 9 Sent Done Done false
28117 8175 Acked Flush 0 Sent Done Done false
28118 8176 Acked Read 6 Sent Done Done false
28119 8177 Acked Read 4 Sent Done Done false
28120 8178 Acked Read 5 Sent Done Done false
28121 8179 Acked Read 7 Sent Done Done false
28122 8180 Acked Read 5 Sent Done Done false
28123 8181 Acked Write 7 Sent Done Done false
28124 8182 Acked Read 1 Sent Done Done false
28125 8183 Acked Read 6 Sent Done Done false
28126 8184 Acked Read 9 Sent Done Done false
28127 8185 Acked Write 1 Sent Done Done false
28128 8186 Acked Write 10 Sent Done Done false
28129 8187 Acked Read 7 Sent Done Done false
28130 8188 Acked Write 6 Sent Done Done false
28131 8189 Acked Write 4 Sent Done Done false
28132 8190 Acked Read 5 Sent Done Done false
28133 8191 Acked Read 2 Sent Done Done false
28134 8192 Acked Write 10 Sent Done Done false
28135 8193 Acked Read 6 Sent Done Done false
28136 8194 Acked Write 10 Sent Done Done false
28137 8195 Acked Write 1 Sent Done Done false
28138 8196 Acked Read 1 Sent Done Done false
28139 8197 Acked Read 4 Sent Done Done false
28140 8198 Acked Write 7 Sent Done Done false
28141 8199 Acked Read 4 Sent Done Done false
28142 8200 Acked Read 5 Sent Done Done false
28143 8201 Acked Write 9 Sent Done Done false
28144 8202 Acked Write 5 Sent Done Done false
28145 8203 Acked Write 6 Sent Done Done false
28146 8204 Acked Write 2 Sent Done Done false
28147 8205 Acked Write 7 Sent Done Done false
28148 8206 Acked Read 10 Sent Done Done false
28149 8207 Acked Read 6 Sent Done Done false
28150 8208 Acked Write 5 Sent Done Done false
28151 8209 Acked Read 10 Sent Done Done false
28152 8210 Acked Read 6 Sent Done Done false
28153 8211 Acked Read 8 Sent Done Done false
28154 8212 Acked Write 10 Sent Done Done false
28155 8213 Acked Write 9 Sent Done Done false
28156 8214 Acked Read 6 Sent Done Done false
28157 8215 Acked Read 4 Sent Done Done false
28158 8216 Acked Write 3 Sent Done Done false
28159 8217 Acked Write 7 Sent Done Done false
28160 8218 Acked Read 7 Sent Done Done false
28161 8219 Acked Write 10 Sent Done Done false
28162 8220 Acked Flush 0 Sent Done Done false
28163 8221 Acked Read 4 Sent Done Done false
28164 8222 Acked Read 7 Sent Done Done false
28165 8223 Acked Read 6 Sent Done Done false
28166 8224 Acked Read 10 Sent Done Done false
28167 8225 Acked Read 7 Sent Done Done false
28168 8226 Acked Write 4 Sent Done Done false
28169 8227 Acked Read 4 Sent Done Done false
28170 8228 Acked Write 3 Sent Done Done false
28171 8229 Acked Read 4 Sent Done Done false
28172 8230 Acked Write 2 Sent Done Done false
28173 8231 Acked Read 4 Sent Done Done false
28174 8232 Acked Flush 0 Sent Done Done false
28175 8233 Acked Flush 0 Sent Done Done false
28176 8234 Acked Flush 0 Sent Done Done false
28177 8235 Acked Write 10 Sent Done Done false
28178 8236 Acked Flush 0 Sent Done Done false
28179 8237 Acked Write 4 Sent Done Done false
28180 8238 Acked Write 2 Sent Done Done false
28181 8239 Acked Write 9 Sent Done Done false
28182 8240 Acked Flush 0 Sent Done Done false
28183 8241 Acked Write 9 Sent Done Done false
28184 8242 Acked Write 2 Sent Done Done false
28185 8243 Acked Flush 0 Sent Done Done false
28186 8244 Acked Read 8 Sent Done Done false
28187 8245 Acked Write 1 Sent Done Done false
28188 STATES DS:0 DS:1 DS:2 TOTAL
28189 Sent 301 0 0 301
28190 Done 0 301 301 602
28191 Skipped 0 0 0 0
28192 Error 0 0 0 0
28193 Last Flush: Some(JobId(7944)) Some(JobId(8243)) Some(JobId(8243))
28194 Downstairs last five completed: 7944 7943 7942 7941 7940
28195 Upstairs last five completed: 8245 8244 8243 8242 8241
28196 CLIENT: Up:0 ds:301 act:2
281972025-01-29T05:16:54.005ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
281982025-01-29T05:16:54.537ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
281992025-01-29T05:16:55.539ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282002025-01-29T05:16:56.541ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282012025-01-29T05:16:57.253ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282022025-01-29T05:16:57.253ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35469) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282032025-01-29T05:16:57.254ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282042025-01-29T05:16:57.255ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282052025-01-29T05:16:57.255ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282062025-01-29T05:16:57.255ZINFOcrucible: send last flush ID to this DS: Some(JobId(7944)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282072025-01-29T05:16:57.255ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282082025-01-29T05:16:57.255ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7944)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282092025-01-29T05:16:57.255ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7944)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
28210 ----------------------------------------------------------------
28211 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
28212 States: Active Active Active
28213 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28214 8244 Acked Read 8 Done Done Done true
28215 8245 Acked Write 1 Done Done Done true
28216 STATES DS:0 DS:1 DS:2 TOTAL
28217 Sent 0 0 0 0
28218 Done 2 2 2 6
28219 Skipped 0 0 0 0
28220 Error 0 0 0 0
28221 Last Flush: Some(JobId(8243)) Some(JobId(8243)) Some(JobId(8243))
28222 Downstairs last five completed: 8243 8242 8241 8240 8239
28223 Upstairs last five completed: 8245 8244 8243 8242 8241
28224 CLIENT: Up:0 ds:2 act:3
28225 ----------------------------------------------------------------
28226 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28227 Downstairs last five completed: 8246 8245 8244 8243 8242
28228 Upstairs last five completed: 8246 8245 8244 8243 8242
28229 CLIENT: Up:0 ds:0 act:3
28230 Replay: All jobs finished
28231 Main task has work to do, go find it
28232 got dsc Stop(0)
28233 stop 0
28234 Jan 29 05:17:01.508 INFO [8810] Got stop action so:false kr:false
28235 request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a2c8d162-9c01-4fbf-aca5-67c7a2d0dc01, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
282362025-01-29T05:17:01.508ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282372025-01-29T05:17:01.508ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
282382025-01-29T05:17:01.509ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
28239 Jan 29 05:17:01.509 INFO request completed{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:17:01.509062324Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"0","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
28240 , latency_us: 151, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9b9d7e4f-ab1d-4630-8410-ba61ce88674e, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
28241 [8810] Exited with: ExitStatus(unix_wait_status(9))
28242 [8810][0] reports Exit
28243 Jan 29 05:17:05.511 INFO request completed, latency_us: 384, response_code: 200, uri: /state/cid/0, method: GET, req_id: d54f2ff4-cedd-4960-9dab-832906b35800, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
28244 001/300 Read block 2135 len 40960
28245 002/300 Read block 2633 len 32768
28246 003/300 Flush
28247 004/300 Write block 4180 len 16384 data: 5 5 6 7
28248 005/300 Read block 4555 len 16384
28249 006/300 Write block 2062 len 24576 data: 6 4 2 2 3 5
28250 007/300 Write block 271 len 20480 data: 4 4 3 4 3
28251 008/300 Write block 4926 len 32768 data: 2 3 4 6 7 7 8 7
28252 009/300 Read block 1773 len 20480
28253 010/300 Write block 1958 len 12288 data: 3 4 4
28254 011/300 Read block 2209 len 36864
28255 012/300 Flush
28256 013/300 Write block 4010 len 8192 data: 7 7
28257 014/300 Write block 4359 len 12288 data: 4 6 5
28258 015/300 Write block 3740 len 16384 data: 2 4 4 4
28259 016/300 Read block 222 len 20480
28260 017/300 Read block 2771 len 40960
28261 018/300 Write block 1286 len 40960 data: 8 7 7 6 4 4 5 5 5 4
28262 019/300 Read block 2643 len 20480
28263 020/300 Read block 751 len 8192
28264 021/300 Write block 2028 len 12288 data: 5 6 5
28265 022/300 Read block 2082 len 24576
28266 023/300 Flush
28267 024/300 Read block 4943 len 4096
28268 025/300 Read block 4051 len 16384
28269 026/300 Read block 2909 len 4096
28270 027/300 Read block 710 len 28672
28271 028/300 Write block 1562 len 32768 data: 5 5 4 4 5 6 7 7
28272 029/300 Flush
28273 030/300 Write block 4085 len 12288 data: 3 3 4
28274 031/300 Write block 4568 len 16384 data: 6 7 8 7
28275 032/300 Read block 2851 len 12288
28276 033/300 Write block 499 len 36864 data: 8 8 6 9 8 8 8 8 10
28277 034/300 Read block 4342 len 32768
28278 035/300 Read block 3722 len 12288
28279 036/300 Read block 2712 len 20480
28280 037/300 Read block 3547 len 12288
28281 038/300 Read block 4247 len 8192
28282 039/300 Read block 4131 len 40960
28283 040/300 Read block 1033 len 36864
28284 041/300 Write block 4299 len 20480 data: 6 5 5 3 5
28285 042/300 Write block 236 len 8192 data: 6 9
28286 043/300 Read block 863 len 28672
28287 044/300 Write block 3201 len 16384 data: 5 6 6 6
28288 045/300 Read block 1867 len 28672
28289 046/300 Write block 2538 len 8192 data: 3 3
28290 047/300 Read block 3496 len 36864
28291 048/300 Read block 3500 len 8192
28292 049/300 Read block 438 len 28672
28293 050/300 Read block 2583 len 40960
28294 051/300 Read block 59 len 4096
28295 052/300 Read block 4634 len 40960
28296 053/300 Write block 856 len 40960 data: 4 4 3 3 3 5 7 6 7 6
28297 054/300 Read block 3818 len 8192
28298 055/300 Write block 1389 len 40960 data: 4 4 4 2 3 3 3 3 4 5
28299 056/300 Write block 1253 len 4096 data: 4
28300 057/300 Read block 2208 len 8192
28301 058/300 Read block 4813 len 20480
28302 059/300 Write block 2091 len 28672 data: 4 5 5 5 3 4 5
28303 060/300 Read block 1304 len 40960
28304 061/300 Read block 3418 len 12288
28305 062/300 Read block 4205 len 12288
28306 063/300 Read block 2358 len 24576
28307 064/300 Read block 1425 len 20480
28308 065/300 Write block 811 len 20480 data: 6 8 8 8 7
28309 066/300 Flush
28310 067/300 Read block 3358 len 12288
28311 068/300 Write block 3613 len 28672 data: 8 8 6 4 4 3 4
28312 069/300 Write block 1779 len 40960 data: 8 6 4 4 5 5 4 3 6 7
28313 070/300 Write block 495 len 24576 data: 9 10 9 10 9 9
28314 071/300 Read block 601 len 28672
28315 072/300 Flush
28316 073/300 Read block 3499 len 20480
28317 074/300 Read block 401 len 24576
28318 075/300 Read block 3232 len 12288
28319 076/300 Write block 526 len 20480 data: 3 3 3 3 3
28320 077/300 Write block 4814 len 12288 data: 4 5 5
28321 078/300 Flush
28322 079/300 Write block 467 len 4096 data: 5
28323 080/300 Flush
28324 081/300 Read block 4064 len 4096
28325 082/300 Read block 4686 len 20480
28326 083/300 Flush
28327 084/300 Read block 4941 len 8192
28328 085/300 Read block 3871 len 8192
28329 086/300 Write block 2595 len 16384 data: 5 5 5 5
28330 087/300 Read block 4720 len 36864
28331 088/300 Write block 678 len 12288 data: 3 3 2
28332 089/300 Read block 2894 len 20480
28333 090/300 Write block 1244 len 36864 data: 6 6 4 5 3 4 4 5 4
28334 091/300 Write block 3986 len 20480 data: 6 6 6 7 7
28335 092/300 Write block 2388 len 28672 data: 2 2 2 2 2 2 2
28336 093/300 Read block 1190 len 4096
28337 094/300 Flush
28338 095/300 Flush
28339 096/300 Write block 1817 len 24576 data: 4 4 5 5 5 4
28340 097/300 Write block 767 len 4096 data: 7
28341 098/300 Write block 4239 len 8192 data: 7 6
28342 099/300 Read block 4307 len 40960
28343 100/300 Write block 978 len 8192 data: 4 6
28344 101/300 Write block 3334 len 24576 data: 5 5 4 5 5 4
28345 102/300 Read block 2074 len 16384
28346 103/300 Write block 4709 len 32768 data: 6 6 5 6 6 6 7 8
28347 104/300 Read block 2856 len 16384
28348 105/300 Read block 2745 len 36864
28349 106/300 Write block 2712 len 20480 data: 4 5 5 4 6
28350 107/300 Read block 3556 len 24576
28351 108/300 Write block 2157 len 16384 data: 4 5 4 3
28352 109/300 Read block 1141 len 8192
28353 110/300 Write block 2908 len 16384 data: 5 5 5 6
28354 111/300 Read block 3272 len 32768
28355 112/300 Read block 2488 len 4096
28356 113/300 Read block 677 len 32768
28357 114/300 Read block 2885 len 16384
28358 115/300 Flush
28359 116/300 Read block 49 len 20480
28360 117/300 Read block 2130 len 12288
28361 118/300 Write block 3174 len 8192 data: 6 5
28362 119/300 Write block 3099 len 4096 data: 4
28363 120/300 Flush
28364 121/300 Write block 1942 len 20480 data: 6 5 5 4 4
28365 122/300 Write block 4308 len 8192 data: 8 7
28366 123/300 Write block 4154 len 12288 data: 3 4 4
28367 124/300 Read block 485 len 28672
28368 125/300 Read block 1776 len 32768
28369 126/300 Read block 118 len 36864
28370 127/300 Read block 4101 len 8192
28371 128/300 Read block 2274 len 16384
28372 129/300 Write block 1480 len 32768 data: 3 3 3 3 3 4 2 3
28373 130/300 Write block 1936 len 36864 data: 9 10 8 9 10 8 7 6 6
28374 131/300 Read block 4228 len 32768
28375 132/300 Write block 873 len 32768 data: 8 7 6 6 7 6 6 8
28376 133/300 Write block 3632 len 8192 data: 5 4
28377 134/300 Write block 2744 len 12288 data: 5 6 6
28378 135/300 Write block 3222 len 32768 data: 8 8 5 5 4 4 7 7
28379 136/300 Write block 4321 len 40960 data: 5 5 5 5 5 5 5 5 7 7
28380 137/300 Read block 3809 len 36864
28381 138/300 Read block 2491 len 16384
28382 139/300 Write block 1200 len 20480 data: 5 5 3 3 2
28383 140/300 Write block 755 len 20480 data: 7 6 6 5 5
28384 141/300 Write block 4285 len 24576 data: 4 4 4 5 6 7
28385 142/300 Write block 4282 len 32768 data: 7 6 6 5 5 5 6 7
28386 143/300 Read block 1775 len 8192
28387 144/300 Write block 2192 len 28672 data: 2 4 4 3 3 3 4
28388 145/300 Read block 2497 len 16384
28389 146/300 Read block 1071 len 28672
28390 147/300 Read block 955 len 12288
28391 148/300 Read block 3349 len 20480
28392 149/300 Read block 325 len 4096
28393 150/300 Read block 4670 len 8192
28394 151/300 Read block 760 len 36864
28395 152/300 Read block 187 len 12288
28396 153/300 Flush
28397 154/300 Write block 1701 len 36864 data: 4 4 6 6 5 5 5 5 5
28398 155/300 Read block 362 len 12288
28399 156/300 Write block 1635 len 36864 data: 3 3 4 5 5 3 4 4 4
28400 157/300 Read block 1450 len 8192
28401 158/300 Flush
28402 159/300 Write block 657 len 8192 data: 4 4
28403 160/300 Read block 3937 len 36864
28404 161/300 Read block 77 len 36864
28405 162/300 Read block 3704 len 4096
28406 163/300 Read block 1164 len 24576
28407 164/300 Write block 2083 len 12288 data: 5 6 6
28408 165/300 Write block 80 len 8192 data: 5 5
28409 166/300 Read block 484 len 20480
28410 167/300 Read block 1085 len 20480
28411 168/300 Read block 3926 len 4096
28412 169/300 Write block 3390 len 12288 data: 6 6 5
28413 170/300 Write block 1510 len 16384 data: 7 6 5 5
28414 171/300 Write block 1423 len 20480 data: 5 6 6 6 6
28415 172/300 Read block 2339 len 40960
28416 173/300 Read block 1172 len 36864
28417 174/300 Read block 2465 len 20480
28418 175/300 Write block 2091 len 32768 data: 5 6 6 6 4 5 6 5
28419 176/300 Write block 4530 len 36864 data: 9 9 9 10 11 9 6 6 5
28420 177/300 Write block 3619 len 28672 data: 5 4 2 2 2 2 2
28421 178/300 Read block 2803 len 16384
28422 179/300 Flush
28423 180/300 Read block 2807 len 40960
28424 181/300 Read block 4216 len 40960
28425 182/300 Read block 3925 len 36864
28426 183/300 Write block 2548 len 20480 data: 8 9 8 6 7
28427 184/300 Read block 2629 len 32768
28428 185/300 Write block 533 len 32768 data: 4 4 4 4 4 4 5 5
28429 186/300 Read block 3593 len 28672
28430 187/300 Read block 4379 len 24576
28431 188/300 Write block 60 len 32768 data: 6 7 6 5 5 5 6 5
28432 189/300 Read block 855 len 16384
28433 190/300 Write block 3646 len 20480 data: 7 7 7 7 7
28434 191/300 Write block 990 len 16384 data: 2 2 3 4
28435 192/300 Read block 2657 len 28672
28436 193/300 Read block 161 len 12288
28437 194/300 Read block 2842 len 20480
28438 195/300 Write block 2481 len 40960 data: 6 6 6 5 5 8 5 5 6 6
28439 196/300 Read block 1380 len 8192
28440 197/300 Flush
28441 198/300 Read block 976 len 32768
28442 199/300 Read block 4203 len 40960
28443 200/300 Write block 407 len 12288 data: 5 5 5
28444 201/300 Write block 2000 len 28672 data: 6 6 7 7 6 9 9
28445 202/300 Read block 3869 len 28672
28446 203/300 Write block 1936 len 40960 data: 10 11 9 10 11 9 8 7 7 5
28447 204/300 Write block 3282 len 40960 data: 4 5 5 5 5 5 5 5 5 4
28448 205/300 Flush
28449 206/300 Read block 2506 len 32768
28450 207/300 Flush
28451 208/300 Write block 2959 len 36864 data: 4 6 7 7 7 6 6 6 5
28452 209/300 Read block 4612 len 36864
28453 210/300 Read block 1487 len 24576
28454 211/300 Read block 2109 len 4096
28455 212/300 Write block 3146 len 12288 data: 5 6 6
28456 213/300 Read block 2938 len 20480
28457 214/300 Read block 3564 len 20480
28458 215/300 Write block 1386 len 40960 data: 6 6 6 5 5 5 3 4 4 4
28459 216/300 Flush
28460 217/300 Write block 4033 len 4096 data: 5
28461 218/300 Write block 1403 len 12288 data: 7 7 6
28462 219/300 Write block 262 len 24576 data: 2 3 3 3 3 4
28463 220/300 Write block 3147 len 40960 data: 7 7 6 4 3 3 4 5 5 5
28464 221/300 Read block 1014 len 8192
28465 222/300 Read block 1371 len 8192
28466 223/300 Read block 4653 len 4096
28467 224/300 Write block 1279 len 36864 data: 3 5 6 6 7 7 8 9 8
28468 225/300 Read block 1810 len 28672
28469 226/300 Read block 1477 len 40960
28470 227/300 Write block 4318 len 8192 data: 3 3
28471 228/300 Read block 4407 len 16384
28472 229/300 Read block 2648 len 20480
28473 230/300 Read block 3420 len 40960
28474 231/300 Read block 2389 len 36864
28475 232/300 Write block 4841 len 16384 data: 7 6 5 6
28476 233/300 Read block 1821 len 16384
28477 234/300 Read block 2803 len 16384
28478 235/300 Write block 321 len 8192 data: 3 5
28479 236/300 Flush
28480 237/300 Read block 1710 len 20480
28481 238/300 Read block 2424 len 12288
28482 239/300 Read block 2494 len 24576
28483 240/300 Read block 1127 len 20480
28484 241/300 Read block 420 len 16384
28485 242/300 Write block 2050 len 24576 data: 4 6 6 6 6 5
28486 243/300 Read block 3645 len 32768
28487 244/300 Read block 1211 len 20480
28488 245/300 Read block 992 len 4096
28489 246/300 Read block 2131 len 20480
28490 247/300 Read block 3766 len 32768
28491 248/300 Read block 4667 len 36864
28492 249/300 Read block 1047 len 24576
28493 250/300 Read block 4687 len 8192
28494 251/300 Read block 3588 len 12288
28495 252/300 Flush
28496 253/300 Read block 704 len 4096
28497 254/300 Write block 4857 len 32768 data: 7 5 6 5 4 4 4 4
28498 255/300 Read block 2778 len 20480
28499 256/300 Flush
28500 257/300 Write block 3229 len 20480 data: 8 8 8 8 9
28501 258/300 Read block 1594 len 12288
28502 259/300 Write block 4353 len 36864 data: 6 4 3 5 5 5 5 7 6
28503 260/300 Write block 4883 len 32768 data: 4 5 5 8 7 7 7 7
28504 261/300 Write block 2471 len 32768 data: 5 4 4 4 4 6 6 6
28505 262/300 Read block 1881 len 40960
28506 263/300 Write block 2797 len 12288 data: 3 4 3
28507 264/300 Write block 1077 len 40960 data: 6 6 6 6 6 5 8 5 5 5
28508 265/300 Read block 1829 len 12288
28509 266/300 Flush
28510 267/300 Write block 4653 len 36864 data: 6 5 6 7 6 6 7 6 6
28511 268/300 Read block 3991 len 28672
28512 269/300 Write block 3413 len 8192 data: 2 3
28513 270/300 Read block 276 len 24576
28514 271/300 Write block 930 len 40960 data: 5 6 5 5 4 4 2 2 2 2
28515 272/300 Read block 3418 len 40960
28516 273/300 Write block 594 len 24576 data: 8 8 6 5 5 3
28517 274/300 Read block 2219 len 40960
28518 275/300 Read block 1103 len 12288
28519 276/300 Write block 4193 len 20480 data: 5 5 5 6 7
28520 277/300 Flush
28521 278/300 Write block 4420 len 4096 data: 4
28522 279/300 Write block 726 len 28672 data: 5 6 7 6 5 5 8
28523 280/300 Write block 4014 len 20480 data: 4 3 4 3 5
28524 281/300 Write block 4064 len 28672 data: 3 3 4 4 3 4 5
28525 282/300 Write block 2491 len 24576 data: 6 7 6 4 4 3
28526 283/300 Write block 3817 len 32768 data: 6 5 6 6 6 6 5 5
28527 284/300 Write block 3929 len 8192 data: 5 6
28528 285/300 Write block 4798 len 8192 data: 3 3
28529 286/300 Write block 2344 len 40960 data: 5 5 6 5 4 4 4 4 4 3
28530 287/300 Flush
28531 288/300 Write block 3971 len 36864 data: 7 6 7 7 8 7 8 8 8
28532 289/300 Write block 4258 len 28672 data: 4 3 3 2 2 2 2
28533 290/300 Read block 1444 len 40960
28534 291/300 Flush
28535 292/300 Write block 2664 len 20480 data: 5 7 5 4 5
28536 293/300 Write block 4669 len 8192 data: 4 4
28537 294/300 Read block 3492 len 36864
28538 295/300 Flush
28539 296/300 Write block 2446 len 24576 data: 4 4 5 4 4 4
28540 297/300 Read block 286 len 40960
28541 298/300 Read block 624 len 36864
28542 299/300 Flush
28543 300/300 Read block 1443 len 8192
28544 301/300 Read block 45 len 16384
28545 Main task has work to do, go find it
28546 got dsc Start(0)
28547 start 0
28548 Jan 29 05:17:05.755 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/0, method: POST, req_id: 58796997-3188-4d6e-a963-3e6046d48b37, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
28549 Replay: started 0, returned:Ok(())
28550 ----------------------------------------------------------------
28551 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28552 States: Offline Active Active
28553 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28554 8247 Acked Read 10 Sent Done Done false
28555 8248 Acked Read 8 Sent Done Done false
28556 8249 Acked Flush 0 Sent Done Done false
28557 8250 Acked Write 4 Sent Done Done false
28558 8251 Acked Read 4 Sent Done Done false
28559 8252 Acked Write 6 Sent Done Done false
28560 8253 Acked Write 5 Sent Done Done false
28561 8254 Acked Write 8 Sent Done Done false
28562 8255 Acked Read 5 Sent Done Done false
28563 8256 Acked Write 3 Sent Done Done false
28564 8257 Acked Read 9 Sent Done Done false
28565 8258 Acked Flush 0 Sent Done Done false
28566 8259 Acked Write 2 Sent Done Done false
28567 8260 Acked Write 3 Sent Done Done false
28568 8261 Acked Write 4 Sent Done Done false
28569 8262 Acked Read 5 Sent Done Done false
28570 8263 Acked Read 10 Sent Done Done false
28571 8264 Acked Write 10 Sent Done Done false
28572 8265 Acked Read 5 Sent Done Done false
28573 8266 Acked Read 2 Sent Done Done false
28574 8267 Acked Write 3 Sent Done Done false
28575 8268 Acked Read 6 Sent Done Done false
28576 8269 Acked Flush 0 Sent Done Done false
28577 8270 Acked Read 1 Sent Done Done false
28578 8271 Acked Read 4 Sent Done Done false
28579 8272 Acked Read 1 Sent Done Done false
28580 8273 Acked Read 7 Sent Done Done false
28581 8274 Acked Write 8 Sent Done Done false
28582 8275 Acked Flush 0 Sent Done Done false
28583 8276 Acked Write 3 Sent Done Done false
28584 8277 Acked Write 4 Sent Done Done false
28585 8278 Acked Read 3 Sent Done Done false
28586 8279 Acked Write 9 Sent Done Done false
28587 8280 Acked Read 8 Sent Done Done false
28588 8281 Acked Read 3 Sent Done Done false
28589 8282 Acked Read 5 Sent Done Done false
28590 8283 Acked Read 3 Sent Done Done false
28591 8284 Acked Read 2 Sent Done Done false
28592 8285 Acked Read 10 Sent Done Done false
28593 8286 Acked Read 9 Sent Done Done false
28594 8287 Acked Write 5 Sent Done Done false
28595 8288 Acked Write 2 Sent Done Done false
28596 8289 Acked Read 7 Sent Done Done false
28597 8290 Acked Write 4 Sent Done Done false
28598 8291 Acked Read 7 Sent Done Done false
28599 8292 Acked Write 2 Sent Done Done false
28600 8293 Acked Read 9 Sent Done Done false
28601 8294 Acked Read 2 Sent Done Done false
28602 8295 Acked Read 7 Sent Done Done false
28603 8296 Acked Read 10 Sent Done Done false
28604 8297 Acked Read 1 Sent Done Done false
28605 8298 Acked Read 10 Sent Done Done false
28606 8299 Acked Write 10 Sent Done Done false
28607 8300 Acked Read 2 Sent Done Done false
28608 8301 Acked Write 10 Sent Done Done false
28609 8302 Acked Write 1 Sent Done Done false
28610 8303 Acked Read 2 Sent Done Done false
28611 8304 Acked Read 5 Sent Done Done false
28612 8305 Acked Write 7 Sent Done Done false
28613 8306 Acked Read 10 Sent Done Done false
28614 8307 Acked Read 3 Sent Done Done false
28615 8308 Acked Read 3 Sent Done Done false
28616 8309 Acked Read 6 Sent Done Done false
28617 8310 Acked Read 5 Sent Done Done false
28618 8311 Acked Write 5 Sent Done Done false
28619 8312 Acked Flush 0 Sent Done Done false
28620 8313 Acked Read 3 Sent Done Done false
28621 8314 Acked Write 7 Sent Done Done false
28622 8315 Acked Write 10 Sent Done Done false
28623 8316 Acked Write 6 Sent Done Done false
28624 8317 Acked Read 7 Sent Done Done false
28625 8318 Acked Flush 0 Sent Done Done false
28626 8319 Acked Read 5 Sent Done Done false
28627 8320 Acked Read 6 Sent Done Done false
28628 8321 Acked Read 3 Sent Done Done false
28629 8322 Acked Write 5 Sent Done Done false
28630 8323 Acked Write 3 Sent Done Done false
28631 8324 Acked Flush 0 Sent Done Done false
28632 8325 Acked Write 1 Sent Done Done false
28633 8326 Acked Flush 0 Sent Done Done false
28634 8327 Acked Read 1 Sent Done Done false
28635 8328 Acked Read 5 Sent Done Done false
28636 8329 Acked Flush 0 Sent Done Done false
28637 8330 Acked Read 2 Sent Done Done false
28638 8331 Acked Read 2 Sent Done Done false
28639 8332 Acked Write 4 Sent Done Done false
28640 8333 Acked Read 9 Sent Done Done false
28641 8334 Acked Write 3 Sent Done Done false
28642 8335 Acked Read 5 Sent Done Done false
28643 8336 Acked Write 9 Sent Done Done false
28644 8337 Acked Write 5 Sent Done Done false
28645 8338 Acked Write 7 Sent Done Done false
28646 8339 Acked Read 1 Sent Done Done false
28647 8340 Acked Flush 0 Sent Done Done false
28648 8341 Acked Flush 0 Sent Done Done false
28649 8342 Acked Write 6 Sent Done Done false
28650 8343 Acked Write 1 Sent Done Done false
28651 8344 Acked Write 2 Sent Done Done false
28652 8345 Acked Read 10 Sent Done Done false
28653 8346 Acked Write 2 Sent Done Done false
28654 8347 Acked Write 6 Sent Done Done false
28655 8348 Acked Read 4 Sent Done Done false
28656 8349 Acked Write 8 Sent Done Done false
28657 8350 Acked Read 4 Sent Done Done false
28658 8351 Acked Read 9 Sent Done Done false
28659 8352 Acked Write 5 Sent Done Done false
28660 8353 Acked Read 6 Sent Done Done false
28661 8354 Acked Write 4 Sent Done Done false
28662 8355 Acked Read 2 Sent Done Done false
28663 8356 Acked Write 4 Sent Done Done false
28664 8357 Acked Read 8 Sent Done Done false
28665 8358 Acked Read 1 Sent Done Done false
28666 8359 Acked Read 8 Sent Done Done false
28667 8360 Acked Read 4 Sent Done Done false
28668 8361 Acked Flush 0 Sent Done Done false
28669 8362 Acked Read 5 Sent Done Done false
28670 8363 Acked Read 3 Sent Done Done false
28671 8364 Acked Write 2 Sent Done Done false
28672 8365 Acked Write 1 Sent Done Done false
28673 8366 Acked Flush 0 Sent Done Done false
28674 8367 Acked Write 5 Sent Done Done false
28675 8368 Acked Write 2 Sent Done Done false
28676 8369 Acked Write 3 Sent Done Done false
28677 8370 Acked Read 7 Sent Done Done false
28678 8371 Acked Read 8 Sent Done Done false
28679 8372 Acked Read 9 Sent Done Done false
28680 8373 Acked Read 2 Sent Done Done false
28681 8374 Acked Read 4 Sent Done Done false
28682 8375 Acked Write 8 Sent Done Done false
28683 8376 Acked Write 9 Sent Done Done false
28684 8377 Acked Read 8 Sent Done Done false
28685 8378 Acked Write 8 Sent Done Done false
28686 8379 Acked Write 2 Sent Done Done false
28687 8380 Acked Write 3 Sent Done Done false
28688 8381 Acked Write 8 Sent Done Done false
28689 8382 Acked Write 10 Sent Done Done false
28690 8383 Acked Read 9 Sent Done Done false
28691 8384 Acked Read 4 Sent Done Done false
28692 8385 Acked Write 5 Sent Done Done false
28693 8386 Acked Write 5 Sent Done Done false
28694 8387 Acked Write 6 Sent Done Done false
28695 8388 Acked Write 8 Sent Done Done false
28696 8389 Acked Read 2 Sent Done Done false
28697 8390 Acked Write 7 Sent Done Done false
28698 8391 Acked Read 4 Sent Done Done false
28699 8392 Acked Read 7 Sent Done Done false
28700 8393 Acked Read 3 Sent Done Done false
28701 8394 Acked Read 5 Sent Done Done false
28702 8395 Acked Read 1 Sent Done Done false
28703 8396 Acked Read 2 Sent Done Done false
28704 8397 Acked Read 9 Sent Done Done false
28705 8398 Acked Read 3 Sent Done Done false
28706 8399 Acked Flush 0 Sent Done Done false
28707 8400 Acked Write 9 Sent Done Done false
28708 8401 Acked Read 3 Sent Done Done false
28709 8402 Acked Write 9 Sent Done Done false
28710 8403 Acked Read 2 Sent Done Done false
28711 8404 Acked Flush 0 Sent Done Done false
28712 8405 Acked Write 2 Sent Done Done false
28713 8406 Acked Read 9 Sent Done Done false
28714 8407 Acked Read 9 Sent Done Done false
28715 8408 Acked Read 1 Sent Done Done false
28716 8409 Acked Read 6 Sent Done Done false
28717 8410 Acked Write 3 Sent Done Done false
28718 8411 Acked Write 2 Sent Done Done false
28719 8412 Acked Read 5 Sent Done Done false
28720 8413 Acked Read 5 Sent Done Done false
28721 8414 Acked Read 1 Sent Done Done false
28722 8415 Acked Write 3 Sent Done Done false
28723 8416 Acked Write 4 Sent Done Done false
28724 8417 Acked Write 5 Sent Done Done false
28725 8418 Acked Read 10 Sent Done Done false
28726 8419 Acked Read 9 Sent Done Done false
28727 8420 Acked Read 5 Sent Done Done false
28728 8421 Acked Write 8 Sent Done Done false
28729 8422 Acked Write 9 Sent Done Done false
28730 8423 Acked Write 7 Sent Done Done false
28731 8424 Acked Read 4 Sent Done Done false
28732 8425 Acked Flush 0 Sent Done Done false
28733 8426 Acked Read 10 Sent Done Done false
28734 8427 Acked Read 10 Sent Done Done false
28735 8428 Acked Read 9 Sent Done Done false
28736 8429 Acked Write 5 Sent Done Done false
28737 8430 Acked Read 8 Sent Done Done false
28738 8431 Acked Write 8 Sent Done Done false
28739 8432 Acked Read 7 Sent Done Done false
28740 8433 Acked Read 6 Sent Done Done false
28741 8434 Acked Write 8 Sent Done Done false
28742 8435 Acked Read 4 Sent Done Done false
28743 8436 Acked Write 5 Sent Done Done false
28744 8437 Acked Write 4 Sent Done Done false
28745 8438 Acked Read 7 Sent Done Done false
28746 8439 Acked Read 3 Sent Done Done false
28747 8440 Acked Read 5 Sent Done Done false
28748 8441 Acked Write 10 Sent Done Done false
28749 8442 Acked Read 2 Sent Done Done false
28750 8443 Acked Flush 0 Sent Done Done false
28751 8444 Acked Read 8 Sent Done Done false
28752 8445 Acked Read 10 Sent Done Done false
28753 8446 Acked Write 3 Sent Done Done false
28754 8447 Acked Write 7 Sent Done Done false
28755 8448 Acked Read 7 Sent Done Done false
28756 8449 Acked Write 10 Sent Done Done false
28757 8450 Acked Write 10 Sent Done Done false
28758 8451 Acked Flush 0 Sent Done Done false
28759 8452 Acked Read 8 Sent Done Done false
28760 8453 Acked Flush 0 Sent Done Done false
28761 8454 Acked Write 9 Sent Done Done false
28762 8455 Acked Read 9 Sent Done Done false
28763 8456 Acked Read 6 Sent Done Done false
28764 8457 Acked Read 1 Sent Done Done false
28765 8458 Acked Write 3 Sent Done Done false
28766 8459 Acked Read 5 Sent Done Done false
28767 8460 Acked Read 5 Sent Done Done false
28768 8461 Acked Write 10 Sent Done Done false
28769 8462 Acked Flush 0 Sent Done Done false
28770 8463 Acked Write 1 Sent Done Done false
28771 8464 Acked Write 3 Sent Done Done false
28772 8465 Acked Write 6 Sent Done Done false
28773 8466 Acked Write 10 Sent Done Done false
28774 8467 Acked Read 2 Sent Done Done false
28775 8468 Acked Read 2 Sent Done Done false
28776 8469 Acked Read 1 Sent Done Done false
28777 8470 Acked Write 9 Sent Done Done false
28778 8471 Acked Read 7 Sent Done Done false
28779 8472 Acked Read 10 Sent Done Done false
28780 8473 Acked Write 2 Sent Done Done false
28781 8474 Acked Read 4 Sent Done Done false
28782 8475 Acked Read 5 Sent Done Done false
28783 8476 Acked Read 10 Sent Done Done false
28784 8477 Acked Read 9 Sent Done Done false
28785 8478 Acked Write 4 Sent Done Done false
28786 8479 Acked Read 4 Sent Done Done false
28787 8480 Acked Read 4 Sent Done Done false
28788 8481 Acked Write 2 Sent Done Done false
28789 8482 Acked Flush 0 Sent Done Done false
28790 8483 Acked Read 5 Sent Done Done false
28791 8484 Acked Read 3 Sent Done Done false
28792 8485 Acked Read 6 Sent Done Done false
28793 8486 Acked Read 5 Sent Done Done false
28794 8487 Acked Read 4 Sent Done Done false
28795 8488 Acked Write 6 Sent Done Done false
28796 8489 Acked Read 8 Sent Done Done false
28797 8490 Acked Read 5 Sent Done Done false
28798 8491 Acked Read 1 Sent Done Done false
28799 8492 Acked Read 5 Sent Done Done false
28800 8493 Acked Read 8 Sent Done Done false
28801 8494 Acked Read 9 Sent Done Done false
28802 8495 Acked Read 6 Sent Done Done false
28803 8496 Acked Read 2 Sent Done Done false
28804 8497 Acked Read 3 Sent Done Done false
28805 8498 Acked Flush 0 Sent Done Done false
28806 8499 Acked Read 1 Sent Done Done false
28807 8500 Acked Write 8 Sent Done Done false
28808 8501 Acked Read 5 Sent Done Done false
28809 8502 Acked Flush 0 Sent Done Done false
28810 8503 Acked Write 5 Sent Done Done false
28811 8504 Acked Read 3 Sent Done Done false
28812 8505 Acked Write 9 Sent Done Done false
28813 8506 Acked Write 8 Sent Done Done false
28814 8507 Acked Write 8 Sent Done Done false
28815 8508 Acked Read 10 Sent Done Done false
28816 8509 Acked Write 3 Sent Done Done false
28817 8510 Acked Write 10 Sent Done Done false
28818 8511 Acked Read 3 Sent Done Done false
28819 8512 Acked Flush 0 Sent Done Done false
28820 8513 Acked Write 9 Sent Done Done false
28821 8514 Acked Read 7 Sent Done Done false
28822 8515 Acked Write 2 Sent Done Done false
28823 8516 Acked Read 6 Sent Done Done false
28824 8517 Acked Write 10 Sent Done Done false
28825 8518 Acked Read 10 Sent Done Done false
28826 8519 Acked Write 6 Sent Done Done false
28827 8520 Acked Read 10 Sent Done Done false
28828 8521 Acked Read 3 Sent Done Done false
28829 8522 Acked Write 5 Sent Done Done false
28830 8523 Acked Flush 0 Sent Done Done false
28831 8524 Acked Write 1 Sent Done Done false
28832 8525 Acked Write 7 Sent Done Done false
28833 8526 Acked Write 5 Sent Done Done false
28834 8527 Acked Write 7 Sent Done Done false
28835 8528 Acked Write 6 Sent Done Done false
28836 8529 Acked Write 8 Sent Done Done false
28837 8530 Acked Write 2 Sent Done Done false
28838 8531 Acked Write 2 Sent Done Done false
28839 8532 Acked Write 10 Sent Done Done false
28840 8533 Acked Flush 0 Sent Done Done false
28841 8534 Acked Write 9 Sent Done Done false
28842 8535 Acked Write 7 Sent Done Done false
28843 8536 Acked Read 10 Sent Done Done false
28844 8537 Acked Flush 0 Sent Done Done false
28845 8538 Acked Write 5 Sent Done Done false
28846 8539 Acked Write 2 Sent Done Done false
28847 8540 Acked Read 9 Sent Done Done false
28848 8541 Acked Flush 0 Sent Done Done false
28849 8542 Acked Write 6 Sent Done Done false
28850 8543 Acked Read 10 Sent Done Done false
28851 8544 Acked Read 9 Sent Done Done false
28852 8545 Acked Flush 0 Sent Done Done false
28853 8546 Acked Read 2 Sent Done Done false
28854 8547 Acked Read 4 Sent Done Done false
28855 STATES DS:0 DS:1 DS:2 TOTAL
28856 Sent 301 0 0 301
28857 Done 0 301 301 602
28858 Skipped 0 0 0 0
28859 Error 0 0 0 0
28860 Last Flush: Some(JobId(8246)) Some(JobId(8545)) Some(JobId(8545))
28861 Downstairs last five completed: 8246 8245 8244 8243 8242
28862 Upstairs last five completed: 8547 8546 8545 8544 8543
28863 CLIENT: Up:0 ds:301 act:2
288642025-01-29T05:17:06.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
28865 [8810] got start action
28866 [8810] I am going to restart
28867 Starting downstairs at port 8810
28868 Make output file at "/tmp/dsc/downstairs-8810.txt"
28869 [8810][0] reports Starting
28870 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1227)
28871 [8810][0] reports Running
288722025-01-29T05:17:07.556ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
288732025-01-29T05:17:08.558ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
288742025-01-29T05:17:09.559ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
28875 ----------------------------------------------------------------
28876 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28877 States: Offline Active Active
28878 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28879 8247 Acked Read 10 Sent Done Done false
28880 8248 Acked Read 8 Sent Done Done false
28881 8249 Acked Flush 0 Sent Done Done false
28882 8250 Acked Write 4 Sent Done Done false
28883 8251 Acked Read 4 Sent Done Done false
28884 8252 Acked Write 6 Sent Done Done false
28885 8253 Acked Write 5 Sent Done Done false
28886 8254 Acked Write 8 Sent Done Done false
28887 8255 Acked Read 5 Sent Done Done false
28888 8256 Acked Write 3 Sent Done Done false
28889 8257 Acked Read 9 Sent Done Done false
28890 8258 Acked Flush 0 Sent Done Done false
28891 8259 Acked Write 2 Sent Done Done false
28892 8260 Acked Write 3 Sent Done Done false
28893 8261 Acked Write 4 Sent Done Done false
28894 8262 Acked Read 5 Sent Done Done false
28895 8263 Acked Read 10 Sent Done Done false
28896 8264 Acked Write 10 Sent Done Done false
28897 8265 Acked Read 5 Sent Done Done false
28898 8266 Acked Read 2 Sent Done Done false
28899 8267 Acked Write 3 Sent Done Done false
28900 8268 Acked Read 6 Sent Done Done false
28901 8269 Acked Flush 0 Sent Done Done false
28902 8270 Acked Read 1 Sent Done Done false
28903 8271 Acked Read 4 Sent Done Done false
28904 8272 Acked Read 1 Sent Done Done false
28905 8273 Acked Read 7 Sent Done Done false
28906 8274 Acked Write 8 Sent Done Done false
28907 8275 Acked Flush 0 Sent Done Done false
28908 8276 Acked Write 3 Sent Done Done false
28909 8277 Acked Write 4 Sent Done Done false
28910 8278 Acked Read 3 Sent Done Done false
28911 8279 Acked Write 9 Sent Done Done false
28912 8280 Acked Read 8 Sent Done Done false
28913 8281 Acked Read 3 Sent Done Done false
28914 8282 Acked Read 5 Sent Done Done false
28915 8283 Acked Read 3 Sent Done Done false
28916 8284 Acked Read 2 Sent Done Done false
28917 8285 Acked Read 10 Sent Done Done false
28918 8286 Acked Read 9 Sent Done Done false
28919 8287 Acked Write 5 Sent Done Done false
28920 8288 Acked Write 2 Sent Done Done false
28921 8289 Acked Read 7 Sent Done Done false
28922 8290 Acked Write 4 Sent Done Done false
28923 8291 Acked Read 7 Sent Done Done false
28924 8292 Acked Write 2 Sent Done Done false
28925 8293 Acked Read 9 Sent Done Done false
28926 8294 Acked Read 2 Sent Done Done false
28927 8295 Acked Read 7 Sent Done Done false
28928 8296 Acked Read 10 Sent Done Done false
28929 8297 Acked Read 1 Sent Done Done false
28930 8298 Acked Read 10 Sent Done Done false
28931 8299 Acked Write 10 Sent Done Done false
28932 8300 Acked Read 2 Sent Done Done false
28933 8301 Acked Write 10 Sent Done Done false
28934 8302 Acked Write 1 Sent Done Done false
28935 8303 Acked Read 2 Sent Done Done false
28936 8304 Acked Read 5 Sent Done Done false
28937 8305 Acked Write 7 Sent Done Done false
28938 8306 Acked Read 10 Sent Done Done false
28939 8307 Acked Read 3 Sent Done Done false
28940 8308 Acked Read 3 Sent Done Done false
28941 8309 Acked Read 6 Sent Done Done false
28942 8310 Acked Read 5 Sent Done Done false
28943 8311 Acked Write 5 Sent Done Done false
28944 8312 Acked Flush 0 Sent Done Done false
28945 8313 Acked Read 3 Sent Done Done false
28946 8314 Acked Write 7 Sent Done Done false
28947 8315 Acked Write 10 Sent Done Done false
28948 8316 Acked Write 6 Sent Done Done false
28949 8317 Acked Read 7 Sent Done Done false
28950 8318 Acked Flush 0 Sent Done Done false
28951 8319 Acked Read 5 Sent Done Done false
28952 8320 Acked Read 6 Sent Done Done false
28953 8321 Acked Read 3 Sent Done Done false
28954 8322 Acked Write 5 Sent Done Done false
28955 8323 Acked Write 3 Sent Done Done false
28956 8324 Acked Flush 0 Sent Done Done false
28957 8325 Acked Write 1 Sent Done Done false
28958 8326 Acked Flush 0 Sent Done Done false
28959 8327 Acked Read 1 Sent Done Done false
28960 8328 Acked Read 5 Sent Done Done false
28961 8329 Acked Flush 0 Sent Done Done false
28962 8330 Acked Read 2 Sent Done Done false
28963 8331 Acked Read 2 Sent Done Done false
28964 8332 Acked Write 4 Sent Done Done false
28965 8333 Acked Read 9 Sent Done Done false
28966 8334 Acked Write 3 Sent Done Done false
28967 8335 Acked Read 5 Sent Done Done false
28968 8336 Acked Write 9 Sent Done Done false
28969 8337 Acked Write 5 Sent Done Done false
28970 8338 Acked Write 7 Sent Done Done false
28971 8339 Acked Read 1 Sent Done Done false
28972 8340 Acked Flush 0 Sent Done Done false
28973 8341 Acked Flush 0 Sent Done Done false
28974 8342 Acked Write 6 Sent Done Done false
28975 8343 Acked Write 1 Sent Done Done false
28976 8344 Acked Write 2 Sent Done Done false
28977 8345 Acked Read 10 Sent Done Done false
28978 8346 Acked Write 2 Sent Done Done false
28979 8347 Acked Write 6 Sent Done Done false
28980 8348 Acked Read 4 Sent Done Done false
28981 8349 Acked Write 8 Sent Done Done false
28982 8350 Acked Read 4 Sent Done Done false
28983 8351 Acked Read 9 Sent Done Done false
28984 8352 Acked Write 5 Sent Done Done false
28985 8353 Acked Read 6 Sent Done Done false
28986 8354 Acked Write 4 Sent Done Done false
28987 8355 Acked Read 2 Sent Done Done false
28988 8356 Acked Write 4 Sent Done Done false
28989 8357 Acked Read 8 Sent Done Done false
28990 8358 Acked Read 1 Sent Done Done false
28991 8359 Acked Read 8 Sent Done Done false
28992 8360 Acked Read 4 Sent Done Done false
28993 8361 Acked Flush 0 Sent Done Done false
28994 8362 Acked Read 5 Sent Done Done false
28995 8363 Acked Read 3 Sent Done Done false
28996 8364 Acked Write 2 Sent Done Done false
28997 8365 Acked Write 1 Sent Done Done false
28998 8366 Acked Flush 0 Sent Done Done false
28999 8367 Acked Write 5 Sent Done Done false
29000 8368 Acked Write 2 Sent Done Done false
29001 8369 Acked Write 3 Sent Done Done false
29002 8370 Acked Read 7 Sent Done Done false
29003 8371 Acked Read 8 Sent Done Done false
29004 8372 Acked Read 9 Sent Done Done false
29005 8373 Acked Read 2 Sent Done Done false
29006 8374 Acked Read 4 Sent Done Done false
29007 8375 Acked Write 8 Sent Done Done false
29008 8376 Acked Write 9 Sent Done Done false
29009 8377 Acked Read 8 Sent Done Done false
29010 8378 Acked Write 8 Sent Done Done false
29011 8379 Acked Write 2 Sent Done Done false
29012 8380 Acked Write 3 Sent Done Done false
29013 8381 Acked Write 8 Sent Done Done false
29014 8382 Acked Write 10 Sent Done Done false
29015 8383 Acked Read 9 Sent Done Done false
29016 8384 Acked Read 4 Sent Done Done false
29017 8385 Acked Write 5 Sent Done Done false
29018 8386 Acked Write 5 Sent Done Done false
29019 8387 Acked Write 6 Sent Done Done false
29020 8388 Acked Write 8 Sent Done Done false
29021 8389 Acked Read 2 Sent Done Done false
29022 8390 Acked Write 7 Sent Done Done false
29023 8391 Acked Read 4 Sent Done Done false
29024 8392 Acked Read 7 Sent Done Done false
29025 8393 Acked Read 3 Sent Done Done false
29026 8394 Acked Read 5 Sent Done Done false
29027 8395 Acked Read 1 Sent Done Done false
29028 8396 Acked Read 2 Sent Done Done false
29029 8397 Acked Read 9 Sent Done Done false
29030 8398 Acked Read 3 Sent Done Done false
29031 8399 Acked Flush 0 Sent Done Done false
29032 8400 Acked Write 9 Sent Done Done false
29033 8401 Acked Read 3 Sent Done Done false
29034 8402 Acked Write 9 Sent Done Done false
29035 8403 Acked Read 2 Sent Done Done false
29036 8404 Acked Flush 0 Sent Done Done false
29037 8405 Acked Write 2 Sent Done Done false
29038 8406 Acked Read 9 Sent Done Done false
29039 8407 Acked Read 9 Sent Done Done false
29040 8408 Acked Read 1 Sent Done Done false
29041 8409 Acked Read 6 Sent Done Done false
29042 8410 Acked Write 3 Sent Done Done false
29043 8411 Acked Write 2 Sent Done Done false
29044 8412 Acked Read 5 Sent Done Done false
29045 8413 Acked Read 5 Sent Done Done false
29046 8414 Acked Read 1 Sent Done Done false
29047 8415 Acked Write 3 Sent Done Done false
29048 8416 Acked Write 4 Sent Done Done false
29049 8417 Acked Write 5 Sent Done Done false
29050 8418 Acked Read 10 Sent Done Done false
29051 8419 Acked Read 9 Sent Done Done false
29052 8420 Acked Read 5 Sent Done Done false
29053 8421 Acked Write 8 Sent Done Done false
29054 8422 Acked Write 9 Sent Done Done false
29055 8423 Acked Write 7 Sent Done Done false
29056 8424 Acked Read 4 Sent Done Done false
29057 8425 Acked Flush 0 Sent Done Done false
29058 8426 Acked Read 10 Sent Done Done false
29059 8427 Acked Read 10 Sent Done Done false
29060 8428 Acked Read 9 Sent Done Done false
29061 8429 Acked Write 5 Sent Done Done false
29062 8430 Acked Read 8 Sent Done Done false
29063 8431 Acked Write 8 Sent Done Done false
29064 8432 Acked Read 7 Sent Done Done false
29065 8433 Acked Read 6 Sent Done Done false
29066 8434 Acked Write 8 Sent Done Done false
29067 8435 Acked Read 4 Sent Done Done false
29068 8436 Acked Write 5 Sent Done Done false
29069 8437 Acked Write 4 Sent Done Done false
29070 8438 Acked Read 7 Sent Done Done false
29071 8439 Acked Read 3 Sent Done Done false
29072 8440 Acked Read 5 Sent Done Done false
29073 8441 Acked Write 10 Sent Done Done false
29074 8442 Acked Read 2 Sent Done Done false
29075 8443 Acked Flush 0 Sent Done Done false
29076 8444 Acked Read 8 Sent Done Done false
29077 8445 Acked Read 10 Sent Done Done false
29078 8446 Acked Write 3 Sent Done Done false
29079 8447 Acked Write 7 Sent Done Done false
29080 8448 Acked Read 7 Sent Done Done false
29081 8449 Acked Write 10 Sent Done Done false
29082 8450 Acked Write 10 Sent Done Done false
29083 8451 Acked Flush 0 Sent Done Done false
29084 8452 Acked Read 8 Sent Done Done false
29085 8453 Acked Flush 0 Sent Done Done false
29086 8454 Acked Write 9 Sent Done Done false
29087 8455 Acked Read 9 Sent Done Done false
29088 8456 Acked Read 6 Sent Done Done false
29089 8457 Acked Read 1 Sent Done Done false
29090 8458 Acked Write 3 Sent Done Done false
29091 8459 Acked Read 5 Sent Done Done false
29092 8460 Acked Read 5 Sent Done Done false
29093 8461 Acked Write 10 Sent Done Done false
29094 8462 Acked Flush 0 Sent Done Done false
29095 8463 Acked Write 1 Sent Done Done false
29096 8464 Acked Write 3 Sent Done Done false
29097 8465 Acked Write 6 Sent Done Done false
29098 8466 Acked Write 10 Sent Done Done false
29099 8467 Acked Read 2 Sent Done Done false
29100 8468 Acked Read 2 Sent Done Done false
29101 8469 Acked Read 1 Sent Done Done false
29102 8470 Acked Write 9 Sent Done Done false
29103 8471 Acked Read 7 Sent Done Done false
29104 8472 Acked Read 10 Sent Done Done false
29105 8473 Acked Write 2 Sent Done Done false
29106 8474 Acked Read 4 Sent Done Done false
29107 8475 Acked Read 5 Sent Done Done false
29108 8476 Acked Read 10 Sent Done Done false
29109 8477 Acked Read 9 Sent Done Done false
29110 8478 Acked Write 4 Sent Done Done false
29111 8479 Acked Read 4 Sent Done Done false
29112 8480 Acked Read 4 Sent Done Done false
29113 8481 Acked Write 2 Sent Done Done false
29114 8482 Acked Flush 0 Sent Done Done false
29115 8483 Acked Read 5 Sent Done Done false
29116 8484 Acked Read 3 Sent Done Done false
29117 8485 Acked Read 6 Sent Done Done false
29118 8486 Acked Read 5 Sent Done Done false
29119 8487 Acked Read 4 Sent Done Done false
29120 8488 Acked Write 6 Sent Done Done false
29121 8489 Acked Read 8 Sent Done Done false
29122 8490 Acked Read 5 Sent Done Done false
29123 8491 Acked Read 1 Sent Done Done false
29124 8492 Acked Read 5 Sent Done Done false
29125 8493 Acked Read 8 Sent Done Done false
29126 8494 Acked Read 9 Sent Done Done false
29127 8495 Acked Read 6 Sent Done Done false
29128 8496 Acked Read 2 Sent Done Done false
29129 8497 Acked Read 3 Sent Done Done false
29130 8498 Acked Flush 0 Sent Done Done false
29131 8499 Acked Read 1 Sent Done Done false
29132 8500 Acked Write 8 Sent Done Done false
29133 8501 Acked Read 5 Sent Done Done false
29134 8502 Acked Flush 0 Sent Done Done false
29135 8503 Acked Write 5 Sent Done Done false
29136 8504 Acked Read 3 Sent Done Done false
29137 8505 Acked Write 9 Sent Done Done false
29138 8506 Acked Write 8 Sent Done Done false
29139 8507 Acked Write 8 Sent Done Done false
29140 8508 Acked Read 10 Sent Done Done false
29141 8509 Acked Write 3 Sent Done Done false
29142 8510 Acked Write 10 Sent Done Done false
29143 8511 Acked Read 3 Sent Done Done false
29144 8512 Acked Flush 0 Sent Done Done false
29145 8513 Acked Write 9 Sent Done Done false
29146 8514 Acked Read 7 Sent Done Done false
29147 8515 Acked Write 2 Sent Done Done false
29148 8516 Acked Read 6 Sent Done Done false
29149 8517 Acked Write 10 Sent Done Done false
29150 8518 Acked Read 10 Sent Done Done false
29151 8519 Acked Write 6 Sent Done Done false
29152 8520 Acked Read 10 Sent Done Done false
29153 8521 Acked Read 3 Sent Done Done false
29154 8522 Acked Write 5 Sent Done Done false
29155 8523 Acked Flush 0 Sent Done Done false
29156 8524 Acked Write 1 Sent Done Done false
29157 8525 Acked Write 7 Sent Done Done false
29158 8526 Acked Write 5 Sent Done Done false
29159 8527 Acked Write 7 Sent Done Done false
29160 8528 Acked Write 6 Sent Done Done false
29161 8529 Acked Write 8 Sent Done Done false
29162 8530 Acked Write 2 Sent Done Done false
29163 8531 Acked Write 2 Sent Done Done false
29164 8532 Acked Write 10 Sent Done Done false
29165 8533 Acked Flush 0 Sent Done Done false
29166 8534 Acked Write 9 Sent Done Done false
29167 8535 Acked Write 7 Sent Done Done false
29168 8536 Acked Read 10 Sent Done Done false
29169 8537 Acked Flush 0 Sent Done Done false
29170 8538 Acked Write 5 Sent Done Done false
29171 8539 Acked Write 2 Sent Done Done false
29172 8540 Acked Read 9 Sent Done Done false
29173 8541 Acked Flush 0 Sent Done Done false
29174 8542 Acked Write 6 Sent Done Done false
29175 8543 Acked Read 10 Sent Done Done false
29176 8544 Acked Read 9 Sent Done Done false
29177 8545 Acked Flush 0 Sent Done Done false
29178 8546 Acked Read 2 Sent Done Done false
29179 8547 Acked Read 4 Sent Done Done false
29180 STATES DS:0 DS:1 DS:2 TOTAL
29181 Sent 301 0 0 301
29182 Done 0 301 301 602
29183 Skipped 0 0 0 0
29184 Error 0 0 0 0
29185 Last Flush: Some(JobId(8246)) Some(JobId(8545)) Some(JobId(8545))
29186 Downstairs last five completed: 8246 8245 8244 8243 8242
29187 Upstairs last five completed: 8547 8546 8545 8544 8543
29188 CLIENT: Up:0 ds:301 act:2
291892025-01-29T05:17:10.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291902025-01-29T05:17:11.509ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291912025-01-29T05:17:11.510ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37610) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291922025-01-29T05:17:11.511ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291932025-01-29T05:17:11.511ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291942025-01-29T05:17:11.511ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291952025-01-29T05:17:11.511ZINFOcrucible: send last flush ID to this DS: Some(JobId(8246)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291962025-01-29T05:17:11.511ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291972025-01-29T05:17:11.511ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8246)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
291982025-01-29T05:17:11.512ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8246)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
29199 ----------------------------------------------------------------
29200 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29201 Downstairs last five completed: 8548 8547 8546 8545 8544
29202 Upstairs last five completed: 8548 8547 8546 8545 8544
29203 CLIENT: Up:0 ds:0 act:3
29204 Replay: All jobs finished
29205 Main task has work to do, go find it
29206 got dsc Stop(2)
29207 stop 2
29208 [8830] Got stop action so:false kr:false
29209 Jan 29 05:17:13.760 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c6be2303-7ee2-49a9-b7da-6f37b5d6c8da, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
292102025-01-29T05:17:13.761ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
292112025-01-29T05:17:13.761ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
29212 Jan 29 05:17:13.761 INFO request completed, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:17:13.761292514Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
29213 latency_us: 147, response_code: 200, uri: /state/cid/2, method: GET, req_id: bedbc2ed-8da6-4239-82a9-8d8c12a1847c, remote_addr: 127{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:17:13.761365668Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
29214 .0.0.1:46995, local_addr: 127.0.0.1:9998
29215 [8830] Exited with: ExitStatus(unix_wait_status(9))
29216 [8830][2] reports Exit
29217 Jan 29 05:17:17.763 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1685bc63-9a61-4707-b878-30ee95e1b318, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
29218 001/300 Read block 2075 len 16384
29219 002/300 Read block 2021 len 40960
29220 003/300 Read block 4713 len 28672
29221 004/300 Read block 4042 len 4096
29222 005/300 Write block 3846 len 4096 data: 5
29223 006/300 Write block 1110 len 12288 data: 5 5 5
29224 007/300 Read block 4022 len 36864
29225 008/300 Read block 4420 len 24576
29226 009/300 Read block 1474 len 12288
29227 010/300 Read block 795 len 16384
29228 011/300 Read block 2901 len 12288
29229 012/300 Write block 4346 len 8192 data: 3 3
29230 013/300 Write block 2974 len 36864 data: 4 6 10 8 7 6 5 5 5
29231 014/300 Write block 2205 len 8192 data: 6 6
29232 015/300 Write block 2325 len 16384 data: 5 5 5 4
29233 016/300 Read block 670 len 32768
29234 017/300 Read block 2324 len 4096
29235 018/300 Write block 711 len 36864 data: 4 3 3 2 5 6 6 6 5
29236 019/300 Read block 3278 len 20480
29237 020/300 Read block 2124 len 20480
29238 021/300 Flush
29239 022/300 Write block 2328 len 28672 data: 5 3 3 5 4 5 4
29240 023/300 Read block 3590 len 12288
29241 024/300 Read block 4242 len 36864
29242 025/300 Read block 3009 len 16384
29243 026/300 Read block 645 len 20480
29244 027/300 Read block 2790 len 24576
29245 028/300 Read block 3779 len 24576
29246 029/300 Write block 774 len 16384 data: 7 7 6 6
29247 030/300 Read block 1335 len 32768
29248 031/300 Write block 3976 len 28672 data: 8 9 9 9 6 7 7
29249 032/300 Write block 214 len 4096 data: 4
29250 033/300 Read block 1356 len 4096
29251 034/300 Flush
29252 035/300 Read block 2155 len 40960
29253 036/300 Read block 1050 len 28672
29254 037/300 Write block 4137 len 28672 data: 7 8 8 10 10 7 5
29255 038/300 Flush
29256 039/300 Read block 2504 len 4096
29257 040/300 Read block 4026 len 36864
29258 041/300 Read block 4360 len 20480
29259 042/300 Write block 1835 len 24576 data: 6 6 5 7 6 7
29260 043/300 Flush
29261 044/300 Flush
29262 045/300 Write block 1504 len 40960 data: 7 7 8 7 8 7 8 7 6 6
29263 046/300 Read block 245 len 40960
29264 047/300 Write block 2312 len 8192 data: 7 6
29265 048/300 Write block 1314 len 24576 data: 5 6 7 8 7 5
29266 049/300 Write block 3100 len 36864 data: 4 5 7 8 7 8 8 7 7
29267 050/300 Read block 1374 len 40960
29268 051/300 Read block 1830 len 28672
29269 052/300 Read block 2988 len 36864
29270 053/300 Read block 4048 len 8192
29271 054/300 Read block 3846 len 40960
29272 055/300 Write block 4184 len 24576 data: 6 5 5 3 5 6
29273 056/300 Write block 3600 len 12288 data: 2 3 3
29274 057/300 Read block 3249 len 32768
29275 058/300 Write block 2508 len 28672 data: 5 4 5 5 6 5 3
29276 059/300 Write block 4703 len 8192 data: 5 5
29277 060/300 Flush
29278 061/300 Write block 1814 len 28672 data: 4 3 3 5 5 6 6
29279 062/300 Read block 4099 len 32768
29280 063/300 Write block 175 len 20480 data: 7 6 6 6 5
29281 064/300 Write block 839 len 8192 data: 6 5
29282 065/300 Write block 857 len 8192 data: 5 4
29283 066/300 Write block 1815 len 24576 data: 4 4 6 6 7 7
29284 067/300 Write block 642 len 24576 data: 7 6 5 3 3 4
29285 068/300 Read block 4217 len 20480
29286 069/300 Read block 2454 len 4096
29287 070/300 Read block 1014 len 8192
29288 071/300 Read block 1795 len 20480
29289 072/300 Read block 2453 len 28672
29290 073/300 Write block 3264 len 16384 data: 3 3 3 4
29291 074/300 Write block 4365 len 4096 data: 7
29292 075/300 Read block 2889 len 8192
29293 076/300 Read block 2723 len 36864
29294 077/300 Read block 4453 len 40960
29295 078/300 Write block 990 len 28672 data: 3 3 4 5 4 4 5
29296 079/300 Read block 4442 len 24576
29297 080/300 Write block 4600 len 20480 data: 10 10 9 10 11
29298 081/300 Read block 3003 len 12288
29299 082/300 Write block 4497 len 20480 data: 5 7 6 7 5
29300 083/300 Write block 1684 len 12288 data: 11 10 9
29301 084/300 Read block 44 len 40960
29302 085/300 Read block 3162 len 4096
29303 086/300 Read block 1561 len 12288
29304 087/300 Read block 4666 len 8192
29305 088/300 Write block 2494 len 28672 data: 5 5 4 4 5 5 5
29306 089/300 Read block 1036 len 8192
29307 090/300 Read block 2906 len 24576
29308 091/300 Write block 554 len 32768 data: 7 5 7 6 8 7 6 4
29309 092/300 Write block 2899 len 36864 data: 3 4 4 5 5 5 5 6 5
29310 093/300 Read block 2571 len 24576
29311 094/300 Read block 4307 len 36864
29312 095/300 Write block 1210 len 20480 data: 2 3 5 6 6
29313 096/300 Read block 1114 len 24576
29314 097/300 Read block 4935 len 20480
29315 098/300 Write block 503 len 8192 data: 9 9
29316 099/300 Flush
29317 100/300 Read block 3722 len 4096
29318 101/300 Write block 2594 len 40960 data: 4 6 6 6 6 5 6 6 7 7
29319 102/300 Read block 1556 len 12288
29320 103/300 Write block 1012 len 28672 data: 7 7 5 6 6 6 6
29321 104/300 Write block 2515 len 8192 data: 4 4
29322 105/300 Write block 796 len 40960 data: 9 8 7 6 4 4 4 3 3 4
29323 106/300 Read block 4034 len 40960
29324 107/300 Read block 1878 len 32768
29325 108/300 Read block 2243 len 28672
29326 109/300 Write block 3828 len 28672 data: 5 5 6 7 7 6 7
29327 110/300 Read block 1262 len 32768
29328 111/300 Flush
29329 112/300 Write block 3415 len 36864 data: 3 4 5 6 7 8 7 7 8
29330 113/300 Write block 2536 len 16384 data: 4 4 4 4
29331 114/300 Write block 1388 len 36864 data: 7 6 6 6 4 5 5 5 4
29332 115/300 Write block 2918 len 8192 data: 7 7
29333 116/300 Read block 2391 len 32768
29334 117/300 Read block 620 len 40960
29335 118/300 Read block 1473 len 32768
29336 119/300 Read block 941 len 16384
29337 120/300 Read block 3978 len 20480
29338 121/300 Read block 2736 len 12288
29339 122/300 Write block 1604 len 20480 data: 2 2 2 3 3
29340 123/300 Write block 4906 len 40960 data: 6 5 5 4 4 2 2 2 2 3
29341 124/300 Read block 1676 len 40960
29342 125/300 Read block 4331 len 20480
29343 126/300 Read block 3434 len 24576
29344 127/300 Write block 956 len 40960 data: 5 7 7 5 5 5 6 6 4 7
29345 128/300 Write block 2465 len 28672 data: 5 4 4 5 6 6 6
29346 129/300 Read block 3333 len 4096
29347 130/300 Write block 1040 len 40960 data: 5 7 8 9 9 10 9 8 6 5
29348 131/300 Write block 2022 len 32768 data: 6 6 7 5 6 6 6 7
29349 132/300 Write block 3283 len 28672 data: 6 6 6 6 6 6 6
29350 133/300 Write block 1094 len 28672 data: 7 6 6 6 5 6 6
29351 134/300 Write block 4045 len 36864 data: 8 9 8 7 6 8 7 6 5
29352 135/300 Read block 1578 len 12288
29353 136/300 Read block 3281 len 36864
29354 137/300 Read block 4088 len 32768
29355 138/300 Flush
29356 139/300 Write block 730 len 36864 data: 6 6 9 7 7 7 9 8 8
29357 140/300 Read block 4550 len 12288
29358 141/300 Write block 4464 len 28672 data: 5 4 4 3 4 5 3
29359 142/300 Read block 998 len 32768
29360 143/300 Flush
29361 144/300 Flush
29362 145/300 Read block 3217 len 40960
29363 146/300 Read block 3071 len 12288
29364 147/300 Read block 467 len 24576
29365 148/300 Write block 2728 len 16384 data: 6 7 7 9
29366 149/300 Flush
29367 150/300 Read block 1366 len 40960
29368 151/300 Write block 3389 len 16384 data: 7 7 7 6
29369 152/300 Read block 3447 len 16384
29370 153/300 Write block 1271 len 12288 data: 6 6 6
29371 154/300 Flush
29372 155/300 Read block 4384 len 20480
29373 156/300 Read block 1005 len 4096
29374 157/300 Read block 3103 len 8192
29375 158/300 Read block 4877 len 12288
29376 159/300 Write block 1808 len 8192 data: 5 5
29377 160/300 Read block 2918 len 40960
29378 161/300 Read block 2390 len 20480
29379 162/300 Write block 2341 len 32768 data: 6 7 5 6 6 7 6 5
29380 163/300 Read block 3719 len 32768
29381 164/300 Write block 4318 len 24576 data: 4 4 5 6 6 6
29382 165/300 Write block 1460 len 24576 data: 5 5 4 4 3 3
29383 166/300 Read block 1959 len 36864
29384 167/300 Write block 1507 len 8192 data: 8 9
29385 168/300 Write block 1660 len 8192 data: 4 4
29386 169/300 Read block 4704 len 8192
29387 170/300 Write block 94 len 4096 data: 3
29388 171/300 Write block 3503 len 20480 data: 7 6 6 4 3
29389 172/300 Write block 4945 len 32768 data: 2 3 4 3 3 3 3 3
29390 173/300 Write block 487 len 12288 data: 9 5 5
29391 174/300 Read block 2893 len 32768
29392 175/300 Read block 1225 len 28672
29393 176/300 Read block 2809 len 32768
29394 177/300 Read block 3862 len 24576
29395 178/300 Read block 3512 len 36864
29396 179/300 Read block 3817 len 24576
29397 180/300 Write block 3856 len 24576 data: 5 5 5 4 3 2
29398 181/300 Write block 1573 len 40960 data: 7 7 6 6 4 3 4 4 5 5
29399 182/300 Read block 3741 len 4096
29400 183/300 Read block 854 len 36864
29401 184/300 Read block 1301 len 36864
29402 185/300 Flush
29403 186/300 Read block 2776 len 24576
29404 187/300 Read block 45 len 4096
29405 188/300 Read block 595 len 36864
29406 189/300 Read block 2887 len 4096
29407 190/300 Write block 2654 len 20480 data: 4 3 3 4 5
29408 191/300 Read block 891 len 32768
29409 192/300 Write block 215 len 8192 data: 5 6
29410 193/300 Read block 4937 len 32768
29411 194/300 Write block 2515 len 36864 data: 5 5 5 6 6 6 7 8 9
29412 195/300 Read block 3190 len 28672
29413 196/300 Write block 3753 len 16384 data: 5 5 5 5
29414 197/300 Write block 2231 len 40960 data: 8 7 6 5 5 5 6 5 4 5
29415 198/300 Flush
29416 199/300 Read block 81 len 8192
29417 200/300 Write block 1580 len 4096 data: 5
29418 201/300 Write block 333 len 16384 data: 7 5 5 5
29419 202/300 Write block 2989 len 24576 data: 6 5 5 4 3 2
29420 203/300 Write block 1194 len 24576 data: 5 5 5 4 5 5
29421 204/300 Write block 217 len 16384 data: 6 5 7 6
29422 205/300 Write block 3161 len 40960 data: 3 3 3 4 3 4 4 5 5 5
29423 206/300 Read block 989 len 28672
29424 207/300 Write block 3844 len 12288 data: 5 5 6
29425 208/300 Read block 284 len 20480
29426 209/300 Write block 1663 len 4096 data: 4
29427 210/300 Write block 85 len 28672 data: 5 5 6 6 5 5 5
29428 211/300 Write block 2740 len 32768 data: 7 7 9 7 6 7 7 8
29429 212/300 Read block 613 len 40960
29430 213/300 Flush
29431 214/300 Read block 1712 len 16384
29432 215/300 Write block 1053 len 16384 data: 4 3 3 5
29433 216/300 Write block 2644 len 40960 data: 5 4 4 5 5 4 5 3 3 4
29434 217/300 Read block 994 len 32768
29435 218/300 Write block 1072 len 16384 data: 6 7 8 7
29436 219/300 Read block 4203 len 40960
29437 220/300 Flush
29438 221/300 Write block 118 len 32768 data: 6 7 6 7 7 8 9 8
29439 222/300 Read block 8 len 16384
29440 223/300 Read block 1662 len 4096
29441 224/300 Read block 4315 len 12288
29442 225/300 Write block 1222 len 12288 data: 3 4 5
29443 226/300 Write block 517 len 32768 data: 6 5 4 4 4 3 2 2
29444 227/300 Read block 4049 len 20480
29445 228/300 Read block 1026 len 8192
29446 229/300 Write block 969 len 28672 data: 10 10 9 9 8 7 5
29447 230/300 Read block 3182 len 24576
29448 231/300 Write block 1614 len 4096 data: 6
29449 232/300 Read block 4144 len 24576
29450 233/300 Flush
29451 234/300 Write block 715 len 32768 data: 6 7 7 7 6 5 7 7
29452 235/300 Read block 1595 len 28672
29453 236/300 Write block 1946 len 16384 data: 5 5 7 7
29454 237/300 Flush
29455 238/300 Read block 1003 len 24576
29456 239/300 Write block 369 len 28672 data: 6 5 4 3 3 4 5
29457 240/300 Read block 760 len 28672
29458 241/300 Write block 4163 len 24576 data: 5 4 5 5 5 5
29459 242/300 Write block 3655 len 36864 data: 4 5 5 5 2 2 2 2 2
29460 243/300 Flush
29461 244/300 Flush
29462 245/300 Write block 1583 len 16384 data: 5 6 6 6
29463 246/300 Flush
29464 247/300 Write block 3351 len 8192 data: 8 7
29465 248/300 Read block 3074 len 28672
29466 249/300 Write block 2583 len 40960 data: 4 4 3 4 5 5 5 5 6 6
29467 250/300 Read block 3964 len 36864
29468 251/300 Read block 73 len 24576
29469 252/300 Flush
29470 253/300 Write block 2314 len 20480 data: 6 6 6 6 5
29471 254/300 Read block 4428 len 4096
29472 255/300 Read block 791 len 16384
29473 256/300 Read block 773 len 24576
29474 257/300 Write block 173 len 20480 data: 8 6 8 7 7
29475 258/300 Write block 2572 len 40960 data: 4 6 6 6 7 7 6 5 6 4
29476 259/300 Read block 2649 len 28672
29477 260/300 Write block 274 len 24576 data: 5 4 3 4 4 4
29478 261/300 Write block 1289 len 12288 data: 7 5 5
29479 262/300 Write block 662 len 12288 data: 6 6 7
29480 263/300 Read block 866 len 4096
29481 264/300 Flush
29482 265/300 Read block 2833 len 20480
29483 266/300 Write block 1292 len 20480 data: 6 6 6 5 4
29484 267/300 Write block 3265 len 40960 data: 4 4 5 3 2 3 4 4 4 2
29485 268/300 Write block 3691 len 28672 data: 4 5 5 5 6 5 4
29486 269/300 Write block 4860 len 36864 data: 6 5 5 5 5 3 2 2 2
29487 270/300 Read block 2465 len 4096
29488 271/300 Read block 3843 len 40960
29489 272/300 Read block 2475 len 4096
29490 273/300 Flush
29491 274/300 Read block 3182 len 12288
29492 275/300 Read block 1827 len 20480
29493 276/300 Write block 692 len 20480 data: 7 5 5 4 3
29494 277/300 Read block 311 len 4096
29495 278/300 Flush
29496 279/300 Write block 1943 len 40960 data: 8 8 6 6 6 8 8 7 8 6
29497 280/300 Write block 3848 len 32768 data: 5 6 5 4 4 2 3 4
29498 281/300 Write block 279 len 8192 data: 5 4
29499 282/300 Read block 2338 len 40960
29500 283/300 Read block 4357 len 32768
29501 284/300 Read block 3297 len 24576
29502 285/300 Read block 648 len 28672
29503 286/300 Write block 41 len 4096 data: 7
29504 287/300 Read block 3050 len 4096
29505 288/300 Read block 2284 len 4096
29506 289/300 Write block 4960 len 40960 data: 6 7 6 6 7 6 5 6 5 5
29507 290/300 Read block 3915 len 12288
29508 291/300 Read block 4026 len 28672
29509 292/300 Read block 512 len 4096
29510 293/300 Read block 3163 len 24576
29511 294/300 Write block 2010 len 32768 data: 6 7 8 6 6 6 5 4
29512 295/300 Write block 4686 len 8192 data: 6 5
29513 296/300 Write block 1564 len 40960 data: 5 5 6 7 8 8 9 9 8 8
29514 297/300 Flush
29515 298/300 Write block 2350 len 28672 data: 5 5 5 4 4 5 5
29516 299/300 Read block 2530 len 36864
29517 300/300 Read block 1843 len 4096
29518 301/300 Read block 3664 len 24576
29519 Main task has work to do, go find it
29520 got dsc Start(2)
29521 start 2
29522 Jan 29 05:17:18.010 INFO request completed, latency_us: 335, response_code: 204, uri: /start/cid/2, method: POST, req_id: b88f349c-2950-4e34-a962-fdb43f7dc12a, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
29523 Replay: started 2, returned:Ok(())
29524 ----------------------------------------------------------------
29525 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29526 States: Active Active Offline
29527 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29528 8549 Acked Read 4 Done Done Sent false
29529 8550 Acked Read 10 Done Done Sent false
29530 8551 Acked Read 7 Done Done Sent false
29531 8552 Acked Read 1 Done Done Sent false
29532 8553 Acked Write 1 Done Done Sent false
29533 8554 Acked Write 3 Done Done Sent false
29534 8555 Acked Read 9 Done Done Sent false
29535 8556 Acked Read 6 Done Done Sent false
29536 8557 Acked Read 3 Done Done Sent false
29537 8558 Acked Read 4 Done Done Sent false
29538 8559 Acked Read 3 Done Done Sent false
29539 8560 Acked Write 2 Done Done Sent false
29540 8561 Acked Write 9 Done Done Sent false
29541 8562 Acked Write 2 Done Done Sent false
29542 8563 Acked Write 4 Done Done Sent false
29543 8564 Acked Read 8 Done Done Sent false
29544 8565 Acked Read 1 Done Done Sent false
29545 8566 Acked Write 9 Done Done Sent false
29546 8567 Acked Read 5 Done Done Sent false
29547 8568 Acked Read 5 Done Done Sent false
29548 8569 Acked Flush 0 Done Done Sent false
29549 8570 Acked Write 7 Done Done Sent false
29550 8571 Acked Read 3 Done Done Sent false
29551 8572 Acked Read 9 Done Done Sent false
29552 8573 Acked Read 4 Done Done Sent false
29553 8574 Acked Read 5 Done Done Sent false
29554 8575 Acked Read 6 Done Done Sent false
29555 8576 Acked Read 6 Done Done Sent false
29556 8577 Acked Write 4 Done Done Sent false
29557 8578 Acked Read 8 Done Done Sent false
29558 8579 Acked Write 7 Done Done Sent false
29559 8580 Acked Write 1 Done Done Sent false
29560 8581 Acked Read 1 Done Done Sent false
29561 8582 Acked Flush 0 Done Done Sent false
29562 8583 Acked Read 10 Done Done Sent false
29563 8584 Acked Read 7 Done Done Sent false
29564 8585 Acked Write 7 Done Done Sent false
29565 8586 Acked Flush 0 Done Done Sent false
29566 8587 Acked Read 1 Done Done Sent false
29567 8588 Acked Read 9 Done Done Sent false
29568 8589 Acked Read 5 Done Done Sent false
29569 8590 Acked Write 6 Done Done Sent false
29570 8591 Acked Flush 0 Done Done Sent false
29571 8592 Acked Flush 0 Done Done Sent false
29572 8593 Acked Write 10 Done Done Sent false
29573 8594 Acked Read 10 Done Done Sent false
29574 8595 Acked Write 2 Done Done Sent false
29575 8596 Acked Write 6 Done Done Sent false
29576 8597 Acked Write 9 Done Done Sent false
29577 8598 Acked Read 10 Done Done Sent false
29578 8599 Acked Read 7 Done Done Sent false
29579 8600 Acked Read 9 Done Done Sent false
29580 8601 Acked Read 2 Done Done Sent false
29581 8602 Acked Read 10 Done Done Sent false
29582 8603 Acked Write 6 Done Done Sent false
29583 8604 Acked Write 3 Done Done Sent false
29584 8605 Acked Read 8 Done Done Sent false
29585 8606 Acked Write 7 Done Done Sent false
29586 8607 Acked Write 2 Done Done Sent false
29587 8608 Acked Flush 0 Done Done Sent false
29588 8609 Acked Write 7 Done Done Sent false
29589 8610 Acked Read 8 Done Done Sent false
29590 8611 Acked Write 5 Done Done Sent false
29591 8612 Acked Write 2 Done Done Sent false
29592 8613 Acked Write 2 Done Done Sent false
29593 8614 Acked Write 6 Done Done Sent false
29594 8615 Acked Write 6 Done Done Sent false
29595 8616 Acked Read 5 Done Done Sent false
29596 8617 Acked Read 1 Done Done Sent false
29597 8618 Acked Read 2 Done Done Sent false
29598 8619 Acked Read 5 Done Done Sent false
29599 8620 Acked Read 7 Done Done Sent false
29600 8621 Acked Write 4 Done Done Sent false
29601 8622 Acked Write 1 Done Done Sent false
29602 8623 Acked Read 2 Done Done Sent false
29603 8624 Acked Read 9 Done Done Sent false
29604 8625 Acked Read 10 Done Done Sent false
29605 8626 Acked Write 7 Done Done Sent false
29606 8627 Acked Read 6 Done Done Sent false
29607 8628 Acked Write 5 Done Done Sent false
29608 8629 Acked Read 3 Done Done Sent false
29609 8630 Acked Write 5 Done Done Sent false
29610 8631 Acked Write 3 Done Done Sent false
29611 8632 Acked Read 10 Done Done Sent false
29612 8633 Acked Read 1 Done Done Sent false
29613 8634 Acked Read 3 Done Done Sent false
29614 8635 Acked Read 2 Done Done Sent false
29615 8636 Acked Write 7 Done Done Sent false
29616 8637 Acked Read 2 Done Done Sent false
29617 8638 Acked Read 6 Done Done Sent false
29618 8639 Acked Write 8 Done Done Sent false
29619 8640 Acked Write 9 Done Done Sent false
29620 8641 Acked Read 6 Done Done Sent false
29621 8642 Acked Read 9 Done Done Sent false
29622 8643 Acked Write 5 Done Done Sent false
29623 8644 Acked Read 6 Done Done Sent false
29624 8645 Acked Read 5 Done Done Sent false
29625 8646 Acked Write 2 Done Done Sent false
29626 8647 Acked Flush 0 Done Done Sent false
29627 8648 Acked Read 1 Done Done Sent false
29628 8649 Acked Write 10 Done Done Sent false
29629 8650 Acked Read 3 Done Done Sent false
29630 8651 Acked Write 7 Done Done Sent false
29631 8652 Acked Write 2 Done Done Sent false
29632 8653 Acked Write 10 Done Done Sent false
29633 8654 Acked Read 10 Done Done Sent false
29634 8655 Acked Read 8 Done Done Sent false
29635 8656 Acked Read 7 Done Done Sent false
29636 8657 Acked Write 7 Done Done Sent false
29637 8658 Acked Read 8 Done Done Sent false
29638 8659 Acked Flush 0 Done Done Sent false
29639 8660 Acked Write 9 Done Done Sent false
29640 8661 Acked Write 4 Done Done Sent false
29641 8662 Acked Write 9 Done Done Sent false
29642 8663 Acked Write 2 Done Done Sent false
29643 8664 Acked Read 8 Done Done Sent false
29644 8665 Acked Read 10 Done Done Sent false
29645 8666 Acked Read 8 Done Done Sent false
29646 8667 Acked Read 4 Done Done Sent false
29647 8668 Acked Read 5 Done Done Sent false
29648 8669 Acked Read 3 Done Done Sent false
29649 8670 Acked Write 5 Done Done Sent false
29650 8671 Acked Write 10 Done Done Sent false
29651 8672 Acked Read 10 Done Done Sent false
29652 8673 Acked Read 5 Done Done Sent false
29653 8674 Acked Read 6 Done Done Sent false
29654 8675 Acked Write 10 Done Done Sent false
29655 8676 Acked Write 7 Done Done Sent false
29656 8677 Acked Read 1 Done Done Sent false
29657 8678 Acked Write 10 Done Done Sent false
29658 8679 Acked Write 8 Done Done Sent false
29659 8680 Acked Write 7 Done Done Sent false
29660 8681 Acked Write 7 Done Done Sent false
29661 8682 Acked Write 9 Done Done Sent false
29662 8683 Acked Read 3 Done Done Sent false
29663 8684 Acked Read 9 Done Done Sent false
29664 8685 Acked Read 8 Done Done Sent false
29665 8686 Acked Flush 0 Done Done Sent false
29666 8687 Acked Write 9 Done Done Sent false
29667 8688 Acked Read 3 Done Done Sent false
29668 8689 Acked Write 7 Done Done Sent false
29669 8690 Acked Read 8 Done Done Sent false
29670 8691 Acked Flush 0 Done Done Sent false
29671 8692 Acked Flush 0 Done Done Sent false
29672 8693 Acked Read 10 Done Done Sent false
29673 8694 Acked Read 3 Done Done Sent false
29674 8695 Acked Read 6 Done Done Sent false
29675 8696 Acked Write 4 Done Done Sent false
29676 8697 Acked Flush 0 Done Done Sent false
29677 8698 Acked Read 10 Done Done Sent false
29678 8699 Acked Write 4 Done Done Sent false
29679 8700 Acked Read 4 Done Done Sent false
29680 8701 Acked Write 3 Done Done Sent false
29681 8702 Acked Flush 0 Done Done Sent false
29682 8703 Acked Read 5 Done Done Sent false
29683 8704 Acked Read 1 Done Done Sent false
29684 8705 Acked Read 2 Done Done Sent false
29685 8706 Acked Read 3 Done Done Sent false
29686 8707 Acked Write 2 Done Done Sent false
29687 8708 Acked Read 10 Done Done Sent false
29688 8709 Acked Read 5 Done Done Sent false
29689 8710 Acked Write 8 Done Done Sent false
29690 8711 Acked Read 8 Done Done Sent false
29691 8712 Acked Write 6 Done Done Sent false
29692 8713 Acked Write 6 Done Done Sent false
29693 8714 Acked Read 9 Done Done Sent false
29694 8715 Acked Write 2 Done Done Sent false
29695 8716 Acked Write 2 Done Done Sent false
29696 8717 Acked Read 2 Done Done Sent false
29697 8718 Acked Write 1 Done Done Sent false
29698 8719 Acked Write 5 Done Done Sent false
29699 8720 Acked Write 8 Done Done Sent false
29700 8721 Acked Write 3 Done Done Sent false
29701 8722 Acked Read 8 Done Done Sent false
29702 8723 Acked Read 7 Done Done Sent false
29703 8724 Acked Read 8 Done Done Sent false
29704 8725 Acked Read 6 Done Done Sent false
29705 8726 Acked Read 9 Done Done Sent false
29706 8727 Acked Read 6 Done Done Sent false
29707 8728 Acked Write 6 Done Done Sent false
29708 8729 Acked Write 10 Done Done Sent false
29709 8730 Acked Read 1 Done Done Sent false
29710 8731 Acked Read 9 Done Done Sent false
29711 8732 Acked Read 9 Done Done Sent false
29712 8733 Acked Flush 0 Done Done Sent false
29713 8734 Acked Read 6 Done Done Sent false
29714 8735 Acked Read 1 Done Done Sent false
29715 8736 Acked Read 9 Done Done Sent false
29716 8737 Acked Read 1 Done Done Sent false
29717 8738 Acked Write 5 Done Done Sent false
29718 8739 Acked Read 8 Done Done Sent false
29719 8740 Acked Write 2 Done Done Sent false
29720 8741 Acked Read 8 Done Done Sent false
29721 8742 Acked Write 9 Done Done Sent false
29722 8743 Acked Read 7 Done Done Sent false
29723 8744 Acked Write 4 Done Done Sent false
29724 8745 Acked Write 10 Done Done Sent false
29725 8746 Acked Flush 0 Done Done Sent false
29726 8747 Acked Read 2 Done Done Sent false
29727 8748 Acked Write 1 Done Done Sent false
29728 8749 Acked Write 4 Done Done Sent false
29729 8750 Acked Write 6 Done Done Sent false
29730 8751 Acked Write 6 Done Done Sent false
29731 8752 Acked Write 4 Done Done Sent false
29732 8753 Acked Write 10 Done Done Sent false
29733 8754 Acked Read 7 Done Done Sent false
29734 8755 Acked Write 3 Done Done Sent false
29735 8756 Acked Read 5 Done Done Sent false
29736 8757 Acked Write 1 Done Done Sent false
29737 8758 Acked Write 7 Done Done Sent false
29738 8759 Acked Write 8 Done Done Sent false
29739 8760 Acked Read 10 Done Done Sent false
29740 8761 Acked Flush 0 Done Done Sent false
29741 8762 Acked Read 4 Done Done Sent false
29742 8763 Acked Write 4 Done Done Sent false
29743 8764 Acked Write 10 Done Done Sent false
29744 8765 Acked Read 8 Done Done Sent false
29745 8766 Acked Write 4 Done Done Sent false
29746 8767 Acked Read 10 Done Done Sent false
29747 8768 Acked Flush 0 Done Done Sent false
29748 8769 Acked Write 8 Done Done Sent false
29749 8770 Acked Read 4 Done Done Sent false
29750 8771 Acked Read 1 Done Done Sent false
29751 8772 Acked Read 3 Done Done Sent false
29752 8773 Acked Write 3 Done Done Sent false
29753 8774 Acked Write 8 Done Done Sent false
29754 8775 Acked Read 5 Done Done Sent false
29755 8776 Acked Read 2 Done Done Sent false
29756 8777 Acked Write 7 Done Done Sent false
29757 8778 Acked Read 6 Done Done Sent false
29758 8779 Acked Write 1 Done Done Sent false
29759 8780 Acked Read 6 Done Done Sent false
29760 8781 Acked Flush 0 Done Done Sent false
29761 8782 Acked Write 8 Done Done Sent false
29762 8783 Acked Read 7 Done Done Sent false
29763 8784 Acked Write 4 Done Done Sent false
29764 8785 Acked Flush 0 Done Done Sent false
29765 8786 Acked Read 6 Done Done Sent false
29766 8787 Acked Write 7 Done Done Sent false
29767 8788 Acked Read 7 Done Done Sent false
29768 8789 Acked Write 6 Done Done Sent false
29769 8790 Acked Write 9 Done Done Sent false
29770 8791 Acked Flush 0 Done Done Sent false
29771 8792 Acked Flush 0 Done Done Sent false
29772 8793 Acked Write 4 Done Done Sent false
29773 8794 Acked Flush 0 Done Done Sent false
29774 8795 Acked Write 2 Done Done Sent false
29775 8796 Acked Read 7 Done Done Sent false
29776 8797 Acked Write 10 Done Done Sent false
29777 8798 Acked Read 9 Done Done Sent false
29778 8799 Acked Read 6 Done Done Sent false
29779 8800 Acked Flush 0 Done Done Sent false
29780 8801 Acked Write 5 Done Done Sent false
29781 8802 Acked Read 1 Done Done Sent false
29782 8803 Acked Read 4 Done Done Sent false
29783 8804 Acked Read 6 Done Done Sent false
29784 8805 Acked Write 5 Done Done Sent false
29785 8806 Acked Write 10 Done Done Sent false
29786 8807 Acked Read 7 Done Done Sent false
29787 8808 Acked Write 6 Done Done Sent false
29788 8809 Acked Write 3 Done Done Sent false
29789 8810 Acked Write 3 Done Done Sent false
29790 8811 Acked Read 1 Done Done Sent false
29791 8812 Acked Flush 0 Done Done Sent false
29792 8813 Acked Read 5 Done Done Sent false
29793 8814 Acked Write 5 Done Done Sent false
29794 8815 Acked Write 10 Done Done Sent false
29795 8816 Acked Write 7 Done Done Sent false
29796 8817 Acked Write 9 Done Done Sent false
29797 8818 Acked Read 1 Done Done Sent false
29798 8819 Acked Read 10 Done Done Sent false
29799 8820 Acked Read 1 Done Done Sent false
29800 8821 Acked Flush 0 Done Done Sent false
29801 8822 Acked Read 3 Done Done Sent false
29802 8823 Acked Read 5 Done Done Sent false
29803 8824 Acked Write 5 Done Done Sent false
29804 8825 Acked Read 1 Done Done Sent false
29805 8826 Acked Flush 0 Done Done Sent false
29806 8827 Acked Write 10 Done Done Sent false
29807 8828 Acked Write 8 Done Done Sent false
29808 8829 Acked Write 2 Done Done Sent false
29809 8830 Acked Read 10 Done Done Sent false
29810 8831 Acked Read 8 Done Done Sent false
29811 8832 Acked Read 6 Done Done Sent false
29812 8833 Acked Read 7 Done Done Sent false
29813 8834 Acked Write 1 Done Done Sent false
29814 8835 Acked Read 1 Done Done Sent false
29815 8836 Acked Read 1 Done Done Sent false
29816 8837 Acked Write 10 Done Done Sent false
29817 8838 Acked Read 3 Done Done Sent false
29818 8839 Acked Read 7 Done Done Sent false
29819 8840 Acked Read 1 Done Done Sent false
29820 8841 Acked Read 6 Done Done Sent false
29821 8842 Acked Write 8 Done Done Sent false
29822 8843 Acked Write 2 Done Done Sent false
29823 8844 Acked Write 10 Done Done Sent false
29824 8845 Acked Flush 0 Done Done Sent false
29825 8846 Acked Write 7 Done Done Sent false
29826 8847 Acked Read 9 Done Done Sent false
29827 8848 Acked Read 1 Done Done Sent false
29828 8849 Acked Read 6 Done Done Sent false
29829 STATES DS:0 DS:1 DS:2 TOTAL
29830 Sent 0 0 301 301
29831 Done 301 301 0 602
29832 Skipped 0 0 0 0
29833 Error 0 0 0 0
29834 Last Flush: Some(JobId(8845)) Some(JobId(8845)) Some(JobId(8548))
29835 Downstairs last five completed: 8548 8547 8546 8545 8544
29836 Upstairs last five completed: 8849 8848 8847 8846 8845
29837 CLIENT: Up:0 ds:301 act:2
298382025-01-29T05:17:18.574ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
298392025-01-29T05:17:19.576ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
298402025-01-29T05:17:20.578ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
29841 [8830] got start action
29842 [8830] I am going to restart
29843 Starting downstairs at port 8830
29844 Make output file at "/tmp/dsc/downstairs-8830.txt"
29845 [8830][2] reports Starting
29846 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
29847 [8830][2] reports Running
29848 ----------------------------------------------------------------
29849 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29850 States: Active Active Offline
29851 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29852 8549 Acked Read 4 Done Done Sent false
29853 8550 Acked Read 10 Done Done Sent false
29854 8551 Acked Read 7 Done Done Sent false
29855 8552 Acked Read 1 Done Done Sent false
29856 8553 Acked Write 1 Done Done Sent false
29857 8554 Acked Write 3 Done Done Sent false
29858 8555 Acked Read 9 Done Done Sent false
29859 8556 Acked Read 6 Done Done Sent false
29860 8557 Acked Read 3 Done Done Sent false
29861 8558 Acked Read 4 Done Done Sent false
29862 8559 Acked Read 3 Done Done Sent false
29863 8560 Acked Write 2 Done Done Sent false
29864 8561 Acked Write 9 Done Done Sent false
29865 8562 Acked Write 2 Done Done Sent false
29866 8563 Acked Write 4 Done Done Sent false
29867 8564 Acked Read 8 Done Done Sent false
29868 8565 Acked Read 1 Done Done Sent false
29869 8566 Acked Write 9 Done Done Sent false
29870 8567 Acked Read 5 Done Done Sent false
29871 8568 Acked Read 5 Done Done Sent false
29872 8569 Acked Flush 0 Done Done Sent false
29873 8570 Acked Write 7 Done Done Sent false
29874 8571 Acked Read 3 Done Done Sent false
29875 8572 Acked Read 9 Done Done Sent false
29876 8573 Acked Read 4 Done Done Sent false
29877 8574 Acked Read 5 Done Done Sent false
29878 8575 Acked Read 6 Done Done Sent false
29879 8576 Acked Read 6 Done Done Sent false
29880 8577 Acked Write 4 Done Done Sent false
29881 8578 Acked Read 8 Done Done Sent false
29882 8579 Acked Write 7 Done Done Sent false
29883 8580 Acked Write 1 Done Done Sent false
29884 8581 Acked Read 1 Done Done Sent false
29885 8582 Acked Flush 0 Done Done Sent false
29886 8583 Acked Read 10 Done Done Sent false
29887 8584 Acked Read 7 Done Done Sent false
29888 8585 Acked Write 7 Done Done Sent false
29889 8586 Acked Flush 0 Done Done Sent false
29890 8587 Acked Read 1 Done Done Sent false
29891 8588 Acked Read 9 Done Done Sent false
29892 8589 Acked Read 5 Done Done Sent false
29893 8590 Acked Write 6 Done Done Sent false
29894 8591 Acked Flush 0 Done Done Sent false
29895 8592 Acked Flush 0 Done Done Sent false
29896 8593 Acked Write 10 Done Done Sent false
29897 8594 Acked Read 10 Done Done Sent false
29898 8595 Acked Write 2 Done Done Sent false
29899 8596 Acked Write 6 Done Done Sent false
29900 8597 Acked Write 9 Done Done Sent false
29901 8598 Acked Read 10 Done Done Sent false
29902 8599 Acked Read 7 Done Done Sent false
29903 8600 Acked Read 9 Done Done Sent false
29904 8601 Acked Read 2 Done Done Sent false
29905 8602 Acked Read 10 Done Done Sent false
29906 8603 Acked Write 6 Done Done Sent false
29907 8604 Acked Write 3 Done Done Sent false
29908 8605 Acked Read 8 Done Done Sent false
29909 8606 Acked Write 7 Done Done Sent false
29910 8607 Acked Write 2 Done Done Sent false
29911 8608 Acked Flush 0 Done Done Sent false
29912 8609 Acked Write 7 Done Done Sent false
29913 8610 Acked Read 8 Done Done Sent false
29914 8611 Acked Write 5 Done Done Sent false
29915 8612 Acked Write 2 Done Done Sent false
29916 8613 Acked Write 2 Done Done Sent false
29917 8614 Acked Write 6 Done Done Sent false
29918 8615 Acked Write 6 Done Done Sent false
29919 8616 Acked Read 5 Done Done Sent false
29920 8617 Acked Read 1 Done Done Sent false
29921 8618 Acked Read 2 Done Done Sent false
29922 8619 Acked Read 5 Done Done Sent false
29923 8620 Acked Read 7 Done Done Sent false
29924 8621 Acked Write 4 Done Done Sent false
29925 8622 Acked Write 1 Done Done Sent false
29926 8623 Acked Read 2 Done Done Sent false
29927 8624 Acked Read 9 Done Done Sent false
29928 8625 Acked Read 10 Done Done Sent false
29929 8626 Acked Write 7 Done Done Sent false
29930 8627 Acked Read 6 Done Done Sent false
29931 8628 Acked Write 5 Done Done Sent false
29932 8629 Acked Read 3 Done Done Sent false
29933 8630 Acked Write 5 Done Done Sent false
29934 8631 Acked Write 3 Done Done Sent false
29935 8632 Acked Read 10 Done Done Sent false
29936 8633 Acked Read 1 Done Done Sent false
29937 8634 Acked Read 3 Done Done Sent false
29938 8635 Acked Read 2 Done Done Sent false
29939 8636 Acked Write 7 Done Done Sent false
29940 8637 Acked Read 2 Done Done Sent false
29941 8638 Acked Read 6 Done Done Sent false
29942 8639 Acked Write 8 Done Done Sent false
29943 8640 Acked Write 9 Done Done Sent false
29944 8641 Acked Read 6 Done Done Sent false
29945 8642 Acked Read 9 Done Done Sent false
29946 8643 Acked Write 5 Done Done Sent false
29947 8644 Acked Read 6 Done Done Sent false
29948 8645 Acked Read 5 Done Done Sent false
29949 8646 Acked Write 2 Done Done Sent false
29950 8647 Acked Flush 0 Done Done Sent false
29951 8648 Acked Read 1 Done Done Sent false
29952 8649 Acked Write 10 Done Done Sent false
29953 8650 Acked Read 3 Done Done Sent false
29954 8651 Acked Write 7 Done Done Sent false
29955 8652 Acked Write 2 Done Done Sent false
29956 8653 Acked Write 10 Done Done Sent false
29957 8654 Acked Read 10 Done Done Sent false
29958 8655 Acked Read 8 Done Done Sent false
29959 8656 Acked Read 7 Done Done Sent false
29960 8657 Acked Write 7 Done Done Sent false
29961 8658 Acked Read 8 Done Done Sent false
29962 8659 Acked Flush 0 Done Done Sent false
29963 8660 Acked Write 9 Done Done Sent false
29964 8661 Acked Write 4 Done Done Sent false
29965 8662 Acked Write 9 Done Done Sent false
29966 8663 Acked Write 2 Done Done Sent false
29967 8664 Acked Read 8 Done Done Sent false
29968 8665 Acked Read 10 Done Done Sent false
29969 8666 Acked Read 8 Done Done Sent false
29970 8667 Acked Read 4 Done Done Sent false
29971 8668 Acked Read 5 Done Done Sent false
29972 8669 Acked Read 3 Done Done Sent false
29973 8670 Acked Write 5 Done Done Sent false
29974 8671 Acked Write 10 Done Done Sent false
29975 8672 Acked Read 10 Done Done Sent false
29976 8673 Acked Read 5 Done Done Sent false
29977 8674 Acked Read 6 Done Done Sent false
29978 8675 Acked Write 10 Done Done Sent false
29979 8676 Acked Write 7 Done Done Sent false
29980 8677 Acked Read 1 Done Done Sent false
29981 8678 Acked Write 10 Done Done Sent false
29982 8679 Acked Write 8 Done Done Sent false
29983 8680 Acked Write 7 Done Done Sent false
29984 8681 Acked Write 7 Done Done Sent false
29985 8682 Acked Write 9 Done Done Sent false
29986 8683 Acked Read 3 Done Done Sent false
29987 8684 Acked Read 9 Done Done Sent false
29988 8685 Acked Read 8 Done Done Sent false
29989 8686 Acked Flush 0 Done Done Sent false
29990 8687 Acked Write 9 Done Done Sent false
29991 8688 Acked Read 3 Done Done Sent false
29992 8689 Acked Write 7 Done Done Sent false
29993 8690 Acked Read 8 Done Done Sent false
29994 8691 Acked Flush 0 Done Done Sent false
29995 8692 Acked Flush 0 Done Done Sent false
29996 8693 Acked Read 10 Done Done Sent false
29997 8694 Acked Read 3 Done Done Sent false
29998 8695 Acked Read 6 Done Done Sent false
29999 8696 Acked Write 4 Done Done Sent false
30000 8697 Acked Flush 0 Done Done Sent false
30001 8698 Acked Read 10 Done Done Sent false
30002 8699 Acked Write 4 Done Done Sent false
30003 8700 Acked Read 4 Done Done Sent false
30004 8701 Acked Write 3 Done Done Sent false
30005 8702 Acked Flush 0 Done Done Sent false
30006 8703 Acked Read 5 Done Done Sent false
30007 8704 Acked Read 1 Done Done Sent false
30008 8705 Acked Read 2 Done Done Sent false
30009 8706 Acked Read 3 Done Done Sent false
30010 8707 Acked Write 2 Done Done Sent false
30011 8708 Acked Read 10 Done Done Sent false
30012 8709 Acked Read 5 Done Done Sent false
30013 8710 Acked Write 8 Done Done Sent false
30014 8711 Acked Read 8 Done Done Sent false
30015 8712 Acked Write 6 Done Done Sent false
30016 8713 Acked Write 6 Done Done Sent false
30017 8714 Acked Read 9 Done Done Sent false
30018 8715 Acked Write 2 Done Done Sent false
30019 8716 Acked Write 2 Done Done Sent false
30020 8717 Acked Read 2 Done Done Sent false
30021 8718 Acked Write 1 Done Done Sent false
30022 8719 Acked Write 5 Done Done Sent false
30023 8720 Acked Write 8 Done Done Sent false
30024 8721 Acked Write 3 Done Done Sent false
30025 8722 Acked Read 8 Done Done Sent false
30026 8723 Acked Read 7 Done Done Sent false
30027 8724 Acked Read 8 Done Done Sent false
30028 8725 Acked Read 6 Done Done Sent false
30029 8726 Acked Read 9 Done Done Sent false
30030 8727 Acked Read 6 Done Done Sent false
30031 8728 Acked Write 6 Done Done Sent false
30032 8729 Acked Write 10 Done Done Sent false
30033 8730 Acked Read 1 Done Done Sent false
30034 8731 Acked Read 9 Done Done Sent false
30035 8732 Acked Read 9 Done Done Sent false
30036 8733 Acked Flush 0 Done Done Sent false
30037 8734 Acked Read 6 Done Done Sent false
30038 8735 Acked Read 1 Done Done Sent false
30039 8736 Acked Read 9 Done Done Sent false
30040 8737 Acked Read 1 Done Done Sent false
30041 8738 Acked Write 5 Done Done Sent false
30042 8739 Acked Read 8 Done Done Sent false
30043 8740 Acked Write 2 Done Done Sent false
30044 8741 Acked Read 8 Done Done Sent false
30045 8742 Acked Write 9 Done Done Sent false
30046 8743 Acked Read 7 Done Done Sent false
30047 8744 Acked Write 4 Done Done Sent false
30048 8745 Acked Write 10 Done Done Sent false
30049 8746 Acked Flush 0 Done Done Sent false
30050 8747 Acked Read 2 Done Done Sent false
30051 8748 Acked Write 1 Done Done Sent false
30052 8749 Acked Write 4 Done Done Sent false
30053 8750 Acked Write 6 Done Done Sent false
30054 8751 Acked Write 6 Done Done Sent false
30055 8752 Acked Write 4 Done Done Sent false
30056 8753 Acked Write 10 Done Done Sent false
30057 8754 Acked Read 7 Done Done Sent false
30058 8755 Acked Write 3 Done Done Sent false
30059 8756 Acked Read 5 Done Done Sent false
30060 8757 Acked Write 1 Done Done Sent false
30061 8758 Acked Write 7 Done Done Sent false
30062 8759 Acked Write 8 Done Done Sent false
30063 8760 Acked Read 10 Done Done Sent false
30064 8761 Acked Flush 0 Done Done Sent false
30065 8762 Acked Read 4 Done Done Sent false
30066 8763 Acked Write 4 Done Done Sent false
30067 8764 Acked Write 10 Done Done Sent false
30068 8765 Acked Read 8 Done Done Sent false
30069 8766 Acked Write 4 Done Done Sent false
30070 8767 Acked Read 10 Done Done Sent false
30071 8768 Acked Flush 0 Done Done Sent false
30072 8769 Acked Write 8 Done Done Sent false
30073 8770 Acked Read 4 Done Done Sent false
30074 8771 Acked Read 1 Done Done Sent false
30075 8772 Acked Read 3 Done Done Sent false
30076 8773 Acked Write 3 Done Done Sent false
30077 8774 Acked Write 8 Done Done Sent false
30078 8775 Acked Read 5 Done Done Sent false
30079 8776 Acked Read 2 Done Done Sent false
30080 8777 Acked Write 7 Done Done Sent false
30081 8778 Acked Read 6 Done Done Sent false
30082 8779 Acked Write 1 Done Done Sent false
30083 8780 Acked Read 6 Done Done Sent false
30084 8781 Acked Flush 0 Done Done Sent false
30085 8782 Acked Write 8 Done Done Sent false
30086 8783 Acked Read 7 Done Done Sent false
30087 8784 Acked Write 4 Done Done Sent false
30088 8785 Acked Flush 0 Done Done Sent false
30089 8786 Acked Read 6 Done Done Sent false
30090 8787 Acked Write 7 Done Done Sent false
30091 8788 Acked Read 7 Done Done Sent false
30092 8789 Acked Write 6 Done Done Sent false
30093 8790 Acked Write 9 Done Done Sent false
30094 8791 Acked Flush 0 Done Done Sent false
30095 8792 Acked Flush 0 Done Done Sent false
30096 8793 Acked Write 4 Done Done Sent false
30097 8794 Acked Flush 0 Done Done Sent false
30098 8795 Acked Write 2 Done Done Sent false
30099 8796 Acked Read 7 Done Done Sent false
30100 8797 Acked Write 10 Done Done Sent false
30101 8798 Acked Read 9 Done Done Sent false
30102 8799 Acked Read 6 Done Done Sent false
30103 8800 Acked Flush 0 Done Done Sent false
30104 8801 Acked Write 5 Done Done Sent false
30105 8802 Acked Read 1 Done Done Sent false
30106 8803 Acked Read 4 Done Done Sent false
30107 8804 Acked Read 6 Done Done Sent false
30108 8805 Acked Write 5 Done Done Sent false
30109 8806 Acked Write 10 Done Done Sent false
30110 8807 Acked Read 7 Done Done Sent false
30111 8808 Acked Write 6 Done Done Sent false
30112 8809 Acked Write 3 Done Done Sent false
30113 8810 Acked Write 3 Done Done Sent false
30114 8811 Acked Read 1 Done Done Sent false
30115 8812 Acked Flush 0 Done Done Sent false
30116 8813 Acked Read 5 Done Done Sent false
30117 8814 Acked Write 5 Done Done Sent false
30118 8815 Acked Write 10 Done Done Sent false
30119 8816 Acked Write 7 Done Done Sent false
30120 8817 Acked Write 9 Done Done Sent false
30121 8818 Acked Read 1 Done Done Sent false
30122 8819 Acked Read 10 Done Done Sent false
30123 8820 Acked Read 1 Done Done Sent false
30124 8821 Acked Flush 0 Done Done Sent false
30125 8822 Acked Read 3 Done Done Sent false
30126 8823 Acked Read 5 Done Done Sent false
30127 8824 Acked Write 5 Done Done Sent false
30128 8825 Acked Read 1 Done Done Sent false
30129 8826 Acked Flush 0 Done Done Sent false
30130 8827 Acked Write 10 Done Done Sent false
30131 8828 Acked Write 8 Done Done Sent false
30132 8829 Acked Write 2 Done Done Sent false
30133 8830 Acked Read 10 Done Done Sent false
30134 8831 Acked Read 8 Done Done Sent false
30135 8832 Acked Read 6 Done Done Sent false
30136 8833 Acked Read 7 Done Done Sent false
30137 8834 Acked Write 1 Done Done Sent false
30138 8835 Acked Read 1 Done Done Sent false
30139 8836 Acked Read 1 Done Done Sent false
30140 8837 Acked Write 10 Done Done Sent false
30141 8838 Acked Read 3 Done Done Sent false
30142 8839 Acked Read 7 Done Done Sent false
30143 8840 Acked Read 1 Done Done Sent false
30144 8841 Acked Read 6 Done Done Sent false
30145 8842 Acked Write 8 Done Done Sent false
30146 8843 Acked Write 2 Done Done Sent false
30147 8844 Acked Write 10 Done Done Sent false
30148 8845 Acked Flush 0 Done Done Sent false
30149 8846 Acked Write 7 Done Done Sent false
30150 8847 Acked Read 9 Done Done Sent false
30151 8848 Acked Read 1 Done Done Sent false
30152 8849 Acked Read 6 Done Done Sent false
30153 STATES DS:0 DS:1 DS:2 TOTAL
30154 Sent 0 0 301 301
30155 Done 301 301 0 602
30156 Skipped 0 0 0 0
30157 Error 0 0 0 0
30158 Last Flush: Some(JobId(8845)) Some(JobId(8845)) Some(JobId(8548))
30159 Downstairs last five completed: 8548 8547 8546 8545 8544
30160 Upstairs last five completed: 8849 8848 8847 8846 8845
30161 CLIENT: Up:0 ds:301 act:2
301622025-01-29T05:17:22.582ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301632025-01-29T05:17:23.584ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301642025-01-29T05:17:23.762ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301652025-01-29T05:17:23.762ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44730) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301662025-01-29T05:17:23.763ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301672025-01-29T05:17:23.764ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301682025-01-29T05:17:23.764ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301692025-01-29T05:17:23.764ZINFOcrucible: send last flush ID to this DS: Some(JobId(8548)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301702025-01-29T05:17:23.764ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301712025-01-29T05:17:23.764ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8548)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301722025-01-29T05:17:23.764ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8548)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
30173 ----------------------------------------------------------------
30174 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30175 Downstairs last five completed: 8850 8849 8848 8847 8846
30176 Upstairs last five completed: 8850 8849 8848 8847 8846
30177 CLIENT: Up:0 ds:0 act:3
30178 Replay: All jobs finished
30179 Main task has work to do, go find it
30180 got dsc Stop(1)
30181 stop 1
30182 [8820] Got stop action so:false kr:false
30183 Jan 29 05:17:26.015 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b97e48c9-42a5-42ba-8a46-c7816ae6e3b2, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
301842025-01-29T05:17:26.016ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301852025-01-29T05:17:26.016ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
301862025-01-29T05:17:26.016ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
30187 Jan 29 05:17:26.016 INFO request completed, latency_us: 167{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:17:26.016429158Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"1","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
30188 , response_code: 200, uri: /state/cid/1, method: GET, req_id: f894d1c1-adb2-4adf-8208-8048c761633d, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
30189 [8820] Exited with: ExitStatus(unix_wait_status(9))
30190 [8820][1] reports Exit
30191 Jan 29 05:17:30.019 INFO request completed, latency_us: 234, response_code: 200, uri: /state/cid/1, method: GET, req_id: d4ca0252-2c74-4840-8afe-361d3427a4bd, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
30192 001/300 Read block 4288 len 32768
30193 002/300 Write block 4492 len 24576 data: 5 5 4 4 6 6
30194 003/300 Read block 4168 len 8192
30195 004/300 Write block 2845 len 40960 data: 4 4 5 4 4 5 5 5 5 4
30196 005/300 Write block 2665 len 8192 data: 8 6
30197 006/300 Read block 4918 len 40960
30198 007/300 Read block 3448 len 12288
30199 008/300 Flush
30200 009/300 Write block 268 len 8192 data: 4 3
30201 010/300 Read block 3702 len 32768
30202 011/300 Read block 62 len 36864
30203 012/300 Flush
30204 013/300 Write block 1811 len 24576 data: 5 5 5 5 5 5
30205 014/300 Read block 1620 len 28672
30206 015/300 Flush
30207 016/300 Read block 1366 len 40960
30208 017/300 Write block 2764 len 4096 data: 3
30209 018/300 Write block 4429 len 32768 data: 2 2 2 2 3 3 3 4
30210 019/300 Read block 3025 len 36864
30211 020/300 Read block 2153 len 8192
30212 021/300 Write block 3863 len 24576 data: 3 3 4 4 4 5
30213 022/300 Read block 2629 len 4096
30214 023/300 Write block 678 len 36864 data: 4 4 3 2 3 3 3 3 4
30215 024/300 Write block 2947 len 40960 data: 8 8 6 5 4 3 4 6 6 5
30216 025/300 Read block 3469 len 20480
30217 026/300 Flush
30218 027/300 Write block 4146 len 12288 data: 5 4 5
30219 028/300 Read block 6 len 12288
30220 029/300 Write block 2091 len 20480 data: 6 7 7 7 5
30221 030/300 Read block 2496 len 36864
30222 031/300 Read block 231 len 4096
30223 032/300 Read block 87 len 40960
30224 033/300 Write block 4699 len 36864 data: 6 6 6 6 6 6 5 4 4
30225 034/300 Write block 712 len 16384 data: 4 4 3 7
30226 035/300 Read block 4177 len 20480
30227 036/300 Read block 1397 len 32768
30228 037/300 Read block 4386 len 36864
30229 038/300 Flush
30230 039/300 Write block 3000 len 24576 data: 6 5 7 7 6 5
30231 040/300 Write block 1905 len 24576 data: 6 8 8 8 7 6
30232 041/300 Write block 3742 len 16384 data: 5 5 5 5
30233 042/300 Write block 821 len 40960 data: 5 4 5 6 7 8 5 4 6 6
30234 043/300 Write block 3455 len 16384 data: 3 3 3 5
30235 044/300 Write block 2660 len 24576 data: 6 5 5 7 6 9
30236 045/300 Read block 4039 len 20480
30237 046/300 Write block 3533 len 36864 data: 6 6 6 3 3 2 2 3 4
30238 047/300 Write block 740 len 28672 data: 8 7 8 8 11 13 8
30239 048/300 Read block 3982 len 20480
30240 049/300 Write block 1156 len 36864 data: 4 5 3 4 4 5 5 4 5
30241 050/300 Read block 3098 len 20480
30242 051/300 Write block 2406 len 24576 data: 4 4 4 6 5 6
30243 052/300 Read block 599 len 32768
30244 053/300 Read block 1687 len 4096
30245 054/300 Read block 3559 len 32768
30246 055/300 Write block 1085 len 36864 data: 6 6 7 5 5 6 6 7 8
30247 056/300 Write block 4526 len 40960 data: 5 7 9 9 10 10 10 11 12 10
30248 057/300 Read block 3806 len 32768
30249 058/300 Read block 347 len 16384
30250 059/300 Read block 4972 len 20480
30251 060/300 Write block 3011 len 4096 data: 4
30252 061/300 Read block 1615 len 40960
30253 062/300 Write block 3941 len 4096 data: 9
30254 063/300 Write block 3802 len 8192 data: 10 9
30255 064/300 Write block 1885 len 40960 data: 7 7 7 8 8 7 6 5 6 4
30256 065/300 Read block 4345 len 12288
30257 066/300 Write block 4307 len 28672 data: 6 9 8 5 5 5 7
30258 067/300 Write block 590 len 36864 data: 10 8 8 7 9 9 7 6 6
30259 068/300 Read block 4804 len 16384
30260 069/300 Read block 3184 len 36864
30261 070/300 Read block 1775 len 16384
30262 071/300 Read block 3500 len 12288
30263 072/300 Read block 1219 len 40960
30264 073/300 Write block 2957 len 36864 data: 5 5 5 7 8 8 8 7 7
30265 074/300 Read block 2832 len 16384
30266 075/300 Read block 416 len 16384
30267 076/300 Write block 3229 len 40960 data: 9 9 9 9 10 8 9 9 9 5
30268 077/300 Read block 3489 len 36864
30269 078/300 Write block 3167 len 20480 data: 5 6 6 6 5
30270 079/300 Read block 193 len 28672
30271 080/300 Write block 293 len 28672 data: 4 5 5 5 4 4 4
30272 081/300 Read block 475 len 28672
30273 082/300 Read block 3685 len 12288
30274 083/300 Flush
30275 084/300 Read block 3613 len 28672
30276 085/300 Read block 662 len 40960
30277 086/300 Write block 3678 len 8192 data: 5 5
30278 087/300 Read block 374 len 28672
30279 088/300 Write block 3824 len 32768 data: 6 5 5 7 6 6 7 8
30280 089/300 Read block 1390 len 32768
30281 090/300 Write block 1538 len 4096 data: 7
30282 091/300 Write block 2012 len 28672 data: 9 7 7 7 6 5 4
30283 092/300 Write block 2287 len 16384 data: 3 4 4 6
30284 093/300 Write block 736 len 20480 data: 10 9 9 8 9
30285 094/300 Read block 4439 len 20480
30286 095/300 Write block 4090 len 8192 data: 5 5
30287 096/300 Write block 1635 len 12288 data: 4 4 5
30288 097/300 Read block 1043 len 28672
30289 098/300 Read block 2963 len 36864
30290 099/300 Write block 4433 len 16384 data: 4 4 4 5
30291 100/300 Write block 416 len 12288 data: 4 5 5
30292 101/300 Read block 572 len 36864
30293 102/300 Read block 114 len 12288
30294 103/300 Read block 315 len 12288
30295 104/300 Write block 2914 len 28672 data: 6 7 8 8 8 8 5
30296 105/300 Read block 60 len 8192
30297 106/300 Write block 2168 len 24576 data: 3 4 7 9 8 7
30298 107/300 Write block 1662 len 20480 data: 4 5 4 4 4
30299 108/300 Read block 4482 len 32768
30300 109/300 Read block 3764 len 20480
30301 110/300 Read block 1806 len 32768
30302 111/300 Flush
30303 112/300 Flush
30304 113/300 Write block 3662 len 24576 data: 3 3 3 2 2 4
30305 114/300 Read block 2803 len 24576
30306 115/300 Write block 2873 len 28672 data: 4 4 5 5 4 4 3
30307 116/300 Read block 295 len 28672
30308 117/300 Read block 2337 len 40960
30309 118/300 Read block 2652 len 20480
30310 119/300 Read block 4317 len 24576
30311 120/300 Write block 2170 len 4096 data: 8
30312 121/300 Read block 988 len 28672
30313 122/300 Write block 2978 len 8192 data: 8 7
30314 123/300 Read block 4015 len 4096
30315 124/300 Flush
30316 125/300 Read block 3072 len 24576
30317 126/300 Write block 4303 len 4096 data: 6
30318 127/300 Write block 1682 len 12288 data: 10 11 12
30319 128/300 Read block 2357 len 40960
30320 129/300 Read block 2617 len 20480
30321 130/300 Write block 3030 len 8192 data: 6 5
30322 131/300 Read block 4981 len 8192
30323 132/300 Write block 4118 len 4096 data: 6
30324 133/300 Flush
30325 134/300 Read block 4092 len 36864
30326 135/300 Read block 453 len 28672
30327 136/300 Write block 2224 len 32768 data: 3 3 4 4 5 6 8 9
30328 137/300 Read block 4406 len 16384
30329 138/300 Write block 2946 len 40960 data: 8 9 9 7 6 5 4 5 7 7
30330 139/300 Write block 1494 len 16384 data: 4 5 6 7
30331 140/300 Read block 30 len 8192
30332 141/300 Write block 242 len 20480 data: 9 8 8 9 8
30333 142/300 Read block 3185 len 8192
30334 143/300 Flush
30335 144/300 Read block 3520 len 4096
30336 145/300 Write block 2207 len 24576 data: 5 6 6 7 6 6
30337 146/300 Read block 1427 len 36864
30338 147/300 Read block 874 len 12288
30339 148/300 Flush
30340 149/300 Flush
30341 150/300 Read block 4320 len 28672
30342 151/300 Read block 1195 len 36864
30343 152/300 Read block 2139 len 28672
30344 153/300 Flush
30345 154/300 Write block 3122 len 28672 data: 6 6 5 4 5 6 6
30346 155/300 Flush
30347 156/300 Write block 2997 len 20480 data: 4 5 6 7 6
30348 157/300 Read block 3187 len 4096
30349 158/300 Read block 2913 len 40960
30350 159/300 Read block 1666 len 16384
30351 160/300 Read block 323 len 12288
30352 161/300 Read block 990 len 24576
30353 162/300 Read block 2416 len 28672
30354 163/300 Read block 4739 len 20480
30355 164/300 Flush
30356 165/300 Read block 646 len 28672
30357 166/300 Read block 2703 len 24576
30358 167/300 Read block 3429 len 32768
30359 168/300 Read block 3308 len 20480
30360 169/300 Write block 1863 len 28672 data: 6 5 5 5 6 7 5
30361 170/300 Read block 1182 len 16384
30362 171/300 Write block 2850 len 24576 data: 6 6 6 6 5 4
30363 172/300 Write block 3320 len 16384 data: 4 4 5 5
30364 173/300 Read block 2059 len 32768
30365 174/300 Write block 4933 len 16384 data: 8 6 6 5
30366 175/300 Read block 3327 len 40960
30367 176/300 Read block 4756 len 20480
30368 177/300 Write block 4228 len 12288 data: 6 5 4
30369 178/300 Read block 219 len 8192
30370 179/300 Read block 4480 len 24576
30371 180/300 Write block 3854 len 8192 data: 4 5
30372 181/300 Write block 4314 len 4096 data: 7
30373 182/300 Write block 3594 len 16384 data: 4 4 4 2
30374 183/300 Write block 3552 len 12288 data: 5 2 2
30375 184/300 Write block 801 len 8192 data: 5 5
30376 185/300 Write block 4132 len 12288 data: 5 5 5
30377 186/300 Read block 4617 len 40960
30378 187/300 Write block 4201 len 8192 data: 9 9
30379 188/300 Read block 1848 len 32768
30380 189/300 Read block 1516 len 32768
30381 190/300 Read block 1503 len 40960
30382 191/300 Read block 845 len 40960
30383 192/300 Write block 1819 len 20480 data: 8 8 6 5 7
30384 193/300 Write block 105 len 4096 data: 5
30385 194/300 Write block 4531 len 28672 data: 11 11 12 13 11 7 7
30386 195/300 Write block 1700 len 36864 data: 4 5 5 7 7 6 6 6 6
30387 196/300 Read block 2017 len 16384
30388 197/300 Write block 3869 len 28672 data: 5 5 4 5 6 8 9
30389 198/300 Write block 3430 len 36864 data: 8 6 6 6 4 4 4 2 3
30390 199/300 Write block 3996 len 32768 data: 4 5 4 4 5 5 6 7
30391 200/300 Read block 2498 len 20480
30392 201/300 Write block 1944 len 28672 data: 9 7 7 7 9 9 8
30393 202/300 Read block 4757 len 12288
30394 203/300 Read block 1705 len 8192
30395 204/300 Read block 2702 len 36864
30396 205/300 Read block 2925 len 32768
30397 206/300 Read block 1985 len 8192
30398 207/300 Read block 424 len 8192
30399 208/300 Read block 4167 len 32768
30400 209/300 Read block 222 len 8192
30401 210/300 Read block 2425 len 40960
30402 211/300 Read block 2728 len 16384
30403 212/300 Write block 2858 len 28672 data: 5 5 5 4 5 6 6
30404 213/300 Read block 4500 len 32768
30405 214/300 Write block 3599 len 24576 data: 2 3 4 4 3 2
30406 215/300 Read block 3073 len 24576
30407 216/300 Write block 4940 len 8192 data: 5 4
30408 217/300 Flush
30409 218/300 Flush
30410 219/300 Read block 4540 len 20480
30411 220/300 Read block 4355 len 28672
30412 221/300 Read block 4240 len 28672
30413 222/300 Read block 1277 len 36864
30414 223/300 Write block 4831 len 40960 data: 5 5 5 4 3 3 3 5 6 7
30415 224/300 Write block 4040 len 8192 data: 4 5
30416 225/300 Flush
30417 226/300 Write block 3320 len 16384 data: 5 5 6 6
30418 227/300 Write block 4055 len 8192 data: 4 5
30419 228/300 Flush
30420 229/300 Read block 2576 len 4096
30421 230/300 Read block 1451 len 24576
30422 231/300 Write block 2250 len 12288 data: 5 4 6
30423 232/300 Write block 4940 len 20480 data: 6 5 4 4 3
30424 233/300 Read block 2929 len 16384
30425 234/300 Read block 820 len 4096
30426 235/300 Read block 2799 len 40960
30427 236/300 Read block 954 len 24576
30428 237/300 Read block 4015 len 20480
30429 238/300 Write block 414 len 36864 data: 4 4 5 6 6 6 7 6 7
30430 239/300 Read block 4597 len 20480
30431 240/300 Write block 752 len 20480 data: 7 7 6 8 7
30432 241/300 Read block 2973 len 4096
30433 242/300 Write block 2076 len 8192 data: 3 3
30434 243/300 Write block 3769 len 28672 data: 8 6 6 6 5 5 5
30435 244/300 Read block 364 len 24576
30436 245/300 Flush
30437 246/300 Read block 3610 len 36864
30438 247/300 Read block 2321 len 36864
30439 248/300 Write block 5 len 24576 data: 5 6 6 5 5 4
30440 249/300 Read block 4381 len 16384
30441 250/300 Read block 1848 len 8192
30442 251/300 Read block 1765 len 40960
30443 252/300 Write block 4011 len 16384 data: 8 6 6 5
30444 253/300 Write block 58 len 20480 data: 4 7 7 8 7
30445 254/300 Read block 4235 len 8192
30446 255/300 Write block 1538 len 40960 data: 8 5 4 4 4 4 6 6 7 7
30447 256/300 Read block 3280 len 32768
30448 257/300 Read block 2221 len 20480
30449 258/300 Flush
30450 259/300 Flush
30451 260/300 Write block 2241 len 4096 data: 6
30452 261/300 Write block 3872 len 12288 data: 6 7 9
30453 262/300 Write block 3378 len 40960 data: 6 7 7 7 9 8 9 7 6 7
30454 263/300 Flush
30455 264/300 Write block 3608 len 4096 data: 7
30456 265/300 Read block 3445 len 36864
30457 266/300 Read block 2547 len 8192
30458 267/300 Read block 458 len 4096
30459 268/300 Read block 4189 len 16384
30460 269/300 Write block 2010 len 12288 data: 7 8 10
30461 270/300 Read block 1751 len 28672
30462 271/300 Read block 2346 len 8192
30463 272/300 Read block 4118 len 40960
30464 273/300 Read block 3958 len 40960
30465 274/300 Flush
30466 275/300 Read block 3126 len 24576
30467 276/300 Write block 2814 len 16384 data: 4 6 5 4
30468 277/300 Read block 4043 len 8192
30469 278/300 Read block 4973 len 8192
30470 279/300 Write block 1853 len 20480 data: 5 5 4 5 4
30471 280/300 Write block 787 len 36864 data: 3 3 3 3 7 8 8 8 8
30472 281/300 Read block 892 len 8192
30473 282/300 Write block 4427 len 24576 data: 6 5 3 3 3 3
30474 283/300 Read block 2554 len 36864
30475 284/300 Write block 933 len 24576 data: 6 5 5 3 3 3
30476 285/300 Read block 2284 len 32768
30477 286/300 Flush
30478 287/300 Read block 4214 len 24576
30479 288/300 Write block 3073 len 16384 data: 4 5 6 6
30480 289/300 Write block 3383 len 32768 data: 9 10 8 7 8 6 8 8
30481 290/300 Read block 2379 len 28672
30482 291/300 Flush
30483 292/300 Read block 1015 len 28672
30484 293/300 Read block 4164 len 24576
30485 294/300 Flush
30486 295/300 Write block 182 len 12288 data: 5 4 4
30487 296/300 Write block 2053 len 20480 data: 7 7 6 3 4
30488 297/300 Write block 1263 len 28672 data: 4 5 6 5 5 4 5
30489 298/300 Write block 4859 len 16384 data: 7 7 6 6
30490 299/300 Read block 2896 len 8192
30491 300/300 Write block 558 len 4096 data: 9
30492 301/300 Read block 2927 len 40960
30493 Main task has work to do, go find it
30494 got dsc Start(1)
30495 start 1
30496 Jan 29 05:17:30.267 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/1, method: POST, req_id: d8f0d145-b4fa-40b7-a18e-304f4d84a3ac, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
30497 Replay: started 1, returned:Ok(())
30498 ----------------------------------------------------------------
30499 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30500 States: Active Offline Active
30501 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30502 8851 Acked Read 8 Done Sent Done false
30503 8852 Acked Write 6 Done Sent Done false
30504 8853 Acked Read 2 Done Sent Done false
30505 8854 Acked Write 10 Done Sent Done false
30506 8855 Acked Write 2 Done Sent Done false
30507 8856 Acked Read 10 Done Sent Done false
30508 8857 Acked Read 3 Done Sent Done false
30509 8858 Acked Flush 0 Done Sent Done false
30510 8859 Acked Write 2 Done Sent Done false
30511 8860 Acked Read 8 Done Sent Done false
30512 8861 Acked Read 9 Done Sent Done false
30513 8862 Acked Flush 0 Done Sent Done false
30514 8863 Acked Write 6 Done Sent Done false
30515 8864 Acked Read 7 Done Sent Done false
30516 8865 Acked Flush 0 Done Sent Done false
30517 8866 Acked Read 10 Done Sent Done false
30518 8867 Acked Write 1 Done Sent Done false
30519 8868 Acked Write 8 Done Sent Done false
30520 8869 Acked Read 9 Done Sent Done false
30521 8870 Acked Read 2 Done Sent Done false
30522 8871 Acked Write 6 Done Sent Done false
30523 8872 Acked Read 1 Done Sent Done false
30524 8873 Acked Write 9 Done Sent Done false
30525 8874 Acked Write 10 Done Sent Done false
30526 8875 Acked Read 5 Done Sent Done false
30527 8876 Acked Flush 0 Done Sent Done false
30528 8877 Acked Write 3 Done Sent Done false
30529 8878 Acked Read 3 Done Sent Done false
30530 8879 Acked Write 5 Done Sent Done false
30531 8880 Acked Read 9 Done Sent Done false
30532 8881 Acked Read 1 Done Sent Done false
30533 8882 Acked Read 10 Done Sent Done false
30534 8883 Acked Write 9 Done Sent Done false
30535 8884 Acked Write 4 Done Sent Done false
30536 8885 Acked Read 5 Done Sent Done false
30537 8886 Acked Read 8 Done Sent Done false
30538 8887 Acked Read 9 Done Sent Done false
30539 8888 Acked Flush 0 Done Sent Done false
30540 8889 Acked Write 6 Done Sent Done false
30541 8890 Acked Write 6 Done Sent Done false
30542 8891 Acked Write 4 Done Sent Done false
30543 8892 Acked Write 10 Done Sent Done false
30544 8893 Acked Write 4 Done Sent Done false
30545 8894 Acked Write 6 Done Sent Done false
30546 8895 Acked Read 5 Done Sent Done false
30547 8896 Acked Write 9 Done Sent Done false
30548 8897 Acked Write 7 Done Sent Done false
30549 8898 Acked Read 5 Done Sent Done false
30550 8899 Acked Write 9 Done Sent Done false
30551 8900 Acked Read 5 Done Sent Done false
30552 8901 Acked Write 6 Done Sent Done false
30553 8902 Acked Read 8 Done Sent Done false
30554 8903 Acked Read 1 Done Sent Done false
30555 8904 Acked Read 8 Done Sent Done false
30556 8905 Acked Write 9 Done Sent Done false
30557 8906 Acked Write 10 Done Sent Done false
30558 8907 Acked Read 8 Done Sent Done false
30559 8908 Acked Read 4 Done Sent Done false
30560 8909 Acked Read 5 Done Sent Done false
30561 8910 Acked Write 1 Done Sent Done false
30562 8911 Acked Read 10 Done Sent Done false
30563 8912 Acked Write 1 Done Sent Done false
30564 8913 Acked Write 2 Done Sent Done false
30565 8914 Acked Write 10 Done Sent Done false
30566 8915 Acked Read 3 Done Sent Done false
30567 8916 Acked Write 7 Done Sent Done false
30568 8917 Acked Write 9 Done Sent Done false
30569 8918 Acked Read 4 Done Sent Done false
30570 8919 Acked Read 9 Done Sent Done false
30571 8920 Acked Read 4 Done Sent Done false
30572 8921 Acked Read 3 Done Sent Done false
30573 8922 Acked Read 10 Done Sent Done false
30574 8923 Acked Write 9 Done Sent Done false
30575 8924 Acked Read 4 Done Sent Done false
30576 8925 Acked Read 4 Done Sent Done false
30577 8926 Acked Write 10 Done Sent Done false
30578 8927 Acked Read 9 Done Sent Done false
30579 8928 Acked Write 5 Done Sent Done false
30580 8929 Acked Read 7 Done Sent Done false
30581 8930 Acked Write 7 Done Sent Done false
30582 8931 Acked Read 7 Done Sent Done false
30583 8932 Acked Read 3 Done Sent Done false
30584 8933 Acked Flush 0 Done Sent Done false
30585 8934 Acked Read 7 Done Sent Done false
30586 8935 Acked Read 10 Done Sent Done false
30587 8936 Acked Write 2 Done Sent Done false
30588 8937 Acked Read 7 Done Sent Done false
30589 8938 Acked Write 8 Done Sent Done false
30590 8939 Acked Read 8 Done Sent Done false
30591 8940 Acked Write 1 Done Sent Done false
30592 8941 Acked Write 7 Done Sent Done false
30593 8942 Acked Write 4 Done Sent Done false
30594 8943 Acked Write 5 Done Sent Done false
30595 8944 Acked Read 5 Done Sent Done false
30596 8945 Acked Write 2 Done Sent Done false
30597 8946 Acked Write 3 Done Sent Done false
30598 8947 Acked Read 7 Done Sent Done false
30599 8948 Acked Read 9 Done Sent Done false
30600 8949 Acked Write 4 Done Sent Done false
30601 8950 Acked Write 3 Done Sent Done false
30602 8951 Acked Read 9 Done Sent Done false
30603 8952 Acked Read 3 Done Sent Done false
30604 8953 Acked Read 3 Done Sent Done false
30605 8954 Acked Write 7 Done Sent Done false
30606 8955 Acked Read 2 Done Sent Done false
30607 8956 Acked Write 6 Done Sent Done false
30608 8957 Acked Write 5 Done Sent Done false
30609 8958 Acked Read 8 Done Sent Done false
30610 8959 Acked Read 5 Done Sent Done false
30611 8960 Acked Read 8 Done Sent Done false
30612 8961 Acked Flush 0 Done Sent Done false
30613 8962 Acked Flush 0 Done Sent Done false
30614 8963 Acked Write 6 Done Sent Done false
30615 8964 Acked Read 6 Done Sent Done false
30616 8965 Acked Write 7 Done Sent Done false
30617 8966 Acked Read 7 Done Sent Done false
30618 8967 Acked Read 10 Done Sent Done false
30619 8968 Acked Read 5 Done Sent Done false
30620 8969 Acked Read 6 Done Sent Done false
30621 8970 Acked Write 1 Done Sent Done false
30622 8971 Acked Read 7 Done Sent Done false
30623 8972 Acked Write 2 Done Sent Done false
30624 8973 Acked Read 1 Done Sent Done false
30625 8974 Acked Flush 0 Done Sent Done false
30626 8975 Acked Read 6 Done Sent Done false
30627 8976 Acked Write 1 Done Sent Done false
30628 8977 Acked Write 3 Done Sent Done false
30629 8978 Acked Read 10 Done Sent Done false
30630 8979 Acked Read 5 Done Sent Done false
30631 8980 Acked Write 2 Done Sent Done false
30632 8981 Acked Read 2 Done Sent Done false
30633 8982 Acked Write 1 Done Sent Done false
30634 8983 Acked Flush 0 Done Sent Done false
30635 8984 Acked Read 9 Done Sent Done false
30636 8985 Acked Read 7 Done Sent Done false
30637 8986 Acked Write 8 Done Sent Done false
30638 8987 Acked Read 4 Done Sent Done false
30639 8988 Acked Write 10 Done Sent Done false
30640 8989 Acked Write 4 Done Sent Done false
30641 8990 Acked Read 2 Done Sent Done false
30642 8991 Acked Write 5 Done Sent Done false
30643 8992 Acked Read 2 Done Sent Done false
30644 8993 Acked Flush 0 Done Sent Done false
30645 8994 Acked Read 1 Done Sent Done false
30646 8995 Acked Write 6 Done Sent Done false
30647 8996 Acked Read 9 Done Sent Done false
30648 8997 Acked Read 3 Done Sent Done false
30649 8998 Acked Flush 0 Done Sent Done false
30650 8999 Acked Flush 0 Done Sent Done false
30651 9000 Acked Read 7 Done Sent Done false
30652 9001 Acked Read 9 Done Sent Done false
30653 9002 Acked Read 7 Done Sent Done false
30654 9003 Acked Flush 0 Done Sent Done false
30655 9004 Acked Write 7 Done Sent Done false
30656 9005 Acked Flush 0 Done Sent Done false
30657 9006 Acked Write 5 Done Sent Done false
30658 9007 Acked Read 1 Done Sent Done false
30659 9008 Acked Read 10 Done Sent Done false
30660 9009 Acked Read 4 Done Sent Done false
30661 9010 Acked Read 3 Done Sent Done false
30662 9011 Acked Read 6 Done Sent Done false
30663 9012 Acked Read 7 Done Sent Done false
30664 9013 Acked Read 5 Done Sent Done false
30665 9014 Acked Flush 0 Done Sent Done false
30666 9015 Acked Read 7 Done Sent Done false
30667 9016 Acked Read 6 Done Sent Done false
30668 9017 Acked Read 8 Done Sent Done false
30669 9018 Acked Read 5 Done Sent Done false
30670 9019 Acked Write 7 Done Sent Done false
30671 9020 Acked Read 4 Done Sent Done false
30672 9021 Acked Write 6 Done Sent Done false
30673 9022 Acked Write 4 Done Sent Done false
30674 9023 Acked Read 8 Done Sent Done false
30675 9024 Acked Write 4 Done Sent Done false
30676 9025 Acked Read 10 Done Sent Done false
30677 9026 Acked Read 5 Done Sent Done false
30678 9027 Acked Write 3 Done Sent Done false
30679 9028 Acked Read 2 Done Sent Done false
30680 9029 Acked Read 6 Done Sent Done false
30681 9030 Acked Write 2 Done Sent Done false
30682 9031 Acked Write 1 Done Sent Done false
30683 9032 Acked Write 4 Done Sent Done false
30684 9033 Acked Write 3 Done Sent Done false
30685 9034 Acked Write 2 Done Sent Done false
30686 9035 Acked Write 3 Done Sent Done false
30687 9036 Acked Read 10 Done Sent Done false
30688 9037 Acked Write 2 Done Sent Done false
30689 9038 Acked Read 8 Done Sent Done false
30690 9039 Acked Read 8 Done Sent Done false
30691 9040 Acked Read 10 Done Sent Done false
30692 9041 Acked Read 10 Done Sent Done false
30693 9042 Acked Write 5 Done Sent Done false
30694 9043 Acked Write 1 Done Sent Done false
30695 9044 Acked Write 7 Done Sent Done false
30696 9045 Acked Write 9 Done Sent Done false
30697 9046 Acked Read 4 Done Sent Done false
30698 9047 Acked Write 7 Done Sent Done false
30699 9048 Acked Write 9 Done Sent Done false
30700 9049 Acked Write 8 Done Sent Done false
30701 9050 Acked Read 5 Done Sent Done false
30702 9051 Acked Write 7 Done Sent Done false
30703 9052 Acked Read 3 Done Sent Done false
30704 9053 Acked Read 2 Done Sent Done false
30705 9054 Acked Read 9 Done Sent Done false
30706 9055 Acked Read 8 Done Sent Done false
30707 9056 Acked Read 2 Done Sent Done false
30708 9057 Acked Read 2 Done Sent Done false
30709 9058 Acked Read 8 Done Sent Done false
30710 9059 Acked Read 2 Done Sent Done false
30711 9060 Acked Read 10 Done Sent Done false
30712 9061 Acked Read 4 Done Sent Done false
30713 9062 Acked Write 7 Done Sent Done false
30714 9063 Acked Read 8 Done Sent Done false
30715 9064 Acked Write 6 Done Sent Done false
30716 9065 Acked Read 6 Done Sent Done false
30717 9066 Acked Write 2 Done Sent Done false
30718 9067 Acked Flush 0 Done Sent Done false
30719 9068 Acked Flush 0 Done Sent Done false
30720 9069 Acked Read 5 Done Sent Done false
30721 9070 Acked Read 7 Done Sent Done false
30722 9071 Acked Read 7 Done Sent Done false
30723 9072 Acked Read 9 Done Sent Done false
30724 9073 Acked Write 10 Done Sent Done false
30725 9074 Acked Write 2 Done Sent Done false
30726 9075 Acked Flush 0 Done Sent Done false
30727 9076 Acked Write 4 Done Sent Done false
30728 9077 Acked Write 2 Done Sent Done false
30729 9078 Acked Flush 0 Done Sent Done false
30730 9079 Acked Read 1 Done Sent Done false
30731 9080 Acked Read 6 Done Sent Done false
30732 9081 Acked Write 3 Done Sent Done false
30733 9082 Acked Write 5 Done Sent Done false
30734 9083 Acked Read 4 Done Sent Done false
30735 9084 Acked Read 1 Done Sent Done false
30736 9085 Acked Read 10 Done Sent Done false
30737 9086 Acked Read 6 Done Sent Done false
30738 9087 Acked Read 5 Done Sent Done false
30739 9088 Acked Write 9 Done Sent Done false
30740 9089 Acked Read 5 Done Sent Done false
30741 9090 Acked Write 5 Done Sent Done false
30742 9091 Acked Read 1 Done Sent Done false
30743 9092 Acked Write 2 Done Sent Done false
30744 9093 Acked Write 7 Done Sent Done false
30745 9094 Acked Read 6 Done Sent Done false
30746 9095 Acked Flush 0 Done Sent Done false
30747 9096 Acked Read 9 Done Sent Done false
30748 9097 Acked Read 9 Done Sent Done false
30749 9098 Acked Write 6 Done Sent Done false
30750 9099 Acked Read 4 Done Sent Done false
30751 9100 Acked Read 2 Done Sent Done false
30752 9101 Acked Read 10 Done Sent Done false
30753 9102 Acked Write 4 Done Sent Done false
30754 9103 Acked Write 5 Done Sent Done false
30755 9104 Acked Read 2 Done Sent Done false
30756 9105 Acked Write 10 Done Sent Done false
30757 9106 Acked Read 8 Done Sent Done false
30758 9107 Acked Read 5 Done Sent Done false
30759 9108 Acked Flush 0 Done Sent Done false
30760 9109 Acked Flush 0 Done Sent Done false
30761 9110 Acked Write 1 Done Sent Done false
30762 9111 Acked Write 3 Done Sent Done false
30763 9112 Acked Write 10 Done Sent Done false
30764 9113 Acked Flush 0 Done Sent Done false
30765 9114 Acked Write 1 Done Sent Done false
30766 9115 Acked Read 9 Done Sent Done false
30767 9116 Acked Read 2 Done Sent Done false
30768 9117 Acked Read 1 Done Sent Done false
30769 9118 Acked Read 4 Done Sent Done false
30770 9119 Acked Write 3 Done Sent Done false
30771 9120 Acked Read 7 Done Sent Done false
30772 9121 Acked Read 2 Done Sent Done false
30773 9122 Acked Read 10 Done Sent Done false
30774 9123 Acked Read 10 Done Sent Done false
30775 9124 Acked Flush 0 Done Sent Done false
30776 9125 Acked Read 6 Done Sent Done false
30777 9126 Acked Write 4 Done Sent Done false
30778 9127 Acked Read 2 Done Sent Done false
30779 9128 Acked Read 2 Done Sent Done false
30780 9129 Acked Write 5 Done Sent Done false
30781 9130 Acked Write 9 Done Sent Done false
30782 9131 Acked Read 2 Done Sent Done false
30783 9132 Acked Write 6 Done Sent Done false
30784 9133 Acked Read 9 Done Sent Done false
30785 9134 Acked Write 6 Done Sent Done false
30786 9135 Acked Read 8 Done Sent Done false
30787 9136 Acked Flush 0 Done Sent Done false
30788 9137 Acked Read 6 Done Sent Done false
30789 9138 Acked Write 4 Done Sent Done false
30790 9139 Acked Write 8 Done Sent Done false
30791 9140 Acked Read 7 Done Sent Done false
30792 9141 Acked Flush 0 Done Sent Done false
30793 9142 Acked Read 7 Done Sent Done false
30794 9143 Acked Read 6 Done Sent Done false
30795 9144 Acked Flush 0 Done Sent Done false
30796 9145 Acked Write 3 Done Sent Done false
30797 9146 Acked Write 5 Done Sent Done false
30798 9147 Acked Write 7 Done Sent Done false
30799 9148 Acked Write 4 Done Sent Done false
30800 9149 Acked Read 2 Done Sent Done false
30801 9150 Acked Write 1 Done Sent Done false
30802 9151 Acked Read 10 Done Sent Done false
30803 STATES DS:0 DS:1 DS:2 TOTAL
30804 Sent 0 301 0 301
30805 Done 301 0 301 602
30806 Skipped 0 0 0 0
30807 Error 0 0 0 0
30808 Last Flush: Some(JobId(9144)) Some(JobId(8850)) Some(JobId(9144))
30809 Downstairs last five completed: 8850 8849 8848 8847 8846
30810 Upstairs last five completed: 9151 9150 9149 9148 9147
30811 CLIENT: Up:0 ds:301 act:2
308122025-01-29T05:17:30.769ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
308132025-01-29T05:17:32.016ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
308142025-01-29T05:17:32.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
308152025-01-29T05:17:33.602ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
30816 ----------------------------------------------------------------
30817 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30818 States: Active Offline Active
30819 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30820 8851 Acked Read 8 Done Sent Done false
30821 8852 Acked Write 6 Done Sent Done false
30822 8853 Acked Read 2 Done Sent Done false
30823 8854 Acked Write 10 Done Sent Done false
30824 8855 Acked Write 2 Done Sent Done false
30825 8856 Acked Read 10 Done Sent Done false
30826 8857 Acked Read 3 Done Sent Done false
30827 8858 Acked Flush 0 Done Sent Done false
30828 8859 Acked Write 2 Done Sent Done false
30829 8860 Acked Read 8 Done Sent Done false
30830 8861 Acked Read 9 Done Sent Done false
30831 8862 Acked Flush 0 Done Sent Done false
30832 8863 Acked Write 6 Done Sent Done false
30833 8864 Acked Read 7 Done Sent Done false
30834 8865 Acked Flush 0 Done Sent Done false
30835 8866 Acked Read 10 Done Sent Done false
30836 8867 Acked Write 1 Done Sent Done false
30837 8868 Acked Write 8 Done Sent Done false
30838 8869 Acked Read 9 Done Sent Done false
30839 8870 Acked Read 2 Done Sent Done false
30840 8871 Acked Write 6 Done Sent Done false
30841 8872 Acked Read 1 Done Sent Done false
30842 8873 Acked Write 9 Done Sent Done false
30843 8874 Acked Write 10 Done Sent Done false
30844 8875 Acked Read 5 Done Sent Done false
30845 8876 Acked Flush 0 Done Sent Done false
30846 8877 Acked Write 3 Done Sent Done false
30847 8878 Acked Read 3 Done Sent Done false
30848 8879 Acked Write 5 Done Sent Done false
30849 8880 Acked Read 9 Done Sent Done false
30850 8881 Acked Read 1 Done Sent Done false
30851 8882 Acked Read 10 Done Sent Done false
30852 8883 Acked Write 9 Done Sent Done false
30853 8884 Acked Write 4 Done Sent Done false
30854 8885 Acked Read 5 Done Sent Done false
30855 8886 Acked Read 8 Done Sent Done false
30856 8887 Acked Read 9 Done Sent Done false
30857 8888 Acked Flush 0 Done Sent Done false
30858 8889 Acked Write 6 Done Sent Done false
30859 8890 Acked Write 6 Done Sent Done false
30860 8891 Acked Write 4 Done Sent Done false
30861 8892 Acked Write 10 Done Sent Done false
30862 8893 Acked Write 4 Done Sent Done false
30863 8894 Acked Write 6 Done Sent Done false
30864 8895 Acked Read 5 Done Sent Done false
30865 8896 Acked Write 9 Done Sent Done false
30866 8897 Acked Write 7 Done Sent Done false
30867 8898 Acked Read 5 Done Sent Done false
30868 8899 Acked Write 9 Done Sent Done false
30869 8900 Acked Read 5 Done Sent Done false
30870 8901 Acked Write 6 Done Sent Done false
30871 8902 Acked Read 8 Done Sent Done false
30872 8903 Acked Read 1 Done Sent Done false
30873 8904 Acked Read 8 Done Sent Done false
30874 8905 Acked Write 9 Done Sent Done false
30875 8906 Acked Write 10 Done Sent Done false
30876 8907 Acked Read 8 Done Sent Done false
30877 8908 Acked Read 4 Done Sent Done false
30878 8909 Acked Read 5 Done Sent Done false
30879 8910 Acked Write 1 Done Sent Done false
30880 8911 Acked Read 10 Done Sent Done false
30881 8912 Acked Write 1 Done Sent Done false
30882 8913 Acked Write 2 Done Sent Done false
30883 8914 Acked Write 10 Done Sent Done false
30884 8915 Acked Read 3 Done Sent Done false
30885 8916 Acked Write 7 Done Sent Done false
30886 8917 Acked Write 9 Done Sent Done false
30887 8918 Acked Read 4 Done Sent Done false
30888 8919 Acked Read 9 Done Sent Done false
30889 8920 Acked Read 4 Done Sent Done false
30890 8921 Acked Read 3 Done Sent Done false
30891 8922 Acked Read 10 Done Sent Done false
30892 8923 Acked Write 9 Done Sent Done false
30893 8924 Acked Read 4 Done Sent Done false
30894 8925 Acked Read 4 Done Sent Done false
30895 8926 Acked Write 10 Done Sent Done false
30896 8927 Acked Read 9 Done Sent Done false
30897 8928 Acked Write 5 Done Sent Done false
30898 8929 Acked Read 7 Done Sent Done false
30899 8930 Acked Write 7 Done Sent Done false
30900 8931 Acked Read 7 Done Sent Done false
30901 8932 Acked Read 3 Done Sent Done false
30902 8933 Acked Flush 0 Done Sent Done false
30903 8934 Acked Read 7 Done Sent Done false
30904 8935 Acked Read 10 Done Sent Done false
30905 8936 Acked Write 2 Done Sent Done false
30906 8937 Acked Read 7 Done Sent Done false
30907 8938 Acked Write 8 Done Sent Done false
30908 8939 Acked Read 8 Done Sent Done false
30909 8940 Acked Write 1 Done Sent Done false
30910 8941 Acked Write 7 Done Sent Done false
30911 8942 Acked Write 4 Done Sent Done false
30912 8943 Acked Write 5 Done Sent Done false
30913 8944 Acked Read 5 Done Sent Done false
30914 8945 Acked Write 2 Done Sent Done false
30915 8946 Acked Write 3 Done Sent Done false
30916 8947 Acked Read 7 Done Sent Done false
30917 8948 Acked Read 9 Done Sent Done false
30918 8949 Acked Write 4 Done Sent Done false
30919 8950 Acked Write 3 Done Sent Done false
30920 8951 Acked Read 9 Done Sent Done false
30921 8952 Acked Read 3 Done Sent Done false
30922 8953 Acked Read 3 Done Sent Done false
30923 8954 Acked Write 7 Done Sent Done false
30924 8955 Acked Read 2 Done Sent Done false
30925 8956 Acked Write 6 Done Sent Done false
30926 8957 Acked Write 5 Done Sent Done false
30927 8958 Acked Read 8 Done Sent Done false
30928 8959 Acked Read 5 Done Sent Done false
30929 8960 Acked Read 8 Done Sent Done false
30930 8961 Acked Flush 0 Done Sent Done false
30931 8962 Acked Flush 0 Done Sent Done false
30932 8963 Acked Write 6 Done Sent Done false
30933 8964 Acked Read 6 Done Sent Done false
30934 8965 Acked Write 7 Done Sent Done false
30935 8966 Acked Read 7 Done Sent Done false
30936 8967 Acked Read 10 Done Sent Done false
30937 8968 Acked Read 5 Done Sent Done false
30938 8969 Acked Read 6 Done Sent Done false
30939 8970 Acked Write 1 Done Sent Done false
30940 8971 Acked Read 7 Done Sent Done false
30941 8972 Acked Write 2 Done Sent Done false
30942 8973 Acked Read 1 Done Sent Done false
30943 8974 Acked Flush 0 Done Sent Done false
30944 8975 Acked Read 6 Done Sent Done false
30945 8976 Acked Write 1 Done Sent Done false
30946 8977 Acked Write 3 Done Sent Done false
30947 8978 Acked Read 10 Done Sent Done false
30948 8979 Acked Read 5 Done Sent Done false
30949 8980 Acked Write 2 Done Sent Done false
30950 8981 Acked Read 2 Done Sent Done false
30951 8982 Acked Write 1 Done Sent Done false
30952 8983 Acked Flush 0 Done Sent Done false
30953 8984 Acked Read 9 Done Sent Done false
30954 8985 Acked Read 7 Done Sent Done false
30955 8986 Acked Write 8 Done Sent Done false
30956 8987 Acked Read 4 Done Sent Done false
30957 8988 Acked Write 10 Done Sent Done false
30958 8989 Acked Write 4 Done Sent Done false
30959 8990 Acked Read 2 Done Sent Done false
30960 8991 Acked Write 5 Done Sent Done false
30961 8992 Acked Read 2 Done Sent Done false
30962 8993 Acked Flush 0 Done Sent Done false
30963 8994 Acked Read 1 Done Sent Done false
30964 8995 Acked Write 6 Done Sent Done false
30965 8996 Acked Read 9 Done Sent Done false
30966 8997 Acked Read 3 Done Sent Done false
30967 8998 Acked Flush 0 Done Sent Done false
30968 8999 Acked Flush 0 Done Sent Done false
30969 9000 Acked Read 7 Done Sent Done false
30970 9001 Acked Read 9 Done Sent Done false
30971 9002 Acked Read 7 Done Sent Done false
30972 9003 Acked Flush 0 Done Sent Done false
30973 9004 Acked Write 7 Done Sent Done false
30974 9005 Acked Flush 0 Done Sent Done false
30975 9006 Acked Write 5 Done Sent Done false
30976 9007 Acked Read 1 Done Sent Done false
30977 9008 Acked Read 10 Done Sent Done false
30978 9009 Acked Read 4 Done Sent Done false
30979 9010 Acked Read 3 Done Sent Done false
30980 9011 Acked Read 6 Done Sent Done false
30981 9012 Acked Read 7 Done Sent Done false
30982 9013 Acked Read 5 Done Sent Done false
30983 9014 Acked Flush 0 Done Sent Done false
30984 9015 Acked Read 7 Done Sent Done false
30985 9016 Acked Read 6 Done Sent Done false
30986 9017 Acked Read 8 Done Sent Done false
30987 9018 Acked Read 5 Done Sent Done false
30988 9019 Acked Write 7 Done Sent Done false
30989 9020 Acked Read 4 Done Sent Done false
30990 9021 Acked Write 6 Done Sent Done false
30991 9022 Acked Write 4 Done Sent Done false
30992 9023 Acked Read 8 Done Sent Done false
30993 9024 Acked Write 4 Done Sent Done false
30994 9025 Acked Read 10 Done Sent Done false
30995 9026 Acked Read 5 Done Sent Done false
30996 9027 Acked Write 3 Done Sent Done false
30997 9028 Acked Read 2 Done Sent Done false
30998 9029 Acked Read 6 Done Sent Done false
30999 9030 Acked Write 2 Done Sent Done false
31000 9031 Acked Write 1 Done Sent Done false
31001 9032 Acked Write 4 Done Sent Done false
31002 9033 Acked Write 3 Done Sent Done false
31003 9034 Acked Write 2 Done Sent Done false
31004 9035 Acked Write 3 Done Sent Done false
31005 9036 Acked Read 10 Done Sent Done false
31006 9037 Acked Write 2 Done Sent Done false
31007 9038 Acked Read 8 Done Sent Done false
31008 9039 Acked Read 8 Done Sent Done false
31009 9040 Acked Read 10 Done Sent Done false
31010 9041 Acked Read 10 Done Sent Done false
31011 9042 Acked Write 5 Done Sent Done false
31012 9043 Acked Write 1 Done Sent Done false
31013 9044 Acked Write 7 Done Sent Done false
31014 9045 Acked Write 9 Done Sent Done false
31015 9046 Acked Read 4 Done Sent Done false
31016 9047 Acked Write 7 Done Sent Done false
31017 9048 Acked Write 9 Done Sent Done false
31018 9049 Acked Write 8 Done Sent Done false
31019 9050 Acked Read 5 Done Sent Done false
31020 9051 Acked Write 7 Done Sent Done false
31021 9052 Acked Read 3 Done Sent Done false
31022 9053 Acked Read 2 Done Sent Done false
31023 9054 Acked Read 9 Done Sent Done false
31024 9055 Acked Read 8 Done Sent Done false
31025 9056 Acked Read 2 Done Sent Done false
31026 9057 Acked Read 2 Done Sent Done false
31027 9058 Acked Read 8 Done Sent Done false
31028 9059 Acked Read 2 Done Sent Done false
31029 9060 Acked Read 10 Done Sent Done false
31030 9061 Acked Read 4 Done Sent Done false
31031 9062 Acked Write 7 Done Sent Done false
31032 9063 Acked Read 8 Done Sent Done false
31033 9064 Acked Write 6 Done Sent Done false
31034 9065 Acked Read 6 Done Sent Done false
31035 9066 Acked Write 2 Done Sent Done false
31036 9067 Acked Flush 0 Done Sent Done false
31037 9068 Acked Flush 0 Done Sent Done false
31038 9069 Acked Read 5 Done Sent Done false
31039 9070 Acked Read 7 Done Sent Done false
31040 9071 Acked Read 7 Done Sent Done false
31041 9072 Acked Read 9 Done Sent Done false
31042 9073 Acked Write 10 Done Sent Done false
31043 9074 Acked Write 2 Done Sent Done false
31044 9075 Acked Flush 0 Done Sent Done false
31045 9076 Acked Write 4 Done Sent Done false
31046 9077 Acked Write 2 Done Sent Done false
31047 9078 Acked Flush 0 Done Sent Done false
31048 9079 Acked Read 1 Done Sent Done false
31049 9080 Acked Read 6 Done Sent Done false
31050 9081 Acked Write 3 Done Sent Done false
31051 9082 Acked Write 5 Done Sent Done false
31052 9083 Acked Read 4 Done Sent Done false
31053 9084 Acked Read 1 Done Sent Done false
31054 9085 Acked Read 10 Done Sent Done false
31055 9086 Acked Read 6 Done Sent Done false
31056 9087 Acked Read 5 Done Sent Done false
31057 9088 Acked Write 9 Done Sent Done false
31058 9089 Acked Read 5 Done Sent Done false
31059 9090 Acked Write 5 Done Sent Done false
31060 9091 Acked Read 1 Done Sent Done false
31061 9092 Acked Write 2 Done Sent Done false
31062 9093 Acked Write 7 Done Sent Done false
31063 9094 Acked Read 6 Done Sent Done false
31064 9095 Acked Flush 0 Done Sent Done false
31065 9096 Acked Read 9 Done Sent Done false
31066 9097 Acked Read 9 Done Sent Done false
31067 9098 Acked Write 6 Done Sent Done false
31068 9099 Acked Read 4 Done Sent Done false
31069 9100 Acked Read 2 Done Sent Done false
31070 9101 Acked Read 10 Done Sent Done false
31071 9102 Acked Write 4 Done Sent Done false
31072 9103 Acked Write 5 Done Sent Done false
31073 9104 Acked Read 2 Done Sent Done false
31074 9105 Acked Write 10 Done Sent Done false
31075 9106 Acked Read 8 Done Sent Done false
31076 9107 Acked Read 5 Done Sent Done false
31077 9108 Acked Flush 0 Done Sent Done false
31078 9109 Acked Flush 0 Done Sent Done false
31079 9110 Acked Write 1 Done Sent Done false
31080 9111 Acked Write 3 Done Sent Done false
31081 9112 Acked Write 10 Done Sent Done false
31082 9113 Acked Flush 0 Done Sent Done false
31083 9114 Acked Write 1 Done Sent Done false
31084 9115 Acked Read 9 Done Sent Done false
31085 9116 Acked Read 2 Done Sent Done false
31086 9117 Acked Read 1 Done Sent Done false
31087 9118 Acked Read 4 Done Sent Done false
31088 9119 Acked Write 3 Done Sent Done false
31089 9120 Acked Read 7 Done Sent Done false
31090 9121 Acked Read 2 Done Sent Done false
31091 9122 Acked Read 10 Done Sent Done false
31092 9123 Acked Read 10 Done Sent Done false
31093 9124 Acked Flush 0 Done Sent Done false
31094 9125 Acked Read 6 Done Sent Done false
31095 9126 Acked Write 4 Done Sent Done false
31096 9127 Acked Read 2 Done Sent Done false
31097 9128 Acked Read 2 Done Sent Done false
31098 9129 Acked Write 5 Done Sent Done false
31099 9130 Acked Write 9 Done Sent Done false
31100 9131 Acked Read 2 Done Sent Done false
31101 9132 Acked Write 6 Done Sent Done false
31102 9133 Acked Read 9 Done Sent Done false
31103 9134 Acked Write 6 Done Sent Done false
31104 9135 Acked Read 8 Done Sent Done false
31105 9136 Acked Flush 0 Done Sent Done false
31106 9137 Acked Read 6 Done Sent Done false
31107 9138 Acked Write 4 Done Sent Done false
31108 9139 Acked Write 8 Done Sent Done false
31109 9140 Acked Read 7 Done Sent Done false
31110 9141 Acked Flush 0 Done Sent Done false
31111 9142 Acked Read 7 Done Sent Done false
31112 9143 Acked Read 6 Done Sent Done false
31113 9144 Acked Flush 0 Done Sent Done false
31114 9145 Acked Write 3 Done Sent Done false
31115 9146 Acked Write 5 Done Sent Done false
31116 9147 Acked Write 7 Done Sent Done false
31117 9148 Acked Write 4 Done Sent Done false
31118 9149 Acked Read 2 Done Sent Done false
31119 9150 Acked Write 1 Done Sent Done false
31120 9151 Acked Read 10 Done Sent Done false
31121 STATES DS:0 DS:1 DS:2 TOTAL
31122 Sent 0 301 0 301
31123 Done 301 0 301 602
31124 Skipped 0 0 0 0
31125 Error 0 0 0 0
31126 Last Flush: Some(JobId(9144)) Some(JobId(8850)) Some(JobId(9144))
31127 Downstairs last five completed: 8850 8849 8848 8847 8846
31128 Upstairs last five completed: 9151 9150 9149 9148 9147
31129 CLIENT: Up:0 ds:301 act:2
311302025-01-29T05:17:34.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311312025-01-29T05:17:35.604ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311322025-01-29T05:17:36.017ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311332025-01-29T05:17:36.017ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311342025-01-29T05:17:36.017ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311352025-01-29T05:17:36.017ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311362025-01-29T05:17:36.017ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31137 [8820] got start action
31138 [8820] I am going to restart
31139 Starting downstairs at port 8820
31140 Make output file at "/tmp/dsc/downstairs-8820.txt"
31141 [8820][1] reports Starting
31142 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
31143 [8820][1] reports Running
311442025-01-29T05:17:37.018ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
311452025-01-29T05:17:37.609ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31146 ----------------------------------------------------------------
31147 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31148 States: Active Offline Active
31149 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31150 8851 Acked Read 8 Done Sent Done false
31151 8852 Acked Write 6 Done Sent Done false
31152 8853 Acked Read 2 Done Sent Done false
31153 8854 Acked Write 10 Done Sent Done false
31154 8855 Acked Write 2 Done Sent Done false
31155 8856 Acked Read 10 Done Sent Done false
31156 8857 Acked Read 3 Done Sent Done false
31157 8858 Acked Flush 0 Done Sent Done false
31158 8859 Acked Write 2 Done Sent Done false
31159 8860 Acked Read 8 Done Sent Done false
31160 8861 Acked Read 9 Done Sent Done false
31161 8862 Acked Flush 0 Done Sent Done false
31162 8863 Acked Write 6 Done Sent Done false
31163 8864 Acked Read 7 Done Sent Done false
31164 8865 Acked Flush 0 Done Sent Done false
31165 8866 Acked Read 10 Done Sent Done false
31166 8867 Acked Write 1 Done Sent Done false
31167 8868 Acked Write 8 Done Sent Done false
31168 8869 Acked Read 9 Done Sent Done false
31169 8870 Acked Read 2 Done Sent Done false
31170 8871 Acked Write 6 Done Sent Done false
31171 8872 Acked Read 1 Done Sent Done false
31172 8873 Acked Write 9 Done Sent Done false
31173 8874 Acked Write 10 Done Sent Done false
31174 8875 Acked Read 5 Done Sent Done false
31175 8876 Acked Flush 0 Done Sent Done false
31176 8877 Acked Write 3 Done Sent Done false
31177 8878 Acked Read 3 Done Sent Done false
31178 8879 Acked Write 5 Done Sent Done false
31179 8880 Acked Read 9 Done Sent Done false
31180 8881 Acked Read 1 Done Sent Done false
31181 8882 Acked Read 10 Done Sent Done false
31182 8883 Acked Write 9 Done Sent Done false
31183 8884 Acked Write 4 Done Sent Done false
31184 8885 Acked Read 5 Done Sent Done false
31185 8886 Acked Read 8 Done Sent Done false
31186 8887 Acked Read 9 Done Sent Done false
31187 8888 Acked Flush 0 Done Sent Done false
31188 8889 Acked Write 6 Done Sent Done false
31189 8890 Acked Write 6 Done Sent Done false
31190 8891 Acked Write 4 Done Sent Done false
31191 8892 Acked Write 10 Done Sent Done false
31192 8893 Acked Write 4 Done Sent Done false
31193 8894 Acked Write 6 Done Sent Done false
31194 8895 Acked Read 5 Done Sent Done false
31195 8896 Acked Write 9 Done Sent Done false
31196 8897 Acked Write 7 Done Sent Done false
31197 8898 Acked Read 5 Done Sent Done false
31198 8899 Acked Write 9 Done Sent Done false
31199 8900 Acked Read 5 Done Sent Done false
31200 8901 Acked Write 6 Done Sent Done false
31201 8902 Acked Read 8 Done Sent Done false
31202 8903 Acked Read 1 Done Sent Done false
31203 8904 Acked Read 8 Done Sent Done false
31204 8905 Acked Write 9 Done Sent Done false
31205 8906 Acked Write 10 Done Sent Done false
31206 8907 Acked Read 8 Done Sent Done false
31207 8908 Acked Read 4 Done Sent Done false
31208 8909 Acked Read 5 Done Sent Done false
31209 8910 Acked Write 1 Done Sent Done false
31210 8911 Acked Read 10 Done Sent Done false
31211 8912 Acked Write 1 Done Sent Done false
31212 8913 Acked Write 2 Done Sent Done false
31213 8914 Acked Write 10 Done Sent Done false
31214 8915 Acked Read 3 Done Sent Done false
31215 8916 Acked Write 7 Done Sent Done false
31216 8917 Acked Write 9 Done Sent Done false
31217 8918 Acked Read 4 Done Sent Done false
31218 8919 Acked Read 9 Done Sent Done false
31219 8920 Acked Read 4 Done Sent Done false
31220 8921 Acked Read 3 Done Sent Done false
31221 8922 Acked Read 10 Done Sent Done false
31222 8923 Acked Write 9 Done Sent Done false
31223 8924 Acked Read 4 Done Sent Done false
31224 8925 Acked Read 4 Done Sent Done false
31225 8926 Acked Write 10 Done Sent Done false
31226 8927 Acked Read 9 Done Sent Done false
31227 8928 Acked Write 5 Done Sent Done false
31228 8929 Acked Read 7 Done Sent Done false
31229 8930 Acked Write 7 Done Sent Done false
31230 8931 Acked Read 7 Done Sent Done false
31231 8932 Acked Read 3 Done Sent Done false
31232 8933 Acked Flush 0 Done Sent Done false
31233 8934 Acked Read 7 Done Sent Done false
31234 8935 Acked Read 10 Done Sent Done false
31235 8936 Acked Write 2 Done Sent Done false
31236 8937 Acked Read 7 Done Sent Done false
31237 8938 Acked Write 8 Done Sent Done false
31238 8939 Acked Read 8 Done Sent Done false
31239 8940 Acked Write 1 Done Sent Done false
31240 8941 Acked Write 7 Done Sent Done false
31241 8942 Acked Write 4 Done Sent Done false
31242 8943 Acked Write 5 Done Sent Done false
31243 8944 Acked Read 5 Done Sent Done false
31244 8945 Acked Write 2 Done Sent Done false
31245 8946 Acked Write 3 Done Sent Done false
31246 8947 Acked Read 7 Done Sent Done false
31247 8948 Acked Read 9 Done Sent Done false
31248 8949 Acked Write 4 Done Sent Done false
31249 8950 Acked Write 3 Done Sent Done false
31250 8951 Acked Read 9 Done Sent Done false
31251 8952 Acked Read 3 Done Sent Done false
31252 8953 Acked Read 3 Done Sent Done false
31253 8954 Acked Write 7 Done Sent Done false
31254 8955 Acked Read 2 Done Sent Done false
31255 8956 Acked Write 6 Done Sent Done false
31256 8957 Acked Write 5 Done Sent Done false
31257 8958 Acked Read 8 Done Sent Done false
31258 8959 Acked Read 5 Done Sent Done false
31259 8960 Acked Read 8 Done Sent Done false
31260 8961 Acked Flush 0 Done Sent Done false
31261 8962 Acked Flush 0 Done Sent Done false
31262 8963 Acked Write 6 Done Sent Done false
31263 8964 Acked Read 6 Done Sent Done false
31264 8965 Acked Write 7 Done Sent Done false
31265 8966 Acked Read 7 Done Sent Done false
31266 8967 Acked Read 10 Done Sent Done false
31267 8968 Acked Read 5 Done Sent Done false
31268 8969 Acked Read 6 Done Sent Done false
31269 8970 Acked Write 1 Done Sent Done false
31270 8971 Acked Read 7 Done Sent Done false
31271 8972 Acked Write 2 Done Sent Done false
31272 8973 Acked Read 1 Done Sent Done false
31273 8974 Acked Flush 0 Done Sent Done false
31274 8975 Acked Read 6 Done Sent Done false
31275 8976 Acked Write 1 Done Sent Done false
31276 8977 Acked Write 3 Done Sent Done false
31277 8978 Acked Read 10 Done Sent Done false
31278 8979 Acked Read 5 Done Sent Done false
31279 8980 Acked Write 2 Done Sent Done false
31280 8981 Acked Read 2 Done Sent Done false
31281 8982 Acked Write 1 Done Sent Done false
31282 8983 Acked Flush 0 Done Sent Done false
31283 8984 Acked Read 9 Done Sent Done false
31284 8985 Acked Read 7 Done Sent Done false
31285 8986 Acked Write 8 Done Sent Done false
31286 8987 Acked Read 4 Done Sent Done false
31287 8988 Acked Write 10 Done Sent Done false
31288 8989 Acked Write 4 Done Sent Done false
31289 8990 Acked Read 2 Done Sent Done false
31290 8991 Acked Write 5 Done Sent Done false
31291 8992 Acked Read 2 Done Sent Done false
31292 8993 Acked Flush 0 Done Sent Done false
31293 8994 Acked Read 1 Done Sent Done false
31294 8995 Acked Write 6 Done Sent Done false
31295 8996 Acked Read 9 Done Sent Done false
31296 8997 Acked Read 3 Done Sent Done false
31297 8998 Acked Flush 0 Done Sent Done false
31298 8999 Acked Flush 0 Done Sent Done false
31299 9000 Acked Read 7 Done Sent Done false
31300 9001 Acked Read 9 Done Sent Done false
31301 9002 Acked Read 7 Done Sent Done false
31302 9003 Acked Flush 0 Done Sent Done false
31303 9004 Acked Write 7 Done Sent Done false
31304 9005 Acked Flush 0 Done Sent Done false
31305 9006 Acked Write 5 Done Sent Done false
31306 9007 Acked Read 1 Done Sent Done false
31307 9008 Acked Read 10 Done Sent Done false
31308 9009 Acked Read 4 Done Sent Done false
31309 9010 Acked Read 3 Done Sent Done false
31310 9011 Acked Read 6 Done Sent Done false
31311 9012 Acked Read 7 Done Sent Done false
31312 9013 Acked Read 5 Done Sent Done false
31313 9014 Acked Flush 0 Done Sent Done false
31314 9015 Acked Read 7 Done Sent Done false
31315 9016 Acked Read 6 Done Sent Done false
31316 9017 Acked Read 8 Done Sent Done false
31317 9018 Acked Read 5 Done Sent Done false
31318 9019 Acked Write 7 Done Sent Done false
31319 9020 Acked Read 4 Done Sent Done false
31320 9021 Acked Write 6 Done Sent Done false
31321 9022 Acked Write 4 Done Sent Done false
31322 9023 Acked Read 8 Done Sent Done false
31323 9024 Acked Write 4 Done Sent Done false
31324 9025 Acked Read 10 Done Sent Done false
31325 9026 Acked Read 5 Done Sent Done false
31326 9027 Acked Write 3 Done Sent Done false
31327 9028 Acked Read 2 Done Sent Done false
31328 9029 Acked Read 6 Done Sent Done false
31329 9030 Acked Write 2 Done Sent Done false
31330 9031 Acked Write 1 Done Sent Done false
31331 9032 Acked Write 4 Done Sent Done false
31332 9033 Acked Write 3 Done Sent Done false
31333 9034 Acked Write 2 Done Sent Done false
31334 9035 Acked Write 3 Done Sent Done false
31335 9036 Acked Read 10 Done Sent Done false
31336 9037 Acked Write 2 Done Sent Done false
31337 9038 Acked Read 8 Done Sent Done false
31338 9039 Acked Read 8 Done Sent Done false
31339 9040 Acked Read 10 Done Sent Done false
31340 9041 Acked Read 10 Done Sent Done false
31341 9042 Acked Write 5 Done Sent Done false
31342 9043 Acked Write 1 Done Sent Done false
31343 9044 Acked Write 7 Done Sent Done false
31344 9045 Acked Write 9 Done Sent Done false
31345 9046 Acked Read 4 Done Sent Done false
31346 9047 Acked Write 7 Done Sent Done false
31347 9048 Acked Write 9 Done Sent Done false
31348 9049 Acked Write 8 Done Sent Done false
31349 9050 Acked Read 5 Done Sent Done false
31350 9051 Acked Write 7 Done Sent Done false
31351 9052 Acked Read 3 Done Sent Done false
31352 9053 Acked Read 2 Done Sent Done false
31353 9054 Acked Read 9 Done Sent Done false
31354 9055 Acked Read 8 Done Sent Done false
31355 9056 Acked Read 2 Done Sent Done false
31356 9057 Acked Read 2 Done Sent Done false
31357 9058 Acked Read 8 Done Sent Done false
31358 9059 Acked Read 2 Done Sent Done false
31359 9060 Acked Read 10 Done Sent Done false
31360 9061 Acked Read 4 Done Sent Done false
31361 9062 Acked Write 7 Done Sent Done false
31362 9063 Acked Read 8 Done Sent Done false
31363 9064 Acked Write 6 Done Sent Done false
31364 9065 Acked Read 6 Done Sent Done false
31365 9066 Acked Write 2 Done Sent Done false
31366 9067 Acked Flush 0 Done Sent Done false
31367 9068 Acked Flush 0 Done Sent Done false
31368 9069 Acked Read 5 Done Sent Done false
31369 9070 Acked Read 7 Done Sent Done false
31370 9071 Acked Read 7 Done Sent Done false
31371 9072 Acked Read 9 Done Sent Done false
31372 9073 Acked Write 10 Done Sent Done false
31373 9074 Acked Write 2 Done Sent Done false
31374 9075 Acked Flush 0 Done Sent Done false
31375 9076 Acked Write 4 Done Sent Done false
31376 9077 Acked Write 2 Done Sent Done false
31377 9078 Acked Flush 0 Done Sent Done false
31378 9079 Acked Read 1 Done Sent Done false
31379 9080 Acked Read 6 Done Sent Done false
31380 9081 Acked Write 3 Done Sent Done false
31381 9082 Acked Write 5 Done Sent Done false
31382 9083 Acked Read 4 Done Sent Done false
31383 9084 Acked Read 1 Done Sent Done false
31384 9085 Acked Read 10 Done Sent Done false
31385 9086 Acked Read 6 Done Sent Done false
31386 9087 Acked Read 5 Done Sent Done false
31387 9088 Acked Write 9 Done Sent Done false
31388 9089 Acked Read 5 Done Sent Done false
31389 9090 Acked Write 5 Done Sent Done false
31390 9091 Acked Read 1 Done Sent Done false
31391 9092 Acked Write 2 Done Sent Done false
31392 9093 Acked Write 7 Done Sent Done false
31393 9094 Acked Read 6 Done Sent Done false
31394 9095 Acked Flush 0 Done Sent Done false
31395 9096 Acked Read 9 Done Sent Done false
31396 9097 Acked Read 9 Done Sent Done false
31397 9098 Acked Write 6 Done Sent Done false
31398 9099 Acked Read 4 Done Sent Done false
31399 9100 Acked Read 2 Done Sent Done false
31400 9101 Acked Read 10 Done Sent Done false
31401 9102 Acked Write 4 Done Sent Done false
31402 9103 Acked Write 5 Done Sent Done false
31403 9104 Acked Read 2 Done Sent Done false
31404 9105 Acked Write 10 Done Sent Done false
31405 9106 Acked Read 8 Done Sent Done false
31406 9107 Acked Read 5 Done Sent Done false
31407 9108 Acked Flush 0 Done Sent Done false
31408 9109 Acked Flush 0 Done Sent Done false
31409 9110 Acked Write 1 Done Sent Done false
31410 9111 Acked Write 3 Done Sent Done false
31411 9112 Acked Write 10 Done Sent Done false
31412 9113 Acked Flush 0 Done Sent Done false
31413 9114 Acked Write 1 Done Sent Done false
31414 9115 Acked Read 9 Done Sent Done false
31415 9116 Acked Read 2 Done Sent Done false
31416 9117 Acked Read 1 Done Sent Done false
31417 9118 Acked Read 4 Done Sent Done false
31418 9119 Acked Write 3 Done Sent Done false
31419 9120 Acked Read 7 Done Sent Done false
31420 9121 Acked Read 2 Done Sent Done false
31421 9122 Acked Read 10 Done Sent Done false
31422 9123 Acked Read 10 Done Sent Done false
31423 9124 Acked Flush 0 Done Sent Done false
31424 9125 Acked Read 6 Done Sent Done false
31425 9126 Acked Write 4 Done Sent Done false
31426 9127 Acked Read 2 Done Sent Done false
31427 9128 Acked Read 2 Done Sent Done false
31428 9129 Acked Write 5 Done Sent Done false
31429 9130 Acked Write 9 Done Sent Done false
31430 9131 Acked Read 2 Done Sent Done false
31431 9132 Acked Write 6 Done Sent Done false
31432 9133 Acked Read 9 Done Sent Done false
31433 9134 Acked Write 6 Done Sent Done false
31434 9135 Acked Read 8 Done Sent Done false
31435 9136 Acked Flush 0 Done Sent Done false
31436 9137 Acked Read 6 Done Sent Done false
31437 9138 Acked Write 4 Done Sent Done false
31438 9139 Acked Write 8 Done Sent Done false
31439 9140 Acked Read 7 Done Sent Done false
31440 9141 Acked Flush 0 Done Sent Done false
31441 9142 Acked Read 7 Done Sent Done false
31442 9143 Acked Read 6 Done Sent Done false
31443 9144 Acked Flush 0 Done Sent Done false
31444 9145 Acked Write 3 Done Sent Done false
31445 9146 Acked Write 5 Done Sent Done false
31446 9147 Acked Write 7 Done Sent Done false
31447 9148 Acked Write 4 Done Sent Done false
31448 9149 Acked Read 2 Done Sent Done false
31449 9150 Acked Write 1 Done Sent Done false
31450 9151 Acked Read 10 Done Sent Done false
31451 STATES DS:0 DS:1 DS:2 TOTAL
31452 Sent 0 301 0 301
31453 Done 301 0 301 602
31454 Skipped 0 0 0 0
31455 Error 0 0 0 0
31456 Last Flush: Some(JobId(9144)) Some(JobId(8850)) Some(JobId(9144))
31457 Downstairs last five completed: 8850 8849 8848 8847 8846
31458 Upstairs last five completed: 9151 9150 9149 9148 9147
31459 CLIENT: Up:0 ds:301 act:2
314602025-01-29T05:17:39.613ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
314612025-01-29T05:17:40.615ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
314622025-01-29T05:17:42.019ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31463 ----------------------------------------------------------------
31464 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31465 States: Active Offline Active
31466 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31467 8851 Acked Read 8 Done Sent Done false
31468 8852 Acked Write 6 Done Sent Done false
31469 8853 Acked Read 2 Done Sent Done false
31470 8854 Acked Write 10 Done Sent Done false
31471 8855 Acked Write 2 Done Sent Done false
31472 8856 Acked Read 10 Done Sent Done false
31473 8857 Acked Read 3 Done Sent Done false
31474 8858 Acked Flush 0 Done Sent Done false
31475 8859 Acked Write 2 Done Sent Done false
31476 8860 Acked Read 8 Done Sent Done false
31477 8861 Acked Read 9 Done Sent Done false
31478 8862 Acked Flush 0 Done Sent Done false
31479 8863 Acked Write 6 Done Sent Done false
31480 8864 Acked Read 7 Done Sent Done false
31481 8865 Acked Flush 0 Done Sent Done false
31482 8866 Acked Read 10 Done Sent Done false
31483 8867 Acked Write 1 Done Sent Done false
31484 8868 Acked Write 8 Done Sent Done false
31485 8869 Acked Read 9 Done Sent Done false
31486 8870 Acked Read 2 Done Sent Done false
31487 8871 Acked Write 6 Done Sent Done false
31488 8872 Acked Read 1 Done Sent Done false
31489 8873 Acked Write 9 Done Sent Done false
31490 8874 Acked Write 10 Done Sent Done false
31491 8875 Acked Read 5 Done Sent Done false
31492 8876 Acked Flush 0 Done Sent Done false
31493 8877 Acked Write 3 Done Sent Done false
31494 8878 Acked Read 3 Done Sent Done false
31495 8879 Acked Write 5 Done Sent Done false
31496 8880 Acked Read 9 Done Sent Done false
31497 8881 Acked Read 1 Done Sent Done false
31498 8882 Acked Read 10 Done Sent Done false
31499 8883 Acked Write 9 Done Sent Done false
31500 8884 Acked Write 4 Done Sent Done false
31501 8885 Acked Read 5 Done Sent Done false
31502 8886 Acked Read 8 Done Sent Done false
31503 8887 Acked Read 9 Done Sent Done false
31504 8888 Acked Flush 0 Done Sent Done false
31505 8889 Acked Write 6 Done Sent Done false
31506 8890 Acked Write 6 Done Sent Done false
31507 8891 Acked Write 4 Done Sent Done false
31508 8892 Acked Write 10 Done Sent Done false
31509 8893 Acked Write 4 Done Sent Done false
31510 8894 Acked Write 6 Done Sent Done false
31511 8895 Acked Read 5 Done Sent Done false
31512 8896 Acked Write 9 Done Sent Done false
31513 8897 Acked Write 7 Done Sent Done false
31514 8898 Acked Read 5 Done Sent Done false
31515 8899 Acked Write 9 Done Sent Done false
31516 8900 Acked Read 5 Done Sent Done false
31517 8901 Acked Write 6 Done Sent Done false
31518 8902 Acked Read 8 Done Sent Done false
31519 8903 Acked Read 1 Done Sent Done false
31520 8904 Acked Read 8 Done Sent Done false
31521 8905 Acked Write 9 Done Sent Done false
31522 8906 Acked Write 10 Done Sent Done false
31523 8907 Acked Read 8 Done Sent Done false
31524 8908 Acked Read 4 Done Sent Done false
31525 8909 Acked Read 5 Done Sent Done false
31526 8910 Acked Write 1 Done Sent Done false
31527 8911 Acked Read 10 Done Sent Done false
31528 8912 Acked Write 1 Done Sent Done false
31529 8913 Acked Write 2 Done Sent Done false
31530 8914 Acked Write 10 Done Sent Done false
31531 8915 Acked Read 3 Done Sent Done false
31532 8916 Acked Write 7 Done Sent Done false
31533 8917 Acked Write 9 Done Sent Done false
31534 8918 Acked Read 4 Done Sent Done false
31535 8919 Acked Read 9 Done Sent Done false
31536 8920 Acked Read 4 Done Sent Done false
31537 8921 Acked Read 3 Done Sent Done false
31538 8922 Acked Read 10 Done Sent Done false
31539 8923 Acked Write 9 Done Sent Done false
31540 8924 Acked Read 4 Done Sent Done false
31541 8925 Acked Read 4 Done Sent Done false
31542 8926 Acked Write 10 Done Sent Done false
31543 8927 Acked Read 9 Done Sent Done false
31544 8928 Acked Write 5 Done Sent Done false
31545 8929 Acked Read 7 Done Sent Done false
31546 8930 Acked Write 7 Done Sent Done false
31547 8931 Acked Read 7 Done Sent Done false
31548 8932 Acked Read 3 Done Sent Done false
31549 8933 Acked Flush 0 Done Sent Done false
31550 8934 Acked Read 7 Done Sent Done false
31551 8935 Acked Read 10 Done Sent Done false
31552 8936 Acked Write 2 Done Sent Done false
31553 8937 Acked Read 7 Done Sent Done false
31554 8938 Acked Write 8 Done Sent Done false
31555 8939 Acked Read 8 Done Sent Done false
31556 8940 Acked Write 1 Done Sent Done false
31557 8941 Acked Write 7 Done Sent Done false
31558 8942 Acked Write 4 Done Sent Done false
31559 8943 Acked Write 5 Done Sent Done false
31560 8944 Acked Read 5 Done Sent Done false
31561 8945 Acked Write 2 Done Sent Done false
31562 8946 Acked Write 3 Done Sent Done false
31563 8947 Acked Read 7 Done Sent Done false
31564 8948 Acked Read 9 Done Sent Done false
31565 8949 Acked Write 4 Done Sent Done false
31566 8950 Acked Write 3 Done Sent Done false
31567 8951 Acked Read 9 Done Sent Done false
31568 8952 Acked Read 3 Done Sent Done false
31569 8953 Acked Read 3 Done Sent Done false
31570 8954 Acked Write 7 Done Sent Done false
31571 8955 Acked Read 2 Done Sent Done false
31572 8956 Acked Write 6 Done Sent Done false
31573 8957 Acked Write 5 Done Sent Done false
31574 8958 Acked Read 8 Done Sent Done false
31575 8959 Acked Read 5 Done Sent Done false
31576 8960 Acked Read 8 Done Sent Done false
31577 8961 Acked Flush 0 Done Sent Done false
31578 8962 Acked Flush 0 Done Sent Done false
31579 8963 Acked Write 6 Done Sent Done false
31580 8964 Acked Read 6 Done Sent Done false
31581 8965 Acked Write 7 Done Sent Done false
31582 8966 Acked Read 7 Done Sent Done false
31583 8967 Acked Read 10 Done Sent Done false
31584 8968 Acked Read 5 Done Sent Done false
31585 8969 Acked Read 6 Done Sent Done false
31586 8970 Acked Write 1 Done Sent Done false
31587 8971 Acked Read 7 Done Sent Done false
31588 8972 Acked Write 2 Done Sent Done false
31589 8973 Acked Read 1 Done Sent Done false
31590 8974 Acked Flush 0 Done Sent Done false
31591 8975 Acked Read 6 Done Sent Done false
31592 8976 Acked Write 1 Done Sent Done false
31593 8977 Acked Write 3 Done Sent Done false
31594 8978 Acked Read 10 Done Sent Done false
31595 8979 Acked Read 5 Done Sent Done false
31596 8980 Acked Write 2 Done Sent Done false
31597 8981 Acked Read 2 Done Sent Done false
31598 8982 Acked Write 1 Done Sent Done false
31599 8983 Acked Flush 0 Done Sent Done false
31600 8984 Acked Read 9 Done Sent Done false
31601 8985 Acked Read 7 Done Sent Done false
31602 8986 Acked Write 8 Done Sent Done false
31603 8987 Acked Read 4 Done Sent Done false
31604 8988 Acked Write 10 Done Sent Done false
31605 8989 Acked Write 4 Done Sent Done false
31606 8990 Acked Read 2 Done Sent Done false
31607 8991 Acked Write 5 Done Sent Done false
31608 8992 Acked Read 2 Done Sent Done false
31609 8993 Acked Flush 0 Done Sent Done false
31610 8994 Acked Read 1 Done Sent Done false
31611 8995 Acked Write 6 Done Sent Done false
31612 8996 Acked Read 9 Done Sent Done false
31613 8997 Acked Read 3 Done Sent Done false
31614 8998 Acked Flush 0 Done Sent Done false
31615 8999 Acked Flush 0 Done Sent Done false
31616 9000 Acked Read 7 Done Sent Done false
31617 9001 Acked Read 9 Done Sent Done false
31618 9002 Acked Read 7 Done Sent Done false
31619 9003 Acked Flush 0 Done Sent Done false
31620 9004 Acked Write 7 Done Sent Done false
31621 9005 Acked Flush 0 Done Sent Done false
31622 9006 Acked Write 5 Done Sent Done false
31623 9007 Acked Read 1 Done Sent Done false
31624 9008 Acked Read 10 Done Sent Done false
31625 9009 Acked Read 4 Done Sent Done false
31626 9010 Acked Read 3 Done Sent Done false
31627 9011 Acked Read 6 Done Sent Done false
31628 9012 Acked Read 7 Done Sent Done false
31629 9013 Acked Read 5 Done Sent Done false
31630 9014 Acked Flush 0 Done Sent Done false
31631 9015 Acked Read 7 Done Sent Done false
31632 9016 Acked Read 6 Done Sent Done false
31633 9017 Acked Read 8 Done Sent Done false
31634 9018 Acked Read 5 Done Sent Done false
31635 9019 Acked Write 7 Done Sent Done false
31636 9020 Acked Read 4 Done Sent Done false
31637 9021 Acked Write 6 Done Sent Done false
31638 9022 Acked Write 4 Done Sent Done false
31639 9023 Acked Read 8 Done Sent Done false
31640 9024 Acked Write 4 Done Sent Done false
31641 9025 Acked Read 10 Done Sent Done false
31642 9026 Acked Read 5 Done Sent Done false
31643 9027 Acked Write 3 Done Sent Done false
31644 9028 Acked Read 2 Done Sent Done false
31645 9029 Acked Read 6 Done Sent Done false
31646 9030 Acked Write 2 Done Sent Done false
31647 9031 Acked Write 1 Done Sent Done false
31648 9032 Acked Write 4 Done Sent Done false
31649 9033 Acked Write 3 Done Sent Done false
31650 9034 Acked Write 2 Done Sent Done false
31651 9035 Acked Write 3 Done Sent Done false
31652 9036 Acked Read 10 Done Sent Done false
31653 9037 Acked Write 2 Done Sent Done false
31654 9038 Acked Read 8 Done Sent Done false
31655 9039 Acked Read 8 Done Sent Done false
31656 9040 Acked Read 10 Done Sent Done false
31657 9041 Acked Read 10 Done Sent Done false
31658 9042 Acked Write 5 Done Sent Done false
31659 9043 Acked Write 1 Done Sent Done false
31660 9044 Acked Write 7 Done Sent Done false
31661 9045 Acked Write 9 Done Sent Done false
31662 9046 Acked Read 4 Done Sent Done false
31663 9047 Acked Write 7 Done Sent Done false
31664 9048 Acked Write 9 Done Sent Done false
31665 9049 Acked Write 8 Done Sent Done false
31666 9050 Acked Read 5 Done Sent Done false
31667 9051 Acked Write 7 Done Sent Done false
31668 9052 Acked Read 3 Done Sent Done false
31669 9053 Acked Read 2 Done Sent Done false
31670 9054 Acked Read 9 Done Sent Done false
31671 9055 Acked Read 8 Done Sent Done false
31672 9056 Acked Read 2 Done Sent Done false
31673 9057 Acked Read 2 Done Sent Done false
31674 9058 Acked Read 8 Done Sent Done false
31675 9059 Acked Read 2 Done Sent Done false
31676 9060 Acked Read 10 Done Sent Done false
31677 9061 Acked Read 4 Done Sent Done false
31678 9062 Acked Write 7 Done Sent Done false
31679 9063 Acked Read 8 Done Sent Done false
31680 9064 Acked Write 6 Done Sent Done false
31681 9065 Acked Read 6 Done Sent Done false
31682 9066 Acked Write 2 Done Sent Done false
31683 9067 Acked Flush 0 Done Sent Done false
31684 9068 Acked Flush 0 Done Sent Done false
31685 9069 Acked Read 5 Done Sent Done false
31686 9070 Acked Read 7 Done Sent Done false
31687 9071 Acked Read 7 Done Sent Done false
31688 9072 Acked Read 9 Done Sent Done false
31689 9073 Acked Write 10 Done Sent Done false
31690 9074 Acked Write 2 Done Sent Done false
31691 9075 Acked Flush 0 Done Sent Done false
31692 9076 Acked Write 4 Done Sent Done false
31693 9077 Acked Write 2 Done Sent Done false
31694 9078 Acked Flush 0 Done Sent Done false
31695 9079 Acked Read 1 Done Sent Done false
31696 9080 Acked Read 6 Done Sent Done false
31697 9081 Acked Write 3 Done Sent Done false
31698 9082 Acked Write 5 Done Sent Done false
31699 9083 Acked Read 4 Done Sent Done false
31700 9084 Acked Read 1 Done Sent Done false
31701 9085 Acked Read 10 Done Sent Done false
31702 9086 Acked Read 6 Done Sent Done false
31703 9087 Acked Read 5 Done Sent Done false
31704 9088 Acked Write 9 Done Sent Done false
31705 9089 Acked Read 5 Done Sent Done false
31706 9090 Acked Write 5 Done Sent Done false
31707 9091 Acked Read 1 Done Sent Done false
31708 9092 Acked Write 2 Done Sent Done false
31709 9093 Acked Write 7 Done Sent Done false
31710 9094 Acked Read 6 Done Sent Done false
31711 9095 Acked Flush 0 Done Sent Done false
31712 9096 Acked Read 9 Done Sent Done false
31713 9097 Acked Read 9 Done Sent Done false
31714 9098 Acked Write 6 Done Sent Done false
31715 9099 Acked Read 4 Done Sent Done false
31716 9100 Acked Read 2 Done Sent Done false
31717 9101 Acked Read 10 Done Sent Done false
31718 9102 Acked Write 4 Done Sent Done false
31719 9103 Acked Write 5 Done Sent Done false
31720 9104 Acked Read 2 Done Sent Done false
31721 9105 Acked Write 10 Done Sent Done false
31722 9106 Acked Read 8 Done Sent Done false
31723 9107 Acked Read 5 Done Sent Done false
31724 9108 Acked Flush 0 Done Sent Done false
31725 9109 Acked Flush 0 Done Sent Done false
31726 9110 Acked Write 1 Done Sent Done false
31727 9111 Acked Write 3 Done Sent Done false
31728 9112 Acked Write 10 Done Sent Done false
31729 9113 Acked Flush 0 Done Sent Done false
31730 9114 Acked Write 1 Done Sent Done false
31731 9115 Acked Read 9 Done Sent Done false
31732 9116 Acked Read 2 Done Sent Done false
31733 9117 Acked Read 1 Done Sent Done false
31734 9118 Acked Read 4 Done Sent Done false
31735 9119 Acked Write 3 Done Sent Done false
31736 9120 Acked Read 7 Done Sent Done false
31737 9121 Acked Read 2 Done Sent Done false
31738 9122 Acked Read 10 Done Sent Done false
31739 9123 Acked Read 10 Done Sent Done false
31740 9124 Acked Flush 0 Done Sent Done false
31741 9125 Acked Read 6 Done Sent Done false
31742 9126 Acked Write 4 Done Sent Done false
31743 9127 Acked Read 2 Done Sent Done false
31744 9128 Acked Read 2 Done Sent Done false
31745 9129 Acked Write 5 Done Sent Done false
31746 9130 Acked Write 9 Done Sent Done false
31747 9131 Acked Read 2 Done Sent Done false
31748 9132 Acked Write 6 Done Sent Done false
31749 9133 Acked Read 9 Done Sent Done false
31750 9134 Acked Write 6 Done Sent Done false
31751 9135 Acked Read 8 Done Sent Done false
31752 9136 Acked Flush 0 Done Sent Done false
31753 9137 Acked Read 6 Done Sent Done false
31754 9138 Acked Write 4 Done Sent Done false
31755 9139 Acked Write 8 Done Sent Done false
31756 9140 Acked Read 7 Done Sent Done false
31757 9141 Acked Flush 0 Done Sent Done false
31758 9142 Acked Read 7 Done Sent Done false
31759 9143 Acked Read 6 Done Sent Done false
31760 9144 Acked Flush 0 Done Sent Done false
31761 9145 Acked Write 3 Done Sent Done false
31762 9146 Acked Write 5 Done Sent Done false
31763 9147 Acked Write 7 Done Sent Done false
31764 9148 Acked Write 4 Done Sent Done false
31765 9149 Acked Read 2 Done Sent Done false
31766 9150 Acked Write 1 Done Sent Done false
31767 9151 Acked Read 10 Done Sent Done false
31768 STATES DS:0 DS:1 DS:2 TOTAL
31769 Sent 0 301 0 301
31770 Done 301 0 301 602
31771 Skipped 0 0 0 0
31772 Error 0 0 0 0
31773 Last Flush: Some(JobId(9144)) Some(JobId(8850)) Some(JobId(9144))
31774 Downstairs last five completed: 8850 8849 8848 8847 8846
31775 Upstairs last five completed: 9151 9150 9149 9148 9147
31776 CLIENT: Up:0 ds:301 act:2
317772025-01-29T05:17:42.776ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317782025-01-29T05:17:43.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317792025-01-29T05:17:44.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317802025-01-29T05:17:45.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317812025-01-29T05:17:46.018ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317822025-01-29T05:17:46.018ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56116) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317832025-01-29T05:17:46.019ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317842025-01-29T05:17:46.020ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317852025-01-29T05:17:46.020ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317862025-01-29T05:17:46.020ZINFOcrucible: send last flush ID to this DS: Some(JobId(8850)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317872025-01-29T05:17:46.020ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317882025-01-29T05:17:46.020ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8850)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
317892025-01-29T05:17:46.020ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8850)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31790 ----------------------------------------------------------------
31791 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:7
31792 States: Active Active Active
31793 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31794 9145 Acked Write 3 Done Done Done true
31795 9146 Acked Write 5 Done Done Done true
31796 9147 Acked Write 7 Done Done Done true
31797 9148 Acked Write 4 Done Done Done true
31798 9149 Acked Read 2 Done Done Done true
31799 9150 Acked Write 1 Done Done Done true
31800 9151 Acked Read 10 Done Done Done true
31801 STATES DS:0 DS:1 DS:2 TOTAL
31802 Sent 0 0 0 0
31803 Done 7 7 7 21
31804 Skipped 0 0 0 0
31805 Error 0 0 0 0
31806 Last Flush: Some(JobId(9144)) Some(JobId(9144)) Some(JobId(9144))
31807 Downstairs last five completed: 9144 9143 9142 9141 9140
31808 Upstairs last five completed: 9151 9150 9149 9148 9147
31809 CLIENT: Up:0 ds:7 act:3
31810 ----------------------------------------------------------------
31811 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31812 Downstairs last five completed: 9152 9151 9150 9149 9148
31813 Upstairs last five completed: 9152 9151 9150 9149 9148
31814 CLIENT: Up:0 ds:0 act:3
31815 Replay: All jobs finished
31816 Main task has work to do, go find it
31817 got dsc Stop(0)
31818 stop 0
31819 Jan 29 05:17:50.278[8810] Got stop action so:false kr:false
31820 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3f2dea0b-a076-40a7-b7e9-1f93c77ea42c, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
318212025-01-29T05:17:50.279ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
318222025-01-29T05:17:50.279ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
318232025-01-29T05:17:50.279ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
318242025-01-29T05:17:50.279ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
31825 Jan 29 05:17:50.279 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/0, method: GET, req_id: 28595ce1-ef34-4d2b-8b93-fc3863c3f367, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
31826 [8810] Exited with: ExitStatus(unix_wait_status(9))
31827 [8810][0] reports Exit
31828 Jan 29 05:17:54.281 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2662f3b7-64db-4f6b-a454-4cb48fa9d1a2, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
31829 001/300 Read block 1302 len 4096
31830 002/300 Flush
31831 003/300 Write block 1966 len 24576 data: 3 5 6 7 8 8
31832 004/300 Read block 1995 len 20480
31833 005/300 Write block 4066 len 32768 data: 5 5 4 5 6 5 6 7
31834 006/300 Write block 2421 len 32768 data: 4 5 8 7 10 9 7 6
31835 007/300 Read block 3319 len 24576
31836 008/300 Read block 914 len 24576
31837 009/300 Read block 4847 len 24576
31838 010/300 Flush
31839 011/300 Write block 4215 len 20480 data: 4 4 5 5 7
31840 012/300 Read block 2792 len 8192
31841 013/300 Read block 1424 len 28672
31842 014/300 Flush
31843 015/300 Read block 1045 len 20480
31844 016/300 Read block 4391 len 28672
31845 017/300 Read block 2898 len 28672
31846 018/300 Read block 2330 len 4096
31847 019/300 Read block 1493 len 20480
31848 020/300 Flush
31849 021/300 Write block 1033 len 4096 data: 6
31850 022/300 Read block 3522 len 20480
31851 023/300 Read block 1900 len 24576
31852 024/300 Read block 3163 len 4096
31853 025/300 Write block 3510 len 36864 data: 3 3 5 6 7 7 8 6 5
31854 026/300 Read block 2897 len 40960
31855 027/300 Read block 2372 len 8192
31856 028/300 Flush
31857 029/300 Read block 727 len 20480
31858 030/300 Read block 4727 len 20480
31859 031/300 Read block 4136 len 4096
31860 032/300 Write block 3030 len 24576 data: 7 6 7 8 6 7
31861 033/300 Read block 2165 len 12288
31862 034/300 Read block 4360 len 4096
31863 035/300 Read block 4333 len 24576
31864 036/300 Read block 3721 len 16384
31865 037/300 Read block 797 len 16384
31866 038/300 Write block 427 len 36864 data: 7 5 5 4 4 3 3 3 2
31867 039/300 Write block 3754 len 12288 data: 6 6 6
31868 040/300 Read block 639 len 24576
31869 041/300 Write block 3182 len 8192 data: 4 2
31870 042/300 Read block 2881 len 12288
31871 043/300 Read block 1043 len 16384
31872 044/300 Read block 1066 len 28672
31873 045/300 Write block 506 len 16384 data: 9 11 8 7
31874 046/300 Write block 1997 len 20480 data: 4 5 6 7 7
31875 047/300 Write block 3634 len 12288 data: 3 4 4
31876 048/300 Write block 1593 len 36864 data: 7 8 6 5 5 5 5 6 2
31877 049/300 Read block 4472 len 24576
31878 050/300 Read block 2503 len 24576
31879 051/300 Read block 4178 len 32768
31880 052/300 Write block 2960 len 28672 data: 8 9 9 9 8 8 7
31881 053/300 Read block 3287 len 32768
31882 054/300 Write block 2256 len 20480 data: 6 5 4 5 4
31883 055/300 Read block 1072 len 40960
31884 056/300 Read block 2159 len 4096
31885 057/300 Write block 3513 len 4096 data: 7
31886 058/300 Read block 2549 len 36864
31887 059/300 Read block 231 len 32768
31888 060/300 Flush
31889 061/300 Read block 3642 len 8192
31890 062/300 Write block 4932 len 20480 data: 9 9 7 7 6
31891 063/300 Write block 3223 len 32768 data: 9 6 6 5 5 8 10 10
31892 064/300 Write block 2466 len 20480 data: 5 5 6 7 7
31893 065/300 Read block 1694 len 20480
31894 066/300 Write block 266 len 4096 data: 4
31895 067/300 Read block 1847 len 24576
31896 068/300 Write block 1458 len 8192 data: 3 4
31897 069/300 Read block 2815 len 4096
31898 070/300 Read block 3264 len 24576
31899 071/300 Write block 348 len 20480 data: 7 6 6 6 6
31900 072/300 Read block 2498 len 8192
31901 073/300 Read block 269 len 20480
31902 074/300 Flush
31903 075/300 Write block 1750 len 28672 data: 6 5 4 4 4 4 4
31904 076/300 Write block 2894 len 8192 data: 5 5
31905 077/300 Flush
31906 078/300 Read block 149 len 16384
31907 079/300 Write block 2624 len 40960 data: 6 4 4 3 3 3 4 3 5 5
31908 080/300 Write block 4418 len 24576 data: 5 6 5 4 4 5
31909 081/300 Read block 3984 len 24576
31910 082/300 Write block 2793 len 36864 data: 7 4 4 4 4 5 4 3 4
31911 083/300 Flush
31912 084/300 Read block 1647 len 16384
31913 085/300 Flush
31914 086/300 Read block 1531 len 8192
31915 087/300 Write block 1353 len 28672 data: 5 6 5 3 4 4 4
31916 088/300 Write block 1919 len 20480 data: 6 6 7 7 8
31917 089/300 Write block 575 len 12288 data: 8 7 10
31918 090/300 Flush
31919 091/300 Read block 4158 len 4096
31920 092/300 Read block 3617 len 28672
31921 093/300 Read block 3645 len 40960
31922 094/300 Write block 4250 len 16384 data: 2 2 4 4
31923 095/300 Read block 225 len 32768
31924 096/300 Read block 4824 len 8192
31925 097/300 Write block 4885 len 28672 data: 6 9 8 8 8 8 5
31926 098/300 Read block 3744 len 8192
31927 099/300 Read block 1974 len 12288
31928 100/300 Read block 4199 len 12288
31929 101/300 Read block 2953 len 40960
31930 102/300 Read block 4176 len 8192
31931 103/300 Read block 4475 len 8192
31932 104/300 Read block 2267 len 28672
31933 105/300 Read block 4820 len 20480
31934 106/300 Read block 3425 len 28672
31935 107/300 Write block 1602 len 36864 data: 2 3 3 3 3 4 4 4 5
31936 108/300 Write block 3085 len 40960 data: 4 3 3 3 4 5 4 4 4 4
31937 109/300 Read block 2696 len 8192
31938 110/300 Write block 2468 len 40960 data: 7 8 8 7 5 5 5 5 7 7
31939 111/300 Write block 661 len 20480 data: 6 7 7 8 6
31940 112/300 Read block 3089 len 40960
31941 113/300 Write block 1583 len 28672 data: 6 7 7 7 5 4 5
31942 114/300 Write block 896 len 20480 data: 5 5 8 8 8
31943 115/300 Read block 1174 len 32768
31944 116/300 Write block 4789 len 40960 data: 3 3 3 2 2 2 3 3 3 4
31945 117/300 Write block 3306 len 20480 data: 2 2 2 2 3
31946 118/300 Write block 676 len 32768 data: 4 4 5 5 4 3 4 4
31947 119/300 Read block 1794 len 16384
31948 120/300 Write block 3407 len 24576 data: 4 4 4 3 2 2
31949 121/300 Read block 3570 len 36864
31950 122/300 Write block 3973 len 40960 data: 8 8 9 9 10 10 10 7 8 8
31951 123/300 Read block 4608 len 12288
31952 124/300 Read block 2930 len 16384
31953 125/300 Write block 959 len 28672 data: 6 6 6 7 7 5 8
31954 126/300 Flush
31955 127/300 Read block 2873 len 20480
31956 128/300 Read block 2005 len 28672
31957 129/300 Read block 3783 len 12288
31958 130/300 Flush
31959 131/300 Write block 434 len 36864 data: 4 3 2 2 3 5 4 4 4
31960 132/300 Read block 4588 len 24576
31961 133/300 Write block 1403 len 20480 data: 8 8 7 6 6
31962 134/300 Read block 4518 len 8192
31963 135/300 Read block 1974 len 24576
31964 136/300 Read block 4830 len 28672
31965 137/300 Read block 4934 len 4096
31966 138/300 Write block 4190 len 4096 data: 6
31967 139/300 Read block 4035 len 24576
31968 140/300 Read block 458 len 16384
31969 141/300 Read block 4646 len 12288
31970 142/300 Write block 4632 len 8192 data: 3 3
31971 143/300 Write block 4249 len 28672 data: 4 3 3 5 5 4 4
31972 144/300 Read block 2707 len 32768
31973 145/300 Write block 3061 len 20480 data: 4 5 5 5 5
31974 146/300 Read block 1811 len 20480
31975 147/300 Write block 192 len 40960 data: 8 6 7 7 7 7 6 4 4 4
31976 148/300 Read block 2462 len 24576
31977 149/300 Write block 2644 len 32768 data: 6 5 5 6 6 5 6 4
31978 150/300 Read block 2562 len 28672
31979 151/300 Read block 930 len 28672
31980 152/300 Write block 2966 len 28672 data: 8 6 4 5 5 6 7
31981 153/300 Read block 3063 len 8192
31982 154/300 Write block 1410 len 28672 data: 7 6 6 7 7 6 9
31983 155/300 Write block 1827 len 28672 data: 5 5 5 5 5 6 3
31984 156/300 Read block 274 len 8192
31985 157/300 Read block 262 len 28672
31986 158/300 Write block 938 len 28672 data: 4 3 3 4 4 4 4
31987 159/300 Write block 4390 len 20480 data: 5 8 9 10 11
31988 160/300 Write block 2117 len 16384 data: 6 4 4 5
31989 161/300 Write block 3281 len 32768 data: 4 5 7 7 7 7 7 7
31990 162/300 Read block 592 len 24576
31991 163/300 Write block 4142 len 12288 data: 8 6 6
31992 164/300 Read block 749 len 8192
31993 165/300 Read block 22 len 32768
31994 166/300 Read block 4619 len 20480
31995 167/300 Flush
31996 168/300 Read block 4353 len 40960
31997 169/300 Read block 3885 len 12288
31998 170/300 Flush
31999 171/300 Read block 4357 len 12288
32000 172/300 Write block 3253 len 8192 data: 3 4
32001 173/300 Write block 1810 len 32768 data: 6 6 6 6 6 6 6 7
32002 174/300 Read block 2597 len 24576
32003 175/300 Read block 2341 len 16384
32004 176/300 Write block 4756 len 20480 data: 5 6 6 4 4
32005 177/300 Write block 1964 len 8192 data: 4 4
32006 178/300 Write block 2107 len 20480 data: 7 7 6 8 8
32007 179/300 Read block 1817 len 36864
32008 180/300 Write block 785 len 24576 data: 3 2 4 4 4 4
32009 181/300 Flush
32010 182/300 Read block 3747 len 28672
32011 183/300 Write block 3543 len 32768 data: 5 7 6 5 5 4 5 5
32012 184/300 Write block 3349 len 32768 data: 8 7 9 8 4 4 4 5
32013 185/300 Write block 3155 len 28672 data: 6 6 4 4 3 3 4
32014 186/300 Write block 589 len 24576 data: 8 11 9 9 8 10
32015 187/300 Read block 2957 len 20480
32016 188/300 Write block 1030 len 28672 data: 5 5 5 7 5 5 3
32017 189/300 Write block 4786 len 32768 data: 5 3 3 4 4 4 3 3
32018 190/300 Read block 1977 len 20480
32019 191/300 Write block 2620 len 16384 data: 6 6 6 6
32020 192/300 Read block 518 len 4096
32021 193/300 Read block 4716 len 24576
32022 194/300 Write block 1417 len 28672 data: 7 7 5 6 6 5 6
32023 195/300 Read block 2351 len 40960
32024 196/300 Write block 640 len 32768 data: 8 7 8 7 6 4 4 5
32025 197/300 Flush
32026 198/300 Write block 3281 len 20480 data: 5 6 8 8 8
32027 199/300 Write block 543 len 24576 data: 7 6 5 6 5 6
32028 200/300 Read block 3760 len 28672
32029 201/300 Write block 3761 len 12288 data: 4 5 7
32030 202/300 Read block 4480 len 40960
32031 203/300 Write block 3343 len 16384 data: 4 6 7 8
32032 204/300 Read block 672 len 36864
32033 205/300 Read block 3202 len 4096
32034 206/300 Flush
32035 207/300 Write block 4203 len 8192 data: 9 8
32036 208/300 Read block 4215 len 28672
32037 209/300 Read block 3635 len 20480
32038 210/300 Read block 2968 len 40960
32039 211/300 Read block 2320 len 36864
32040 212/300 Read block 832 len 32768
32041 213/300 Write block 3078 len 12288 data: 6 6 7
32042 214/300 Read block 3858 len 20480
32043 215/300 Write block 26 len 24576 data: 5 5 5 6 4 4
32044 216/300 Write block 1284 len 32768 data: 8 9 10 9 8 8 6 6
32045 217/300 Write block 2732 len 40960 data: 9 9 10 10 9 8 9 9 8 8
32046 218/300 Write block 1808 len 12288 data: 6 6 7
32047 219/300 Flush
32048 220/300 Read block 1173 len 24576
32049 221/300 Write block 4464 len 28672 data: 6 5 5 4 5 6 4
32050 222/300 Read block 1539 len 12288
32051 223/300 Read block 1236 len 40960
32052 224/300 Read block 4918 len 36864
32053 225/300 Read block 933 len 12288
32054 226/300 Read block 3814 len 20480
32055 227/300 Read block 3574 len 4096
32056 228/300 Write block 2747 len 32768 data: 9 8 8 7 8 9 8 7
32057 229/300 Read block 4485 len 32768
32058 230/300 Write block 1757 len 16384 data: 3 5 5 6
32059 231/300 Write block 568 len 32768 data: 3 5 5 5 6 7 7 9
32060 232/300 Flush
32061 233/300 Write block 1105 len 40960 data: 4 4 5 6 6 6 6 6 3 4
32062 234/300 Read block 1310 len 24576
32063 235/300 Write block 391 len 12288 data: 5 4 5
32064 236/300 Read block 1829 len 32768
32065 237/300 Flush
32066 238/300 Read block 1545 len 4096
32067 239/300 Write block 4041 len 4096 data: 6
32068 240/300 Read block 2900 len 20480
32069 241/300 Read block 4292 len 12288
32070 242/300 Read block 3907 len 40960
32071 243/300 Read block 4687 len 16384
32072 244/300 Write block 747 len 32768 data: 8 6 9 8 7 8 8 7
32073 245/300 Read block 4535 len 28672
32074 246/300 Write block 4558 len 20480 data: 6 5 5 5 3
32075 247/300 Flush
32076 248/300 Write block 2553 len 20480 data: 7 6 6 5 6
32077 249/300 Read block 579 len 4096
32078 250/300 Read block 3746 len 36864
32079 251/300 Read block 4018 len 8192
32080 252/300 Read block 1139 len 32768
32081 253/300 Read block 3123 len 32768
32082 254/300 Read block 3200 len 36864
32083 255/300 Write block 1685 len 40960 data: 11 10 11 11 13 13 13 8 9 5
32084 256/300 Flush
32085 257/300 Write block 2117 len 40960 data: 7 5 5 6 5 5 5 5 5 5
32086 258/300 Flush
32087 259/300 Write block 4642 len 32768 data: 10 10 9 8 7 7 7 5
32088 260/300 Read block 364 len 36864
32089 261/300 Read block 82 len 24576
32090 262/300 Read block 4229 len 16384
32091 263/300 Write block 1486 len 12288 data: 3 4 3
32092 264/300 Write block 4953 len 8192 data: 5 5
32093 265/300 Write block 1568 len 40960 data: 9 9 10 10 9 9 8 7 7 5
32094 266/300 Read block 3078 len 20480
32095 267/300 Write block 526 len 16384 data: 4 4 4 4
32096 268/300 Write block 990 len 8192 data: 4 4
32097 269/300 Read block 4491 len 12288
32098 270/300 Write block 3701 len 8192 data: 6 7
32099 271/300 Flush
32100 272/300 Write block 4962 len 32768 data: 7 7 8 7 6 7 6 6
32101 273/300 Write block 3471 len 40960 data: 6 4 4 5 3 5 4 5 6 7
32102 274/300 Write block 1468 len 16384 data: 4 3 4 4
32103 275/300 Read block 1466 len 20480
32104 276/300 Read block 4396 len 24576
32105 277/300 Read block 755 len 16384
32106 278/300 Read block 166 len 32768
32107 279/300 Read block 4495 len 24576
32108 280/300 Read block 3703 len 40960
32109 281/300 Read block 3202 len 24576
32110 282/300 Read block 4274 len 12288
32111 283/300 Write block 1574 len 24576 data: 9 8 8 6 4 5
32112 284/300 Read block 749 len 32768
32113 285/300 Read block 355 len 16384
32114 286/300 Read block 1831 len 16384
32115 287/300 Read block 4821 len 12288
32116 288/300 Write block 4289 len 32768 data: 8 8 7 6 7 7 7 7
32117 289/300 Read block 875 len 4096
32118 290/300 Read block 4227 len 12288
32119 291/300 Write block 2440 len 28672 data: 6 4 5 5 6 5 5
32120 292/300 Read block 4167 len 24576
32121 293/300 Write block 1764 len 24576 data: 6 6 7 6 5 6
32122 294/300 Read block 4194 len 4096
32123 295/300 Flush
32124 296/300 Write block 2549 len 16384 data: 10 9 7 8
32125 297/300 Write block 4422 len 32768 data: 5 6 6 6 5 7 6 4
32126 298/300 Read block 3847 len 4096
32127 299/300 Flush
32128 300/300 Flush
32129 301/300 Write block 1956 len 20480 data: 5 4 4 5 5
32130 Main task has work to do, go find it
32131 got dsc Start(0)
32132 start 0
32133 Jan 29 05:17:54.526 INFO request completed, latency_us: 193, response_code: 204, uri: /start/cid/0, method: POST, req_id: fcf09944-eb3b-4de9-8e60-82d3931bd977, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
32134 Replay: started 0, returned:Ok(())
32135 ----------------------------------------------------------------
32136 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32137 States: Offline Active Active
32138 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32139 9153 Acked Read 1 Sent Done Done false
32140 9154 Acked Flush 0 Sent Done Done false
32141 9155 Acked Write 6 Sent Done Done false
32142 9156 Acked Read 5 Sent Done Done false
32143 9157 Acked Write 8 Sent Done Done false
32144 9158 Acked Write 8 Sent Done Done false
32145 9159 Acked Read 6 Sent Done Done false
32146 9160 Acked Read 6 Sent Done Done false
32147 9161 Acked Read 6 Sent Done Done false
32148 9162 Acked Flush 0 Sent Done Done false
32149 9163 Acked Write 5 Sent Done Done false
32150 9164 Acked Read 2 Sent Done Done false
32151 9165 Acked Read 7 Sent Done Done false
32152 9166 Acked Flush 0 Sent Done Done false
32153 9167 Acked Read 5 Sent Done Done false
32154 9168 Acked Read 7 Sent Done Done false
32155 9169 Acked Read 7 Sent Done Done false
32156 9170 Acked Read 1 Sent Done Done false
32157 9171 Acked Read 5 Sent Done Done false
32158 9172 Acked Flush 0 Sent Done Done false
32159 9173 Acked Write 1 Sent Done Done false
32160 9174 Acked Read 5 Sent Done Done false
32161 9175 Acked Read 6 Sent Done Done false
32162 9176 Acked Read 1 Sent Done Done false
32163 9177 Acked Write 9 Sent Done Done false
32164 9178 Acked Read 10 Sent Done Done false
32165 9179 Acked Read 2 Sent Done Done false
32166 9180 Acked Flush 0 Sent Done Done false
32167 9181 Acked Read 5 Sent Done Done false
32168 9182 Acked Read 5 Sent Done Done false
32169 9183 Acked Read 1 Sent Done Done false
32170 9184 Acked Write 6 Sent Done Done false
32171 9185 Acked Read 3 Sent Done Done false
32172 9186 Acked Read 1 Sent Done Done false
32173 9187 Acked Read 6 Sent Done Done false
32174 9188 Acked Read 4 Sent Done Done false
32175 9189 Acked Read 4 Sent Done Done false
32176 9190 Acked Write 9 Sent Done Done false
32177 9191 Acked Write 3 Sent Done Done false
32178 9192 Acked Read 6 Sent Done Done false
32179 9193 Acked Write 2 Sent Done Done false
32180 9194 Acked Read 3 Sent Done Done false
32181 9195 Acked Read 4 Sent Done Done false
32182 9196 Acked Read 7 Sent Done Done false
32183 9197 Acked Write 4 Sent Done Done false
32184 9198 Acked Write 5 Sent Done Done false
32185 9199 Acked Write 3 Sent Done Done false
32186 9200 Acked Write 9 Sent Done Done false
32187 9201 Acked Read 6 Sent Done Done false
32188 9202 Acked Read 6 Sent Done Done false
32189 9203 Acked Read 8 Sent Done Done false
32190 9204 Acked Write 7 Sent Done Done false
32191 9205 Acked Read 8 Sent Done Done false
32192 9206 Acked Write 5 Sent Done Done false
32193 9207 Acked Read 10 Sent Done Done false
32194 9208 Acked Read 1 Sent Done Done false
32195 9209 Acked Write 1 Sent Done Done false
32196 9210 Acked Read 9 Sent Done Done false
32197 9211 Acked Read 8 Sent Done Done false
32198 9212 Acked Flush 0 Sent Done Done false
32199 9213 Acked Read 2 Sent Done Done false
32200 9214 Acked Write 5 Sent Done Done false
32201 9215 Acked Write 8 Sent Done Done false
32202 9216 Acked Write 5 Sent Done Done false
32203 9217 Acked Read 5 Sent Done Done false
32204 9218 Acked Write 1 Sent Done Done false
32205 9219 Acked Read 6 Sent Done Done false
32206 9220 Acked Write 2 Sent Done Done false
32207 9221 Acked Read 1 Sent Done Done false
32208 9222 Acked Read 6 Sent Done Done false
32209 9223 Acked Write 5 Sent Done Done false
32210 9224 Acked Read 2 Sent Done Done false
32211 9225 Acked Read 5 Sent Done Done false
32212 9226 Acked Flush 0 Sent Done Done false
32213 9227 Acked Write 7 Sent Done Done false
32214 9228 Acked Write 2 Sent Done Done false
32215 9229 Acked Flush 0 Sent Done Done false
32216 9230 Acked Read 4 Sent Done Done false
32217 9231 Acked Write 10 Sent Done Done false
32218 9232 Acked Write 6 Sent Done Done false
32219 9233 Acked Read 6 Sent Done Done false
32220 9234 Acked Write 9 Sent Done Done false
32221 9235 Acked Flush 0 Sent Done Done false
32222 9236 Acked Read 4 Sent Done Done false
32223 9237 Acked Flush 0 Sent Done Done false
32224 9238 Acked Read 2 Sent Done Done false
32225 9239 Acked Write 7 Sent Done Done false
32226 9240 Acked Write 5 Sent Done Done false
32227 9241 Acked Write 3 Sent Done Done false
32228 9242 Acked Flush 0 Sent Done Done false
32229 9243 Acked Read 1 Sent Done Done false
32230 9244 Acked Read 7 Sent Done Done false
32231 9245 Acked Read 10 Sent Done Done false
32232 9246 Acked Write 4 Sent Done Done false
32233 9247 Acked Read 8 Sent Done Done false
32234 9248 Acked Read 2 Sent Done Done false
32235 9249 Acked Write 7 Sent Done Done false
32236 9250 Acked Read 2 Sent Done Done false
32237 9251 Acked Read 3 Sent Done Done false
32238 9252 Acked Read 3 Sent Done Done false
32239 9253 Acked Read 10 Sent Done Done false
32240 9254 Acked Read 2 Sent Done Done false
32241 9255 Acked Read 2 Sent Done Done false
32242 9256 Acked Read 7 Sent Done Done false
32243 9257 Acked Read 5 Sent Done Done false
32244 9258 Acked Read 7 Sent Done Done false
32245 9259 Acked Write 9 Sent Done Done false
32246 9260 Acked Write 10 Sent Done Done false
32247 9261 Acked Read 2 Sent Done Done false
32248 9262 Acked Write 10 Sent Done Done false
32249 9263 Acked Write 5 Sent Done Done false
32250 9264 Acked Read 10 Sent Done Done false
32251 9265 Acked Write 7 Sent Done Done false
32252 9266 Acked Write 5 Sent Done Done false
32253 9267 Acked Read 8 Sent Done Done false
32254 9268 Acked Write 10 Sent Done Done false
32255 9269 Acked Write 5 Sent Done Done false
32256 9270 Acked Write 8 Sent Done Done false
32257 9271 Acked Read 4 Sent Done Done false
32258 9272 Acked Write 6 Sent Done Done false
32259 9273 Acked Read 9 Sent Done Done false
32260 9274 Acked Write 10 Sent Done Done false
32261 9275 Acked Read 3 Sent Done Done false
32262 9276 Acked Read 4 Sent Done Done false
32263 9277 Acked Write 7 Sent Done Done false
32264 9278 Acked Flush 0 Sent Done Done false
32265 9279 Acked Read 5 Sent Done Done false
32266 9280 Acked Read 7 Sent Done Done false
32267 9281 Acked Read 3 Sent Done Done false
32268 9282 Acked Flush 0 Sent Done Done false
32269 9283 Acked Write 9 Sent Done Done false
32270 9284 Acked Read 6 Sent Done Done false
32271 9285 Acked Write 5 Sent Done Done false
32272 9286 Acked Read 2 Sent Done Done false
32273 9287 Acked Read 6 Sent Done Done false
32274 9288 Acked Read 7 Sent Done Done false
32275 9289 Acked Read 1 Sent Done Done false
32276 9290 Acked Write 1 Sent Done Done false
32277 9291 Acked Read 6 Sent Done Done false
32278 9292 Acked Read 4 Sent Done Done false
32279 9293 Acked Read 3 Sent Done Done false
32280 9294 Acked Write 2 Sent Done Done false
32281 9295 Acked Write 7 Sent Done Done false
32282 9296 Acked Read 8 Sent Done Done false
32283 9297 Acked Write 5 Sent Done Done false
32284 9298 Acked Read 5 Sent Done Done false
32285 9299 Acked Write 10 Sent Done Done false
32286 9300 Acked Read 6 Sent Done Done false
32287 9301 Acked Write 8 Sent Done Done false
32288 9302 Acked Read 7 Sent Done Done false
32289 9303 Acked Read 7 Sent Done Done false
32290 9304 Acked Write 7 Sent Done Done false
32291 9305 Acked Read 2 Sent Done Done false
32292 9306 Acked Write 7 Sent Done Done false
32293 9307 Acked Write 7 Sent Done Done false
32294 9308 Acked Read 2 Sent Done Done false
32295 9309 Acked Read 7 Sent Done Done false
32296 9310 Acked Write 7 Sent Done Done false
32297 9311 Acked Write 5 Sent Done Done false
32298 9312 Acked Write 4 Sent Done Done false
32299 9313 Acked Write 8 Sent Done Done false
32300 9314 Acked Read 6 Sent Done Done false
32301 9315 Acked Write 3 Sent Done Done false
32302 9316 Acked Read 2 Sent Done Done false
32303 9317 Acked Read 8 Sent Done Done false
32304 9318 Acked Read 5 Sent Done Done false
32305 9319 Acked Flush 0 Sent Done Done false
32306 9320 Acked Read 10 Sent Done Done false
32307 9321 Acked Read 3 Sent Done Done false
32308 9322 Acked Flush 0 Sent Done Done false
32309 9323 Acked Read 3 Sent Done Done false
32310 9324 Acked Write 2 Sent Done Done false
32311 9325 Acked Write 8 Sent Done Done false
32312 9326 Acked Read 6 Sent Done Done false
32313 9327 Acked Read 4 Sent Done Done false
32314 9328 Acked Write 5 Sent Done Done false
32315 9329 Acked Write 2 Sent Done Done false
32316 9330 Acked Write 5 Sent Done Done false
32317 9331 Acked Read 9 Sent Done Done false
32318 9332 Acked Write 6 Sent Done Done false
32319 9333 Acked Flush 0 Sent Done Done false
32320 9334 Acked Read 7 Sent Done Done false
32321 9335 Acked Write 8 Sent Done Done false
32322 9336 Acked Write 8 Sent Done Done false
32323 9337 Acked Write 7 Sent Done Done false
32324 9338 Acked Write 6 Sent Done Done false
32325 9339 Acked Read 5 Sent Done Done false
32326 9340 Acked Write 7 Sent Done Done false
32327 9341 Acked Write 8 Sent Done Done false
32328 9342 Acked Read 5 Sent Done Done false
32329 9343 Acked Write 4 Sent Done Done false
32330 9344 Acked Read 1 Sent Done Done false
32331 9345 Acked Read 6 Sent Done Done false
32332 9346 Acked Write 7 Sent Done Done false
32333 9347 Acked Read 10 Sent Done Done false
32334 9348 Acked Write 8 Sent Done Done false
32335 9349 Acked Flush 0 Sent Done Done false
32336 9350 Acked Write 5 Sent Done Done false
32337 9351 Acked Write 6 Sent Done Done false
32338 9352 Acked Read 7 Sent Done Done false
32339 9353 Acked Write 3 Sent Done Done false
32340 9354 Acked Read 10 Sent Done Done false
32341 9355 Acked Write 4 Sent Done Done false
32342 9356 Acked Read 9 Sent Done Done false
32343 9357 Acked Read 1 Sent Done Done false
32344 9358 Acked Flush 0 Sent Done Done false
32345 9359 Acked Write 2 Sent Done Done false
32346 9360 Acked Read 7 Sent Done Done false
32347 9361 Acked Read 5 Sent Done Done false
32348 9362 Acked Read 10 Sent Done Done false
32349 9363 Acked Read 9 Sent Done Done false
32350 9364 Acked Read 8 Sent Done Done false
32351 9365 Acked Write 3 Sent Done Done false
32352 9366 Acked Read 5 Sent Done Done false
32353 9367 Acked Write 6 Sent Done Done false
32354 9368 Acked Write 8 Sent Done Done false
32355 9369 Acked Write 10 Sent Done Done false
32356 9370 Acked Write 3 Sent Done Done false
32357 9371 Acked Flush 0 Sent Done Done false
32358 9372 Acked Read 6 Sent Done Done false
32359 9373 Acked Write 7 Sent Done Done false
32360 9374 Acked Read 3 Sent Done Done false
32361 9375 Acked Read 10 Sent Done Done false
32362 9376 Acked Read 9 Sent Done Done false
32363 9377 Acked Read 3 Sent Done Done false
32364 9378 Acked Read 5 Sent Done Done false
32365 9379 Acked Read 1 Sent Done Done false
32366 9380 Acked Write 8 Sent Done Done false
32367 9381 Acked Read 8 Sent Done Done false
32368 9382 Acked Write 4 Sent Done Done false
32369 9383 Acked Write 8 Sent Done Done false
32370 9384 Acked Flush 0 Sent Done Done false
32371 9385 Acked Write 10 Sent Done Done false
32372 9386 Acked Read 6 Sent Done Done false
32373 9387 Acked Write 3 Sent Done Done false
32374 9388 Acked Read 8 Sent Done Done false
32375 9389 Acked Flush 0 Sent Done Done false
32376 9390 Acked Read 1 Sent Done Done false
32377 9391 Acked Write 1 Sent Done Done false
32378 9392 Acked Read 5 Sent Done Done false
32379 9393 Acked Read 3 Sent Done Done false
32380 9394 Acked Read 10 Sent Done Done false
32381 9395 Acked Read 4 Sent Done Done false
32382 9396 Acked Write 8 Sent Done Done false
32383 9397 Acked Read 7 Sent Done Done false
32384 9398 Acked Write 5 Sent Done Done false
32385 9399 Acked Flush 0 Sent Done Done false
32386 9400 Acked Write 5 Sent Done Done false
32387 9401 Acked Read 1 Sent Done Done false
32388 9402 Acked Read 9 Sent Done Done false
32389 9403 Acked Read 2 Sent Done Done false
32390 9404 Acked Read 8 Sent Done Done false
32391 9405 Acked Read 8 Sent Done Done false
32392 9406 Acked Read 9 Sent Done Done false
32393 9407 Acked Write 10 Sent Done Done false
32394 9408 Acked Flush 0 Sent Done Done false
32395 9409 Acked Write 10 Sent Done Done false
32396 9410 Acked Flush 0 Sent Done Done false
32397 9411 Acked Write 8 Sent Done Done false
32398 9412 Acked Read 9 Sent Done Done false
32399 9413 Acked Read 6 Sent Done Done false
32400 9414 Acked Read 4 Sent Done Done false
32401 9415 Acked Write 3 Sent Done Done false
32402 9416 Acked Write 2 Sent Done Done false
32403 9417 Acked Write 10 Sent Done Done false
32404 9418 Acked Read 5 Sent Done Done false
32405 9419 Acked Write 4 Sent Done Done false
32406 9420 Acked Write 2 Sent Done Done false
32407 9421 Acked Read 3 Sent Done Done false
32408 9422 Acked Write 2 Sent Done Done false
32409 9423 Acked Flush 0 Sent Done Done false
32410 9424 Acked Write 8 Sent Done Done false
32411 9425 Acked Write 10 Sent Done Done false
32412 9426 Acked Write 4 Sent Done Done false
32413 9427 Acked Read 5 Sent Done Done false
32414 9428 Acked Read 6 Sent Done Done false
32415 9429 Acked Read 4 Sent Done Done false
32416 9430 Acked Read 8 Sent Done Done false
32417 9431 Acked Read 6 Sent Done Done false
32418 9432 Acked Read 10 Sent Done Done false
32419 9433 Acked Read 6 Sent Done Done false
32420 9434 Acked Read 3 Sent Done Done false
32421 9435 Acked Write 6 Sent Done Done false
32422 9436 Acked Read 8 Sent Done Done false
32423 9437 Acked Read 4 Sent Done Done false
32424 9438 Acked Read 4 Sent Done Done false
32425 9439 Acked Read 3 Sent Done Done false
32426 9440 Acked Write 8 Sent Done Done false
32427 9441 Acked Read 1 Sent Done Done false
32428 9442 Acked Read 3 Sent Done Done false
32429 9443 Acked Write 7 Sent Done Done false
32430 9444 Acked Read 6 Sent Done Done false
32431 9445 Acked Write 6 Sent Done Done false
32432 9446 Acked Read 1 Sent Done Done false
32433 9447 Acked Flush 0 Sent Done Done false
32434 9448 Acked Write 4 Sent Done Done false
32435 9449 Acked Write 8 Sent Done Done false
32436 9450 Acked Read 1 Sent Done Done false
32437 9451 Acked Flush 0 Sent Done Done false
32438 9452 Acked Flush 0 Sent Done Done false
32439 9453 Acked Write 5 Sent Done Done false
32440 STATES DS:0 DS:1 DS:2 TOTAL
32441 Sent 301 0 0 301
32442 Done 0 301 301 602
32443 Skipped 0 0 0 0
32444 Error 0 0 0 0
32445 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
32446 Downstairs last five completed: 9152 9151 9150 9149 9148
32447 Upstairs last five completed: 9453 9452 9451 9450 9449
32448 CLIENT: Up:0 ds:301 act:2
324492025-01-29T05:17:55.028ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
324502025-01-29T05:17:55.642ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
324512025-01-29T05:17:56.643ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
324522025-01-29T05:17:57.645ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
32453 ----------------------------------------------------------------
32454 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32455 States: Offline Active Active
32456 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32457 9153 Acked Read 1 Sent Done Done false
32458 9154 Acked Flush 0 Sent Done Done false
32459 9155 Acked Write 6 Sent Done Done false
32460 9156 Acked Read 5 Sent Done Done false
32461 9157 Acked Write 8 Sent Done Done false
32462 9158 Acked Write 8 Sent Done Done false
32463 9159 Acked Read 6 Sent Done Done false
32464 9160 Acked Read 6 Sent Done Done false
32465 9161 Acked Read 6 Sent Done Done false
32466 9162 Acked Flush 0 Sent Done Done false
32467 9163 Acked Write 5 Sent Done Done false
32468 9164 Acked Read 2 Sent Done Done false
32469 9165 Acked Read 7 Sent Done Done false
32470 9166 Acked Flush 0 Sent Done Done false
32471 9167 Acked Read 5 Sent Done Done false
32472 9168 Acked Read 7 Sent Done Done false
32473 9169 Acked Read 7 Sent Done Done false
32474 9170 Acked Read 1 Sent Done Done false
32475 9171 Acked Read 5 Sent Done Done false
32476 9172 Acked Flush 0 Sent Done Done false
32477 9173 Acked Write 1 Sent Done Done false
32478 9174 Acked Read 5 Sent Done Done false
32479 9175 Acked Read 6 Sent Done Done false
32480 9176 Acked Read 1 Sent Done Done false
32481 9177 Acked Write 9 Sent Done Done false
32482 9178 Acked Read 10 Sent Done Done false
32483 9179 Acked Read 2 Sent Done Done false
32484 9180 Acked Flush 0 Sent Done Done false
32485 9181 Acked Read 5 Sent Done Done false
32486 9182 Acked Read 5 Sent Done Done false
32487 9183 Acked Read 1 Sent Done Done false
32488 9184 Acked Write 6 Sent Done Done false
32489 9185 Acked Read 3 Sent Done Done false
32490 9186 Acked Read 1 Sent Done Done false
32491 9187 Acked Read 6 Sent Done Done false
32492 9188 Acked Read 4 Sent Done Done false
32493 9189 Acked Read 4 Sent Done Done false
32494 9190 Acked Write 9 Sent Done Done false
32495 9191 Acked Write 3 Sent Done Done false
32496 9192 Acked Read 6 Sent Done Done false
32497 9193 Acked Write 2 Sent Done Done false
32498 9194 Acked Read 3 Sent Done Done false
32499 9195 Acked Read 4 Sent Done Done false
32500 9196 Acked Read 7 Sent Done Done false
32501 9197 Acked Write 4 Sent Done Done false
32502 9198 Acked Write 5 Sent Done Done false
32503 9199 Acked Write 3 Sent Done Done false
32504 9200 Acked Write 9 Sent Done Done false
32505 9201 Acked Read 6 Sent Done Done false
32506 9202 Acked Read 6 Sent Done Done false
32507 9203 Acked Read 8 Sent Done Done false
32508 9204 Acked Write 7 Sent Done Done false
32509 9205 Acked Read 8 Sent Done Done false
32510 9206 Acked Write 5 Sent Done Done false
32511 9207 Acked Read 10 Sent Done Done false
32512 9208 Acked Read 1 Sent Done Done false
32513 9209 Acked Write 1 Sent Done Done false
32514 9210 Acked Read 9 Sent Done Done false
32515 9211 Acked Read 8 Sent Done Done false
32516 9212 Acked Flush 0 Sent Done Done false
32517 9213 Acked Read 2 Sent Done Done false
32518 9214 Acked Write 5 Sent Done Done false
32519 9215 Acked Write 8 Sent Done Done false
32520 9216 Acked Write 5 Sent Done Done false
32521 9217 Acked Read 5 Sent Done Done false
32522 9218 Acked Write 1 Sent Done Done false
32523 9219 Acked Read 6 Sent Done Done false
32524 9220 Acked Write 2 Sent Done Done false
32525 9221 Acked Read 1 Sent Done Done false
32526 9222 Acked Read 6 Sent Done Done false
32527 9223 Acked Write 5 Sent Done Done false
32528 9224 Acked Read 2 Sent Done Done false
32529 9225 Acked Read 5 Sent Done Done false
32530 9226 Acked Flush 0 Sent Done Done false
32531 9227 Acked Write 7 Sent Done Done false
32532 9228 Acked Write 2 Sent Done Done false
32533 9229 Acked Flush 0 Sent Done Done false
32534 9230 Acked Read 4 Sent Done Done false
32535 9231 Acked Write 10 Sent Done Done false
32536 9232 Acked Write 6 Sent Done Done false
32537 9233 Acked Read 6 Sent Done Done false
32538 9234 Acked Write 9 Sent Done Done false
32539 9235 Acked Flush 0 Sent Done Done false
32540 9236 Acked Read 4 Sent Done Done false
32541 9237 Acked Flush 0 Sent Done Done false
32542 9238 Acked Read 2 Sent Done Done false
32543 9239 Acked Write 7 Sent Done Done false
32544 9240 Acked Write 5 Sent Done Done false
32545 9241 Acked Write 3 Sent Done Done false
32546 9242 Acked Flush 0 Sent Done Done false
32547 9243 Acked Read 1 Sent Done Done false
32548 9244 Acked Read 7 Sent Done Done false
32549 9245 Acked Read 10 Sent Done Done false
32550 9246 Acked Write 4 Sent Done Done false
32551 9247 Acked Read 8 Sent Done Done false
32552 9248 Acked Read 2 Sent Done Done false
32553 9249 Acked Write 7 Sent Done Done false
32554 9250 Acked Read 2 Sent Done Done false
32555 9251 Acked Read 3 Sent Done Done false
32556 9252 Acked Read 3 Sent Done Done false
32557 9253 Acked Read 10 Sent Done Done false
32558 9254 Acked Read 2 Sent Done Done false
32559 9255 Acked Read 2 Sent Done Done false
32560 9256 Acked Read 7 Sent Done Done false
32561 9257 Acked Read 5 Sent Done Done false
32562 9258 Acked Read 7 Sent Done Done false
32563 9259 Acked Write 9 Sent Done Done false
32564 9260 Acked Write 10 Sent Done Done false
32565 9261 Acked Read 2 Sent Done Done false
32566 9262 Acked Write 10 Sent Done Done false
32567 9263 Acked Write 5 Sent Done Done false
32568 9264 Acked Read 10 Sent Done Done false
32569 9265 Acked Write 7 Sent Done Done false
32570 9266 Acked Write 5 Sent Done Done false
32571 9267 Acked Read 8 Sent Done Done false
32572 9268 Acked Write 10 Sent Done Done false
32573 9269 Acked Write 5 Sent Done Done false
32574 9270 Acked Write 8 Sent Done Done false
32575 9271 Acked Read 4 Sent Done Done false
32576 9272 Acked Write 6 Sent Done Done false
32577 9273 Acked Read 9 Sent Done Done false
32578 9274 Acked Write 10 Sent Done Done false
32579 9275 Acked Read 3 Sent Done Done false
32580 9276 Acked Read 4 Sent Done Done false
32581 9277 Acked Write 7 Sent Done Done false
32582 9278 Acked Flush 0 Sent Done Done false
32583 9279 Acked Read 5 Sent Done Done false
32584 9280 Acked Read 7 Sent Done Done false
32585 9281 Acked Read 3 Sent Done Done false
32586 9282 Acked Flush 0 Sent Done Done false
32587 9283 Acked Write 9 Sent Done Done false
32588 9284 Acked Read 6 Sent Done Done false
32589 9285 Acked Write 5 Sent Done Done false
32590 9286 Acked Read 2 Sent Done Done false
32591 9287 Acked Read 6 Sent Done Done false
32592 9288 Acked Read 7 Sent Done Done false
32593 9289 Acked Read 1 Sent Done Done false
32594 9290 Acked Write 1 Sent Done Done false
32595 9291 Acked Read 6 Sent Done Done false
32596 9292 Acked Read 4 Sent Done Done false
32597 9293 Acked Read 3 Sent Done Done false
32598 9294 Acked Write 2 Sent Done Done false
32599 9295 Acked Write 7 Sent Done Done false
32600 9296 Acked Read 8 Sent Done Done false
32601 9297 Acked Write 5 Sent Done Done false
32602 9298 Acked Read 5 Sent Done Done false
32603 9299 Acked Write 10 Sent Done Done false
32604 9300 Acked Read 6 Sent Done Done false
32605 9301 Acked Write 8 Sent Done Done false
32606 9302 Acked Read 7 Sent Done Done false
32607 9303 Acked Read 7 Sent Done Done false
32608 9304 Acked Write 7 Sent Done Done false
32609 9305 Acked Read 2 Sent Done Done false
32610 9306 Acked Write 7 Sent Done Done false
32611 9307 Acked Write 7 Sent Done Done false
32612 9308 Acked Read 2 Sent Done Done false
32613 9309 Acked Read 7 Sent Done Done false
32614 9310 Acked Write 7 Sent Done Done false
32615 9311 Acked Write 5 Sent Done Done false
32616 9312 Acked Write 4 Sent Done Done false
32617 9313 Acked Write 8 Sent Done Done false
32618 9314 Acked Read 6 Sent Done Done false
32619 9315 Acked Write 3 Sent Done Done false
32620 9316 Acked Read 2 Sent Done Done false
32621 9317 Acked Read 8 Sent Done Done false
32622 9318 Acked Read 5 Sent Done Done false
32623 9319 Acked Flush 0 Sent Done Done false
32624 9320 Acked Read 10 Sent Done Done false
32625 9321 Acked Read 3 Sent Done Done false
32626 9322 Acked Flush 0 Sent Done Done false
32627 9323 Acked Read 3 Sent Done Done false
32628 9324 Acked Write 2 Sent Done Done false
32629 9325 Acked Write 8 Sent Done Done false
32630 9326 Acked Read 6 Sent Done Done false
32631 9327 Acked Read 4 Sent Done Done false
32632 9328 Acked Write 5 Sent Done Done false
32633 9329 Acked Write 2 Sent Done Done false
32634 9330 Acked Write 5 Sent Done Done false
32635 9331 Acked Read 9 Sent Done Done false
32636 9332 Acked Write 6 Sent Done Done false
32637 9333 Acked Flush 0 Sent Done Done false
32638 9334 Acked Read 7 Sent Done Done false
32639 9335 Acked Write 8 Sent Done Done false
32640 9336 Acked Write 8 Sent Done Done false
32641 9337 Acked Write 7 Sent Done Done false
32642 9338 Acked Write 6 Sent Done Done false
32643 9339 Acked Read 5 Sent Done Done false
32644 9340 Acked Write 7 Sent Done Done false
32645 9341 Acked Write 8 Sent Done Done false
32646 9342 Acked Read 5 Sent Done Done false
32647 9343 Acked Write 4 Sent Done Done false
32648 9344 Acked Read 1 Sent Done Done false
32649 9345 Acked Read 6 Sent Done Done false
32650 9346 Acked Write 7 Sent Done Done false
32651 9347 Acked Read 10 Sent Done Done false
32652 9348 Acked Write 8 Sent Done Done false
32653 9349 Acked Flush 0 Sent Done Done false
32654 9350 Acked Write 5 Sent Done Done false
32655 9351 Acked Write 6 Sent Done Done false
32656 9352 Acked Read 7 Sent Done Done false
32657 9353 Acked Write 3 Sent Done Done false
32658 9354 Acked Read 10 Sent Done Done false
32659 9355 Acked Write 4 Sent Done Done false
32660 9356 Acked Read 9 Sent Done Done false
32661 9357 Acked Read 1 Sent Done Done false
32662 9358 Acked Flush 0 Sent Done Done false
32663 9359 Acked Write 2 Sent Done Done false
32664 9360 Acked Read 7 Sent Done Done false
32665 9361 Acked Read 5 Sent Done Done false
32666 9362 Acked Read 10 Sent Done Done false
32667 9363 Acked Read 9 Sent Done Done false
32668 9364 Acked Read 8 Sent Done Done false
32669 9365 Acked Write 3 Sent Done Done false
32670 9366 Acked Read 5 Sent Done Done false
32671 9367 Acked Write 6 Sent Done Done false
32672 9368 Acked Write 8 Sent Done Done false
32673 9369 Acked Write 10 Sent Done Done false
32674 9370 Acked Write 3 Sent Done Done false
32675 9371 Acked Flush 0 Sent Done Done false
32676 9372 Acked Read 6 Sent Done Done false
32677 9373 Acked Write 7 Sent Done Done false
32678 9374 Acked Read 3 Sent Done Done false
32679 9375 Acked Read 10 Sent Done Done false
32680 9376 Acked Read 9 Sent Done Done false
32681 9377 Acked Read 3 Sent Done Done false
32682 9378 Acked Read 5 Sent Done Done false
32683 9379 Acked Read 1 Sent Done Done false
32684 9380 Acked Write 8 Sent Done Done false
32685 9381 Acked Read 8 Sent Done Done false
32686 9382 Acked Write 4 Sent Done Done false
32687 9383 Acked Write 8 Sent Done Done false
32688 9384 Acked Flush 0 Sent Done Done false
32689 9385 Acked Write 10 Sent Done Done false
32690 9386 Acked Read 6 Sent Done Done false
32691 9387 Acked Write 3 Sent Done Done false
32692 9388 Acked Read 8 Sent Done Done false
32693 9389 Acked Flush 0 Sent Done Done false
32694 9390 Acked Read 1 Sent Done Done false
32695 9391 Acked Write 1 Sent Done Done false
32696 9392 Acked Read 5 Sent Done Done false
32697 9393 Acked Read 3 Sent Done Done false
32698 9394 Acked Read 10 Sent Done Done false
32699 9395 Acked Read 4 Sent Done Done false
32700 9396 Acked Write 8 Sent Done Done false
32701 9397 Acked Read 7 Sent Done Done false
32702 9398 Acked Write 5 Sent Done Done false
32703 9399 Acked Flush 0 Sent Done Done false
32704 9400 Acked Write 5 Sent Done Done false
32705 9401 Acked Read 1 Sent Done Done false
32706 9402 Acked Read 9 Sent Done Done false
32707 9403 Acked Read 2 Sent Done Done false
32708 9404 Acked Read 8 Sent Done Done false
32709 9405 Acked Read 8 Sent Done Done false
32710 9406 Acked Read 9 Sent Done Done false
32711 9407 Acked Write 10 Sent Done Done false
32712 9408 Acked Flush 0 Sent Done Done false
32713 9409 Acked Write 10 Sent Done Done false
32714 9410 Acked Flush 0 Sent Done Done false
32715 9411 Acked Write 8 Sent Done Done false
32716 9412 Acked Read 9 Sent Done Done false
32717 9413 Acked Read 6 Sent Done Done false
32718 9414 Acked Read 4 Sent Done Done false
32719 9415 Acked Write 3 Sent Done Done false
32720 9416 Acked Write 2 Sent Done Done false
32721 9417 Acked Write 10 Sent Done Done false
32722 9418 Acked Read 5 Sent Done Done false
32723 9419 Acked Write 4 Sent Done Done false
32724 9420 Acked Write 2 Sent Done Done false
32725 9421 Acked Read 3 Sent Done Done false
32726 9422 Acked Write 2 Sent Done Done false
32727 9423 Acked Flush 0 Sent Done Done false
32728 9424 Acked Write 8 Sent Done Done false
32729 9425 Acked Write 10 Sent Done Done false
32730 9426 Acked Write 4 Sent Done Done false
32731 9427 Acked Read 5 Sent Done Done false
32732 9428 Acked Read 6 Sent Done Done false
32733 9429 Acked Read 4 Sent Done Done false
32734 9430 Acked Read 8 Sent Done Done false
32735 9431 Acked Read 6 Sent Done Done false
32736 9432 Acked Read 10 Sent Done Done false
32737 9433 Acked Read 6 Sent Done Done false
32738 9434 Acked Read 3 Sent Done Done false
32739 9435 Acked Write 6 Sent Done Done false
32740 9436 Acked Read 8 Sent Done Done false
32741 9437 Acked Read 4 Sent Done Done false
32742 9438 Acked Read 4 Sent Done Done false
32743 9439 Acked Read 3 Sent Done Done false
32744 9440 Acked Write 8 Sent Done Done false
32745 9441 Acked Read 1 Sent Done Done false
32746 9442 Acked Read 3 Sent Done Done false
32747 9443 Acked Write 7 Sent Done Done false
32748 9444 Acked Read 6 Sent Done Done false
32749 9445 Acked Write 6 Sent Done Done false
32750 9446 Acked Read 1 Sent Done Done false
32751 9447 Acked Flush 0 Sent Done Done false
32752 9448 Acked Write 4 Sent Done Done false
32753 9449 Acked Write 8 Sent Done Done false
32754 9450 Acked Read 1 Sent Done Done false
32755 9451 Acked Flush 0 Sent Done Done false
32756 9452 Acked Flush 0 Sent Done Done false
32757 9453 Acked Write 5 Sent Done Done false
32758 STATES DS:0 DS:1 DS:2 TOTAL
32759 Sent 301 0 0 301
32760 Done 0 301 301 602
32761 Skipped 0 0 0 0
32762 Error 0 0 0 0
32763 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
32764 Downstairs last five completed: 9152 9151 9150 9149 9148
32765 Upstairs last five completed: 9453 9452 9451 9450 9449
32766 CLIENT: Up:0 ds:301 act:2
327672025-01-29T05:17:59.649ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327682025-01-29T05:18:00.280ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327692025-01-29T05:18:00.280ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327702025-01-29T05:18:00.280ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327712025-01-29T05:18:00.280ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327722025-01-29T05:18:00.280ZINFOcrucible: client task is exiting = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
32773 [8810] got start action
32774 [8810] I am going to restart
32775 Starting downstairs at port 8810
32776 Make output file at "/tmp/dsc/downstairs-8810.txt"
32777 [8810][0] reports Starting
32778 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1230)
32779 [8810][0] reports Running
327802025-01-29T05:18:00.781ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
327812025-01-29T05:18:01.652ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
32782 ----------------------------------------------------------------
32783 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32784 States: Offline Active Active
32785 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32786 9153 Acked Read 1 Sent Done Done false
32787 9154 Acked Flush 0 Sent Done Done false
32788 9155 Acked Write 6 Sent Done Done false
32789 9156 Acked Read 5 Sent Done Done false
32790 9157 Acked Write 8 Sent Done Done false
32791 9158 Acked Write 8 Sent Done Done false
32792 9159 Acked Read 6 Sent Done Done false
32793 9160 Acked Read 6 Sent Done Done false
32794 9161 Acked Read 6 Sent Done Done false
32795 9162 Acked Flush 0 Sent Done Done false
32796 9163 Acked Write 5 Sent Done Done false
32797 9164 Acked Read 2 Sent Done Done false
32798 9165 Acked Read 7 Sent Done Done false
32799 9166 Acked Flush 0 Sent Done Done false
32800 9167 Acked Read 5 Sent Done Done false
32801 9168 Acked Read 7 Sent Done Done false
32802 9169 Acked Read 7 Sent Done Done false
32803 9170 Acked Read 1 Sent Done Done false
32804 9171 Acked Read 5 Sent Done Done false
32805 9172 Acked Flush 0 Sent Done Done false
32806 9173 Acked Write 1 Sent Done Done false
32807 9174 Acked Read 5 Sent Done Done false
32808 9175 Acked Read 6 Sent Done Done false
32809 9176 Acked Read 1 Sent Done Done false
32810 9177 Acked Write 9 Sent Done Done false
32811 9178 Acked Read 10 Sent Done Done false
32812 9179 Acked Read 2 Sent Done Done false
32813 9180 Acked Flush 0 Sent Done Done false
32814 9181 Acked Read 5 Sent Done Done false
32815 9182 Acked Read 5 Sent Done Done false
32816 9183 Acked Read 1 Sent Done Done false
32817 9184 Acked Write 6 Sent Done Done false
32818 9185 Acked Read 3 Sent Done Done false
32819 9186 Acked Read 1 Sent Done Done false
32820 9187 Acked Read 6 Sent Done Done false
32821 9188 Acked Read 4 Sent Done Done false
32822 9189 Acked Read 4 Sent Done Done false
32823 9190 Acked Write 9 Sent Done Done false
32824 9191 Acked Write 3 Sent Done Done false
32825 9192 Acked Read 6 Sent Done Done false
32826 9193 Acked Write 2 Sent Done Done false
32827 9194 Acked Read 3 Sent Done Done false
32828 9195 Acked Read 4 Sent Done Done false
32829 9196 Acked Read 7 Sent Done Done false
32830 9197 Acked Write 4 Sent Done Done false
32831 9198 Acked Write 5 Sent Done Done false
32832 9199 Acked Write 3 Sent Done Done false
32833 9200 Acked Write 9 Sent Done Done false
32834 9201 Acked Read 6 Sent Done Done false
32835 9202 Acked Read 6 Sent Done Done false
32836 9203 Acked Read 8 Sent Done Done false
32837 9204 Acked Write 7 Sent Done Done false
32838 9205 Acked Read 8 Sent Done Done false
32839 9206 Acked Write 5 Sent Done Done false
32840 9207 Acked Read 10 Sent Done Done false
32841 9208 Acked Read 1 Sent Done Done false
32842 9209 Acked Write 1 Sent Done Done false
32843 9210 Acked Read 9 Sent Done Done false
32844 9211 Acked Read 8 Sent Done Done false
32845 9212 Acked Flush 0 Sent Done Done false
32846 9213 Acked Read 2 Sent Done Done false
32847 9214 Acked Write 5 Sent Done Done false
32848 9215 Acked Write 8 Sent Done Done false
32849 9216 Acked Write 5 Sent Done Done false
32850 9217 Acked Read 5 Sent Done Done false
32851 9218 Acked Write 1 Sent Done Done false
32852 9219 Acked Read 6 Sent Done Done false
32853 9220 Acked Write 2 Sent Done Done false
32854 9221 Acked Read 1 Sent Done Done false
32855 9222 Acked Read 6 Sent Done Done false
32856 9223 Acked Write 5 Sent Done Done false
32857 9224 Acked Read 2 Sent Done Done false
32858 9225 Acked Read 5 Sent Done Done false
32859 9226 Acked Flush 0 Sent Done Done false
32860 9227 Acked Write 7 Sent Done Done false
32861 9228 Acked Write 2 Sent Done Done false
32862 9229 Acked Flush 0 Sent Done Done false
32863 9230 Acked Read 4 Sent Done Done false
32864 9231 Acked Write 10 Sent Done Done false
32865 9232 Acked Write 6 Sent Done Done false
32866 9233 Acked Read 6 Sent Done Done false
32867 9234 Acked Write 9 Sent Done Done false
32868 9235 Acked Flush 0 Sent Done Done false
32869 9236 Acked Read 4 Sent Done Done false
32870 9237 Acked Flush 0 Sent Done Done false
32871 9238 Acked Read 2 Sent Done Done false
32872 9239 Acked Write 7 Sent Done Done false
32873 9240 Acked Write 5 Sent Done Done false
32874 9241 Acked Write 3 Sent Done Done false
32875 9242 Acked Flush 0 Sent Done Done false
32876 9243 Acked Read 1 Sent Done Done false
32877 9244 Acked Read 7 Sent Done Done false
32878 9245 Acked Read 10 Sent Done Done false
32879 9246 Acked Write 4 Sent Done Done false
32880 9247 Acked Read 8 Sent Done Done false
32881 9248 Acked Read 2 Sent Done Done false
32882 9249 Acked Write 7 Sent Done Done false
32883 9250 Acked Read 2 Sent Done Done false
32884 9251 Acked Read 3 Sent Done Done false
32885 9252 Acked Read 3 Sent Done Done false
32886 9253 Acked Read 10 Sent Done Done false
32887 9254 Acked Read 2 Sent Done Done false
32888 9255 Acked Read 2 Sent Done Done false
32889 9256 Acked Read 7 Sent Done Done false
32890 9257 Acked Read 5 Sent Done Done false
32891 9258 Acked Read 7 Sent Done Done false
32892 9259 Acked Write 9 Sent Done Done false
32893 9260 Acked Write 10 Sent Done Done false
32894 9261 Acked Read 2 Sent Done Done false
32895 9262 Acked Write 10 Sent Done Done false
32896 9263 Acked Write 5 Sent Done Done false
32897 9264 Acked Read 10 Sent Done Done false
32898 9265 Acked Write 7 Sent Done Done false
32899 9266 Acked Write 5 Sent Done Done false
32900 9267 Acked Read 8 Sent Done Done false
32901 9268 Acked Write 10 Sent Done Done false
32902 9269 Acked Write 5 Sent Done Done false
32903 9270 Acked Write 8 Sent Done Done false
32904 9271 Acked Read 4 Sent Done Done false
32905 9272 Acked Write 6 Sent Done Done false
32906 9273 Acked Read 9 Sent Done Done false
32907 9274 Acked Write 10 Sent Done Done false
32908 9275 Acked Read 3 Sent Done Done false
32909 9276 Acked Read 4 Sent Done Done false
32910 9277 Acked Write 7 Sent Done Done false
32911 9278 Acked Flush 0 Sent Done Done false
32912 9279 Acked Read 5 Sent Done Done false
32913 9280 Acked Read 7 Sent Done Done false
32914 9281 Acked Read 3 Sent Done Done false
32915 9282 Acked Flush 0 Sent Done Done false
32916 9283 Acked Write 9 Sent Done Done false
32917 9284 Acked Read 6 Sent Done Done false
32918 9285 Acked Write 5 Sent Done Done false
32919 9286 Acked Read 2 Sent Done Done false
32920 9287 Acked Read 6 Sent Done Done false
32921 9288 Acked Read 7 Sent Done Done false
32922 9289 Acked Read 1 Sent Done Done false
32923 9290 Acked Write 1 Sent Done Done false
32924 9291 Acked Read 6 Sent Done Done false
32925 9292 Acked Read 4 Sent Done Done false
32926 9293 Acked Read 3 Sent Done Done false
32927 9294 Acked Write 2 Sent Done Done false
32928 9295 Acked Write 7 Sent Done Done false
32929 9296 Acked Read 8 Sent Done Done false
32930 9297 Acked Write 5 Sent Done Done false
32931 9298 Acked Read 5 Sent Done Done false
32932 9299 Acked Write 10 Sent Done Done false
32933 9300 Acked Read 6 Sent Done Done false
32934 9301 Acked Write 8 Sent Done Done false
32935 9302 Acked Read 7 Sent Done Done false
32936 9303 Acked Read 7 Sent Done Done false
32937 9304 Acked Write 7 Sent Done Done false
32938 9305 Acked Read 2 Sent Done Done false
32939 9306 Acked Write 7 Sent Done Done false
32940 9307 Acked Write 7 Sent Done Done false
32941 9308 Acked Read 2 Sent Done Done false
32942 9309 Acked Read 7 Sent Done Done false
32943 9310 Acked Write 7 Sent Done Done false
32944 9311 Acked Write 5 Sent Done Done false
32945 9312 Acked Write 4 Sent Done Done false
32946 9313 Acked Write 8 Sent Done Done false
32947 9314 Acked Read 6 Sent Done Done false
32948 9315 Acked Write 3 Sent Done Done false
32949 9316 Acked Read 2 Sent Done Done false
32950 9317 Acked Read 8 Sent Done Done false
32951 9318 Acked Read 5 Sent Done Done false
32952 9319 Acked Flush 0 Sent Done Done false
32953 9320 Acked Read 10 Sent Done Done false
32954 9321 Acked Read 3 Sent Done Done false
32955 9322 Acked Flush 0 Sent Done Done false
32956 9323 Acked Read 3 Sent Done Done false
32957 9324 Acked Write 2 Sent Done Done false
32958 9325 Acked Write 8 Sent Done Done false
32959 9326 Acked Read 6 Sent Done Done false
32960 9327 Acked Read 4 Sent Done Done false
32961 9328 Acked Write 5 Sent Done Done false
32962 9329 Acked Write 2 Sent Done Done false
32963 9330 Acked Write 5 Sent Done Done false
32964 9331 Acked Read 9 Sent Done Done false
32965 9332 Acked Write 6 Sent Done Done false
32966 9333 Acked Flush 0 Sent Done Done false
32967 9334 Acked Read 7 Sent Done Done false
32968 9335 Acked Write 8 Sent Done Done false
32969 9336 Acked Write 8 Sent Done Done false
32970 9337 Acked Write 7 Sent Done Done false
32971 9338 Acked Write 6 Sent Done Done false
32972 9339 Acked Read 5 Sent Done Done false
32973 9340 Acked Write 7 Sent Done Done false
32974 9341 Acked Write 8 Sent Done Done false
32975 9342 Acked Read 5 Sent Done Done false
32976 9343 Acked Write 4 Sent Done Done false
32977 9344 Acked Read 1 Sent Done Done false
32978 9345 Acked Read 6 Sent Done Done false
32979 9346 Acked Write 7 Sent Done Done false
32980 9347 Acked Read 10 Sent Done Done false
32981 9348 Acked Write 8 Sent Done Done false
32982 9349 Acked Flush 0 Sent Done Done false
32983 9350 Acked Write 5 Sent Done Done false
32984 9351 Acked Write 6 Sent Done Done false
32985 9352 Acked Read 7 Sent Done Done false
32986 9353 Acked Write 3 Sent Done Done false
32987 9354 Acked Read 10 Sent Done Done false
32988 9355 Acked Write 4 Sent Done Done false
32989 9356 Acked Read 9 Sent Done Done false
32990 9357 Acked Read 1 Sent Done Done false
32991 9358 Acked Flush 0 Sent Done Done false
32992 9359 Acked Write 2 Sent Done Done false
32993 9360 Acked Read 7 Sent Done Done false
32994 9361 Acked Read 5 Sent Done Done false
32995 9362 Acked Read 10 Sent Done Done false
32996 9363 Acked Read 9 Sent Done Done false
32997 9364 Acked Read 8 Sent Done Done false
32998 9365 Acked Write 3 Sent Done Done false
32999 9366 Acked Read 5 Sent Done Done false
33000 9367 Acked Write 6 Sent Done Done false
33001 9368 Acked Write 8 Sent Done Done false
33002 9369 Acked Write 10 Sent Done Done false
33003 9370 Acked Write 3 Sent Done Done false
33004 9371 Acked Flush 0 Sent Done Done false
33005 9372 Acked Read 6 Sent Done Done false
33006 9373 Acked Write 7 Sent Done Done false
33007 9374 Acked Read 3 Sent Done Done false
33008 9375 Acked Read 10 Sent Done Done false
33009 9376 Acked Read 9 Sent Done Done false
33010 9377 Acked Read 3 Sent Done Done false
33011 9378 Acked Read 5 Sent Done Done false
33012 9379 Acked Read 1 Sent Done Done false
33013 9380 Acked Write 8 Sent Done Done false
33014 9381 Acked Read 8 Sent Done Done false
33015 9382 Acked Write 4 Sent Done Done false
33016 9383 Acked Write 8 Sent Done Done false
33017 9384 Acked Flush 0 Sent Done Done false
33018 9385 Acked Write 10 Sent Done Done false
33019 9386 Acked Read 6 Sent Done Done false
33020 9387 Acked Write 3 Sent Done Done false
33021 9388 Acked Read 8 Sent Done Done false
33022 9389 Acked Flush 0 Sent Done Done false
33023 9390 Acked Read 1 Sent Done Done false
33024 9391 Acked Write 1 Sent Done Done false
33025 9392 Acked Read 5 Sent Done Done false
33026 9393 Acked Read 3 Sent Done Done false
33027 9394 Acked Read 10 Sent Done Done false
33028 9395 Acked Read 4 Sent Done Done false
33029 9396 Acked Write 8 Sent Done Done false
33030 9397 Acked Read 7 Sent Done Done false
33031 9398 Acked Write 5 Sent Done Done false
33032 9399 Acked Flush 0 Sent Done Done false
33033 9400 Acked Write 5 Sent Done Done false
33034 9401 Acked Read 1 Sent Done Done false
33035 9402 Acked Read 9 Sent Done Done false
33036 9403 Acked Read 2 Sent Done Done false
33037 9404 Acked Read 8 Sent Done Done false
33038 9405 Acked Read 8 Sent Done Done false
33039 9406 Acked Read 9 Sent Done Done false
33040 9407 Acked Write 10 Sent Done Done false
33041 9408 Acked Flush 0 Sent Done Done false
33042 9409 Acked Write 10 Sent Done Done false
33043 9410 Acked Flush 0 Sent Done Done false
33044 9411 Acked Write 8 Sent Done Done false
33045 9412 Acked Read 9 Sent Done Done false
33046 9413 Acked Read 6 Sent Done Done false
33047 9414 Acked Read 4 Sent Done Done false
33048 9415 Acked Write 3 Sent Done Done false
33049 9416 Acked Write 2 Sent Done Done false
33050 9417 Acked Write 10 Sent Done Done false
33051 9418 Acked Read 5 Sent Done Done false
33052 9419 Acked Write 4 Sent Done Done false
33053 9420 Acked Write 2 Sent Done Done false
33054 9421 Acked Read 3 Sent Done Done false
33055 9422 Acked Write 2 Sent Done Done false
33056 9423 Acked Flush 0 Sent Done Done false
33057 9424 Acked Write 8 Sent Done Done false
33058 9425 Acked Write 10 Sent Done Done false
33059 9426 Acked Write 4 Sent Done Done false
33060 9427 Acked Read 5 Sent Done Done false
33061 9428 Acked Read 6 Sent Done Done false
33062 9429 Acked Read 4 Sent Done Done false
33063 9430 Acked Read 8 Sent Done Done false
33064 9431 Acked Read 6 Sent Done Done false
33065 9432 Acked Read 10 Sent Done Done false
33066 9433 Acked Read 6 Sent Done Done false
33067 9434 Acked Read 3 Sent Done Done false
33068 9435 Acked Write 6 Sent Done Done false
33069 9436 Acked Read 8 Sent Done Done false
33070 9437 Acked Read 4 Sent Done Done false
33071 9438 Acked Read 4 Sent Done Done false
33072 9439 Acked Read 3 Sent Done Done false
33073 9440 Acked Write 8 Sent Done Done false
33074 9441 Acked Read 1 Sent Done Done false
33075 9442 Acked Read 3 Sent Done Done false
33076 9443 Acked Write 7 Sent Done Done false
33077 9444 Acked Read 6 Sent Done Done false
33078 9445 Acked Write 6 Sent Done Done false
33079 9446 Acked Read 1 Sent Done Done false
33080 9447 Acked Flush 0 Sent Done Done false
33081 9448 Acked Write 4 Sent Done Done false
33082 9449 Acked Write 8 Sent Done Done false
33083 9450 Acked Read 1 Sent Done Done false
33084 9451 Acked Flush 0 Sent Done Done false
33085 9452 Acked Flush 0 Sent Done Done false
33086 9453 Acked Write 5 Sent Done Done false
33087 STATES DS:0 DS:1 DS:2 TOTAL
33088 Sent 301 0 0 301
33089 Done 0 301 301 602
33090 Skipped 0 0 0 0
33091 Error 0 0 0 0
33092 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
33093 Downstairs last five completed: 9152 9151 9150 9149 9148
33094 Upstairs last five completed: 9453 9452 9451 9450 9449
33095 CLIENT: Up:0 ds:301 act:2
330962025-01-29T05:18:03.031ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
330972025-01-29T05:18:03.655ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
330982025-01-29T05:18:04.657ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
330992025-01-29T05:18:05.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
33100 ----------------------------------------------------------------
33101 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33102 States: Offline Active Active
33103 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33104 9153 Acked Read 1 Sent Done Done false
33105 9154 Acked Flush 0 Sent Done Done false
33106 9155 Acked Write 6 Sent Done Done false
33107 9156 Acked Read 5 Sent Done Done false
33108 9157 Acked Write 8 Sent Done Done false
33109 9158 Acked Write 8 Sent Done Done false
33110 9159 Acked Read 6 Sent Done Done false
33111 9160 Acked Read 6 Sent Done Done false
33112 9161 Acked Read 6 Sent Done Done false
33113 9162 Acked Flush 0 Sent Done Done false
33114 9163 Acked Write 5 Sent Done Done false
33115 9164 Acked Read 2 Sent Done Done false
33116 9165 Acked Read 7 Sent Done Done false
33117 9166 Acked Flush 0 Sent Done Done false
33118 9167 Acked Read 5 Sent Done Done false
33119 9168 Acked Read 7 Sent Done Done false
33120 9169 Acked Read 7 Sent Done Done false
33121 9170 Acked Read 1 Sent Done Done false
33122 9171 Acked Read 5 Sent Done Done false
33123 9172 Acked Flush 0 Sent Done Done false
33124 9173 Acked Write 1 Sent Done Done false
33125 9174 Acked Read 5 Sent Done Done false
33126 9175 Acked Read 6 Sent Done Done false
33127 9176 Acked Read 1 Sent Done Done false
33128 9177 Acked Write 9 Sent Done Done false
33129 9178 Acked Read 10 Sent Done Done false
33130 9179 Acked Read 2 Sent Done Done false
33131 9180 Acked Flush 0 Sent Done Done false
33132 9181 Acked Read 5 Sent Done Done false
33133 9182 Acked Read 5 Sent Done Done false
33134 9183 Acked Read 1 Sent Done Done false
33135 9184 Acked Write 6 Sent Done Done false
33136 9185 Acked Read 3 Sent Done Done false
33137 9186 Acked Read 1 Sent Done Done false
33138 9187 Acked Read 6 Sent Done Done false
33139 9188 Acked Read 4 Sent Done Done false
33140 9189 Acked Read 4 Sent Done Done false
33141 9190 Acked Write 9 Sent Done Done false
33142 9191 Acked Write 3 Sent Done Done false
33143 9192 Acked Read 6 Sent Done Done false
33144 9193 Acked Write 2 Sent Done Done false
33145 9194 Acked Read 3 Sent Done Done false
33146 9195 Acked Read 4 Sent Done Done false
33147 9196 Acked Read 7 Sent Done Done false
33148 9197 Acked Write 4 Sent Done Done false
33149 9198 Acked Write 5 Sent Done Done false
33150 9199 Acked Write 3 Sent Done Done false
33151 9200 Acked Write 9 Sent Done Done false
33152 9201 Acked Read 6 Sent Done Done false
33153 9202 Acked Read 6 Sent Done Done false
33154 9203 Acked Read 8 Sent Done Done false
33155 9204 Acked Write 7 Sent Done Done false
33156 9205 Acked Read 8 Sent Done Done false
33157 9206 Acked Write 5 Sent Done Done false
33158 9207 Acked Read 10 Sent Done Done false
33159 9208 Acked Read 1 Sent Done Done false
33160 9209 Acked Write 1 Sent Done Done false
33161 9210 Acked Read 9 Sent Done Done false
33162 9211 Acked Read 8 Sent Done Done false
33163 9212 Acked Flush 0 Sent Done Done false
33164 9213 Acked Read 2 Sent Done Done false
33165 9214 Acked Write 5 Sent Done Done false
33166 9215 Acked Write 8 Sent Done Done false
33167 9216 Acked Write 5 Sent Done Done false
33168 9217 Acked Read 5 Sent Done Done false
33169 9218 Acked Write 1 Sent Done Done false
33170 9219 Acked Read 6 Sent Done Done false
33171 9220 Acked Write 2 Sent Done Done false
33172 9221 Acked Read 1 Sent Done Done false
33173 9222 Acked Read 6 Sent Done Done false
33174 9223 Acked Write 5 Sent Done Done false
33175 9224 Acked Read 2 Sent Done Done false
33176 9225 Acked Read 5 Sent Done Done false
33177 9226 Acked Flush 0 Sent Done Done false
33178 9227 Acked Write 7 Sent Done Done false
33179 9228 Acked Write 2 Sent Done Done false
33180 9229 Acked Flush 0 Sent Done Done false
33181 9230 Acked Read 4 Sent Done Done false
33182 9231 Acked Write 10 Sent Done Done false
33183 9232 Acked Write 6 Sent Done Done false
33184 9233 Acked Read 6 Sent Done Done false
33185 9234 Acked Write 9 Sent Done Done false
33186 9235 Acked Flush 0 Sent Done Done false
33187 9236 Acked Read 4 Sent Done Done false
33188 9237 Acked Flush 0 Sent Done Done false
33189 9238 Acked Read 2 Sent Done Done false
33190 9239 Acked Write 7 Sent Done Done false
33191 9240 Acked Write 5 Sent Done Done false
33192 9241 Acked Write 3 Sent Done Done false
33193 9242 Acked Flush 0 Sent Done Done false
33194 9243 Acked Read 1 Sent Done Done false
33195 9244 Acked Read 7 Sent Done Done false
33196 9245 Acked Read 10 Sent Done Done false
33197 9246 Acked Write 4 Sent Done Done false
33198 9247 Acked Read 8 Sent Done Done false
33199 9248 Acked Read 2 Sent Done Done false
33200 9249 Acked Write 7 Sent Done Done false
33201 9250 Acked Read 2 Sent Done Done false
33202 9251 Acked Read 3 Sent Done Done false
33203 9252 Acked Read 3 Sent Done Done false
33204 9253 Acked Read 10 Sent Done Done false
33205 9254 Acked Read 2 Sent Done Done false
33206 9255 Acked Read 2 Sent Done Done false
33207 9256 Acked Read 7 Sent Done Done false
33208 9257 Acked Read 5 Sent Done Done false
33209 9258 Acked Read 7 Sent Done Done false
33210 9259 Acked Write 9 Sent Done Done false
33211 9260 Acked Write 10 Sent Done Done false
33212 9261 Acked Read 2 Sent Done Done false
33213 9262 Acked Write 10 Sent Done Done false
33214 9263 Acked Write 5 Sent Done Done false
33215 9264 Acked Read 10 Sent Done Done false
33216 9265 Acked Write 7 Sent Done Done false
33217 9266 Acked Write 5 Sent Done Done false
33218 9267 Acked Read 8 Sent Done Done false
33219 9268 Acked Write 10 Sent Done Done false
33220 9269 Acked Write 5 Sent Done Done false
33221 9270 Acked Write 8 Sent Done Done false
33222 9271 Acked Read 4 Sent Done Done false
33223 9272 Acked Write 6 Sent Done Done false
33224 9273 Acked Read 9 Sent Done Done false
33225 9274 Acked Write 10 Sent Done Done false
33226 9275 Acked Read 3 Sent Done Done false
33227 9276 Acked Read 4 Sent Done Done false
33228 9277 Acked Write 7 Sent Done Done false
33229 9278 Acked Flush 0 Sent Done Done false
33230 9279 Acked Read 5 Sent Done Done false
33231 9280 Acked Read 7 Sent Done Done false
33232 9281 Acked Read 3 Sent Done Done false
33233 9282 Acked Flush 0 Sent Done Done false
33234 9283 Acked Write 9 Sent Done Done false
33235 9284 Acked Read 6 Sent Done Done false
33236 9285 Acked Write 5 Sent Done Done false
33237 9286 Acked Read 2 Sent Done Done false
33238 9287 Acked Read 6 Sent Done Done false
33239 9288 Acked Read 7 Sent Done Done false
33240 9289 Acked Read 1 Sent Done Done false
33241 9290 Acked Write 1 Sent Done Done false
33242 9291 Acked Read 6 Sent Done Done false
33243 9292 Acked Read 4 Sent Done Done false
33244 9293 Acked Read 3 Sent Done Done false
33245 9294 Acked Write 2 Sent Done Done false
33246 9295 Acked Write 7 Sent Done Done false
33247 9296 Acked Read 8 Sent Done Done false
33248 9297 Acked Write 5 Sent Done Done false
33249 9298 Acked Read 5 Sent Done Done false
33250 9299 Acked Write 10 Sent Done Done false
33251 9300 Acked Read 6 Sent Done Done false
33252 9301 Acked Write 8 Sent Done Done false
33253 9302 Acked Read 7 Sent Done Done false
33254 9303 Acked Read 7 Sent Done Done false
33255 9304 Acked Write 7 Sent Done Done false
33256 9305 Acked Read 2 Sent Done Done false
33257 9306 Acked Write 7 Sent Done Done false
33258 9307 Acked Write 7 Sent Done Done false
33259 9308 Acked Read 2 Sent Done Done false
33260 9309 Acked Read 7 Sent Done Done false
33261 9310 Acked Write 7 Sent Done Done false
33262 9311 Acked Write 5 Sent Done Done false
33263 9312 Acked Write 4 Sent Done Done false
33264 9313 Acked Write 8 Sent Done Done false
33265 9314 Acked Read 6 Sent Done Done false
33266 9315 Acked Write 3 Sent Done Done false
33267 9316 Acked Read 2 Sent Done Done false
33268 9317 Acked Read 8 Sent Done Done false
33269 9318 Acked Read 5 Sent Done Done false
33270 9319 Acked Flush 0 Sent Done Done false
33271 9320 Acked Read 10 Sent Done Done false
33272 9321 Acked Read 3 Sent Done Done false
33273 9322 Acked Flush 0 Sent Done Done false
33274 9323 Acked Read 3 Sent Done Done false
33275 9324 Acked Write 2 Sent Done Done false
33276 9325 Acked Write 8 Sent Done Done false
33277 9326 Acked Read 6 Sent Done Done false
33278 9327 Acked Read 4 Sent Done Done false
33279 9328 Acked Write 5 Sent Done Done false
33280 9329 Acked Write 2 Sent Done Done false
33281 9330 Acked Write 5 Sent Done Done false
33282 9331 Acked Read 9 Sent Done Done false
33283 9332 Acked Write 6 Sent Done Done false
33284 9333 Acked Flush 0 Sent Done Done false
33285 9334 Acked Read 7 Sent Done Done false
33286 9335 Acked Write 8 Sent Done Done false
33287 9336 Acked Write 8 Sent Done Done false
33288 9337 Acked Write 7 Sent Done Done false
33289 9338 Acked Write 6 Sent Done Done false
33290 9339 Acked Read 5 Sent Done Done false
33291 9340 Acked Write 7 Sent Done Done false
33292 9341 Acked Write 8 Sent Done Done false
33293 9342 Acked Read 5 Sent Done Done false
33294 9343 Acked Write 4 Sent Done Done false
33295 9344 Acked Read 1 Sent Done Done false
33296 9345 Acked Read 6 Sent Done Done false
33297 9346 Acked Write 7 Sent Done Done false
33298 9347 Acked Read 10 Sent Done Done false
33299 9348 Acked Write 8 Sent Done Done false
33300 9349 Acked Flush 0 Sent Done Done false
33301 9350 Acked Write 5 Sent Done Done false
33302 9351 Acked Write 6 Sent Done Done false
33303 9352 Acked Read 7 Sent Done Done false
33304 9353 Acked Write 3 Sent Done Done false
33305 9354 Acked Read 10 Sent Done Done false
33306 9355 Acked Write 4 Sent Done Done false
33307 9356 Acked Read 9 Sent Done Done false
33308 9357 Acked Read 1 Sent Done Done false
33309 9358 Acked Flush 0 Sent Done Done false
33310 9359 Acked Write 2 Sent Done Done false
33311 9360 Acked Read 7 Sent Done Done false
33312 9361 Acked Read 5 Sent Done Done false
33313 9362 Acked Read 10 Sent Done Done false
33314 9363 Acked Read 9 Sent Done Done false
33315 9364 Acked Read 8 Sent Done Done false
33316 9365 Acked Write 3 Sent Done Done false
33317 9366 Acked Read 5 Sent Done Done false
33318 9367 Acked Write 6 Sent Done Done false
33319 9368 Acked Write 8 Sent Done Done false
33320 9369 Acked Write 10 Sent Done Done false
33321 9370 Acked Write 3 Sent Done Done false
33322 9371 Acked Flush 0 Sent Done Done false
33323 9372 Acked Read 6 Sent Done Done false
33324 9373 Acked Write 7 Sent Done Done false
33325 9374 Acked Read 3 Sent Done Done false
33326 9375 Acked Read 10 Sent Done Done false
33327 9376 Acked Read 9 Sent Done Done false
33328 9377 Acked Read 3 Sent Done Done false
33329 9378 Acked Read 5 Sent Done Done false
33330 9379 Acked Read 1 Sent Done Done false
33331 9380 Acked Write 8 Sent Done Done false
33332 9381 Acked Read 8 Sent Done Done false
33333 9382 Acked Write 4 Sent Done Done false
33334 9383 Acked Write 8 Sent Done Done false
33335 9384 Acked Flush 0 Sent Done Done false
33336 9385 Acked Write 10 Sent Done Done false
33337 9386 Acked Read 6 Sent Done Done false
33338 9387 Acked Write 3 Sent Done Done false
33339 9388 Acked Read 8 Sent Done Done false
33340 9389 Acked Flush 0 Sent Done Done false
33341 9390 Acked Read 1 Sent Done Done false
33342 9391 Acked Write 1 Sent Done Done false
33343 9392 Acked Read 5 Sent Done Done false
33344 9393 Acked Read 3 Sent Done Done false
33345 9394 Acked Read 10 Sent Done Done false
33346 9395 Acked Read 4 Sent Done Done false
33347 9396 Acked Write 8 Sent Done Done false
33348 9397 Acked Read 7 Sent Done Done false
33349 9398 Acked Write 5 Sent Done Done false
33350 9399 Acked Flush 0 Sent Done Done false
33351 9400 Acked Write 5 Sent Done Done false
33352 9401 Acked Read 1 Sent Done Done false
33353 9402 Acked Read 9 Sent Done Done false
33354 9403 Acked Read 2 Sent Done Done false
33355 9404 Acked Read 8 Sent Done Done false
33356 9405 Acked Read 8 Sent Done Done false
33357 9406 Acked Read 9 Sent Done Done false
33358 9407 Acked Write 10 Sent Done Done false
33359 9408 Acked Flush 0 Sent Done Done false
33360 9409 Acked Write 10 Sent Done Done false
33361 9410 Acked Flush 0 Sent Done Done false
33362 9411 Acked Write 8 Sent Done Done false
33363 9412 Acked Read 9 Sent Done Done false
33364 9413 Acked Read 6 Sent Done Done false
33365 9414 Acked Read 4 Sent Done Done false
33366 9415 Acked Write 3 Sent Done Done false
33367 9416 Acked Write 2 Sent Done Done false
33368 9417 Acked Write 10 Sent Done Done false
33369 9418 Acked Read 5 Sent Done Done false
33370 9419 Acked Write 4 Sent Done Done false
33371 9420 Acked Write 2 Sent Done Done false
33372 9421 Acked Read 3 Sent Done Done false
33373 9422 Acked Write 2 Sent Done Done false
33374 9423 Acked Flush 0 Sent Done Done false
33375 9424 Acked Write 8 Sent Done Done false
33376 9425 Acked Write 10 Sent Done Done false
33377 9426 Acked Write 4 Sent Done Done false
33378 9427 Acked Read 5 Sent Done Done false
33379 9428 Acked Read 6 Sent Done Done false
33380 9429 Acked Read 4 Sent Done Done false
33381 9430 Acked Read 8 Sent Done Done false
33382 9431 Acked Read 6 Sent Done Done false
33383 9432 Acked Read 10 Sent Done Done false
33384 9433 Acked Read 6 Sent Done Done false
33385 9434 Acked Read 3 Sent Done Done false
33386 9435 Acked Write 6 Sent Done Done false
33387 9436 Acked Read 8 Sent Done Done false
33388 9437 Acked Read 4 Sent Done Done false
33389 9438 Acked Read 4 Sent Done Done false
33390 9439 Acked Read 3 Sent Done Done false
33391 9440 Acked Write 8 Sent Done Done false
33392 9441 Acked Read 1 Sent Done Done false
33393 9442 Acked Read 3 Sent Done Done false
33394 9443 Acked Write 7 Sent Done Done false
33395 9444 Acked Read 6 Sent Done Done false
33396 9445 Acked Write 6 Sent Done Done false
33397 9446 Acked Read 1 Sent Done Done false
33398 9447 Acked Flush 0 Sent Done Done false
33399 9448 Acked Write 4 Sent Done Done false
33400 9449 Acked Write 8 Sent Done Done false
33401 9450 Acked Read 1 Sent Done Done false
33402 9451 Acked Flush 0 Sent Done Done false
33403 9452 Acked Flush 0 Sent Done Done false
33404 9453 Acked Write 5 Sent Done Done false
33405 STATES DS:0 DS:1 DS:2 TOTAL
33406 Sent 301 0 0 301
33407 Done 0 301 301 602
33408 Skipped 0 0 0 0
33409 Error 0 0 0 0
33410 Last Flush: Some(JobId(9152)) Some(JobId(9452)) Some(JobId(9452))
33411 Downstairs last five completed: 9152 9151 9150 9149 9148
33412 Upstairs last five completed: 9453 9452 9451 9450 9449
33413 CLIENT: Up:0 ds:301 act:2
334142025-01-29T05:18:07.034ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334152025-01-29T05:18:07.662ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334162025-01-29T05:18:08.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334172025-01-29T05:18:09.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334182025-01-29T05:18:10.281ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334192025-01-29T05:18:10.281ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39613) = io task client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334202025-01-29T05:18:10.282ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334212025-01-29T05:18:10.282ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334222025-01-29T05:18:10.282ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334232025-01-29T05:18:10.282ZINFOcrucible: send last flush ID to this DS: Some(JobId(9152)) = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334242025-01-29T05:18:10.282ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334252025-01-29T05:18:10.282ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9152)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334262025-01-29T05:18:10.283ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9152)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
33427 ----------------------------------------------------------------
33428 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:1
33429 States: Active Active Active
33430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33431 9453 Acked Write 5 Done Done Done true
33432 STATES DS:0 DS:1 DS:2 TOTAL
33433 Sent 0 0 0 0
33434 Done 1 1 1 3
33435 Skipped 0 0 0 0
33436 Error 0 0 0 0
33437 Last Flush: Some(JobId(9452)) Some(JobId(9452)) Some(JobId(9452))
33438 Downstairs last five completed: 9452 9451 9450 9449 9448
33439 Upstairs last five completed: 9453 9452 9451 9450 9449
33440 CLIENT: Up:0 ds:1 act:3
33441 ----------------------------------------------------------------
33442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33443 Downstairs last five completed: 9454 9453 9452 9451 9450
33444 Upstairs last five completed: 9454 9453 9452 9451 9450
33445 CLIENT: Up:0 ds:0 act:3
33446 Replay: All jobs finished
33447 Main task has work to do, go find it
33448 got dsc Stop(2)
33449 stop 2
33450 Jan 29 05:18:14.537 INFO request completed, latency_us: 411, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 56502495-f415-42a2-bc37-3897faa95f52[8830] Got stop action so:false kr:false
33451 , remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
33452 Jan 29 05:18:14.537 INFO request completed, latency_us: 122, response_code: 200, uri: /state/cid/2, method: GET, req_id: e04dddcf-a8b8-4dd7-a91a-d44fcbcb24df, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
334532025-01-29T05:18:14.538ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334542025-01-29T05:18:14.538ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334552025-01-29T05:18:14.538ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
334562025-01-29T05:18:14.538ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
33457 [8830] Exited with: ExitStatus(unix_wait_status(9))
33458 [8830][2] reports Exit
33459 Jan 29 05:18:18.540 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5749a0f2-9107-472a-b45c-90be3584b313, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
33460 001/300 Read block 265 len 24576
33461 002/300 Write block 4963 len 20480 data: 8 9 8 7 8
33462 003/300 Write block 2521 len 36864 data: 8 9 10 7 6 7 6 7 6
33463 004/300 Write block 4772 len 24576 data: 7 7 7 7 7 9
33464 005/300 Write block 3716 len 4096 data: 6
33465 006/300 Read block 2278 len 36864
33466 007/300 Write block 35 len 8192 data: 6 7
33467 008/300 Read block 2318 len 40960
33468 009/300 Write block 2311 len 12288 data: 7 8 7
33469 010/300 Write block 1810 len 12288 data: 8 7 7
33470 011/300 Read block 2462 len 16384
33471 012/300 Write block 2186 len 16384 data: 5 6 4 4
33472 013/300 Flush
33473 014/300 Read block 4641 len 12288
33474 015/300 Flush
33475 016/300 Write block 3794 len 28672 data: 5 4 4 5 6 4 8
33476 017/300 Read block 1529 len 32768
33477 018/300 Flush
33478 019/300 Write block 619 len 24576 data: 5 4 4 4 4 4
33479 020/300 Read block 83 len 8192
33480 021/300 Write block 452 len 32768 data: 4 5 7 7 7 7 6 6
33481 022/300 Write block 2556 len 36864 data: 6 7 6 7 6 5 5 6 3
33482 023/300 Read block 1025 len 40960
33483 024/300 Write block 4253 len 40960 data: 6 5 5 4 4 5 4 4 3 3
33484 025/300 Write block 4056 len 8192 data: 6 4
33485 026/300 Write block 1695 len 20480 data: 6 7 7 7 6
33486 027/300 Write block 2166 len 16384 data: 5 4 4 5
33487 028/300 Write block 2779 len 16384 data: 5 7 6 6
33488 029/300 Read block 3171 len 32768
33489 030/300 Read block 2679 len 20480
33490 031/300 Write block 1240 len 12288 data: 3 5 6
33491 032/300 Write block 2002 len 20480 data: 8 8 7 10 10
33492 033/300 Read block 1164 len 16384
33493 034/300 Read block 4727 len 40960
33494 035/300 Flush
33495 036/300 Write block 2457 len 20480 data: 9 8 5 7 5
33496 037/300 Read block 2540 len 16384
33497 038/300 Read block 3855 len 24576
33498 039/300 Read block 1549 len 28672
33499 040/300 Write block 3654 len 20480 data: 4 5 6 6 6
33500 041/300 Read block 1181 len 24576
33501 042/300 Write block 1236 len 4096 data: 4
33502 043/300 Read block 818 len 40960
33503 044/300 Write block 2039 len 24576 data: 6 5 5 4 4 4
33504 045/300 Read block 1922 len 32768
33505 046/300 Read block 4737 len 28672
33506 047/300 Read block 1672 len 32768
33507 048/300 Write block 2347 len 24576 data: 7 6 5 6 6 6
33508 049/300 Read block 2680 len 20480
33509 050/300 Write block 4672 len 20480 data: 3 2 3 4 4
33510 051/300 Read block 4806 len 8192
33511 052/300 Read block 3218 len 4096
33512 053/300 Read block 2635 len 36864
33513 054/300 Write block 163 len 28672 data: 4 5 5 5 5 5 7
33514 055/300 Write block 4127 len 20480 data: 5 5 5 5 4
33515 056/300 Write block 3289 len 4096 data: 7
33516 057/300 Read block 3130 len 24576
33517 058/300 Read block 2678 len 24576
33518 059/300 Read block 2151 len 24576
33519 060/300 Read block 4062 len 8192
33520 061/300 Read block 3722 len 12288
33521 062/300 Read block 3527 len 20480
33522 063/300 Write block 115 len 40960 data: 6 4 4 7 8 7 8 8 9 10
33523 064/300 Flush
33524 065/300 Flush
33525 066/300 Read block 1773 len 28672
33526 067/300 Read block 2992 len 8192
33527 068/300 Read block 961 len 24576
33528 069/300 Read block 872 len 24576
33529 070/300 Flush
33530 071/300 Read block 1401 len 28672
33531 072/300 Write block 325 len 32768 data: 6 6 4 4 4 4 5 6
33532 073/300 Write block 4375 len 40960 data: 4 4 4 4 4 5 5 4 4 5
33533 074/300 Read block 145 len 36864
33534 075/300 Flush
33535 076/300 Read block 222 len 12288
33536 077/300 Read block 3099 len 40960
33537 078/300 Read block 4928 len 20480
33538 079/300 Write block 3997 len 8192 data: 6 5
33539 080/300 Read block 1444 len 40960
33540 081/300 Write block 912 len 8192 data: 5 6
33541 082/300 Read block 2467 len 20480
33542 083/300 Flush
33543 084/300 Read block 1640 len 32768
33544 085/300 Write block 3366 len 16384 data: 2 3 4 5
33545 086/300 Flush
33546 087/300 Write block 4931 len 36864 data: 8 10 10 8 8 7 4 4 5
33547 088/300 Read block 3946 len 20480
33548 089/300 Read block 3463 len 8192
33549 090/300 Flush
33550 091/300 Read block 1059 len 4096
33551 092/300 Flush
33552 093/300 Write block 2686 len 24576 data: 6 7 6 5 4 4
33553 094/300 Read block 208 len 12288
33554 095/300 Write block 3921 len 32768 data: 3 3 3 3 4 4 5 6
33555 096/300 Read block 1355 len 40960
33556 097/300 Read block 1569 len 16384
33557 098/300 Write block 98 len 12288 data: 6 5 5
33558 099/300 Flush
33559 100/300 Read block 1215 len 32768
33560 101/300 Write block 4178 len 12288 data: 5 5 6
33561 102/300 Write block 1166 len 16384 data: 5 4 6 9
33562 103/300 Write block 1508 len 8192 data: 10 8
33563 104/300 Read block 3227 len 12288
33564 105/300 Read block 178 len 32768
33565 106/300 Write block 3700 len 24576 data: 6 7 8 5 4 4
33566 107/300 Read block 4550 len 8192
33567 108/300 Read block 243 len 12288
33568 109/300 Flush
33569 110/300 Flush
33570 111/300 Read block 1149 len 4096
33571 112/300 Read block 3282 len 32768
33572 113/300 Read block 370 len 16384
33573 114/300 Read block 3269 len 32768
33574 115/300 Read block 2476 len 28672
33575 116/300 Read block 259 len 16384
33576 117/300 Read block 2332 len 32768
33577 118/300 Read block 994 len 12288
33578 119/300 Read block 2309 len 16384
33579 120/300 Read block 3282 len 16384
33580 121/300 Write block 1033 len 24576 data: 8 6 6 4 2 3
33581 122/300 Read block 2547 len 24576
33582 123/300 Read block 340 len 28672
33583 124/300 Read block 818 len 28672
33584 125/300 Write block 4152 len 40960 data: 4 4 4 5 5 3 2 3 3 5
33585 126/300 Write block 1124 len 40960 data: 9 9 8 8 7 5 5 4 4 4
33586 127/300 Read block 1944 len 12288
33587 128/300 Write block 4457 len 40960 data: 9 5 5 6 6 6 5 7 6 6
33588 129/300 Write block 1967 len 36864 data: 6 7 8 9 9 6 9 9 5
33589 130/300 Read block 2103 len 4096
33590 131/300 Read block 1301 len 16384
33591 132/300 Flush
33592 133/300 Read block 2689 len 28672
33593 134/300 Write block 625 len 4096 data: 3
33594 135/300 Read block 1719 len 20480
33595 136/300 Write block 2404 len 28672 data: 3 3 5 5 5 7 6
33596 137/300 Flush
33597 138/300 Write block 3708 len 8192 data: 2 2
33598 139/300 Read block 4858 len 24576
33599 140/300 Write block 3830 len 4096 data: 8
33600 141/300 Write block 3106 len 28672 data: 9 8 8 5 7 6 6
33601 142/300 Read block 2715 len 36864
33602 143/300 Read block 1008 len 28672
33603 144/300 Write block 1697 len 16384 data: 8 8 7 5
33604 145/300 Read block 2561 len 28672
33605 146/300 Read block 1368 len 24576
33606 147/300 Read block 1931 len 16384
33607 148/300 Read block 4243 len 40960
33608 149/300 Read block 2597 len 8192
33609 150/300 Flush
33610 151/300 Write block 4902 len 4096 data: 6
33611 152/300 Write block 3374 len 36864 data: 4 4 6 5 7 8 8 8 10
33612 153/300 Read block 1826 len 24576
33613 154/300 Flush
33614 155/300 Read block 1227 len 40960
33615 156/300 Read block 2647 len 28672
33616 157/300 Flush
33617 158/300 Write block 3304 len 32768 data: 2 2 3 3 3 3 4 3
33618 159/300 Write block 1673 len 40960 data: 4 5 7 6 5 5 5 6 7 11
33619 160/300 Read block 4711 len 28672
33620 161/300 Read block 1337 len 40960
33621 162/300 Flush
33622 163/300 Read block 1144 len 4096
33623 164/300 Write block 3220 len 12288 data: 10 8 9
33624 165/300 Flush
33625 166/300 Write block 3916 len 4096 data: 4
33626 167/300 Read block 227 len 16384
33627 168/300 Flush
33628 169/300 Read block 3399 len 8192
33629 170/300 Flush
33630 171/300 Read block 1088 len 40960
33631 172/300 Read block 1394 len 36864
33632 173/300 Read block 3369 len 16384
33633 174/300 Write block 717 len 24576 data: 8 8 7 6 8 8
33634 175/300 Read block 2605 len 4096
33635 176/300 Flush
33636 177/300 Write block 4849 len 32768 data: 5 5 6 6 6 5 6 6
33637 178/300 Read block 4446 len 12288
33638 179/300 Read block 1142 len 12288
33639 180/300 Read block 4158 len 4096
33640 181/300 Write block 320 len 16384 data: 4 4 6 4
33641 182/300 Write block 3728 len 40960 data: 6 6 6 6 8 8 7 7 6 4
33642 183/300 Flush
33643 184/300 Read block 2751 len 36864
33644 185/300 Read block 1363 len 12288
33645 186/300 Read block 2965 len 16384
33646 187/300 Write block 4858 len 24576 data: 6 8 8 7 7 6
33647 188/300 Read block 3566 len 24576
33648 189/300 Write block 4807 len 40960 data: 4 4 5 7 6 6 6 5 6 6
33649 190/300 Read block 1534 len 36864
33650 191/300 Read block 198 len 32768
33651 192/300 Read block 3409 len 32768
33652 193/300 Write block 2971 len 16384 data: 7 8 5 5
33653 194/300 Flush
33654 195/300 Write block 2447 len 4096 data: 5
33655 196/300 Write block 1350 len 16384 data: 3 4 5 6
33656 197/300 Read block 590 len 24576
33657 198/300 Read block 1300 len 24576
33658 199/300 Read block 1861 len 28672
33659 200/300 Write block 3337 len 4096 data: 6
33660 201/300 Read block 2879 len 40960
33661 202/300 Write block 4590 len 28672 data: 5 5 5 4 5 4 4
33662 203/300 Read block 2338 len 20480
33663 204/300 Read block 4512 len 36864
33664 205/300 Flush
33665 206/300 Read block 1601 len 16384
33666 207/300 Write block 1890 len 24576 data: 8 7 6 7 5 4
33667 208/300 Read block 1626 len 8192
33668 209/300 Read block 2612 len 16384
33669 210/300 Write block 46 len 4096 data: 4
33670 211/300 Write block 1866 len 32768 data: 6 7 8 6 5 5 6 7
33671 212/300 Write block 3942 len 12288 data: 10 7 8
33672 213/300 Write block 1659 len 32768 data: 4 5 5 5 6 5 5 5
33673 214/300 Read block 3687 len 40960
33674 215/300 Read block 3642 len 20480
33675 216/300 Read block 1162 len 8192
33676 217/300 Read block 4744 len 24576
33677 218/300 Write block 2468 len 40960 data: 8 9 9 8 6 6 6 6 8 8
33678 219/300 Write block 1551 len 4096 data: 7
33679 220/300 Read block 2699 len 40960
33680 221/300 Flush
33681 222/300 Read block 3055 len 36864
33682 223/300 Write block 4479 len 32768 data: 6 7 7 6 6 5 6 6
33683 224/300 Read block 4619 len 24576
33684 225/300 Read block 4940 len 4096
33685 226/300 Write block 465 len 16384 data: 2 3 6 7
33686 227/300 Read block 2493 len 16384
33687 228/300 Flush
33688 229/300 Read block 4758 len 36864
33689 230/300 Read block 3447 len 12288
33690 231/300 Write block 49 len 20480 data: 6 6 7 6 4
33691 232/300 Read block 2281 len 36864
33692 233/300 Flush
33693 234/300 Write block 328 len 4096 data: 5
33694 235/300 Read block 636 len 16384
33695 236/300 Flush
33696 237/300 Read block 3701 len 32768
33697 238/300 Read block 3682 len 28672
33698 239/300 Read block 4929 len 32768
33699 240/300 Read block 29 len 24576
33700 241/300 Write block 2455 len 20480 data: 9 9 10 9 6
33701 242/300 Write block 99 len 12288 data: 6 6 6
33702 243/300 Read block 3542 len 24576
33703 244/300 Read block 1646 len 40960
33704 245/300 Write block 19 len 32768 data: 6 6 6 4 7 7 5 6
33705 246/300 Read block 3058 len 12288
33706 247/300 Read block 2293 len 36864
33707 248/300 Flush
33708 249/300 Read block 2419 len 40960
33709 250/300 Read block 3801 len 40960
33710 251/300 Write block 1691 len 4096 data: 14
33711 252/300 Read block 4095 len 16384
33712 253/300 Write block 2140 len 16384 data: 9 11 10 8
33713 254/300 Write block 4002 len 8192 data: 7 8
33714 255/300 Read block 888 len 24576
33715 256/300 Flush
33716 257/300 Read block 2628 len 32768
33717 258/300 Read block 2455 len 40960
33718 259/300 Read block 866 len 24576
33719 260/300 Write block 1745 len 32768 data: 3 4 4 4 4 7 6 5
33720 261/300 Read block 326 len 16384
33721 262/300 Read block 3976 len 40960
33722 263/300 Read block 290 len 40960
33723 264/300 Read block 4664 len 4096
33724 265/300 Read block 3557 len 32768
33725 266/300 Read block 15 len 36864
33726 267/300 Write block 767 len 36864 data: 8 5 4 4 4 6 6 8 8
33727 268/300 Flush
33728 269/300 Write block 3496 len 40960 data: 8 9 9 9 9 9 7 8 7 7
33729 270/300 Flush
33730 271/300 Flush
33731 272/300 Read block 3687 len 8192
33732 273/300 Write block 655 len 20480 data: 3 4 5 5 6
33733 274/300 Read block 3971 len 36864
33734 275/300 Write block 4723 len 20480 data: 3 5 4 3 4
33735 276/300 Read block 59 len 8192
33736 277/300 Write block 273 len 32768 data: 4 6 5 4 5 5 6 5
33737 278/300 Write block 4424 len 40960 data: 7 7 6 8 7 5 4 4 4 5
33738 279/300 Write block 791 len 32768 data: 8 9 9 9 9 10 9 8
33739 280/300 Write block 998 len 32768 data: 6 5 6 5 4 4 5 6
33740 281/300 Read block 2280 len 28672
33741 282/300 Write block 2354 len 28672 data: 5 6 6 4 4 4 4
33742 283/300 Write block 1939 len 24576 data: 11 12 10 9 9 10
33743 284/300 Write block 3430 len 8192 data: 9 7
33744 285/300 Write block 2849 len 4096 data: 5
33745 286/300 Write block 2403 len 20480 data: 3 4 4 6 6
33746 287/300 Write block 3092 len 20480 data: 5 5 5 3 4
33747 288/300 Read block 905 len 20480
33748 289/300 Write block 122 len 12288 data: 9 10 11
33749 290/300 Read block 1919 len 32768
33750 291/300 Read block 4876 len 16384
33751 292/300 Write block 3196 len 20480 data: 6 5 6 5 5
33752 293/300 Read block 927 len 8192
33753 294/300 Read block 316 len 24576
33754 295/300 Read block 4983 len 20480
33755 296/300 Write block 4074 len 36864 data: 8 7 6 5 6 3 4 4 4
33756 297/300 Read block 2253 len 40960
33757 298/300 Write block 4047 len 24576 data: 9 8 7 9 8 7
33758 299/300 Write block 2250 len 8192 data: 6 5
33759 300/300 Read block 3638 len 24576
33760 301/300 Read block 677 len 24576
33761 Main task has work to do, go find it
33762 got dsc Start(2)
33763 start 2
33764 Jan 29 05:18:18.789 INFO request completed, latency_us: 172, response_code: 204, uri: /start/cid/2, method: POST, req_id: 4c4a53d5-de52-439b-be6c-9f269ade6907, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
33765 Replay: started 2, returned:Ok(())
33766 ----------------------------------------------------------------
33767 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33768 States: Active Active Offline
33769 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33770 9455 Acked Read 6 Done Done Sent false
33771 9456 Acked Write 5 Done Done Sent false
33772 9457 Acked Write 9 Done Done Sent false
33773 9458 Acked Write 6 Done Done Sent false
33774 9459 Acked Write 1 Done Done Sent false
33775 9460 Acked Read 9 Done Done Sent false
33776 9461 Acked Write 2 Done Done Sent false
33777 9462 Acked Read 10 Done Done Sent false
33778 9463 Acked Write 3 Done Done Sent false
33779 9464 Acked Write 3 Done Done Sent false
33780 9465 Acked Read 4 Done Done Sent false
33781 9466 Acked Write 4 Done Done Sent false
33782 9467 Acked Flush 0 Done Done Sent false
33783 9468 Acked Read 3 Done Done Sent false
33784 9469 Acked Flush 0 Done Done Sent false
33785 9470 Acked Write 7 Done Done Sent false
33786 9471 Acked Read 8 Done Done Sent false
33787 9472 Acked Flush 0 Done Done Sent false
33788 9473 Acked Write 6 Done Done Sent false
33789 9474 Acked Read 2 Done Done Sent false
33790 9475 Acked Write 8 Done Done Sent false
33791 9476 Acked Write 9 Done Done Sent false
33792 9477 Acked Read 10 Done Done Sent false
33793 9478 Acked Write 10 Done Done Sent false
33794 9479 Acked Write 2 Done Done Sent false
33795 9480 Acked Write 5 Done Done Sent false
33796 9481 Acked Write 4 Done Done Sent false
33797 9482 Acked Write 4 Done Done Sent false
33798 9483 Acked Read 8 Done Done Sent false
33799 9484 Acked Read 5 Done Done Sent false
33800 9485 Acked Write 3 Done Done Sent false
33801 9486 Acked Write 5 Done Done Sent false
33802 9487 Acked Read 4 Done Done Sent false
33803 9488 Acked Read 10 Done Done Sent false
33804 9489 Acked Flush 0 Done Done Sent false
33805 9490 Acked Write 5 Done Done Sent false
33806 9491 Acked Read 4 Done Done Sent false
33807 9492 Acked Read 6 Done Done Sent false
33808 9493 Acked Read 7 Done Done Sent false
33809 9494 Acked Write 5 Done Done Sent false
33810 9495 Acked Read 6 Done Done Sent false
33811 9496 Acked Write 1 Done Done Sent false
33812 9497 Acked Read 10 Done Done Sent false
33813 9498 Acked Write 6 Done Done Sent false
33814 9499 Acked Read 8 Done Done Sent false
33815 9500 Acked Read 7 Done Done Sent false
33816 9501 Acked Read 8 Done Done Sent false
33817 9502 Acked Write 6 Done Done Sent false
33818 9503 Acked Read 5 Done Done Sent false
33819 9504 Acked Write 5 Done Done Sent false
33820 9505 Acked Read 2 Done Done Sent false
33821 9506 Acked Read 1 Done Done Sent false
33822 9507 Acked Read 9 Done Done Sent false
33823 9508 Acked Write 7 Done Done Sent false
33824 9509 Acked Write 5 Done Done Sent false
33825 9510 Acked Write 1 Done Done Sent false
33826 9511 Acked Read 6 Done Done Sent false
33827 9512 Acked Read 6 Done Done Sent false
33828 9513 Acked Read 6 Done Done Sent false
33829 9514 Acked Read 2 Done Done Sent false
33830 9515 Acked Read 3 Done Done Sent false
33831 9516 Acked Read 5 Done Done Sent false
33832 9517 Acked Write 10 Done Done Sent false
33833 9518 Acked Flush 0 Done Done Sent false
33834 9519 Acked Flush 0 Done Done Sent false
33835 9520 Acked Read 7 Done Done Sent false
33836 9521 Acked Read 2 Done Done Sent false
33837 9522 Acked Read 6 Done Done Sent false
33838 9523 Acked Read 6 Done Done Sent false
33839 9524 Acked Flush 0 Done Done Sent false
33840 9525 Acked Read 7 Done Done Sent false
33841 9526 Acked Write 8 Done Done Sent false
33842 9527 Acked Write 10 Done Done Sent false
33843 9528 Acked Read 9 Done Done Sent false
33844 9529 Acked Flush 0 Done Done Sent false
33845 9530 Acked Read 3 Done Done Sent false
33846 9531 Acked Read 10 Done Done Sent false
33847 9532 Acked Read 5 Done Done Sent false
33848 9533 Acked Write 2 Done Done Sent false
33849 9534 Acked Read 10 Done Done Sent false
33850 9535 Acked Write 2 Done Done Sent false
33851 9536 Acked Read 5 Done Done Sent false
33852 9537 Acked Flush 0 Done Done Sent false
33853 9538 Acked Read 8 Done Done Sent false
33854 9539 Acked Write 4 Done Done Sent false
33855 9540 Acked Flush 0 Done Done Sent false
33856 9541 Acked Write 9 Done Done Sent false
33857 9542 Acked Read 5 Done Done Sent false
33858 9543 Acked Read 2 Done Done Sent false
33859 9544 Acked Flush 0 Done Done Sent false
33860 9545 Acked Read 1 Done Done Sent false
33861 9546 Acked Flush 0 Done Done Sent false
33862 9547 Acked Write 6 Done Done Sent false
33863 9548 Acked Read 3 Done Done Sent false
33864 9549 Acked Write 8 Done Done Sent false
33865 9550 Acked Read 10 Done Done Sent false
33866 9551 Acked Read 4 Done Done Sent false
33867 9552 Acked Write 3 Done Done Sent false
33868 9553 Acked Flush 0 Done Done Sent false
33869 9554 Acked Read 8 Done Done Sent false
33870 9555 Acked Write 3 Done Done Sent false
33871 9556 Acked Write 4 Done Done Sent false
33872 9557 Acked Write 2 Done Done Sent false
33873 9558 Acked Read 3 Done Done Sent false
33874 9559 Acked Read 8 Done Done Sent false
33875 9560 Acked Write 6 Done Done Sent false
33876 9561 Acked Read 2 Done Done Sent false
33877 9562 Acked Read 3 Done Done Sent false
33878 9563 Acked Flush 0 Done Done Sent false
33879 9564 Acked Flush 0 Done Done Sent false
33880 9565 Acked Read 1 Done Done Sent false
33881 9566 Acked Read 8 Done Done Sent false
33882 9567 Acked Read 4 Done Done Sent false
33883 9568 Acked Read 8 Done Done Sent false
33884 9569 Acked Read 7 Done Done Sent false
33885 9570 Acked Read 4 Done Done Sent false
33886 9571 Acked Read 8 Done Done Sent false
33887 9572 Acked Read 3 Done Done Sent false
33888 9573 Acked Read 4 Done Done Sent false
33889 9574 Acked Read 4 Done Done Sent false
33890 9575 Acked Write 6 Done Done Sent false
33891 9576 Acked Read 6 Done Done Sent false
33892 9577 Acked Read 7 Done Done Sent false
33893 9578 Acked Read 7 Done Done Sent false
33894 9579 Acked Write 10 Done Done Sent false
33895 9580 Acked Write 10 Done Done Sent false
33896 9581 Acked Read 3 Done Done Sent false
33897 9582 Acked Write 10 Done Done Sent false
33898 9583 Acked Write 9 Done Done Sent false
33899 9584 Acked Read 1 Done Done Sent false
33900 9585 Acked Read 4 Done Done Sent false
33901 9586 Acked Flush 0 Done Done Sent false
33902 9587 Acked Read 7 Done Done Sent false
33903 9588 Acked Write 1 Done Done Sent false
33904 9589 Acked Read 5 Done Done Sent false
33905 9590 Acked Write 7 Done Done Sent false
33906 9591 Acked Flush 0 Done Done Sent false
33907 9592 Acked Write 2 Done Done Sent false
33908 9593 Acked Read 6 Done Done Sent false
33909 9594 Acked Write 1 Done Done Sent false
33910 9595 Acked Write 7 Done Done Sent false
33911 9596 Acked Read 9 Done Done Sent false
33912 9597 Acked Read 7 Done Done Sent false
33913 9598 Acked Write 4 Done Done Sent false
33914 9599 Acked Read 7 Done Done Sent false
33915 9600 Acked Read 6 Done Done Sent false
33916 9601 Acked Read 4 Done Done Sent false
33917 9602 Acked Read 10 Done Done Sent false
33918 9603 Acked Read 2 Done Done Sent false
33919 9604 Acked Flush 0 Done Done Sent false
33920 9605 Acked Write 1 Done Done Sent false
33921 9606 Acked Write 9 Done Done Sent false
33922 9607 Acked Read 6 Done Done Sent false
33923 9608 Acked Flush 0 Done Done Sent false
33924 9609 Acked Read 10 Done Done Sent false
33925 9610 Acked Read 7 Done Done Sent false
33926 9611 Acked Flush 0 Done Done Sent false
33927 9612 Acked Write 8 Done Done Sent false
33928 9613 Acked Write 10 Done Done Sent false
33929 9614 Acked Read 7 Done Done Sent false
33930 9615 Acked Read 10 Done Done Sent false
33931 9616 Acked Flush 0 Done Done Sent false
33932 9617 Acked Read 1 Done Done Sent false
33933 9618 Acked Write 3 Done Done Sent false
33934 9619 Acked Flush 0 Done Done Sent false
33935 9620 Acked Write 1 Done Done Sent false
33936 9621 Acked Read 4 Done Done Sent false
33937 9622 Acked Flush 0 Done Done Sent false
33938 9623 Acked Read 2 Done Done Sent false
33939 9624 Acked Flush 0 Done Done Sent false
33940 9625 Acked Read 10 Done Done Sent false
33941 9626 Acked Read 9 Done Done Sent false
33942 9627 Acked Read 4 Done Done Sent false
33943 9628 Acked Write 6 Done Done Sent false
33944 9629 Acked Read 1 Done Done Sent false
33945 9630 Acked Flush 0 Done Done Sent false
33946 9631 Acked Write 8 Done Done Sent false
33947 9632 Acked Read 3 Done Done Sent false
33948 9633 Acked Read 3 Done Done Sent false
33949 9634 Acked Read 1 Done Done Sent false
33950 9635 Acked Write 4 Done Done Sent false
33951 9636 Acked Write 10 Done Done Sent false
33952 9637 Acked Flush 0 Done Done Sent false
33953 9638 Acked Read 9 Done Done Sent false
33954 9639 Acked Read 3 Done Done Sent false
33955 9640 Acked Read 4 Done Done Sent false
33956 9641 Acked Write 6 Done Done Sent false
33957 9642 Acked Read 6 Done Done Sent false
33958 9643 Acked Write 10 Done Done Sent false
33959 9644 Acked Read 9 Done Done Sent false
33960 9645 Acked Read 8 Done Done Sent false
33961 9646 Acked Read 8 Done Done Sent false
33962 9647 Acked Write 4 Done Done Sent false
33963 9648 Acked Flush 0 Done Done Sent false
33964 9649 Acked Write 1 Done Done Sent false
33965 9650 Acked Write 4 Done Done Sent false
33966 9651 Acked Read 6 Done Done Sent false
33967 9652 Acked Read 6 Done Done Sent false
33968 9653 Acked Read 7 Done Done Sent false
33969 9654 Acked Write 1 Done Done Sent false
33970 9655 Acked Read 10 Done Done Sent false
33971 9656 Acked Write 7 Done Done Sent false
33972 9657 Acked Read 5 Done Done Sent false
33973 9658 Acked Read 9 Done Done Sent false
33974 9659 Acked Flush 0 Done Done Sent false
33975 9660 Acked Read 4 Done Done Sent false
33976 9661 Acked Write 6 Done Done Sent false
33977 9662 Acked Read 2 Done Done Sent false
33978 9663 Acked Read 4 Done Done Sent false
33979 9664 Acked Write 1 Done Done Sent false
33980 9665 Acked Write 8 Done Done Sent false
33981 9666 Acked Write 3 Done Done Sent false
33982 9667 Acked Write 8 Done Done Sent false
33983 9668 Acked Read 10 Done Done Sent false
33984 9669 Acked Read 5 Done Done Sent false
33985 9670 Acked Read 2 Done Done Sent false
33986 9671 Acked Read 6 Done Done Sent false
33987 9672 Acked Write 10 Done Done Sent false
33988 9673 Acked Write 1 Done Done Sent false
33989 9674 Acked Read 10 Done Done Sent false
33990 9675 Acked Flush 0 Done Done Sent false
33991 9676 Acked Read 9 Done Done Sent false
33992 9677 Acked Write 8 Done Done Sent false
33993 9678 Acked Read 6 Done Done Sent false
33994 9679 Acked Read 1 Done Done Sent false
33995 9680 Acked Write 4 Done Done Sent false
33996 9681 Acked Read 4 Done Done Sent false
33997 9682 Acked Flush 0 Done Done Sent false
33998 9683 Acked Read 9 Done Done Sent false
33999 9684 Acked Read 3 Done Done Sent false
34000 9685 Acked Write 5 Done Done Sent false
34001 9686 Acked Read 9 Done Done Sent false
34002 9687 Acked Flush 0 Done Done Sent false
34003 9688 Acked Write 1 Done Done Sent false
34004 9689 Acked Read 4 Done Done Sent false
34005 9690 Acked Flush 0 Done Done Sent false
34006 9691 Acked Read 8 Done Done Sent false
34007 9692 Acked Read 7 Done Done Sent false
34008 9693 Acked Read 8 Done Done Sent false
34009 9694 Acked Read 6 Done Done Sent false
34010 9695 Acked Write 5 Done Done Sent false
34011 9696 Acked Write 3 Done Done Sent false
34012 9697 Acked Read 6 Done Done Sent false
34013 9698 Acked Read 10 Done Done Sent false
34014 9699 Acked Write 8 Done Done Sent false
34015 9700 Acked Read 3 Done Done Sent false
34016 9701 Acked Read 9 Done Done Sent false
34017 9702 Acked Flush 0 Done Done Sent false
34018 9703 Acked Read 10 Done Done Sent false
34019 9704 Acked Read 10 Done Done Sent false
34020 9705 Acked Write 1 Done Done Sent false
34021 9706 Acked Read 4 Done Done Sent false
34022 9707 Acked Write 4 Done Done Sent false
34023 9708 Acked Write 2 Done Done Sent false
34024 9709 Acked Read 6 Done Done Sent false
34025 9710 Acked Flush 0 Done Done Sent false
34026 9711 Acked Read 8 Done Done Sent false
34027 9712 Acked Read 10 Done Done Sent false
34028 9713 Acked Read 6 Done Done Sent false
34029 9714 Acked Write 8 Done Done Sent false
34030 9715 Acked Read 4 Done Done Sent false
34031 9716 Acked Read 10 Done Done Sent false
34032 9717 Acked Read 10 Done Done Sent false
34033 9718 Acked Read 1 Done Done Sent false
34034 9719 Acked Read 8 Done Done Sent false
34035 9720 Acked Read 9 Done Done Sent false
34036 9721 Acked Write 9 Done Done Sent false
34037 9722 Acked Flush 0 Done Done Sent false
34038 9723 Acked Write 10 Done Done Sent false
34039 9724 Acked Flush 0 Done Done Sent false
34040 9725 Acked Flush 0 Done Done Sent false
34041 9726 Acked Read 2 Done Done Sent false
34042 9727 Acked Write 5 Done Done Sent false
34043 9728 Acked Read 9 Done Done Sent false
34044 9729 Acked Write 5 Done Done Sent false
34045 9730 Acked Read 2 Done Done Sent false
34046 9731 Acked Write 8 Done Done Sent false
34047 9732 Acked Write 10 Done Done Sent false
34048 9733 Acked Write 8 Done Done Sent false
34049 9734 Acked Write 8 Done Done Sent false
34050 9735 Acked Read 7 Done Done Sent false
34051 9736 Acked Write 7 Done Done Sent false
34052 9737 Acked Write 6 Done Done Sent false
34053 9738 Acked Write 2 Done Done Sent false
34054 9739 Acked Write 1 Done Done Sent false
34055 9740 Acked Write 5 Done Done Sent false
34056 9741 Acked Write 5 Done Done Sent false
34057 9742 Acked Read 5 Done Done Sent false
34058 9743 Acked Write 3 Done Done Sent false
34059 9744 Acked Read 8 Done Done Sent false
34060 9745 Acked Read 4 Done Done Sent false
34061 9746 Acked Write 5 Done Done Sent false
34062 9747 Acked Read 2 Done Done Sent false
34063 9748 Acked Read 6 Done Done Sent false
34064 9749 Acked Read 5 Done Done Sent false
34065 9750 Acked Write 9 Done Done Sent false
34066 9751 Acked Read 10 Done Done Sent false
34067 9752 Acked Write 6 Done Done Sent false
34068 9753 Acked Write 2 Done Done Sent false
34069 9754 Acked Read 6 Done Done Sent false
34070 9755 Acked Read 6 Done Done Sent false
34071 STATES DS:0 DS:1 DS:2 TOTAL
34072 Sent 0 0 301 301
34073 Done 301 301 0 602
34074 Skipped 0 0 0 0
34075 Error 0 0 0 0
34076 Last Flush: Some(JobId(9725)) Some(JobId(9725)) Some(JobId(9454))
34077 Downstairs last five completed: 9454 9453 9452 9451 9450
34078 Upstairs last five completed: 9755 9754 9753 9752 9751
34079 CLIENT: Up:0 ds:301 act:2
340802025-01-29T05:18:19.680ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
340812025-01-29T05:18:20.786ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
340822025-01-29T05:18:21.683ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
340832025-01-29T05:18:22.685ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
34084 ----------------------------------------------------------------
34085 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34086 States: Active Active Offline
34087 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34088 9455 Acked Read 6 Done Done Sent false
34089 9456 Acked Write 5 Done Done Sent false
34090 9457 Acked Write 9 Done Done Sent false
34091 9458 Acked Write 6 Done Done Sent false
34092 9459 Acked Write 1 Done Done Sent false
34093 9460 Acked Read 9 Done Done Sent false
34094 9461 Acked Write 2 Done Done Sent false
34095 9462 Acked Read 10 Done Done Sent false
34096 9463 Acked Write 3 Done Done Sent false
34097 9464 Acked Write 3 Done Done Sent false
34098 9465 Acked Read 4 Done Done Sent false
34099 9466 Acked Write 4 Done Done Sent false
34100 9467 Acked Flush 0 Done Done Sent false
34101 9468 Acked Read 3 Done Done Sent false
34102 9469 Acked Flush 0 Done Done Sent false
34103 9470 Acked Write 7 Done Done Sent false
34104 9471 Acked Read 8 Done Done Sent false
34105 9472 Acked Flush 0 Done Done Sent false
34106 9473 Acked Write 6 Done Done Sent false
34107 9474 Acked Read 2 Done Done Sent false
34108 9475 Acked Write 8 Done Done Sent false
34109 9476 Acked Write 9 Done Done Sent false
34110 9477 Acked Read 10 Done Done Sent false
34111 9478 Acked Write 10 Done Done Sent false
34112 9479 Acked Write 2 Done Done Sent false
34113 9480 Acked Write 5 Done Done Sent false
34114 9481 Acked Write 4 Done Done Sent false
34115 9482 Acked Write 4 Done Done Sent false
34116 9483 Acked Read 8 Done Done Sent false
34117 9484 Acked Read 5 Done Done Sent false
34118 9485 Acked Write 3 Done Done Sent false
34119 9486 Acked Write 5 Done Done Sent false
34120 9487 Acked Read 4 Done Done Sent false
34121 9488 Acked Read 10 Done Done Sent false
34122 9489 Acked Flush 0 Done Done Sent false
34123 9490 Acked Write 5 Done Done Sent false
34124 9491 Acked Read 4 Done Done Sent false
34125 9492 Acked Read 6 Done Done Sent false
34126 9493 Acked Read 7 Done Done Sent false
34127 9494 Acked Write 5 Done Done Sent false
34128 9495 Acked Read 6 Done Done Sent false
34129 9496 Acked Write 1 Done Done Sent false
34130 9497 Acked Read 10 Done Done Sent false
34131 9498 Acked Write 6 Done Done Sent false
34132 9499 Acked Read 8 Done Done Sent false
34133 9500 Acked Read 7 Done Done Sent false
34134 9501 Acked Read 8 Done Done Sent false
34135 9502 Acked Write 6 Done Done Sent false
34136 9503 Acked Read 5 Done Done Sent false
34137 9504 Acked Write 5 Done Done Sent false
34138 9505 Acked Read 2 Done Done Sent false
34139 9506 Acked Read 1 Done Done Sent false
34140 9507 Acked Read 9 Done Done Sent false
34141 9508 Acked Write 7 Done Done Sent false
34142 9509 Acked Write 5 Done Done Sent false
34143 9510 Acked Write 1 Done Done Sent false
34144 9511 Acked Read 6 Done Done Sent false
34145 9512 Acked Read 6 Done Done Sent false
34146 9513 Acked Read 6 Done Done Sent false
34147 9514 Acked Read 2 Done Done Sent false
34148 9515 Acked Read 3 Done Done Sent false
34149 9516 Acked Read 5 Done Done Sent false
34150 9517 Acked Write 10 Done Done Sent false
34151 9518 Acked Flush 0 Done Done Sent false
34152 9519 Acked Flush 0 Done Done Sent false
34153 9520 Acked Read 7 Done Done Sent false
34154 9521 Acked Read 2 Done Done Sent false
34155 9522 Acked Read 6 Done Done Sent false
34156 9523 Acked Read 6 Done Done Sent false
34157 9524 Acked Flush 0 Done Done Sent false
34158 9525 Acked Read 7 Done Done Sent false
34159 9526 Acked Write 8 Done Done Sent false
34160 9527 Acked Write 10 Done Done Sent false
34161 9528 Acked Read 9 Done Done Sent false
34162 9529 Acked Flush 0 Done Done Sent false
34163 9530 Acked Read 3 Done Done Sent false
34164 9531 Acked Read 10 Done Done Sent false
34165 9532 Acked Read 5 Done Done Sent false
34166 9533 Acked Write 2 Done Done Sent false
34167 9534 Acked Read 10 Done Done Sent false
34168 9535 Acked Write 2 Done Done Sent false
34169 9536 Acked Read 5 Done Done Sent false
34170 9537 Acked Flush 0 Done Done Sent false
34171 9538 Acked Read 8 Done Done Sent false
34172 9539 Acked Write 4 Done Done Sent false
34173 9540 Acked Flush 0 Done Done Sent false
34174 9541 Acked Write 9 Done Done Sent false
34175 9542 Acked Read 5 Done Done Sent false
34176 9543 Acked Read 2 Done Done Sent false
34177 9544 Acked Flush 0 Done Done Sent false
34178 9545 Acked Read 1 Done Done Sent false
34179 9546 Acked Flush 0 Done Done Sent false
34180 9547 Acked Write 6 Done Done Sent false
34181 9548 Acked Read 3 Done Done Sent false
34182 9549 Acked Write 8 Done Done Sent false
34183 9550 Acked Read 10 Done Done Sent false
34184 9551 Acked Read 4 Done Done Sent false
34185 9552 Acked Write 3 Done Done Sent false
34186 9553 Acked Flush 0 Done Done Sent false
34187 9554 Acked Read 8 Done Done Sent false
34188 9555 Acked Write 3 Done Done Sent false
34189 9556 Acked Write 4 Done Done Sent false
34190 9557 Acked Write 2 Done Done Sent false
34191 9558 Acked Read 3 Done Done Sent false
34192 9559 Acked Read 8 Done Done Sent false
34193 9560 Acked Write 6 Done Done Sent false
34194 9561 Acked Read 2 Done Done Sent false
34195 9562 Acked Read 3 Done Done Sent false
34196 9563 Acked Flush 0 Done Done Sent false
34197 9564 Acked Flush 0 Done Done Sent false
34198 9565 Acked Read 1 Done Done Sent false
34199 9566 Acked Read 8 Done Done Sent false
34200 9567 Acked Read 4 Done Done Sent false
34201 9568 Acked Read 8 Done Done Sent false
34202 9569 Acked Read 7 Done Done Sent false
34203 9570 Acked Read 4 Done Done Sent false
34204 9571 Acked Read 8 Done Done Sent false
34205 9572 Acked Read 3 Done Done Sent false
34206 9573 Acked Read 4 Done Done Sent false
34207 9574 Acked Read 4 Done Done Sent false
34208 9575 Acked Write 6 Done Done Sent false
34209 9576 Acked Read 6 Done Done Sent false
34210 9577 Acked Read 7 Done Done Sent false
34211 9578 Acked Read 7 Done Done Sent false
34212 9579 Acked Write 10 Done Done Sent false
34213 9580 Acked Write 10 Done Done Sent false
34214 9581 Acked Read 3 Done Done Sent false
34215 9582 Acked Write 10 Done Done Sent false
34216 9583 Acked Write 9 Done Done Sent false
34217 9584 Acked Read 1 Done Done Sent false
34218 9585 Acked Read 4 Done Done Sent false
34219 9586 Acked Flush 0 Done Done Sent false
34220 9587 Acked Read 7 Done Done Sent false
34221 9588 Acked Write 1 Done Done Sent false
34222 9589 Acked Read 5 Done Done Sent false
34223 9590 Acked Write 7 Done Done Sent false
34224 9591 Acked Flush 0 Done Done Sent false
34225 9592 Acked Write 2 Done Done Sent false
34226 9593 Acked Read 6 Done Done Sent false
34227 9594 Acked Write 1 Done Done Sent false
34228 9595 Acked Write 7 Done Done Sent false
34229 9596 Acked Read 9 Done Done Sent false
34230 9597 Acked Read 7 Done Done Sent false
34231 9598 Acked Write 4 Done Done Sent false
34232 9599 Acked Read 7 Done Done Sent false
34233 9600 Acked Read 6 Done Done Sent false
34234 9601 Acked Read 4 Done Done Sent false
34235 9602 Acked Read 10 Done Done Sent false
34236 9603 Acked Read 2 Done Done Sent false
34237 9604 Acked Flush 0 Done Done Sent false
34238 9605 Acked Write 1 Done Done Sent false
34239 9606 Acked Write 9 Done Done Sent false
34240 9607 Acked Read 6 Done Done Sent false
34241 9608 Acked Flush 0 Done Done Sent false
34242 9609 Acked Read 10 Done Done Sent false
34243 9610 Acked Read 7 Done Done Sent false
34244 9611 Acked Flush 0 Done Done Sent false
34245 9612 Acked Write 8 Done Done Sent false
34246 9613 Acked Write 10 Done Done Sent false
34247 9614 Acked Read 7 Done Done Sent false
34248 9615 Acked Read 10 Done Done Sent false
34249 9616 Acked Flush 0 Done Done Sent false
34250 9617 Acked Read 1 Done Done Sent false
34251 9618 Acked Write 3 Done Done Sent false
34252 9619 Acked Flush 0 Done Done Sent false
34253 9620 Acked Write 1 Done Done Sent false
34254 9621 Acked Read 4 Done Done Sent false
34255 9622 Acked Flush 0 Done Done Sent false
34256 9623 Acked Read 2 Done Done Sent false
34257 9624 Acked Flush 0 Done Done Sent false
34258 9625 Acked Read 10 Done Done Sent false
34259 9626 Acked Read 9 Done Done Sent false
34260 9627 Acked Read 4 Done Done Sent false
34261 9628 Acked Write 6 Done Done Sent false
34262 9629 Acked Read 1 Done Done Sent false
34263 9630 Acked Flush 0 Done Done Sent false
34264 9631 Acked Write 8 Done Done Sent false
34265 9632 Acked Read 3 Done Done Sent false
34266 9633 Acked Read 3 Done Done Sent false
34267 9634 Acked Read 1 Done Done Sent false
34268 9635 Acked Write 4 Done Done Sent false
34269 9636 Acked Write 10 Done Done Sent false
34270 9637 Acked Flush 0 Done Done Sent false
34271 9638 Acked Read 9 Done Done Sent false
34272 9639 Acked Read 3 Done Done Sent false
34273 9640 Acked Read 4 Done Done Sent false
34274 9641 Acked Write 6 Done Done Sent false
34275 9642 Acked Read 6 Done Done Sent false
34276 9643 Acked Write 10 Done Done Sent false
34277 9644 Acked Read 9 Done Done Sent false
34278 9645 Acked Read 8 Done Done Sent false
34279 9646 Acked Read 8 Done Done Sent false
34280 9647 Acked Write 4 Done Done Sent false
34281 9648 Acked Flush 0 Done Done Sent false
34282 9649 Acked Write 1 Done Done Sent false
34283 9650 Acked Write 4 Done Done Sent false
34284 9651 Acked Read 6 Done Done Sent false
34285 9652 Acked Read 6 Done Done Sent false
34286 9653 Acked Read 7 Done Done Sent false
34287 9654 Acked Write 1 Done Done Sent false
34288 9655 Acked Read 10 Done Done Sent false
34289 9656 Acked Write 7 Done Done Sent false
34290 9657 Acked Read 5 Done Done Sent false
34291 9658 Acked Read 9 Done Done Sent false
34292 9659 Acked Flush 0 Done Done Sent false
34293 9660 Acked Read 4 Done Done Sent false
34294 9661 Acked Write 6 Done Done Sent false
34295 9662 Acked Read 2 Done Done Sent false
34296 9663 Acked Read 4 Done Done Sent false
34297 9664 Acked Write 1 Done Done Sent false
34298 9665 Acked Write 8 Done Done Sent false
34299 9666 Acked Write 3 Done Done Sent false
34300 9667 Acked Write 8 Done Done Sent false
34301 9668 Acked Read 10 Done Done Sent false
34302 9669 Acked Read 5 Done Done Sent false
34303 9670 Acked Read 2 Done Done Sent false
34304 9671 Acked Read 6 Done Done Sent false
34305 9672 Acked Write 10 Done Done Sent false
34306 9673 Acked Write 1 Done Done Sent false
34307 9674 Acked Read 10 Done Done Sent false
34308 9675 Acked Flush 0 Done Done Sent false
34309 9676 Acked Read 9 Done Done Sent false
34310 9677 Acked Write 8 Done Done Sent false
34311 9678 Acked Read 6 Done Done Sent false
34312 9679 Acked Read 1 Done Done Sent false
34313 9680 Acked Write 4 Done Done Sent false
34314 9681 Acked Read 4 Done Done Sent false
34315 9682 Acked Flush 0 Done Done Sent false
34316 9683 Acked Read 9 Done Done Sent false
34317 9684 Acked Read 3 Done Done Sent false
34318 9685 Acked Write 5 Done Done Sent false
34319 9686 Acked Read 9 Done Done Sent false
34320 9687 Acked Flush 0 Done Done Sent false
34321 9688 Acked Write 1 Done Done Sent false
34322 9689 Acked Read 4 Done Done Sent false
34323 9690 Acked Flush 0 Done Done Sent false
34324 9691 Acked Read 8 Done Done Sent false
34325 9692 Acked Read 7 Done Done Sent false
34326 9693 Acked Read 8 Done Done Sent false
34327 9694 Acked Read 6 Done Done Sent false
34328 9695 Acked Write 5 Done Done Sent false
34329 9696 Acked Write 3 Done Done Sent false
34330 9697 Acked Read 6 Done Done Sent false
34331 9698 Acked Read 10 Done Done Sent false
34332 9699 Acked Write 8 Done Done Sent false
34333 9700 Acked Read 3 Done Done Sent false
34334 9701 Acked Read 9 Done Done Sent false
34335 9702 Acked Flush 0 Done Done Sent false
34336 9703 Acked Read 10 Done Done Sent false
34337 9704 Acked Read 10 Done Done Sent false
34338 9705 Acked Write 1 Done Done Sent false
34339 9706 Acked Read 4 Done Done Sent false
34340 9707 Acked Write 4 Done Done Sent false
34341 9708 Acked Write 2 Done Done Sent false
34342 9709 Acked Read 6 Done Done Sent false
34343 9710 Acked Flush 0 Done Done Sent false
34344 9711 Acked Read 8 Done Done Sent false
34345 9712 Acked Read 10 Done Done Sent false
34346 9713 Acked Read 6 Done Done Sent false
34347 9714 Acked Write 8 Done Done Sent false
34348 9715 Acked Read 4 Done Done Sent false
34349 9716 Acked Read 10 Done Done Sent false
34350 9717 Acked Read 10 Done Done Sent false
34351 9718 Acked Read 1 Done Done Sent false
34352 9719 Acked Read 8 Done Done Sent false
34353 9720 Acked Read 9 Done Done Sent false
34354 9721 Acked Write 9 Done Done Sent false
34355 9722 Acked Flush 0 Done Done Sent false
34356 9723 Acked Write 10 Done Done Sent false
34357 9724 Acked Flush 0 Done Done Sent false
34358 9725 Acked Flush 0 Done Done Sent false
34359 9726 Acked Read 2 Done Done Sent false
34360 9727 Acked Write 5 Done Done Sent false
34361 9728 Acked Read 9 Done Done Sent false
34362 9729 Acked Write 5 Done Done Sent false
34363 9730 Acked Read 2 Done Done Sent false
34364 9731 Acked Write 8 Done Done Sent false
34365 9732 Acked Write 10 Done Done Sent false
34366 9733 Acked Write 8 Done Done Sent false
34367 9734 Acked Write 8 Done Done Sent false
34368 9735 Acked Read 7 Done Done Sent false
34369 9736 Acked Write 7 Done Done Sent false
34370 9737 Acked Write 6 Done Done Sent false
34371 9738 Acked Write 2 Done Done Sent false
34372 9739 Acked Write 1 Done Done Sent false
34373 9740 Acked Write 5 Done Done Sent false
34374 9741 Acked Write 5 Done Done Sent false
34375 9742 Acked Read 5 Done Done Sent false
34376 9743 Acked Write 3 Done Done Sent false
34377 9744 Acked Read 8 Done Done Sent false
34378 9745 Acked Read 4 Done Done Sent false
34379 9746 Acked Write 5 Done Done Sent false
34380 9747 Acked Read 2 Done Done Sent false
34381 9748 Acked Read 6 Done Done Sent false
34382 9749 Acked Read 5 Done Done Sent false
34383 9750 Acked Write 9 Done Done Sent false
34384 9751 Acked Read 10 Done Done Sent false
34385 9752 Acked Write 6 Done Done Sent false
34386 9753 Acked Write 2 Done Done Sent false
34387 9754 Acked Read 6 Done Done Sent false
34388 9755 Acked Read 6 Done Done Sent false
34389 STATES DS:0 DS:1 DS:2 TOTAL
34390 Sent 0 0 301 301
34391 Done 301 301 0 602
34392 Skipped 0 0 0 0
34393 Error 0 0 0 0
34394 Last Flush: Some(JobId(9725)) Some(JobId(9725)) Some(JobId(9454))
34395 Downstairs last five completed: 9454 9453 9452 9451 9450
34396 Upstairs last five completed: 9755 9754 9753 9752 9751
34397 CLIENT: Up:0 ds:301 act:2
343982025-01-29T05:18:23.687ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
343992025-01-29T05:18:24.539ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344002025-01-29T05:18:24.539ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344012025-01-29T05:18:24.539ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344022025-01-29T05:18:24.539ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344032025-01-29T05:18:24.539ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
34404 [8830] got start action
34405 [8830] I am going to restart
34406 Starting downstairs at port 8830
34407 Make output file at "/tmp/dsc/downstairs-8830.txt"
34408 [8830][2] reports Starting
34409 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1231)
34410 [8830][2] reports Running
344112025-01-29T05:18:25.039ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344122025-01-29T05:18:25.786ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
344132025-01-29T05:18:26.691ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
34414 ----------------------------------------------------------------
34415 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34416 States: Active Active Offline
34417 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34418 9455 Acked Read 6 Done Done Sent false
34419 9456 Acked Write 5 Done Done Sent false
34420 9457 Acked Write 9 Done Done Sent false
34421 9458 Acked Write 6 Done Done Sent false
34422 9459 Acked Write 1 Done Done Sent false
34423 9460 Acked Read 9 Done Done Sent false
34424 9461 Acked Write 2 Done Done Sent false
34425 9462 Acked Read 10 Done Done Sent false
34426 9463 Acked Write 3 Done Done Sent false
34427 9464 Acked Write 3 Done Done Sent false
34428 9465 Acked Read 4 Done Done Sent false
34429 9466 Acked Write 4 Done Done Sent false
34430 9467 Acked Flush 0 Done Done Sent false
34431 9468 Acked Read 3 Done Done Sent false
34432 9469 Acked Flush 0 Done Done Sent false
34433 9470 Acked Write 7 Done Done Sent false
34434 9471 Acked Read 8 Done Done Sent false
34435 9472 Acked Flush 0 Done Done Sent false
34436 9473 Acked Write 6 Done Done Sent false
34437 9474 Acked Read 2 Done Done Sent false
34438 9475 Acked Write 8 Done Done Sent false
34439 9476 Acked Write 9 Done Done Sent false
34440 9477 Acked Read 10 Done Done Sent false
34441 9478 Acked Write 10 Done Done Sent false
34442 9479 Acked Write 2 Done Done Sent false
34443 9480 Acked Write 5 Done Done Sent false
34444 9481 Acked Write 4 Done Done Sent false
34445 9482 Acked Write 4 Done Done Sent false
34446 9483 Acked Read 8 Done Done Sent false
34447 9484 Acked Read 5 Done Done Sent false
34448 9485 Acked Write 3 Done Done Sent false
34449 9486 Acked Write 5 Done Done Sent false
34450 9487 Acked Read 4 Done Done Sent false
34451 9488 Acked Read 10 Done Done Sent false
34452 9489 Acked Flush 0 Done Done Sent false
34453 9490 Acked Write 5 Done Done Sent false
34454 9491 Acked Read 4 Done Done Sent false
34455 9492 Acked Read 6 Done Done Sent false
34456 9493 Acked Read 7 Done Done Sent false
34457 9494 Acked Write 5 Done Done Sent false
34458 9495 Acked Read 6 Done Done Sent false
34459 9496 Acked Write 1 Done Done Sent false
34460 9497 Acked Read 10 Done Done Sent false
34461 9498 Acked Write 6 Done Done Sent false
34462 9499 Acked Read 8 Done Done Sent false
34463 9500 Acked Read 7 Done Done Sent false
34464 9501 Acked Read 8 Done Done Sent false
34465 9502 Acked Write 6 Done Done Sent false
34466 9503 Acked Read 5 Done Done Sent false
34467 9504 Acked Write 5 Done Done Sent false
34468 9505 Acked Read 2 Done Done Sent false
34469 9506 Acked Read 1 Done Done Sent false
34470 9507 Acked Read 9 Done Done Sent false
34471 9508 Acked Write 7 Done Done Sent false
34472 9509 Acked Write 5 Done Done Sent false
34473 9510 Acked Write 1 Done Done Sent false
34474 9511 Acked Read 6 Done Done Sent false
34475 9512 Acked Read 6 Done Done Sent false
34476 9513 Acked Read 6 Done Done Sent false
34477 9514 Acked Read 2 Done Done Sent false
34478 9515 Acked Read 3 Done Done Sent false
34479 9516 Acked Read 5 Done Done Sent false
34480 9517 Acked Write 10 Done Done Sent false
34481 9518 Acked Flush 0 Done Done Sent false
34482 9519 Acked Flush 0 Done Done Sent false
34483 9520 Acked Read 7 Done Done Sent false
34484 9521 Acked Read 2 Done Done Sent false
34485 9522 Acked Read 6 Done Done Sent false
34486 9523 Acked Read 6 Done Done Sent false
34487 9524 Acked Flush 0 Done Done Sent false
34488 9525 Acked Read 7 Done Done Sent false
34489 9526 Acked Write 8 Done Done Sent false
34490 9527 Acked Write 10 Done Done Sent false
34491 9528 Acked Read 9 Done Done Sent false
34492 9529 Acked Flush 0 Done Done Sent false
34493 9530 Acked Read 3 Done Done Sent false
34494 9531 Acked Read 10 Done Done Sent false
34495 9532 Acked Read 5 Done Done Sent false
34496 9533 Acked Write 2 Done Done Sent false
34497 9534 Acked Read 10 Done Done Sent false
34498 9535 Acked Write 2 Done Done Sent false
34499 9536 Acked Read 5 Done Done Sent false
34500 9537 Acked Flush 0 Done Done Sent false
34501 9538 Acked Read 8 Done Done Sent false
34502 9539 Acked Write 4 Done Done Sent false
34503 9540 Acked Flush 0 Done Done Sent false
34504 9541 Acked Write 9 Done Done Sent false
34505 9542 Acked Read 5 Done Done Sent false
34506 9543 Acked Read 2 Done Done Sent false
34507 9544 Acked Flush 0 Done Done Sent false
34508 9545 Acked Read 1 Done Done Sent false
34509 9546 Acked Flush 0 Done Done Sent false
34510 9547 Acked Write 6 Done Done Sent false
34511 9548 Acked Read 3 Done Done Sent false
34512 9549 Acked Write 8 Done Done Sent false
34513 9550 Acked Read 10 Done Done Sent false
34514 9551 Acked Read 4 Done Done Sent false
34515 9552 Acked Write 3 Done Done Sent false
34516 9553 Acked Flush 0 Done Done Sent false
34517 9554 Acked Read 8 Done Done Sent false
34518 9555 Acked Write 3 Done Done Sent false
34519 9556 Acked Write 4 Done Done Sent false
34520 9557 Acked Write 2 Done Done Sent false
34521 9558 Acked Read 3 Done Done Sent false
34522 9559 Acked Read 8 Done Done Sent false
34523 9560 Acked Write 6 Done Done Sent false
34524 9561 Acked Read 2 Done Done Sent false
34525 9562 Acked Read 3 Done Done Sent false
34526 9563 Acked Flush 0 Done Done Sent false
34527 9564 Acked Flush 0 Done Done Sent false
34528 9565 Acked Read 1 Done Done Sent false
34529 9566 Acked Read 8 Done Done Sent false
34530 9567 Acked Read 4 Done Done Sent false
34531 9568 Acked Read 8 Done Done Sent false
34532 9569 Acked Read 7 Done Done Sent false
34533 9570 Acked Read 4 Done Done Sent false
34534 9571 Acked Read 8 Done Done Sent false
34535 9572 Acked Read 3 Done Done Sent false
34536 9573 Acked Read 4 Done Done Sent false
34537 9574 Acked Read 4 Done Done Sent false
34538 9575 Acked Write 6 Done Done Sent false
34539 9576 Acked Read 6 Done Done Sent false
34540 9577 Acked Read 7 Done Done Sent false
34541 9578 Acked Read 7 Done Done Sent false
34542 9579 Acked Write 10 Done Done Sent false
34543 9580 Acked Write 10 Done Done Sent false
34544 9581 Acked Read 3 Done Done Sent false
34545 9582 Acked Write 10 Done Done Sent false
34546 9583 Acked Write 9 Done Done Sent false
34547 9584 Acked Read 1 Done Done Sent false
34548 9585 Acked Read 4 Done Done Sent false
34549 9586 Acked Flush 0 Done Done Sent false
34550 9587 Acked Read 7 Done Done Sent false
34551 9588 Acked Write 1 Done Done Sent false
34552 9589 Acked Read 5 Done Done Sent false
34553 9590 Acked Write 7 Done Done Sent false
34554 9591 Acked Flush 0 Done Done Sent false
34555 9592 Acked Write 2 Done Done Sent false
34556 9593 Acked Read 6 Done Done Sent false
34557 9594 Acked Write 1 Done Done Sent false
34558 9595 Acked Write 7 Done Done Sent false
34559 9596 Acked Read 9 Done Done Sent false
34560 9597 Acked Read 7 Done Done Sent false
34561 9598 Acked Write 4 Done Done Sent false
34562 9599 Acked Read 7 Done Done Sent false
34563 9600 Acked Read 6 Done Done Sent false
34564 9601 Acked Read 4 Done Done Sent false
34565 9602 Acked Read 10 Done Done Sent false
34566 9603 Acked Read 2 Done Done Sent false
34567 9604 Acked Flush 0 Done Done Sent false
34568 9605 Acked Write 1 Done Done Sent false
34569 9606 Acked Write 9 Done Done Sent false
34570 9607 Acked Read 6 Done Done Sent false
34571 9608 Acked Flush 0 Done Done Sent false
34572 9609 Acked Read 10 Done Done Sent false
34573 9610 Acked Read 7 Done Done Sent false
34574 9611 Acked Flush 0 Done Done Sent false
34575 9612 Acked Write 8 Done Done Sent false
34576 9613 Acked Write 10 Done Done Sent false
34577 9614 Acked Read 7 Done Done Sent false
34578 9615 Acked Read 10 Done Done Sent false
34579 9616 Acked Flush 0 Done Done Sent false
34580 9617 Acked Read 1 Done Done Sent false
34581 9618 Acked Write 3 Done Done Sent false
34582 9619 Acked Flush 0 Done Done Sent false
34583 9620 Acked Write 1 Done Done Sent false
34584 9621 Acked Read 4 Done Done Sent false
34585 9622 Acked Flush 0 Done Done Sent false
34586 9623 Acked Read 2 Done Done Sent false
34587 9624 Acked Flush 0 Done Done Sent false
34588 9625 Acked Read 10 Done Done Sent false
34589 9626 Acked Read 9 Done Done Sent false
34590 9627 Acked Read 4 Done Done Sent false
34591 9628 Acked Write 6 Done Done Sent false
34592 9629 Acked Read 1 Done Done Sent false
34593 9630 Acked Flush 0 Done Done Sent false
34594 9631 Acked Write 8 Done Done Sent false
34595 9632 Acked Read 3 Done Done Sent false
34596 9633 Acked Read 3 Done Done Sent false
34597 9634 Acked Read 1 Done Done Sent false
34598 9635 Acked Write 4 Done Done Sent false
34599 9636 Acked Write 10 Done Done Sent false
34600 9637 Acked Flush 0 Done Done Sent false
34601 9638 Acked Read 9 Done Done Sent false
34602 9639 Acked Read 3 Done Done Sent false
34603 9640 Acked Read 4 Done Done Sent false
34604 9641 Acked Write 6 Done Done Sent false
34605 9642 Acked Read 6 Done Done Sent false
34606 9643 Acked Write 10 Done Done Sent false
34607 9644 Acked Read 9 Done Done Sent false
34608 9645 Acked Read 8 Done Done Sent false
34609 9646 Acked Read 8 Done Done Sent false
34610 9647 Acked Write 4 Done Done Sent false
34611 9648 Acked Flush 0 Done Done Sent false
34612 9649 Acked Write 1 Done Done Sent false
34613 9650 Acked Write 4 Done Done Sent false
34614 9651 Acked Read 6 Done Done Sent false
34615 9652 Acked Read 6 Done Done Sent false
34616 9653 Acked Read 7 Done Done Sent false
34617 9654 Acked Write 1 Done Done Sent false
34618 9655 Acked Read 10 Done Done Sent false
34619 9656 Acked Write 7 Done Done Sent false
34620 9657 Acked Read 5 Done Done Sent false
34621 9658 Acked Read 9 Done Done Sent false
34622 9659 Acked Flush 0 Done Done Sent false
34623 9660 Acked Read 4 Done Done Sent false
34624 9661 Acked Write 6 Done Done Sent false
34625 9662 Acked Read 2 Done Done Sent false
34626 9663 Acked Read 4 Done Done Sent false
34627 9664 Acked Write 1 Done Done Sent false
34628 9665 Acked Write 8 Done Done Sent false
34629 9666 Acked Write 3 Done Done Sent false
34630 9667 Acked Write 8 Done Done Sent false
34631 9668 Acked Read 10 Done Done Sent false
34632 9669 Acked Read 5 Done Done Sent false
34633 9670 Acked Read 2 Done Done Sent false
34634 9671 Acked Read 6 Done Done Sent false
34635 9672 Acked Write 10 Done Done Sent false
34636 9673 Acked Write 1 Done Done Sent false
34637 9674 Acked Read 10 Done Done Sent false
34638 9675 Acked Flush 0 Done Done Sent false
34639 9676 Acked Read 9 Done Done Sent false
34640 9677 Acked Write 8 Done Done Sent false
34641 9678 Acked Read 6 Done Done Sent false
34642 9679 Acked Read 1 Done Done Sent false
34643 9680 Acked Write 4 Done Done Sent false
34644 9681 Acked Read 4 Done Done Sent false
34645 9682 Acked Flush 0 Done Done Sent false
34646 9683 Acked Read 9 Done Done Sent false
34647 9684 Acked Read 3 Done Done Sent false
34648 9685 Acked Write 5 Done Done Sent false
34649 9686 Acked Read 9 Done Done Sent false
34650 9687 Acked Flush 0 Done Done Sent false
34651 9688 Acked Write 1 Done Done Sent false
34652 9689 Acked Read 4 Done Done Sent false
34653 9690 Acked Flush 0 Done Done Sent false
34654 9691 Acked Read 8 Done Done Sent false
34655 9692 Acked Read 7 Done Done Sent false
34656 9693 Acked Read 8 Done Done Sent false
34657 9694 Acked Read 6 Done Done Sent false
34658 9695 Acked Write 5 Done Done Sent false
34659 9696 Acked Write 3 Done Done Sent false
34660 9697 Acked Read 6 Done Done Sent false
34661 9698 Acked Read 10 Done Done Sent false
34662 9699 Acked Write 8 Done Done Sent false
34663 9700 Acked Read 3 Done Done Sent false
34664 9701 Acked Read 9 Done Done Sent false
34665 9702 Acked Flush 0 Done Done Sent false
34666 9703 Acked Read 10 Done Done Sent false
34667 9704 Acked Read 10 Done Done Sent false
34668 9705 Acked Write 1 Done Done Sent false
34669 9706 Acked Read 4 Done Done Sent false
34670 9707 Acked Write 4 Done Done Sent false
34671 9708 Acked Write 2 Done Done Sent false
34672 9709 Acked Read 6 Done Done Sent false
34673 9710 Acked Flush 0 Done Done Sent false
34674 9711 Acked Read 8 Done Done Sent false
34675 9712 Acked Read 10 Done Done Sent false
34676 9713 Acked Read 6 Done Done Sent false
34677 9714 Acked Write 8 Done Done Sent false
34678 9715 Acked Read 4 Done Done Sent false
34679 9716 Acked Read 10 Done Done Sent false
34680 9717 Acked Read 10 Done Done Sent false
34681 9718 Acked Read 1 Done Done Sent false
34682 9719 Acked Read 8 Done Done Sent false
34683 9720 Acked Read 9 Done Done Sent false
34684 9721 Acked Write 9 Done Done Sent false
34685 9722 Acked Flush 0 Done Done Sent false
34686 9723 Acked Write 10 Done Done Sent false
34687 9724 Acked Flush 0 Done Done Sent false
34688 9725 Acked Flush 0 Done Done Sent false
34689 9726 Acked Read 2 Done Done Sent false
34690 9727 Acked Write 5 Done Done Sent false
34691 9728 Acked Read 9 Done Done Sent false
34692 9729 Acked Write 5 Done Done Sent false
34693 9730 Acked Read 2 Done Done Sent false
34694 9731 Acked Write 8 Done Done Sent false
34695 9732 Acked Write 10 Done Done Sent false
34696 9733 Acked Write 8 Done Done Sent false
34697 9734 Acked Write 8 Done Done Sent false
34698 9735 Acked Read 7 Done Done Sent false
34699 9736 Acked Write 7 Done Done Sent false
34700 9737 Acked Write 6 Done Done Sent false
34701 9738 Acked Write 2 Done Done Sent false
34702 9739 Acked Write 1 Done Done Sent false
34703 9740 Acked Write 5 Done Done Sent false
34704 9741 Acked Write 5 Done Done Sent false
34705 9742 Acked Read 5 Done Done Sent false
34706 9743 Acked Write 3 Done Done Sent false
34707 9744 Acked Read 8 Done Done Sent false
34708 9745 Acked Read 4 Done Done Sent false
34709 9746 Acked Write 5 Done Done Sent false
34710 9747 Acked Read 2 Done Done Sent false
34711 9748 Acked Read 6 Done Done Sent false
34712 9749 Acked Read 5 Done Done Sent false
34713 9750 Acked Write 9 Done Done Sent false
34714 9751 Acked Read 10 Done Done Sent false
34715 9752 Acked Write 6 Done Done Sent false
34716 9753 Acked Write 2 Done Done Sent false
34717 9754 Acked Read 6 Done Done Sent false
34718 9755 Acked Read 6 Done Done Sent false
34719 STATES DS:0 DS:1 DS:2 TOTAL
34720 Sent 0 0 301 301
34721 Done 301 301 0 602
34722 Skipped 0 0 0 0
34723 Error 0 0 0 0
34724 Last Flush: Some(JobId(9725)) Some(JobId(9725)) Some(JobId(9454))
34725 Downstairs last five completed: 9454 9453 9452 9451 9450
34726 Upstairs last five completed: 9755 9754 9753 9752 9751
34727 CLIENT: Up:0 ds:301 act:2
347282025-01-29T05:18:27.694ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
347292025-01-29T05:18:28.696ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
347302025-01-29T05:18:29.698ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
347312025-01-29T05:18:30.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
34732 ----------------------------------------------------------------
34733 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34734 States: Active Active Offline
34735 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34736 9455 Acked Read 6 Done Done Sent false
34737 9456 Acked Write 5 Done Done Sent false
34738 9457 Acked Write 9 Done Done Sent false
34739 9458 Acked Write 6 Done Done Sent false
34740 9459 Acked Write 1 Done Done Sent false
34741 9460 Acked Read 9 Done Done Sent false
34742 9461 Acked Write 2 Done Done Sent false
34743 9462 Acked Read 10 Done Done Sent false
34744 9463 Acked Write 3 Done Done Sent false
34745 9464 Acked Write 3 Done Done Sent false
34746 9465 Acked Read 4 Done Done Sent false
34747 9466 Acked Write 4 Done Done Sent false
34748 9467 Acked Flush 0 Done Done Sent false
34749 9468 Acked Read 3 Done Done Sent false
34750 9469 Acked Flush 0 Done Done Sent false
34751 9470 Acked Write 7 Done Done Sent false
34752 9471 Acked Read 8 Done Done Sent false
34753 9472 Acked Flush 0 Done Done Sent false
34754 9473 Acked Write 6 Done Done Sent false
34755 9474 Acked Read 2 Done Done Sent false
34756 9475 Acked Write 8 Done Done Sent false
34757 9476 Acked Write 9 Done Done Sent false
34758 9477 Acked Read 10 Done Done Sent false
34759 9478 Acked Write 10 Done Done Sent false
34760 9479 Acked Write 2 Done Done Sent false
34761 9480 Acked Write 5 Done Done Sent false
34762 9481 Acked Write 4 Done Done Sent false
34763 9482 Acked Write 4 Done Done Sent false
34764 9483 Acked Read 8 Done Done Sent false
34765 9484 Acked Read 5 Done Done Sent false
34766 9485 Acked Write 3 Done Done Sent false
34767 9486 Acked Write 5 Done Done Sent false
34768 9487 Acked Read 4 Done Done Sent false
34769 9488 Acked Read 10 Done Done Sent false
34770 9489 Acked Flush 0 Done Done Sent false
34771 9490 Acked Write 5 Done Done Sent false
34772 9491 Acked Read 4 Done Done Sent false
34773 9492 Acked Read 6 Done Done Sent false
34774 9493 Acked Read 7 Done Done Sent false
34775 9494 Acked Write 5 Done Done Sent false
34776 9495 Acked Read 6 Done Done Sent false
34777 9496 Acked Write 1 Done Done Sent false
34778 9497 Acked Read 10 Done Done Sent false
34779 9498 Acked Write 6 Done Done Sent false
34780 9499 Acked Read 8 Done Done Sent false
34781 9500 Acked Read 7 Done Done Sent false
34782 9501 Acked Read 8 Done Done Sent false
34783 9502 Acked Write 6 Done Done Sent false
34784 9503 Acked Read 5 Done Done Sent false
34785 9504 Acked Write 5 Done Done Sent false
34786 9505 Acked Read 2 Done Done Sent false
34787 9506 Acked Read 1 Done Done Sent false
34788 9507 Acked Read 9 Done Done Sent false
34789 9508 Acked Write 7 Done Done Sent false
34790 9509 Acked Write 5 Done Done Sent false
34791 9510 Acked Write 1 Done Done Sent false
34792 9511 Acked Read 6 Done Done Sent false
34793 9512 Acked Read 6 Done Done Sent false
34794 9513 Acked Read 6 Done Done Sent false
34795 9514 Acked Read 2 Done Done Sent false
34796 9515 Acked Read 3 Done Done Sent false
34797 9516 Acked Read 5 Done Done Sent false
34798 9517 Acked Write 10 Done Done Sent false
34799 9518 Acked Flush 0 Done Done Sent false
34800 9519 Acked Flush 0 Done Done Sent false
34801 9520 Acked Read 7 Done Done Sent false
34802 9521 Acked Read 2 Done Done Sent false
34803 9522 Acked Read 6 Done Done Sent false
34804 9523 Acked Read 6 Done Done Sent false
34805 9524 Acked Flush 0 Done Done Sent false
34806 9525 Acked Read 7 Done Done Sent false
34807 9526 Acked Write 8 Done Done Sent false
34808 9527 Acked Write 10 Done Done Sent false
34809 9528 Acked Read 9 Done Done Sent false
34810 9529 Acked Flush 0 Done Done Sent false
34811 9530 Acked Read 3 Done Done Sent false
34812 9531 Acked Read 10 Done Done Sent false
34813 9532 Acked Read 5 Done Done Sent false
34814 9533 Acked Write 2 Done Done Sent false
34815 9534 Acked Read 10 Done Done Sent false
34816 9535 Acked Write 2 Done Done Sent false
34817 9536 Acked Read 5 Done Done Sent false
34818 9537 Acked Flush 0 Done Done Sent false
34819 9538 Acked Read 8 Done Done Sent false
34820 9539 Acked Write 4 Done Done Sent false
34821 9540 Acked Flush 0 Done Done Sent false
34822 9541 Acked Write 9 Done Done Sent false
34823 9542 Acked Read 5 Done Done Sent false
34824 9543 Acked Read 2 Done Done Sent false
34825 9544 Acked Flush 0 Done Done Sent false
34826 9545 Acked Read 1 Done Done Sent false
34827 9546 Acked Flush 0 Done Done Sent false
34828 9547 Acked Write 6 Done Done Sent false
34829 9548 Acked Read 3 Done Done Sent false
34830 9549 Acked Write 8 Done Done Sent false
34831 9550 Acked Read 10 Done Done Sent false
34832 9551 Acked Read 4 Done Done Sent false
34833 9552 Acked Write 3 Done Done Sent false
34834 9553 Acked Flush 0 Done Done Sent false
34835 9554 Acked Read 8 Done Done Sent false
34836 9555 Acked Write 3 Done Done Sent false
34837 9556 Acked Write 4 Done Done Sent false
34838 9557 Acked Write 2 Done Done Sent false
34839 9558 Acked Read 3 Done Done Sent false
34840 9559 Acked Read 8 Done Done Sent false
34841 9560 Acked Write 6 Done Done Sent false
34842 9561 Acked Read 2 Done Done Sent false
34843 9562 Acked Read 3 Done Done Sent false
34844 9563 Acked Flush 0 Done Done Sent false
34845 9564 Acked Flush 0 Done Done Sent false
34846 9565 Acked Read 1 Done Done Sent false
34847 9566 Acked Read 8 Done Done Sent false
34848 9567 Acked Read 4 Done Done Sent false
34849 9568 Acked Read 8 Done Done Sent false
34850 9569 Acked Read 7 Done Done Sent false
34851 9570 Acked Read 4 Done Done Sent false
34852 9571 Acked Read 8 Done Done Sent false
34853 9572 Acked Read 3 Done Done Sent false
34854 9573 Acked Read 4 Done Done Sent false
34855 9574 Acked Read 4 Done Done Sent false
34856 9575 Acked Write 6 Done Done Sent false
34857 9576 Acked Read 6 Done Done Sent false
34858 9577 Acked Read 7 Done Done Sent false
34859 9578 Acked Read 7 Done Done Sent false
34860 9579 Acked Write 10 Done Done Sent false
34861 9580 Acked Write 10 Done Done Sent false
34862 9581 Acked Read 3 Done Done Sent false
34863 9582 Acked Write 10 Done Done Sent false
34864 9583 Acked Write 9 Done Done Sent false
34865 9584 Acked Read 1 Done Done Sent false
34866 9585 Acked Read 4 Done Done Sent false
34867 9586 Acked Flush 0 Done Done Sent false
34868 9587 Acked Read 7 Done Done Sent false
34869 9588 Acked Write 1 Done Done Sent false
34870 9589 Acked Read 5 Done Done Sent false
34871 9590 Acked Write 7 Done Done Sent false
34872 9591 Acked Flush 0 Done Done Sent false
34873 9592 Acked Write 2 Done Done Sent false
34874 9593 Acked Read 6 Done Done Sent false
34875 9594 Acked Write 1 Done Done Sent false
34876 9595 Acked Write 7 Done Done Sent false
34877 9596 Acked Read 9 Done Done Sent false
34878 9597 Acked Read 7 Done Done Sent false
34879 9598 Acked Write 4 Done Done Sent false
34880 9599 Acked Read 7 Done Done Sent false
34881 9600 Acked Read 6 Done Done Sent false
34882 9601 Acked Read 4 Done Done Sent false
34883 9602 Acked Read 10 Done Done Sent false
34884 9603 Acked Read 2 Done Done Sent false
34885 9604 Acked Flush 0 Done Done Sent false
34886 9605 Acked Write 1 Done Done Sent false
34887 9606 Acked Write 9 Done Done Sent false
34888 9607 Acked Read 6 Done Done Sent false
34889 9608 Acked Flush 0 Done Done Sent false
34890 9609 Acked Read 10 Done Done Sent false
34891 9610 Acked Read 7 Done Done Sent false
34892 9611 Acked Flush 0 Done Done Sent false
34893 9612 Acked Write 8 Done Done Sent false
34894 9613 Acked Write 10 Done Done Sent false
34895 9614 Acked Read 7 Done Done Sent false
34896 9615 Acked Read 10 Done Done Sent false
34897 9616 Acked Flush 0 Done Done Sent false
34898 9617 Acked Read 1 Done Done Sent false
34899 9618 Acked Write 3 Done Done Sent false
34900 9619 Acked Flush 0 Done Done Sent false
34901 9620 Acked Write 1 Done Done Sent false
34902 9621 Acked Read 4 Done Done Sent false
34903 9622 Acked Flush 0 Done Done Sent false
34904 9623 Acked Read 2 Done Done Sent false
34905 9624 Acked Flush 0 Done Done Sent false
34906 9625 Acked Read 10 Done Done Sent false
34907 9626 Acked Read 9 Done Done Sent false
34908 9627 Acked Read 4 Done Done Sent false
34909 9628 Acked Write 6 Done Done Sent false
34910 9629 Acked Read 1 Done Done Sent false
34911 9630 Acked Flush 0 Done Done Sent false
34912 9631 Acked Write 8 Done Done Sent false
34913 9632 Acked Read 3 Done Done Sent false
34914 9633 Acked Read 3 Done Done Sent false
34915 9634 Acked Read 1 Done Done Sent false
34916 9635 Acked Write 4 Done Done Sent false
34917 9636 Acked Write 10 Done Done Sent false
34918 9637 Acked Flush 0 Done Done Sent false
34919 9638 Acked Read 9 Done Done Sent false
34920 9639 Acked Read 3 Done Done Sent false
34921 9640 Acked Read 4 Done Done Sent false
34922 9641 Acked Write 6 Done Done Sent false
34923 9642 Acked Read 6 Done Done Sent false
34924 9643 Acked Write 10 Done Done Sent false
34925 9644 Acked Read 9 Done Done Sent false
34926 9645 Acked Read 8 Done Done Sent false
34927 9646 Acked Read 8 Done Done Sent false
34928 9647 Acked Write 4 Done Done Sent false
34929 9648 Acked Flush 0 Done Done Sent false
34930 9649 Acked Write 1 Done Done Sent false
34931 9650 Acked Write 4 Done Done Sent false
34932 9651 Acked Read 6 Done Done Sent false
34933 9652 Acked Read 6 Done Done Sent false
34934 9653 Acked Read 7 Done Done Sent false
34935 9654 Acked Write 1 Done Done Sent false
34936 9655 Acked Read 10 Done Done Sent false
34937 9656 Acked Write 7 Done Done Sent false
34938 9657 Acked Read 5 Done Done Sent false
34939 9658 Acked Read 9 Done Done Sent false
34940 9659 Acked Flush 0 Done Done Sent false
34941 9660 Acked Read 4 Done Done Sent false
34942 9661 Acked Write 6 Done Done Sent false
34943 9662 Acked Read 2 Done Done Sent false
34944 9663 Acked Read 4 Done Done Sent false
34945 9664 Acked Write 1 Done Done Sent false
34946 9665 Acked Write 8 Done Done Sent false
34947 9666 Acked Write 3 Done Done Sent false
34948 9667 Acked Write 8 Done Done Sent false
34949 9668 Acked Read 10 Done Done Sent false
34950 9669 Acked Read 5 Done Done Sent false
34951 9670 Acked Read 2 Done Done Sent false
34952 9671 Acked Read 6 Done Done Sent false
34953 9672 Acked Write 10 Done Done Sent false
34954 9673 Acked Write 1 Done Done Sent false
34955 9674 Acked Read 10 Done Done Sent false
34956 9675 Acked Flush 0 Done Done Sent false
34957 9676 Acked Read 9 Done Done Sent false
34958 9677 Acked Write 8 Done Done Sent false
34959 9678 Acked Read 6 Done Done Sent false
34960 9679 Acked Read 1 Done Done Sent false
34961 9680 Acked Write 4 Done Done Sent false
34962 9681 Acked Read 4 Done Done Sent false
34963 9682 Acked Flush 0 Done Done Sent false
34964 9683 Acked Read 9 Done Done Sent false
34965 9684 Acked Read 3 Done Done Sent false
34966 9685 Acked Write 5 Done Done Sent false
34967 9686 Acked Read 9 Done Done Sent false
34968 9687 Acked Flush 0 Done Done Sent false
34969 9688 Acked Write 1 Done Done Sent false
34970 9689 Acked Read 4 Done Done Sent false
34971 9690 Acked Flush 0 Done Done Sent false
34972 9691 Acked Read 8 Done Done Sent false
34973 9692 Acked Read 7 Done Done Sent false
34974 9693 Acked Read 8 Done Done Sent false
34975 9694 Acked Read 6 Done Done Sent false
34976 9695 Acked Write 5 Done Done Sent false
34977 9696 Acked Write 3 Done Done Sent false
34978 9697 Acked Read 6 Done Done Sent false
34979 9698 Acked Read 10 Done Done Sent false
34980 9699 Acked Write 8 Done Done Sent false
34981 9700 Acked Read 3 Done Done Sent false
34982 9701 Acked Read 9 Done Done Sent false
34983 9702 Acked Flush 0 Done Done Sent false
34984 9703 Acked Read 10 Done Done Sent false
34985 9704 Acked Read 10 Done Done Sent false
34986 9705 Acked Write 1 Done Done Sent false
34987 9706 Acked Read 4 Done Done Sent false
34988 9707 Acked Write 4 Done Done Sent false
34989 9708 Acked Write 2 Done Done Sent false
34990 9709 Acked Read 6 Done Done Sent false
34991 9710 Acked Flush 0 Done Done Sent false
34992 9711 Acked Read 8 Done Done Sent false
34993 9712 Acked Read 10 Done Done Sent false
34994 9713 Acked Read 6 Done Done Sent false
34995 9714 Acked Write 8 Done Done Sent false
34996 9715 Acked Read 4 Done Done Sent false
34997 9716 Acked Read 10 Done Done Sent false
34998 9717 Acked Read 10 Done Done Sent false
34999 9718 Acked Read 1 Done Done Sent false
35000 9719 Acked Read 8 Done Done Sent false
35001 9720 Acked Read 9 Done Done Sent false
35002 9721 Acked Write 9 Done Done Sent false
35003 9722 Acked Flush 0 Done Done Sent false
35004 9723 Acked Write 10 Done Done Sent false
35005 9724 Acked Flush 0 Done Done Sent false
35006 9725 Acked Flush 0 Done Done Sent false
35007 9726 Acked Read 2 Done Done Sent false
35008 9727 Acked Write 5 Done Done Sent false
35009 9728 Acked Read 9 Done Done Sent false
35010 9729 Acked Write 5 Done Done Sent false
35011 9730 Acked Read 2 Done Done Sent false
35012 9731 Acked Write 8 Done Done Sent false
35013 9732 Acked Write 10 Done Done Sent false
35014 9733 Acked Write 8 Done Done Sent false
35015 9734 Acked Write 8 Done Done Sent false
35016 9735 Acked Read 7 Done Done Sent false
35017 9736 Acked Write 7 Done Done Sent false
35018 9737 Acked Write 6 Done Done Sent false
35019 9738 Acked Write 2 Done Done Sent false
35020 9739 Acked Write 1 Done Done Sent false
35021 9740 Acked Write 5 Done Done Sent false
35022 9741 Acked Write 5 Done Done Sent false
35023 9742 Acked Read 5 Done Done Sent false
35024 9743 Acked Write 3 Done Done Sent false
35025 9744 Acked Read 8 Done Done Sent false
35026 9745 Acked Read 4 Done Done Sent false
35027 9746 Acked Write 5 Done Done Sent false
35028 9747 Acked Read 2 Done Done Sent false
35029 9748 Acked Read 6 Done Done Sent false
35030 9749 Acked Read 5 Done Done Sent false
35031 9750 Acked Write 9 Done Done Sent false
35032 9751 Acked Read 10 Done Done Sent false
35033 9752 Acked Write 6 Done Done Sent false
35034 9753 Acked Write 2 Done Done Sent false
35035 9754 Acked Read 6 Done Done Sent false
35036 9755 Acked Read 6 Done Done Sent false
35037 STATES DS:0 DS:1 DS:2 TOTAL
35038 Sent 0 0 301 301
35039 Done 301 301 0 602
35040 Skipped 0 0 0 0
35041 Error 0 0 0 0
35042 Last Flush: Some(JobId(9725)) Some(JobId(9725)) Some(JobId(9454))
35043 Downstairs last five completed: 9454 9453 9452 9451 9450
35044 Upstairs last five completed: 9755 9754 9753 9752 9751
35045 CLIENT: Up:0 ds:301 act:2
350462025-01-29T05:18:31.700ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350472025-01-29T05:18:32.701ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350482025-01-29T05:18:33.703ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350492025-01-29T05:18:34.540ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350502025-01-29T05:18:34.540ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39454) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350512025-01-29T05:18:34.541ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350522025-01-29T05:18:34.542ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350532025-01-29T05:18:34.542ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350542025-01-29T05:18:34.542ZINFOcrucible: send last flush ID to this DS: Some(JobId(9454)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350552025-01-29T05:18:34.542ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350562025-01-29T05:18:34.542ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9454)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
350572025-01-29T05:18:34.542ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9454)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
35058 ----------------------------------------------------------------
35059 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:30
35060 States: Active Active Active
35061 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35062 9726 Acked Read 2 Done Done Done true
35063 9727 Acked Write 5 Done Done Done true
35064 9728 Acked Read 9 Done Done Done true
35065 9729 Acked Write 5 Done Done Done true
35066 9730 Acked Read 2 Done Done Done true
35067 9731 Acked Write 8 Done Done Done true
35068 9732 Acked Write 10 Done Done Done true
35069 9733 Acked Write 8 Done Done Done true
35070 9734 Acked Write 8 Done Done Done true
35071 9735 Acked Read 7 Done Done Done true
35072 9736 Acked Write 7 Done Done Done true
35073 9737 Acked Write 6 Done Done Done true
35074 9738 Acked Write 2 Done Done Done true
35075 9739 Acked Write 1 Done Done Done true
35076 9740 Acked Write 5 Done Done Done true
35077 9741 Acked Write 5 Done Done Done true
35078 9742 Acked Read 5 Done Done Done true
35079 9743 Acked Write 3 Done Done Done true
35080 9744 Acked Read 8 Done Done Done true
35081 9745 Acked Read 4 Done Done Done true
35082 9746 Acked Write 5 Done Done Done true
35083 9747 Acked Read 2 Done Done Done true
35084 9748 Acked Read 6 Done Done Done true
35085 9749 Acked Read 5 Done Done Done true
35086 9750 Acked Write 9 Done Done Done true
35087 9751 Acked Read 10 Done Done Done true
35088 9752 Acked Write 6 Done Done Done true
35089 9753 Acked Write 2 Done Done Done true
35090 9754 Acked Read 6 Done Done Done true
35091 9755 Acked Read 6 Done Done Done true
35092 STATES DS:0 DS:1 DS:2 TOTAL
35093 Sent 0 0 0 0
35094 Done 30 30 30 90
35095 Skipped 0 0 0 0
35096 Error 0 0 0 0
35097 Last Flush: Some(JobId(9725)) Some(JobId(9725)) Some(JobId(9725))
35098 Downstairs last five completed: 9725 9724 9723 9722 9721
35099 Upstairs last five completed: 9755 9754 9753 9752 9751
35100 CLIENT: Up:0 ds:30 act:3
35101 ----------------------------------------------------------------
35102 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35103 Downstairs last five completed: 9756 9755 9754 9753 9752
35104 Upstairs last five completed: 9756 9755 9754 9753 9752
35105 CLIENT: Up:0 ds:0 act:3
35106 Replay: All jobs finished
35107 Main task has work to do, go find it
35108 got dsc Stop(1)
35109 stop 1
35110 Jan 29 05:18:38.802 [8820] Got stop action so:false kr:false
35111 INFO request completed, latency_us: 377, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 33f44b59-6e23-4300-9b46-2f40278726a5, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
351122025-01-29T05:18:38.803ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
351132025-01-29T05:18:38.803ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
35114 Jan 29 05:18:38.803 INFO request completed, latency_us: 125, response_code: 200, uri: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-01-29T05:18:38.803349449Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
35115 /state/cid/1, method: GET, req_id: ff4aa8fb-c562-4f6d-b6ba-91751fb2fb7b, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
351162025-01-29T05:18:38.803ZINFOcrucible: client task is exiting = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
35117 [8820] Exited with: ExitStatus(unix_wait_status(9))
35118 [8820][1] reports Exit
35119 Jan 29 05:18:42.805 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/1, method: GET, req_id: 60f7afac-6d6d-4fb0-a086-fa3fbf607851, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
35120 001/300 Read block 2202 len 28672
35121 002/300 Write block 3769 len 16384 data: 9 7 7 7
35122 003/300 Read block 1107 len 24576
35123 004/300 Flush
35124 005/300 Read block 4602 len 24576
35125 006/300 Write block 2430 len 32768 data: 5 6 5 5 6 6 6 6
35126 007/300 Write block 1691 len 16384 data: 15 9 10 6
35127 008/300 Write block 1599 len 32768 data: 6 7 3 3 4 4 4 4
35128 009/300 Read block 3215 len 32768
35129 010/300 Write block 4806 len 28672 data: 4 5 5 6 8 7 7
35130 011/300 Write block 3456 len 32768 data: 4 4 6 6 8 8 9 9
35131 012/300 Write block 4944 len 12288 data: 4 3 4
35132 013/300 Read block 894 len 8192
35133 014/300 Write block 3715 len 12288 data: 5 7 6
35134 015/300 Read block 2315 len 16384
35135 016/300 Read block 749 len 20480
35136 017/300 Write block 2766 len 16384 data: 3 3 3 3
35137 018/300 Write block 540 len 28672 data: 6 5 7 8 7 6 7
35138 019/300 Read block 2230 len 24576
35139 020/300 Write block 1152 len 16384 data: 4 3 4 4
35140 021/300 Read block 4027 len 4096
35141 022/300 Read block 1972 len 8192
35142 023/300 Read block 2636 len 32768
35143 024/300 Read block 839 len 32768
35144 025/300 Read block 4647 len 36864
35145 026/300 Write block 406 len 20480 data: 5 6 6 6 5
35146 027/300 Read block 2014 len 36864
35147 028/300 Write block 2176 len 4096 data: 2
35148 029/300 Write block 4568 len 32768 data: 7 8 9 8 6 4 2 4
35149 030/300 Read block 2401 len 16384
35150 031/300 Flush
35151 032/300 Write block 2639 len 24576 data: 8 5 5 4 5 7
35152 033/300 Flush
35153 034/300 Read block 3019 len 28672
35154 035/300 Write block 2345 len 32768 data: 7 8 8 7 6 7 7 7
35155 036/300 Read block 337 len 36864
35156 037/300 Read block 1787 len 32768
35157 038/300 Read block 1569 len 40960
35158 039/300 Read block 4880 len 12288
35159 040/300 Read block 3866 len 28672
35160 041/300 Flush
35161 042/300 Write block 223 len 16384 data: 4 6 7 8
35162 043/300 Read block 2052 len 8192
35163 044/300 Write block 3709 len 12288 data: 3 2 4
35164 045/300 Read block 3024 len 32768
35165 046/300 Write block 4488 len 20480 data: 4 4 6 6 6
35166 047/300 Write block 4659 len 28672 data: 8 7 7 6 7 5 6
35167 048/300 Write block 1221 len 4096 data: 4
35168 049/300 Write block 1123 len 12288 data: 8 10 10
35169 050/300 Read block 3313 len 4096
35170 051/300 Read block 4226 len 32768
35171 052/300 Read block 2920 len 12288
35172 053/300 Read block 2364 len 24576
35173 054/300 Flush
35174 055/300 Read block 4857 len 4096
35175 056/300 Write block 3526 len 32768 data: 7 7 7 8 4 7 7 7
35176 057/300 Write block 4964 len 36864 data: 10 9 8 9 7 7 4 4 3
35177 058/300 Read block 2059 len 40960
35178 059/300 Read block 1472 len 8192
35179 060/300 Write block 205 len 28672 data: 3 3 3 3 3 2 3
35180 061/300 Write block 2302 len 40960 data: 5 3 3 3 3 3 3 5 7 8
35181 062/300 Write block 3976 len 16384 data: 10 11 11 11
35182 063/300 Write block 861 len 12288 data: 6 8 7
35183 064/300 Read block 4202 len 4096
35184 065/300 Read block 1472 len 8192
35185 066/300 Write block 1200 len 32768 data: 6 6 4 4 3 2 3 3
35186 067/300 Read block 4560 len 24576
35187 068/300 Read block 2651 len 12288
35188 069/300 Read block 4126 len 36864
35189 070/300 Read block 4144 len 40960
35190 071/300 Read block 1266 len 12288
35191 072/300 Read block 1634 len 8192
35192 073/300 Write block 1554 len 36864 data: 6 5 4 4 5 6 6 6 6
35193 074/300 Write block 2394 len 4096 data: 3
35194 075/300 Write block 2541 len 32768 data: 2 2 3 4 4 6 7 9
35195 076/300 Write block 4537 len 32768 data: 8 6 6 5 5 5 5 5
35196 077/300 Read block 3340 len 24576
35197 078/300 Write block 4437 len 36864 data: 3 4 6 5 4 3 3 3 4
35198 079/300 Read block 4803 len 36864
35199 080/300 Read block 4436 len 8192
35200 081/300 Read block 1440 len 16384
35201 082/300 Read block 3085 len 12288
35202 083/300 Write block 2673 len 24576 data: 6 6 5 5 4 3
35203 084/300 Write block 242 len 32768 data: 10 9 9 10 9 6 5 5
35204 085/300 Flush
35205 086/300 Read block 2742 len 20480
35206 087/300 Write block 4030 len 40960 data: 3 3 3 6 5 4 4 3 4 4
35207 088/300 Read block 1559 len 40960
35208 089/300 Write block 2086 len 24576 data: 6 5 3 4 4 7
35209 090/300 Write block 91 len 16384 data: 6 4 3 4
35210 091/300 Read block 659 len 24576
35211 092/300 Flush
35212 093/300 Write block 4674 len 12288 data: 4 5 5
35213 094/300 Write block 2006 len 8192 data: 11 7
35214 095/300 Write block 882 len 8192 data: 5 6
35215 096/300 Write block 3483 len 40960 data: 10 8 7 8 6 5 6 7 6 6
35216 097/300 Write block 90 len 12288 data: 6 7 5
35217 098/300 Read block 4888 len 32768
35218 099/300 Flush
35219 100/300 Read block 1235 len 24576
35220 101/300 Read block 2779 len 32768
35221 102/300 Read block 487 len 32768
35222 103/300 Read block 4209 len 32768
35223 104/300 Read block 2685 len 32768
35224 105/300 Flush
35225 106/300 Read block 3200 len 36864
35226 107/300 Read block 3496 len 4096
35227 108/300 Write block 1215 len 40960 data: 5 5 5 5 5 5 5 4 5 6
35228 109/300 Write block 160 len 16384 data: 5 5 4 5
35229 110/300 Write block 759 len 20480 data: 6 5 4 4 4
35230 111/300 Write block 1714 len 12288 data: 5 5 5
35231 112/300 Write block 3556 len 12288 data: 2 2 3
35232 113/300 Read block 2868 len 36864
35233 114/300 Write block 2861 len 24576 data: 5 6 7 7 7 6
35234 115/300 Read block 1054 len 20480
35235 116/300 Read block 4513 len 20480
35236 117/300 Read block 4576 len 8192
35237 118/300 Read block 4601 len 20480
35238 119/300 Write block 896 len 20480 data: 6 6 9 9 9
35239 120/300 Read block 745 len 40960
35240 121/300 Read block 126 len 12288
35241 122/300 Read block 4778 len 32768
35242 123/300 Flush
35243 124/300 Flush
35244 125/300 Read block 1825 len 36864
35245 126/300 Write block 231 len 8192 data: 7 7
35246 127/300 Read block 1500 len 24576
35247 128/300 Write block 3543 len 16384 data: 6 8 7 6
35248 129/300 Read block 2027 len 12288
35249 130/300 Read block 3662 len 8192
35250 131/300 Write block 4348 len 24576 data: 3 4 6 5 5 7
35251 132/300 Flush
35252 133/300 Read block 2278 len 36864
35253 134/300 Read block 1730 len 12288
35254 135/300 Write block 749 len 24576 data: 10 9 8 9 9 8
35255 136/300 Write block 778 len 40960 data: 10 9 7 7 6 6 4 4 3 5
35256 137/300 Read block 786 len 40960
35257 138/300 Flush
35258 139/300 Read block 4454 len 20480
35259 140/300 Flush
35260 141/300 Write block 3273 len 4096 data: 5
35261 142/300 Read block 2613 len 12288
35262 143/300 Read block 2114 len 16384
35263 144/300 Read block 3788 len 32768
35264 145/300 Write block 477 len 28672 data: 6 8 9 9 11 11 10
35265 146/300 Read block 1694 len 24576
35266 147/300 Read block 3262 len 28672
35267 148/300 Write block 1792 len 32768 data: 11 10 9 10 7 7 5 4
35268 149/300 Write block 4823 len 32768 data: 4 4 3 3 3 2 3 4
35269 150/300 Write block 4167 len 8192 data: 6 6
35270 151/300 Read block 3958 len 28672
35271 152/300 Read block 638 len 32768
35272 153/300 Write block 925 len 20480 data: 4 4 2 3 3
35273 154/300 Write block 2466 len 8192 data: 6 6
35274 155/300 Read block 3926 len 12288
35275 156/300 Flush
35276 157/300 Flush
35277 158/300 Read block 1646 len 36864
35278 159/300 Read block 3592 len 8192
35279 160/300 Write block 4888 len 32768 data: 9 9 9 6 4 4 5 6
35280 161/300 Read block 3628 len 32768
35281 162/300 Flush
35282 163/300 Read block 2228 len 36864
35283 164/300 Read block 757 len 40960
35284 165/300 Write block 4841 len 40960 data: 8 7 6 7 5 5 4 6 6 6
35285 166/300 Write block 2867 len 16384 data: 6 7 8 5
35286 167/300 Write block 2545 len 12288 data: 5 7 8
35287 168/300 Write block 3084 len 20480 data: 4 5 4 4 4
35288 169/300 Read block 16 len 36864
35289 170/300 Flush
35290 171/300 Read block 1842 len 8192
35291 172/300 Read block 1492 len 8192
35292 173/300 Read block 1133 len 28672
35293 174/300 Write block 2207 len 32768 data: 6 7 7 8 7 7 4 5
35294 175/300 Read block 3457 len 16384
35295 176/300 Read block 1437 len 36864
35296 177/300 Read block 1121 len 36864
35297 178/300 Write block 4199 len 4096 data: 7
35298 179/300 Read block 523 len 40960
35299 180/300 Read block 3125 len 32768
35300 181/300 Read block 4523 len 8192
35301 182/300 Read block 63 len 8192
35302 183/300 Flush
35303 184/300 Write block 887 len 32768 data: 7 6 3 6 7 7 7 7
35304 185/300 Read block 4632 len 16384
35305 186/300 Read block 1633 len 12288
35306 187/300 Write block 4259 len 28672 data: 5 5 4 4 3 3 2
35307 188/300 Read block 1036 len 36864
35308 189/300 Write block 952 len 40960 data: 8 8 7 6 6 8 8 7 7 7
35309 190/300 Flush
35310 191/300 Read block 1745 len 20480
35311 192/300 Read block 4054 len 28672
35312 193/300 Write block 3941 len 32768 data: 10 11 8 9 7 7 7 7
35313 194/300 Write block 630 len 20480 data: 6 4 3 3 6
35314 195/300 Write block 3916 len 4096 data: 5
35315 196/300 Flush
35316 197/300 Read block 3876 len 28672
35317 198/300 Write block 1536 len 28672 data: 5 5 9 6 5 5 5
35318 199/300 Write block 577 len 16384 data: 11 7 7 7
35319 200/300 Read block 735 len 4096
35320 201/300 Read block 1937 len 28672
35321 202/300 Write block 3028 len 24576 data: 5 6 8 7 8 9
35322 203/300 Write block 4610 len 28672 data: 5 6 7 6 6 7 6
35323 204/300 Read block 3729 len 24576
35324 205/300 Read block 2940 len 28672
35325 206/300 Write block 1048 len 12288 data: 7 6 6
35326 207/300 Write block 3252 len 40960 data: 4 4 5 3 3 5 6 7 7 6
35327 208/300 Read block 1235 len 32768
35328 209/300 Read block 884 len 16384
35329 210/300 Read block 4137 len 36864
35330 211/300 Read block 1819 len 16384
35331 212/300 Write block 581 len 20480 data: 6 8 7 6 7
35332 213/300 Write block 4551 len 24576 data: 4 6 5 6 7 7
35333 214/300 Write block 2828 len 12288 data: 4 5 6
35334 215/300 Write block 2595 len 36864 data: 7 7 7 7 6 7 7 8 8
35335 216/300 Write block 1200 len 32768 data: 7 7 5 5 4 3 4 4
35336 217/300 Read block 1857 len 16384
35337 218/300 Write block 4612 len 40960 data: 8 7 7 8 7 6 5 6 5 5
35338 219/300 Write block 1278 len 20480 data: 3 4 6 7 7
35339 220/300 Write block 4894 len 4096 data: 6
35340 221/300 Read block 4695 len 12288
35341 222/300 Write block 3338 len 8192 data: 6 5
35342 223/300 Flush
35343 224/300 Write block 1175 len 4096 data: 10
35344 225/300 Read block 4970 len 28672
35345 226/300 Write block 4225 len 36864 data: 7 6 7 7 6 5 4 3 4
35346 227/300 Flush
35347 228/300 Flush
35348 229/300 Read block 4708 len 40960
35349 230/300 Flush
35350 231/300 Read block 1609 len 4096
35351 232/300 Write block 1281 len 40960 data: 8 8 8 9 10 11 10 9 9 7
35352 233/300 Read block 4648 len 40960
35353 234/300 Read block 296 len 20480
35354 235/300 Write block 315 len 28672 data: 5 5 4 4 4 5 5
35355 236/300 Read block 4495 len 12288
35356 237/300 Read block 502 len 4096
35357 238/300 Write block 2245 len 28672 data: 5 5 4 5 4 7 6
35358 239/300 Flush
35359 240/300 Write block 3013 len 24576 data: 5 5 5 6 7 6
35360 241/300 Flush
35361 242/300 Read block 1366 len 40960
35362 243/300 Write block 3058 len 24576 data: 5 5 4 5 6 6
35363 244/300 Write block 1636 len 40960 data: 5 6 6 6 4 5 5 5 3 3
35364 245/300 Write block 2512 len 24576 data: 7 6 4 6 6 6
35365 246/300 Write block 3779 len 16384 data: 6 7 4 4
35366 247/300 Flush
35367 248/300 Read block 4955 len 32768
35368 249/300 Write block 1295 len 24576 data: 6 5 4 5 8 8
35369 250/300 Read block 4030 len 12288
35370 251/300 Read block 1289 len 36864
35371 252/300 Read block 640 len 8192
35372 253/300 Read block 238 len 40960
35373 254/300 Read block 3748 len 20480
35374 255/300 Write block 1641 len 24576 data: 6 6 6 4 4 3
35375 256/300 Read block 3189 len 8192
35376 257/300 Read block 4938 len 12288
35377 258/300 Write block 687 len 12288 data: 5 6 7
35378 259/300 Write block 3642 len 28672 data: 10 10 8 7 8 8 8
35379 260/300 Write block 3671 len 16384 data: 5 5 5 5
35380 261/300 Read block 829 len 20480
35381 262/300 Write block 2096 len 36864 data: 6 7 6 4 6 4 4 6 5
35382 263/300 Read block 2434 len 4096
35383 264/300 Write block 2965 len 24576 data: 9 9 7 5 6 6
35384 265/300 Write block 1890 len 12288 data: 9 8 7
35385 266/300 Write block 4185 len 24576 data: 6 6 4 6 7 7
35386 267/300 Read block 369 len 24576
35387 268/300 Read block 2021 len 32768
35388 269/300 Read block 2308 len 32768
35389 270/300 Write block 2382 len 20480 data: 9 6 6 5 4
35390 271/300 Read block 1318 len 16384
35391 272/300 Read block 2225 len 40960
35392 273/300 Write block 2255 len 8192 data: 5 7
35393 274/300 Read block 3793 len 36864
35394 275/300 Read block 4456 len 20480
35395 276/300 Write block 1433 len 20480 data: 5 4 5 7 6
35396 277/300 Write block 4792 len 40960 data: 4 4 3 4 4 4 5 4 4 4
35397 278/300 Read block 1955 len 12288
35398 279/300 Read block 159 len 12288
35399 280/300 Write block 3171 len 28672 data: 6 5 6 7 6 4 5
35400 281/300 Write block 2260 len 24576 data: 5 7 8 7 6 5
35401 282/300 Read block 258 len 24576
35402 283/300 Write block 4371 len 32768 data: 6 5 5 5 5 5 5 5
35403 284/300 Flush
35404 285/300 Read block 4509 len 24576
35405 286/300 Write block 51 len 40960 data: 8 7 5 5 5 5 4 5 8 8
35406 287/300 Read block 4633 len 24576
35407 288/300 Read block 623 len 8192
35408 289/300 Write block 1506 len 24576 data: 9 9 11 9 9 8
35409 290/300 Write block 118 len 20480 data: 8 9 8 9 10
35410 291/300 Read block 3032 len 20480
35411 292/300 Flush
35412 293/300 Read block 3987 len 32768
35413 294/300 Write block 3886 len 16384 data: 6 4 4 5
35414 295/300 Read block 1706 len 4096
35415 296/300 Write block 2134 len 12288 data: 3 4 6
35416 297/300 Write block 879 len 12288 data: 7 9 6
35417 298/300 Read block 4245 len 40960
35418 299/300 Write block 1945 len 36864 data: 8 8 8 10 10 9 9 7 4
35419 300/300 Read block 2408 len 32768
35420 301/300 Read block 3567 len 36864
35421 Main task has work to do, go find it
35422 got dsc Start(1)
35423 start 1
35424 Jan 29 05:18:43.051 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/1, method: POST, req_id: 40f4edf5-5f78-480d-b7b2-5ce48b812e29, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
35425 Replay: started 1, returned:Ok(())
35426 ----------------------------------------------------------------
35427 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35428 States: Active Offline Active
35429 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35430 9757 Acked Read 7 Done Sent Done false
35431 9758 Acked Write 4 Done Sent Done false
35432 9759 Acked Read 6 Done Sent Done false
35433 9760 Acked Flush 0 Done Sent Done false
35434 9761 Acked Read 6 Done Sent Done false
35435 9762 Acked Write 8 Done Sent Done false
35436 9763 Acked Write 4 Done Sent Done false
35437 9764 Acked Write 8 Done Sent Done false
35438 9765 Acked Read 8 Done Sent Done false
35439 9766 Acked Write 7 Done Sent Done false
35440 9767 Acked Write 8 Done Sent Done false
35441 9768 Acked Write 3 Done Sent Done false
35442 9769 Acked Read 2 Done Sent Done false
35443 9770 Acked Write 3 Done Sent Done false
35444 9771 Acked Read 4 Done Sent Done false
35445 9772 Acked Read 5 Done Sent Done false
35446 9773 Acked Write 4 Done Sent Done false
35447 9774 Acked Write 7 Done Sent Done false
35448 9775 Acked Read 6 Done Sent Done false
35449 9776 Acked Write 4 Done Sent Done false
35450 9777 Acked Read 1 Done Sent Done false
35451 9778 Acked Read 2 Done Sent Done false
35452 9779 Acked Read 8 Done Sent Done false
35453 9780 Acked Read 8 Done Sent Done false
35454 9781 Acked Read 9 Done Sent Done false
35455 9782 Acked Write 5 Done Sent Done false
35456 9783 Acked Read 9 Done Sent Done false
35457 9784 Acked Write 1 Done Sent Done false
35458 9785 Acked Write 8 Done Sent Done false
35459 9786 Acked Read 4 Done Sent Done false
35460 9787 Acked Flush 0 Done Sent Done false
35461 9788 Acked Write 6 Done Sent Done false
35462 9789 Acked Flush 0 Done Sent Done false
35463 9790 Acked Read 7 Done Sent Done false
35464 9791 Acked Write 8 Done Sent Done false
35465 9792 Acked Read 9 Done Sent Done false
35466 9793 Acked Read 8 Done Sent Done false
35467 9794 Acked Read 10 Done Sent Done false
35468 9795 Acked Read 3 Done Sent Done false
35469 9796 Acked Read 7 Done Sent Done false
35470 9797 Acked Flush 0 Done Sent Done false
35471 9798 Acked Write 4 Done Sent Done false
35472 9799 Acked Read 2 Done Sent Done false
35473 9800 Acked Write 3 Done Sent Done false
35474 9801 Acked Read 8 Done Sent Done false
35475 9802 Acked Write 5 Done Sent Done false
35476 9803 Acked Write 7 Done Sent Done false
35477 9804 Acked Write 1 Done Sent Done false
35478 9805 Acked Write 3 Done Sent Done false
35479 9806 Acked Read 1 Done Sent Done false
35480 9807 Acked Read 8 Done Sent Done false
35481 9808 Acked Read 3 Done Sent Done false
35482 9809 Acked Read 6 Done Sent Done false
35483 9810 Acked Flush 0 Done Sent Done false
35484 9811 Acked Read 1 Done Sent Done false
35485 9812 Acked Write 8 Done Sent Done false
35486 9813 Acked Write 9 Done Sent Done false
35487 9814 Acked Read 10 Done Sent Done false
35488 9815 Acked Read 2 Done Sent Done false
35489 9816 Acked Write 7 Done Sent Done false
35490 9817 Acked Write 10 Done Sent Done false
35491 9818 Acked Write 4 Done Sent Done false
35492 9819 Acked Write 3 Done Sent Done false
35493 9820 Acked Read 1 Done Sent Done false
35494 9821 Acked Read 2 Done Sent Done false
35495 9822 Acked Write 8 Done Sent Done false
35496 9823 Acked Read 6 Done Sent Done false
35497 9824 Acked Read 3 Done Sent Done false
35498 9825 Acked Read 9 Done Sent Done false
35499 9826 Acked Read 10 Done Sent Done false
35500 9827 Acked Read 3 Done Sent Done false
35501 9828 Acked Read 2 Done Sent Done false
35502 9829 Acked Write 9 Done Sent Done false
35503 9830 Acked Write 1 Done Sent Done false
35504 9831 Acked Write 8 Done Sent Done false
35505 9832 Acked Write 8 Done Sent Done false
35506 9833 Acked Read 6 Done Sent Done false
35507 9834 Acked Write 9 Done Sent Done false
35508 9835 Acked Read 9 Done Sent Done false
35509 9836 Acked Read 2 Done Sent Done false
35510 9837 Acked Read 4 Done Sent Done false
35511 9838 Acked Read 3 Done Sent Done false
35512 9839 Acked Write 6 Done Sent Done false
35513 9840 Acked Write 8 Done Sent Done false
35514 9841 Acked Flush 0 Done Sent Done false
35515 9842 Acked Read 5 Done Sent Done false
35516 9843 Acked Write 10 Done Sent Done false
35517 9844 Acked Read 10 Done Sent Done false
35518 9845 Acked Write 6 Done Sent Done false
35519 9846 Acked Write 4 Done Sent Done false
35520 9847 Acked Read 6 Done Sent Done false
35521 9848 Acked Flush 0 Done Sent Done false
35522 9849 Acked Write 3 Done Sent Done false
35523 9850 Acked Write 2 Done Sent Done false
35524 9851 Acked Write 2 Done Sent Done false
35525 9852 Acked Write 10 Done Sent Done false
35526 9853 Acked Write 3 Done Sent Done false
35527 9854 Acked Read 8 Done Sent Done false
35528 9855 Acked Flush 0 Done Sent Done false
35529 9856 Acked Read 6 Done Sent Done false
35530 9857 Acked Read 8 Done Sent Done false
35531 9858 Acked Read 8 Done Sent Done false
35532 9859 Acked Read 8 Done Sent Done false
35533 9860 Acked Read 8 Done Sent Done false
35534 9861 Acked Flush 0 Done Sent Done false
35535 9862 Acked Read 9 Done Sent Done false
35536 9863 Acked Read 1 Done Sent Done false
35537 9864 Acked Write 10 Done Sent Done false
35538 9865 Acked Write 4 Done Sent Done false
35539 9866 Acked Write 5 Done Sent Done false
35540 9867 Acked Write 3 Done Sent Done false
35541 9868 Acked Write 3 Done Sent Done false
35542 9869 Acked Read 9 Done Sent Done false
35543 9870 Acked Write 6 Done Sent Done false
35544 9871 Acked Read 5 Done Sent Done false
35545 9872 Acked Read 5 Done Sent Done false
35546 9873 Acked Read 2 Done Sent Done false
35547 9874 Acked Read 5 Done Sent Done false
35548 9875 Acked Write 5 Done Sent Done false
35549 9876 Acked Read 10 Done Sent Done false
35550 9877 Acked Read 3 Done Sent Done false
35551 9878 Acked Read 8 Done Sent Done false
35552 9879 Acked Flush 0 Done Sent Done false
35553 9880 Acked Flush 0 Done Sent Done false
35554 9881 Acked Read 9 Done Sent Done false
35555 9882 Acked Write 2 Done Sent Done false
35556 9883 Acked Read 6 Done Sent Done false
35557 9884 Acked Write 4 Done Sent Done false
35558 9885 Acked Read 3 Done Sent Done false
35559 9886 Acked Read 2 Done Sent Done false
35560 9887 Acked Write 6 Done Sent Done false
35561 9888 Acked Flush 0 Done Sent Done false
35562 9889 Acked Read 9 Done Sent Done false
35563 9890 Acked Read 3 Done Sent Done false
35564 9891 Acked Write 6 Done Sent Done false
35565 9892 Acked Write 10 Done Sent Done false
35566 9893 Acked Read 10 Done Sent Done false
35567 9894 Acked Flush 0 Done Sent Done false
35568 9895 Acked Read 5 Done Sent Done false
35569 9896 Acked Flush 0 Done Sent Done false
35570 9897 Acked Write 1 Done Sent Done false
35571 9898 Acked Read 3 Done Sent Done false
35572 9899 Acked Read 4 Done Sent Done false
35573 9900 Acked Read 8 Done Sent Done false
35574 9901 Acked Write 7 Done Sent Done false
35575 9902 Acked Read 6 Done Sent Done false
35576 9903 Acked Read 7 Done Sent Done false
35577 9904 Acked Write 8 Done Sent Done false
35578 9905 Acked Write 8 Done Sent Done false
35579 9906 Acked Write 2 Done Sent Done false
35580 9907 Acked Read 7 Done Sent Done false
35581 9908 Acked Read 8 Done Sent Done false
35582 9909 Acked Write 5 Done Sent Done false
35583 9910 Acked Write 2 Done Sent Done false
35584 9911 Acked Read 3 Done Sent Done false
35585 9912 Acked Flush 0 Done Sent Done false
35586 9913 Acked Flush 0 Done Sent Done false
35587 9914 Acked Read 9 Done Sent Done false
35588 9915 Acked Read 2 Done Sent Done false
35589 9916 Acked Write 8 Done Sent Done false
35590 9917 Acked Read 8 Done Sent Done false
35591 9918 Acked Flush 0 Done Sent Done false
35592 9919 Acked Read 9 Done Sent Done false
35593 9920 Acked Read 10 Done Sent Done false
35594 9921 Acked Write 10 Done Sent Done false
35595 9922 Acked Write 4 Done Sent Done false
35596 9923 Acked Write 3 Done Sent Done false
35597 9924 Acked Write 5 Done Sent Done false
35598 9925 Acked Read 9 Done Sent Done false
35599 9926 Acked Flush 0 Done Sent Done false
35600 9927 Acked Read 2 Done Sent Done false
35601 9928 Acked Read 2 Done Sent Done false
35602 9929 Acked Read 7 Done Sent Done false
35603 9930 Acked Write 8 Done Sent Done false
35604 9931 Acked Read 4 Done Sent Done false
35605 9932 Acked Read 9 Done Sent Done false
35606 9933 Acked Read 9 Done Sent Done false
35607 9934 Acked Write 1 Done Sent Done false
35608 9935 Acked Read 10 Done Sent Done false
35609 9936 Acked Read 8 Done Sent Done false
35610 9937 Acked Read 2 Done Sent Done false
35611 9938 Acked Read 2 Done Sent Done false
35612 9939 Acked Flush 0 Done Sent Done false
35613 9940 Acked Write 8 Done Sent Done false
35614 9941 Acked Read 4 Done Sent Done false
35615 9942 Acked Read 3 Done Sent Done false
35616 9943 Acked Write 7 Done Sent Done false
35617 9944 Acked Read 9 Done Sent Done false
35618 9945 Acked Write 10 Done Sent Done false
35619 9946 Acked Flush 0 Done Sent Done false
35620 9947 Acked Read 5 Done Sent Done false
35621 9948 Acked Read 7 Done Sent Done false
35622 9949 Acked Write 8 Done Sent Done false
35623 9950 Acked Write 5 Done Sent Done false
35624 9951 Acked Write 1 Done Sent Done false
35625 9952 Acked Flush 0 Done Sent Done false
35626 9953 Acked Read 7 Done Sent Done false
35627 9954 Acked Write 7 Done Sent Done false
35628 9955 Acked Write 4 Done Sent Done false
35629 9956 Acked Read 1 Done Sent Done false
35630 9957 Acked Read 7 Done Sent Done false
35631 9958 Acked Write 6 Done Sent Done false
35632 9959 Acked Write 7 Done Sent Done false
35633 9960 Acked Read 6 Done Sent Done false
35634 9961 Acked Read 7 Done Sent Done false
35635 9962 Acked Write 3 Done Sent Done false
35636 9963 Acked Write 10 Done Sent Done false
35637 9964 Acked Read 8 Done Sent Done false
35638 9965 Acked Read 4 Done Sent Done false
35639 9966 Acked Read 9 Done Sent Done false
35640 9967 Acked Read 4 Done Sent Done false
35641 9968 Acked Write 5 Done Sent Done false
35642 9969 Acked Write 6 Done Sent Done false
35643 9970 Acked Write 3 Done Sent Done false
35644 9971 Acked Write 9 Done Sent Done false
35645 9972 Acked Write 8 Done Sent Done false
35646 9973 Acked Read 4 Done Sent Done false
35647 9974 Acked Write 10 Done Sent Done false
35648 9975 Acked Write 5 Done Sent Done false
35649 9976 Acked Write 1 Done Sent Done false
35650 9977 Acked Read 3 Done Sent Done false
35651 9978 Acked Write 2 Done Sent Done false
35652 9979 Acked Flush 0 Done Sent Done false
35653 9980 Acked Write 1 Done Sent Done false
35654 9981 Acked Read 7 Done Sent Done false
35655 9982 Acked Write 9 Done Sent Done false
35656 9983 Acked Flush 0 Done Sent Done false
35657 9984 Acked Flush 0 Done Sent Done false
35658 9985 Acked Read 10 Done Sent Done false
35659 9986 Acked Flush 0 Done Sent Done false
35660 9987 Acked Read 1 Done Sent Done false
35661 9988 Acked Write 10 Done Sent Done false
35662 9989 Acked Read 10 Done Sent Done false
35663 9990 Acked Read 5 Done Sent Done false
35664 9991 Acked Write 7 Done Sent Done false
35665 9992 Acked Read 3 Done Sent Done false
35666 9993 Acked Read 1 Done Sent Done false
35667 9994 Acked Write 7 Done Sent Done false
35668 9995 Acked Flush 0 Done Sent Done false
35669 9996 Acked Write 6 Done Sent Done false
35670 9997 Acked Flush 0 Done Sent Done false
35671 9998 Acked Read 10 Done Sent Done false
35672 9999 Acked Write 6 Done Sent Done false
35673 10000 Acked Write 10 Done Sent Done false
35674 10001 Acked Write 6 Done Sent Done false
35675 10002 Acked Write 4 Done Sent Done false
35676 10003 Acked Flush 0 Done Sent Done false
35677 10004 Acked Read 8 Done Sent Done false
35678 10005 Acked Write 6 Done Sent Done false
35679 10006 Acked Read 3 Done Sent Done false
35680 10007 Acked Read 9 Done Sent Done false
35681 10008 Acked Read 2 Done Sent Done false
35682 10009 Acked Read 10 Done Sent Done false
35683 10010 Acked Read 5 Done Sent Done false
35684 10011 Acked Write 6 Done Sent Done false
35685 10012 Acked Read 2 Done Sent Done false
35686 10013 Acked Read 3 Done Sent Done false
35687 10014 Acked Write 3 Done Sent Done false
35688 10015 Acked Write 7 Done Sent Done false
35689 10016 Acked Write 4 Done Sent Done false
35690 10017 Acked Read 5 Done Sent Done false
35691 10018 Acked Write 9 Done Sent Done false
35692 10019 Acked Read 1 Done Sent Done false
35693 10020 Acked Write 6 Done Sent Done false
35694 10021 Acked Write 3 Done Sent Done false
35695 10022 Acked Write 6 Done Sent Done false
35696 10023 Acked Read 6 Done Sent Done false
35697 10024 Acked Read 8 Done Sent Done false
35698 10025 Acked Read 8 Done Sent Done false
35699 10026 Acked Write 5 Done Sent Done false
35700 10027 Acked Read 4 Done Sent Done false
35701 10028 Acked Read 10 Done Sent Done false
35702 10029 Acked Write 2 Done Sent Done false
35703 10030 Acked Read 9 Done Sent Done false
35704 10031 Acked Read 5 Done Sent Done false
35705 10032 Acked Write 5 Done Sent Done false
35706 10033 Acked Write 10 Done Sent Done false
35707 10034 Acked Read 3 Done Sent Done false
35708 10035 Acked Read 3 Done Sent Done false
35709 10036 Acked Write 7 Done Sent Done false
35710 10037 Acked Write 6 Done Sent Done false
35711 10038 Acked Read 6 Done Sent Done false
35712 10039 Acked Write 8 Done Sent Done false
35713 10040 Acked Flush 0 Done Sent Done false
35714 10041 Acked Read 6 Done Sent Done false
35715 10042 Acked Write 10 Done Sent Done false
35716 10043 Acked Read 6 Done Sent Done false
35717 10044 Acked Read 2 Done Sent Done false
35718 10045 Acked Write 6 Done Sent Done false
35719 10046 Acked Write 5 Done Sent Done false
35720 10047 Acked Read 5 Done Sent Done false
35721 10048 Acked Flush 0 Done Sent Done false
35722 10049 Acked Read 8 Done Sent Done false
35723 10050 Acked Write 4 Done Sent Done false
35724 10051 Acked Read 1 Done Sent Done false
35725 10052 Acked Write 3 Done Sent Done false
35726 10053 Acked Write 3 Done Sent Done false
35727 10054 Acked Read 10 Done Sent Done false
35728 10055 Acked Write 9 Done Sent Done false
35729 10056 Acked Read 8 Done Sent Done false
35730 10057 Acked Read 9 Done Sent Done false
35731 STATES DS:0 DS:1 DS:2 TOTAL
35732 Sent 0 301 0 301
35733 Done 301 0 301 602
35734 Skipped 0 0 0 0
35735 Error 0 0 0 0
35736 Last Flush: Some(JobId(10048)) Some(JobId(9756)) Some(JobId(10048))
35737 Downstairs last five completed: 9756 9755 9754 9753 9752
35738 Upstairs last five completed: 10057 10056 10055 10054 10053
35739 CLIENT: Up:0 ds:301 act:2
357402025-01-29T05:18:43.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
35741 [8820] got start action
35742 [8820] I am going to restart
35743 Starting downstairs at port 8820
35744 Make output file at "/tmp/dsc/downstairs-8820.txt"
35745 [8820][1] reports Starting
35746 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
35747 [8820][1] reports Running
357482025-01-29T05:18:45.044ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
357492025-01-29T05:18:45.791ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
357502025-01-29T05:18:46.726ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
35751 ----------------------------------------------------------------
35752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35753 States: Active Offline Active
35754 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35755 9757 Acked Read 7 Done Sent Done false
35756 9758 Acked Write 4 Done Sent Done false
35757 9759 Acked Read 6 Done Sent Done false
35758 9760 Acked Flush 0 Done Sent Done false
35759 9761 Acked Read 6 Done Sent Done false
35760 9762 Acked Write 8 Done Sent Done false
35761 9763 Acked Write 4 Done Sent Done false
35762 9764 Acked Write 8 Done Sent Done false
35763 9765 Acked Read 8 Done Sent Done false
35764 9766 Acked Write 7 Done Sent Done false
35765 9767 Acked Write 8 Done Sent Done false
35766 9768 Acked Write 3 Done Sent Done false
35767 9769 Acked Read 2 Done Sent Done false
35768 9770 Acked Write 3 Done Sent Done false
35769 9771 Acked Read 4 Done Sent Done false
35770 9772 Acked Read 5 Done Sent Done false
35771 9773 Acked Write 4 Done Sent Done false
35772 9774 Acked Write 7 Done Sent Done false
35773 9775 Acked Read 6 Done Sent Done false
35774 9776 Acked Write 4 Done Sent Done false
35775 9777 Acked Read 1 Done Sent Done false
35776 9778 Acked Read 2 Done Sent Done false
35777 9779 Acked Read 8 Done Sent Done false
35778 9780 Acked Read 8 Done Sent Done false
35779 9781 Acked Read 9 Done Sent Done false
35780 9782 Acked Write 5 Done Sent Done false
35781 9783 Acked Read 9 Done Sent Done false
35782 9784 Acked Write 1 Done Sent Done false
35783 9785 Acked Write 8 Done Sent Done false
35784 9786 Acked Read 4 Done Sent Done false
35785 9787 Acked Flush 0 Done Sent Done false
35786 9788 Acked Write 6 Done Sent Done false
35787 9789 Acked Flush 0 Done Sent Done false
35788 9790 Acked Read 7 Done Sent Done false
35789 9791 Acked Write 8 Done Sent Done false
35790 9792 Acked Read 9 Done Sent Done false
35791 9793 Acked Read 8 Done Sent Done false
35792 9794 Acked Read 10 Done Sent Done false
35793 9795 Acked Read 3 Done Sent Done false
35794 9796 Acked Read 7 Done Sent Done false
35795 9797 Acked Flush 0 Done Sent Done false
35796 9798 Acked Write 4 Done Sent Done false
35797 9799 Acked Read 2 Done Sent Done false
35798 9800 Acked Write 3 Done Sent Done false
35799 9801 Acked Read 8 Done Sent Done false
35800 9802 Acked Write 5 Done Sent Done false
35801 9803 Acked Write 7 Done Sent Done false
35802 9804 Acked Write 1 Done Sent Done false
35803 9805 Acked Write 3 Done Sent Done false
35804 9806 Acked Read 1 Done Sent Done false
35805 9807 Acked Read 8 Done Sent Done false
35806 9808 Acked Read 3 Done Sent Done false
35807 9809 Acked Read 6 Done Sent Done false
35808 9810 Acked Flush 0 Done Sent Done false
35809 9811 Acked Read 1 Done Sent Done false
35810 9812 Acked Write 8 Done Sent Done false
35811 9813 Acked Write 9 Done Sent Done false
35812 9814 Acked Read 10 Done Sent Done false
35813 9815 Acked Read 2 Done Sent Done false
35814 9816 Acked Write 7 Done Sent Done false
35815 9817 Acked Write 10 Done Sent Done false
35816 9818 Acked Write 4 Done Sent Done false
35817 9819 Acked Write 3 Done Sent Done false
35818 9820 Acked Read 1 Done Sent Done false
35819 9821 Acked Read 2 Done Sent Done false
35820 9822 Acked Write 8 Done Sent Done false
35821 9823 Acked Read 6 Done Sent Done false
35822 9824 Acked Read 3 Done Sent Done false
35823 9825 Acked Read 9 Done Sent Done false
35824 9826 Acked Read 10 Done Sent Done false
35825 9827 Acked Read 3 Done Sent Done false
35826 9828 Acked Read 2 Done Sent Done false
35827 9829 Acked Write 9 Done Sent Done false
35828 9830 Acked Write 1 Done Sent Done false
35829 9831 Acked Write 8 Done Sent Done false
35830 9832 Acked Write 8 Done Sent Done false
35831 9833 Acked Read 6 Done Sent Done false
35832 9834 Acked Write 9 Done Sent Done false
35833 9835 Acked Read 9 Done Sent Done false
35834 9836 Acked Read 2 Done Sent Done false
35835 9837 Acked Read 4 Done Sent Done false
35836 9838 Acked Read 3 Done Sent Done false
35837 9839 Acked Write 6 Done Sent Done false
35838 9840 Acked Write 8 Done Sent Done false
35839 9841 Acked Flush 0 Done Sent Done false
35840 9842 Acked Read 5 Done Sent Done false
35841 9843 Acked Write 10 Done Sent Done false
35842 9844 Acked Read 10 Done Sent Done false
35843 9845 Acked Write 6 Done Sent Done false
35844 9846 Acked Write 4 Done Sent Done false
35845 9847 Acked Read 6 Done Sent Done false
35846 9848 Acked Flush 0 Done Sent Done false
35847 9849 Acked Write 3 Done Sent Done false
35848 9850 Acked Write 2 Done Sent Done false
35849 9851 Acked Write 2 Done Sent Done false
35850 9852 Acked Write 10 Done Sent Done false
35851 9853 Acked Write 3 Done Sent Done false
35852 9854 Acked Read 8 Done Sent Done false
35853 9855 Acked Flush 0 Done Sent Done false
35854 9856 Acked Read 6 Done Sent Done false
35855 9857 Acked Read 8 Done Sent Done false
35856 9858 Acked Read 8 Done Sent Done false
35857 9859 Acked Read 8 Done Sent Done false
35858 9860 Acked Read 8 Done Sent Done false
35859 9861 Acked Flush 0 Done Sent Done false
35860 9862 Acked Read 9 Done Sent Done false
35861 9863 Acked Read 1 Done Sent Done false
35862 9864 Acked Write 10 Done Sent Done false
35863 9865 Acked Write 4 Done Sent Done false
35864 9866 Acked Write 5 Done Sent Done false
35865 9867 Acked Write 3 Done Sent Done false
35866 9868 Acked Write 3 Done Sent Done false
35867 9869 Acked Read 9 Done Sent Done false
35868 9870 Acked Write 6 Done Sent Done false
35869 9871 Acked Read 5 Done Sent Done false
35870 9872 Acked Read 5 Done Sent Done false
35871 9873 Acked Read 2 Done Sent Done false
35872 9874 Acked Read 5 Done Sent Done false
35873 9875 Acked Write 5 Done Sent Done false
35874 9876 Acked Read 10 Done Sent Done false
35875 9877 Acked Read 3 Done Sent Done false
35876 9878 Acked Read 8 Done Sent Done false
35877 9879 Acked Flush 0 Done Sent Done false
35878 9880 Acked Flush 0 Done Sent Done false
35879 9881 Acked Read 9 Done Sent Done false
35880 9882 Acked Write 2 Done Sent Done false
35881 9883 Acked Read 6 Done Sent Done false
35882 9884 Acked Write 4 Done Sent Done false
35883 9885 Acked Read 3 Done Sent Done false
35884 9886 Acked Read 2 Done Sent Done false
35885 9887 Acked Write 6 Done Sent Done false
35886 9888 Acked Flush 0 Done Sent Done false
35887 9889 Acked Read 9 Done Sent Done false
35888 9890 Acked Read 3 Done Sent Done false
35889 9891 Acked Write 6 Done Sent Done false
35890 9892 Acked Write 10 Done Sent Done false
35891 9893 Acked Read 10 Done Sent Done false
35892 9894 Acked Flush 0 Done Sent Done false
35893 9895 Acked Read 5 Done Sent Done false
35894 9896 Acked Flush 0 Done Sent Done false
35895 9897 Acked Write 1 Done Sent Done false
35896 9898 Acked Read 3 Done Sent Done false
35897 9899 Acked Read 4 Done Sent Done false
35898 9900 Acked Read 8 Done Sent Done false
35899 9901 Acked Write 7 Done Sent Done false
35900 9902 Acked Read 6 Done Sent Done false
35901 9903 Acked Read 7 Done Sent Done false
35902 9904 Acked Write 8 Done Sent Done false
35903 9905 Acked Write 8 Done Sent Done false
35904 9906 Acked Write 2 Done Sent Done false
35905 9907 Acked Read 7 Done Sent Done false
35906 9908 Acked Read 8 Done Sent Done false
35907 9909 Acked Write 5 Done Sent Done false
35908 9910 Acked Write 2 Done Sent Done false
35909 9911 Acked Read 3 Done Sent Done false
35910 9912 Acked Flush 0 Done Sent Done false
35911 9913 Acked Flush 0 Done Sent Done false
35912 9914 Acked Read 9 Done Sent Done false
35913 9915 Acked Read 2 Done Sent Done false
35914 9916 Acked Write 8 Done Sent Done false
35915 9917 Acked Read 8 Done Sent Done false
35916 9918 Acked Flush 0 Done Sent Done false
35917 9919 Acked Read 9 Done Sent Done false
35918 9920 Acked Read 10 Done Sent Done false
35919 9921 Acked Write 10 Done Sent Done false
35920 9922 Acked Write 4 Done Sent Done false
35921 9923 Acked Write 3 Done Sent Done false
35922 9924 Acked Write 5 Done Sent Done false
35923 9925 Acked Read 9 Done Sent Done false
35924 9926 Acked Flush 0 Done Sent Done false
35925 9927 Acked Read 2 Done Sent Done false
35926 9928 Acked Read 2 Done Sent Done false
35927 9929 Acked Read 7 Done Sent Done false
35928 9930 Acked Write 8 Done Sent Done false
35929 9931 Acked Read 4 Done Sent Done false
35930 9932 Acked Read 9 Done Sent Done false
35931 9933 Acked Read 9 Done Sent Done false
35932 9934 Acked Write 1 Done Sent Done false
35933 9935 Acked Read 10 Done Sent Done false
35934 9936 Acked Read 8 Done Sent Done false
35935 9937 Acked Read 2 Done Sent Done false
35936 9938 Acked Read 2 Done Sent Done false
35937 9939 Acked Flush 0 Done Sent Done false
35938 9940 Acked Write 8 Done Sent Done false
35939 9941 Acked Read 4 Done Sent Done false
35940 9942 Acked Read 3 Done Sent Done false
35941 9943 Acked Write 7 Done Sent Done false
35942 9944 Acked Read 9 Done Sent Done false
35943 9945 Acked Write 10 Done Sent Done false
35944 9946 Acked Flush 0 Done Sent Done false
35945 9947 Acked Read 5 Done Sent Done false
35946 9948 Acked Read 7 Done Sent Done false
35947 9949 Acked Write 8 Done Sent Done false
35948 9950 Acked Write 5 Done Sent Done false
35949 9951 Acked Write 1 Done Sent Done false
35950 9952 Acked Flush 0 Done Sent Done false
35951 9953 Acked Read 7 Done Sent Done false
35952 9954 Acked Write 7 Done Sent Done false
35953 9955 Acked Write 4 Done Sent Done false
35954 9956 Acked Read 1 Done Sent Done false
35955 9957 Acked Read 7 Done Sent Done false
35956 9958 Acked Write 6 Done Sent Done false
35957 9959 Acked Write 7 Done Sent Done false
35958 9960 Acked Read 6 Done Sent Done false
35959 9961 Acked Read 7 Done Sent Done false
35960 9962 Acked Write 3 Done Sent Done false
35961 9963 Acked Write 10 Done Sent Done false
35962 9964 Acked Read 8 Done Sent Done false
35963 9965 Acked Read 4 Done Sent Done false
35964 9966 Acked Read 9 Done Sent Done false
35965 9967 Acked Read 4 Done Sent Done false
35966 9968 Acked Write 5 Done Sent Done false
35967 9969 Acked Write 6 Done Sent Done false
35968 9970 Acked Write 3 Done Sent Done false
35969 9971 Acked Write 9 Done Sent Done false
35970 9972 Acked Write 8 Done Sent Done false
35971 9973 Acked Read 4 Done Sent Done false
35972 9974 Acked Write 10 Done Sent Done false
35973 9975 Acked Write 5 Done Sent Done false
35974 9976 Acked Write 1 Done Sent Done false
35975 9977 Acked Read 3 Done Sent Done false
35976 9978 Acked Write 2 Done Sent Done false
35977 9979 Acked Flush 0 Done Sent Done false
35978 9980 Acked Write 1 Done Sent Done false
35979 9981 Acked Read 7 Done Sent Done false
35980 9982 Acked Write 9 Done Sent Done false
35981 9983 Acked Flush 0 Done Sent Done false
35982 9984 Acked Flush 0 Done Sent Done false
35983 9985 Acked Read 10 Done Sent Done false
35984 9986 Acked Flush 0 Done Sent Done false
35985 9987 Acked Read 1 Done Sent Done false
35986 9988 Acked Write 10 Done Sent Done false
35987 9989 Acked Read 10 Done Sent Done false
35988 9990 Acked Read 5 Done Sent Done false
35989 9991 Acked Write 7 Done Sent Done false
35990 9992 Acked Read 3 Done Sent Done false
35991 9993 Acked Read 1 Done Sent Done false
35992 9994 Acked Write 7 Done Sent Done false
35993 9995 Acked Flush 0 Done Sent Done false
35994 9996 Acked Write 6 Done Sent Done false
35995 9997 Acked Flush 0 Done Sent Done false
35996 9998 Acked Read 10 Done Sent Done false
35997 9999 Acked Write 6 Done Sent Done false
35998 10000 Acked Write 10 Done Sent Done false
35999 10001 Acked Write 6 Done Sent Done false
36000 10002 Acked Write 4 Done Sent Done false
36001 10003 Acked Flush 0 Done Sent Done false
36002 10004 Acked Read 8 Done Sent Done false
36003 10005 Acked Write 6 Done Sent Done false
36004 10006 Acked Read 3 Done Sent Done false
36005 10007 Acked Read 9 Done Sent Done false
36006 10008 Acked Read 2 Done Sent Done false
36007 10009 Acked Read 10 Done Sent Done false
36008 10010 Acked Read 5 Done Sent Done false
36009 10011 Acked Write 6 Done Sent Done false
36010 10012 Acked Read 2 Done Sent Done false
36011 10013 Acked Read 3 Done Sent Done false
36012 10014 Acked Write 3 Done Sent Done false
36013 10015 Acked Write 7 Done Sent Done false
36014 10016 Acked Write 4 Done Sent Done false
36015 10017 Acked Read 5 Done Sent Done false
36016 10018 Acked Write 9 Done Sent Done false
36017 10019 Acked Read 1 Done Sent Done false
36018 10020 Acked Write 6 Done Sent Done false
36019 10021 Acked Write 3 Done Sent Done false
36020 10022 Acked Write 6 Done Sent Done false
36021 10023 Acked Read 6 Done Sent Done false
36022 10024 Acked Read 8 Done Sent Done false
36023 10025 Acked Read 8 Done Sent Done false
36024 10026 Acked Write 5 Done Sent Done false
36025 10027 Acked Read 4 Done Sent Done false
36026 10028 Acked Read 10 Done Sent Done false
36027 10029 Acked Write 2 Done Sent Done false
36028 10030 Acked Read 9 Done Sent Done false
36029 10031 Acked Read 5 Done Sent Done false
36030 10032 Acked Write 5 Done Sent Done false
36031 10033 Acked Write 10 Done Sent Done false
36032 10034 Acked Read 3 Done Sent Done false
36033 10035 Acked Read 3 Done Sent Done false
36034 10036 Acked Write 7 Done Sent Done false
36035 10037 Acked Write 6 Done Sent Done false
36036 10038 Acked Read 6 Done Sent Done false
36037 10039 Acked Write 8 Done Sent Done false
36038 10040 Acked Flush 0 Done Sent Done false
36039 10041 Acked Read 6 Done Sent Done false
36040 10042 Acked Write 10 Done Sent Done false
36041 10043 Acked Read 6 Done Sent Done false
36042 10044 Acked Read 2 Done Sent Done false
36043 10045 Acked Write 6 Done Sent Done false
36044 10046 Acked Write 5 Done Sent Done false
36045 10047 Acked Read 5 Done Sent Done false
36046 10048 Acked Flush 0 Done Sent Done false
36047 10049 Acked Read 8 Done Sent Done false
36048 10050 Acked Write 4 Done Sent Done false
36049 10051 Acked Read 1 Done Sent Done false
36050 10052 Acked Write 3 Done Sent Done false
36051 10053 Acked Write 3 Done Sent Done false
36052 10054 Acked Read 10 Done Sent Done false
36053 10055 Acked Write 9 Done Sent Done false
36054 10056 Acked Read 8 Done Sent Done false
36055 10057 Acked Read 9 Done Sent Done false
36056 STATES DS:0 DS:1 DS:2 TOTAL
36057 Sent 0 301 0 301
36058 Done 301 0 301 602
36059 Skipped 0 0 0 0
36060 Error 0 0 0 0
36061 Last Flush: Some(JobId(10048)) Some(JobId(9756)) Some(JobId(10048))
36062 Downstairs last five completed: 9756 9755 9754 9753 9752
36063 Upstairs last five completed: 10057 10056 10055 10054 10053
36064 CLIENT: Up:0 ds:301 act:2
360652025-01-29T05:18:47.728ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360662025-01-29T05:18:48.730ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360672025-01-29T05:18:48.804ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360682025-01-29T05:18:48.805ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39251) = io task client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360692025-01-29T05:18:48.806ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360702025-01-29T05:18:48.806ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360712025-01-29T05:18:48.807ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360722025-01-29T05:18:48.807ZINFOcrucible: send last flush ID to this DS: Some(JobId(9756)) = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360732025-01-29T05:18:48.807ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360742025-01-29T05:18:48.807ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9756)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360752025-01-29T05:18:48.807ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9756)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
36076 ----------------------------------------------------------------
36077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36078 Downstairs last five completed: 10058 10057 10056 10055 10054
36079 Upstairs last five completed: 10058 10057 10056 10055 10054
36080 CLIENT: Up:0 ds:0 act:3
36081 Replay: All jobs finished
36082 Main task has work to do, go find it
36083 got dsc Stop(2)
36084 stop 2
36085 Jan 29 05:18:51.058 INFO[8830] Got stop action so:false kr:false
36086 request completed, latency_us: 393, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 9279f688-3f89-4def-9dac-29bfdfd28d82, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
360872025-01-29T05:18:51.059ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360882025-01-29T05:18:51.059ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
360892025-01-29T05:18:51.059ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
36090 Jan 29 05:18:51.059 INFO {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-01-29T05:18:51.059609205Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1202,"":"io task","client":"2","":"downstairs","session_id":"3b9f53a2-1179-49c9-b40f-5354d75015be"}
36091 request completed, latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: 925b1e4f-d4e2-4352-9372-eb82399c3d68, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
36092 [8830] Exited with: ExitStatus(unix_wait_status(9))
36093 [8830][2] reports Exit
36094 Jan 29 05:18:55.062 INFO request completed, latency_us: 333, response_code: 200, uri: /state/cid/2, method: GET, req_id: 879c8bb7-467e-4b40-8cb1-d59d9fe5cec9, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
36095 001/300 Write block 385 len 12288 data: 3 3 3
36096 002/300 Write block 477 len 20480 data: 7 9 10 10 12
36097 003/300 Read block 4452 len 20480
36098 004/300 Write block 1188 len 36864 data: 4 4 4 5 5 6 6 6 6
36099 005/300 Write block 2144 len 8192 data: 8 7
36100 006/300 Read block 1823 len 24576
36101 007/300 Read block 3694 len 16384
36102 008/300 Write block 3350 len 12288 data: 8 10 9
36103 009/300 Read block 28 len 36864
36104 010/300 Read block 418 len 8192
36105 011/300 Write block 1793 len 36864 data: 11 10 11 8 8 6 5 3 2
36106 012/300 Write block 4449 len 32768 data: 6 8 9 10 10 11 10 9
36107 013/300 Read block 4804 len 36864
36108 014/300 Read block 3502 len 36864
36109 015/300 Flush
36110 016/300 Write block 4153 len 36864 data: 5 5 6 6 4 3 4 4 6
36111 017/300 Flush
36112 018/300 Read block 2465 len 36864
36113 019/300 Flush
36114 020/300 Read block 141 len 20480
36115 021/300 Write block 2427 len 4096 data: 8
36116 022/300 Write block 3112 len 28672 data: 7 5 5 6 5 5 7
36117 023/300 Read block 4923 len 40960
36118 024/300 Read block 236 len 12288
36119 025/300 Read block 1646 len 28672
36120 026/300 Read block 151 len 4096
36121 027/300 Flush
36122 028/300 Write block 2129 len 36864 data: 3 3 2 2 3 4 5 7 5
36123 029/300 Write block 3090 len 28672 data: 6 5 6 6 6 4 5
36124 030/300 Write block 491 len 28672 data: 9 9 9 9 10 11 10
36125 031/300 Read block 2322 len 12288
36126 032/300 Write block 2026 len 28672 data: 7 7 7 8 6 6 6
36127 033/300 Read block 4588 len 36864
36128 034/300 Read block 1852 len 20480
36129 035/300 Read block 4529 len 28672
36130 036/300 Read block 2406 len 4096
36131 037/300 Write block 2371 len 28672 data: 6 5 5 5 4 5 6
36132 038/300 Read block 800 len 24576
36133 039/300 Flush
36134 040/300 Read block 866 len 12288
36135 041/300 Write block 1776 len 40960 data: 9 8 8 9 7 5 5 6 6 5
36136 042/300 Flush
36137 043/300 Read block 2402 len 8192
36138 044/300 Read block 3030 len 8192
36139 045/300 Read block 2302 len 32768
36140 046/300 Write block 4938 len 16384 data: 5 6 7 6
36141 047/300 Read block 3495 len 40960
36142 048/300 Write block 1954 len 4096 data: 4
36143 049/300 Read block 2179 len 24576
36144 050/300 Read block 868 len 36864
36145 051/300 Read block 2413 len 28672
36146 052/300 Write block 4380 len 40960 data: 6 6 5 5 6 5 6 7 6 5
36147 053/300 Write block 1100 len 36864 data: 7 5 5 5 4 5 5 6 7
36148 054/300 Read block 212 len 40960
36149 055/300 Read block 2448 len 4096
36150 056/300 Read block 1502 len 40960
36151 057/300 Write block 588 len 8192 data: 8 9
36152 058/300 Read block 680 len 12288
36153 059/300 Flush
36154 060/300 Write block 3496 len 4096 data: 9
36155 061/300 Flush
36156 062/300 Read block 3630 len 36864
36157 063/300 Read block 4342 len 20480
36158 064/300 Write block 4126 len 20480 data: 6 6 6 6 6
36159 065/300 Write block 4415 len 24576 data: 5 4 4 6 7 6
36160 066/300 Read block 1733 len 8192
36161 067/300 Write block 4809 len 28672 data: 7 9 8 8 7 6 7
36162 068/300 Write block 1609 len 40960 data: 5 6 5 6 6 7 6 7 7 5
36163 069/300 Flush
36164 070/300 Write block 4986 len 16384 data: 6 6 7 8
36165 071/300 Read block 3185 len 4096
36166 072/300 Write block 4551 len 36864 data: 5 7 6 7 8 8 6 7 6
36167 073/300 Flush
36168 074/300 Write block 3697 len 40960 data: 5 3 6 7 8 9 6 5 5 4
36169 075/300 Write block 3791 len 40960 data: 6 6 6 6 5 5 6 7 5 9
36170 076/300 Read block 4885 len 32768
36171 077/300 Read block 4788 len 32768
36172 078/300 Read block 2685 len 32768
36173 079/300 Read block 4398 len 28672
36174 080/300 Read block 4630 len 40960
36175 081/300 Flush
36176 082/300 Write block 2140 len 36864 data: 10 12 11 9 9 8 5 7 7
36177 083/300 Read block 1221 len 32768
36178 084/300 Flush
36179 085/300 Read block 3547 len 12288
36180 086/300 Write block 3916 len 28672 data: 6 3 3 4 3 4 4
36181 087/300 Read block 2593 len 16384
36182 088/300 Flush
36183 089/300 Read block 3076 len 12288
36184 090/300 Read block 869 len 32768
36185 091/300 Write block 12 len 40960 data: 4 5 6 6 7 7 6 7 7 7
36186 092/300 Write block 3238 len 12288 data: 6 5 4
36187 093/300 Write block 384 len 20480 data: 3 4 4 4 5
36188 094/300 Read block 984 len 12288
36189 095/300 Write block 4214 len 36864 data: 4 5 5 6 6 8 8 7 6
36190 096/300 Flush
36191 097/300 Read block 2989 len 36864
36192 098/300 Read block 2957 len 16384
36193 099/300 Write block 3560 len 36864 data: 3 3 4 4 4 4 4 5 7
36194 100/300 Read block 317 len 12288
36195 101/300 Read block 3590 len 40960
36196 102/300 Flush
36197 103/300 Write block 4200 len 36864 data: 9 10 10 10 9 9 9 7 6
36198 104/300 Write block 4657 len 8192 data: 7 7
36199 105/300 Write block 3341 len 12288 data: 3 3 5
36200 106/300 Read block 4313 len 12288
36201 107/300 Write block 344 len 36864 data: 7 6 7 7 8 7 7 7 7
36202 108/300 Write block 3401 len 8192 data: 3 4
36203 109/300 Read block 1662 len 16384
36204 110/300 Read block 3840 len 24576
36205 111/300 Write block 3786 len 16384 data: 4 5 7 7
36206 112/300 Read block 2798 len 16384
36207 113/300 Write block 6 len 32768 data: 7 7 6 6 5 3 5 6
36208 114/300 Read block 280 len 36864
36209 115/300 Flush
36210 116/300 Write block 3295 len 32768 data: 3 3 3 3 3 3 4 3
36211 117/300 Read block 3060 len 20480
36212 118/300 Write block 2847 len 16384 data: 6 5 6 7
36213 119/300 Read block 3294 len 4096
36214 120/300 Read block 3855 len 4096
36215 121/300 Write block 2404 len 4096 data: 5
36216 122/300 Read block 4903 len 16384
36217 123/300 Write block 4907 len 28672 data: 6 6 5 5 3 3 3
36218 124/300 Flush
36219 125/300 Write block 2460 len 24576 data: 8 6 6 6 4 6
36220 126/300 Read block 1292 len 4096
36221 127/300 Read block 1548 len 12288
36222 128/300 Read block 946 len 32768
36223 129/300 Read block 2231 len 28672
36224 130/300 Write block 212 len 20480 data: 3 4 5 6 7
36225 131/300 Write block 3387 len 16384 data: 9 7 9 9
36226 132/300 Write block 4601 len 40960 data: 11 10 11 12 8 6 5 4 4 6
36227 133/300 Read block 3729 len 12288
36228 134/300 Read block 3258 len 24576
36229 135/300 Write block 3151 len 12288 data: 4 4 5
36230 136/300 Read block 2593 len 12288
36231 137/300 Read block 3735 len 28672
36232 138/300 Read block 4879 len 12288
36233 139/300 Read block 2983 len 8192
36234 140/300 Write block 915 len 12288 data: 5 4 4
36235 141/300 Write block 3204 len 24576 data: 7 5 4 4 5 3
36236 142/300 Write block 2876 len 8192 data: 6 5
36237 143/300 Write block 4172 len 16384 data: 5 6 6 5
36238 144/300 Read block 1139 len 32768
36239 145/300 Write block 1522 len 36864 data: 5 5 5 5 5 6 5 5 5
36240 146/300 Write block 2443 len 24576 data: 6 7 6 6 6 6
36241 147/300 Read block 4340 len 4096
36242 148/300 Write block 1046 len 36864 data: 10 9 8 7 7 4 5 5 4
36243 149/300 Read block 1067 len 24576
36244 150/300 Flush
36245 151/300 Flush
36246 152/300 Write block 1945 len 16384 data: 9 9 9 11
36247 153/300 Write block 3030 len 32768 data: 9 8 9 10 7 8 7 5
36248 154/300 Write block 3738 len 32768 data: 3 3 3 5 6 6 6 6
36249 155/300 Write block 1212 len 20480 data: 6 7 7 6 6
36250 156/300 Read block 3129 len 4096
36251 157/300 Read block 2251 len 24576
36252 158/300 Write block 4371 len 4096 data: 7
36253 159/300 Read block 2239 len 12288
36254 160/300 Read block 2746 len 40960
36255 161/300 Read block 385 len 4096
36256 162/300 Read block 1850 len 4096
36257 163/300 Write block 32 len 8192 data: 5 4
36258 164/300 Read block 1194 len 4096
36259 165/300 Write block 1664 len 40960 data: 6 6 6 4 4 3 3 3 3 5
36260 166/300 Flush
36261 167/300 Write block 4259 len 20480 data: 6 6 5 5 4
36262 168/300 Write block 3223 len 28672 data: 10 7 7 6 6 9 11
36263 169/300 Write block 3859 len 8192 data: 5 4
36264 170/300 Flush
36265 171/300 Read block 1128 len 8192
36266 172/300 Flush
36267 173/300 Read block 2721 len 8192
36268 174/300 Flush
36269 175/300 Read block 4067 len 28672
36270 176/300 Read block 3875 len 36864
36271 177/300 Write block 3945 len 8192 data: 8 8
36272 178/300 Read block 3272 len 20480
36273 179/300 Write block 3046 len 8192 data: 4 4
36274 180/300 Write block 3881 len 16384 data: 6 6 6 7
36275 181/300 Write block 3802 len 32768 data: 11 10 9 10 10 10 9 7
36276 182/300 Write block 4241 len 24576 data: 5 3 3 2 4 5
36277 183/300 Read block 276 len 4096
36278 184/300 Read block 1242 len 28672
36279 185/300 Write block 2787 len 16384 data: 5 5 5 7
36280 186/300 Read block 489 len 36864
36281 187/300 Write block 2211 len 8192 data: 8 8
36282 188/300 Read block 1108 len 12288
36283 189/300 Read block 4867 len 20480
36284 190/300 Read block 355 len 8192
36285 191/300 Write block 3779 len 4096 data: 7
36286 192/300 Flush
36287 193/300 Read block 1629 len 16384
36288 194/300 Flush
36289 195/300 Read block 3568 len 40960
36290 196/300 Flush
36291 197/300 Write block 4518 len 32768 data: 5 5 3 3 3 3 3 4
36292 198/300 Write block 1383 len 28672 data: 4 4 6 7 7 8 7
36293 199/300 Read block 1927 len 4096
36294 200/300 Write block 975 len 12288 data: 6 4 4
36295 201/300 Write block 3872 len 16384 data: 7 8 10 10
36296 202/300 Write block 432 len 24576 data: 4 4 5 4 3 3
36297 203/300 Read block 2065 len 8192
36298 204/300 Write block 3099 len 16384 data: 5 5 6 8
36299 205/300 Write block 1137 len 28672 data: 5 5 5 5 5 5 6
36300 206/300 Read block 3160 len 36864
36301 207/300 Read block 2500 len 32768
36302 208/300 Read block 2736 len 16384
36303 209/300 Write block 2844 len 16384 data: 4 5 5 7
36304 210/300 Write block 3750 len 32768 data: 8 6 8 6 7 7 7 6
36305 211/300 Read block 2487 len 40960
36306 212/300 Read block 2616 len 20480
36307 213/300 Read block 4160 len 40960
36308 214/300 Read block 3035 len 12288
36309 215/300 Write block 2805 len 24576 data: 5 4 4 3 7 7
36310 216/300 Read block 3521 len 12288
36311 217/300 Write block 2632 len 20480 data: 6 6 4 6 6
36312 218/300 Write block 4881 len 16384 data: 2 4 5 6
36313 219/300 Read block 3824 len 36864
36314 220/300 Write block 3364 len 36864 data: 5 3 3 4 5 6 6 6 4
36315 221/300 Write block 1774 len 32768 data: 7 9 10 9 9 10 8 6
36316 222/300 Write block 1705 len 36864 data: 7 7 7 7 6 4 6 7 7
36317 223/300 Read block 287 len 20480
36318 224/300 Flush
36319 225/300 Write block 3603 len 24576 data: 4 3 4 5 4 8
36320 226/300 Write block 4048 len 36864 data: 9 8 10 9 8 6 4 5 7
36321 227/300 Read block 4052 len 12288
36322 228/300 Write block 3535 len 4096 data: 7
36323 229/300 Read block 3459 len 40960
36324 230/300 Read block 1428 len 36864
36325 231/300 Write block 4884 len 24576 data: 7 7 10 9 10 10
36326 232/300 Read block 4038 len 40960
36327 233/300 Write block 4612 len 40960 data: 9 8 8 9 8 7 6 7 6 6
36328 234/300 Read block 2243 len 40960
36329 235/300 Write block 346 len 40960 data: 8 8 9 8 8 8 8 5 5 4
36330 236/300 Read block 3998 len 12288
36331 237/300 Read block 2026 len 24576
36332 238/300 Write block 4340 len 8192 data: 3 3
36333 239/300 Read block 4274 len 16384
36334 240/300 Write block 3810 len 24576 data: 5 5 4 3 3 4
36335 241/300 Read block 2170 len 8192
36336 242/300 Read block 2598 len 20480
36337 243/300 Read block 3494 len 40960
36338 244/300 Write block 2388 len 40960 data: 3 3 3 3 3 3 4 5 5 4
36339 245/300 Write block 4402 len 36864 data: 8 7 7 5 5 4 4 4 4
36340 246/300 Read block 3772 len 36864
36341 247/300 Write block 3507 len 12288 data: 4 2 3
36342 248/300 Read block 4497 len 40960
36343 249/300 Read block 2467 len 24576
36344 250/300 Read block 3765 len 4096
36345 251/300 Read block 796 len 28672
36346 252/300 Read block 4824 len 40960
36347 253/300 Write block 3305 len 12288 data: 3 4 4
36348 254/300 Read block 1178 len 32768
36349 255/300 Write block 2093 len 12288 data: 8 8 6
36350 256/300 Read block 50 len 36864
36351 257/300 Write block 3808 len 12288 data: 10 8 6
36352 258/300 Write block 220 len 8192 data: 7 5
36353 259/300 Write block 746 len 8192 data: 9 9
36354 260/300 Write block 2179 len 12288 data: 4 4 4
36355 261/300 Flush
36356 262/300 Write block 99 len 8192 data: 7 7
36357 263/300 Read block 3140 len 12288
36358 264/300 Read block 4503 len 36864
36359 265/300 Read block 3564 len 20480
36360 266/300 Read block 1841 len 24576
36361 267/300 Read block 4583 len 40960
36362 268/300 Read block 1386 len 32768
36363 269/300 Read block 3778 len 4096
36364 270/300 Read block 3448 len 16384
36365 271/300 Write block 1643 len 16384 data: 7 5 5 4
36366 272/300 Read block 1268 len 4096
36367 273/300 Read block 141 len 28672
36368 274/300 Write block 3781 len 8192 data: 5 5
36369 275/300 Read block 1383 len 36864
36370 276/300 Write block 1284 len 12288 data: 10 11 12
36371 277/300 Write block 4356 len 12288 data: 6 6 6
36372 278/300 Read block 1317 len 36864
36373 279/300 Write block 2411 len 8192 data: 7 7
36374 280/300 Write block 2870 len 36864 data: 6 4 3 5 5 6 7 6 5
36375 281/300 Write block 2490 len 36864 data: 7 7 8 7 6 6 5 5 6
36376 282/300 Read block 1987 len 12288
36377 283/300 Write block 4626 len 12288 data: 3 4 4
36378 284/300 Write block 4952 len 12288 data: 4 6 6
36379 285/300 Read block 4379 len 28672
36380 286/300 Write block 2987 len 36864 data: 7 6 7 6 6 5 4 3 3
36381 287/300 Read block 1426 len 12288
36382 288/300 Write block 1294 len 28672 data: 7 7 6 5 6 9 9
36383 289/300 Write block 1252 len 24576 data: 5 5 4 5 5 4
36384 290/300 Write block 1977 len 28672 data: 4 4 4 3 3 5 7
36385 291/300 Write block 1927 len 32768 data: 3 3 3 3 4 4 4 5
36386 292/300 Read block 1151 len 4096
36387 293/300 Read block 4354 len 20480
36388 294/300 Read block 1556 len 20480
36389 295/300 Flush
36390 296/300 Read block 3508 len 40960
36391 297/300 Read block 147 len 20480
36392 298/300 Write block 2651 len 16384 data: 5 4 5 5
36393 299/300 Read block 628 len 36864
36394 300/300 Read block 260 len 28672
36395 301/300 Read block 4448 len 32768
36396 Main task has work to do, go find it
36397 got dsc Start(2)
36398 start 2
36399 Jan 29 05:18:55.304 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: cb4ba64e-fb07-4ef0-bb23-954b4d7e2254, remote_addr: 127.0.0.1:46995, local_addr: 127.0.0.1:9998
36400 Replay: started 2, returned:Ok(())
36401 ----------------------------------------------------------------
36402 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36403 States: Active Active Offline
36404 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36405 10059 Acked Write 3 Done Done Sent false
36406 10060 Acked Write 5 Done Done Sent false
36407 10061 Acked Read 5 Done Done Sent false
36408 10062 Acked Write 9 Done Done Sent false
36409 10063 Acked Write 2 Done Done Sent false
36410 10064 Acked Read 6 Done Done Sent false
36411 10065 Acked Read 4 Done Done Sent false
36412 10066 Acked Write 3 Done Done Sent false
36413 10067 Acked Read 9 Done Done Sent false
36414 10068 Acked Read 2 Done Done Sent false
36415 10069 Acked Write 9 Done Done Sent false
36416 10070 Acked Write 8 Done Done Sent false
36417 10071 Acked Read 9 Done Done Sent false
36418 10072 Acked Read 9 Done Done Sent false
36419 10073 Acked Flush 0 Done Done Sent false
36420 10074 Acked Write 9 Done Done Sent false
36421 10075 Acked Flush 0 Done Done Sent false
36422 10076 Acked Read 9 Done Done Sent false
36423 10077 Acked Flush 0 Done Done Sent false
36424 10078 Acked Read 5 Done Done Sent false
36425 10079 Acked Write 1 Done Done Sent false
36426 10080 Acked Write 7 Done Done Sent false
36427 10081 Acked Read 10 Done Done Sent false
36428 10082 Acked Read 3 Done Done Sent false
36429 10083 Acked Read 7 Done Done Sent false
36430 10084 Acked Read 1 Done Done Sent false
36431 10085 Acked Flush 0 Done Done Sent false
36432 10086 Acked Write 9 Done Done Sent false
36433 10087 Acked Write 7 Done Done Sent false
36434 10088 Acked Write 7 Done Done Sent false
36435 10089 Acked Read 3 Done Done Sent false
36436 10090 Acked Write 7 Done Done Sent false
36437 10091 Acked Read 9 Done Done Sent false
36438 10092 Acked Read 5 Done Done Sent false
36439 10093 Acked Read 7 Done Done Sent false
36440 10094 Acked Read 1 Done Done Sent false
36441 10095 Acked Write 7 Done Done Sent false
36442 10096 Acked Read 6 Done Done Sent false
36443 10097 Acked Flush 0 Done Done Sent false
36444 10098 Acked Read 3 Done Done Sent false
36445 10099 Acked Write 10 Done Done Sent false
36446 10100 Acked Flush 0 Done Done Sent false
36447 10101 Acked Read 2 Done Done Sent false
36448 10102 Acked Read 2 Done Done Sent false
36449 10103 Acked Read 8 Done Done Sent false
36450 10104 Acked Write 4 Done Done Sent false
36451 10105 Acked Read 10 Done Done Sent false
36452 10106 Acked Write 1 Done Done Sent false
36453 10107 Acked Read 6 Done Done Sent false
36454 10108 Acked Read 9 Done Done Sent false
36455 10109 Acked Read 7 Done Done Sent false
36456 10110 Acked Write 10 Done Done Sent false
36457 10111 Acked Write 9 Done Done Sent false
36458 10112 Acked Read 10 Done Done Sent false
36459 10113 Acked Read 1 Done Done Sent false
36460 10114 Acked Read 10 Done Done Sent false
36461 10115 Acked Write 2 Done Done Sent false
36462 10116 Acked Read 3 Done Done Sent false
36463 10117 Acked Flush 0 Done Done Sent false
36464 10118 Acked Write 1 Done Done Sent false
36465 10119 Acked Flush 0 Done Done Sent false
36466 10120 Acked Read 9 Done Done Sent false
36467 10121 Acked Read 5 Done Done Sent false
36468 10122 Acked Write 5 Done Done Sent false
36469 10123 Acked Write 6 Done Done Sent false
36470 10124 Acked Read 2 Done Done Sent false
36471 10125 Acked Write 7 Done Done Sent false
36472 10126 Acked Write 10 Done Done Sent false
36473 10127 Acked Flush 0 Done Done Sent false
36474 10128 Acked Write 4 Done Done Sent false
36475 10129 Acked Read 1 Done Done Sent false
36476 10130 Acked Write 9 Done Done Sent false
36477 10131 Acked Flush 0 Done Done Sent false
36478 10132 Acked Write 10 Done Done Sent false
36479 10133 Acked Write 10 Done Done Sent false
36480 10134 Acked Read 8 Done Done Sent false
36481 10135 Acked Read 8 Done Done Sent false
36482 10136 Acked Read 8 Done Done Sent false
36483 10137 Acked Read 7 Done Done Sent false
36484 10138 Acked Read 10 Done Done Sent false
36485 10139 Acked Flush 0 Done Done Sent false
36486 10140 Acked Write 9 Done Done Sent false
36487 10141 Acked Read 8 Done Done Sent false
36488 10142 Acked Flush 0 Done Done Sent false
36489 10143 Acked Read 3 Done Done Sent false
36490 10144 Acked Write 7 Done Done Sent false
36491 10145 Acked Read 4 Done Done Sent false
36492 10146 Acked Flush 0 Done Done Sent false
36493 10147 Acked Read 3 Done Done Sent false
36494 10148 Acked Read 8 Done Done Sent false
36495 10149 Acked Write 10 Done Done Sent false
36496 10150 Acked Write 3 Done Done Sent false
36497 10151 Acked Write 5 Done Done Sent false
36498 10152 Acked Read 3 Done Done Sent false
36499 10153 Acked Write 9 Done Done Sent false
36500 10154 Acked Flush 0 Done Done Sent false
36501 10155 Acked Read 9 Done Done Sent false
36502 10156 Acked Read 4 Done Done Sent false
36503 10157 Acked Write 9 Done Done Sent false
36504 10158 Acked Read 3 Done Done Sent false
36505 10159 Acked Read 10 Done Done Sent false
36506 10160 Acked Flush 0 Done Done Sent false
36507 10161 Acked Write 9 Done Done Sent false
36508 10162 Acked Write 2 Done Done Sent false
36509 10163 Acked Write 3 Done Done Sent false
36510 10164 Acked Read 3 Done Done Sent false
36511 10165 Acked Write 9 Done Done Sent false
36512 10166 Acked Write 2 Done Done Sent false
36513 10167 Acked Read 4 Done Done Sent false
36514 10168 Acked Read 6 Done Done Sent false
36515 10169 Acked Write 4 Done Done Sent false
36516 10170 Acked Read 4 Done Done Sent false
36517 10171 Acked Write 8 Done Done Sent false
36518 10172 Acked Read 9 Done Done Sent false
36519 10173 Acked Flush 0 Done Done Sent false
36520 10174 Acked Write 8 Done Done Sent false
36521 10175 Acked Read 5 Done Done Sent false
36522 10176 Acked Write 4 Done Done Sent false
36523 10177 Acked Read 1 Done Done Sent false
36524 10178 Acked Read 1 Done Done Sent false
36525 10179 Acked Write 1 Done Done Sent false
36526 10180 Acked Read 4 Done Done Sent false
36527 10181 Acked Write 7 Done Done Sent false
36528 10182 Acked Flush 0 Done Done Sent false
36529 10183 Acked Write 6 Done Done Sent false
36530 10184 Acked Read 1 Done Done Sent false
36531 10185 Acked Read 3 Done Done Sent false
36532 10186 Acked Read 8 Done Done Sent false
36533 10187 Acked Read 7 Done Done Sent false
36534 10188 Acked Write 5 Done Done Sent false
36535 10189 Acked Write 4 Done Done Sent false
36536 10190 Acked Write 10 Done Done Sent false
36537 10191 Acked Read 3 Done Done Sent false
36538 10192 Acked Read 6 Done Done Sent false
36539 10193 Acked Write 3 Done Done Sent false
36540 10194 Acked Read 3 Done Done Sent false
36541 10195 Acked Read 7 Done Done Sent false
36542 10196 Acked Read 3 Done Done Sent false
36543 10197 Acked Read 2 Done Done Sent false
36544 10198 Acked Write 3 Done Done Sent false
36545 10199 Acked Write 6 Done Done Sent false
36546 10200 Acked Write 2 Done Done Sent false
36547 10201 Acked Write 4 Done Done Sent false
36548 10202 Acked Read 8 Done Done Sent false
36549 10203 Acked Write 9 Done Done Sent false
36550 10204 Acked Write 6 Done Done Sent false
36551 10205 Acked Read 1 Done Done Sent false
36552 10206 Acked Write 9 Done Done Sent false
36553 10207 Acked Read 6 Done Done Sent false
36554 10208 Acked Flush 0 Done Done Sent false
36555 10209 Acked Flush 0 Done Done Sent false
36556 10210 Acked Write 4 Done Done Sent false
36557 10211 Acked Write 8 Done Done Sent false
36558 10212 Acked Write 8 Done Done Sent false
36559 10213 Acked Write 5 Done Done Sent false
36560 10214 Acked Read 1 Done Done Sent false
36561 10215 Acked Read 6 Done Done Sent false
36562 10216 Acked Write 1 Done Done Sent false
36563 10217 Acked Read 3 Done Done Sent false
36564 10218 Acked Read 10 Done Done Sent false
36565 10219 Acked Read 1 Done Done Sent false
36566 10220 Acked Read 1 Done Done Sent false
36567 10221 Acked Write 2 Done Done Sent false
36568 10222 Acked Read 1 Done Done Sent false
36569 10223 Acked Write 10 Done Done Sent false
36570 10224 Acked Flush 0 Done Done Sent false
36571 10225 Acked Write 5 Done Done Sent false
36572 10226 Acked Write 7 Done Done Sent false
36573 10227 Acked Write 2 Done Done Sent false
36574 10228 Acked Flush 0 Done Done Sent false
36575 10229 Acked Read 2 Done Done Sent false
36576 10230 Acked Flush 0 Done Done Sent false
36577 10231 Acked Read 2 Done Done Sent false
36578 10232 Acked Flush 0 Done Done Sent false
36579 10233 Acked Read 7 Done Done Sent false
36580 10234 Acked Read 9 Done Done Sent false
36581 10235 Acked Write 2 Done Done Sent false
36582 10236 Acked Read 5 Done Done Sent false
36583 10237 Acked Write 2 Done Done Sent false
36584 10238 Acked Write 4 Done Done Sent false
36585 10239 Acked Write 8 Done Done Sent false
36586 10240 Acked Write 6 Done Done Sent false
36587 10241 Acked Read 1 Done Done Sent false
36588 10242 Acked Read 7 Done Done Sent false
36589 10243 Acked Write 4 Done Done Sent false
36590 10244 Acked Read 9 Done Done Sent false
36591 10245 Acked Write 2 Done Done Sent false
36592 10246 Acked Read 3 Done Done Sent false
36593 10247 Acked Read 5 Done Done Sent false
36594 10248 Acked Read 2 Done Done Sent false
36595 10249 Acked Write 1 Done Done Sent false
36596 10250 Acked Flush 0 Done Done Sent false
36597 10251 Acked Read 4 Done Done Sent false
36598 10252 Acked Flush 0 Done Done Sent false
36599 10253 Acked Read 10 Done Done Sent false
36600 10254 Acked Flush 0 Done Done Sent false
36601 10255 Acked Write 8 Done Done Sent false
36602 10256 Acked Write 7 Done Done Sent false
36603 10257 Acked Read 1 Done Done Sent false
36604 10258 Acked Write 3 Done Done Sent false
36605 10259 Acked Write 4 Done Done Sent false
36606 10260 Acked Write 6 Done Done Sent false
36607 10261 Acked Read 2 Done Done Sent false
36608 10262 Acked Write 4 Done Done Sent false
36609 10263 Acked Write 7 Done Done Sent false
36610 10264 Acked Read 9 Done Done Sent false
36611 10265 Acked Read 8 Done Done Sent false
36612 10266 Acked Read 4 Done Done Sent false
36613 10267 Acked Write 4 Done Done Sent false
36614 10268 Acked Write 8 Done Done Sent false
36615 10269 Acked Read 10 Done Done Sent false
36616 10270 Acked Read 5 Done Done Sent false
36617 10271 Acked Read 10 Done Done Sent false
36618 10272 Acked Read 3 Done Done Sent false
36619 10273 Acked Write 6 Done Done Sent false
36620 10274 Acked Read 3 Done Done Sent false
36621 10275 Acked Write 5 Done Done Sent false
36622 10276 Acked Write 4 Done Done Sent false
36623 10277 Acked Read 9 Done Done Sent false
36624 10278 Acked Write 9 Done Done Sent false
36625 10279 Acked Write 8 Done Done Sent false
36626 10280 Acked Write 9 Done Done Sent false
36627 10281 Acked Read 5 Done Done Sent false
36628 10282 Acked Flush 0 Done Done Sent false
36629 10283 Acked Write 6 Done Done Sent false
36630 10284 Acked Write 9 Done Done Sent false
36631 10285 Acked Read 3 Done Done Sent false
36632 10286 Acked Write 1 Done Done Sent false
36633 10287 Acked Read 10 Done Done Sent false
36634 10288 Acked Read 9 Done Done Sent false
36635 10289 Acked Write 6 Done Done Sent false
36636 10290 Acked Read 10 Done Done Sent false
36637 10291 Acked Write 10 Done Done Sent false
36638 10292 Acked Read 10 Done Done Sent false
36639 10293 Acked Write 10 Done Done Sent false
36640 10294 Acked Read 3 Done Done Sent false
36641 10295 Acked Read 6 Done Done Sent false
36642 10296 Acked Write 2 Done Done Sent false
36643 10297 Acked Read 4 Done Done Sent false
36644 10298 Acked Write 6 Done Done Sent false
36645 10299 Acked Read 2 Done Done Sent false
36646 10300 Acked Read 5 Done Done Sent false
36647 10301 Acked Read 10 Done Done Sent false
36648 10302 Acked Write 10 Done Done Sent false
36649 10303 Acked Write 9 Done Done Sent false
36650 10304 Acked Read 9 Done Done Sent false
36651 10305 Acked Write 3 Done Done Sent false
36652 10306 Acked Read 10 Done Done Sent false
36653 10307 Acked Read 6 Done Done Sent false
36654 10308 Acked Read 1 Done Done Sent false
36655 10309 Acked Read 7 Done Done Sent false
36656 10310 Acked Read 10 Done Done Sent false
36657 10311 Acked Write 3 Done Done Sent false
36658 10312 Acked Read 8 Done Done Sent false
36659 10313 Acked Write 3 Done Done Sent false
36660 10314 Acked Read 9 Done Done Sent false
36661 10315 Acked Write 3 Done Done Sent false
36662 10316 Acked Write 2 Done Done Sent false
36663 10317 Acked Write 2 Done Done Sent false
36664 10318 Acked Write 3 Done Done Sent false
36665 10319 Acked Flush 0 Done Done Sent false
36666 10320 Acked Write 2 Done Done Sent false
36667 10321 Acked Read 3 Done Done Sent false
36668 10322 Acked Read 9 Done Done Sent false
36669 10323 Acked Read 5 Done Done Sent false
36670 10324 Acked Read 6 Done Done Sent false
36671 10325 Acked Read 10 Done Done Sent false
36672 10326 Acked Read 8 Done Done Sent false
36673 10327 Acked Read 1 Done Done Sent false
36674 10328 Acked Read 4 Done Done Sent false
36675 10329 Acked Write 4 Done Done Sent false
36676 10330 Acked Read 1 Done Done Sent false
36677 10331 Acked Read 7 Done Done Sent false
36678 10332 Acked Write 2 Done Done Sent false
36679 10333 Acked Read 9 Done Done Sent false
36680 10334 Acked Write 3 Done Done Sent false
36681 10335 Acked Write 3 Done Done Sent false
36682 10336 Acked Read 9 Done Done Sent false
36683 10337 Acked Write 2 Done Done Sent false
36684 10338 Acked Write 9 Done Done Sent false
36685 10339 Acked Write 9 Done Done Sent false
36686 10340 Acked Read 3 Done Done Sent false
36687 10341 Acked Write 3 Done Done Sent false
36688 10342 Acked Write 3 Done Done Sent false
36689 10343 Acked Read 7 Done Done Sent false
36690 10344 Acked Write 9 Done Done Sent false
36691 10345 Acked Read 3 Done Done Sent false
36692 10346 Acked Write 7 Done Done Sent false
36693 10347 Acked Write 6 Done Done Sent false
36694 10348 Acked Write 7 Done Done Sent false
36695 10349 Acked Write 8 Done Done Sent false
36696 10350 Acked Read 1 Done Done Sent false
36697 10351 Acked Read 5 Done Done Sent false
36698 10352 Acked Read 5 Done Done Sent false
36699 10353 Acked Flush 0 Done Done Sent false
36700 10354 Acked Read 10 Done Done Sent false
36701 10355 Acked Read 5 Done Done Sent false
36702 10356 Acked Write 4 Done Done Sent false
36703 10357 Acked Read 9 Done Done Sent false
36704 10358 Acked Read 7 Done Done Sent false
36705 10359 Acked Read 8 Done Done Sent false
36706 STATES DS:0 DS:1 DS:2 TOTAL
36707 Sent 0 0 301 301
36708 Done 301 301 0 602
36709 Skipped 0 0 0 0
36710 Error 0 0 0 0
36711 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10058))
36712 Downstairs last five completed: 10058 10057 10056 10055 10054
36713 Upstairs last five completed: 10359 10358 10357 10356 10355
36714 CLIENT: Up:0 ds:301 act:2
367152025-01-29T05:18:55.806ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
367162025-01-29T05:18:56.744ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
367172025-01-29T05:18:57.745ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
367182025-01-29T05:18:58.745ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
36719 ----------------------------------------------------------------
36720 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36721 States: Active Active Offline
36722 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36723 10059 Acked Write 3 Done Done Sent false
36724 10060 Acked Write 5 Done Done Sent false
36725 10061 Acked Read 5 Done Done Sent false
36726 10062 Acked Write 9 Done Done Sent false
36727 10063 Acked Write 2 Done Done Sent false
36728 10064 Acked Read 6 Done Done Sent false
36729 10065 Acked Read 4 Done Done Sent false
36730 10066 Acked Write 3 Done Done Sent false
36731 10067 Acked Read 9 Done Done Sent false
36732 10068 Acked Read 2 Done Done Sent false
36733 10069 Acked Write 9 Done Done Sent false
36734 10070 Acked Write 8 Done Done Sent false
36735 10071 Acked Read 9 Done Done Sent false
36736 10072 Acked Read 9 Done Done Sent false
36737 10073 Acked Flush 0 Done Done Sent false
36738 10074 Acked Write 9 Done Done Sent false
36739 10075 Acked Flush 0 Done Done Sent false
36740 10076 Acked Read 9 Done Done Sent false
36741 10077 Acked Flush 0 Done Done Sent false
36742 10078 Acked Read 5 Done Done Sent false
36743 10079 Acked Write 1 Done Done Sent false
36744 10080 Acked Write 7 Done Done Sent false
36745 10081 Acked Read 10 Done Done Sent false
36746 10082 Acked Read 3 Done Done Sent false
36747 10083 Acked Read 7 Done Done Sent false
36748 10084 Acked Read 1 Done Done Sent false
36749 10085 Acked Flush 0 Done Done Sent false
36750 10086 Acked Write 9 Done Done Sent false
36751 10087 Acked Write 7 Done Done Sent false
36752 10088 Acked Write 7 Done Done Sent false
36753 10089 Acked Read 3 Done Done Sent false
36754 10090 Acked Write 7 Done Done Sent false
36755 10091 Acked Read 9 Done Done Sent false
36756 10092 Acked Read 5 Done Done Sent false
36757 10093 Acked Read 7 Done Done Sent false
36758 10094 Acked Read 1 Done Done Sent false
36759 10095 Acked Write 7 Done Done Sent false
36760 10096 Acked Read 6 Done Done Sent false
36761 10097 Acked Flush 0 Done Done Sent false
36762 10098 Acked Read 3 Done Done Sent false
36763 10099 Acked Write 10 Done Done Sent false
36764 10100 Acked Flush 0 Done Done Sent false
36765 10101 Acked Read 2 Done Done Sent false
36766 10102 Acked Read 2 Done Done Sent false
36767 10103 Acked Read 8 Done Done Sent false
36768 10104 Acked Write 4 Done Done Sent false
36769 10105 Acked Read 10 Done Done Sent false
36770 10106 Acked Write 1 Done Done Sent false
36771 10107 Acked Read 6 Done Done Sent false
36772 10108 Acked Read 9 Done Done Sent false
36773 10109 Acked Read 7 Done Done Sent false
36774 10110 Acked Write 10 Done Done Sent false
36775 10111 Acked Write 9 Done Done Sent false
36776 10112 Acked Read 10 Done Done Sent false
36777 10113 Acked Read 1 Done Done Sent false
36778 10114 Acked Read 10 Done Done Sent false
36779 10115 Acked Write 2 Done Done Sent false
36780 10116 Acked Read 3 Done Done Sent false
36781 10117 Acked Flush 0 Done Done Sent false
36782 10118 Acked Write 1 Done Done Sent false
36783 10119 Acked Flush 0 Done Done Sent false
36784 10120 Acked Read 9 Done Done Sent false
36785 10121 Acked Read 5 Done Done Sent false
36786 10122 Acked Write 5 Done Done Sent false
36787 10123 Acked Write 6 Done Done Sent false
36788 10124 Acked Read 2 Done Done Sent false
36789 10125 Acked Write 7 Done Done Sent false
36790 10126 Acked Write 10 Done Done Sent false
36791 10127 Acked Flush 0 Done Done Sent false
36792 10128 Acked Write 4 Done Done Sent false
36793 10129 Acked Read 1 Done Done Sent false
36794 10130 Acked Write 9 Done Done Sent false
36795 10131 Acked Flush 0 Done Done Sent false
36796 10132 Acked Write 10 Done Done Sent false
36797 10133 Acked Write 10 Done Done Sent false
36798 10134 Acked Read 8 Done Done Sent false
36799 10135 Acked Read 8 Done Done Sent false
36800 10136 Acked Read 8 Done Done Sent false
36801 10137 Acked Read 7 Done Done Sent false
36802 10138 Acked Read 10 Done Done Sent false
36803 10139 Acked Flush 0 Done Done Sent false
36804 10140 Acked Write 9 Done Done Sent false
36805 10141 Acked Read 8 Done Done Sent false
36806 10142 Acked Flush 0 Done Done Sent false
36807 10143 Acked Read 3 Done Done Sent false
36808 10144 Acked Write 7 Done Done Sent false
36809 10145 Acked Read 4 Done Done Sent false
36810 10146 Acked Flush 0 Done Done Sent false
36811 10147 Acked Read 3 Done Done Sent false
36812 10148 Acked Read 8 Done Done Sent false
36813 10149 Acked Write 10 Done Done Sent false
36814 10150 Acked Write 3 Done Done Sent false
36815 10151 Acked Write 5 Done Done Sent false
36816 10152 Acked Read 3 Done Done Sent false
36817 10153 Acked Write 9 Done Done Sent false
36818 10154 Acked Flush 0 Done Done Sent false
36819 10155 Acked Read 9 Done Done Sent false
36820 10156 Acked Read 4 Done Done Sent false
36821 10157 Acked Write 9 Done Done Sent false
36822 10158 Acked Read 3 Done Done Sent false
36823 10159 Acked Read 10 Done Done Sent false
36824 10160 Acked Flush 0 Done Done Sent false
36825 10161 Acked Write 9 Done Done Sent false
36826 10162 Acked Write 2 Done Done Sent false
36827 10163 Acked Write 3 Done Done Sent false
36828 10164 Acked Read 3 Done Done Sent false
36829 10165 Acked Write 9 Done Done Sent false
36830 10166 Acked Write 2 Done Done Sent false
36831 10167 Acked Read 4 Done Done Sent false
36832 10168 Acked Read 6 Done Done Sent false
36833 10169 Acked Write 4 Done Done Sent false
36834 10170 Acked Read 4 Done Done Sent false
36835 10171 Acked Write 8 Done Done Sent false
36836 10172 Acked Read 9 Done Done Sent false
36837 10173 Acked Flush 0 Done Done Sent false
36838 10174 Acked Write 8 Done Done Sent false
36839 10175 Acked Read 5 Done Done Sent false
36840 10176 Acked Write 4 Done Done Sent false
36841 10177 Acked Read 1 Done Done Sent false
36842 10178 Acked Read 1 Done Done Sent false
36843 10179 Acked Write 1 Done Done Sent false
36844 10180 Acked Read 4 Done Done Sent false
36845 10181 Acked Write 7 Done Done Sent false
36846 10182 Acked Flush 0 Done Done Sent false
36847 10183 Acked Write 6 Done Done Sent false
36848 10184 Acked Read 1 Done Done Sent false
36849 10185 Acked Read 3 Done Done Sent false
36850 10186 Acked Read 8 Done Done Sent false
36851 10187 Acked Read 7 Done Done Sent false
36852 10188 Acked Write 5 Done Done Sent false
36853 10189 Acked Write 4 Done Done Sent false
36854 10190 Acked Write 10 Done Done Sent false
36855 10191 Acked Read 3 Done Done Sent false
36856 10192 Acked Read 6 Done Done Sent false
36857 10193 Acked Write 3 Done Done Sent false
36858 10194 Acked Read 3 Done Done Sent false
36859 10195 Acked Read 7 Done Done Sent false
36860 10196 Acked Read 3 Done Done Sent false
36861 10197 Acked Read 2 Done Done Sent false
36862 10198 Acked Write 3 Done Done Sent false
36863 10199 Acked Write 6 Done Done Sent false
36864 10200 Acked Write 2 Done Done Sent false
36865 10201 Acked Write 4 Done Done Sent false
36866 10202 Acked Read 8 Done Done Sent false
36867 10203 Acked Write 9 Done Done Sent false
36868 10204 Acked Write 6 Done Done Sent false
36869 10205 Acked Read 1 Done Done Sent false
36870 10206 Acked Write 9 Done Done Sent false
36871 10207 Acked Read 6 Done Done Sent false
36872 10208 Acked Flush 0 Done Done Sent false
36873 10209 Acked Flush 0 Done Done Sent false
36874 10210 Acked Write 4 Done Done Sent false
36875 10211 Acked Write 8 Done Done Sent false
36876 10212 Acked Write 8 Done Done Sent false
36877 10213 Acked Write 5 Done Done Sent false
36878 10214 Acked Read 1 Done Done Sent false
36879 10215 Acked Read 6 Done Done Sent false
36880 10216 Acked Write 1 Done Done Sent false
36881 10217 Acked Read 3 Done Done Sent false
36882 10218 Acked Read 10 Done Done Sent false
36883 10219 Acked Read 1 Done Done Sent false
36884 10220 Acked Read 1 Done Done Sent false
36885 10221 Acked Write 2 Done Done Sent false
36886 10222 Acked Read 1 Done Done Sent false
36887 10223 Acked Write 10 Done Done Sent false
36888 10224 Acked Flush 0 Done Done Sent false
36889 10225 Acked Write 5 Done Done Sent false
36890 10226 Acked Write 7 Done Done Sent false
36891 10227 Acked Write 2 Done Done Sent false
36892 10228 Acked Flush 0 Done Done Sent false
36893 10229 Acked Read 2 Done Done Sent false
36894 10230 Acked Flush 0 Done Done Sent false
36895 10231 Acked Read 2 Done Done Sent false
36896 10232 Acked Flush 0 Done Done Sent false
36897 10233 Acked Read 7 Done Done Sent false
36898 10234 Acked Read 9 Done Done Sent false
36899 10235 Acked Write 2 Done Done Sent false
36900 10236 Acked Read 5 Done Done Sent false
36901 10237 Acked Write 2 Done Done Sent false
36902 10238 Acked Write 4 Done Done Sent false
36903 10239 Acked Write 8 Done Done Sent false
36904 10240 Acked Write 6 Done Done Sent false
36905 10241 Acked Read 1 Done Done Sent false
36906 10242 Acked Read 7 Done Done Sent false
36907 10243 Acked Write 4 Done Done Sent false
36908 10244 Acked Read 9 Done Done Sent false
36909 10245 Acked Write 2 Done Done Sent false
36910 10246 Acked Read 3 Done Done Sent false
36911 10247 Acked Read 5 Done Done Sent false
36912 10248 Acked Read 2 Done Done Sent false
36913 10249 Acked Write 1 Done Done Sent false
36914 10250 Acked Flush 0 Done Done Sent false
36915 10251 Acked Read 4 Done Done Sent false
36916 10252 Acked Flush 0 Done Done Sent false
36917 10253 Acked Read 10 Done Done Sent false
36918 10254 Acked Flush 0 Done Done Sent false
36919 10255 Acked Write 8 Done Done Sent false
36920 10256 Acked Write 7 Done Done Sent false
36921 10257 Acked Read 1 Done Done Sent false
36922 10258 Acked Write 3 Done Done Sent false
36923 10259 Acked Write 4 Done Done Sent false
36924 10260 Acked Write 6 Done Done Sent false
36925 10261 Acked Read 2 Done Done Sent false
36926 10262 Acked Write 4 Done Done Sent false
36927 10263 Acked Write 7 Done Done Sent false
36928 10264 Acked Read 9 Done Done Sent false
36929 10265 Acked Read 8 Done Done Sent false
36930 10266 Acked Read 4 Done Done Sent false
36931 10267 Acked Write 4 Done Done Sent false
36932 10268 Acked Write 8 Done Done Sent false
36933 10269 Acked Read 10 Done Done Sent false
36934 10270 Acked Read 5 Done Done Sent false
36935 10271 Acked Read 10 Done Done Sent false
36936 10272 Acked Read 3 Done Done Sent false
36937 10273 Acked Write 6 Done Done Sent false
36938 10274 Acked Read 3 Done Done Sent false
36939 10275 Acked Write 5 Done Done Sent false
36940 10276 Acked Write 4 Done Done Sent false
36941 10277 Acked Read 9 Done Done Sent false
36942 10278 Acked Write 9 Done Done Sent false
36943 10279 Acked Write 8 Done Done Sent false
36944 10280 Acked Write 9 Done Done Sent false
36945 10281 Acked Read 5 Done Done Sent false
36946 10282 Acked Flush 0 Done Done Sent false
36947 10283 Acked Write 6 Done Done Sent false
36948 10284 Acked Write 9 Done Done Sent false
36949 10285 Acked Read 3 Done Done Sent false
36950 10286 Acked Write 1 Done Done Sent false
36951 10287 Acked Read 10 Done Done Sent false
36952 10288 Acked Read 9 Done Done Sent false
36953 10289 Acked Write 6 Done Done Sent false
36954 10290 Acked Read 10 Done Done Sent false
36955 10291 Acked Write 10 Done Done Sent false
36956 10292 Acked Read 10 Done Done Sent false
36957 10293 Acked Write 10 Done Done Sent false
36958 10294 Acked Read 3 Done Done Sent false
36959 10295 Acked Read 6 Done Done Sent false
36960 10296 Acked Write 2 Done Done Sent false
36961 10297 Acked Read 4 Done Done Sent false
36962 10298 Acked Write 6 Done Done Sent false
36963 10299 Acked Read 2 Done Done Sent false
36964 10300 Acked Read 5 Done Done Sent false
36965 10301 Acked Read 10 Done Done Sent false
36966 10302 Acked Write 10 Done Done Sent false
36967 10303 Acked Write 9 Done Done Sent false
36968 10304 Acked Read 9 Done Done Sent false
36969 10305 Acked Write 3 Done Done Sent false
36970 10306 Acked Read 10 Done Done Sent false
36971 10307 Acked Read 6 Done Done Sent false
36972 10308 Acked Read 1 Done Done Sent false
36973 10309 Acked Read 7 Done Done Sent false
36974 10310 Acked Read 10 Done Done Sent false
36975 10311 Acked Write 3 Done Done Sent false
36976 10312 Acked Read 8 Done Done Sent false
36977 10313 Acked Write 3 Done Done Sent false
36978 10314 Acked Read 9 Done Done Sent false
36979 10315 Acked Write 3 Done Done Sent false
36980 10316 Acked Write 2 Done Done Sent false
36981 10317 Acked Write 2 Done Done Sent false
36982 10318 Acked Write 3 Done Done Sent false
36983 10319 Acked Flush 0 Done Done Sent false
36984 10320 Acked Write 2 Done Done Sent false
36985 10321 Acked Read 3 Done Done Sent false
36986 10322 Acked Read 9 Done Done Sent false
36987 10323 Acked Read 5 Done Done Sent false
36988 10324 Acked Read 6 Done Done Sent false
36989 10325 Acked Read 10 Done Done Sent false
36990 10326 Acked Read 8 Done Done Sent false
36991 10327 Acked Read 1 Done Done Sent false
36992 10328 Acked Read 4 Done Done Sent false
36993 10329 Acked Write 4 Done Done Sent false
36994 10330 Acked Read 1 Done Done Sent false
36995 10331 Acked Read 7 Done Done Sent false
36996 10332 Acked Write 2 Done Done Sent false
36997 10333 Acked Read 9 Done Done Sent false
36998 10334 Acked Write 3 Done Done Sent false
36999 10335 Acked Write 3 Done Done Sent false
37000 10336 Acked Read 9 Done Done Sent false
37001 10337 Acked Write 2 Done Done Sent false
37002 10338 Acked Write 9 Done Done Sent false
37003 10339 Acked Write 9 Done Done Sent false
37004 10340 Acked Read 3 Done Done Sent false
37005 10341 Acked Write 3 Done Done Sent false
37006 10342 Acked Write 3 Done Done Sent false
37007 10343 Acked Read 7 Done Done Sent false
37008 10344 Acked Write 9 Done Done Sent false
37009 10345 Acked Read 3 Done Done Sent false
37010 10346 Acked Write 7 Done Done Sent false
37011 10347 Acked Write 6 Done Done Sent false
37012 10348 Acked Write 7 Done Done Sent false
37013 10349 Acked Write 8 Done Done Sent false
37014 10350 Acked Read 1 Done Done Sent false
37015 10351 Acked Read 5 Done Done Sent false
37016 10352 Acked Read 5 Done Done Sent false
37017 10353 Acked Flush 0 Done Done Sent false
37018 10354 Acked Read 10 Done Done Sent false
37019 10355 Acked Read 5 Done Done Sent false
37020 10356 Acked Write 4 Done Done Sent false
37021 10357 Acked Read 9 Done Done Sent false
37022 10358 Acked Read 7 Done Done Sent false
37023 10359 Acked Read 8 Done Done Sent false
37024 STATES DS:0 DS:1 DS:2 TOTAL
37025 Sent 0 0 301 301
37026 Done 301 301 0 602
37027 Skipped 0 0 0 0
37028 Error 0 0 0 0
37029 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10058))
37030 Downstairs last five completed: 10058 10057 10056 10055 10054
37031 Upstairs last five completed: 10359 10358 10357 10356 10355
37032 CLIENT: Up:0 ds:301 act:2
370332025-01-29T05:18:59.808ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370342025-01-29T05:19:00.793ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370352025-01-29T05:19:01.059ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370362025-01-29T05:19:01.060ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370372025-01-29T05:19:01.060ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370382025-01-29T05:19:01.060ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370392025-01-29T05:19:01.060ZINFOcrucible: client task is exiting = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37040 [8830] got start action
37041 [8830] I am going to restart
37042 Starting downstairs at port 8830
37043 Make output file at "/tmp/dsc/downstairs-8830.txt"
37044 [8830][2] reports Starting
37045 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1233)
37046 [8830][2] reports Running
370472025-01-29T05:19:01.750ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
370482025-01-29T05:19:02.751ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37049 ----------------------------------------------------------------
37050 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37051 States: Active Active Offline
37052 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37053 10059 Acked Write 3 Done Done Sent false
37054 10060 Acked Write 5 Done Done Sent false
37055 10061 Acked Read 5 Done Done Sent false
37056 10062 Acked Write 9 Done Done Sent false
37057 10063 Acked Write 2 Done Done Sent false
37058 10064 Acked Read 6 Done Done Sent false
37059 10065 Acked Read 4 Done Done Sent false
37060 10066 Acked Write 3 Done Done Sent false
37061 10067 Acked Read 9 Done Done Sent false
37062 10068 Acked Read 2 Done Done Sent false
37063 10069 Acked Write 9 Done Done Sent false
37064 10070 Acked Write 8 Done Done Sent false
37065 10071 Acked Read 9 Done Done Sent false
37066 10072 Acked Read 9 Done Done Sent false
37067 10073 Acked Flush 0 Done Done Sent false
37068 10074 Acked Write 9 Done Done Sent false
37069 10075 Acked Flush 0 Done Done Sent false
37070 10076 Acked Read 9 Done Done Sent false
37071 10077 Acked Flush 0 Done Done Sent false
37072 10078 Acked Read 5 Done Done Sent false
37073 10079 Acked Write 1 Done Done Sent false
37074 10080 Acked Write 7 Done Done Sent false
37075 10081 Acked Read 10 Done Done Sent false
37076 10082 Acked Read 3 Done Done Sent false
37077 10083 Acked Read 7 Done Done Sent false
37078 10084 Acked Read 1 Done Done Sent false
37079 10085 Acked Flush 0 Done Done Sent false
37080 10086 Acked Write 9 Done Done Sent false
37081 10087 Acked Write 7 Done Done Sent false
37082 10088 Acked Write 7 Done Done Sent false
37083 10089 Acked Read 3 Done Done Sent false
37084 10090 Acked Write 7 Done Done Sent false
37085 10091 Acked Read 9 Done Done Sent false
37086 10092 Acked Read 5 Done Done Sent false
37087 10093 Acked Read 7 Done Done Sent false
37088 10094 Acked Read 1 Done Done Sent false
37089 10095 Acked Write 7 Done Done Sent false
37090 10096 Acked Read 6 Done Done Sent false
37091 10097 Acked Flush 0 Done Done Sent false
37092 10098 Acked Read 3 Done Done Sent false
37093 10099 Acked Write 10 Done Done Sent false
37094 10100 Acked Flush 0 Done Done Sent false
37095 10101 Acked Read 2 Done Done Sent false
37096 10102 Acked Read 2 Done Done Sent false
37097 10103 Acked Read 8 Done Done Sent false
37098 10104 Acked Write 4 Done Done Sent false
37099 10105 Acked Read 10 Done Done Sent false
37100 10106 Acked Write 1 Done Done Sent false
37101 10107 Acked Read 6 Done Done Sent false
37102 10108 Acked Read 9 Done Done Sent false
37103 10109 Acked Read 7 Done Done Sent false
37104 10110 Acked Write 10 Done Done Sent false
37105 10111 Acked Write 9 Done Done Sent false
37106 10112 Acked Read 10 Done Done Sent false
37107 10113 Acked Read 1 Done Done Sent false
37108 10114 Acked Read 10 Done Done Sent false
37109 10115 Acked Write 2 Done Done Sent false
37110 10116 Acked Read 3 Done Done Sent false
37111 10117 Acked Flush 0 Done Done Sent false
37112 10118 Acked Write 1 Done Done Sent false
37113 10119 Acked Flush 0 Done Done Sent false
37114 10120 Acked Read 9 Done Done Sent false
37115 10121 Acked Read 5 Done Done Sent false
37116 10122 Acked Write 5 Done Done Sent false
37117 10123 Acked Write 6 Done Done Sent false
37118 10124 Acked Read 2 Done Done Sent false
37119 10125 Acked Write 7 Done Done Sent false
37120 10126 Acked Write 10 Done Done Sent false
37121 10127 Acked Flush 0 Done Done Sent false
37122 10128 Acked Write 4 Done Done Sent false
37123 10129 Acked Read 1 Done Done Sent false
37124 10130 Acked Write 9 Done Done Sent false
37125 10131 Acked Flush 0 Done Done Sent false
37126 10132 Acked Write 10 Done Done Sent false
37127 10133 Acked Write 10 Done Done Sent false
37128 10134 Acked Read 8 Done Done Sent false
37129 10135 Acked Read 8 Done Done Sent false
37130 10136 Acked Read 8 Done Done Sent false
37131 10137 Acked Read 7 Done Done Sent false
37132 10138 Acked Read 10 Done Done Sent false
37133 10139 Acked Flush 0 Done Done Sent false
37134 10140 Acked Write 9 Done Done Sent false
37135 10141 Acked Read 8 Done Done Sent false
37136 10142 Acked Flush 0 Done Done Sent false
37137 10143 Acked Read 3 Done Done Sent false
37138 10144 Acked Write 7 Done Done Sent false
37139 10145 Acked Read 4 Done Done Sent false
37140 10146 Acked Flush 0 Done Done Sent false
37141 10147 Acked Read 3 Done Done Sent false
37142 10148 Acked Read 8 Done Done Sent false
37143 10149 Acked Write 10 Done Done Sent false
37144 10150 Acked Write 3 Done Done Sent false
37145 10151 Acked Write 5 Done Done Sent false
37146 10152 Acked Read 3 Done Done Sent false
37147 10153 Acked Write 9 Done Done Sent false
37148 10154 Acked Flush 0 Done Done Sent false
37149 10155 Acked Read 9 Done Done Sent false
37150 10156 Acked Read 4 Done Done Sent false
37151 10157 Acked Write 9 Done Done Sent false
37152 10158 Acked Read 3 Done Done Sent false
37153 10159 Acked Read 10 Done Done Sent false
37154 10160 Acked Flush 0 Done Done Sent false
37155 10161 Acked Write 9 Done Done Sent false
37156 10162 Acked Write 2 Done Done Sent false
37157 10163 Acked Write 3 Done Done Sent false
37158 10164 Acked Read 3 Done Done Sent false
37159 10165 Acked Write 9 Done Done Sent false
37160 10166 Acked Write 2 Done Done Sent false
37161 10167 Acked Read 4 Done Done Sent false
37162 10168 Acked Read 6 Done Done Sent false
37163 10169 Acked Write 4 Done Done Sent false
37164 10170 Acked Read 4 Done Done Sent false
37165 10171 Acked Write 8 Done Done Sent false
37166 10172 Acked Read 9 Done Done Sent false
37167 10173 Acked Flush 0 Done Done Sent false
37168 10174 Acked Write 8 Done Done Sent false
37169 10175 Acked Read 5 Done Done Sent false
37170 10176 Acked Write 4 Done Done Sent false
37171 10177 Acked Read 1 Done Done Sent false
37172 10178 Acked Read 1 Done Done Sent false
37173 10179 Acked Write 1 Done Done Sent false
37174 10180 Acked Read 4 Done Done Sent false
37175 10181 Acked Write 7 Done Done Sent false
37176 10182 Acked Flush 0 Done Done Sent false
37177 10183 Acked Write 6 Done Done Sent false
37178 10184 Acked Read 1 Done Done Sent false
37179 10185 Acked Read 3 Done Done Sent false
37180 10186 Acked Read 8 Done Done Sent false
37181 10187 Acked Read 7 Done Done Sent false
37182 10188 Acked Write 5 Done Done Sent false
37183 10189 Acked Write 4 Done Done Sent false
37184 10190 Acked Write 10 Done Done Sent false
37185 10191 Acked Read 3 Done Done Sent false
37186 10192 Acked Read 6 Done Done Sent false
37187 10193 Acked Write 3 Done Done Sent false
37188 10194 Acked Read 3 Done Done Sent false
37189 10195 Acked Read 7 Done Done Sent false
37190 10196 Acked Read 3 Done Done Sent false
37191 10197 Acked Read 2 Done Done Sent false
37192 10198 Acked Write 3 Done Done Sent false
37193 10199 Acked Write 6 Done Done Sent false
37194 10200 Acked Write 2 Done Done Sent false
37195 10201 Acked Write 4 Done Done Sent false
37196 10202 Acked Read 8 Done Done Sent false
37197 10203 Acked Write 9 Done Done Sent false
37198 10204 Acked Write 6 Done Done Sent false
37199 10205 Acked Read 1 Done Done Sent false
37200 10206 Acked Write 9 Done Done Sent false
37201 10207 Acked Read 6 Done Done Sent false
37202 10208 Acked Flush 0 Done Done Sent false
37203 10209 Acked Flush 0 Done Done Sent false
37204 10210 Acked Write 4 Done Done Sent false
37205 10211 Acked Write 8 Done Done Sent false
37206 10212 Acked Write 8 Done Done Sent false
37207 10213 Acked Write 5 Done Done Sent false
37208 10214 Acked Read 1 Done Done Sent false
37209 10215 Acked Read 6 Done Done Sent false
37210 10216 Acked Write 1 Done Done Sent false
37211 10217 Acked Read 3 Done Done Sent false
37212 10218 Acked Read 10 Done Done Sent false
37213 10219 Acked Read 1 Done Done Sent false
37214 10220 Acked Read 1 Done Done Sent false
37215 10221 Acked Write 2 Done Done Sent false
37216 10222 Acked Read 1 Done Done Sent false
37217 10223 Acked Write 10 Done Done Sent false
37218 10224 Acked Flush 0 Done Done Sent false
37219 10225 Acked Write 5 Done Done Sent false
37220 10226 Acked Write 7 Done Done Sent false
37221 10227 Acked Write 2 Done Done Sent false
37222 10228 Acked Flush 0 Done Done Sent false
37223 10229 Acked Read 2 Done Done Sent false
37224 10230 Acked Flush 0 Done Done Sent false
37225 10231 Acked Read 2 Done Done Sent false
37226 10232 Acked Flush 0 Done Done Sent false
37227 10233 Acked Read 7 Done Done Sent false
37228 10234 Acked Read 9 Done Done Sent false
37229 10235 Acked Write 2 Done Done Sent false
37230 10236 Acked Read 5 Done Done Sent false
37231 10237 Acked Write 2 Done Done Sent false
37232 10238 Acked Write 4 Done Done Sent false
37233 10239 Acked Write 8 Done Done Sent false
37234 10240 Acked Write 6 Done Done Sent false
37235 10241 Acked Read 1 Done Done Sent false
37236 10242 Acked Read 7 Done Done Sent false
37237 10243 Acked Write 4 Done Done Sent false
37238 10244 Acked Read 9 Done Done Sent false
37239 10245 Acked Write 2 Done Done Sent false
37240 10246 Acked Read 3 Done Done Sent false
37241 10247 Acked Read 5 Done Done Sent false
37242 10248 Acked Read 2 Done Done Sent false
37243 10249 Acked Write 1 Done Done Sent false
37244 10250 Acked Flush 0 Done Done Sent false
37245 10251 Acked Read 4 Done Done Sent false
37246 10252 Acked Flush 0 Done Done Sent false
37247 10253 Acked Read 10 Done Done Sent false
37248 10254 Acked Flush 0 Done Done Sent false
37249 10255 Acked Write 8 Done Done Sent false
37250 10256 Acked Write 7 Done Done Sent false
37251 10257 Acked Read 1 Done Done Sent false
37252 10258 Acked Write 3 Done Done Sent false
37253 10259 Acked Write 4 Done Done Sent false
37254 10260 Acked Write 6 Done Done Sent false
37255 10261 Acked Read 2 Done Done Sent false
37256 10262 Acked Write 4 Done Done Sent false
37257 10263 Acked Write 7 Done Done Sent false
37258 10264 Acked Read 9 Done Done Sent false
37259 10265 Acked Read 8 Done Done Sent false
37260 10266 Acked Read 4 Done Done Sent false
37261 10267 Acked Write 4 Done Done Sent false
37262 10268 Acked Write 8 Done Done Sent false
37263 10269 Acked Read 10 Done Done Sent false
37264 10270 Acked Read 5 Done Done Sent false
37265 10271 Acked Read 10 Done Done Sent false
37266 10272 Acked Read 3 Done Done Sent false
37267 10273 Acked Write 6 Done Done Sent false
37268 10274 Acked Read 3 Done Done Sent false
37269 10275 Acked Write 5 Done Done Sent false
37270 10276 Acked Write 4 Done Done Sent false
37271 10277 Acked Read 9 Done Done Sent false
37272 10278 Acked Write 9 Done Done Sent false
37273 10279 Acked Write 8 Done Done Sent false
37274 10280 Acked Write 9 Done Done Sent false
37275 10281 Acked Read 5 Done Done Sent false
37276 10282 Acked Flush 0 Done Done Sent false
37277 10283 Acked Write 6 Done Done Sent false
37278 10284 Acked Write 9 Done Done Sent false
37279 10285 Acked Read 3 Done Done Sent false
37280 10286 Acked Write 1 Done Done Sent false
37281 10287 Acked Read 10 Done Done Sent false
37282 10288 Acked Read 9 Done Done Sent false
37283 10289 Acked Write 6 Done Done Sent false
37284 10290 Acked Read 10 Done Done Sent false
37285 10291 Acked Write 10 Done Done Sent false
37286 10292 Acked Read 10 Done Done Sent false
37287 10293 Acked Write 10 Done Done Sent false
37288 10294 Acked Read 3 Done Done Sent false
37289 10295 Acked Read 6 Done Done Sent false
37290 10296 Acked Write 2 Done Done Sent false
37291 10297 Acked Read 4 Done Done Sent false
37292 10298 Acked Write 6 Done Done Sent false
37293 10299 Acked Read 2 Done Done Sent false
37294 10300 Acked Read 5 Done Done Sent false
37295 10301 Acked Read 10 Done Done Sent false
37296 10302 Acked Write 10 Done Done Sent false
37297 10303 Acked Write 9 Done Done Sent false
37298 10304 Acked Read 9 Done Done Sent false
37299 10305 Acked Write 3 Done Done Sent false
37300 10306 Acked Read 10 Done Done Sent false
37301 10307 Acked Read 6 Done Done Sent false
37302 10308 Acked Read 1 Done Done Sent false
37303 10309 Acked Read 7 Done Done Sent false
37304 10310 Acked Read 10 Done Done Sent false
37305 10311 Acked Write 3 Done Done Sent false
37306 10312 Acked Read 8 Done Done Sent false
37307 10313 Acked Write 3 Done Done Sent false
37308 10314 Acked Read 9 Done Done Sent false
37309 10315 Acked Write 3 Done Done Sent false
37310 10316 Acked Write 2 Done Done Sent false
37311 10317 Acked Write 2 Done Done Sent false
37312 10318 Acked Write 3 Done Done Sent false
37313 10319 Acked Flush 0 Done Done Sent false
37314 10320 Acked Write 2 Done Done Sent false
37315 10321 Acked Read 3 Done Done Sent false
37316 10322 Acked Read 9 Done Done Sent false
37317 10323 Acked Read 5 Done Done Sent false
37318 10324 Acked Read 6 Done Done Sent false
37319 10325 Acked Read 10 Done Done Sent false
37320 10326 Acked Read 8 Done Done Sent false
37321 10327 Acked Read 1 Done Done Sent false
37322 10328 Acked Read 4 Done Done Sent false
37323 10329 Acked Write 4 Done Done Sent false
37324 10330 Acked Read 1 Done Done Sent false
37325 10331 Acked Read 7 Done Done Sent false
37326 10332 Acked Write 2 Done Done Sent false
37327 10333 Acked Read 9 Done Done Sent false
37328 10334 Acked Write 3 Done Done Sent false
37329 10335 Acked Write 3 Done Done Sent false
37330 10336 Acked Read 9 Done Done Sent false
37331 10337 Acked Write 2 Done Done Sent false
37332 10338 Acked Write 9 Done Done Sent false
37333 10339 Acked Write 9 Done Done Sent false
37334 10340 Acked Read 3 Done Done Sent false
37335 10341 Acked Write 3 Done Done Sent false
37336 10342 Acked Write 3 Done Done Sent false
37337 10343 Acked Read 7 Done Done Sent false
37338 10344 Acked Write 9 Done Done Sent false
37339 10345 Acked Read 3 Done Done Sent false
37340 10346 Acked Write 7 Done Done Sent false
37341 10347 Acked Write 6 Done Done Sent false
37342 10348 Acked Write 7 Done Done Sent false
37343 10349 Acked Write 8 Done Done Sent false
37344 10350 Acked Read 1 Done Done Sent false
37345 10351 Acked Read 5 Done Done Sent false
37346 10352 Acked Read 5 Done Done Sent false
37347 10353 Acked Flush 0 Done Done Sent false
37348 10354 Acked Read 10 Done Done Sent false
37349 10355 Acked Read 5 Done Done Sent false
37350 10356 Acked Write 4 Done Done Sent false
37351 10357 Acked Read 9 Done Done Sent false
37352 10358 Acked Read 7 Done Done Sent false
37353 10359 Acked Read 8 Done Done Sent false
37354 STATES DS:0 DS:1 DS:2 TOTAL
37355 Sent 0 0 301 301
37356 Done 301 301 0 602
37357 Skipped 0 0 0 0
37358 Error 0 0 0 0
37359 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10058))
37360 Downstairs last five completed: 10058 10057 10056 10055 10054
37361 Upstairs last five completed: 10359 10358 10357 10356 10355
37362 CLIENT: Up:0 ds:301 act:2
373632025-01-29T05:19:04.754ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
373642025-01-29T05:19:05.795ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
373652025-01-29T05:19:06.758ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37366 ----------------------------------------------------------------
37367 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37368 States: Active Active Offline
37369 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37370 10059 Acked Write 3 Done Done Sent false
37371 10060 Acked Write 5 Done Done Sent false
37372 10061 Acked Read 5 Done Done Sent false
37373 10062 Acked Write 9 Done Done Sent false
37374 10063 Acked Write 2 Done Done Sent false
37375 10064 Acked Read 6 Done Done Sent false
37376 10065 Acked Read 4 Done Done Sent false
37377 10066 Acked Write 3 Done Done Sent false
37378 10067 Acked Read 9 Done Done Sent false
37379 10068 Acked Read 2 Done Done Sent false
37380 10069 Acked Write 9 Done Done Sent false
37381 10070 Acked Write 8 Done Done Sent false
37382 10071 Acked Read 9 Done Done Sent false
37383 10072 Acked Read 9 Done Done Sent false
37384 10073 Acked Flush 0 Done Done Sent false
37385 10074 Acked Write 9 Done Done Sent false
37386 10075 Acked Flush 0 Done Done Sent false
37387 10076 Acked Read 9 Done Done Sent false
37388 10077 Acked Flush 0 Done Done Sent false
37389 10078 Acked Read 5 Done Done Sent false
37390 10079 Acked Write 1 Done Done Sent false
37391 10080 Acked Write 7 Done Done Sent false
37392 10081 Acked Read 10 Done Done Sent false
37393 10082 Acked Read 3 Done Done Sent false
37394 10083 Acked Read 7 Done Done Sent false
37395 10084 Acked Read 1 Done Done Sent false
37396 10085 Acked Flush 0 Done Done Sent false
37397 10086 Acked Write 9 Done Done Sent false
37398 10087 Acked Write 7 Done Done Sent false
37399 10088 Acked Write 7 Done Done Sent false
37400 10089 Acked Read 3 Done Done Sent false
37401 10090 Acked Write 7 Done Done Sent false
37402 10091 Acked Read 9 Done Done Sent false
37403 10092 Acked Read 5 Done Done Sent false
37404 10093 Acked Read 7 Done Done Sent false
37405 10094 Acked Read 1 Done Done Sent false
37406 10095 Acked Write 7 Done Done Sent false
37407 10096 Acked Read 6 Done Done Sent false
37408 10097 Acked Flush 0 Done Done Sent false
37409 10098 Acked Read 3 Done Done Sent false
37410 10099 Acked Write 10 Done Done Sent false
37411 10100 Acked Flush 0 Done Done Sent false
37412 10101 Acked Read 2 Done Done Sent false
37413 10102 Acked Read 2 Done Done Sent false
37414 10103 Acked Read 8 Done Done Sent false
37415 10104 Acked Write 4 Done Done Sent false
37416 10105 Acked Read 10 Done Done Sent false
37417 10106 Acked Write 1 Done Done Sent false
37418 10107 Acked Read 6 Done Done Sent false
37419 10108 Acked Read 9 Done Done Sent false
37420 10109 Acked Read 7 Done Done Sent false
37421 10110 Acked Write 10 Done Done Sent false
37422 10111 Acked Write 9 Done Done Sent false
37423 10112 Acked Read 10 Done Done Sent false
37424 10113 Acked Read 1 Done Done Sent false
37425 10114 Acked Read 10 Done Done Sent false
37426 10115 Acked Write 2 Done Done Sent false
37427 10116 Acked Read 3 Done Done Sent false
37428 10117 Acked Flush 0 Done Done Sent false
37429 10118 Acked Write 1 Done Done Sent false
37430 10119 Acked Flush 0 Done Done Sent false
37431 10120 Acked Read 9 Done Done Sent false
37432 10121 Acked Read 5 Done Done Sent false
37433 10122 Acked Write 5 Done Done Sent false
37434 10123 Acked Write 6 Done Done Sent false
37435 10124 Acked Read 2 Done Done Sent false
37436 10125 Acked Write 7 Done Done Sent false
37437 10126 Acked Write 10 Done Done Sent false
37438 10127 Acked Flush 0 Done Done Sent false
37439 10128 Acked Write 4 Done Done Sent false
37440 10129 Acked Read 1 Done Done Sent false
37441 10130 Acked Write 9 Done Done Sent false
37442 10131 Acked Flush 0 Done Done Sent false
37443 10132 Acked Write 10 Done Done Sent false
37444 10133 Acked Write 10 Done Done Sent false
37445 10134 Acked Read 8 Done Done Sent false
37446 10135 Acked Read 8 Done Done Sent false
37447 10136 Acked Read 8 Done Done Sent false
37448 10137 Acked Read 7 Done Done Sent false
37449 10138 Acked Read 10 Done Done Sent false
37450 10139 Acked Flush 0 Done Done Sent false
37451 10140 Acked Write 9 Done Done Sent false
37452 10141 Acked Read 8 Done Done Sent false
37453 10142 Acked Flush 0 Done Done Sent false
37454 10143 Acked Read 3 Done Done Sent false
37455 10144 Acked Write 7 Done Done Sent false
37456 10145 Acked Read 4 Done Done Sent false
37457 10146 Acked Flush 0 Done Done Sent false
37458 10147 Acked Read 3 Done Done Sent false
37459 10148 Acked Read 8 Done Done Sent false
37460 10149 Acked Write 10 Done Done Sent false
37461 10150 Acked Write 3 Done Done Sent false
37462 10151 Acked Write 5 Done Done Sent false
37463 10152 Acked Read 3 Done Done Sent false
37464 10153 Acked Write 9 Done Done Sent false
37465 10154 Acked Flush 0 Done Done Sent false
37466 10155 Acked Read 9 Done Done Sent false
37467 10156 Acked Read 4 Done Done Sent false
37468 10157 Acked Write 9 Done Done Sent false
37469 10158 Acked Read 3 Done Done Sent false
37470 10159 Acked Read 10 Done Done Sent false
37471 10160 Acked Flush 0 Done Done Sent false
37472 10161 Acked Write 9 Done Done Sent false
37473 10162 Acked Write 2 Done Done Sent false
37474 10163 Acked Write 3 Done Done Sent false
37475 10164 Acked Read 3 Done Done Sent false
37476 10165 Acked Write 9 Done Done Sent false
37477 10166 Acked Write 2 Done Done Sent false
37478 10167 Acked Read 4 Done Done Sent false
37479 10168 Acked Read 6 Done Done Sent false
37480 10169 Acked Write 4 Done Done Sent false
37481 10170 Acked Read 4 Done Done Sent false
37482 10171 Acked Write 8 Done Done Sent false
37483 10172 Acked Read 9 Done Done Sent false
37484 10173 Acked Flush 0 Done Done Sent false
37485 10174 Acked Write 8 Done Done Sent false
37486 10175 Acked Read 5 Done Done Sent false
37487 10176 Acked Write 4 Done Done Sent false
37488 10177 Acked Read 1 Done Done Sent false
37489 10178 Acked Read 1 Done Done Sent false
37490 10179 Acked Write 1 Done Done Sent false
37491 10180 Acked Read 4 Done Done Sent false
37492 10181 Acked Write 7 Done Done Sent false
37493 10182 Acked Flush 0 Done Done Sent false
37494 10183 Acked Write 6 Done Done Sent false
37495 10184 Acked Read 1 Done Done Sent false
37496 10185 Acked Read 3 Done Done Sent false
37497 10186 Acked Read 8 Done Done Sent false
37498 10187 Acked Read 7 Done Done Sent false
37499 10188 Acked Write 5 Done Done Sent false
37500 10189 Acked Write 4 Done Done Sent false
37501 10190 Acked Write 10 Done Done Sent false
37502 10191 Acked Read 3 Done Done Sent false
37503 10192 Acked Read 6 Done Done Sent false
37504 10193 Acked Write 3 Done Done Sent false
37505 10194 Acked Read 3 Done Done Sent false
37506 10195 Acked Read 7 Done Done Sent false
37507 10196 Acked Read 3 Done Done Sent false
37508 10197 Acked Read 2 Done Done Sent false
37509 10198 Acked Write 3 Done Done Sent false
37510 10199 Acked Write 6 Done Done Sent false
37511 10200 Acked Write 2 Done Done Sent false
37512 10201 Acked Write 4 Done Done Sent false
37513 10202 Acked Read 8 Done Done Sent false
37514 10203 Acked Write 9 Done Done Sent false
37515 10204 Acked Write 6 Done Done Sent false
37516 10205 Acked Read 1 Done Done Sent false
37517 10206 Acked Write 9 Done Done Sent false
37518 10207 Acked Read 6 Done Done Sent false
37519 10208 Acked Flush 0 Done Done Sent false
37520 10209 Acked Flush 0 Done Done Sent false
37521 10210 Acked Write 4 Done Done Sent false
37522 10211 Acked Write 8 Done Done Sent false
37523 10212 Acked Write 8 Done Done Sent false
37524 10213 Acked Write 5 Done Done Sent false
37525 10214 Acked Read 1 Done Done Sent false
37526 10215 Acked Read 6 Done Done Sent false
37527 10216 Acked Write 1 Done Done Sent false
37528 10217 Acked Read 3 Done Done Sent false
37529 10218 Acked Read 10 Done Done Sent false
37530 10219 Acked Read 1 Done Done Sent false
37531 10220 Acked Read 1 Done Done Sent false
37532 10221 Acked Write 2 Done Done Sent false
37533 10222 Acked Read 1 Done Done Sent false
37534 10223 Acked Write 10 Done Done Sent false
37535 10224 Acked Flush 0 Done Done Sent false
37536 10225 Acked Write 5 Done Done Sent false
37537 10226 Acked Write 7 Done Done Sent false
37538 10227 Acked Write 2 Done Done Sent false
37539 10228 Acked Flush 0 Done Done Sent false
37540 10229 Acked Read 2 Done Done Sent false
37541 10230 Acked Flush 0 Done Done Sent false
37542 10231 Acked Read 2 Done Done Sent false
37543 10232 Acked Flush 0 Done Done Sent false
37544 10233 Acked Read 7 Done Done Sent false
37545 10234 Acked Read 9 Done Done Sent false
37546 10235 Acked Write 2 Done Done Sent false
37547 10236 Acked Read 5 Done Done Sent false
37548 10237 Acked Write 2 Done Done Sent false
37549 10238 Acked Write 4 Done Done Sent false
37550 10239 Acked Write 8 Done Done Sent false
37551 10240 Acked Write 6 Done Done Sent false
37552 10241 Acked Read 1 Done Done Sent false
37553 10242 Acked Read 7 Done Done Sent false
37554 10243 Acked Write 4 Done Done Sent false
37555 10244 Acked Read 9 Done Done Sent false
37556 10245 Acked Write 2 Done Done Sent false
37557 10246 Acked Read 3 Done Done Sent false
37558 10247 Acked Read 5 Done Done Sent false
37559 10248 Acked Read 2 Done Done Sent false
37560 10249 Acked Write 1 Done Done Sent false
37561 10250 Acked Flush 0 Done Done Sent false
37562 10251 Acked Read 4 Done Done Sent false
37563 10252 Acked Flush 0 Done Done Sent false
37564 10253 Acked Read 10 Done Done Sent false
37565 10254 Acked Flush 0 Done Done Sent false
37566 10255 Acked Write 8 Done Done Sent false
37567 10256 Acked Write 7 Done Done Sent false
37568 10257 Acked Read 1 Done Done Sent false
37569 10258 Acked Write 3 Done Done Sent false
37570 10259 Acked Write 4 Done Done Sent false
37571 10260 Acked Write 6 Done Done Sent false
37572 10261 Acked Read 2 Done Done Sent false
37573 10262 Acked Write 4 Done Done Sent false
37574 10263 Acked Write 7 Done Done Sent false
37575 10264 Acked Read 9 Done Done Sent false
37576 10265 Acked Read 8 Done Done Sent false
37577 10266 Acked Read 4 Done Done Sent false
37578 10267 Acked Write 4 Done Done Sent false
37579 10268 Acked Write 8 Done Done Sent false
37580 10269 Acked Read 10 Done Done Sent false
37581 10270 Acked Read 5 Done Done Sent false
37582 10271 Acked Read 10 Done Done Sent false
37583 10272 Acked Read 3 Done Done Sent false
37584 10273 Acked Write 6 Done Done Sent false
37585 10274 Acked Read 3 Done Done Sent false
37586 10275 Acked Write 5 Done Done Sent false
37587 10276 Acked Write 4 Done Done Sent false
37588 10277 Acked Read 9 Done Done Sent false
37589 10278 Acked Write 9 Done Done Sent false
37590 10279 Acked Write 8 Done Done Sent false
37591 10280 Acked Write 9 Done Done Sent false
37592 10281 Acked Read 5 Done Done Sent false
37593 10282 Acked Flush 0 Done Done Sent false
37594 10283 Acked Write 6 Done Done Sent false
37595 10284 Acked Write 9 Done Done Sent false
37596 10285 Acked Read 3 Done Done Sent false
37597 10286 Acked Write 1 Done Done Sent false
37598 10287 Acked Read 10 Done Done Sent false
37599 10288 Acked Read 9 Done Done Sent false
37600 10289 Acked Write 6 Done Done Sent false
37601 10290 Acked Read 10 Done Done Sent false
37602 10291 Acked Write 10 Done Done Sent false
37603 10292 Acked Read 10 Done Done Sent false
37604 10293 Acked Write 10 Done Done Sent false
37605 10294 Acked Read 3 Done Done Sent false
37606 10295 Acked Read 6 Done Done Sent false
37607 10296 Acked Write 2 Done Done Sent false
37608 10297 Acked Read 4 Done Done Sent false
37609 10298 Acked Write 6 Done Done Sent false
37610 10299 Acked Read 2 Done Done Sent false
37611 10300 Acked Read 5 Done Done Sent false
37612 10301 Acked Read 10 Done Done Sent false
37613 10302 Acked Write 10 Done Done Sent false
37614 10303 Acked Write 9 Done Done Sent false
37615 10304 Acked Read 9 Done Done Sent false
37616 10305 Acked Write 3 Done Done Sent false
37617 10306 Acked Read 10 Done Done Sent false
37618 10307 Acked Read 6 Done Done Sent false
37619 10308 Acked Read 1 Done Done Sent false
37620 10309 Acked Read 7 Done Done Sent false
37621 10310 Acked Read 10 Done Done Sent false
37622 10311 Acked Write 3 Done Done Sent false
37623 10312 Acked Read 8 Done Done Sent false
37624 10313 Acked Write 3 Done Done Sent false
37625 10314 Acked Read 9 Done Done Sent false
37626 10315 Acked Write 3 Done Done Sent false
37627 10316 Acked Write 2 Done Done Sent false
37628 10317 Acked Write 2 Done Done Sent false
37629 10318 Acked Write 3 Done Done Sent false
37630 10319 Acked Flush 0 Done Done Sent false
37631 10320 Acked Write 2 Done Done Sent false
37632 10321 Acked Read 3 Done Done Sent false
37633 10322 Acked Read 9 Done Done Sent false
37634 10323 Acked Read 5 Done Done Sent false
37635 10324 Acked Read 6 Done Done Sent false
37636 10325 Acked Read 10 Done Done Sent false
37637 10326 Acked Read 8 Done Done Sent false
37638 10327 Acked Read 1 Done Done Sent false
37639 10328 Acked Read 4 Done Done Sent false
37640 10329 Acked Write 4 Done Done Sent false
37641 10330 Acked Read 1 Done Done Sent false
37642 10331 Acked Read 7 Done Done Sent false
37643 10332 Acked Write 2 Done Done Sent false
37644 10333 Acked Read 9 Done Done Sent false
37645 10334 Acked Write 3 Done Done Sent false
37646 10335 Acked Write 3 Done Done Sent false
37647 10336 Acked Read 9 Done Done Sent false
37648 10337 Acked Write 2 Done Done Sent false
37649 10338 Acked Write 9 Done Done Sent false
37650 10339 Acked Write 9 Done Done Sent false
37651 10340 Acked Read 3 Done Done Sent false
37652 10341 Acked Write 3 Done Done Sent false
37653 10342 Acked Write 3 Done Done Sent false
37654 10343 Acked Read 7 Done Done Sent false
37655 10344 Acked Write 9 Done Done Sent false
37656 10345 Acked Read 3 Done Done Sent false
37657 10346 Acked Write 7 Done Done Sent false
37658 10347 Acked Write 6 Done Done Sent false
37659 10348 Acked Write 7 Done Done Sent false
37660 10349 Acked Write 8 Done Done Sent false
37661 10350 Acked Read 1 Done Done Sent false
37662 10351 Acked Read 5 Done Done Sent false
37663 10352 Acked Read 5 Done Done Sent false
37664 10353 Acked Flush 0 Done Done Sent false
37665 10354 Acked Read 10 Done Done Sent false
37666 10355 Acked Read 5 Done Done Sent false
37667 10356 Acked Write 4 Done Done Sent false
37668 10357 Acked Read 9 Done Done Sent false
37669 10358 Acked Read 7 Done Done Sent false
37670 10359 Acked Read 8 Done Done Sent false
37671 STATES DS:0 DS:1 DS:2 TOTAL
37672 Sent 0 0 301 301
37673 Done 301 301 0 602
37674 Skipped 0 0 0 0
37675 Error 0 0 0 0
37676 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10058))
37677 Downstairs last five completed: 10058 10057 10056 10055 10054
37678 Upstairs last five completed: 10359 10358 10357 10356 10355
37679 CLIENT: Up:0 ds:301 act:2
376802025-01-29T05:19:07.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376812025-01-29T05:19:08.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376822025-01-29T05:19:09.764ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376832025-01-29T05:19:10.795ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376842025-01-29T05:19:11.061ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376852025-01-29T05:19:11.061ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33989) = io task client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376862025-01-29T05:19:11.062ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376872025-01-29T05:19:11.062ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376882025-01-29T05:19:11.062ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376892025-01-29T05:19:11.062ZINFOcrucible: send last flush ID to this DS: Some(JobId(10058)) = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376902025-01-29T05:19:11.062ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376912025-01-29T05:19:11.062ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(10058)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
376922025-01-29T05:19:11.063ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(10058)) = downstairs session_id = 3b9f53a2-1179-49c9-b40f-5354d75015be
37693 ----------------------------------------------------------------
37694 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:6
37695 States: Active Active Active
37696 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37697 10354 Acked Read 10 Done Done Done true
37698 10355 Acked Read 5 Done Done Done true
37699 10356 Acked Write 4 Done Done Done true
37700 10357 Acked Read 9 Done Done Done true
37701 10358 Acked Read 7 Done Done Done true
37702 10359 Acked Read 8 Done Done Done true
37703 STATES DS:0 DS:1 DS:2 TOTAL
37704 Sent 0 0 0 0
37705 Done 6 6 6 18
37706 Skipped 0 0 0 0
37707 Error 0 0 0 0
37708 Last Flush: Some(JobId(10353)) Some(JobId(10353)) Some(JobId(10353))
37709 Downstairs last five completed: 10353 10352 10351 10350 10349
37710 Upstairs last five completed: 10359 10358 10357 10356 10355
37711 CLIENT: Up:0 ds:6 act:3
37712 ----------------------------------------------------------------
37713 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37714 Downstairs last five completed: 10360 10359 10358 10357 10356
37715 Upstairs last five completed: 10360 10359 10358 10357 10356
37716 CLIENT: Up:0 ds:0 act:3
37717 Replay: All jobs finished
37718 Test replay has completed
37719 Wrote out file "/tmp/test_replay_verify.log"
37720 CLIENT: Tests done. All submitted work has been ACK'd
37721 ----------------------------------------------------------------
37722 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37723 Downstairs last five completed: 10360 10359 10358 10357 10356
37724 Upstairs last five completed: 10360 10359 10358 10357 10356
37725 CLIENT: Up:0 ds:0 act:3
37726 CLIENT: All jobs finished, all DS active.
37727 Replays:30 time: 8:32
37728 Do final verify
37729 Jan 29 05:19:15.392 INFO accepted connection, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
37730 Jan 29 05:19:15.393 INFO request completed, latency_us: 186, response_code: 200, uri: /regioninfo, method: GET, req_id: 00f45c22-06bd-47cf-ae8e-50733e4f5cbf, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
377312025-01-29T05:19:15.394ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
37732 Jan 29 05:19:15.394 INFO request completed, latency_us: 87, response_code: 200, uri: /regioncount, method: GET, req_id: b3967817-34bb-40c4-b209-c2c7a78f94b6, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
377332025-01-29T05:19:15.394ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
37734 Jan 29 05:19:15.394 INFO request completed, latency_us: 108, response_code: 200, uri: /port/cid/0, method: GET, req_id: 701818c0-9d3e-487c-9677-7a32543ecf29, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
37735 Jan 29 05:19:15.395 INFO request completed, latency_us: 93, response_code: 200, uri: /port/cid/1, method: GET, req_id: ab8a6a8e-7794-4dfb-af1b-90723640742b, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
37736 Jan 29 05:19:15.396 INFO request completed, latency_us: 100, response_code: 200, uri: /port/cid/2, method: GET, req_id: 3567da8c-299d-499a-b8b3-263114ec9359, remote_addr: 127.0.0.1:40682, local_addr: 127.0.0.1:9998
377372025-01-29T05:19:15.396ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
377382025-01-29T05:19:15.396ZINFOcrucible: Upstairs starts
377392025-01-29T05:19:15.396ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "442be2395eb3bf8ebe11d34acf0678a1aae4fa71", git_commit_timestamp: "2025-01-29T04:48:13.000000000Z", git_branch: "main", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
377402025-01-29T05:19:15.396ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
377412025-01-29T05:19:15.396ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
377422025-01-29T05:19:15.396ZINFOcrucible: Crucible ff40d0e9-ce6c-40bb-8585-986ee42c0687 has session id: 3470096f-08e1-489f-9f4f-32a5f31195ab session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377432025-01-29T05:19:15.397ZINFOcrucible: Upstairs opts: Upstairs UUID: ff40d0e9-ce6c-40bb-8585-986ee42c0687, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377442025-01-29T05:19:15.397ZINFOcrucible: Crucible stats registered with UUID: ff40d0e9-ce6c-40bb-8585-986ee42c0687 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377452025-01-29T05:19:15.397ZWARNcrucible: could not find Downstairs address for Nexus session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377462025-01-29T05:19:15.397ZINFOcrucible: Downstairs in volume = 3 task = crutest
377472025-01-29T05:19:15.397ZINFOcrucible: The guest has requested activation with gen:3
377482025-01-29T05:19:15.397ZINFOcrucible: ff40d0e9-ce6c-40bb-8585-986ee42c0687 active request set session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377492025-01-29T05:19:15.397ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377502025-01-29T05:19:15.398ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377512025-01-29T05:19:15.398ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377522025-01-29T05:19:15.398ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377532025-01-29T05:19:15.398ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377542025-01-29T05:19:15.398ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377552025-01-29T05:19:15.398ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377562025-01-29T05:19:15.398ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377572025-01-29T05:19:15.398ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377582025-01-29T05:19:15.398ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44181) = io task client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377592025-01-29T05:19:15.398ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50667) = io task client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377602025-01-29T05:19:15.398ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41027) = io task client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377612025-01-29T05:19:15.399ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377622025-01-29T05:19:15.399ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377632025-01-29T05:19:15.399ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377642025-01-29T05:19:15.399ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377652025-01-29T05:19:15.399ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377662025-01-29T05:19:15.399ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377672025-01-29T05:19:15.400ZINFOcrucible: Waiting for more clients to be ready session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377682025-01-29T05:19:15.400ZINFOcrucible: Waiting for more clients to be ready session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377692025-01-29T05:19:15.400ZINFOcrucible: [0]R flush_numbers[0..12]: [966, 966, 965, 965, 964, 944, 935, 965, 937, 964, 955, 964] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377702025-01-29T05:19:15.400ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377712025-01-29T05:19:15.400ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377722025-01-29T05:19:15.400ZINFOcrucible: [1]R flush_numbers[0..12]: [966, 966, 965, 965, 964, 944, 935, 965, 937, 964, 955, 964] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377732025-01-29T05:19:15.400ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377742025-01-29T05:19:15.400ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377752025-01-29T05:19:15.400ZINFOcrucible: [2]R flush_numbers[0..12]: [966, 966, 965, 965, 964, 944, 935, 965, 937, 964, 955, 964] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
37776 {"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30Wait for a query_work_queue command to finish before sending IO
37777 ,"time":"2025-01-29T05:19:15.400763486Z","hostname":"w-01JJR7DDKGFJDPMCDR9NFSE52V","pid":1238,"":"downstairs","":"downstairs","session_id":"3470096f-08e1-489f-9f4f-32a5f31195ab"}
377782025-01-29T05:19:15.400ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377792025-01-29T05:19:15.400ZINFOcrucible: Max found gen is 3 = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377802025-01-29T05:19:15.400ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377812025-01-29T05:19:15.400ZINFOcrucible: Next flush: 968 = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377822025-01-29T05:19:15.400ZINFOcrucible: All extents match = downstairs session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377832025-01-29T05:19:15.400ZINFOcrucible: No downstairs reconciliation required session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377842025-01-29T05:19:15.400ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377852025-01-29T05:19:15.400ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377862025-01-29T05:19:15.400ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377872025-01-29T05:19:15.400ZINFOcrucible: All required reconciliation work is completed session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377882025-01-29T05:19:15.400ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377892025-01-29T05:19:15.400ZINFOcrucible: ff40d0e9-ce6c-40bb-8585-986ee42c0687 is now active with session: 3470096f-08e1-489f-9f4f-32a5f31195ab session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377902025-01-29T05:19:15.400ZINFOcrucible: Set Active after no reconciliation session_id = 3470096f-08e1-489f-9f4f-32a5f31195ab
377912025-01-29T05:19:15.400ZINFOcrucible: The guest has finished waiting for activation with:3
37792 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
37793 Loading write count information from file "/tmp/test_replay_verify.log"
37794 Read and Verify all blocks (0..5000 range:false)
37795 Wrote out file "/tmp/test_replay_verify.log"
37796 Verify test completed
37797 Wrote out file "/tmp/test_replay_verify.log"
37798 CLIENT: Tests done. All submitted work has been ACK'd
37799 ----------------------------------------------------------------
37800 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
37801 States: Active Active Active
37802 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37803 1000 Acked Read 256 Done Done Done false
37804 1001 Acked Read 256 Done Done Done false
37805 1002 Acked Read 256 Done Done Done false
37806 1003 Acked Read 256 Done Done Done false
37807 1004 Acked Read 256 Done Done Done false
37808 1005 Acked Read 256 Done Done Done false
37809 1006 Acked Read 256 Done Done Done false
37810 1007 Acked Read 256 Done Done Done false
37811 1008 Acked Read 256 Done Done Done false
37812 1009 Acked Read 256 Done Done Done false
37813 1010 Acked Read 256 Done Done Done false
37814 1011 Acked Read 256 Done Done Done false
37815 1012 Acked Read 256 Done Done Done false
37816 1013 Acked Read 256 Done Done Done false
37817 1014 Acked Read 256 Done Done Done false
37818 1015 Acked Read 256 Done Done Done false
37819 1016 Acked Read 256 Done Done Done false
37820 1017 Acked Read 256 Done Done Done false
37821 1018 Acked Read 136 Done Done Done false
37822 1019 Acked Read 256 Done Done Done false
37823 STATES DS:0 DS:1 DS:2 TOTAL
37824 Sent 0 0 0 0
37825 Done 20 20 20 60
37826 Skipped 0 0 0 0
37827 Error 0 0 0 0
37828 Last Flush: None None None
37829 Downstairs last five completed:
37830 Upstairs last five completed: 1019 1018 1017 1016 1015
37831 CLIENT: Up:0 ds:20 act:3
37832 ----------------------------------------------------------------
37833 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
37834 Downstairs last five completed: 1020 1019 1018 1017 1016
37835 Upstairs last five completed: 1020 1019 1018 1017 1016
37836 CLIENT: Up:0 ds:0 act:3
37837 CLIENT: All crucible jobs finished, exiting program
37838 Jan 29 05:19:19.875 INFO accepted connection, remote_addr: 127.0.0.1:37255, local_addr: 127.0.0.1:9998
37839 Main task has work to do, go find it
37840 got dsc Shutdown
37841 Shutdown
37842 Jan 29 05:19:19.876 INFO request completed, latency_us: 170, response_code: 204, uri: /shutdownShut it down
37843 , method: POST, req_id: 952e747b-e805-4a86-9774-7a4b04e2c226, remote_addr: 127.[8810] Disable keep_running
37844 0.0.1:37255, local_addr: 127.0.0.1:9998
37845 [8810] Got stop action so:false kr:false
37846 [8820] Disable keep_running
37847 [8820] Got stop action so:false kr:false
37848 [8830] Disable keep_running
37849 [8830] Got stop action so:false kr:false
37850 [8820] Exited with: ExitStatus(unix_wait_status(9))
37851 [8820][1] reports Exit
37852 [8830] Exited with: ExitStatus(unix_wait_status(9))
37853 [8830][2] reports Exit
37854 [8810] Exited with: ExitStatus(unix_wait_status(9))
37855 [8810][0] reports Exit
37856 January 29, 2025 at 05:19:24 AM UTC Test ends with 0