Max level shown:
1 starting March 7, 2025 at 06:12:26 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.26569542
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.021976031
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.019918755
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8830
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Starting downstairs at port 8820
20 Make output file at "/tmp/dsc/downstairs-8820.txt"
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8830][2] initial start wait reports Starting
24 [8820][1] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1198)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1199)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1200)
30 [8830][2] initial start wait reports Running
31 All downstairs are running
32 Mar 07 18:12:26.688 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Mar 07 18:12:31.788 INFO accepted connection, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
36 Mar 07 18:12:31.791 INFO request completed, latency_us: 445, response_code: 200, uri: /regioninfo, method: GET, req_id: 40a5d599-4fa7-496b-92cc-ae2974e59d47, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
37 Mar 07 18:12:31.794 INFO request completed, latency_us: 126, response_code: 200, uri: /regioncount, method: GET, req_id: ef67dffe-21de-4584-a367-3f9da21751ea, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
382025-03-07T18:12:31.794ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-03-07T18:12:31.795ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Mar 07 18:12:31.795 INFO request completed, latency_us: 165, response_code: 200, uri: /port/cid/0, method: GET, req_id: 09a33555-ce4e-42b3-a150-fc87ec249e53, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
41 Mar 07 18:12:31.796 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/1, method: GET, req_id: 05d10c93-128e-4c9c-9308-97571f256011, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
42 Mar 07 18:12:31.796 INFO request completed, latency_us: 86, response_code: 200, uri: /port/cid/2, method: GET, req_id: 2775e2df-cfe7-4913-928c-eb7ef196eb12, remote_addr: 127.0.0.1:60304, local_addr: 127.0.0.1:9998
432025-03-07T18:12:31.796ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-03-07T18:12:31.798ZINFOcrucible: Upstairs starts
452025-03-07T18:12:31.798ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "6c903058314cb23845d8ec9d1b70e497f207c60e", git_commit_timestamp: "2025-03-07T17:56:28.000000000Z", git_branch: "renovate/dropshot-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-03-07T18:12:31.798ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-03-07T18:12:31.798ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-03-07T18:12:31.799ZINFOcrucible: Crucible 57e5a875-f7b1-4735-a6d1-5be2ee30a948 has session id: 76942cb8-80db-41e5-abf0-a1b3f4c60fa1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
492025-03-07T18:12:31.799ZINFOcrucible: Upstairs opts: Upstairs UUID: 57e5a875-f7b1-4735-a6d1-5be2ee30a948, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
502025-03-07T18:12:31.799ZINFOcrucible: Crucible stats registered with UUID: 57e5a875-f7b1-4735-a6d1-5be2ee30a948 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
512025-03-07T18:12:31.799ZWARNcrucible: could not find Downstairs address for Nexus session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
522025-03-07T18:12:31.801ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-03-07T18:12:31.801ZINFOcrucible: The guest has requested activation with gen:1
542025-03-07T18:12:31.801ZINFOcrucible: 57e5a875-f7b1-4735-a6d1-5be2ee30a948 active request set session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
552025-03-07T18:12:31.801ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
562025-03-07T18:12:31.802ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
572025-03-07T18:12:31.802ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
582025-03-07T18:12:31.802ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
592025-03-07T18:12:31.802ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
602025-03-07T18:12:31.802ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
612025-03-07T18:12:31.802ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
622025-03-07T18:12:31.802ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
632025-03-07T18:12:31.802ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
642025-03-07T18:12:31.802ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52672) = io task client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
652025-03-07T18:12:31.802ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39105) = io task client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
662025-03-07T18:12:31.802ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53466) = io task client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
672025-03-07T18:12:31.804ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
682025-03-07T18:12:31.804ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
692025-03-07T18:12:31.804ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
702025-03-07T18:12:31.805ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
712025-03-07T18:12:31.805ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
722025-03-07T18:12:31.805ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
732025-03-07T18:12:31.807ZINFOcrucible: Waiting for more clients to be ready session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
742025-03-07T18:12:31.807ZINFOcrucible: Waiting for more clients to be ready session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
752025-03-07T18:12:31.807ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
762025-03-07T18:12:31.807ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
772025-03-07T18:12:31.807ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
782025-03-07T18:12:31.807ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
792025-03-07T18:12:31.807ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
802025-03-07T18:12:31.807ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
812025-03-07T18:12:31.807ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
822025-03-07T18:12:31.807ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
832025-03-07T18:12:31.807ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
842025-03-07T18:12:31.807ZINFOcrucible: Max found gen is 1 = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
852025-03-07T18:12:31.807ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
862025-03-07T18:12:31.807ZINFOcrucible: Next flush: 1 = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
872025-03-07T18:12:31.807ZINFOcrucible: All extents match = downstairs session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
882025-03-07T18:12:31.807ZINFOcrucible: No downstairs reconciliation required session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
892025-03-07T18:12:31.807ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
902025-03-07T18:12:31.807ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
912025-03-07T18:12:31.807ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
922025-03-07T18:12:31.807ZINFOcrucible: All required reconciliation work is completed session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
932025-03-07T18:12:31.807ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
942025-03-07T18:12:31.807ZINFOcrucible: 57e5a875-f7b1-4735-a6d1-5be2ee30a948 is now active with session: 76942cb8-80db-41e5-abf0-a1b3f4c60fa1 session_id = 76942cb8-80db-41e5-abf0-a1b3f4c60fa1
95 {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level"Activate successful
96 :Wait for a query_work_queue command to finish before sending IO
97 30,"time":"2025-03-07T18:12:31.807710134Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1204,"session_id":"76942cb8-80db-41e5-abf0-a1b3f4c60fa1"}
982025-03-07T18:12:31.807ZINFOcrucible: The guest has finished waiting for activation with:1
99 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
100 Fill test
101 Read and Verify all blocks (0..5000 range:false)
102 Wrote out file "/tmp/test_replay_verify.log"
103 CLIENT: Tests done. All submitted work has been ACK'd
104 ----------------------------------------------------------------
105 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
106 States: Active Active Active
107 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
108 1021 Acked Read 256 Done Done Done false
109 1022 Acked Read 256 Done Done Done false
110 1023 Acked Read 256 Done Done Done false
111 1024 Acked Read 256 Done Done Done false
112 1025 Acked Read 256 Done Done Done false
113 1026 Acked Read 256 Done Done Done false
114 1027 Acked Read 256 Done Done Done false
115 1028 Acked Read 256 Done Done Done false
116 1029 Acked Read 256 Done Done Done false
117 1030 Acked Read 256 Done Done Done false
118 1031 Acked Read 256 Done Done Done false
119 1032 Acked Read 256 Done Done Done false
120 1033 Acked Read 256 Done Done Done false
121 1034 Acked Read 256 Done Done Done false
122 1035 Acked Read 256 Done Done Done false
123 1036 Acked Read 256 Done Done Done false
124 1037 Acked Read 256 Done Done Done false
125 1038 Acked Read 256 Done Done Done false
126 1039 Acked Read 256 Done Done Done false
127 1040 Acked Read 136 Done Done Done false
128 STATES DS:0 DS:1 DS:2 TOTAL
129 Sent 0 0 0 0
130 Done 20 20 20 60
131 Skipped 0 0 0 0
132 Error 0 0 0 0
133 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
134 Downstairs last five completed: 1020 1019 1018 1017 1016
135 Upstairs last five completed: 1040 1039 1038 1037 1036
136 CLIENT: Up:0 ds:20 act:3
137 ----------------------------------------------------------------
138 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
139 Downstairs last five completed: 1041 1040 1039 1038 1037
140 Upstairs last five completed: 1041 1040 1039 1038 1037
141 CLIENT: Up:0 ds:0 act:3
142 CLIENT: All crucible jobs finished, exiting program
1432025-03-07T18:12:36.372ZWARNcrucible: Guest handle has been dropped
144 Replay loop starts now March 7, 2025 at 06:12:36 PM UTC
145 Mar 07 18:12:36.441 INFO accepted connection, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
146 Mar 07 18:12:36.442 INFO request completed, latency_us: 141, response_code: 200, uri: /regioninfo, method: GET, req_id: 13233a90-12b7-4a5a-a4a3-7ee79fc0154a, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
1472025-03-07T18:12:36.443ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
148 Mar 07 18:12:36.443 INFO request completed, latency_us: 84, response_code: 200, uri: /regioncount, method: GET, req_id: 4f7767e3-01b6-435e-aac7-89b82b69b137, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
1492025-03-07T18:12:36.443ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
150 Mar 07 18:12:36.444 INFO request completed, latency_us: 106, response_code: 200, uri: /port/cid/0, method: GET, req_id: ac841770-3777-4fcb-9695-325365b82fbf, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
151 Mar 07 18:12:36.444 INFO request completed, latency_us: 100, response_code: 200, uri: /port/cid/1, method: GET, req_id: 6fc2cdc6-498f-4abb-a9a0-7c8b6bc19cc8, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
152 Mar 07 18:12:36.445 INFO request completed, latency_us: 84, response_code: 200, uri: /port/cid/2, method: GET, req_id: fdf58686-4ca0-4e8e-a287-20d5723c0ed9, remote_addr: 127.0.0.1:41645, local_addr: 127.0.0.1:9998
1532025-03-07T18:12:36.445ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1542025-03-07T18:12:36.445ZINFOcrucible: Upstairs starts
1552025-03-07T18:12:36.445ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "6c903058314cb23845d8ec9d1b70e497f207c60e", git_commit_timestamp: "2025-03-07T17:56:28.000000000Z", git_branch: "renovate/dropshot-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1562025-03-07T18:12:36.445ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1572025-03-07T18:12:36.445ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1582025-03-07T18:12:36.446ZINFOcrucible: Crucible 73fa22a5-b287-43f7-9d6f-29b9b8e2c2f7 has session id: 5e9909a1-dc0d-4821-9383-88a904f8ac68 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1592025-03-07T18:12:36.446ZINFOcrucible: Upstairs opts: Upstairs UUID: 73fa22a5-b287-43f7-9d6f-29b9b8e2c2f7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1602025-03-07T18:12:36.446ZINFOcrucible: Crucible stats registered with UUID: 73fa22a5-b287-43f7-9d6f-29b9b8e2c2f7 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1612025-03-07T18:12:36.446ZWARNcrucible: could not find Downstairs address for Nexus session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1622025-03-07T18:12:36.446ZINFOcrucible: Downstairs in volume = 3 task = crutest
1632025-03-07T18:12:36.447ZINFOcrucible: The guest has requested activation with gen:2
1642025-03-07T18:12:36.447ZINFOcrucible: 73fa22a5-b287-43f7-9d6f-29b9b8e2c2f7 active request set session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1652025-03-07T18:12:36.447ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1662025-03-07T18:12:36.447ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1672025-03-07T18:12:36.447ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1682025-03-07T18:12:36.447ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1692025-03-07T18:12:36.447ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1702025-03-07T18:12:36.447ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1712025-03-07T18:12:36.447ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1722025-03-07T18:12:36.447ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1732025-03-07T18:12:36.447ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1742025-03-07T18:12:36.447ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42969) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1752025-03-07T18:12:36.447ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38467) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1762025-03-07T18:12:36.447ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43013) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1772025-03-07T18:12:36.448ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1782025-03-07T18:12:36.448ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1792025-03-07T18:12:36.448ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1802025-03-07T18:12:36.448ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1812025-03-07T18:12:36.448ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1822025-03-07T18:12:36.448ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1832025-03-07T18:12:36.449ZINFOcrucible: Waiting for more clients to be ready session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1842025-03-07T18:12:36.449ZINFOcrucible: Waiting for more clients to be ready session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1852025-03-07T18:12:36.449ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1862025-03-07T18:12:36.449ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1872025-03-07T18:12:36.449ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1882025-03-07T18:12:36.449ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1892025-03-07T18:12:36.449ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1902025-03-07T18:12:36.449ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1912025-03-07T18:12:36.449ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1922025-03-07T18:12:36.449ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
193 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30Activate successful
194 ,Wait for a query_work_queue command to finish before sending IO
195 "time":"2025-03-07T18:12:36.449760619Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"downstairs","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
1962025-03-07T18:12:36.449ZINFOcrucible: Max found gen is 2 = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1972025-03-07T18:12:36.449ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1982025-03-07T18:12:36.449ZINFOcrucible: Next flush: 2 = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1992025-03-07T18:12:36.449ZINFOcrucible: All extents match = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2002025-03-07T18:12:36.449ZINFOcrucible: No downstairs reconciliation required session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2012025-03-07T18:12:36.449ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2022025-03-07T18:12:36.449ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2032025-03-07T18:12:36.449ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2042025-03-07T18:12:36.449ZINFOcrucible: All required reconciliation work is completed session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2052025-03-07T18:12:36.449ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2062025-03-07T18:12:36.449ZINFOcrucible: 73fa22a5-b287-43f7-9d6f-29b9b8e2c2f7 is now active with session: 5e9909a1-dc0d-4821-9383-88a904f8ac68 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2072025-03-07T18:12:36.449ZINFOcrucible: Set Active after no reconciliation session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2082025-03-07T18:12:36.449ZINFOcrucible: The guest has finished waiting for activation with:2
209 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "/tmp/test_replay_verify.log"
211 Mar 07 18:12:36.483 INFO accepted connection, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(0)
214 stop 0
215 Mar 07 18:12:36.484 INFO request completed, latency_us: 194, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 34a083da-10a4-4af5-a3ce-dc78b9568a4e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
216 [8810] Got stop action so:false kr:false
2172025-03-07T18:12:36.484ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2182025-03-07T18:12:36.484ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2192025-03-07T18:12:36.484ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2202025-03-07T18:12:36.485ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
221 Mar 07 18:12:36.485 INFO request completed, latency_us: 289, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9c9d06bf-2a93-41d1-8ef1-765b83692866, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
222 [8810] Exited with: ExitStatus(unix_wait_status(9))
223 [8810][0] reports Exit
224 Mar 07 18:12:40.486 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9bc34e68-6d30-42c7-841a-5b52c41deda4, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
225 001/300 Read block 3216 len 36864
226 002/300 Flush
227 003/300 Write block 1480 len 20480 data: 2 2 2 2 2
228 004/300 Write block 1907 len 20480 data: 2 2 2 2 2
229 005/300 Write block 3853 len 4096 data: 2
230 006/300 Write block 2762 len 28672 data: 2 2 2 2 2 2 2
231 007/300 Flush
232 008/300 Write block 3057 len 20480 data: 2 2 2 2 2
233 009/300 Read block 2829 len 40960
234 010/300 Write block 2574 len 24576 data: 2 2 2 2 2 2
235 011/300 Write block 2659 len 32768 data: 2 2 2 2 2 2 2 2
236 012/300 Read block 4692 len 20480
237 013/300 Write block 4356 len 32768 data: 2 2 2 2 2 2 2 2
238 014/300 Write block 4887 len 16384 data: 2 2 2 2
239 015/300 Read block 3871 len 16384
240 016/300 Read block 3323 len 16384
241 017/300 Read block 3342 len 32768
242 018/300 Write block 1958 len 12288 data: 2 2 2
243 019/300 Write block 2578 len 36864 data: 3 3 2 2 2 2 2 2 2
244 020/300 Read block 263 len 16384
245 021/300 Read block 4679 len 8192
246 022/300 Write block 903 len 12288 data: 2 2 2
247 023/300 Write block 4592 len 40960 data: 2 2 2 2 2 2 2 2 2 2
248 024/300 Read block 4764 len 20480
249 025/300 Read block 4265 len 32768
250 026/300 Write block 1512 len 16384 data: 2 2 2 2
251 027/300 Write block 3281 len 4096 data: 2
252 028/300 Read block 4656 len 24576
253 029/300 Read block 4068 len 4096
254 030/300 Read block 2086 len 32768
255 031/300 Write block 160 len 12288 data: 2 2 2
256 032/300 Write block 3084 len 36864 data: 2 2 2 2 2 2 2 2 2
257 033/300 Flush
258 034/300 Read block 4505 len 36864
259 035/300 Flush
260 036/300 Read block 230 len 32768
261 037/300 Read block 4455 len 24576
262 038/300 Write block 1629 len 4096 data: 2
263 039/300 Write block 500 len 28672 data: 2 2 2 2 2 2 2
264 040/300 Read block 3928 len 36864
265 041/300 Write block 1360 len 32768 data: 2 2 2 2 2 2 2 2
266 042/300 Read block 3024 len 16384
267 043/300 Write block 4075 len 36864 data: 2 2 2 2 2 2 2 2 2
268 044/300 Read block 3688 len 24576
269 045/300 Read block 3705 len 16384
270 046/300 Write block 1708 len 12288 data: 2 2 2
271 047/300 Flush
272 048/300 Read block 4094 len 12288
273 049/300 Read block 4005 len 20480
274 050/300 Write block 1846 len 28672 data: 2 2 2 2 2 2 2
275 051/300 Flush
276 052/300 Write block 2249 len 24576 data: 2 2 2 2 2 2
277 053/300 Flush
278 054/300 Write block 1949 len 12288 data: 2 2 2
279 055/300 Write block 3159 len 12288 data: 2 2 2
280 056/300 Write block 4048 len 8192 data: 2 2
281 057/300 Read block 2858 len 4096
282 058/300 Flush
283 059/300 Read block 3966 len 12288
284 060/300 Read block 4071 len 16384
285 061/300 Write block 429 len 28672 data: 2 2 2 2 2 2 2
286 062/300 Write block 1889 len 4096 data: 2
287 063/300 Read block 3300 len 28672
288 064/300 Write block 2463 len 20480 data: 2 2 2 2 2
289 065/300 Write block 4279 len 24576 data: 2 2 2 2 2 2
290 066/300 Write block 2207 len 12288 data: 2 2 2
291 067/300 Write block 2965 len 8192 data: 2 2
292 068/300 Write block 2437 len 24576 data: 2 2 2 2 2 2
293 069/300 Flush
294 070/300 Read block 3528 len 28672
295 071/300 Read block 3992 len 8192
296 072/300 Read block 1325 len 28672
297 073/300 Write block 4897 len 40960 data: 2 2 2 2 2 2 2 2 2 2
298 074/300 Write block 120 len 36864 data: 2 2 2 2 2 2 2 2 2
299 075/300 Write block 4182 len 28672 data: 2 2 2 2 2 2 2
300 076/300 Write block 4424 len 16384 data: 2 2 2 2
301 077/300 Read block 1898 len 8192
302 078/300 Read block 1243 len 28672
303 079/300 Read block 940 len 8192
304 080/300 Write block 3338 len 28672 data: 2 2 2 2 2 2 2
305 081/300 Write block 3180 len 4096 data: 2
306 082/300 Read block 3690 len 36864
307 083/300 Read block 3144 len 32768
308 084/300 Read block 3099 len 36864
309 085/300 Write block 4943 len 16384 data: 2 2 2 2
310 086/300 Read block 4247 len 16384
311 087/300 Write block 3383 len 32768 data: 2 2 2 2 2 2 2 2
312 088/300 Read block 1505 len 40960
313 089/300 Read block 2042 len 8192
314 090/300 Write block 2943 len 40960 data: 2 2 2 2 2 2 2 2 2 2
315 091/300 Write block 4202 len 28672 data: 2 2 2 2 2 2 2
316 092/300 Flush
317 093/300 Read block 1068 len 40960
318 094/300 Write block 4765 len 32768 data: 2 2 2 2 2 2 2 2
319 095/300 Read block 1221 len 32768
320 096/300 Write block 4755 len 36864 data: 2 2 2 2 2 2 2 2 2
321 097/300 Read block 2871 len 32768
322 098/300 Read block 3180 len 24576
323 099/300 Read block 935 len 8192
324 100/300 Flush
325 101/300 Read block 442 len 28672
326 102/300 Read block 1119 len 16384
327 103/300 Read block 3240 len 8192
328 104/300 Write block 2487 len 32768 data: 2 2 2 2 2 2 2 2
329 105/300 Read block 2077 len 36864
330 106/300 Write block 2807 len 12288 data: 2 2 2
331 107/300 Write block 2591 len 12288 data: 2 2 2
332 108/300 Flush
333 109/300 Write block 4283 len 40960 data: 3 3 2 2 2 2 2 2 2 2
334 110/300 Read block 850 len 36864
335 111/300 Read block 3038 len 32768
336 112/300 Read block 4802 len 24576
337 113/300 Read block 2043 len 8192
338 114/300 Read block 2808 len 16384
339 115/300 Read block 1586 len 8192
340 116/300 Read block 1857 len 32768
341 117/300 Read block 2547 len 36864
342 118/300 Write block 2393 len 12288 data: 2 2 2
343 119/300 Write block 2183 len 20480 data: 2 2 2 2 2
344 120/300 Read block 4244 len 40960
345 121/300 Read block 4003 len 28672
346 122/300 Write block 1485 len 16384 data: 2 2 2 2
347 123/300 Read block 2472 len 20480
348 124/300 Write block 817 len 16384 data: 2 2 2 2
349 125/300 Read block 2979 len 28672
350 126/300 Write block 3854 len 16384 data: 2 2 2 2
351 127/300 Read block 3395 len 28672
352 128/300 Read block 1808 len 4096
353 129/300 Write block 2000 len 8192 data: 2 2
354 130/300 Write block 824 len 36864 data: 2 2 2 2 2 2 2 2 2
355 131/300 Read block 4500 len 20480
356 132/300 Write block 2889 len 24576 data: 2 2 2 2 2 2
357 133/300 Read block 4674 len 24576
358 134/300 Flush
359 135/300 Write block 1150 len 24576 data: 2 2 2 2 2 2
360 136/300 Read block 1303 len 8192
361 137/300 Read block 4811 len 24576
362 138/300 Write block 4006 len 20480 data: 2 2 2 2 2
363 139/300 Write block 2303 len 4096 data: 2
364 140/300 Read block 1315 len 4096
365 141/300 Write block 3062 len 24576 data: 2 2 2 2 2 2
366 142/300 Write block 4342 len 12288 data: 2 2 2
367 143/300 Read block 2135 len 32768
368 144/300 Read block 2853 len 24576
369 145/300 Read block 1666 len 8192
370 146/300 Read block 4010 len 24576
371 147/300 Write block 203 len 16384 data: 2 2 2 2
372 148/300 Write block 4606 len 28672 data: 2 2 2 2 2 2 2
373 149/300 Read block 2158 len 8192
374 150/300 Read block 1641 len 8192
375 151/300 Write block 4470 len 32768 data: 2 2 2 2 2 2 2 2
376 152/300 Read block 840 len 36864
377 153/300 Read block 3060 len 36864
378 154/300 Read block 3416 len 28672
379 155/300 Write block 4186 len 8192 data: 3 3
380 156/300 Read block 3961 len 16384
381 157/300 Write block 4519 len 24576 data: 2 2 2 2 2 2
382 158/300 Flush
383 159/300 Write block 393 len 28672 data: 2 2 2 2 2 2 2
384 160/300 Write block 253 len 16384 data: 2 2 2 2
385 161/300 Write block 2142 len 36864 data: 2 2 2 2 2 2 2 2 2
386 162/300 Write block 3719 len 28672 data: 2 2 2 2 2 2 2
387 163/300 Write block 1893 len 20480 data: 2 2 2 2 2
388 164/300 Read block 4770 len 32768
389 165/300 Read block 124 len 12288
390 166/300 Write block 1516 len 40960 data: 2 2 2 2 2 2 2 2 2 2
391 167/300 Write block 1724 len 20480 data: 2 2 2 2 2
392 168/300 Read block 1627 len 28672
393 169/300 Read block 702 len 32768
394 170/300 Write block 3002 len 40960 data: 2 2 2 2 2 2 2 2 2 2
395 171/300 Read block 1485 len 36864
396 172/300 Write block 4737 len 8192 data: 2 2
397 173/300 Write block 3552 len 32768 data: 2 2 2 2 2 2 2 2
398 174/300 Write block 1405 len 40960 data: 2 2 2 2 2 2 2 2 2 2
399 175/300 Write block 1287 len 4096 data: 2
400 176/300 Read block 2128 len 24576
401 177/300 Write block 4463 len 32768 data: 2 2 2 2 2 2 2 3
402 178/300 Write block 2032 len 24576 data: 2 2 2 2 2 2
403 179/300 Read block 4880 len 8192
404 180/300 Read block 2617 len 28672
405 181/300 Write block 2598 len 28672 data: 2 2 2 2 2 2 2
406 182/300 Write block 2648 len 16384 data: 2 2 2 2
407 183/300 Read block 755 len 28672
408 184/300 Write block 3240 len 16384 data: 2 2 2 2
409 185/300 Read block 1542 len 4096
410 186/300 Flush
411 187/300 Read block 2276 len 4096
412 188/300 Read block 922 len 8192
413 189/300 Read block 3939 len 40960
414 190/300 Flush
415 191/300 Read block 4760 len 28672
416 192/300 Write block 4335 len 16384 data: 2 2 2 2
417 193/300 Read block 3504 len 32768
418 194/300 Flush
419 195/300 Write block 307 len 20480 data: 2 2 2 2 2
420 196/300 Write block 4244 len 8192 data: 2 2
421 197/300 Read block 4071 len 40960
422 198/300 Read block 2451 len 12288
423 199/300 Write block 261 len 32768 data: 2 2 2 2 2 2 2 2
424 200/300 Write block 3332 len 24576 data: 2 2 2 2 2 2
425 201/300 Flush
426 202/300 Read block 3871 len 20480
427 203/300 Write block 1610 len 40960 data: 2 2 2 2 2 2 2 2 2 2
428 204/300 Write block 302 len 32768 data: 2 2 2 2 2 3 3 3
429 205/300 Flush
430 206/300 Write block 3957 len 32768 data: 2 2 2 2 2 2 2 2
431 207/300 Flush
432 208/300 Flush
433 209/300 Read block 3042 len 24576
434 210/300 Write block 1699 len 28672 data: 2 2 2 2 2 2 2
435 211/300 Read block 495 len 24576
436 212/300 Flush
437 213/300 Read block 2410 len 12288
438 214/300 Write block 4891 len 24576 data: 2 2 2 2 2 2
439 215/300 Read block 2852 len 8192
440 216/300 Flush
441 217/300 Write block 324 len 40960 data: 2 2 2 2 2 2 2 2 2 2
442 218/300 Read block 3546 len 16384
443 219/300 Write block 2800 len 4096 data: 2
444 220/300 Read block 3717 len 32768
445 221/300 Read block 3075 len 8192
446 222/300 Flush
447 223/300 Read block 1900 len 16384
448 224/300 Write block 3024 len 20480 data: 2 2 2 2 2
449 225/300 Write block 1599 len 40960 data: 2 2 2 2 2 2 2 2 2 2
450 226/300 Read block 4961 len 28672
451 227/300 Read block 3552 len 24576
452 228/300 Read block 304 len 4096
453 229/300 Write block 2931 len 20480 data: 2 2 2 2 2
454 230/300 Read block 707 len 36864
455 231/300 Read block 3572 len 32768
456 232/300 Read block 3529 len 28672
457 233/300 Read block 2822 len 28672
458 234/300 Read block 541 len 36864
459 235/300 Read block 1420 len 32768
460 236/300 Read block 2231 len 4096
461 237/300 Read block 1975 len 16384
462 238/300 Read block 2071 len 8192
463 239/300 Write block 2279 len 40960 data: 2 2 2 2 2 2 2 2 2 2
464 240/300 Flush
465 241/300 Read block 4702 len 20480
466 242/300 Read block 2440 len 40960
467 243/300 Read block 2128 len 24576
468 244/300 Read block 55 len 36864
469 245/300 Read block 4046 len 8192
470 246/300 Read block 4197 len 40960
471 247/300 Read block 3597 len 20480
472 248/300 Read block 4235 len 28672
473 249/300 Write block 1328 len 16384 data: 2 2 2 2
474 250/300 Read block 3440 len 28672
475 251/300 Read block 2508 len 32768
476 252/300 Write block 4319 len 40960 data: 2 2 2 2 2 2 2 2 2 2
477 253/300 Write block 2513 len 4096 data: 2
478 254/300 Read block 599 len 40960
479 255/300 Flush
480 256/300 Read block 4962 len 24576
481 257/300 Read block 2112 len 40960
482 258/300 Write block 2270 len 20480 data: 2 2 2 2 2
483 259/300 Read block 1766 len 4096
484 260/300 Write block 3076 len 40960 data: 2 2 2 2 2 2 2 2 3 3
485 261/300 Write block 2314 len 4096 data: 2
486 262/300 Write block 979 len 40960 data: 2 2 2 2 2 2 2 2 2 2
487 263/300 Write block 676 len 16384 data: 2 2 2 2
488 264/300 Read block 178 len 12288
489 265/300 Write block 1693 len 20480 data: 2 2 2 2 2
490 266/300 Read block 1325 len 40960
491 267/300 Read block 4742 len 4096
492 268/300 Write block 2779 len 4096 data: 2
493 269/300 Read block 2744 len 20480
494 270/300 Write block 3100 len 40960 data: 2 2 2 2 2 2 2 2 2 2
495 271/300 Write block 2618 len 16384 data: 2 2 2 2
496 272/300 Read block 2606 len 32768
497 273/300 Read block 693 len 12288
498 274/300 Write block 3322 len 28672 data: 2 2 2 2 2 2 2
499 275/300 Write block 4373 len 32768 data: 2 2 2 2 2 2 2 2
500 276/300 Read block 3245 len 4096
501 277/300 Read block 398 len 16384
502 278/300 Write block 335 len 4096 data: 2
503 279/300 Write block 4421 len 28672 data: 2 2 2 3 3 3 3
504 280/300 Write block 2512 len 36864 data: 2 3 2 2 2 2 2 2 2
505 281/300 Write block 726 len 16384 data: 2 2 2 2
506 282/300 Read block 4300 len 24576
507 283/300 Write block 212 len 20480 data: 2 2 2 2 2
508 284/300 Write block 1161 len 4096 data: 2
509 285/300 Flush
510 286/300 Write block 2241 len 16384 data: 2 2 2 2
511 287/300 Read block 4884 len 16384
512 288/300 Read block 4966 len 28672
513 289/300 Write block 307 len 40960 data: 4 4 4 3 3 2 2 2 2 2
514 290/300 Write block 1902 len 12288 data: 2 2 2
515 291/300 Flush
516 292/300 Read block 905 len 8192
517 293/300 Read block 245 len 40960
518 294/300 Read block 2346 len 24576
519 295/300 Flush
520 296/300 Write block 3092 len 36864 data: 3 2 2 2 2 2 2 2 3
521 297/300 Write block 2799 len 16384 data: 2 3 2 2
522 298/300 Write block 1531 len 40960 data: 2 2 2 2 2 2 2 2 2 2
523 299/300 Write block 4240 len 12288 data: 2 2 2
524 300/300 Read block 4323 len 24576
525 301/300 Flush
526 Main task has work to do, go find it
527 got dsc Start(0)
528 start 0
529 Mar 07 18:12:40.733 INFO request completed, latency_us: 187, response_code: 204, uri: /start/cid/0, method: POST, req_id: 03d2ab00-76e8-44cb-814e-9149e7389d46, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
530 Replay: started 0, returned:Ok(())
531 ----------------------------------------------------------------
532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
533 States: Offline Active Active
534 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
535 1000 Acked Read 9 Sent Done Done false
536 1001 Acked Flush 0 Sent Done Done false
537 1002 Acked Write 5 Sent Done Done false
538 1003 Acked Write 5 Sent Done Done false
539 1004 Acked Write 1 Sent Done Done false
540 1005 Acked Write 7 Sent Done Done false
541 1006 Acked Flush 0 Sent Done Done false
542 1007 Acked Write 5 Sent Done Done false
543 1008 Acked Read 10 Sent Done Done false
544 1009 Acked Write 6 Sent Done Done false
545 1010 Acked Write 8 Sent Done Done false
546 1011 Acked Read 5 Sent Done Done false
547 1012 Acked Write 8 Sent Done Done false
548 1013 Acked Write 4 Sent Done Done false
549 1014 Acked Read 4 Sent Done Done false
550 1015 Acked Read 4 Sent Done Done false
551 1016 Acked Read 8 Sent Done Done false
552 1017 Acked Write 3 Sent Done Done false
553 1018 Acked Write 9 Sent Done Done false
554 1019 Acked Read 4 Sent Done Done false
555 1020 Acked Read 2 Sent Done Done false
556 1021 Acked Write 3 Sent Done Done false
557 1022 Acked Write 10 Sent Done Done false
558 1023 Acked Read 5 Sent Done Done false
559 1024 Acked Read 8 Sent Done Done false
560 1025 Acked Write 4 Sent Done Done false
561 1026 Acked Write 1 Sent Done Done false
562 1027 Acked Read 6 Sent Done Done false
563 1028 Acked Read 1 Sent Done Done false
564 1029 Acked Read 8 Sent Done Done false
565 1030 Acked Write 3 Sent Done Done false
566 1031 Acked Write 9 Sent Done Done false
567 1032 Acked Flush 0 Sent Done Done false
568 1033 Acked Read 9 Sent Done Done false
569 1034 Acked Flush 0 Sent Done Done false
570 1035 Acked Read 8 Sent Done Done false
571 1036 Acked Read 6 Sent Done Done false
572 1037 Acked Write 1 Sent Done Done false
573 1038 Acked Write 7 Sent Done Done false
574 1039 Acked Read 9 Sent Done Done false
575 1040 Acked Write 8 Sent Done Done false
576 1041 Acked Read 4 Sent Done Done false
577 1042 Acked Write 9 Sent Done Done false
578 1043 Acked Read 6 Sent Done Done false
579 1044 Acked Read 4 Sent Done Done false
580 1045 Acked Write 3 Sent Done Done false
581 1046 Acked Flush 0 Sent Done Done false
582 1047 Acked Read 3 Sent Done Done false
583 1048 Acked Read 5 Sent Done Done false
584 1049 Acked Write 7 Sent Done Done false
585 1050 Acked Flush 0 Sent Done Done false
586 1051 Acked Write 6 Sent Done Done false
587 1052 Acked Flush 0 Sent Done Done false
588 1053 Acked Write 3 Sent Done Done false
589 1054 Acked Write 3 Sent Done Done false
590 1055 Acked Write 2 Sent Done Done false
591 1056 Acked Read 1 Sent Done Done false
592 1057 Acked Flush 0 Sent Done Done false
593 1058 Acked Read 3 Sent Done Done false
594 1059 Acked Read 4 Sent Done Done false
595 1060 Acked Write 7 Sent Done Done false
596 1061 Acked Write 1 Sent Done Done false
597 1062 Acked Read 7 Sent Done Done false
598 1063 Acked Write 5 Sent Done Done false
599 1064 Acked Write 6 Sent Done Done false
600 1065 Acked Write 3 Sent Done Done false
601 1066 Acked Write 2 Sent Done Done false
602 1067 Acked Write 6 Sent Done Done false
603 1068 Acked Flush 0 Sent Done Done false
604 1069 Acked Read 7 Sent Done Done false
605 1070 Acked Read 2 Sent Done Done false
606 1071 Acked Read 7 Sent Done Done false
607 1072 Acked Write 10 Sent Done Done false
608 1073 Acked Write 9 Sent Done Done false
609 1074 Acked Write 7 Sent Done Done false
610 1075 Acked Write 4 Sent Done Done false
611 1076 Acked Read 2 Sent Done Done false
612 1077 Acked Read 7 Sent Done Done false
613 1078 Acked Read 2 Sent Done Done false
614 1079 Acked Write 7 Sent Done Done false
615 1080 Acked Write 1 Sent Done Done false
616 1081 Acked Read 9 Sent Done Done false
617 1082 Acked Read 8 Sent Done Done false
618 1083 Acked Read 9 Sent Done Done false
619 1084 Acked Write 4 Sent Done Done false
620 1085 Acked Read 4 Sent Done Done false
621 1086 Acked Write 8 Sent Done Done false
622 1087 Acked Read 10 Sent Done Done false
623 1088 Acked Read 2 Sent Done Done false
624 1089 Acked Write 10 Sent Done Done false
625 1090 Acked Write 7 Sent Done Done false
626 1091 Acked Flush 0 Sent Done Done false
627 1092 Acked Read 10 Sent Done Done false
628 1093 Acked Write 8 Sent Done Done false
629 1094 Acked Read 8 Sent Done Done false
630 1095 Acked Write 9 Sent Done Done false
631 1096 Acked Read 8 Sent Done Done false
632 1097 Acked Read 6 Sent Done Done false
633 1098 Acked Read 2 Sent Done Done false
634 1099 Acked Flush 0 Sent Done Done false
635 1100 Acked Read 7 Sent Done Done false
636 1101 Acked Read 4 Sent Done Done false
637 1102 Acked Read 2 Sent Done Done false
638 1103 Acked Write 8 Sent Done Done false
639 1104 Acked Read 9 Sent Done Done false
640 1105 Acked Write 3 Sent Done Done false
641 1106 Acked Write 3 Sent Done Done false
642 1107 Acked Flush 0 Sent Done Done false
643 1108 Acked Write 10 Sent Done Done false
644 1109 Acked Read 9 Sent Done Done false
645 1110 Acked Read 8 Sent Done Done false
646 1111 Acked Read 6 Sent Done Done false
647 1112 Acked Read 2 Sent Done Done false
648 1113 Acked Read 4 Sent Done Done false
649 1114 Acked Read 2 Sent Done Done false
650 1115 Acked Read 8 Sent Done Done false
651 1116 Acked Read 9 Sent Done Done false
652 1117 Acked Write 3 Sent Done Done false
653 1118 Acked Write 5 Sent Done Done false
654 1119 Acked Read 10 Sent Done Done false
655 1120 Acked Read 7 Sent Done Done false
656 1121 Acked Write 4 Sent Done Done false
657 1122 Acked Read 5 Sent Done Done false
658 1123 Acked Write 4 Sent Done Done false
659 1124 Acked Read 7 Sent Done Done false
660 1125 Acked Write 4 Sent Done Done false
661 1126 Acked Read 7 Sent Done Done false
662 1127 Acked Read 1 Sent Done Done false
663 1128 Acked Write 2 Sent Done Done false
664 1129 Acked Write 9 Sent Done Done false
665 1130 Acked Read 5 Sent Done Done false
666 1131 Acked Write 6 Sent Done Done false
667 1132 Acked Read 6 Sent Done Done false
668 1133 Acked Flush 0 Sent Done Done false
669 1134 Acked Write 6 Sent Done Done false
670 1135 Acked Read 2 Sent Done Done false
671 1136 Acked Read 6 Sent Done Done false
672 1137 Acked Write 5 Sent Done Done false
673 1138 Acked Write 1 Sent Done Done false
674 1139 Acked Read 1 Sent Done Done false
675 1140 Acked Write 6 Sent Done Done false
676 1141 Acked Write 3 Sent Done Done false
677 1142 Acked Read 8 Sent Done Done false
678 1143 Acked Read 6 Sent Done Done false
679 1144 Acked Read 2 Sent Done Done false
680 1145 Acked Read 6 Sent Done Done false
681 1146 Acked Write 4 Sent Done Done false
682 1147 Acked Write 7 Sent Done Done false
683 1148 Acked Read 2 Sent Done Done false
684 1149 Acked Read 2 Sent Done Done false
685 1150 Acked Write 8 Sent Done Done false
686 1151 Acked Read 9 Sent Done Done false
687 1152 Acked Read 9 Sent Done Done false
688 1153 Acked Read 7 Sent Done Done false
689 1154 Acked Write 2 Sent Done Done false
690 1155 Acked Read 4 Sent Done Done false
691 1156 Acked Write 6 Sent Done Done false
692 1157 Acked Flush 0 Sent Done Done false
693 1158 Acked Write 7 Sent Done Done false
694 1159 Acked Write 4 Sent Done Done false
695 1160 Acked Write 9 Sent Done Done false
696 1161 Acked Write 7 Sent Done Done false
697 1162 Acked Write 5 Sent Done Done false
698 1163 Acked Read 8 Sent Done Done false
699 1164 Acked Read 3 Sent Done Done false
700 1165 Acked Write 10 Sent Done Done false
701 1166 Acked Write 5 Sent Done Done false
702 1167 Acked Read 7 Sent Done Done false
703 1168 Acked Read 8 Sent Done Done false
704 1169 Acked Write 10 Sent Done Done false
705 1170 Acked Read 9 Sent Done Done false
706 1171 Acked Write 2 Sent Done Done false
707 1172 Acked Write 8 Sent Done Done false
708 1173 Acked Write 10 Sent Done Done false
709 1174 Acked Write 1 Sent Done Done false
710 1175 Acked Read 6 Sent Done Done false
711 1176 Acked Write 8 Sent Done Done false
712 1177 Acked Write 6 Sent Done Done false
713 1178 Acked Read 2 Sent Done Done false
714 1179 Acked Read 7 Sent Done Done false
715 1180 Acked Write 7 Sent Done Done false
716 1181 Acked Write 4 Sent Done Done false
717 1182 Acked Read 7 Sent Done Done false
718 1183 Acked Write 4 Sent Done Done false
719 1184 Acked Read 1 Sent Done Done false
720 1185 Acked Flush 0 Sent Done Done false
721 1186 Acked Read 1 Sent Done Done false
722 1187 Acked Read 2 Sent Done Done false
723 1188 Acked Read 10 Sent Done Done false
724 1189 Acked Flush 0 Sent Done Done false
725 1190 Acked Read 7 Sent Done Done false
726 1191 Acked Write 4 Sent Done Done false
727 1192 Acked Read 8 Sent Done Done false
728 1193 Acked Flush 0 Sent Done Done false
729 1194 Acked Write 5 Sent Done Done false
730 1195 Acked Write 2 Sent Done Done false
731 1196 Acked Read 10 Sent Done Done false
732 1197 Acked Read 3 Sent Done Done false
733 1198 Acked Write 8 Sent Done Done false
734 1199 Acked Write 6 Sent Done Done false
735 1200 Acked Flush 0 Sent Done Done false
736 1201 Acked Read 5 Sent Done Done false
737 1202 Acked Write 10 Sent Done Done false
738 1203 Acked Write 8 Sent Done Done false
739 1204 Acked Flush 0 Sent Done Done false
740 1205 Acked Write 8 Sent Done Done false
741 1206 Acked Flush 0 Sent Done Done false
742 1207 Acked Flush 0 Sent Done Done false
743 1208 Acked Read 6 Sent Done Done false
744 1209 Acked Write 7 Sent Done Done false
745 1210 Acked Read 6 Sent Done Done false
746 1211 Acked Flush 0 Sent Done Done false
747 1212 Acked Read 3 Sent Done Done false
748 1213 Acked Write 6 Sent Done Done false
749 1214 Acked Read 2 Sent Done Done false
750 1215 Acked Flush 0 Sent Done Done false
751 1216 Acked Write 10 Sent Done Done false
752 1217 Acked Read 4 Sent Done Done false
753 1218 Acked Write 1 Sent Done Done false
754 1219 Acked Read 8 Sent Done Done false
755 1220 Acked Read 2 Sent Done Done false
756 1221 Acked Flush 0 Sent Done Done false
757 1222 Acked Read 4 Sent Done Done false
758 1223 Acked Write 5 Sent Done Done false
759 1224 Acked Write 10 Sent Done Done false
760 1225 Acked Read 7 Sent Done Done false
761 1226 Acked Read 6 Sent Done Done false
762 1227 Acked Read 1 Sent Done Done false
763 1228 Acked Write 5 Sent Done Done false
764 1229 Acked Read 9 Sent Done Done false
765 1230 Acked Read 8 Sent Done Done false
766 1231 Acked Read 7 Sent Done Done false
767 1232 Acked Read 7 Sent Done Done false
768 1233 Acked Read 9 Sent Done Done false
769 1234 Acked Read 8 Sent Done Done false
770 1235 Acked Read 1 Sent Done Done false
771 1236 Acked Read 4 Sent Done Done false
772 1237 Acked Read 2 Sent Done Done false
773 1238 Acked Write 10 Sent Done Done false
774 1239 Acked Flush 0 Sent Done Done false
775 1240 Acked Read 5 Sent Done Done false
776 1241 Acked Read 10 Sent Done Done false
777 1242 Acked Read 6 Sent Done Done false
778 1243 Acked Read 9 Sent Done Done false
779 1244 Acked Read 2 Sent Done Done false
780 1245 Acked Read 10 Sent Done Done false
781 1246 Acked Read 5 Sent Done Done false
782 1247 Acked Read 7 Sent Done Done false
783 1248 Acked Write 4 Sent Done Done false
784 1249 Acked Read 7 Sent Done Done false
785 1250 Acked Read 8 Sent Done Done false
786 1251 Acked Write 10 Sent Done Done false
787 1252 Acked Write 1 Sent Done Done false
788 1253 Acked Read 10 Sent Done Done false
789 1254 Acked Flush 0 Sent Done Done false
790 1255 Acked Read 6 Sent Done Done false
791 1256 Acked Read 10 Sent Done Done false
792 1257 Acked Write 5 Sent Done Done false
793 1258 Acked Read 1 Sent Done Done false
794 1259 Acked Write 10 Sent Done Done false
795 1260 Acked Write 1 Sent Done Done false
796 1261 Acked Write 10 Sent Done Done false
797 1262 Acked Write 4 Sent Done Done false
798 1263 Acked Read 3 Sent Done Done false
799 1264 Acked Write 5 Sent Done Done false
800 1265 Acked Read 10 Sent Done Done false
801 1266 Acked Read 1 Sent Done Done false
802 1267 Acked Write 1 Sent Done Done false
803 1268 Acked Read 5 Sent Done Done false
804 1269 Acked Write 10 Sent Done Done false
805 1270 Acked Write 4 Sent Done Done false
806 1271 Acked Read 8 Sent Done Done false
807 1272 Acked Read 3 Sent Done Done false
808 1273 Acked Write 7 Sent Done Done false
809 1274 Acked Write 8 Sent Done Done false
810 1275 Acked Read 1 Sent Done Done false
811 1276 Acked Read 4 Sent Done Done false
812 1277 Acked Write 1 Sent Done Done false
813 1278 Acked Write 7 Sent Done Done false
814 1279 Acked Write 9 Sent Done Done false
815 1280 Acked Write 4 Sent Done Done false
816 1281 Acked Read 6 Sent Done Done false
817 1282 Acked Write 5 Sent Done Done false
818 1283 Acked Write 1 Sent Done Done false
819 1284 Acked Flush 0 Sent Done Done false
820 1285 Acked Write 4 Sent Done Done false
821 1286 Acked Read 4 Sent Done Done false
822 1287 Acked Read 7 Sent Done Done false
823 1288 Acked Write 10 Sent Done Done false
824 1289 Acked Write 3 Sent Done Done false
825 1290 Acked Flush 0 Sent Done Done false
826 1291 Acked Read 2 Sent Done Done false
827 1292 Acked Read 10 Sent Done Done false
828 1293 Acked Read 6 Sent Done Done false
829 1294 Acked Flush 0 Sent Done Done false
830 1295 Acked Write 9 Sent Done Done false
831 1296 Acked Write 4 Sent Done Done false
832 1297 Acked Write 10 Sent Done Done false
833 1298 Acked Write 3 Sent Done Done false
834 1299 Acked Read 6 Sent Done Done false
835 1300 Acked Flush 0 Sent Done Done false
836 STATES DS:0 DS:1 DS:2 TOTAL
837 Sent 301 0 0 301
838 Done 0 301 301 602
839 Skipped 0 0 0 0
840 Error 0 0 0 0
841 Last Flush: None Some(JobId(1300)) Some(JobId(1300))
842 Downstairs last five completed:
843 Upstairs last five completed: 1300 1299 1298 1297 1296
844 CLIENT: Up:0 ds:301 act:2
8452025-03-07T18:12:41.236ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8462025-03-07T18:12:41.956ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8472025-03-07T18:12:42.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8482025-03-07T18:12:43.458ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8492025-03-07T18:12:43.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
850 ----------------------------------------------------------------
851 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
852 States: Offline Active Active
853 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
854 1000 Acked Read 9 Sent Done Done false
855 1001 Acked Flush 0 Sent Done Done false
856 1002 Acked Write 5 Sent Done Done false
857 1003 Acked Write 5 Sent Done Done false
858 1004 Acked Write 1 Sent Done Done false
859 1005 Acked Write 7 Sent Done Done false
860 1006 Acked Flush 0 Sent Done Done false
861 1007 Acked Write 5 Sent Done Done false
862 1008 Acked Read 10 Sent Done Done false
863 1009 Acked Write 6 Sent Done Done false
864 1010 Acked Write 8 Sent Done Done false
865 1011 Acked Read 5 Sent Done Done false
866 1012 Acked Write 8 Sent Done Done false
867 1013 Acked Write 4 Sent Done Done false
868 1014 Acked Read 4 Sent Done Done false
869 1015 Acked Read 4 Sent Done Done false
870 1016 Acked Read 8 Sent Done Done false
871 1017 Acked Write 3 Sent Done Done false
872 1018 Acked Write 9 Sent Done Done false
873 1019 Acked Read 4 Sent Done Done false
874 1020 Acked Read 2 Sent Done Done false
875 1021 Acked Write 3 Sent Done Done false
876 1022 Acked Write 10 Sent Done Done false
877 1023 Acked Read 5 Sent Done Done false
878 1024 Acked Read 8 Sent Done Done false
879 1025 Acked Write 4 Sent Done Done false
880 1026 Acked Write 1 Sent Done Done false
881 1027 Acked Read 6 Sent Done Done false
882 1028 Acked Read 1 Sent Done Done false
883 1029 Acked Read 8 Sent Done Done false
884 1030 Acked Write 3 Sent Done Done false
885 1031 Acked Write 9 Sent Done Done false
886 1032 Acked Flush 0 Sent Done Done false
887 1033 Acked Read 9 Sent Done Done false
888 1034 Acked Flush 0 Sent Done Done false
889 1035 Acked Read 8 Sent Done Done false
890 1036 Acked Read 6 Sent Done Done false
891 1037 Acked Write 1 Sent Done Done false
892 1038 Acked Write 7 Sent Done Done false
893 1039 Acked Read 9 Sent Done Done false
894 1040 Acked Write 8 Sent Done Done false
895 1041 Acked Read 4 Sent Done Done false
896 1042 Acked Write 9 Sent Done Done false
897 1043 Acked Read 6 Sent Done Done false
898 1044 Acked Read 4 Sent Done Done false
899 1045 Acked Write 3 Sent Done Done false
900 1046 Acked Flush 0 Sent Done Done false
901 1047 Acked Read 3 Sent Done Done false
902 1048 Acked Read 5 Sent Done Done false
903 1049 Acked Write 7 Sent Done Done false
904 1050 Acked Flush 0 Sent Done Done false
905 1051 Acked Write 6 Sent Done Done false
906 1052 Acked Flush 0 Sent Done Done false
907 1053 Acked Write 3 Sent Done Done false
908 1054 Acked Write 3 Sent Done Done false
909 1055 Acked Write 2 Sent Done Done false
910 1056 Acked Read 1 Sent Done Done false
911 1057 Acked Flush 0 Sent Done Done false
912 1058 Acked Read 3 Sent Done Done false
913 1059 Acked Read 4 Sent Done Done false
914 1060 Acked Write 7 Sent Done Done false
915 1061 Acked Write 1 Sent Done Done false
916 1062 Acked Read 7 Sent Done Done false
917 1063 Acked Write 5 Sent Done Done false
918 1064 Acked Write 6 Sent Done Done false
919 1065 Acked Write 3 Sent Done Done false
920 1066 Acked Write 2 Sent Done Done false
921 1067 Acked Write 6 Sent Done Done false
922 1068 Acked Flush 0 Sent Done Done false
923 1069 Acked Read 7 Sent Done Done false
924 1070 Acked Read 2 Sent Done Done false
925 1071 Acked Read 7 Sent Done Done false
926 1072 Acked Write 10 Sent Done Done false
927 1073 Acked Write 9 Sent Done Done false
928 1074 Acked Write 7 Sent Done Done false
929 1075 Acked Write 4 Sent Done Done false
930 1076 Acked Read 2 Sent Done Done false
931 1077 Acked Read 7 Sent Done Done false
932 1078 Acked Read 2 Sent Done Done false
933 1079 Acked Write 7 Sent Done Done false
934 1080 Acked Write 1 Sent Done Done false
935 1081 Acked Read 9 Sent Done Done false
936 1082 Acked Read 8 Sent Done Done false
937 1083 Acked Read 9 Sent Done Done false
938 1084 Acked Write 4 Sent Done Done false
939 1085 Acked Read 4 Sent Done Done false
940 1086 Acked Write 8 Sent Done Done false
941 1087 Acked Read 10 Sent Done Done false
942 1088 Acked Read 2 Sent Done Done false
943 1089 Acked Write 10 Sent Done Done false
944 1090 Acked Write 7 Sent Done Done false
945 1091 Acked Flush 0 Sent Done Done false
946 1092 Acked Read 10 Sent Done Done false
947 1093 Acked Write 8 Sent Done Done false
948 1094 Acked Read 8 Sent Done Done false
949 1095 Acked Write 9 Sent Done Done false
950 1096 Acked Read 8 Sent Done Done false
951 1097 Acked Read 6 Sent Done Done false
952 1098 Acked Read 2 Sent Done Done false
953 1099 Acked Flush 0 Sent Done Done false
954 1100 Acked Read 7 Sent Done Done false
955 1101 Acked Read 4 Sent Done Done false
956 1102 Acked Read 2 Sent Done Done false
957 1103 Acked Write 8 Sent Done Done false
958 1104 Acked Read 9 Sent Done Done false
959 1105 Acked Write 3 Sent Done Done false
960 1106 Acked Write 3 Sent Done Done false
961 1107 Acked Flush 0 Sent Done Done false
962 1108 Acked Write 10 Sent Done Done false
963 1109 Acked Read 9 Sent Done Done false
964 1110 Acked Read 8 Sent Done Done false
965 1111 Acked Read 6 Sent Done Done false
966 1112 Acked Read 2 Sent Done Done false
967 1113 Acked Read 4 Sent Done Done false
968 1114 Acked Read 2 Sent Done Done false
969 1115 Acked Read 8 Sent Done Done false
970 1116 Acked Read 9 Sent Done Done false
971 1117 Acked Write 3 Sent Done Done false
972 1118 Acked Write 5 Sent Done Done false
973 1119 Acked Read 10 Sent Done Done false
974 1120 Acked Read 7 Sent Done Done false
975 1121 Acked Write 4 Sent Done Done false
976 1122 Acked Read 5 Sent Done Done false
977 1123 Acked Write 4 Sent Done Done false
978 1124 Acked Read 7 Sent Done Done false
979 1125 Acked Write 4 Sent Done Done false
980 1126 Acked Read 7 Sent Done Done false
981 1127 Acked Read 1 Sent Done Done false
982 1128 Acked Write 2 Sent Done Done false
983 1129 Acked Write 9 Sent Done Done false
984 1130 Acked Read 5 Sent Done Done false
985 1131 Acked Write 6 Sent Done Done false
986 1132 Acked Read 6 Sent Done Done false
987 1133 Acked Flush 0 Sent Done Done false
988 1134 Acked Write 6 Sent Done Done false
989 1135 Acked Read 2 Sent Done Done false
990 1136 Acked Read 6 Sent Done Done false
991 1137 Acked Write 5 Sent Done Done false
992 1138 Acked Write 1 Sent Done Done false
993 1139 Acked Read 1 Sent Done Done false
994 1140 Acked Write 6 Sent Done Done false
995 1141 Acked Write 3 Sent Done Done false
996 1142 Acked Read 8 Sent Done Done false
997 1143 Acked Read 6 Sent Done Done false
998 1144 Acked Read 2 Sent Done Done false
999 1145 Acked Read 6 Sent Done Done false
1000 1146 Acked Write 4 Sent Done Done false
1001 1147 Acked Write 7 Sent Done Done false
1002 1148 Acked Read 2 Sent Done Done false
1003 1149 Acked Read 2 Sent Done Done false
1004 1150 Acked Write 8 Sent Done Done false
1005 1151 Acked Read 9 Sent Done Done false
1006 1152 Acked Read 9 Sent Done Done false
1007 1153 Acked Read 7 Sent Done Done false
1008 1154 Acked Write 2 Sent Done Done false
1009 1155 Acked Read 4 Sent Done Done false
1010 1156 Acked Write 6 Sent Done Done false
1011 1157 Acked Flush 0 Sent Done Done false
1012 1158 Acked Write 7 Sent Done Done false
1013 1159 Acked Write 4 Sent Done Done false
1014 1160 Acked Write 9 Sent Done Done false
1015 1161 Acked Write 7 Sent Done Done false
1016 1162 Acked Write 5 Sent Done Done false
1017 1163 Acked Read 8 Sent Done Done false
1018 1164 Acked Read 3 Sent Done Done false
1019 1165 Acked Write 10 Sent Done Done false
1020 1166 Acked Write 5 Sent Done Done false
1021 1167 Acked Read 7 Sent Done Done false
1022 1168 Acked Read 8 Sent Done Done false
1023 1169 Acked Write 10 Sent Done Done false
1024 1170 Acked Read 9 Sent Done Done false
1025 1171 Acked Write 2 Sent Done Done false
1026 1172 Acked Write 8 Sent Done Done false
1027 1173 Acked Write 10 Sent Done Done false
1028 1174 Acked Write 1 Sent Done Done false
1029 1175 Acked Read 6 Sent Done Done false
1030 1176 Acked Write 8 Sent Done Done false
1031 1177 Acked Write 6 Sent Done Done false
1032 1178 Acked Read 2 Sent Done Done false
1033 1179 Acked Read 7 Sent Done Done false
1034 1180 Acked Write 7 Sent Done Done false
1035 1181 Acked Write 4 Sent Done Done false
1036 1182 Acked Read 7 Sent Done Done false
1037 1183 Acked Write 4 Sent Done Done false
1038 1184 Acked Read 1 Sent Done Done false
1039 1185 Acked Flush 0 Sent Done Done false
1040 1186 Acked Read 1 Sent Done Done false
1041 1187 Acked Read 2 Sent Done Done false
1042 1188 Acked Read 10 Sent Done Done false
1043 1189 Acked Flush 0 Sent Done Done false
1044 1190 Acked Read 7 Sent Done Done false
1045 1191 Acked Write 4 Sent Done Done false
1046 1192 Acked Read 8 Sent Done Done false
1047 1193 Acked Flush 0 Sent Done Done false
1048 1194 Acked Write 5 Sent Done Done false
1049 1195 Acked Write 2 Sent Done Done false
1050 1196 Acked Read 10 Sent Done Done false
1051 1197 Acked Read 3 Sent Done Done false
1052 1198 Acked Write 8 Sent Done Done false
1053 1199 Acked Write 6 Sent Done Done false
1054 1200 Acked Flush 0 Sent Done Done false
1055 1201 Acked Read 5 Sent Done Done false
1056 1202 Acked Write 10 Sent Done Done false
1057 1203 Acked Write 8 Sent Done Done false
1058 1204 Acked Flush 0 Sent Done Done false
1059 1205 Acked Write 8 Sent Done Done false
1060 1206 Acked Flush 0 Sent Done Done false
1061 1207 Acked Flush 0 Sent Done Done false
1062 1208 Acked Read 6 Sent Done Done false
1063 1209 Acked Write 7 Sent Done Done false
1064 1210 Acked Read 6 Sent Done Done false
1065 1211 Acked Flush 0 Sent Done Done false
1066 1212 Acked Read 3 Sent Done Done false
1067 1213 Acked Write 6 Sent Done Done false
1068 1214 Acked Read 2 Sent Done Done false
1069 1215 Acked Flush 0 Sent Done Done false
1070 1216 Acked Write 10 Sent Done Done false
1071 1217 Acked Read 4 Sent Done Done false
1072 1218 Acked Write 1 Sent Done Done false
1073 1219 Acked Read 8 Sent Done Done false
1074 1220 Acked Read 2 Sent Done Done false
1075 1221 Acked Flush 0 Sent Done Done false
1076 1222 Acked Read 4 Sent Done Done false
1077 1223 Acked Write 5 Sent Done Done false
1078 1224 Acked Write 10 Sent Done Done false
1079 1225 Acked Read 7 Sent Done Done false
1080 1226 Acked Read 6 Sent Done Done false
1081 1227 Acked Read 1 Sent Done Done false
1082 1228 Acked Write 5 Sent Done Done false
1083 1229 Acked Read 9 Sent Done Done false
1084 1230 Acked Read 8 Sent Done Done false
1085 1231 Acked Read 7 Sent Done Done false
1086 1232 Acked Read 7 Sent Done Done false
1087 1233 Acked Read 9 Sent Done Done false
1088 1234 Acked Read 8 Sent Done Done false
1089 1235 Acked Read 1 Sent Done Done false
1090 1236 Acked Read 4 Sent Done Done false
1091 1237 Acked Read 2 Sent Done Done false
1092 1238 Acked Write 10 Sent Done Done false
1093 1239 Acked Flush 0 Sent Done Done false
1094 1240 Acked Read 5 Sent Done Done false
1095 1241 Acked Read 10 Sent Done Done false
1096 1242 Acked Read 6 Sent Done Done false
1097 1243 Acked Read 9 Sent Done Done false
1098 1244 Acked Read 2 Sent Done Done false
1099 1245 Acked Read 10 Sent Done Done false
1100 1246 Acked Read 5 Sent Done Done false
1101 1247 Acked Read 7 Sent Done Done false
1102 1248 Acked Write 4 Sent Done Done false
1103 1249 Acked Read 7 Sent Done Done false
1104 1250 Acked Read 8 Sent Done Done false
1105 1251 Acked Write 10 Sent Done Done false
1106 1252 Acked Write 1 Sent Done Done false
1107 1253 Acked Read 10 Sent Done Done false
1108 1254 Acked Flush 0 Sent Done Done false
1109 1255 Acked Read 6 Sent Done Done false
1110 1256 Acked Read 10 Sent Done Done false
1111 1257 Acked Write 5 Sent Done Done false
1112 1258 Acked Read 1 Sent Done Done false
1113 1259 Acked Write 10 Sent Done Done false
1114 1260 Acked Write 1 Sent Done Done false
1115 1261 Acked Write 10 Sent Done Done false
1116 1262 Acked Write 4 Sent Done Done false
1117 1263 Acked Read 3 Sent Done Done false
1118 1264 Acked Write 5 Sent Done Done false
1119 1265 Acked Read 10 Sent Done Done false
1120 1266 Acked Read 1 Sent Done Done false
1121 1267 Acked Write 1 Sent Done Done false
1122 1268 Acked Read 5 Sent Done Done false
1123 1269 Acked Write 10 Sent Done Done false
1124 1270 Acked Write 4 Sent Done Done false
1125 1271 Acked Read 8 Sent Done Done false
1126 1272 Acked Read 3 Sent Done Done false
1127 1273 Acked Write 7 Sent Done Done false
1128 1274 Acked Write 8 Sent Done Done false
1129 1275 Acked Read 1 Sent Done Done false
1130 1276 Acked Read 4 Sent Done Done false
1131 1277 Acked Write 1 Sent Done Done false
1132 1278 Acked Write 7 Sent Done Done false
1133 1279 Acked Write 9 Sent Done Done false
1134 1280 Acked Write 4 Sent Done Done false
1135 1281 Acked Read 6 Sent Done Done false
1136 1282 Acked Write 5 Sent Done Done false
1137 1283 Acked Write 1 Sent Done Done false
1138 1284 Acked Flush 0 Sent Done Done false
1139 1285 Acked Write 4 Sent Done Done false
1140 1286 Acked Read 4 Sent Done Done false
1141 1287 Acked Read 7 Sent Done Done false
1142 1288 Acked Write 10 Sent Done Done false
1143 1289 Acked Write 3 Sent Done Done false
1144 1290 Acked Flush 0 Sent Done Done false
1145 1291 Acked Read 2 Sent Done Done false
1146 1292 Acked Read 10 Sent Done Done false
1147 1293 Acked Read 6 Sent Done Done false
1148 1294 Acked Flush 0 Sent Done Done false
1149 1295 Acked Write 9 Sent Done Done false
1150 1296 Acked Write 4 Sent Done Done false
1151 1297 Acked Write 10 Sent Done Done false
1152 1298 Acked Write 3 Sent Done Done false
1153 1299 Acked Read 6 Sent Done Done false
1154 1300 Acked Flush 0 Sent Done Done false
1155 STATES DS:0 DS:1 DS:2 TOTAL
1156 Sent 301 0 0 301
1157 Done 0 301 301 602
1158 Skipped 0 0 0 0
1159 Error 0 0 0 0
1160 Last Flush: None Some(JobId(1300)) Some(JobId(1300))
1161 Downstairs last five completed:
1162 Upstairs last five completed: 1300 1299 1298 1297 1296
1163 CLIENT: Up:0 ds:301 act:2
11642025-03-07T18:12:45.238ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11652025-03-07T18:12:45.964ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11662025-03-07T18:12:46.486ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11672025-03-07T18:12:46.486ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11682025-03-07T18:12:46.486ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11692025-03-07T18:12:46.486ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11702025-03-07T18:12:46.487ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1171 [8810] got start action
1172 [8810] I am going to restart
1173 Starting downstairs at port 8810
1174 Make output file at "/tmp/dsc/downstairs-8810.txt"
1175 [8810][0] reports Starting
1176 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1209)
1177 [8810][0] reports Running
11782025-03-07T18:12:46.988ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11792025-03-07T18:12:47.968ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1180 ----------------------------------------------------------------
1181 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1182 States: Offline Active Active
1183 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1184 1000 Acked Read 9 Sent Done Done false
1185 1001 Acked Flush 0 Sent Done Done false
1186 1002 Acked Write 5 Sent Done Done false
1187 1003 Acked Write 5 Sent Done Done false
1188 1004 Acked Write 1 Sent Done Done false
1189 1005 Acked Write 7 Sent Done Done false
1190 1006 Acked Flush 0 Sent Done Done false
1191 1007 Acked Write 5 Sent Done Done false
1192 1008 Acked Read 10 Sent Done Done false
1193 1009 Acked Write 6 Sent Done Done false
1194 1010 Acked Write 8 Sent Done Done false
1195 1011 Acked Read 5 Sent Done Done false
1196 1012 Acked Write 8 Sent Done Done false
1197 1013 Acked Write 4 Sent Done Done false
1198 1014 Acked Read 4 Sent Done Done false
1199 1015 Acked Read 4 Sent Done Done false
1200 1016 Acked Read 8 Sent Done Done false
1201 1017 Acked Write 3 Sent Done Done false
1202 1018 Acked Write 9 Sent Done Done false
1203 1019 Acked Read 4 Sent Done Done false
1204 1020 Acked Read 2 Sent Done Done false
1205 1021 Acked Write 3 Sent Done Done false
1206 1022 Acked Write 10 Sent Done Done false
1207 1023 Acked Read 5 Sent Done Done false
1208 1024 Acked Read 8 Sent Done Done false
1209 1025 Acked Write 4 Sent Done Done false
1210 1026 Acked Write 1 Sent Done Done false
1211 1027 Acked Read 6 Sent Done Done false
1212 1028 Acked Read 1 Sent Done Done false
1213 1029 Acked Read 8 Sent Done Done false
1214 1030 Acked Write 3 Sent Done Done false
1215 1031 Acked Write 9 Sent Done Done false
1216 1032 Acked Flush 0 Sent Done Done false
1217 1033 Acked Read 9 Sent Done Done false
1218 1034 Acked Flush 0 Sent Done Done false
1219 1035 Acked Read 8 Sent Done Done false
1220 1036 Acked Read 6 Sent Done Done false
1221 1037 Acked Write 1 Sent Done Done false
1222 1038 Acked Write 7 Sent Done Done false
1223 1039 Acked Read 9 Sent Done Done false
1224 1040 Acked Write 8 Sent Done Done false
1225 1041 Acked Read 4 Sent Done Done false
1226 1042 Acked Write 9 Sent Done Done false
1227 1043 Acked Read 6 Sent Done Done false
1228 1044 Acked Read 4 Sent Done Done false
1229 1045 Acked Write 3 Sent Done Done false
1230 1046 Acked Flush 0 Sent Done Done false
1231 1047 Acked Read 3 Sent Done Done false
1232 1048 Acked Read 5 Sent Done Done false
1233 1049 Acked Write 7 Sent Done Done false
1234 1050 Acked Flush 0 Sent Done Done false
1235 1051 Acked Write 6 Sent Done Done false
1236 1052 Acked Flush 0 Sent Done Done false
1237 1053 Acked Write 3 Sent Done Done false
1238 1054 Acked Write 3 Sent Done Done false
1239 1055 Acked Write 2 Sent Done Done false
1240 1056 Acked Read 1 Sent Done Done false
1241 1057 Acked Flush 0 Sent Done Done false
1242 1058 Acked Read 3 Sent Done Done false
1243 1059 Acked Read 4 Sent Done Done false
1244 1060 Acked Write 7 Sent Done Done false
1245 1061 Acked Write 1 Sent Done Done false
1246 1062 Acked Read 7 Sent Done Done false
1247 1063 Acked Write 5 Sent Done Done false
1248 1064 Acked Write 6 Sent Done Done false
1249 1065 Acked Write 3 Sent Done Done false
1250 1066 Acked Write 2 Sent Done Done false
1251 1067 Acked Write 6 Sent Done Done false
1252 1068 Acked Flush 0 Sent Done Done false
1253 1069 Acked Read 7 Sent Done Done false
1254 1070 Acked Read 2 Sent Done Done false
1255 1071 Acked Read 7 Sent Done Done false
1256 1072 Acked Write 10 Sent Done Done false
1257 1073 Acked Write 9 Sent Done Done false
1258 1074 Acked Write 7 Sent Done Done false
1259 1075 Acked Write 4 Sent Done Done false
1260 1076 Acked Read 2 Sent Done Done false
1261 1077 Acked Read 7 Sent Done Done false
1262 1078 Acked Read 2 Sent Done Done false
1263 1079 Acked Write 7 Sent Done Done false
1264 1080 Acked Write 1 Sent Done Done false
1265 1081 Acked Read 9 Sent Done Done false
1266 1082 Acked Read 8 Sent Done Done false
1267 1083 Acked Read 9 Sent Done Done false
1268 1084 Acked Write 4 Sent Done Done false
1269 1085 Acked Read 4 Sent Done Done false
1270 1086 Acked Write 8 Sent Done Done false
1271 1087 Acked Read 10 Sent Done Done false
1272 1088 Acked Read 2 Sent Done Done false
1273 1089 Acked Write 10 Sent Done Done false
1274 1090 Acked Write 7 Sent Done Done false
1275 1091 Acked Flush 0 Sent Done Done false
1276 1092 Acked Read 10 Sent Done Done false
1277 1093 Acked Write 8 Sent Done Done false
1278 1094 Acked Read 8 Sent Done Done false
1279 1095 Acked Write 9 Sent Done Done false
1280 1096 Acked Read 8 Sent Done Done false
1281 1097 Acked Read 6 Sent Done Done false
1282 1098 Acked Read 2 Sent Done Done false
1283 1099 Acked Flush 0 Sent Done Done false
1284 1100 Acked Read 7 Sent Done Done false
1285 1101 Acked Read 4 Sent Done Done false
1286 1102 Acked Read 2 Sent Done Done false
1287 1103 Acked Write 8 Sent Done Done false
1288 1104 Acked Read 9 Sent Done Done false
1289 1105 Acked Write 3 Sent Done Done false
1290 1106 Acked Write 3 Sent Done Done false
1291 1107 Acked Flush 0 Sent Done Done false
1292 1108 Acked Write 10 Sent Done Done false
1293 1109 Acked Read 9 Sent Done Done false
1294 1110 Acked Read 8 Sent Done Done false
1295 1111 Acked Read 6 Sent Done Done false
1296 1112 Acked Read 2 Sent Done Done false
1297 1113 Acked Read 4 Sent Done Done false
1298 1114 Acked Read 2 Sent Done Done false
1299 1115 Acked Read 8 Sent Done Done false
1300 1116 Acked Read 9 Sent Done Done false
1301 1117 Acked Write 3 Sent Done Done false
1302 1118 Acked Write 5 Sent Done Done false
1303 1119 Acked Read 10 Sent Done Done false
1304 1120 Acked Read 7 Sent Done Done false
1305 1121 Acked Write 4 Sent Done Done false
1306 1122 Acked Read 5 Sent Done Done false
1307 1123 Acked Write 4 Sent Done Done false
1308 1124 Acked Read 7 Sent Done Done false
1309 1125 Acked Write 4 Sent Done Done false
1310 1126 Acked Read 7 Sent Done Done false
1311 1127 Acked Read 1 Sent Done Done false
1312 1128 Acked Write 2 Sent Done Done false
1313 1129 Acked Write 9 Sent Done Done false
1314 1130 Acked Read 5 Sent Done Done false
1315 1131 Acked Write 6 Sent Done Done false
1316 1132 Acked Read 6 Sent Done Done false
1317 1133 Acked Flush 0 Sent Done Done false
1318 1134 Acked Write 6 Sent Done Done false
1319 1135 Acked Read 2 Sent Done Done false
1320 1136 Acked Read 6 Sent Done Done false
1321 1137 Acked Write 5 Sent Done Done false
1322 1138 Acked Write 1 Sent Done Done false
1323 1139 Acked Read 1 Sent Done Done false
1324 1140 Acked Write 6 Sent Done Done false
1325 1141 Acked Write 3 Sent Done Done false
1326 1142 Acked Read 8 Sent Done Done false
1327 1143 Acked Read 6 Sent Done Done false
1328 1144 Acked Read 2 Sent Done Done false
1329 1145 Acked Read 6 Sent Done Done false
1330 1146 Acked Write 4 Sent Done Done false
1331 1147 Acked Write 7 Sent Done Done false
1332 1148 Acked Read 2 Sent Done Done false
1333 1149 Acked Read 2 Sent Done Done false
1334 1150 Acked Write 8 Sent Done Done false
1335 1151 Acked Read 9 Sent Done Done false
1336 1152 Acked Read 9 Sent Done Done false
1337 1153 Acked Read 7 Sent Done Done false
1338 1154 Acked Write 2 Sent Done Done false
1339 1155 Acked Read 4 Sent Done Done false
1340 1156 Acked Write 6 Sent Done Done false
1341 1157 Acked Flush 0 Sent Done Done false
1342 1158 Acked Write 7 Sent Done Done false
1343 1159 Acked Write 4 Sent Done Done false
1344 1160 Acked Write 9 Sent Done Done false
1345 1161 Acked Write 7 Sent Done Done false
1346 1162 Acked Write 5 Sent Done Done false
1347 1163 Acked Read 8 Sent Done Done false
1348 1164 Acked Read 3 Sent Done Done false
1349 1165 Acked Write 10 Sent Done Done false
1350 1166 Acked Write 5 Sent Done Done false
1351 1167 Acked Read 7 Sent Done Done false
1352 1168 Acked Read 8 Sent Done Done false
1353 1169 Acked Write 10 Sent Done Done false
1354 1170 Acked Read 9 Sent Done Done false
1355 1171 Acked Write 2 Sent Done Done false
1356 1172 Acked Write 8 Sent Done Done false
1357 1173 Acked Write 10 Sent Done Done false
1358 1174 Acked Write 1 Sent Done Done false
1359 1175 Acked Read 6 Sent Done Done false
1360 1176 Acked Write 8 Sent Done Done false
1361 1177 Acked Write 6 Sent Done Done false
1362 1178 Acked Read 2 Sent Done Done false
1363 1179 Acked Read 7 Sent Done Done false
1364 1180 Acked Write 7 Sent Done Done false
1365 1181 Acked Write 4 Sent Done Done false
1366 1182 Acked Read 7 Sent Done Done false
1367 1183 Acked Write 4 Sent Done Done false
1368 1184 Acked Read 1 Sent Done Done false
1369 1185 Acked Flush 0 Sent Done Done false
1370 1186 Acked Read 1 Sent Done Done false
1371 1187 Acked Read 2 Sent Done Done false
1372 1188 Acked Read 10 Sent Done Done false
1373 1189 Acked Flush 0 Sent Done Done false
1374 1190 Acked Read 7 Sent Done Done false
1375 1191 Acked Write 4 Sent Done Done false
1376 1192 Acked Read 8 Sent Done Done false
1377 1193 Acked Flush 0 Sent Done Done false
1378 1194 Acked Write 5 Sent Done Done false
1379 1195 Acked Write 2 Sent Done Done false
1380 1196 Acked Read 10 Sent Done Done false
1381 1197 Acked Read 3 Sent Done Done false
1382 1198 Acked Write 8 Sent Done Done false
1383 1199 Acked Write 6 Sent Done Done false
1384 1200 Acked Flush 0 Sent Done Done false
1385 1201 Acked Read 5 Sent Done Done false
1386 1202 Acked Write 10 Sent Done Done false
1387 1203 Acked Write 8 Sent Done Done false
1388 1204 Acked Flush 0 Sent Done Done false
1389 1205 Acked Write 8 Sent Done Done false
1390 1206 Acked Flush 0 Sent Done Done false
1391 1207 Acked Flush 0 Sent Done Done false
1392 1208 Acked Read 6 Sent Done Done false
1393 1209 Acked Write 7 Sent Done Done false
1394 1210 Acked Read 6 Sent Done Done false
1395 1211 Acked Flush 0 Sent Done Done false
1396 1212 Acked Read 3 Sent Done Done false
1397 1213 Acked Write 6 Sent Done Done false
1398 1214 Acked Read 2 Sent Done Done false
1399 1215 Acked Flush 0 Sent Done Done false
1400 1216 Acked Write 10 Sent Done Done false
1401 1217 Acked Read 4 Sent Done Done false
1402 1218 Acked Write 1 Sent Done Done false
1403 1219 Acked Read 8 Sent Done Done false
1404 1220 Acked Read 2 Sent Done Done false
1405 1221 Acked Flush 0 Sent Done Done false
1406 1222 Acked Read 4 Sent Done Done false
1407 1223 Acked Write 5 Sent Done Done false
1408 1224 Acked Write 10 Sent Done Done false
1409 1225 Acked Read 7 Sent Done Done false
1410 1226 Acked Read 6 Sent Done Done false
1411 1227 Acked Read 1 Sent Done Done false
1412 1228 Acked Write 5 Sent Done Done false
1413 1229 Acked Read 9 Sent Done Done false
1414 1230 Acked Read 8 Sent Done Done false
1415 1231 Acked Read 7 Sent Done Done false
1416 1232 Acked Read 7 Sent Done Done false
1417 1233 Acked Read 9 Sent Done Done false
1418 1234 Acked Read 8 Sent Done Done false
1419 1235 Acked Read 1 Sent Done Done false
1420 1236 Acked Read 4 Sent Done Done false
1421 1237 Acked Read 2 Sent Done Done false
1422 1238 Acked Write 10 Sent Done Done false
1423 1239 Acked Flush 0 Sent Done Done false
1424 1240 Acked Read 5 Sent Done Done false
1425 1241 Acked Read 10 Sent Done Done false
1426 1242 Acked Read 6 Sent Done Done false
1427 1243 Acked Read 9 Sent Done Done false
1428 1244 Acked Read 2 Sent Done Done false
1429 1245 Acked Read 10 Sent Done Done false
1430 1246 Acked Read 5 Sent Done Done false
1431 1247 Acked Read 7 Sent Done Done false
1432 1248 Acked Write 4 Sent Done Done false
1433 1249 Acked Read 7 Sent Done Done false
1434 1250 Acked Read 8 Sent Done Done false
1435 1251 Acked Write 10 Sent Done Done false
1436 1252 Acked Write 1 Sent Done Done false
1437 1253 Acked Read 10 Sent Done Done false
1438 1254 Acked Flush 0 Sent Done Done false
1439 1255 Acked Read 6 Sent Done Done false
1440 1256 Acked Read 10 Sent Done Done false
1441 1257 Acked Write 5 Sent Done Done false
1442 1258 Acked Read 1 Sent Done Done false
1443 1259 Acked Write 10 Sent Done Done false
1444 1260 Acked Write 1 Sent Done Done false
1445 1261 Acked Write 10 Sent Done Done false
1446 1262 Acked Write 4 Sent Done Done false
1447 1263 Acked Read 3 Sent Done Done false
1448 1264 Acked Write 5 Sent Done Done false
1449 1265 Acked Read 10 Sent Done Done false
1450 1266 Acked Read 1 Sent Done Done false
1451 1267 Acked Write 1 Sent Done Done false
1452 1268 Acked Read 5 Sent Done Done false
1453 1269 Acked Write 10 Sent Done Done false
1454 1270 Acked Write 4 Sent Done Done false
1455 1271 Acked Read 8 Sent Done Done false
1456 1272 Acked Read 3 Sent Done Done false
1457 1273 Acked Write 7 Sent Done Done false
1458 1274 Acked Write 8 Sent Done Done false
1459 1275 Acked Read 1 Sent Done Done false
1460 1276 Acked Read 4 Sent Done Done false
1461 1277 Acked Write 1 Sent Done Done false
1462 1278 Acked Write 7 Sent Done Done false
1463 1279 Acked Write 9 Sent Done Done false
1464 1280 Acked Write 4 Sent Done Done false
1465 1281 Acked Read 6 Sent Done Done false
1466 1282 Acked Write 5 Sent Done Done false
1467 1283 Acked Write 1 Sent Done Done false
1468 1284 Acked Flush 0 Sent Done Done false
1469 1285 Acked Write 4 Sent Done Done false
1470 1286 Acked Read 4 Sent Done Done false
1471 1287 Acked Read 7 Sent Done Done false
1472 1288 Acked Write 10 Sent Done Done false
1473 1289 Acked Write 3 Sent Done Done false
1474 1290 Acked Flush 0 Sent Done Done false
1475 1291 Acked Read 2 Sent Done Done false
1476 1292 Acked Read 10 Sent Done Done false
1477 1293 Acked Read 6 Sent Done Done false
1478 1294 Acked Flush 0 Sent Done Done false
1479 1295 Acked Write 9 Sent Done Done false
1480 1296 Acked Write 4 Sent Done Done false
1481 1297 Acked Write 10 Sent Done Done false
1482 1298 Acked Write 3 Sent Done Done false
1483 1299 Acked Read 6 Sent Done Done false
1484 1300 Acked Flush 0 Sent Done Done false
1485 STATES DS:0 DS:1 DS:2 TOTAL
1486 Sent 301 0 0 301
1487 Done 0 301 301 602
1488 Skipped 0 0 0 0
1489 Error 0 0 0 0
1490 Last Flush: None Some(JobId(1300)) Some(JobId(1300))
1491 Downstairs last five completed:
1492 Upstairs last five completed: 1300 1299 1298 1297 1296
1493 CLIENT: Up:0 ds:301 act:2
14942025-03-07T18:12:49.240ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14952025-03-07T18:12:49.971ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14962025-03-07T18:12:50.973ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14972025-03-07T18:12:51.974ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1498 ----------------------------------------------------------------
1499 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1500 States: Offline Active Active
1501 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1502 1000 Acked Read 9 Sent Done Done false
1503 1001 Acked Flush 0 Sent Done Done false
1504 1002 Acked Write 5 Sent Done Done false
1505 1003 Acked Write 5 Sent Done Done false
1506 1004 Acked Write 1 Sent Done Done false
1507 1005 Acked Write 7 Sent Done Done false
1508 1006 Acked Flush 0 Sent Done Done false
1509 1007 Acked Write 5 Sent Done Done false
1510 1008 Acked Read 10 Sent Done Done false
1511 1009 Acked Write 6 Sent Done Done false
1512 1010 Acked Write 8 Sent Done Done false
1513 1011 Acked Read 5 Sent Done Done false
1514 1012 Acked Write 8 Sent Done Done false
1515 1013 Acked Write 4 Sent Done Done false
1516 1014 Acked Read 4 Sent Done Done false
1517 1015 Acked Read 4 Sent Done Done false
1518 1016 Acked Read 8 Sent Done Done false
1519 1017 Acked Write 3 Sent Done Done false
1520 1018 Acked Write 9 Sent Done Done false
1521 1019 Acked Read 4 Sent Done Done false
1522 1020 Acked Read 2 Sent Done Done false
1523 1021 Acked Write 3 Sent Done Done false
1524 1022 Acked Write 10 Sent Done Done false
1525 1023 Acked Read 5 Sent Done Done false
1526 1024 Acked Read 8 Sent Done Done false
1527 1025 Acked Write 4 Sent Done Done false
1528 1026 Acked Write 1 Sent Done Done false
1529 1027 Acked Read 6 Sent Done Done false
1530 1028 Acked Read 1 Sent Done Done false
1531 1029 Acked Read 8 Sent Done Done false
1532 1030 Acked Write 3 Sent Done Done false
1533 1031 Acked Write 9 Sent Done Done false
1534 1032 Acked Flush 0 Sent Done Done false
1535 1033 Acked Read 9 Sent Done Done false
1536 1034 Acked Flush 0 Sent Done Done false
1537 1035 Acked Read 8 Sent Done Done false
1538 1036 Acked Read 6 Sent Done Done false
1539 1037 Acked Write 1 Sent Done Done false
1540 1038 Acked Write 7 Sent Done Done false
1541 1039 Acked Read 9 Sent Done Done false
1542 1040 Acked Write 8 Sent Done Done false
1543 1041 Acked Read 4 Sent Done Done false
1544 1042 Acked Write 9 Sent Done Done false
1545 1043 Acked Read 6 Sent Done Done false
1546 1044 Acked Read 4 Sent Done Done false
1547 1045 Acked Write 3 Sent Done Done false
1548 1046 Acked Flush 0 Sent Done Done false
1549 1047 Acked Read 3 Sent Done Done false
1550 1048 Acked Read 5 Sent Done Done false
1551 1049 Acked Write 7 Sent Done Done false
1552 1050 Acked Flush 0 Sent Done Done false
1553 1051 Acked Write 6 Sent Done Done false
1554 1052 Acked Flush 0 Sent Done Done false
1555 1053 Acked Write 3 Sent Done Done false
1556 1054 Acked Write 3 Sent Done Done false
1557 1055 Acked Write 2 Sent Done Done false
1558 1056 Acked Read 1 Sent Done Done false
1559 1057 Acked Flush 0 Sent Done Done false
1560 1058 Acked Read 3 Sent Done Done false
1561 1059 Acked Read 4 Sent Done Done false
1562 1060 Acked Write 7 Sent Done Done false
1563 1061 Acked Write 1 Sent Done Done false
1564 1062 Acked Read 7 Sent Done Done false
1565 1063 Acked Write 5 Sent Done Done false
1566 1064 Acked Write 6 Sent Done Done false
1567 1065 Acked Write 3 Sent Done Done false
1568 1066 Acked Write 2 Sent Done Done false
1569 1067 Acked Write 6 Sent Done Done false
1570 1068 Acked Flush 0 Sent Done Done false
1571 1069 Acked Read 7 Sent Done Done false
1572 1070 Acked Read 2 Sent Done Done false
1573 1071 Acked Read 7 Sent Done Done false
1574 1072 Acked Write 10 Sent Done Done false
1575 1073 Acked Write 9 Sent Done Done false
1576 1074 Acked Write 7 Sent Done Done false
1577 1075 Acked Write 4 Sent Done Done false
1578 1076 Acked Read 2 Sent Done Done false
1579 1077 Acked Read 7 Sent Done Done false
1580 1078 Acked Read 2 Sent Done Done false
1581 1079 Acked Write 7 Sent Done Done false
1582 1080 Acked Write 1 Sent Done Done false
1583 1081 Acked Read 9 Sent Done Done false
1584 1082 Acked Read 8 Sent Done Done false
1585 1083 Acked Read 9 Sent Done Done false
1586 1084 Acked Write 4 Sent Done Done false
1587 1085 Acked Read 4 Sent Done Done false
1588 1086 Acked Write 8 Sent Done Done false
1589 1087 Acked Read 10 Sent Done Done false
1590 1088 Acked Read 2 Sent Done Done false
1591 1089 Acked Write 10 Sent Done Done false
1592 1090 Acked Write 7 Sent Done Done false
1593 1091 Acked Flush 0 Sent Done Done false
1594 1092 Acked Read 10 Sent Done Done false
1595 1093 Acked Write 8 Sent Done Done false
1596 1094 Acked Read 8 Sent Done Done false
1597 1095 Acked Write 9 Sent Done Done false
1598 1096 Acked Read 8 Sent Done Done false
1599 1097 Acked Read 6 Sent Done Done false
1600 1098 Acked Read 2 Sent Done Done false
1601 1099 Acked Flush 0 Sent Done Done false
1602 1100 Acked Read 7 Sent Done Done false
1603 1101 Acked Read 4 Sent Done Done false
1604 1102 Acked Read 2 Sent Done Done false
1605 1103 Acked Write 8 Sent Done Done false
1606 1104 Acked Read 9 Sent Done Done false
1607 1105 Acked Write 3 Sent Done Done false
1608 1106 Acked Write 3 Sent Done Done false
1609 1107 Acked Flush 0 Sent Done Done false
1610 1108 Acked Write 10 Sent Done Done false
1611 1109 Acked Read 9 Sent Done Done false
1612 1110 Acked Read 8 Sent Done Done false
1613 1111 Acked Read 6 Sent Done Done false
1614 1112 Acked Read 2 Sent Done Done false
1615 1113 Acked Read 4 Sent Done Done false
1616 1114 Acked Read 2 Sent Done Done false
1617 1115 Acked Read 8 Sent Done Done false
1618 1116 Acked Read 9 Sent Done Done false
1619 1117 Acked Write 3 Sent Done Done false
1620 1118 Acked Write 5 Sent Done Done false
1621 1119 Acked Read 10 Sent Done Done false
1622 1120 Acked Read 7 Sent Done Done false
1623 1121 Acked Write 4 Sent Done Done false
1624 1122 Acked Read 5 Sent Done Done false
1625 1123 Acked Write 4 Sent Done Done false
1626 1124 Acked Read 7 Sent Done Done false
1627 1125 Acked Write 4 Sent Done Done false
1628 1126 Acked Read 7 Sent Done Done false
1629 1127 Acked Read 1 Sent Done Done false
1630 1128 Acked Write 2 Sent Done Done false
1631 1129 Acked Write 9 Sent Done Done false
1632 1130 Acked Read 5 Sent Done Done false
1633 1131 Acked Write 6 Sent Done Done false
1634 1132 Acked Read 6 Sent Done Done false
1635 1133 Acked Flush 0 Sent Done Done false
1636 1134 Acked Write 6 Sent Done Done false
1637 1135 Acked Read 2 Sent Done Done false
1638 1136 Acked Read 6 Sent Done Done false
1639 1137 Acked Write 5 Sent Done Done false
1640 1138 Acked Write 1 Sent Done Done false
1641 1139 Acked Read 1 Sent Done Done false
1642 1140 Acked Write 6 Sent Done Done false
1643 1141 Acked Write 3 Sent Done Done false
1644 1142 Acked Read 8 Sent Done Done false
1645 1143 Acked Read 6 Sent Done Done false
1646 1144 Acked Read 2 Sent Done Done false
1647 1145 Acked Read 6 Sent Done Done false
1648 1146 Acked Write 4 Sent Done Done false
1649 1147 Acked Write 7 Sent Done Done false
1650 1148 Acked Read 2 Sent Done Done false
1651 1149 Acked Read 2 Sent Done Done false
1652 1150 Acked Write 8 Sent Done Done false
1653 1151 Acked Read 9 Sent Done Done false
1654 1152 Acked Read 9 Sent Done Done false
1655 1153 Acked Read 7 Sent Done Done false
1656 1154 Acked Write 2 Sent Done Done false
1657 1155 Acked Read 4 Sent Done Done false
1658 1156 Acked Write 6 Sent Done Done false
1659 1157 Acked Flush 0 Sent Done Done false
1660 1158 Acked Write 7 Sent Done Done false
1661 1159 Acked Write 4 Sent Done Done false
1662 1160 Acked Write 9 Sent Done Done false
1663 1161 Acked Write 7 Sent Done Done false
1664 1162 Acked Write 5 Sent Done Done false
1665 1163 Acked Read 8 Sent Done Done false
1666 1164 Acked Read 3 Sent Done Done false
1667 1165 Acked Write 10 Sent Done Done false
1668 1166 Acked Write 5 Sent Done Done false
1669 1167 Acked Read 7 Sent Done Done false
1670 1168 Acked Read 8 Sent Done Done false
1671 1169 Acked Write 10 Sent Done Done false
1672 1170 Acked Read 9 Sent Done Done false
1673 1171 Acked Write 2 Sent Done Done false
1674 1172 Acked Write 8 Sent Done Done false
1675 1173 Acked Write 10 Sent Done Done false
1676 1174 Acked Write 1 Sent Done Done false
1677 1175 Acked Read 6 Sent Done Done false
1678 1176 Acked Write 8 Sent Done Done false
1679 1177 Acked Write 6 Sent Done Done false
1680 1178 Acked Read 2 Sent Done Done false
1681 1179 Acked Read 7 Sent Done Done false
1682 1180 Acked Write 7 Sent Done Done false
1683 1181 Acked Write 4 Sent Done Done false
1684 1182 Acked Read 7 Sent Done Done false
1685 1183 Acked Write 4 Sent Done Done false
1686 1184 Acked Read 1 Sent Done Done false
1687 1185 Acked Flush 0 Sent Done Done false
1688 1186 Acked Read 1 Sent Done Done false
1689 1187 Acked Read 2 Sent Done Done false
1690 1188 Acked Read 10 Sent Done Done false
1691 1189 Acked Flush 0 Sent Done Done false
1692 1190 Acked Read 7 Sent Done Done false
1693 1191 Acked Write 4 Sent Done Done false
1694 1192 Acked Read 8 Sent Done Done false
1695 1193 Acked Flush 0 Sent Done Done false
1696 1194 Acked Write 5 Sent Done Done false
1697 1195 Acked Write 2 Sent Done Done false
1698 1196 Acked Read 10 Sent Done Done false
1699 1197 Acked Read 3 Sent Done Done false
1700 1198 Acked Write 8 Sent Done Done false
1701 1199 Acked Write 6 Sent Done Done false
1702 1200 Acked Flush 0 Sent Done Done false
1703 1201 Acked Read 5 Sent Done Done false
1704 1202 Acked Write 10 Sent Done Done false
1705 1203 Acked Write 8 Sent Done Done false
1706 1204 Acked Flush 0 Sent Done Done false
1707 1205 Acked Write 8 Sent Done Done false
1708 1206 Acked Flush 0 Sent Done Done false
1709 1207 Acked Flush 0 Sent Done Done false
1710 1208 Acked Read 6 Sent Done Done false
1711 1209 Acked Write 7 Sent Done Done false
1712 1210 Acked Read 6 Sent Done Done false
1713 1211 Acked Flush 0 Sent Done Done false
1714 1212 Acked Read 3 Sent Done Done false
1715 1213 Acked Write 6 Sent Done Done false
1716 1214 Acked Read 2 Sent Done Done false
1717 1215 Acked Flush 0 Sent Done Done false
1718 1216 Acked Write 10 Sent Done Done false
1719 1217 Acked Read 4 Sent Done Done false
1720 1218 Acked Write 1 Sent Done Done false
1721 1219 Acked Read 8 Sent Done Done false
1722 1220 Acked Read 2 Sent Done Done false
1723 1221 Acked Flush 0 Sent Done Done false
1724 1222 Acked Read 4 Sent Done Done false
1725 1223 Acked Write 5 Sent Done Done false
1726 1224 Acked Write 10 Sent Done Done false
1727 1225 Acked Read 7 Sent Done Done false
1728 1226 Acked Read 6 Sent Done Done false
1729 1227 Acked Read 1 Sent Done Done false
1730 1228 Acked Write 5 Sent Done Done false
1731 1229 Acked Read 9 Sent Done Done false
1732 1230 Acked Read 8 Sent Done Done false
1733 1231 Acked Read 7 Sent Done Done false
1734 1232 Acked Read 7 Sent Done Done false
1735 1233 Acked Read 9 Sent Done Done false
1736 1234 Acked Read 8 Sent Done Done false
1737 1235 Acked Read 1 Sent Done Done false
1738 1236 Acked Read 4 Sent Done Done false
1739 1237 Acked Read 2 Sent Done Done false
1740 1238 Acked Write 10 Sent Done Done false
1741 1239 Acked Flush 0 Sent Done Done false
1742 1240 Acked Read 5 Sent Done Done false
1743 1241 Acked Read 10 Sent Done Done false
1744 1242 Acked Read 6 Sent Done Done false
1745 1243 Acked Read 9 Sent Done Done false
1746 1244 Acked Read 2 Sent Done Done false
1747 1245 Acked Read 10 Sent Done Done false
1748 1246 Acked Read 5 Sent Done Done false
1749 1247 Acked Read 7 Sent Done Done false
1750 1248 Acked Write 4 Sent Done Done false
1751 1249 Acked Read 7 Sent Done Done false
1752 1250 Acked Read 8 Sent Done Done false
1753 1251 Acked Write 10 Sent Done Done false
1754 1252 Acked Write 1 Sent Done Done false
1755 1253 Acked Read 10 Sent Done Done false
1756 1254 Acked Flush 0 Sent Done Done false
1757 1255 Acked Read 6 Sent Done Done false
1758 1256 Acked Read 10 Sent Done Done false
1759 1257 Acked Write 5 Sent Done Done false
1760 1258 Acked Read 1 Sent Done Done false
1761 1259 Acked Write 10 Sent Done Done false
1762 1260 Acked Write 1 Sent Done Done false
1763 1261 Acked Write 10 Sent Done Done false
1764 1262 Acked Write 4 Sent Done Done false
1765 1263 Acked Read 3 Sent Done Done false
1766 1264 Acked Write 5 Sent Done Done false
1767 1265 Acked Read 10 Sent Done Done false
1768 1266 Acked Read 1 Sent Done Done false
1769 1267 Acked Write 1 Sent Done Done false
1770 1268 Acked Read 5 Sent Done Done false
1771 1269 Acked Write 10 Sent Done Done false
1772 1270 Acked Write 4 Sent Done Done false
1773 1271 Acked Read 8 Sent Done Done false
1774 1272 Acked Read 3 Sent Done Done false
1775 1273 Acked Write 7 Sent Done Done false
1776 1274 Acked Write 8 Sent Done Done false
1777 1275 Acked Read 1 Sent Done Done false
1778 1276 Acked Read 4 Sent Done Done false
1779 1277 Acked Write 1 Sent Done Done false
1780 1278 Acked Write 7 Sent Done Done false
1781 1279 Acked Write 9 Sent Done Done false
1782 1280 Acked Write 4 Sent Done Done false
1783 1281 Acked Read 6 Sent Done Done false
1784 1282 Acked Write 5 Sent Done Done false
1785 1283 Acked Write 1 Sent Done Done false
1786 1284 Acked Flush 0 Sent Done Done false
1787 1285 Acked Write 4 Sent Done Done false
1788 1286 Acked Read 4 Sent Done Done false
1789 1287 Acked Read 7 Sent Done Done false
1790 1288 Acked Write 10 Sent Done Done false
1791 1289 Acked Write 3 Sent Done Done false
1792 1290 Acked Flush 0 Sent Done Done false
1793 1291 Acked Read 2 Sent Done Done false
1794 1292 Acked Read 10 Sent Done Done false
1795 1293 Acked Read 6 Sent Done Done false
1796 1294 Acked Flush 0 Sent Done Done false
1797 1295 Acked Write 9 Sent Done Done false
1798 1296 Acked Write 4 Sent Done Done false
1799 1297 Acked Write 10 Sent Done Done false
1800 1298 Acked Write 3 Sent Done Done false
1801 1299 Acked Read 6 Sent Done Done false
1802 1300 Acked Flush 0 Sent Done Done false
1803 STATES DS:0 DS:1 DS:2 TOTAL
1804 Sent 301 0 0 301
1805 Done 0 301 301 602
1806 Skipped 0 0 0 0
1807 Error 0 0 0 0
1808 Last Flush: None Some(JobId(1300)) Some(JobId(1300))
1809 Downstairs last five completed:
1810 Upstairs last five completed: 1300 1299 1298 1297 1296
1811 CLIENT: Up:0 ds:301 act:2
18122025-03-07T18:12:53.243ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18132025-03-07T18:12:53.978ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18142025-03-07T18:12:54.980ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18152025-03-07T18:12:55.983ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18162025-03-07T18:12:56.488ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18172025-03-07T18:12:56.488ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56595) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18182025-03-07T18:12:56.489ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18192025-03-07T18:12:56.490ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18202025-03-07T18:12:56.490ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18212025-03-07T18:12:56.490ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18222025-03-07T18:12:56.490ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18232025-03-07T18:12:56.490ZINFOcrucible: [0] client re-new 301 jobs since flush None = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18242025-03-07T18:12:56.490ZINFOcrucible: [0] Replayed 301 jobs since flush: None = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1825 ----------------------------------------------------------------
1826 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1827 Downstairs last five completed: 1300 1299 1298 1297 1296
1828 Upstairs last five completed: 1300 1299 1298 1297 1296
1829 CLIENT: Up:0 ds:0 act:3
1830 Replay: All jobs finished
1831 Main task has work to do, go find it
1832 got dsc Stop(2)
1833 stop 2
1834 Mar 07 18:12:56.744 INFO request completed, latency_us: 307, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
1835 , req_id: 3d5df832-113a-40ba-84d1-cf3ec165e630, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
18362025-03-07T18:12:56.745ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18372025-03-07T18:12:56.745ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18382025-03-07T18:12:56.745ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18392025-03-07T18:12:56.745ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
1840 Mar 07 18:12:56.745 INFO request completed, latency_us: 173, response_code: 200, uri: /state/cid/2, method: GET, req_id: 78a20655-2117-45dc-a1be-9cb158786c43, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
1841 [8830] Exited with: ExitStatus(unix_wait_status(9))
1842 [8830][2] reports Exit
1843 Mar 07 18:13:00.747 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/2, method: GET, req_id: f74204ce-f120-496a-a834-fb41cd32672b, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
1844 001/300 Read block 1565 len 4096
1845 002/300 Flush
1846 003/300 Read block 1462 len 16384
1847 004/300 Write block 1870 len 16384 data: 2 2 2 2
1848 005/300 Write block 3185 len 20480 data: 2 2 2 2 2
1849 006/300 Read block 708 len 20480
1850 007/300 Write block 2663 len 32768 data: 3 3 3 3 2 2 2 2
1851 008/300 Write block 18 len 32768 data: 2 2 2 2 2 2 2 2
1852 009/300 Read block 1443 len 16384
1853 010/300 Read block 4498 len 32768
1854 011/300 Read block 2487 len 40960
1855 012/300 Read block 3469 len 4096
1856 013/300 Read block 427 len 12288
1857 014/300 Write block 368 len 12288 data: 2 2 2
1858 015/300 Read block 1937 len 8192
1859 016/300 Write block 2436 len 32768 data: 2 3 3 3 3 3 3 2
1860 017/300 Read block 3908 len 8192
1861 018/300 Write block 3530 len 16384 data: 2 2 2 2
1862 019/300 Read block 2407 len 12288
1863 020/300 Read block 3898 len 36864
1864 021/300 Read block 3414 len 40960
1865 022/300 Write block 4505 len 28672 data: 2 2 2 2 2 2 2
1866 023/300 Read block 2558 len 28672
1867 024/300 Flush
1868 025/300 Write block 4017 len 36864 data: 2 2 2 2 2 2 2 2 2
1869 026/300 Write block 4680 len 36864 data: 2 2 2 2 2 2 2 2 2
1870 027/300 Read block 4880 len 12288
1871 028/300 Flush
1872 029/300 Read block 3776 len 40960
1873 030/300 Read block 1791 len 8192
1874 031/300 Write block 1234 len 8192 data: 2 2
1875 032/300 Write block 3793 len 20480 data: 2 2 2 2 2
1876 033/300 Read block 829 len 32768
1877 034/300 Write block 4365 len 8192 data: 2 2
1878 035/300 Read block 4258 len 40960
1879 036/300 Write block 3187 len 24576 data: 3 3 3 2 2 2
1880 037/300 Write block 1189 len 20480 data: 2 2 2 2 2
1881 038/300 Read block 807 len 36864
1882 039/300 Read block 634 len 32768
1883 040/300 Read block 4655 len 20480
1884 041/300 Read block 4139 len 16384
1885 042/300 Write block 3111 len 24576 data: 2 2 2 2 2 2
1886 043/300 Flush
1887 044/300 Read block 1770 len 40960
1888 045/300 Read block 2096 len 40960
1889 046/300 Write block 338 len 36864 data: 2 2 2 2 2 2 2 2 2
1890 047/300 Read block 582 len 8192
1891 048/300 Write block 1576 len 32768 data: 2 2 2 2 2 2 2 2
1892 049/300 Write block 643 len 32768 data: 2 2 2 2 2 2 2 2
1893 050/300 Write block 3508 len 28672 data: 2 2 2 2 2 2 2
1894 051/300 Write block 2494 len 8192 data: 3 2
1895 052/300 Read block 408 len 40960
1896 053/300 Write block 215 len 8192 data: 3 3
1897 054/300 Read block 3726 len 4096
1898 055/300 Read block 3624 len 8192
1899 056/300 Read block 789 len 16384
1900 057/300 Write block 4371 len 28672 data: 2 2 3 3 3 3 3
1901 058/300 Read block 792 len 8192
1902 059/300 Write block 619 len 36864 data: 2 2 2 2 2 2 2 2 2
1903 060/300 Read block 2195 len 40960
1904 061/300 Write block 1539 len 12288 data: 3 3 2
1905 062/300 Read block 1107 len 24576
1906 063/300 Write block 673 len 4096 data: 2
1907 064/300 Read block 2504 len 24576
1908 065/300 Flush
1909 066/300 Write block 4136 len 24576 data: 2 2 2 2 2 2
1910 067/300 Read block 235 len 24576
1911 068/300 Write block 229 len 4096 data: 2
1912 069/300 Read block 2226 len 24576
1913 070/300 Flush
1914 071/300 Flush
1915 072/300 Read block 3200 len 8192
1916 073/300 Write block 170 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1917 074/300 Read block 500 len 12288
1918 075/300 Read block 1361 len 8192
1919 076/300 Flush
1920 077/300 Read block 4003 len 20480
1921 078/300 Read block 3888 len 4096
1922 079/300 Write block 179 len 12288 data: 3 2 2
1923 080/300 Flush
1924 081/300 Read block 4386 len 36864
1925 082/300 Write block 4454 len 16384 data: 2 2 2 2
1926 083/300 Read block 3920 len 36864
1927 084/300 Read block 291 len 20480
1928 085/300 Flush
1929 086/300 Read block 4653 len 32768
1930 087/300 Flush
1931 088/300 Read block 710 len 12288
1932 089/300 Write block 4389 len 12288 data: 2 2 2
1933 090/300 Flush
1934 091/300 Write block 3368 len 8192 data: 2 2
1935 092/300 Read block 4317 len 32768
1936 093/300 Read block 2285 len 36864
1937 094/300 Write block 209 len 20480 data: 2 2 2 3 3
1938 095/300 Read block 4916 len 20480
1939 096/300 Read block 378 len 4096
1940 097/300 Write block 1680 len 8192 data: 2 2
1941 098/300 Write block 4064 len 24576 data: 2 2 2 2 2 2
1942 099/300 Write block 3041 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1943 100/300 Read block 2499 len 28672
1944 101/300 Write block 897 len 4096 data: 2
1945 102/300 Read block 697 len 24576
1946 103/300 Write block 735 len 16384 data: 2 2 2 2
1947 104/300 Read block 4637 len 12288
1948 105/300 Read block 2662 len 32768
1949 106/300 Flush
1950 107/300 Write block 825 len 12288 data: 3 3 3
1951 108/300 Read block 4848 len 16384
1952 109/300 Read block 3784 len 32768
1953 110/300 Read block 4341 len 20480
1954 111/300 Read block 4777 len 4096
1955 112/300 Read block 3048 len 28672
1956 113/300 Write block 1162 len 36864 data: 2 2 2 2 2 2 2 2 2
1957 114/300 Read block 2504 len 24576
1958 115/300 Read block 648 len 40960
1959 116/300 Write block 1979 len 24576 data: 2 2 2 2 2 2
1960 117/300 Write block 3113 len 20480 data: 3 3 3 3 2
1961 118/300 Read block 3804 len 8192
1962 119/300 Write block 1594 len 16384 data: 2 2 2 2
1963 120/300 Read block 3549 len 28672
1964 121/300 Flush
1965 122/300 Read block 720 len 32768
1966 123/300 Read block 3544 len 20480
1967 124/300 Read block 1866 len 20480
1968 125/300 Read block 4419 len 40960
1969 126/300 Flush
1970 127/300 Flush
1971 128/300 Read block 2175 len 36864
1972 129/300 Write block 2752 len 4096 data: 2
1973 130/300 Write block 72 len 32768 data: 2 2 2 2 2 2 2 2
1974 131/300 Read block 914 len 24576
1975 132/300 Read block 1043 len 16384
1976 133/300 Read block 192 len 20480
1977 134/300 Read block 4086 len 24576
1978 135/300 Read block 809 len 28672
1979 136/300 Read block 1425 len 4096
1980 137/300 Read block 131 len 12288
1981 138/300 Flush
1982 139/300 Write block 3605 len 20480 data: 2 2 2 2 2
1983 140/300 Write block 1146 len 12288 data: 2 2 2
1984 141/300 Read block 851 len 20480
1985 142/300 Read block 4367 len 24576
1986 143/300 Flush
1987 144/300 Read block 177 len 36864
1988 145/300 Write block 1600 len 20480 data: 3 3 3 3 3
1989 146/300 Write block 2908 len 12288 data: 2 2 2
1990 147/300 Read block 1135 len 28672
1991 148/300 Write block 2942 len 40960 data: 2 3 3 3 3 3 3 3 3 3
1992 149/300 Write block 1397 len 4096 data: 2
1993 150/300 Write block 509 len 32768 data: 2 2 2 2 2 2 2 2
1994 151/300 Read block 3612 len 36864
1995 152/300 Write block 4691 len 8192 data: 2 2
1996 153/300 Write block 4305 len 32768 data: 2 2 2 2 2 2 2 2
1997 154/300 Read block 360 len 40960
1998 155/300 Read block 1220 len 12288
1999 156/300 Write block 1067 len 28672 data: 2 2 2 2 2 2 2
2000 157/300 Read block 870 len 36864
2001 158/300 Read block 2655 len 24576
2002 159/300 Write block 3073 len 8192 data: 2 2
2003 160/300 Read block 2339 len 20480
2004 161/300 Flush
2005 162/300 Read block 766 len 32768
2006 163/300 Read block 2535 len 20480
2007 164/300 Write block 2914 len 20480 data: 2 2 2 2 2
2008 165/300 Read block 2140 len 20480
2009 166/300 Read block 4975 len 40960
2010 167/300 Read block 1257 len 40960
2011 168/300 Write block 956 len 20480 data: 2 2 2 2 2
2012 169/300 Flush
2013 170/300 Read block 84 len 16384
2014 171/300 Write block 4254 len 24576 data: 2 2 2 2 2 2
2015 172/300 Read block 3292 len 4096
2016 173/300 Write block 663 len 12288 data: 2 2 2
2017 174/300 Read block 1135 len 20480
2018 175/300 Flush
2019 176/300 Write block 1888 len 12288 data: 2 3 2
2020 177/300 Flush
2021 178/300 Read block 3852 len 32768
2022 179/300 Read block 4381 len 8192
2023 180/300 Read block 4198 len 24576
2024 181/300 Read block 2759 len 28672
2025 182/300 Write block 4839 len 4096 data: 2
2026 183/300 Write block 3044 len 32768 data: 3 3 3 3 3 3 3 2
2027 184/300 Read block 4902 len 32768
2028 185/300 Read block 2493 len 20480
2029 186/300 Write block 2239 len 12288 data: 2 2 3
2030 187/300 Write block 4155 len 32768 data: 2 2 2 2 2 2 2 2
2031 188/300 Flush
2032 189/300 Read block 3046 len 28672
2033 190/300 Read block 2440 len 4096
2034 191/300 Write block 4895 len 40960 data: 3 3 3 3 3 3 3 3 3 3
2035 192/300 Read block 739 len 40960
2036 193/300 Read block 4608 len 28672
2037 194/300 Read block 581 len 24576
2038 195/300 Read block 4937 len 32768
2039 196/300 Write block 3938 len 36864 data: 2 2 2 2 2 2 2 2 2
2040 197/300 Write block 1267 len 12288 data: 2 2 2
2041 198/300 Flush
2042 199/300 Write block 3919 len 28672 data: 2 2 2 2 2 2 2
2043 200/300 Read block 2973 len 12288
2044 201/300 Read block 4100 len 36864
2045 202/300 Read block 1359 len 36864
2046 203/300 Read block 1685 len 16384
2047 204/300 Flush
2048 205/300 Write block 2924 len 8192 data: 2 2
2049 206/300 Write block 518 len 4096 data: 2
2050 207/300 Read block 3917 len 20480
2051 208/300 Write block 4229 len 4096 data: 2
2052 209/300 Write block 1466 len 20480 data: 2 2 2 2 2
2053 210/300 Read block 816 len 4096
2054 211/300 Flush
2055 212/300 Read block 3359 len 4096
2056 213/300 Read block 2472 len 36864
2057 214/300 Read block 4922 len 8192
2058 215/300 Read block 1017 len 8192
2059 216/300 Read block 1625 len 36864
2060 217/300 Read block 3552 len 32768
2061 218/300 Write block 777 len 16384 data: 2 2 2 2
2062 219/300 Read block 1150 len 16384
2063 220/300 Read block 4621 len 20480
2064 221/300 Write block 3990 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2065 222/300 Read block 2417 len 12288
2066 223/300 Flush
2067 224/300 Read block 2329 len 24576
2068 225/300 Write block 1198 len 16384 data: 2 2 2 2
2069 226/300 Write block 4365 len 16384 data: 3 3 2 2
2070 227/300 Read block 2700 len 24576
2071 228/300 Read block 4351 len 24576
2072 229/300 Read block 537 len 36864
2073 230/300 Write block 3455 len 20480 data: 2 2 2 2 2
2074 231/300 Read block 1189 len 8192
2075 232/300 Read block 905 len 24576
2076 233/300 Read block 447 len 36864
2077 234/300 Read block 3039 len 28672
2078 235/300 Write block 2198 len 20480 data: 2 2 2 2 2
2079 236/300 Flush
2080 237/300 Read block 1248 len 40960
2081 238/300 Read block 1465 len 28672
2082 239/300 Read block 3440 len 32768
2083 240/300 Read block 2886 len 16384
2084 241/300 Write block 1958 len 8192 data: 3 3
2085 242/300 Read block 1329 len 4096
2086 243/300 Read block 1700 len 32768
2087 244/300 Read block 3640 len 28672
2088 245/300 Flush
2089 246/300 Write block 3198 len 24576 data: 2 2 2 2 2 2
2090 247/300 Read block 489 len 4096
2091 248/300 Write block 4067 len 20480 data: 3 3 3 2 2
2092 249/300 Write block 308 len 8192 data: 5 5
2093 250/300 Write block 2950 len 36864 data: 4 4 3 2 2 2 2 2 2
2094 251/300 Read block 3331 len 16384
2095 252/300 Write block 2047 len 24576 data: 2 2 2 2 2 2
2096 253/300 Read block 749 len 8192
2097 254/300 Read block 2040 len 24576
2098 255/300 Read block 2823 len 28672
2099 256/300 Read block 1671 len 24576
2100 257/300 Flush
2101 258/300 Write block 4680 len 16384 data: 3 3 3 3
2102 259/300 Read block 231 len 24576
2103 260/300 Write block 601 len 12288 data: 2 2 2
2104 261/300 Write block 3475 len 12288 data: 2 2 2
2105 262/300 Write block 4122 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2106 263/300 Write block 4909 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2107 264/300 Read block 3693 len 20480
2108 265/300 Read block 2623 len 8192
2109 266/300 Read block 4300 len 32768
2110 267/300 Write block 4765 len 16384 data: 3 3 3 3
2111 268/300 Read block 917 len 20480
2112 269/300 Read block 3636 len 8192
2113 270/300 Read block 716 len 16384
2114 271/300 Write block 3941 len 36864 data: 3 3 3 3 3 3 2 2 2
2115 272/300 Write block 2885 len 8192 data: 2 2
2116 273/300 Read block 4249 len 16384
2117 274/300 Read block 526 len 32768
2118 275/300 Write block 2237 len 36864 data: 2 2 3 3 4 3 3 3 2
2119 276/300 Write block 159 len 32768 data: 2 3 3 3 2 2 2 2
2120 277/300 Write block 4800 len 8192 data: 2 2
2121 278/300 Flush
2122 279/300 Write block 4934 len 4096 data: 2
2123 280/300 Read block 279 len 4096
2124 281/300 Read block 4771 len 16384
2125 282/300 Read block 2847 len 4096
2126 283/300 Write block 2657 len 28672 data: 2 2 3 3 3 3 4
2127 284/300 Write block 1458 len 32768 data: 2 2 2 2 2 2 2 2
2128 285/300 Read block 391 len 40960
2129 286/300 Flush
2130 287/300 Flush
2131 288/300 Read block 3521 len 36864
2132 289/300 Write block 2754 len 16384 data: 2 2 2 2
2133 290/300 Write block 2364 len 32768 data: 2 2 2 2 2 2 2 2
2134 291/300 Write block 4691 len 20480 data: 3 3 2 2 2
2135 292/300 Write block 2797 len 8192 data: 2 2
2136 293/300 Write block 1805 len 4096 data: 2
2137 294/300 Read block 1184 len 16384
2138 295/300 Write block 2164 len 20480 data: 2 2 2 2 2
2139 296/300 Read block 614 len 32768
2140 297/300 Read block 3815 len 4096
2141 298/300 Write block 3336 len 24576 data: 3 3 3 3 3 3
2142 299/300 Read block 1175 len 16384
2143 300/300 Read block 2614 len 36864
2144 301/300 Write block 1564 len 36864 data: 2 2 2 2 2 2 2 2 2
2145 Main task has work to do, go find it
2146 got dsc Start(2)
2147 start 2
2148 Mar 07 18:13:00.999 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/2, method: POST, req_id: ef366b85-72c7-4d67-8e52-72d3a57d91d5, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
2149 Replay: started 2, returned:Ok(())
2150 ----------------------------------------------------------------
2151 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2152 States: Active Active Offline
2153 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2154 1301 Acked Read 1 Done Done Sent false
2155 1302 Acked Flush 0 Done Done Sent false
2156 1303 Acked Read 4 Done Done Sent false
2157 1304 Acked Write 4 Done Done Sent false
2158 1305 Acked Write 5 Done Done Sent false
2159 1306 Acked Read 5 Done Done Sent false
2160 1307 Acked Write 8 Done Done Sent false
2161 1308 Acked Write 8 Done Done Sent false
2162 1309 Acked Read 4 Done Done Sent false
2163 1310 Acked Read 8 Done Done Sent false
2164 1311 Acked Read 10 Done Done Sent false
2165 1312 Acked Read 1 Done Done Sent false
2166 1313 Acked Read 3 Done Done Sent false
2167 1314 Acked Write 3 Done Done Sent false
2168 1315 Acked Read 2 Done Done Sent false
2169 1316 Acked Write 8 Done Done Sent false
2170 1317 Acked Read 2 Done Done Sent false
2171 1318 Acked Write 4 Done Done Sent false
2172 1319 Acked Read 3 Done Done Sent false
2173 1320 Acked Read 9 Done Done Sent false
2174 1321 Acked Read 10 Done Done Sent false
2175 1322 Acked Write 7 Done Done Sent false
2176 1323 Acked Read 7 Done Done Sent false
2177 1324 Acked Flush 0 Done Done Sent false
2178 1325 Acked Write 9 Done Done Sent false
2179 1326 Acked Write 9 Done Done Sent false
2180 1327 Acked Read 3 Done Done Sent false
2181 1328 Acked Flush 0 Done Done Sent false
2182 1329 Acked Read 10 Done Done Sent false
2183 1330 Acked Read 2 Done Done Sent false
2184 1331 Acked Write 2 Done Done Sent false
2185 1332 Acked Write 5 Done Done Sent false
2186 1333 Acked Read 8 Done Done Sent false
2187 1334 Acked Write 2 Done Done Sent false
2188 1335 Acked Read 10 Done Done Sent false
2189 1336 Acked Write 6 Done Done Sent false
2190 1337 Acked Write 5 Done Done Sent false
2191 1338 Acked Read 9 Done Done Sent false
2192 1339 Acked Read 8 Done Done Sent false
2193 1340 Acked Read 5 Done Done Sent false
2194 1341 Acked Read 4 Done Done Sent false
2195 1342 Acked Write 6 Done Done Sent false
2196 1343 Acked Flush 0 Done Done Sent false
2197 1344 Acked Read 10 Done Done Sent false
2198 1345 Acked Read 10 Done Done Sent false
2199 1346 Acked Write 9 Done Done Sent false
2200 1347 Acked Read 2 Done Done Sent false
2201 1348 Acked Write 8 Done Done Sent false
2202 1349 Acked Write 8 Done Done Sent false
2203 1350 Acked Write 7 Done Done Sent false
2204 1351 Acked Write 2 Done Done Sent false
2205 1352 Acked Read 10 Done Done Sent false
2206 1353 Acked Write 2 Done Done Sent false
2207 1354 Acked Read 1 Done Done Sent false
2208 1355 Acked Read 2 Done Done Sent false
2209 1356 Acked Read 4 Done Done Sent false
2210 1357 Acked Write 7 Done Done Sent false
2211 1358 Acked Read 2 Done Done Sent false
2212 1359 Acked Write 9 Done Done Sent false
2213 1360 Acked Read 10 Done Done Sent false
2214 1361 Acked Write 3 Done Done Sent false
2215 1362 Acked Read 6 Done Done Sent false
2216 1363 Acked Write 1 Done Done Sent false
2217 1364 Acked Read 6 Done Done Sent false
2218 1365 Acked Flush 0 Done Done Sent false
2219 1366 Acked Write 6 Done Done Sent false
2220 1367 Acked Read 6 Done Done Sent false
2221 1368 Acked Write 1 Done Done Sent false
2222 1369 Acked Read 6 Done Done Sent false
2223 1370 Acked Flush 0 Done Done Sent false
2224 1371 Acked Flush 0 Done Done Sent false
2225 1372 Acked Read 2 Done Done Sent false
2226 1373 Acked Write 10 Done Done Sent false
2227 1374 Acked Read 3 Done Done Sent false
2228 1375 Acked Read 2 Done Done Sent false
2229 1376 Acked Flush 0 Done Done Sent false
2230 1377 Acked Read 5 Done Done Sent false
2231 1378 Acked Read 1 Done Done Sent false
2232 1379 Acked Write 3 Done Done Sent false
2233 1380 Acked Flush 0 Done Done Sent false
2234 1381 Acked Read 9 Done Done Sent false
2235 1382 Acked Write 4 Done Done Sent false
2236 1383 Acked Read 9 Done Done Sent false
2237 1384 Acked Read 5 Done Done Sent false
2238 1385 Acked Flush 0 Done Done Sent false
2239 1386 Acked Read 8 Done Done Sent false
2240 1387 Acked Flush 0 Done Done Sent false
2241 1388 Acked Read 3 Done Done Sent false
2242 1389 Acked Write 3 Done Done Sent false
2243 1390 Acked Flush 0 Done Done Sent false
2244 1391 Acked Write 2 Done Done Sent false
2245 1392 Acked Read 8 Done Done Sent false
2246 1393 Acked Read 9 Done Done Sent false
2247 1394 Acked Write 5 Done Done Sent false
2248 1395 Acked Read 5 Done Done Sent false
2249 1396 Acked Read 1 Done Done Sent false
2250 1397 Acked Write 2 Done Done Sent false
2251 1398 Acked Write 6 Done Done Sent false
2252 1399 Acked Write 10 Done Done Sent false
2253 1400 Acked Read 7 Done Done Sent false
2254 1401 Acked Write 1 Done Done Sent false
2255 1402 Acked Read 6 Done Done Sent false
2256 1403 Acked Write 4 Done Done Sent false
2257 1404 Acked Read 3 Done Done Sent false
2258 1405 Acked Read 8 Done Done Sent false
2259 1406 Acked Flush 0 Done Done Sent false
2260 1407 Acked Write 3 Done Done Sent false
2261 1408 Acked Read 4 Done Done Sent false
2262 1409 Acked Read 8 Done Done Sent false
2263 1410 Acked Read 5 Done Done Sent false
2264 1411 Acked Read 1 Done Done Sent false
2265 1412 Acked Read 7 Done Done Sent false
2266 1413 Acked Write 9 Done Done Sent false
2267 1414 Acked Read 6 Done Done Sent false
2268 1415 Acked Read 10 Done Done Sent false
2269 1416 Acked Write 6 Done Done Sent false
2270 1417 Acked Write 5 Done Done Sent false
2271 1418 Acked Read 2 Done Done Sent false
2272 1419 Acked Write 4 Done Done Sent false
2273 1420 Acked Read 7 Done Done Sent false
2274 1421 Acked Flush 0 Done Done Sent false
2275 1422 Acked Read 8 Done Done Sent false
2276 1423 Acked Read 5 Done Done Sent false
2277 1424 Acked Read 5 Done Done Sent false
2278 1425 Acked Read 10 Done Done Sent false
2279 1426 Acked Flush 0 Done Done Sent false
2280 1427 Acked Flush 0 Done Done Sent false
2281 1428 Acked Read 9 Done Done Sent false
2282 1429 Acked Write 1 Done Done Sent false
2283 1430 Acked Write 8 Done Done Sent false
2284 1431 Acked Read 6 Done Done Sent false
2285 1432 Acked Read 4 Done Done Sent false
2286 1433 Acked Read 5 Done Done Sent false
2287 1434 Acked Read 6 Done Done Sent false
2288 1435 Acked Read 7 Done Done Sent false
2289 1436 Acked Read 1 Done Done Sent false
2290 1437 Acked Read 3 Done Done Sent false
2291 1438 Acked Flush 0 Done Done Sent false
2292 1439 Acked Write 5 Done Done Sent false
2293 1440 Acked Write 3 Done Done Sent false
2294 1441 Acked Read 5 Done Done Sent false
2295 1442 Acked Read 6 Done Done Sent false
2296 1443 Acked Flush 0 Done Done Sent false
2297 1444 Acked Read 9 Done Done Sent false
2298 1445 Acked Write 5 Done Done Sent false
2299 1446 Acked Write 3 Done Done Sent false
2300 1447 Acked Read 7 Done Done Sent false
2301 1448 Acked Write 10 Done Done Sent false
2302 1449 Acked Write 1 Done Done Sent false
2303 1450 Acked Write 8 Done Done Sent false
2304 1451 Acked Read 9 Done Done Sent false
2305 1452 Acked Write 2 Done Done Sent false
2306 1453 Acked Write 8 Done Done Sent false
2307 1454 Acked Read 10 Done Done Sent false
2308 1455 Acked Read 3 Done Done Sent false
2309 1456 Acked Write 7 Done Done Sent false
2310 1457 Acked Read 9 Done Done Sent false
2311 1458 Acked Read 6 Done Done Sent false
2312 1459 Acked Write 2 Done Done Sent false
2313 1460 Acked Read 5 Done Done Sent false
2314 1461 Acked Flush 0 Done Done Sent false
2315 1462 Acked Read 8 Done Done Sent false
2316 1463 Acked Read 5 Done Done Sent false
2317 1464 Acked Write 5 Done Done Sent false
2318 1465 Acked Read 5 Done Done Sent false
2319 1466 Acked Read 10 Done Done Sent false
2320 1467 Acked Read 10 Done Done Sent false
2321 1468 Acked Write 5 Done Done Sent false
2322 1469 Acked Flush 0 Done Done Sent false
2323 1470 Acked Read 4 Done Done Sent false
2324 1471 Acked Write 6 Done Done Sent false
2325 1472 Acked Read 1 Done Done Sent false
2326 1473 Acked Write 3 Done Done Sent false
2327 1474 Acked Read 5 Done Done Sent false
2328 1475 Acked Flush 0 Done Done Sent false
2329 1476 Acked Write 3 Done Done Sent false
2330 1477 Acked Flush 0 Done Done Sent false
2331 1478 Acked Read 8 Done Done Sent false
2332 1479 Acked Read 2 Done Done Sent false
2333 1480 Acked Read 6 Done Done Sent false
2334 1481 Acked Read 7 Done Done Sent false
2335 1482 Acked Write 1 Done Done Sent false
2336 1483 Acked Write 8 Done Done Sent false
2337 1484 Acked Read 8 Done Done Sent false
2338 1485 Acked Read 5 Done Done Sent false
2339 1486 Acked Write 3 Done Done Sent false
2340 1487 Acked Write 8 Done Done Sent false
2341 1488 Acked Flush 0 Done Done Sent false
2342 1489 Acked Read 7 Done Done Sent false
2343 1490 Acked Read 1 Done Done Sent false
2344 1491 Acked Write 10 Done Done Sent false
2345 1492 Acked Read 10 Done Done Sent false
2346 1493 Acked Read 7 Done Done Sent false
2347 1494 Acked Read 6 Done Done Sent false
2348 1495 Acked Read 8 Done Done Sent false
2349 1496 Acked Write 9 Done Done Sent false
2350 1497 Acked Write 3 Done Done Sent false
2351 1498 Acked Flush 0 Done Done Sent false
2352 1499 Acked Write 7 Done Done Sent false
2353 1500 Acked Read 3 Done Done Sent false
2354 1501 Acked Read 9 Done Done Sent false
2355 1502 Acked Read 9 Done Done Sent false
2356 1503 Acked Read 4 Done Done Sent false
2357 1504 Acked Flush 0 Done Done Sent false
2358 1505 Acked Write 2 Done Done Sent false
2359 1506 Acked Write 1 Done Done Sent false
2360 1507 Acked Read 5 Done Done Sent false
2361 1508 Acked Write 1 Done Done Sent false
2362 1509 Acked Write 5 Done Done Sent false
2363 1510 Acked Read 1 Done Done Sent false
2364 1511 Acked Flush 0 Done Done Sent false
2365 1512 Acked Read 1 Done Done Sent false
2366 1513 Acked Read 9 Done Done Sent false
2367 1514 Acked Read 2 Done Done Sent false
2368 1515 Acked Read 2 Done Done Sent false
2369 1516 Acked Read 9 Done Done Sent false
2370 1517 Acked Read 8 Done Done Sent false
2371 1518 Acked Write 4 Done Done Sent false
2372 1519 Acked Read 4 Done Done Sent false
2373 1520 Acked Read 5 Done Done Sent false
2374 1521 Acked Write 10 Done Done Sent false
2375 1522 Acked Read 3 Done Done Sent false
2376 1523 Acked Flush 0 Done Done Sent false
2377 1524 Acked Read 6 Done Done Sent false
2378 1525 Acked Write 4 Done Done Sent false
2379 1526 Acked Write 4 Done Done Sent false
2380 1527 Acked Read 6 Done Done Sent false
2381 1528 Acked Read 6 Done Done Sent false
2382 1529 Acked Read 9 Done Done Sent false
2383 1530 Acked Write 5 Done Done Sent false
2384 1531 Acked Read 2 Done Done Sent false
2385 1532 Acked Read 6 Done Done Sent false
2386 1533 Acked Read 9 Done Done Sent false
2387 1534 Acked Read 7 Done Done Sent false
2388 1535 Acked Write 5 Done Done Sent false
2389 1536 Acked Flush 0 Done Done Sent false
2390 1537 Acked Read 10 Done Done Sent false
2391 1538 Acked Read 7 Done Done Sent false
2392 1539 Acked Read 8 Done Done Sent false
2393 1540 Acked Read 4 Done Done Sent false
2394 1541 Acked Write 2 Done Done Sent false
2395 1542 Acked Read 1 Done Done Sent false
2396 1543 Acked Read 8 Done Done Sent false
2397 1544 Acked Read 7 Done Done Sent false
2398 1545 Acked Flush 0 Done Done Sent false
2399 1546 Acked Write 6 Done Done Sent false
2400 1547 Acked Read 1 Done Done Sent false
2401 1548 Acked Write 5 Done Done Sent false
2402 1549 Acked Write 2 Done Done Sent false
2403 1550 Acked Write 9 Done Done Sent false
2404 1551 Acked Read 4 Done Done Sent false
2405 1552 Acked Write 6 Done Done Sent false
2406 1553 Acked Read 2 Done Done Sent false
2407 1554 Acked Read 6 Done Done Sent false
2408 1555 Acked Read 7 Done Done Sent false
2409 1556 Acked Read 6 Done Done Sent false
2410 1557 Acked Flush 0 Done Done Sent false
2411 1558 Acked Write 4 Done Done Sent false
2412 1559 Acked Read 6 Done Done Sent false
2413 1560 Acked Write 3 Done Done Sent false
2414 1561 Acked Write 3 Done Done Sent false
2415 1562 Acked Write 10 Done Done Sent false
2416 1563 Acked Write 10 Done Done Sent false
2417 1564 Acked Read 5 Done Done Sent false
2418 1565 Acked Read 2 Done Done Sent false
2419 1566 Acked Read 8 Done Done Sent false
2420 1567 Acked Write 4 Done Done Sent false
2421 1568 Acked Read 5 Done Done Sent false
2422 1569 Acked Read 2 Done Done Sent false
2423 1570 Acked Read 4 Done Done Sent false
2424 1571 Acked Write 9 Done Done Sent false
2425 1572 Acked Write 2 Done Done Sent false
2426 1573 Acked Read 4 Done Done Sent false
2427 1574 Acked Read 8 Done Done Sent false
2428 1575 Acked Write 9 Done Done Sent false
2429 1576 Acked Write 8 Done Done Sent false
2430 1577 Acked Write 2 Done Done Sent false
2431 1578 Acked Flush 0 Done Done Sent false
2432 1579 Acked Write 1 Done Done Sent false
2433 1580 Acked Read 1 Done Done Sent false
2434 1581 Acked Read 4 Done Done Sent false
2435 1582 Acked Read 1 Done Done Sent false
2436 1583 Acked Write 7 Done Done Sent false
2437 1584 Acked Write 8 Done Done Sent false
2438 1585 Acked Read 10 Done Done Sent false
2439 1586 Acked Flush 0 Done Done Sent false
2440 1587 Acked Flush 0 Done Done Sent false
2441 1588 Acked Read 9 Done Done Sent false
2442 1589 Acked Write 4 Done Done Sent false
2443 1590 Acked Write 8 Done Done Sent false
2444 1591 Acked Write 5 Done Done Sent false
2445 1592 Acked Write 2 Done Done Sent false
2446 1593 Acked Write 1 Done Done Sent false
2447 1594 Acked Read 4 Done Done Sent false
2448 1595 Acked Write 5 Done Done Sent false
2449 1596 Acked Read 8 Done Done Sent false
2450 1597 Acked Read 1 Done Done Sent false
2451 1598 Acked Write 6 Done Done Sent false
2452 1599 Acked Read 4 Done Done Sent false
2453 1600 Acked Read 9 Done Done Sent false
2454 1601 Acked Write 9 Sent Done Sent false
2455 STATES DS:0 DS:1 DS:2 TOTAL
2456 Sent 1 0 301 302
2457 Done 300 301 0 601
2458 Skipped 0 0 0 0
2459 Error 0 0 0 0
2460 Last Flush: Some(JobId(1587)) Some(JobId(1587)) Some(JobId(1300))
2461 Downstairs last five completed: 1300 1299 1298 1297 1296
2462 Upstairs last five completed: 1601 1600 1599 1598 1597
2463 CLIENT: Up:0 ds:301 act:2
24642025-03-07T18:13:01.993ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24652025-03-07T18:13:02.995ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24662025-03-07T18:13:03.996ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24672025-03-07T18:13:04.998ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2468 ----------------------------------------------------------------
2469 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2470 States: Active Active Offline
2471 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2472 1301 Acked Read 1 Done Done Sent false
2473 1302 Acked Flush 0 Done Done Sent false
2474 1303 Acked Read 4 Done Done Sent false
2475 1304 Acked Write 4 Done Done Sent false
2476 1305 Acked Write 5 Done Done Sent false
2477 1306 Acked Read 5 Done Done Sent false
2478 1307 Acked Write 8 Done Done Sent false
2479 1308 Acked Write 8 Done Done Sent false
2480 1309 Acked Read 4 Done Done Sent false
2481 1310 Acked Read 8 Done Done Sent false
2482 1311 Acked Read 10 Done Done Sent false
2483 1312 Acked Read 1 Done Done Sent false
2484 1313 Acked Read 3 Done Done Sent false
2485 1314 Acked Write 3 Done Done Sent false
2486 1315 Acked Read 2 Done Done Sent false
2487 1316 Acked Write 8 Done Done Sent false
2488 1317 Acked Read 2 Done Done Sent false
2489 1318 Acked Write 4 Done Done Sent false
2490 1319 Acked Read 3 Done Done Sent false
2491 1320 Acked Read 9 Done Done Sent false
2492 1321 Acked Read 10 Done Done Sent false
2493 1322 Acked Write 7 Done Done Sent false
2494 1323 Acked Read 7 Done Done Sent false
2495 1324 Acked Flush 0 Done Done Sent false
2496 1325 Acked Write 9 Done Done Sent false
2497 1326 Acked Write 9 Done Done Sent false
2498 1327 Acked Read 3 Done Done Sent false
2499 1328 Acked Flush 0 Done Done Sent false
2500 1329 Acked Read 10 Done Done Sent false
2501 1330 Acked Read 2 Done Done Sent false
2502 1331 Acked Write 2 Done Done Sent false
2503 1332 Acked Write 5 Done Done Sent false
2504 1333 Acked Read 8 Done Done Sent false
2505 1334 Acked Write 2 Done Done Sent false
2506 1335 Acked Read 10 Done Done Sent false
2507 1336 Acked Write 6 Done Done Sent false
2508 1337 Acked Write 5 Done Done Sent false
2509 1338 Acked Read 9 Done Done Sent false
2510 1339 Acked Read 8 Done Done Sent false
2511 1340 Acked Read 5 Done Done Sent false
2512 1341 Acked Read 4 Done Done Sent false
2513 1342 Acked Write 6 Done Done Sent false
2514 1343 Acked Flush 0 Done Done Sent false
2515 1344 Acked Read 10 Done Done Sent false
2516 1345 Acked Read 10 Done Done Sent false
2517 1346 Acked Write 9 Done Done Sent false
2518 1347 Acked Read 2 Done Done Sent false
2519 1348 Acked Write 8 Done Done Sent false
2520 1349 Acked Write 8 Done Done Sent false
2521 1350 Acked Write 7 Done Done Sent false
2522 1351 Acked Write 2 Done Done Sent false
2523 1352 Acked Read 10 Done Done Sent false
2524 1353 Acked Write 2 Done Done Sent false
2525 1354 Acked Read 1 Done Done Sent false
2526 1355 Acked Read 2 Done Done Sent false
2527 1356 Acked Read 4 Done Done Sent false
2528 1357 Acked Write 7 Done Done Sent false
2529 1358 Acked Read 2 Done Done Sent false
2530 1359 Acked Write 9 Done Done Sent false
2531 1360 Acked Read 10 Done Done Sent false
2532 1361 Acked Write 3 Done Done Sent false
2533 1362 Acked Read 6 Done Done Sent false
2534 1363 Acked Write 1 Done Done Sent false
2535 1364 Acked Read 6 Done Done Sent false
2536 1365 Acked Flush 0 Done Done Sent false
2537 1366 Acked Write 6 Done Done Sent false
2538 1367 Acked Read 6 Done Done Sent false
2539 1368 Acked Write 1 Done Done Sent false
2540 1369 Acked Read 6 Done Done Sent false
2541 1370 Acked Flush 0 Done Done Sent false
2542 1371 Acked Flush 0 Done Done Sent false
2543 1372 Acked Read 2 Done Done Sent false
2544 1373 Acked Write 10 Done Done Sent false
2545 1374 Acked Read 3 Done Done Sent false
2546 1375 Acked Read 2 Done Done Sent false
2547 1376 Acked Flush 0 Done Done Sent false
2548 1377 Acked Read 5 Done Done Sent false
2549 1378 Acked Read 1 Done Done Sent false
2550 1379 Acked Write 3 Done Done Sent false
2551 1380 Acked Flush 0 Done Done Sent false
2552 1381 Acked Read 9 Done Done Sent false
2553 1382 Acked Write 4 Done Done Sent false
2554 1383 Acked Read 9 Done Done Sent false
2555 1384 Acked Read 5 Done Done Sent false
2556 1385 Acked Flush 0 Done Done Sent false
2557 1386 Acked Read 8 Done Done Sent false
2558 1387 Acked Flush 0 Done Done Sent false
2559 1388 Acked Read 3 Done Done Sent false
2560 1389 Acked Write 3 Done Done Sent false
2561 1390 Acked Flush 0 Done Done Sent false
2562 1391 Acked Write 2 Done Done Sent false
2563 1392 Acked Read 8 Done Done Sent false
2564 1393 Acked Read 9 Done Done Sent false
2565 1394 Acked Write 5 Done Done Sent false
2566 1395 Acked Read 5 Done Done Sent false
2567 1396 Acked Read 1 Done Done Sent false
2568 1397 Acked Write 2 Done Done Sent false
2569 1398 Acked Write 6 Done Done Sent false
2570 1399 Acked Write 10 Done Done Sent false
2571 1400 Acked Read 7 Done Done Sent false
2572 1401 Acked Write 1 Done Done Sent false
2573 1402 Acked Read 6 Done Done Sent false
2574 1403 Acked Write 4 Done Done Sent false
2575 1404 Acked Read 3 Done Done Sent false
2576 1405 Acked Read 8 Done Done Sent false
2577 1406 Acked Flush 0 Done Done Sent false
2578 1407 Acked Write 3 Done Done Sent false
2579 1408 Acked Read 4 Done Done Sent false
2580 1409 Acked Read 8 Done Done Sent false
2581 1410 Acked Read 5 Done Done Sent false
2582 1411 Acked Read 1 Done Done Sent false
2583 1412 Acked Read 7 Done Done Sent false
2584 1413 Acked Write 9 Done Done Sent false
2585 1414 Acked Read 6 Done Done Sent false
2586 1415 Acked Read 10 Done Done Sent false
2587 1416 Acked Write 6 Done Done Sent false
2588 1417 Acked Write 5 Done Done Sent false
2589 1418 Acked Read 2 Done Done Sent false
2590 1419 Acked Write 4 Done Done Sent false
2591 1420 Acked Read 7 Done Done Sent false
2592 1421 Acked Flush 0 Done Done Sent false
2593 1422 Acked Read 8 Done Done Sent false
2594 1423 Acked Read 5 Done Done Sent false
2595 1424 Acked Read 5 Done Done Sent false
2596 1425 Acked Read 10 Done Done Sent false
2597 1426 Acked Flush 0 Done Done Sent false
2598 1427 Acked Flush 0 Done Done Sent false
2599 1428 Acked Read 9 Done Done Sent false
2600 1429 Acked Write 1 Done Done Sent false
2601 1430 Acked Write 8 Done Done Sent false
2602 1431 Acked Read 6 Done Done Sent false
2603 1432 Acked Read 4 Done Done Sent false
2604 1433 Acked Read 5 Done Done Sent false
2605 1434 Acked Read 6 Done Done Sent false
2606 1435 Acked Read 7 Done Done Sent false
2607 1436 Acked Read 1 Done Done Sent false
2608 1437 Acked Read 3 Done Done Sent false
2609 1438 Acked Flush 0 Done Done Sent false
2610 1439 Acked Write 5 Done Done Sent false
2611 1440 Acked Write 3 Done Done Sent false
2612 1441 Acked Read 5 Done Done Sent false
2613 1442 Acked Read 6 Done Done Sent false
2614 1443 Acked Flush 0 Done Done Sent false
2615 1444 Acked Read 9 Done Done Sent false
2616 1445 Acked Write 5 Done Done Sent false
2617 1446 Acked Write 3 Done Done Sent false
2618 1447 Acked Read 7 Done Done Sent false
2619 1448 Acked Write 10 Done Done Sent false
2620 1449 Acked Write 1 Done Done Sent false
2621 1450 Acked Write 8 Done Done Sent false
2622 1451 Acked Read 9 Done Done Sent false
2623 1452 Acked Write 2 Done Done Sent false
2624 1453 Acked Write 8 Done Done Sent false
2625 1454 Acked Read 10 Done Done Sent false
2626 1455 Acked Read 3 Done Done Sent false
2627 1456 Acked Write 7 Done Done Sent false
2628 1457 Acked Read 9 Done Done Sent false
2629 1458 Acked Read 6 Done Done Sent false
2630 1459 Acked Write 2 Done Done Sent false
2631 1460 Acked Read 5 Done Done Sent false
2632 1461 Acked Flush 0 Done Done Sent false
2633 1462 Acked Read 8 Done Done Sent false
2634 1463 Acked Read 5 Done Done Sent false
2635 1464 Acked Write 5 Done Done Sent false
2636 1465 Acked Read 5 Done Done Sent false
2637 1466 Acked Read 10 Done Done Sent false
2638 1467 Acked Read 10 Done Done Sent false
2639 1468 Acked Write 5 Done Done Sent false
2640 1469 Acked Flush 0 Done Done Sent false
2641 1470 Acked Read 4 Done Done Sent false
2642 1471 Acked Write 6 Done Done Sent false
2643 1472 Acked Read 1 Done Done Sent false
2644 1473 Acked Write 3 Done Done Sent false
2645 1474 Acked Read 5 Done Done Sent false
2646 1475 Acked Flush 0 Done Done Sent false
2647 1476 Acked Write 3 Done Done Sent false
2648 1477 Acked Flush 0 Done Done Sent false
2649 1478 Acked Read 8 Done Done Sent false
2650 1479 Acked Read 2 Done Done Sent false
2651 1480 Acked Read 6 Done Done Sent false
2652 1481 Acked Read 7 Done Done Sent false
2653 1482 Acked Write 1 Done Done Sent false
2654 1483 Acked Write 8 Done Done Sent false
2655 1484 Acked Read 8 Done Done Sent false
2656 1485 Acked Read 5 Done Done Sent false
2657 1486 Acked Write 3 Done Done Sent false
2658 1487 Acked Write 8 Done Done Sent false
2659 1488 Acked Flush 0 Done Done Sent false
2660 1489 Acked Read 7 Done Done Sent false
2661 1490 Acked Read 1 Done Done Sent false
2662 1491 Acked Write 10 Done Done Sent false
2663 1492 Acked Read 10 Done Done Sent false
2664 1493 Acked Read 7 Done Done Sent false
2665 1494 Acked Read 6 Done Done Sent false
2666 1495 Acked Read 8 Done Done Sent false
2667 1496 Acked Write 9 Done Done Sent false
2668 1497 Acked Write 3 Done Done Sent false
2669 1498 Acked Flush 0 Done Done Sent false
2670 1499 Acked Write 7 Done Done Sent false
2671 1500 Acked Read 3 Done Done Sent false
2672 1501 Acked Read 9 Done Done Sent false
2673 1502 Acked Read 9 Done Done Sent false
2674 1503 Acked Read 4 Done Done Sent false
2675 1504 Acked Flush 0 Done Done Sent false
2676 1505 Acked Write 2 Done Done Sent false
2677 1506 Acked Write 1 Done Done Sent false
2678 1507 Acked Read 5 Done Done Sent false
2679 1508 Acked Write 1 Done Done Sent false
2680 1509 Acked Write 5 Done Done Sent false
2681 1510 Acked Read 1 Done Done Sent false
2682 1511 Acked Flush 0 Done Done Sent false
2683 1512 Acked Read 1 Done Done Sent false
2684 1513 Acked Read 9 Done Done Sent false
2685 1514 Acked Read 2 Done Done Sent false
2686 1515 Acked Read 2 Done Done Sent false
2687 1516 Acked Read 9 Done Done Sent false
2688 1517 Acked Read 8 Done Done Sent false
2689 1518 Acked Write 4 Done Done Sent false
2690 1519 Acked Read 4 Done Done Sent false
2691 1520 Acked Read 5 Done Done Sent false
2692 1521 Acked Write 10 Done Done Sent false
2693 1522 Acked Read 3 Done Done Sent false
2694 1523 Acked Flush 0 Done Done Sent false
2695 1524 Acked Read 6 Done Done Sent false
2696 1525 Acked Write 4 Done Done Sent false
2697 1526 Acked Write 4 Done Done Sent false
2698 1527 Acked Read 6 Done Done Sent false
2699 1528 Acked Read 6 Done Done Sent false
2700 1529 Acked Read 9 Done Done Sent false
2701 1530 Acked Write 5 Done Done Sent false
2702 1531 Acked Read 2 Done Done Sent false
2703 1532 Acked Read 6 Done Done Sent false
2704 1533 Acked Read 9 Done Done Sent false
2705 1534 Acked Read 7 Done Done Sent false
2706 1535 Acked Write 5 Done Done Sent false
2707 1536 Acked Flush 0 Done Done Sent false
2708 1537 Acked Read 10 Done Done Sent false
2709 1538 Acked Read 7 Done Done Sent false
2710 1539 Acked Read 8 Done Done Sent false
2711 1540 Acked Read 4 Done Done Sent false
2712 1541 Acked Write 2 Done Done Sent false
2713 1542 Acked Read 1 Done Done Sent false
2714 1543 Acked Read 8 Done Done Sent false
2715 1544 Acked Read 7 Done Done Sent false
2716 1545 Acked Flush 0 Done Done Sent false
2717 1546 Acked Write 6 Done Done Sent false
2718 1547 Acked Read 1 Done Done Sent false
2719 1548 Acked Write 5 Done Done Sent false
2720 1549 Acked Write 2 Done Done Sent false
2721 1550 Acked Write 9 Done Done Sent false
2722 1551 Acked Read 4 Done Done Sent false
2723 1552 Acked Write 6 Done Done Sent false
2724 1553 Acked Read 2 Done Done Sent false
2725 1554 Acked Read 6 Done Done Sent false
2726 1555 Acked Read 7 Done Done Sent false
2727 1556 Acked Read 6 Done Done Sent false
2728 1557 Acked Flush 0 Done Done Sent false
2729 1558 Acked Write 4 Done Done Sent false
2730 1559 Acked Read 6 Done Done Sent false
2731 1560 Acked Write 3 Done Done Sent false
2732 1561 Acked Write 3 Done Done Sent false
2733 1562 Acked Write 10 Done Done Sent false
2734 1563 Acked Write 10 Done Done Sent false
2735 1564 Acked Read 5 Done Done Sent false
2736 1565 Acked Read 2 Done Done Sent false
2737 1566 Acked Read 8 Done Done Sent false
2738 1567 Acked Write 4 Done Done Sent false
2739 1568 Acked Read 5 Done Done Sent false
2740 1569 Acked Read 2 Done Done Sent false
2741 1570 Acked Read 4 Done Done Sent false
2742 1571 Acked Write 9 Done Done Sent false
2743 1572 Acked Write 2 Done Done Sent false
2744 1573 Acked Read 4 Done Done Sent false
2745 1574 Acked Read 8 Done Done Sent false
2746 1575 Acked Write 9 Done Done Sent false
2747 1576 Acked Write 8 Done Done Sent false
2748 1577 Acked Write 2 Done Done Sent false
2749 1578 Acked Flush 0 Done Done Sent false
2750 1579 Acked Write 1 Done Done Sent false
2751 1580 Acked Read 1 Done Done Sent false
2752 1581 Acked Read 4 Done Done Sent false
2753 1582 Acked Read 1 Done Done Sent false
2754 1583 Acked Write 7 Done Done Sent false
2755 1584 Acked Write 8 Done Done Sent false
2756 1585 Acked Read 10 Done Done Sent false
2757 1586 Acked Flush 0 Done Done Sent false
2758 1587 Acked Flush 0 Done Done Sent false
2759 1588 Acked Read 9 Done Done Sent false
2760 1589 Acked Write 4 Done Done Sent false
2761 1590 Acked Write 8 Done Done Sent false
2762 1591 Acked Write 5 Done Done Sent false
2763 1592 Acked Write 2 Done Done Sent false
2764 1593 Acked Write 1 Done Done Sent false
2765 1594 Acked Read 4 Done Done Sent false
2766 1595 Acked Write 5 Done Done Sent false
2767 1596 Acked Read 8 Done Done Sent false
2768 1597 Acked Read 1 Done Done Sent false
2769 1598 Acked Write 6 Done Done Sent false
2770 1599 Acked Read 4 Done Done Sent false
2771 1600 Acked Read 9 Done Done Sent false
2772 1601 Acked Write 9 Done Done Sent false
2773 STATES DS:0 DS:1 DS:2 TOTAL
2774 Sent 0 0 301 301
2775 Done 301 301 0 602
2776 Skipped 0 0 0 0
2777 Error 0 0 0 0
2778 Last Flush: Some(JobId(1587)) Some(JobId(1587)) Some(JobId(1300))
2779 Downstairs last five completed: 1300 1299 1298 1297 1296
2780 Upstairs last five completed: 1601 1600 1599 1598 1597
2781 CLIENT: Up:0 ds:301 act:2
27822025-03-07T18:13:06.000ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27832025-03-07T18:13:06.746ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27842025-03-07T18:13:06.746ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27852025-03-07T18:13:06.746ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27862025-03-07T18:13:06.746ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27872025-03-07T18:13:06.746ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27882025-03-07T18:13:07.247ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27892025-03-07T18:13:08.004ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
2790 [8830] got start action
2791 [8830] I am going to restart
2792 Starting downstairs at port 8830
2793 Make output file at "/tmp/dsc/downstairs-8830.txt"
2794 [8830][2] reports Starting
2795 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1210)
2796 [8830][2] reports Running
2797 ----------------------------------------------------------------
2798 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2799 States: Active Active Offline
2800 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2801 1301 Acked Read 1 Done Done Sent false
2802 1302 Acked Flush 0 Done Done Sent false
2803 1303 Acked Read 4 Done Done Sent false
2804 1304 Acked Write 4 Done Done Sent false
2805 1305 Acked Write 5 Done Done Sent false
2806 1306 Acked Read 5 Done Done Sent false
2807 1307 Acked Write 8 Done Done Sent false
2808 1308 Acked Write 8 Done Done Sent false
2809 1309 Acked Read 4 Done Done Sent false
2810 1310 Acked Read 8 Done Done Sent false
2811 1311 Acked Read 10 Done Done Sent false
2812 1312 Acked Read 1 Done Done Sent false
2813 1313 Acked Read 3 Done Done Sent false
2814 1314 Acked Write 3 Done Done Sent false
2815 1315 Acked Read 2 Done Done Sent false
2816 1316 Acked Write 8 Done Done Sent false
2817 1317 Acked Read 2 Done Done Sent false
2818 1318 Acked Write 4 Done Done Sent false
2819 1319 Acked Read 3 Done Done Sent false
2820 1320 Acked Read 9 Done Done Sent false
2821 1321 Acked Read 10 Done Done Sent false
2822 1322 Acked Write 7 Done Done Sent false
2823 1323 Acked Read 7 Done Done Sent false
2824 1324 Acked Flush 0 Done Done Sent false
2825 1325 Acked Write 9 Done Done Sent false
2826 1326 Acked Write 9 Done Done Sent false
2827 1327 Acked Read 3 Done Done Sent false
2828 1328 Acked Flush 0 Done Done Sent false
2829 1329 Acked Read 10 Done Done Sent false
2830 1330 Acked Read 2 Done Done Sent false
2831 1331 Acked Write 2 Done Done Sent false
2832 1332 Acked Write 5 Done Done Sent false
2833 1333 Acked Read 8 Done Done Sent false
2834 1334 Acked Write 2 Done Done Sent false
2835 1335 Acked Read 10 Done Done Sent false
2836 1336 Acked Write 6 Done Done Sent false
2837 1337 Acked Write 5 Done Done Sent false
2838 1338 Acked Read 9 Done Done Sent false
2839 1339 Acked Read 8 Done Done Sent false
2840 1340 Acked Read 5 Done Done Sent false
2841 1341 Acked Read 4 Done Done Sent false
2842 1342 Acked Write 6 Done Done Sent false
2843 1343 Acked Flush 0 Done Done Sent false
2844 1344 Acked Read 10 Done Done Sent false
2845 1345 Acked Read 10 Done Done Sent false
2846 1346 Acked Write 9 Done Done Sent false
2847 1347 Acked Read 2 Done Done Sent false
2848 1348 Acked Write 8 Done Done Sent false
2849 1349 Acked Write 8 Done Done Sent false
2850 1350 Acked Write 7 Done Done Sent false
2851 1351 Acked Write 2 Done Done Sent false
2852 1352 Acked Read 10 Done Done Sent false
2853 1353 Acked Write 2 Done Done Sent false
2854 1354 Acked Read 1 Done Done Sent false
2855 1355 Acked Read 2 Done Done Sent false
2856 1356 Acked Read 4 Done Done Sent false
2857 1357 Acked Write 7 Done Done Sent false
2858 1358 Acked Read 2 Done Done Sent false
2859 1359 Acked Write 9 Done Done Sent false
2860 1360 Acked Read 10 Done Done Sent false
2861 1361 Acked Write 3 Done Done Sent false
2862 1362 Acked Read 6 Done Done Sent false
2863 1363 Acked Write 1 Done Done Sent false
2864 1364 Acked Read 6 Done Done Sent false
2865 1365 Acked Flush 0 Done Done Sent false
2866 1366 Acked Write 6 Done Done Sent false
2867 1367 Acked Read 6 Done Done Sent false
2868 1368 Acked Write 1 Done Done Sent false
2869 1369 Acked Read 6 Done Done Sent false
2870 1370 Acked Flush 0 Done Done Sent false
2871 1371 Acked Flush 0 Done Done Sent false
2872 1372 Acked Read 2 Done Done Sent false
2873 1373 Acked Write 10 Done Done Sent false
2874 1374 Acked Read 3 Done Done Sent false
2875 1375 Acked Read 2 Done Done Sent false
2876 1376 Acked Flush 0 Done Done Sent false
2877 1377 Acked Read 5 Done Done Sent false
2878 1378 Acked Read 1 Done Done Sent false
2879 1379 Acked Write 3 Done Done Sent false
2880 1380 Acked Flush 0 Done Done Sent false
2881 1381 Acked Read 9 Done Done Sent false
2882 1382 Acked Write 4 Done Done Sent false
2883 1383 Acked Read 9 Done Done Sent false
2884 1384 Acked Read 5 Done Done Sent false
2885 1385 Acked Flush 0 Done Done Sent false
2886 1386 Acked Read 8 Done Done Sent false
2887 1387 Acked Flush 0 Done Done Sent false
2888 1388 Acked Read 3 Done Done Sent false
2889 1389 Acked Write 3 Done Done Sent false
2890 1390 Acked Flush 0 Done Done Sent false
2891 1391 Acked Write 2 Done Done Sent false
2892 1392 Acked Read 8 Done Done Sent false
2893 1393 Acked Read 9 Done Done Sent false
2894 1394 Acked Write 5 Done Done Sent false
2895 1395 Acked Read 5 Done Done Sent false
2896 1396 Acked Read 1 Done Done Sent false
2897 1397 Acked Write 2 Done Done Sent false
2898 1398 Acked Write 6 Done Done Sent false
2899 1399 Acked Write 10 Done Done Sent false
2900 1400 Acked Read 7 Done Done Sent false
2901 1401 Acked Write 1 Done Done Sent false
2902 1402 Acked Read 6 Done Done Sent false
2903 1403 Acked Write 4 Done Done Sent false
2904 1404 Acked Read 3 Done Done Sent false
2905 1405 Acked Read 8 Done Done Sent false
2906 1406 Acked Flush 0 Done Done Sent false
2907 1407 Acked Write 3 Done Done Sent false
2908 1408 Acked Read 4 Done Done Sent false
2909 1409 Acked Read 8 Done Done Sent false
2910 1410 Acked Read 5 Done Done Sent false
2911 1411 Acked Read 1 Done Done Sent false
2912 1412 Acked Read 7 Done Done Sent false
2913 1413 Acked Write 9 Done Done Sent false
2914 1414 Acked Read 6 Done Done Sent false
2915 1415 Acked Read 10 Done Done Sent false
2916 1416 Acked Write 6 Done Done Sent false
2917 1417 Acked Write 5 Done Done Sent false
2918 1418 Acked Read 2 Done Done Sent false
2919 1419 Acked Write 4 Done Done Sent false
2920 1420 Acked Read 7 Done Done Sent false
2921 1421 Acked Flush 0 Done Done Sent false
2922 1422 Acked Read 8 Done Done Sent false
2923 1423 Acked Read 5 Done Done Sent false
2924 1424 Acked Read 5 Done Done Sent false
2925 1425 Acked Read 10 Done Done Sent false
2926 1426 Acked Flush 0 Done Done Sent false
2927 1427 Acked Flush 0 Done Done Sent false
2928 1428 Acked Read 9 Done Done Sent false
2929 1429 Acked Write 1 Done Done Sent false
2930 1430 Acked Write 8 Done Done Sent false
2931 1431 Acked Read 6 Done Done Sent false
2932 1432 Acked Read 4 Done Done Sent false
2933 1433 Acked Read 5 Done Done Sent false
2934 1434 Acked Read 6 Done Done Sent false
2935 1435 Acked Read 7 Done Done Sent false
2936 1436 Acked Read 1 Done Done Sent false
2937 1437 Acked Read 3 Done Done Sent false
2938 1438 Acked Flush 0 Done Done Sent false
2939 1439 Acked Write 5 Done Done Sent false
2940 1440 Acked Write 3 Done Done Sent false
2941 1441 Acked Read 5 Done Done Sent false
2942 1442 Acked Read 6 Done Done Sent false
2943 1443 Acked Flush 0 Done Done Sent false
2944 1444 Acked Read 9 Done Done Sent false
2945 1445 Acked Write 5 Done Done Sent false
2946 1446 Acked Write 3 Done Done Sent false
2947 1447 Acked Read 7 Done Done Sent false
2948 1448 Acked Write 10 Done Done Sent false
2949 1449 Acked Write 1 Done Done Sent false
2950 1450 Acked Write 8 Done Done Sent false
2951 1451 Acked Read 9 Done Done Sent false
2952 1452 Acked Write 2 Done Done Sent false
2953 1453 Acked Write 8 Done Done Sent false
2954 1454 Acked Read 10 Done Done Sent false
2955 1455 Acked Read 3 Done Done Sent false
2956 1456 Acked Write 7 Done Done Sent false
2957 1457 Acked Read 9 Done Done Sent false
2958 1458 Acked Read 6 Done Done Sent false
2959 1459 Acked Write 2 Done Done Sent false
2960 1460 Acked Read 5 Done Done Sent false
2961 1461 Acked Flush 0 Done Done Sent false
2962 1462 Acked Read 8 Done Done Sent false
2963 1463 Acked Read 5 Done Done Sent false
2964 1464 Acked Write 5 Done Done Sent false
2965 1465 Acked Read 5 Done Done Sent false
2966 1466 Acked Read 10 Done Done Sent false
2967 1467 Acked Read 10 Done Done Sent false
2968 1468 Acked Write 5 Done Done Sent false
2969 1469 Acked Flush 0 Done Done Sent false
2970 1470 Acked Read 4 Done Done Sent false
2971 1471 Acked Write 6 Done Done Sent false
2972 1472 Acked Read 1 Done Done Sent false
2973 1473 Acked Write 3 Done Done Sent false
2974 1474 Acked Read 5 Done Done Sent false
2975 1475 Acked Flush 0 Done Done Sent false
2976 1476 Acked Write 3 Done Done Sent false
2977 1477 Acked Flush 0 Done Done Sent false
2978 1478 Acked Read 8 Done Done Sent false
2979 1479 Acked Read 2 Done Done Sent false
2980 1480 Acked Read 6 Done Done Sent false
2981 1481 Acked Read 7 Done Done Sent false
2982 1482 Acked Write 1 Done Done Sent false
2983 1483 Acked Write 8 Done Done Sent false
2984 1484 Acked Read 8 Done Done Sent false
2985 1485 Acked Read 5 Done Done Sent false
2986 1486 Acked Write 3 Done Done Sent false
2987 1487 Acked Write 8 Done Done Sent false
2988 1488 Acked Flush 0 Done Done Sent false
2989 1489 Acked Read 7 Done Done Sent false
2990 1490 Acked Read 1 Done Done Sent false
2991 1491 Acked Write 10 Done Done Sent false
2992 1492 Acked Read 10 Done Done Sent false
2993 1493 Acked Read 7 Done Done Sent false
2994 1494 Acked Read 6 Done Done Sent false
2995 1495 Acked Read 8 Done Done Sent false
2996 1496 Acked Write 9 Done Done Sent false
2997 1497 Acked Write 3 Done Done Sent false
2998 1498 Acked Flush 0 Done Done Sent false
2999 1499 Acked Write 7 Done Done Sent false
3000 1500 Acked Read 3 Done Done Sent false
3001 1501 Acked Read 9 Done Done Sent false
3002 1502 Acked Read 9 Done Done Sent false
3003 1503 Acked Read 4 Done Done Sent false
3004 1504 Acked Flush 0 Done Done Sent false
3005 1505 Acked Write 2 Done Done Sent false
3006 1506 Acked Write 1 Done Done Sent false
3007 1507 Acked Read 5 Done Done Sent false
3008 1508 Acked Write 1 Done Done Sent false
3009 1509 Acked Write 5 Done Done Sent false
3010 1510 Acked Read 1 Done Done Sent false
3011 1511 Acked Flush 0 Done Done Sent false
3012 1512 Acked Read 1 Done Done Sent false
3013 1513 Acked Read 9 Done Done Sent false
3014 1514 Acked Read 2 Done Done Sent false
3015 1515 Acked Read 2 Done Done Sent false
3016 1516 Acked Read 9 Done Done Sent false
3017 1517 Acked Read 8 Done Done Sent false
3018 1518 Acked Write 4 Done Done Sent false
3019 1519 Acked Read 4 Done Done Sent false
3020 1520 Acked Read 5 Done Done Sent false
3021 1521 Acked Write 10 Done Done Sent false
3022 1522 Acked Read 3 Done Done Sent false
3023 1523 Acked Flush 0 Done Done Sent false
3024 1524 Acked Read 6 Done Done Sent false
3025 1525 Acked Write 4 Done Done Sent false
3026 1526 Acked Write 4 Done Done Sent false
3027 1527 Acked Read 6 Done Done Sent false
3028 1528 Acked Read 6 Done Done Sent false
3029 1529 Acked Read 9 Done Done Sent false
3030 1530 Acked Write 5 Done Done Sent false
3031 1531 Acked Read 2 Done Done Sent false
3032 1532 Acked Read 6 Done Done Sent false
3033 1533 Acked Read 9 Done Done Sent false
3034 1534 Acked Read 7 Done Done Sent false
3035 1535 Acked Write 5 Done Done Sent false
3036 1536 Acked Flush 0 Done Done Sent false
3037 1537 Acked Read 10 Done Done Sent false
3038 1538 Acked Read 7 Done Done Sent false
3039 1539 Acked Read 8 Done Done Sent false
3040 1540 Acked Read 4 Done Done Sent false
3041 1541 Acked Write 2 Done Done Sent false
3042 1542 Acked Read 1 Done Done Sent false
3043 1543 Acked Read 8 Done Done Sent false
3044 1544 Acked Read 7 Done Done Sent false
3045 1545 Acked Flush 0 Done Done Sent false
3046 1546 Acked Write 6 Done Done Sent false
3047 1547 Acked Read 1 Done Done Sent false
3048 1548 Acked Write 5 Done Done Sent false
3049 1549 Acked Write 2 Done Done Sent false
3050 1550 Acked Write 9 Done Done Sent false
3051 1551 Acked Read 4 Done Done Sent false
3052 1552 Acked Write 6 Done Done Sent false
3053 1553 Acked Read 2 Done Done Sent false
3054 1554 Acked Read 6 Done Done Sent false
3055 1555 Acked Read 7 Done Done Sent false
3056 1556 Acked Read 6 Done Done Sent false
3057 1557 Acked Flush 0 Done Done Sent false
3058 1558 Acked Write 4 Done Done Sent false
3059 1559 Acked Read 6 Done Done Sent false
3060 1560 Acked Write 3 Done Done Sent false
3061 1561 Acked Write 3 Done Done Sent false
3062 1562 Acked Write 10 Done Done Sent false
3063 1563 Acked Write 10 Done Done Sent false
3064 1564 Acked Read 5 Done Done Sent false
3065 1565 Acked Read 2 Done Done Sent false
3066 1566 Acked Read 8 Done Done Sent false
3067 1567 Acked Write 4 Done Done Sent false
3068 1568 Acked Read 5 Done Done Sent false
3069 1569 Acked Read 2 Done Done Sent false
3070 1570 Acked Read 4 Done Done Sent false
3071 1571 Acked Write 9 Done Done Sent false
3072 1572 Acked Write 2 Done Done Sent false
3073 1573 Acked Read 4 Done Done Sent false
3074 1574 Acked Read 8 Done Done Sent false
3075 1575 Acked Write 9 Done Done Sent false
3076 1576 Acked Write 8 Done Done Sent false
3077 1577 Acked Write 2 Done Done Sent false
3078 1578 Acked Flush 0 Done Done Sent false
3079 1579 Acked Write 1 Done Done Sent false
3080 1580 Acked Read 1 Done Done Sent false
3081 1581 Acked Read 4 Done Done Sent false
3082 1582 Acked Read 1 Done Done Sent false
3083 1583 Acked Write 7 Done Done Sent false
3084 1584 Acked Write 8 Done Done Sent false
3085 1585 Acked Read 10 Done Done Sent false
3086 1586 Acked Flush 0 Done Done Sent false
3087 1587 Acked Flush 0 Done Done Sent false
3088 1588 Acked Read 9 Done Done Sent false
3089 1589 Acked Write 4 Done Done Sent false
3090 1590 Acked Write 8 Done Done Sent false
3091 1591 Acked Write 5 Done Done Sent false
3092 1592 Acked Write 2 Done Done Sent false
3093 1593 Acked Write 1 Done Done Sent false
3094 1594 Acked Read 4 Done Done Sent false
3095 1595 Acked Write 5 Done Done Sent false
3096 1596 Acked Read 8 Done Done Sent false
3097 1597 Acked Read 1 Done Done Sent false
3098 1598 Acked Write 6 Done Done Sent false
3099 1599 Acked Read 4 Done Done Sent false
3100 1600 Acked Read 9 Done Done Sent false
3101 1601 Acked Write 9 Done Done Sent false
3102 STATES DS:0 DS:1 DS:2 TOTAL
3103 Sent 0 0 301 301
3104 Done 301 301 0 602
3105 Skipped 0 0 0 0
3106 Error 0 0 0 0
3107 Last Flush: Some(JobId(1587)) Some(JobId(1587)) Some(JobId(1300))
3108 Downstairs last five completed: 1300 1299 1298 1297 1296
3109 Upstairs last five completed: 1601 1600 1599 1598 1597
3110 CLIENT: Up:0 ds:301 act:2
31112025-03-07T18:13:09.505ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31122025-03-07T18:13:10.007ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31132025-03-07T18:13:11.010ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31142025-03-07T18:13:12.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
3115 ----------------------------------------------------------------
3116 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3117 States: Active Active Offline
3118 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3119 1301 Acked Read 1 Done Done Sent false
3120 1302 Acked Flush 0 Done Done Sent false
3121 1303 Acked Read 4 Done Done Sent false
3122 1304 Acked Write 4 Done Done Sent false
3123 1305 Acked Write 5 Done Done Sent false
3124 1306 Acked Read 5 Done Done Sent false
3125 1307 Acked Write 8 Done Done Sent false
3126 1308 Acked Write 8 Done Done Sent false
3127 1309 Acked Read 4 Done Done Sent false
3128 1310 Acked Read 8 Done Done Sent false
3129 1311 Acked Read 10 Done Done Sent false
3130 1312 Acked Read 1 Done Done Sent false
3131 1313 Acked Read 3 Done Done Sent false
3132 1314 Acked Write 3 Done Done Sent false
3133 1315 Acked Read 2 Done Done Sent false
3134 1316 Acked Write 8 Done Done Sent false
3135 1317 Acked Read 2 Done Done Sent false
3136 1318 Acked Write 4 Done Done Sent false
3137 1319 Acked Read 3 Done Done Sent false
3138 1320 Acked Read 9 Done Done Sent false
3139 1321 Acked Read 10 Done Done Sent false
3140 1322 Acked Write 7 Done Done Sent false
3141 1323 Acked Read 7 Done Done Sent false
3142 1324 Acked Flush 0 Done Done Sent false
3143 1325 Acked Write 9 Done Done Sent false
3144 1326 Acked Write 9 Done Done Sent false
3145 1327 Acked Read 3 Done Done Sent false
3146 1328 Acked Flush 0 Done Done Sent false
3147 1329 Acked Read 10 Done Done Sent false
3148 1330 Acked Read 2 Done Done Sent false
3149 1331 Acked Write 2 Done Done Sent false
3150 1332 Acked Write 5 Done Done Sent false
3151 1333 Acked Read 8 Done Done Sent false
3152 1334 Acked Write 2 Done Done Sent false
3153 1335 Acked Read 10 Done Done Sent false
3154 1336 Acked Write 6 Done Done Sent false
3155 1337 Acked Write 5 Done Done Sent false
3156 1338 Acked Read 9 Done Done Sent false
3157 1339 Acked Read 8 Done Done Sent false
3158 1340 Acked Read 5 Done Done Sent false
3159 1341 Acked Read 4 Done Done Sent false
3160 1342 Acked Write 6 Done Done Sent false
3161 1343 Acked Flush 0 Done Done Sent false
3162 1344 Acked Read 10 Done Done Sent false
3163 1345 Acked Read 10 Done Done Sent false
3164 1346 Acked Write 9 Done Done Sent false
3165 1347 Acked Read 2 Done Done Sent false
3166 1348 Acked Write 8 Done Done Sent false
3167 1349 Acked Write 8 Done Done Sent false
3168 1350 Acked Write 7 Done Done Sent false
3169 1351 Acked Write 2 Done Done Sent false
3170 1352 Acked Read 10 Done Done Sent false
3171 1353 Acked Write 2 Done Done Sent false
3172 1354 Acked Read 1 Done Done Sent false
3173 1355 Acked Read 2 Done Done Sent false
3174 1356 Acked Read 4 Done Done Sent false
3175 1357 Acked Write 7 Done Done Sent false
3176 1358 Acked Read 2 Done Done Sent false
3177 1359 Acked Write 9 Done Done Sent false
3178 1360 Acked Read 10 Done Done Sent false
3179 1361 Acked Write 3 Done Done Sent false
3180 1362 Acked Read 6 Done Done Sent false
3181 1363 Acked Write 1 Done Done Sent false
3182 1364 Acked Read 6 Done Done Sent false
3183 1365 Acked Flush 0 Done Done Sent false
3184 1366 Acked Write 6 Done Done Sent false
3185 1367 Acked Read 6 Done Done Sent false
3186 1368 Acked Write 1 Done Done Sent false
3187 1369 Acked Read 6 Done Done Sent false
3188 1370 Acked Flush 0 Done Done Sent false
3189 1371 Acked Flush 0 Done Done Sent false
3190 1372 Acked Read 2 Done Done Sent false
3191 1373 Acked Write 10 Done Done Sent false
3192 1374 Acked Read 3 Done Done Sent false
3193 1375 Acked Read 2 Done Done Sent false
3194 1376 Acked Flush 0 Done Done Sent false
3195 1377 Acked Read 5 Done Done Sent false
3196 1378 Acked Read 1 Done Done Sent false
3197 1379 Acked Write 3 Done Done Sent false
3198 1380 Acked Flush 0 Done Done Sent false
3199 1381 Acked Read 9 Done Done Sent false
3200 1382 Acked Write 4 Done Done Sent false
3201 1383 Acked Read 9 Done Done Sent false
3202 1384 Acked Read 5 Done Done Sent false
3203 1385 Acked Flush 0 Done Done Sent false
3204 1386 Acked Read 8 Done Done Sent false
3205 1387 Acked Flush 0 Done Done Sent false
3206 1388 Acked Read 3 Done Done Sent false
3207 1389 Acked Write 3 Done Done Sent false
3208 1390 Acked Flush 0 Done Done Sent false
3209 1391 Acked Write 2 Done Done Sent false
3210 1392 Acked Read 8 Done Done Sent false
3211 1393 Acked Read 9 Done Done Sent false
3212 1394 Acked Write 5 Done Done Sent false
3213 1395 Acked Read 5 Done Done Sent false
3214 1396 Acked Read 1 Done Done Sent false
3215 1397 Acked Write 2 Done Done Sent false
3216 1398 Acked Write 6 Done Done Sent false
3217 1399 Acked Write 10 Done Done Sent false
3218 1400 Acked Read 7 Done Done Sent false
3219 1401 Acked Write 1 Done Done Sent false
3220 1402 Acked Read 6 Done Done Sent false
3221 1403 Acked Write 4 Done Done Sent false
3222 1404 Acked Read 3 Done Done Sent false
3223 1405 Acked Read 8 Done Done Sent false
3224 1406 Acked Flush 0 Done Done Sent false
3225 1407 Acked Write 3 Done Done Sent false
3226 1408 Acked Read 4 Done Done Sent false
3227 1409 Acked Read 8 Done Done Sent false
3228 1410 Acked Read 5 Done Done Sent false
3229 1411 Acked Read 1 Done Done Sent false
3230 1412 Acked Read 7 Done Done Sent false
3231 1413 Acked Write 9 Done Done Sent false
3232 1414 Acked Read 6 Done Done Sent false
3233 1415 Acked Read 10 Done Done Sent false
3234 1416 Acked Write 6 Done Done Sent false
3235 1417 Acked Write 5 Done Done Sent false
3236 1418 Acked Read 2 Done Done Sent false
3237 1419 Acked Write 4 Done Done Sent false
3238 1420 Acked Read 7 Done Done Sent false
3239 1421 Acked Flush 0 Done Done Sent false
3240 1422 Acked Read 8 Done Done Sent false
3241 1423 Acked Read 5 Done Done Sent false
3242 1424 Acked Read 5 Done Done Sent false
3243 1425 Acked Read 10 Done Done Sent false
3244 1426 Acked Flush 0 Done Done Sent false
3245 1427 Acked Flush 0 Done Done Sent false
3246 1428 Acked Read 9 Done Done Sent false
3247 1429 Acked Write 1 Done Done Sent false
3248 1430 Acked Write 8 Done Done Sent false
3249 1431 Acked Read 6 Done Done Sent false
3250 1432 Acked Read 4 Done Done Sent false
3251 1433 Acked Read 5 Done Done Sent false
3252 1434 Acked Read 6 Done Done Sent false
3253 1435 Acked Read 7 Done Done Sent false
3254 1436 Acked Read 1 Done Done Sent false
3255 1437 Acked Read 3 Done Done Sent false
3256 1438 Acked Flush 0 Done Done Sent false
3257 1439 Acked Write 5 Done Done Sent false
3258 1440 Acked Write 3 Done Done Sent false
3259 1441 Acked Read 5 Done Done Sent false
3260 1442 Acked Read 6 Done Done Sent false
3261 1443 Acked Flush 0 Done Done Sent false
3262 1444 Acked Read 9 Done Done Sent false
3263 1445 Acked Write 5 Done Done Sent false
3264 1446 Acked Write 3 Done Done Sent false
3265 1447 Acked Read 7 Done Done Sent false
3266 1448 Acked Write 10 Done Done Sent false
3267 1449 Acked Write 1 Done Done Sent false
3268 1450 Acked Write 8 Done Done Sent false
3269 1451 Acked Read 9 Done Done Sent false
3270 1452 Acked Write 2 Done Done Sent false
3271 1453 Acked Write 8 Done Done Sent false
3272 1454 Acked Read 10 Done Done Sent false
3273 1455 Acked Read 3 Done Done Sent false
3274 1456 Acked Write 7 Done Done Sent false
3275 1457 Acked Read 9 Done Done Sent false
3276 1458 Acked Read 6 Done Done Sent false
3277 1459 Acked Write 2 Done Done Sent false
3278 1460 Acked Read 5 Done Done Sent false
3279 1461 Acked Flush 0 Done Done Sent false
3280 1462 Acked Read 8 Done Done Sent false
3281 1463 Acked Read 5 Done Done Sent false
3282 1464 Acked Write 5 Done Done Sent false
3283 1465 Acked Read 5 Done Done Sent false
3284 1466 Acked Read 10 Done Done Sent false
3285 1467 Acked Read 10 Done Done Sent false
3286 1468 Acked Write 5 Done Done Sent false
3287 1469 Acked Flush 0 Done Done Sent false
3288 1470 Acked Read 4 Done Done Sent false
3289 1471 Acked Write 6 Done Done Sent false
3290 1472 Acked Read 1 Done Done Sent false
3291 1473 Acked Write 3 Done Done Sent false
3292 1474 Acked Read 5 Done Done Sent false
3293 1475 Acked Flush 0 Done Done Sent false
3294 1476 Acked Write 3 Done Done Sent false
3295 1477 Acked Flush 0 Done Done Sent false
3296 1478 Acked Read 8 Done Done Sent false
3297 1479 Acked Read 2 Done Done Sent false
3298 1480 Acked Read 6 Done Done Sent false
3299 1481 Acked Read 7 Done Done Sent false
3300 1482 Acked Write 1 Done Done Sent false
3301 1483 Acked Write 8 Done Done Sent false
3302 1484 Acked Read 8 Done Done Sent false
3303 1485 Acked Read 5 Done Done Sent false
3304 1486 Acked Write 3 Done Done Sent false
3305 1487 Acked Write 8 Done Done Sent false
3306 1488 Acked Flush 0 Done Done Sent false
3307 1489 Acked Read 7 Done Done Sent false
3308 1490 Acked Read 1 Done Done Sent false
3309 1491 Acked Write 10 Done Done Sent false
3310 1492 Acked Read 10 Done Done Sent false
3311 1493 Acked Read 7 Done Done Sent false
3312 1494 Acked Read 6 Done Done Sent false
3313 1495 Acked Read 8 Done Done Sent false
3314 1496 Acked Write 9 Done Done Sent false
3315 1497 Acked Write 3 Done Done Sent false
3316 1498 Acked Flush 0 Done Done Sent false
3317 1499 Acked Write 7 Done Done Sent false
3318 1500 Acked Read 3 Done Done Sent false
3319 1501 Acked Read 9 Done Done Sent false
3320 1502 Acked Read 9 Done Done Sent false
3321 1503 Acked Read 4 Done Done Sent false
3322 1504 Acked Flush 0 Done Done Sent false
3323 1505 Acked Write 2 Done Done Sent false
3324 1506 Acked Write 1 Done Done Sent false
3325 1507 Acked Read 5 Done Done Sent false
3326 1508 Acked Write 1 Done Done Sent false
3327 1509 Acked Write 5 Done Done Sent false
3328 1510 Acked Read 1 Done Done Sent false
3329 1511 Acked Flush 0 Done Done Sent false
3330 1512 Acked Read 1 Done Done Sent false
3331 1513 Acked Read 9 Done Done Sent false
3332 1514 Acked Read 2 Done Done Sent false
3333 1515 Acked Read 2 Done Done Sent false
3334 1516 Acked Read 9 Done Done Sent false
3335 1517 Acked Read 8 Done Done Sent false
3336 1518 Acked Write 4 Done Done Sent false
3337 1519 Acked Read 4 Done Done Sent false
3338 1520 Acked Read 5 Done Done Sent false
3339 1521 Acked Write 10 Done Done Sent false
3340 1522 Acked Read 3 Done Done Sent false
3341 1523 Acked Flush 0 Done Done Sent false
3342 1524 Acked Read 6 Done Done Sent false
3343 1525 Acked Write 4 Done Done Sent false
3344 1526 Acked Write 4 Done Done Sent false
3345 1527 Acked Read 6 Done Done Sent false
3346 1528 Acked Read 6 Done Done Sent false
3347 1529 Acked Read 9 Done Done Sent false
3348 1530 Acked Write 5 Done Done Sent false
3349 1531 Acked Read 2 Done Done Sent false
3350 1532 Acked Read 6 Done Done Sent false
3351 1533 Acked Read 9 Done Done Sent false
3352 1534 Acked Read 7 Done Done Sent false
3353 1535 Acked Write 5 Done Done Sent false
3354 1536 Acked Flush 0 Done Done Sent false
3355 1537 Acked Read 10 Done Done Sent false
3356 1538 Acked Read 7 Done Done Sent false
3357 1539 Acked Read 8 Done Done Sent false
3358 1540 Acked Read 4 Done Done Sent false
3359 1541 Acked Write 2 Done Done Sent false
3360 1542 Acked Read 1 Done Done Sent false
3361 1543 Acked Read 8 Done Done Sent false
3362 1544 Acked Read 7 Done Done Sent false
3363 1545 Acked Flush 0 Done Done Sent false
3364 1546 Acked Write 6 Done Done Sent false
3365 1547 Acked Read 1 Done Done Sent false
3366 1548 Acked Write 5 Done Done Sent false
3367 1549 Acked Write 2 Done Done Sent false
3368 1550 Acked Write 9 Done Done Sent false
3369 1551 Acked Read 4 Done Done Sent false
3370 1552 Acked Write 6 Done Done Sent false
3371 1553 Acked Read 2 Done Done Sent false
3372 1554 Acked Read 6 Done Done Sent false
3373 1555 Acked Read 7 Done Done Sent false
3374 1556 Acked Read 6 Done Done Sent false
3375 1557 Acked Flush 0 Done Done Sent false
3376 1558 Acked Write 4 Done Done Sent false
3377 1559 Acked Read 6 Done Done Sent false
3378 1560 Acked Write 3 Done Done Sent false
3379 1561 Acked Write 3 Done Done Sent false
3380 1562 Acked Write 10 Done Done Sent false
3381 1563 Acked Write 10 Done Done Sent false
3382 1564 Acked Read 5 Done Done Sent false
3383 1565 Acked Read 2 Done Done Sent false
3384 1566 Acked Read 8 Done Done Sent false
3385 1567 Acked Write 4 Done Done Sent false
3386 1568 Acked Read 5 Done Done Sent false
3387 1569 Acked Read 2 Done Done Sent false
3388 1570 Acked Read 4 Done Done Sent false
3389 1571 Acked Write 9 Done Done Sent false
3390 1572 Acked Write 2 Done Done Sent false
3391 1573 Acked Read 4 Done Done Sent false
3392 1574 Acked Read 8 Done Done Sent false
3393 1575 Acked Write 9 Done Done Sent false
3394 1576 Acked Write 8 Done Done Sent false
3395 1577 Acked Write 2 Done Done Sent false
3396 1578 Acked Flush 0 Done Done Sent false
3397 1579 Acked Write 1 Done Done Sent false
3398 1580 Acked Read 1 Done Done Sent false
3399 1581 Acked Read 4 Done Done Sent false
3400 1582 Acked Read 1 Done Done Sent false
3401 1583 Acked Write 7 Done Done Sent false
3402 1584 Acked Write 8 Done Done Sent false
3403 1585 Acked Read 10 Done Done Sent false
3404 1586 Acked Flush 0 Done Done Sent false
3405 1587 Acked Flush 0 Done Done Sent false
3406 1588 Acked Read 9 Done Done Sent false
3407 1589 Acked Write 4 Done Done Sent false
3408 1590 Acked Write 8 Done Done Sent false
3409 1591 Acked Write 5 Done Done Sent false
3410 1592 Acked Write 2 Done Done Sent false
3411 1593 Acked Write 1 Done Done Sent false
3412 1594 Acked Read 4 Done Done Sent false
3413 1595 Acked Write 5 Done Done Sent false
3414 1596 Acked Read 8 Done Done Sent false
3415 1597 Acked Read 1 Done Done Sent false
3416 1598 Acked Write 6 Done Done Sent false
3417 1599 Acked Read 4 Done Done Sent false
3418 1600 Acked Read 9 Done Done Sent false
3419 1601 Acked Write 9 Done Done Sent false
3420 STATES DS:0 DS:1 DS:2 TOTAL
3421 Sent 0 0 301 301
3422 Done 301 301 0 602
3423 Skipped 0 0 0 0
3424 Error 0 0 0 0
3425 Last Flush: Some(JobId(1587)) Some(JobId(1587)) Some(JobId(1300))
3426 Downstairs last five completed: 1300 1299 1298 1297 1296
3427 Upstairs last five completed: 1601 1600 1599 1598 1597
3428 CLIENT: Up:0 ds:301 act:2
34292025-03-07T18:13:13.508ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34302025-03-07T18:13:14.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34312025-03-07T18:13:15.017ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34322025-03-07T18:13:16.018ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34332025-03-07T18:13:16.747ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34342025-03-07T18:13:16.748ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39613) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34352025-03-07T18:13:16.749ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34362025-03-07T18:13:16.749ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34372025-03-07T18:13:16.750ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34382025-03-07T18:13:16.750ZINFOcrucible: send last flush ID to this DS: Some(JobId(1300)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34392025-03-07T18:13:16.750ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34402025-03-07T18:13:16.750ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1300)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34412025-03-07T18:13:16.750ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1300)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
3442 ----------------------------------------------------------------
3443 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
3444 States: Active Active Active
3445 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3446 1588 Acked Read 9 Done Done Done true
3447 1589 Acked Write 4 Done Done Done true
3448 1590 Acked Write 8 Done Done Done true
3449 1591 Acked Write 5 Done Done Done true
3450 1592 Acked Write 2 Done Done Done true
3451 1593 Acked Write 1 Done Done Done true
3452 1594 Acked Read 4 Done Done Done true
3453 1595 Acked Write 5 Done Done Done true
3454 1596 Acked Read 8 Done Done Done true
3455 1597 Acked Read 1 Done Done Done true
3456 1598 Acked Write 6 Done Done Done true
3457 1599 Acked Read 4 Done Done Done true
3458 1600 Acked Read 9 Done Done Done true
3459 1601 Acked Write 9 Done Done Done true
3460 STATES DS:0 DS:1 DS:2 TOTAL
3461 Sent 0 0 0 0
3462 Done 14 14 14 42
3463 Skipped 0 0 0 0
3464 Error 0 0 0 0
3465 Last Flush: Some(JobId(1587)) Some(JobId(1587)) Some(JobId(1587))
3466 Downstairs last five completed: 1587 1586 1585 1584 1583
3467 Upstairs last five completed: 1601 1600 1599 1598 1597
3468 CLIENT: Up:0 ds:14 act:3
3469 ----------------------------------------------------------------
3470 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3471 Downstairs last five completed: 1602 1601 1600 1599 1598
3472 Upstairs last five completed: 1602 1601 1600 1599 1598
3473 CLIENT: Up:0 ds:0 act:3
3474 Replay: All jobs finished
3475 Main task has work to do, go find it
3476 got dsc Stop(0)
3477 stop 0
3478 Mar 07 18:13:21.011 INFO request completed, latency_us: 336, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 124aa30a-f5b0-4905-a91b-e968a6a0aeba, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
3479 [8810] Got stop action so:false kr:false
34802025-03-07T18:13:21.012ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34812025-03-07T18:13:21.012ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34822025-03-07T18:13:21.012ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34832025-03-07T18:13:21.012ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
3484 Mar 07 18:13:21.012 INFO request completed, latency_us: 183, response_code: 200, uri: /state/cid/0, method: GET, req_id: 544b8bc4-de30-4450-80e4-4f7c4bde4e92, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
3485 [8810] Exited with: ExitStatus(unix_wait_status(9))
3486 [8810][0] reports Exit
3487 Mar 07 18:13:25.015 INFO request completed, latency_us: 280, response_code: 200, uri: /state/cid/0, method: GET, req_id: 800a60bc-5e87-43ea-82e7-f89b9b3a8345, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
3488 001/300 Read block 4064 len 32768
3489 002/300 Read block 4261 len 36864
3490 003/300 Read block 201 len 16384
3491 004/300 Write block 757 len 36864 data: 2 2 2 2 2 2 2 2 2
3492 005/300 Flush
3493 006/300 Read block 2384 len 28672
3494 007/300 Flush
3495 008/300 Write block 4561 len 16384 data: 2 2 2 2
3496 009/300 Write block 4701 len 20480 data: 2 2 2 2 2
3497 010/300 Write block 3266 len 16384 data: 2 2 2 2
3498 011/300 Read block 3924 len 36864
3499 012/300 Read block 4672 len 28672
3500 013/300 Read block 1366 len 8192
3501 014/300 Read block 772 len 20480
3502 015/300 Write block 283 len 8192 data: 2 2
3503 016/300 Write block 1837 len 20480 data: 2 2 2 2 2
3504 017/300 Read block 4558 len 8192
3505 018/300 Write block 4547 len 8192 data: 2 2
3506 019/300 Read block 3379 len 12288
3507 020/300 Write block 1334 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3508 021/300 Read block 1001 len 40960
3509 022/300 Read block 216 len 4096
3510 023/300 Write block 1977 len 4096 data: 2
3511 024/300 Read block 2087 len 32768
3512 025/300 Read block 2911 len 32768
3513 026/300 Write block 3793 len 24576 data: 3 3 3 3 3 2
3514 027/300 Read block 4368 len 24576
3515 028/300 Read block 3112 len 40960
3516 029/300 Read block 4947 len 8192
3517 030/300 Write block 2157 len 12288 data: 2 2 2
3518 031/300 Write block 1565 len 28672 data: 3 3 3 3 3 3 3
3519 032/300 Read block 2861 len 12288
3520 033/300 Read block 626 len 20480
3521 034/300 Write block 4213 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3522 035/300 Read block 1386 len 12288
3523 036/300 Write block 2909 len 28672 data: 3 3 2 2 2 3 3
3524 037/300 Write block 2804 len 36864 data: 2 2 2 3 3 3 2 2 2
3525 038/300 Write block 712 len 28672 data: 2 2 2 2 2 2 2
3526 039/300 Read block 2842 len 36864
3527 040/300 Read block 3040 len 8192
3528 041/300 Write block 834 len 36864 data: 2 2 2 2 2 2 2 2 2
3529 042/300 Read block 1804 len 16384
3530 043/300 Read block 1669 len 4096
3531 044/300 Read block 4264 len 40960
3532 045/300 Read block 4869 len 4096
3533 046/300 Read block 3748 len 28672
3534 047/300 Read block 3759 len 40960
3535 048/300 Flush
3536 049/300 Write block 817 len 12288 data: 3 3 3
3537 050/300 Write block 2686 len 4096 data: 2
3538 051/300 Read block 3876 len 12288
3539 052/300 Write block 1284 len 12288 data: 2 2 2
3540 053/300 Flush
3541 054/300 Write block 3350 len 12288 data: 2 2 2
3542 055/300 Read block 2992 len 16384
3543 056/300 Write block 2494 len 24576 data: 4 3 2 2 2 2
3544 057/300 Write block 4653 len 24576 data: 2 2 2 2 2 2
3545 058/300 Read block 4956 len 36864
3546 059/300 Read block 1144 len 4096
3547 060/300 Flush
3548 061/300 Write block 148 len 12288 data: 2 2 2
3549 062/300 Write block 4328 len 40960 data: 3 2 2 2 2 2 2 3 3 3
3550 063/300 Read block 739 len 24576
3551 064/300 Flush
3552 065/300 Write block 3567 len 12288 data: 2 2 2
3553 066/300 Write block 4870 len 32768 data: 2 2 2 2 2 2 2 2
3554 067/300 Read block 3743 len 36864
3555 068/300 Flush
3556 069/300 Read block 4641 len 8192
3557 070/300 Flush
3558 071/300 Read block 4247 len 36864
3559 072/300 Write block 3080 len 8192 data: 3 3
3560 073/300 Write block 4743 len 24576 data: 2 2 2 2 2 2
3561 074/300 Flush
3562 075/300 Read block 3201 len 36864
3563 076/300 Write block 44 len 20480 data: 2 2 2 2 2
3564 077/300 Write block 2503 len 32768 data: 2 2 2 2 2 2 2 2
3565 078/300 Write block 4110 len 4096 data: 2
3566 079/300 Read block 3567 len 32768
3567 080/300 Read block 249 len 36864
3568 081/300 Read block 3508 len 12288
3569 082/300 Write block 4106 len 36864 data: 2 2 2 2 3 2 2 2 2
3570 083/300 Read block 832 len 20480
3571 084/300 Write block 2539 len 36864 data: 2 2 2 2 2 2 2 2 2
3572 085/300 Write block 2781 len 4096 data: 2
3573 086/300 Flush
3574 087/300 Flush
3575 088/300 Write block 3622 len 32768 data: 2 2 2 2 2 2 2 2
3576 089/300 Flush
3577 090/300 Read block 3171 len 32768
3578 091/300 Write block 2920 len 20480 data: 2 2 2 2 3
3579 092/300 Read block 1036 len 24576
3580 093/300 Write block 2816 len 20480 data: 2 2 2 2 2
3581 094/300 Read block 917 len 32768
3582 095/300 Read block 1143 len 16384
3583 096/300 Write block 4540 len 28672 data: 2 2 2 2 2 2 2
3584 097/300 Write block 2496 len 12288 data: 3 3 3
3585 098/300 Read block 1896 len 20480
3586 099/300 Read block 3180 len 28672
3587 100/300 Write block 3061 len 12288 data: 3 3 3
3588 101/300 Read block 2212 len 32768
3589 102/300 Read block 4892 len 32768
3590 103/300 Read block 4919 len 24576
3591 104/300 Write block 64 len 4096 data: 2
3592 105/300 Read block 4389 len 16384
3593 106/300 Write block 3523 len 4096 data: 2
3594 107/300 Read block 2398 len 24576
3595 108/300 Read block 4080 len 28672
3596 109/300 Read block 4691 len 24576
3597 110/300 Read block 2516 len 36864
3598 111/300 Read block 3530 len 24576
3599 112/300 Write block 274 len 20480 data: 2 2 2 2 2
3600 113/300 Write block 1773 len 32768 data: 2 2 2 2 2 2 2 2
3601 114/300 Write block 1007 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3602 115/300 Read block 4906 len 24576
3603 116/300 Read block 3977 len 12288
3604 117/300 Read block 946 len 8192
3605 118/300 Flush
3606 119/300 Read block 4480 len 16384
3607 120/300 Read block 532 len 12288
3608 121/300 Read block 3650 len 28672
3609 122/300 Read block 4875 len 32768
3610 123/300 Write block 2453 len 20480 data: 2 2 2 2 2
3611 124/300 Read block 4097 len 16384
3612 125/300 Read block 4195 len 12288
3613 126/300 Read block 1016 len 28672
3614 127/300 Write block 3292 len 36864 data: 2 2 2 2 2 2 2 2 2
3615 128/300 Write block 4504 len 40960 data: 2 3 3 3 3 3 3 3 2 2
3616 129/300 Read block 3606 len 32768
3617 130/300 Write block 212 len 16384 data: 4 4 3 4
3618 131/300 Write block 2468 len 4096 data: 2
3619 132/300 Read block 3552 len 36864
3620 133/300 Read block 1448 len 24576
3621 134/300 Read block 1848 len 16384
3622 135/300 Read block 4252 len 8192
3623 136/300 Read block 1821 len 4096
3624 137/300 Flush
3625 138/300 Read block 3448 len 12288
3626 139/300 Write block 1949 len 12288 data: 3 3 3
3627 140/300 Write block 3559 len 36864 data: 3 2 2 2 2 2 2 2 3
3628 141/300 Read block 2365 len 16384
3629 142/300 Read block 1108 len 4096
3630 143/300 Write block 4377 len 4096 data: 4
3631 144/300 Read block 4566 len 24576
3632 145/300 Write block 1353 len 12288 data: 2 2 2
3633 146/300 Read block 1952 len 24576
3634 147/300 Write block 2553 len 12288 data: 2 2 2
3635 148/300 Flush
3636 149/300 Write block 2541 len 16384 data: 3 3 3 3
3637 150/300 Read block 2357 len 36864
3638 151/300 Read block 4736 len 20480
3639 152/300 Write block 604 len 16384 data: 2 2 2 2
3640 153/300 Write block 189 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3641 154/300 Read block 3544 len 32768
3642 155/300 Read block 806 len 4096
3643 156/300 Read block 3660 len 28672
3644 157/300 Write block 3219 len 28672 data: 2 2 2 2 2 2 2
3645 158/300 Write block 1251 len 4096 data: 2
3646 159/300 Write block 2370 len 40960 data: 3 3 2 2 2 2 2 2 2 2
3647 160/300 Write block 2114 len 8192 data: 2 2
3648 161/300 Write block 2801 len 16384 data: 3 3 2 3
3649 162/300 Write block 663 len 4096 data: 3
3650 163/300 Write block 3278 len 24576 data: 2 2 2 3 2 2
3651 164/300 Write block 1720 len 40960 data: 2 2 2 2 3 3 3 3 3 2
3652 165/300 Write block 2450 len 40960 data: 2 2 2 3 3 3 3 3 2 2
3653 166/300 Read block 1759 len 28672
3654 167/300 Read block 2811 len 36864
3655 168/300 Read block 4701 len 4096
3656 169/300 Flush
3657 170/300 Write block 1590 len 36864 data: 2 2 2 2 3 3 3 3 2
3658 171/300 Write block 125 len 28672 data: 3 3 3 3 2 2 2
3659 172/300 Write block 244 len 36864 data: 2 2 2 2 2 2 2 2 2
3660 173/300 Write block 4885 len 16384 data: 2 2 3 3
3661 174/300 Read block 1364 len 8192
3662 175/300 Write block 2432 len 4096 data: 2
3663 176/300 Write block 1294 len 24576 data: 2 2 2 2 2 2
3664 177/300 Write block 2773 len 24576 data: 2 2 2 2 2 2
3665 178/300 Flush
3666 179/300 Read block 1659 len 8192
3667 180/300 Read block 1072 len 12288
3668 181/300 Write block 4624 len 24576 data: 2 2 2 2 2 2
3669 182/300 Read block 2589 len 32768
3670 183/300 Read block 3085 len 28672
3671 184/300 Write block 1614 len 12288 data: 3 3 3
3672 185/300 Read block 2869 len 28672
3673 186/300 Write block 3232 len 8192 data: 2 2
3674 187/300 Flush
3675 188/300 Flush
3676 189/300 Read block 2657 len 4096
3677 190/300 Write block 310 len 8192 data: 4 4
3678 191/300 Write block 2063 len 16384 data: 2 2 2 2
3679 192/300 Read block 2736 len 40960
3680 193/300 Write block 2679 len 24576 data: 2 2 2 2 2 2
3681 194/300 Read block 4201 len 8192
3682 195/300 Write block 4451 len 16384 data: 2 2 2 3
3683 196/300 Read block 4537 len 4096
3684 197/300 Write block 1830 len 16384 data: 2 2 2 2
3685 198/300 Flush
3686 199/300 Read block 3891 len 20480
3687 200/300 Read block 2934 len 28672
3688 201/300 Flush
3689 202/300 Read block 3241 len 4096
3690 203/300 Write block 3720 len 28672 data: 3 3 3 3 3 3 2
3691 204/300 Write block 156 len 8192 data: 2 2
3692 205/300 Flush
3693 206/300 Flush
3694 207/300 Read block 4807 len 32768
3695 208/300 Read block 4396 len 28672
3696 209/300 Flush
3697 210/300 Read block 1631 len 8192
3698 211/300 Read block 1235 len 4096
3699 212/300 Read block 2778 len 4096
3700 213/300 Read block 4740 len 28672
3701 214/300 Write block 3750 len 12288 data: 2 2 2
3702 215/300 Write block 909 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3703 216/300 Write block 1901 len 24576 data: 2 3 3 3 2 2
3704 217/300 Write block 3273 len 8192 data: 2 2
3705 218/300 Flush
3706 219/300 Read block 1323 len 36864
3707 220/300 Write block 2317 len 4096 data: 2
3708 221/300 Read block 4490 len 12288
3709 222/300 Write block 2382 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3710 223/300 Flush
3711 224/300 Write block 2024 len 28672 data: 2 2 2 2 2 2 2
3712 225/300 Read block 799 len 16384
3713 226/300 Read block 1261 len 36864
3714 227/300 Read block 4484 len 24576
3715 228/300 Write block 2883 len 40960 data: 2 2 3 3 2 2 3 3 3 3
3716 229/300 Read block 3359 len 24576
3717 230/300 Flush
3718 231/300 Write block 1462 len 12288 data: 3 3 3
3719 232/300 Read block 3898 len 12288
3720 233/300 Read block 4469 len 16384
3721 234/300 Write block 866 len 24576 data: 2 2 2 2 2 2
3722 235/300 Read block 4963 len 28672
3723 236/300 Read block 960 len 40960
3724 237/300 Read block 4960 len 4096
3725 238/300 Write block 3699 len 36864 data: 2 2 2 2 2 2 2 2 2
3726 239/300 Flush
3727 240/300 Write block 3598 len 12288 data: 2 2 2
3728 241/300 Write block 4787 len 32768 data: 2 2 2 2 2 2 2 2
3729 242/300 Read block 1326 len 4096
3730 243/300 Read block 440 len 8192
3731 244/300 Write block 1810 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3732 245/300 Write block 325 len 36864 data: 3 3 3 3 3 3 3 3 3
3733 246/300 Write block 3266 len 8192 data: 3 3
3734 247/300 Write block 4682 len 32768 data: 4 4 3 3 3 3 3 2
3735 248/300 Write block 4449 len 36864 data: 2 2 3 3 3 4 3 3 3
3736 249/300 Write block 4461 len 20480 data: 2 2 3 3 3
3737 250/300 Write block 4705 len 12288 data: 3 2 2
3738 251/300 Read block 2527 len 36864
3739 252/300 Read block 4733 len 8192
3740 253/300 Read block 4403 len 4096
3741 254/300 Read block 803 len 12288
3742 255/300 Read block 1229 len 16384
3743 256/300 Read block 3600 len 24576
3744 257/300 Read block 3879 len 12288
3745 258/300 Read block 1 len 40960
3746 259/300 Read block 2209 len 40960
3747 260/300 Read block 4739 len 4096
3748 261/300 Write block 3053 len 32768 data: 2 2 2 2 3 3 3 3
3749 262/300 Write block 1539 len 40960 data: 4 4 3 2 2 2 2 2 2 2
3750 263/300 Write block 4460 len 24576 data: 2 3 3 4 4 4
3751 264/300 Read block 20 len 32768
3752 265/300 Flush
3753 266/300 Write block 4545 len 32768 data: 3 3 3 3 2 2 2 2
3754 267/300 Read block 3415 len 20480
3755 268/300 Flush
3756 269/300 Read block 691 len 40960
3757 270/300 Flush
3758 271/300 Read block 4141 len 24576
3759 272/300 Write block 4663 len 16384 data: 2 2 2 2
3760 273/300 Write block 1909 len 20480 data: 3 3 3 2 2
3761 274/300 Read block 1952 len 24576
3762 275/300 Read block 3402 len 24576
3763 276/300 Read block 3048 len 40960
3764 277/300 Read block 1104 len 32768
3765 278/300 Read block 3666 len 8192
3766 279/300 Read block 197 len 28672
3767 280/300 Read block 1995 len 36864
3768 281/300 Write block 2921 len 32768 data: 3 3 3 4 3 2 2 2
3769 282/300 Read block 3322 len 8192
3770 283/300 Write block 4442 len 12288 data: 2 2 2
3771 284/300 Write block 2514 len 16384 data: 3 3 3 3
3772 285/300 Read block 1418 len 12288
3773 286/300 Read block 2198 len 24576
3774 287/300 Write block 2997 len 36864 data: 2 2 2 2 2 3 3 3 3
3775 288/300 Read block 901 len 32768
3776 289/300 Write block 4258 len 32768 data: 3 3 2 2 2 2 2 2
3777 290/300 Write block 4103 len 12288 data: 2 2 2
3778 291/300 Read block 3800 len 28672
3779 292/300 Write block 1915 len 32768 data: 2 2 2 2 2 2 2 2
3780 293/300 Read block 4909 len 20480
3781 294/300 Write block 4502 len 28672 data: 2 2 3 4 4 4 4
3782 295/300 Write block 2188 len 8192 data: 2 2
3783 296/300 Write block 3010 len 28672 data: 3 3 2 2 2 2 2
3784 297/300 Read block 1989 len 32768
3785 298/300 Read block 984 len 20480
3786 299/300 Write block 1923 len 4096 data: 2
3787 300/300 Flush
3788 301/300 Read block 641 len 4096
3789 Main task has work to do, go find it
3790 got dsc Start(0)
3791 start 0
3792 Mar 07 18:13:25.256 INFO request completed, latency_us: 195, response_code: 204, uri: /start/cid/0, method: POST, req_id: ac86d77f-6c30-4a7d-b849-cb20d06db4e4, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
3793 Replay: started 0, returned:Ok(())
3794 ----------------------------------------------------------------
3795 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3796 States: Offline Active Active
3797 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3798 1603 Acked Read 8 Sent Done Done false
3799 1604 Acked Read 9 Sent Done Done false
3800 1605 Acked Read 4 Sent Done Done false
3801 1606 Acked Write 9 Sent Done Done false
3802 1607 Acked Flush 0 Sent Done Done false
3803 1608 Acked Read 7 Sent Done Done false
3804 1609 Acked Flush 0 Sent Done Done false
3805 1610 Acked Write 4 Sent Done Done false
3806 1611 Acked Write 5 Sent Done Done false
3807 1612 Acked Write 4 Sent Done Done false
3808 1613 Acked Read 9 Sent Done Done false
3809 1614 Acked Read 7 Sent Done Done false
3810 1615 Acked Read 2 Sent Done Done false
3811 1616 Acked Read 5 Sent Done Done false
3812 1617 Acked Write 2 Sent Done Done false
3813 1618 Acked Write 5 Sent Done Done false
3814 1619 Acked Read 2 Sent Done Done false
3815 1620 Acked Write 2 Sent Done Done false
3816 1621 Acked Read 3 Sent Done Done false
3817 1622 Acked Write 10 Sent Done Done false
3818 1623 Acked Read 10 Sent Done Done false
3819 1624 Acked Read 1 Sent Done Done false
3820 1625 Acked Write 1 Sent Done Done false
3821 1626 Acked Read 8 Sent Done Done false
3822 1627 Acked Read 8 Sent Done Done false
3823 1628 Acked Write 6 Sent Done Done false
3824 1629 Acked Read 6 Sent Done Done false
3825 1630 Acked Read 10 Sent Done Done false
3826 1631 Acked Read 2 Sent Done Done false
3827 1632 Acked Write 3 Sent Done Done false
3828 1633 Acked Write 7 Sent Done Done false
3829 1634 Acked Read 3 Sent Done Done false
3830 1635 Acked Read 5 Sent Done Done false
3831 1636 Acked Write 10 Sent Done Done false
3832 1637 Acked Read 3 Sent Done Done false
3833 1638 Acked Write 7 Sent Done Done false
3834 1639 Acked Write 9 Sent Done Done false
3835 1640 Acked Write 7 Sent Done Done false
3836 1641 Acked Read 9 Sent Done Done false
3837 1642 Acked Read 2 Sent Done Done false
3838 1643 Acked Write 9 Sent Done Done false
3839 1644 Acked Read 4 Sent Done Done false
3840 1645 Acked Read 1 Sent Done Done false
3841 1646 Acked Read 10 Sent Done Done false
3842 1647 Acked Read 1 Sent Done Done false
3843 1648 Acked Read 7 Sent Done Done false
3844 1649 Acked Read 10 Sent Done Done false
3845 1650 Acked Flush 0 Sent Done Done false
3846 1651 Acked Write 3 Sent Done Done false
3847 1652 Acked Write 1 Sent Done Done false
3848 1653 Acked Read 3 Sent Done Done false
3849 1654 Acked Write 3 Sent Done Done false
3850 1655 Acked Flush 0 Sent Done Done false
3851 1656 Acked Write 3 Sent Done Done false
3852 1657 Acked Read 4 Sent Done Done false
3853 1658 Acked Write 6 Sent Done Done false
3854 1659 Acked Write 6 Sent Done Done false
3855 1660 Acked Read 9 Sent Done Done false
3856 1661 Acked Read 1 Sent Done Done false
3857 1662 Acked Flush 0 Sent Done Done false
3858 1663 Acked Write 3 Sent Done Done false
3859 1664 Acked Write 10 Sent Done Done false
3860 1665 Acked Read 6 Sent Done Done false
3861 1666 Acked Flush 0 Sent Done Done false
3862 1667 Acked Write 3 Sent Done Done false
3863 1668 Acked Write 8 Sent Done Done false
3864 1669 Acked Read 9 Sent Done Done false
3865 1670 Acked Flush 0 Sent Done Done false
3866 1671 Acked Read 2 Sent Done Done false
3867 1672 Acked Flush 0 Sent Done Done false
3868 1673 Acked Read 9 Sent Done Done false
3869 1674 Acked Write 2 Sent Done Done false
3870 1675 Acked Write 6 Sent Done Done false
3871 1676 Acked Flush 0 Sent Done Done false
3872 1677 Acked Read 9 Sent Done Done false
3873 1678 Acked Write 5 Sent Done Done false
3874 1679 Acked Write 8 Sent Done Done false
3875 1680 Acked Write 1 Sent Done Done false
3876 1681 Acked Read 8 Sent Done Done false
3877 1682 Acked Read 9 Sent Done Done false
3878 1683 Acked Read 3 Sent Done Done false
3879 1684 Acked Write 9 Sent Done Done false
3880 1685 Acked Read 5 Sent Done Done false
3881 1686 Acked Write 9 Sent Done Done false
3882 1687 Acked Write 1 Sent Done Done false
3883 1688 Acked Flush 0 Sent Done Done false
3884 1689 Acked Flush 0 Sent Done Done false
3885 1690 Acked Write 8 Sent Done Done false
3886 1691 Acked Flush 0 Sent Done Done false
3887 1692 Acked Read 8 Sent Done Done false
3888 1693 Acked Write 5 Sent Done Done false
3889 1694 Acked Read 6 Sent Done Done false
3890 1695 Acked Write 5 Sent Done Done false
3891 1696 Acked Read 8 Sent Done Done false
3892 1697 Acked Read 4 Sent Done Done false
3893 1698 Acked Write 7 Sent Done Done false
3894 1699 Acked Write 3 Sent Done Done false
3895 1700 Acked Read 5 Sent Done Done false
3896 1701 Acked Read 7 Sent Done Done false
3897 1702 Acked Write 3 Sent Done Done false
3898 1703 Acked Read 8 Sent Done Done false
3899 1704 Acked Read 8 Sent Done Done false
3900 1705 Acked Read 6 Sent Done Done false
3901 1706 Acked Write 1 Sent Done Done false
3902 1707 Acked Read 4 Sent Done Done false
3903 1708 Acked Write 1 Sent Done Done false
3904 1709 Acked Read 6 Sent Done Done false
3905 1710 Acked Read 7 Sent Done Done false
3906 1711 Acked Read 6 Sent Done Done false
3907 1712 Acked Read 9 Sent Done Done false
3908 1713 Acked Read 6 Sent Done Done false
3909 1714 Acked Write 5 Sent Done Done false
3910 1715 Acked Write 8 Sent Done Done false
3911 1716 Acked Write 10 Sent Done Done false
3912 1717 Acked Read 6 Sent Done Done false
3913 1718 Acked Read 3 Sent Done Done false
3914 1719 Acked Read 2 Sent Done Done false
3915 1720 Acked Flush 0 Sent Done Done false
3916 1721 Acked Read 4 Sent Done Done false
3917 1722 Acked Read 3 Sent Done Done false
3918 1723 Acked Read 7 Sent Done Done false
3919 1724 Acked Read 8 Sent Done Done false
3920 1725 Acked Write 5 Sent Done Done false
3921 1726 Acked Read 4 Sent Done Done false
3922 1727 Acked Read 3 Sent Done Done false
3923 1728 Acked Read 7 Sent Done Done false
3924 1729 Acked Write 9 Sent Done Done false
3925 1730 Acked Write 10 Sent Done Done false
3926 1731 Acked Read 8 Sent Done Done false
3927 1732 Acked Write 4 Sent Done Done false
3928 1733 Acked Write 1 Sent Done Done false
3929 1734 Acked Read 9 Sent Done Done false
3930 1735 Acked Read 6 Sent Done Done false
3931 1736 Acked Read 4 Sent Done Done false
3932 1737 Acked Read 2 Sent Done Done false
3933 1738 Acked Read 1 Sent Done Done false
3934 1739 Acked Flush 0 Sent Done Done false
3935 1740 Acked Read 3 Sent Done Done false
3936 1741 Acked Write 3 Sent Done Done false
3937 1742 Acked Write 9 Sent Done Done false
3938 1743 Acked Read 4 Sent Done Done false
3939 1744 Acked Read 1 Sent Done Done false
3940 1745 Acked Write 1 Sent Done Done false
3941 1746 Acked Read 6 Sent Done Done false
3942 1747 Acked Write 3 Sent Done Done false
3943 1748 Acked Read 6 Sent Done Done false
3944 1749 Acked Write 3 Sent Done Done false
3945 1750 Acked Flush 0 Sent Done Done false
3946 1751 Acked Write 4 Sent Done Done false
3947 1752 Acked Read 9 Sent Done Done false
3948 1753 Acked Read 5 Sent Done Done false
3949 1754 Acked Write 4 Sent Done Done false
3950 1755 Acked Write 10 Sent Done Done false
3951 1756 Acked Read 8 Sent Done Done false
3952 1757 Acked Read 1 Sent Done Done false
3953 1758 Acked Read 7 Sent Done Done false
3954 1759 Acked Write 7 Sent Done Done false
3955 1760 Acked Write 1 Sent Done Done false
3956 1761 Acked Write 10 Sent Done Done false
3957 1762 Acked Write 2 Sent Done Done false
3958 1763 Acked Write 4 Sent Done Done false
3959 1764 Acked Write 1 Sent Done Done false
3960 1765 Acked Write 6 Sent Done Done false
3961 1766 Acked Write 10 Sent Done Done false
3962 1767 Acked Write 10 Sent Done Done false
3963 1768 Acked Read 7 Sent Done Done false
3964 1769 Acked Read 9 Sent Done Done false
3965 1770 Acked Read 1 Sent Done Done false
3966 1771 Acked Flush 0 Sent Done Done false
3967 1772 Acked Write 9 Sent Done Done false
3968 1773 Acked Write 7 Sent Done Done false
3969 1774 Acked Write 9 Sent Done Done false
3970 1775 Acked Write 4 Sent Done Done false
3971 1776 Acked Read 2 Sent Done Done false
3972 1777 Acked Write 1 Sent Done Done false
3973 1778 Acked Write 6 Sent Done Done false
3974 1779 Acked Write 6 Sent Done Done false
3975 1780 Acked Flush 0 Sent Done Done false
3976 1781 Acked Read 2 Sent Done Done false
3977 1782 Acked Read 3 Sent Done Done false
3978 1783 Acked Write 6 Sent Done Done false
3979 1784 Acked Read 8 Sent Done Done false
3980 1785 Acked Read 7 Sent Done Done false
3981 1786 Acked Write 3 Sent Done Done false
3982 1787 Acked Read 7 Sent Done Done false
3983 1788 Acked Write 2 Sent Done Done false
3984 1789 Acked Flush 0 Sent Done Done false
3985 1790 Acked Flush 0 Sent Done Done false
3986 1791 Acked Read 1 Sent Done Done false
3987 1792 Acked Write 2 Sent Done Done false
3988 1793 Acked Write 4 Sent Done Done false
3989 1794 Acked Read 10 Sent Done Done false
3990 1795 Acked Write 6 Sent Done Done false
3991 1796 Acked Read 2 Sent Done Done false
3992 1797 Acked Write 4 Sent Done Done false
3993 1798 Acked Read 1 Sent Done Done false
3994 1799 Acked Write 4 Sent Done Done false
3995 1800 Acked Flush 0 Sent Done Done false
3996 1801 Acked Read 5 Sent Done Done false
3997 1802 Acked Read 7 Sent Done Done false
3998 1803 Acked Flush 0 Sent Done Done false
3999 1804 Acked Read 1 Sent Done Done false
4000 1805 Acked Write 7 Sent Done Done false
4001 1806 Acked Write 2 Sent Done Done false
4002 1807 Acked Flush 0 Sent Done Done false
4003 1808 Acked Flush 0 Sent Done Done false
4004 1809 Acked Read 8 Sent Done Done false
4005 1810 Acked Read 7 Sent Done Done false
4006 1811 Acked Flush 0 Sent Done Done false
4007 1812 Acked Read 2 Sent Done Done false
4008 1813 Acked Read 1 Sent Done Done false
4009 1814 Acked Read 1 Sent Done Done false
4010 1815 Acked Read 7 Sent Done Done false
4011 1816 Acked Write 3 Sent Done Done false
4012 1817 Acked Write 10 Sent Done Done false
4013 1818 Acked Write 6 Sent Done Done false
4014 1819 Acked Write 2 Sent Done Done false
4015 1820 Acked Flush 0 Sent Done Done false
4016 1821 Acked Read 9 Sent Done Done false
4017 1822 Acked Write 1 Sent Done Done false
4018 1823 Acked Read 3 Sent Done Done false
4019 1824 Acked Write 10 Sent Done Done false
4020 1825 Acked Flush 0 Sent Done Done false
4021 1826 Acked Write 7 Sent Done Done false
4022 1827 Acked Read 4 Sent Done Done false
4023 1828 Acked Read 9 Sent Done Done false
4024 1829 Acked Read 6 Sent Done Done false
4025 1830 Acked Write 10 Sent Done Done false
4026 1831 Acked Read 6 Sent Done Done false
4027 1832 Acked Flush 0 Sent Done Done false
4028 1833 Acked Write 3 Sent Done Done false
4029 1834 Acked Read 3 Sent Done Done false
4030 1835 Acked Read 4 Sent Done Done false
4031 1836 Acked Write 6 Sent Done Done false
4032 1837 Acked Read 7 Sent Done Done false
4033 1838 Acked Read 10 Sent Done Done false
4034 1839 Acked Read 1 Sent Done Done false
4035 1840 Acked Write 9 Sent Done Done false
4036 1841 Acked Flush 0 Sent Done Done false
4037 1842 Acked Write 3 Sent Done Done false
4038 1843 Acked Write 8 Sent Done Done false
4039 1844 Acked Read 1 Sent Done Done false
4040 1845 Acked Read 2 Sent Done Done false
4041 1846 Acked Write 10 Sent Done Done false
4042 1847 Acked Write 9 Sent Done Done false
4043 1848 Acked Write 2 Sent Done Done false
4044 1849 Acked Write 8 Sent Done Done false
4045 1850 Acked Write 9 Sent Done Done false
4046 1851 Acked Write 5 Sent Done Done false
4047 1852 Acked Write 3 Sent Done Done false
4048 1853 Acked Read 9 Sent Done Done false
4049 1854 Acked Read 2 Sent Done Done false
4050 1855 Acked Read 1 Sent Done Done false
4051 1856 Acked Read 3 Sent Done Done false
4052 1857 Acked Read 4 Sent Done Done false
4053 1858 Acked Read 6 Sent Done Done false
4054 1859 Acked Read 3 Sent Done Done false
4055 1860 Acked Read 10 Sent Done Done false
4056 1861 Acked Read 10 Sent Done Done false
4057 1862 Acked Read 1 Sent Done Done false
4058 1863 Acked Write 8 Sent Done Done false
4059 1864 Acked Write 10 Sent Done Done false
4060 1865 Acked Write 6 Sent Done Done false
4061 1866 Acked Read 8 Sent Done Done false
4062 1867 Acked Flush 0 Sent Done Done false
4063 1868 Acked Write 8 Sent Done Done false
4064 1869 Acked Read 5 Sent Done Done false
4065 1870 Acked Flush 0 Sent Done Done false
4066 1871 Acked Read 10 Sent Done Done false
4067 1872 Acked Flush 0 Sent Done Done false
4068 1873 Acked Read 6 Sent Done Done false
4069 1874 Acked Write 4 Sent Done Done false
4070 1875 Acked Write 5 Sent Done Done false
4071 1876 Acked Read 6 Sent Done Done false
4072 1877 Acked Read 6 Sent Done Done false
4073 1878 Acked Read 10 Sent Done Done false
4074 1879 Acked Read 8 Sent Done Done false
4075 1880 Acked Read 2 Sent Done Done false
4076 1881 Acked Read 7 Sent Done Done false
4077 1882 Acked Read 9 Sent Done Done false
4078 1883 Acked Write 8 Sent Done Done false
4079 1884 Acked Read 2 Sent Done Done false
4080 1885 Acked Write 3 Sent Done Done false
4081 1886 Acked Write 4 Sent Done Done false
4082 1887 Acked Read 3 Sent Done Done false
4083 1888 Acked Read 6 Sent Done Done false
4084 1889 Acked Write 9 Sent Done Done false
4085 1890 Acked Read 8 Sent Done Done false
4086 1891 Acked Write 8 Sent Done Done false
4087 1892 Acked Write 3 Sent Done Done false
4088 1893 Acked Read 7 Sent Done Done false
4089 1894 Acked Write 8 Sent Done Done false
4090 1895 Acked Read 5 Sent Done Done false
4091 1896 Acked Write 7 Sent Done Done false
4092 1897 Acked Write 2 Sent Done Done false
4093 1898 Acked Write 7 Sent Done Done false
4094 1899 Acked Read 8 Sent Done Done false
4095 1900 Acked Read 5 Sent Done Done false
4096 1901 Acked Write 1 Sent Done Done false
4097 1902 Acked Flush 0 Sent Done Done false
4098 1903 Acked Read 1 Sent Done Done false
4099 STATES DS:0 DS:1 DS:2 TOTAL
4100 Sent 301 0 0 301
4101 Done 0 301 301 602
4102 Skipped 0 0 0 0
4103 Error 0 0 0 0
4104 Last Flush: Some(JobId(1602)) Some(JobId(1902)) Some(JobId(1902))
4105 Downstairs last five completed: 1602 1601 1600 1599 1598
4106 Upstairs last five completed: 1903 1902 1901 1900 1899
4107 CLIENT: Up:0 ds:301 act:2
41082025-03-07T18:13:26.034ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
41092025-03-07T18:13:27.251ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
41102025-03-07T18:13:28.038ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
41112025-03-07T18:13:29.040ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
4112 ----------------------------------------------------------------
4113 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4114 States: Offline Active Active
4115 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4116 1603 Acked Read 8 Sent Done Done false
4117 1604 Acked Read 9 Sent Done Done false
4118 1605 Acked Read 4 Sent Done Done false
4119 1606 Acked Write 9 Sent Done Done false
4120 1607 Acked Flush 0 Sent Done Done false
4121 1608 Acked Read 7 Sent Done Done false
4122 1609 Acked Flush 0 Sent Done Done false
4123 1610 Acked Write 4 Sent Done Done false
4124 1611 Acked Write 5 Sent Done Done false
4125 1612 Acked Write 4 Sent Done Done false
4126 1613 Acked Read 9 Sent Done Done false
4127 1614 Acked Read 7 Sent Done Done false
4128 1615 Acked Read 2 Sent Done Done false
4129 1616 Acked Read 5 Sent Done Done false
4130 1617 Acked Write 2 Sent Done Done false
4131 1618 Acked Write 5 Sent Done Done false
4132 1619 Acked Read 2 Sent Done Done false
4133 1620 Acked Write 2 Sent Done Done false
4134 1621 Acked Read 3 Sent Done Done false
4135 1622 Acked Write 10 Sent Done Done false
4136 1623 Acked Read 10 Sent Done Done false
4137 1624 Acked Read 1 Sent Done Done false
4138 1625 Acked Write 1 Sent Done Done false
4139 1626 Acked Read 8 Sent Done Done false
4140 1627 Acked Read 8 Sent Done Done false
4141 1628 Acked Write 6 Sent Done Done false
4142 1629 Acked Read 6 Sent Done Done false
4143 1630 Acked Read 10 Sent Done Done false
4144 1631 Acked Read 2 Sent Done Done false
4145 1632 Acked Write 3 Sent Done Done false
4146 1633 Acked Write 7 Sent Done Done false
4147 1634 Acked Read 3 Sent Done Done false
4148 1635 Acked Read 5 Sent Done Done false
4149 1636 Acked Write 10 Sent Done Done false
4150 1637 Acked Read 3 Sent Done Done false
4151 1638 Acked Write 7 Sent Done Done false
4152 1639 Acked Write 9 Sent Done Done false
4153 1640 Acked Write 7 Sent Done Done false
4154 1641 Acked Read 9 Sent Done Done false
4155 1642 Acked Read 2 Sent Done Done false
4156 1643 Acked Write 9 Sent Done Done false
4157 1644 Acked Read 4 Sent Done Done false
4158 1645 Acked Read 1 Sent Done Done false
4159 1646 Acked Read 10 Sent Done Done false
4160 1647 Acked Read 1 Sent Done Done false
4161 1648 Acked Read 7 Sent Done Done false
4162 1649 Acked Read 10 Sent Done Done false
4163 1650 Acked Flush 0 Sent Done Done false
4164 1651 Acked Write 3 Sent Done Done false
4165 1652 Acked Write 1 Sent Done Done false
4166 1653 Acked Read 3 Sent Done Done false
4167 1654 Acked Write 3 Sent Done Done false
4168 1655 Acked Flush 0 Sent Done Done false
4169 1656 Acked Write 3 Sent Done Done false
4170 1657 Acked Read 4 Sent Done Done false
4171 1658 Acked Write 6 Sent Done Done false
4172 1659 Acked Write 6 Sent Done Done false
4173 1660 Acked Read 9 Sent Done Done false
4174 1661 Acked Read 1 Sent Done Done false
4175 1662 Acked Flush 0 Sent Done Done false
4176 1663 Acked Write 3 Sent Done Done false
4177 1664 Acked Write 10 Sent Done Done false
4178 1665 Acked Read 6 Sent Done Done false
4179 1666 Acked Flush 0 Sent Done Done false
4180 1667 Acked Write 3 Sent Done Done false
4181 1668 Acked Write 8 Sent Done Done false
4182 1669 Acked Read 9 Sent Done Done false
4183 1670 Acked Flush 0 Sent Done Done false
4184 1671 Acked Read 2 Sent Done Done false
4185 1672 Acked Flush 0 Sent Done Done false
4186 1673 Acked Read 9 Sent Done Done false
4187 1674 Acked Write 2 Sent Done Done false
4188 1675 Acked Write 6 Sent Done Done false
4189 1676 Acked Flush 0 Sent Done Done false
4190 1677 Acked Read 9 Sent Done Done false
4191 1678 Acked Write 5 Sent Done Done false
4192 1679 Acked Write 8 Sent Done Done false
4193 1680 Acked Write 1 Sent Done Done false
4194 1681 Acked Read 8 Sent Done Done false
4195 1682 Acked Read 9 Sent Done Done false
4196 1683 Acked Read 3 Sent Done Done false
4197 1684 Acked Write 9 Sent Done Done false
4198 1685 Acked Read 5 Sent Done Done false
4199 1686 Acked Write 9 Sent Done Done false
4200 1687 Acked Write 1 Sent Done Done false
4201 1688 Acked Flush 0 Sent Done Done false
4202 1689 Acked Flush 0 Sent Done Done false
4203 1690 Acked Write 8 Sent Done Done false
4204 1691 Acked Flush 0 Sent Done Done false
4205 1692 Acked Read 8 Sent Done Done false
4206 1693 Acked Write 5 Sent Done Done false
4207 1694 Acked Read 6 Sent Done Done false
4208 1695 Acked Write 5 Sent Done Done false
4209 1696 Acked Read 8 Sent Done Done false
4210 1697 Acked Read 4 Sent Done Done false
4211 1698 Acked Write 7 Sent Done Done false
4212 1699 Acked Write 3 Sent Done Done false
4213 1700 Acked Read 5 Sent Done Done false
4214 1701 Acked Read 7 Sent Done Done false
4215 1702 Acked Write 3 Sent Done Done false
4216 1703 Acked Read 8 Sent Done Done false
4217 1704 Acked Read 8 Sent Done Done false
4218 1705 Acked Read 6 Sent Done Done false
4219 1706 Acked Write 1 Sent Done Done false
4220 1707 Acked Read 4 Sent Done Done false
4221 1708 Acked Write 1 Sent Done Done false
4222 1709 Acked Read 6 Sent Done Done false
4223 1710 Acked Read 7 Sent Done Done false
4224 1711 Acked Read 6 Sent Done Done false
4225 1712 Acked Read 9 Sent Done Done false
4226 1713 Acked Read 6 Sent Done Done false
4227 1714 Acked Write 5 Sent Done Done false
4228 1715 Acked Write 8 Sent Done Done false
4229 1716 Acked Write 10 Sent Done Done false
4230 1717 Acked Read 6 Sent Done Done false
4231 1718 Acked Read 3 Sent Done Done false
4232 1719 Acked Read 2 Sent Done Done false
4233 1720 Acked Flush 0 Sent Done Done false
4234 1721 Acked Read 4 Sent Done Done false
4235 1722 Acked Read 3 Sent Done Done false
4236 1723 Acked Read 7 Sent Done Done false
4237 1724 Acked Read 8 Sent Done Done false
4238 1725 Acked Write 5 Sent Done Done false
4239 1726 Acked Read 4 Sent Done Done false
4240 1727 Acked Read 3 Sent Done Done false
4241 1728 Acked Read 7 Sent Done Done false
4242 1729 Acked Write 9 Sent Done Done false
4243 1730 Acked Write 10 Sent Done Done false
4244 1731 Acked Read 8 Sent Done Done false
4245 1732 Acked Write 4 Sent Done Done false
4246 1733 Acked Write 1 Sent Done Done false
4247 1734 Acked Read 9 Sent Done Done false
4248 1735 Acked Read 6 Sent Done Done false
4249 1736 Acked Read 4 Sent Done Done false
4250 1737 Acked Read 2 Sent Done Done false
4251 1738 Acked Read 1 Sent Done Done false
4252 1739 Acked Flush 0 Sent Done Done false
4253 1740 Acked Read 3 Sent Done Done false
4254 1741 Acked Write 3 Sent Done Done false
4255 1742 Acked Write 9 Sent Done Done false
4256 1743 Acked Read 4 Sent Done Done false
4257 1744 Acked Read 1 Sent Done Done false
4258 1745 Acked Write 1 Sent Done Done false
4259 1746 Acked Read 6 Sent Done Done false
4260 1747 Acked Write 3 Sent Done Done false
4261 1748 Acked Read 6 Sent Done Done false
4262 1749 Acked Write 3 Sent Done Done false
4263 1750 Acked Flush 0 Sent Done Done false
4264 1751 Acked Write 4 Sent Done Done false
4265 1752 Acked Read 9 Sent Done Done false
4266 1753 Acked Read 5 Sent Done Done false
4267 1754 Acked Write 4 Sent Done Done false
4268 1755 Acked Write 10 Sent Done Done false
4269 1756 Acked Read 8 Sent Done Done false
4270 1757 Acked Read 1 Sent Done Done false
4271 1758 Acked Read 7 Sent Done Done false
4272 1759 Acked Write 7 Sent Done Done false
4273 1760 Acked Write 1 Sent Done Done false
4274 1761 Acked Write 10 Sent Done Done false
4275 1762 Acked Write 2 Sent Done Done false
4276 1763 Acked Write 4 Sent Done Done false
4277 1764 Acked Write 1 Sent Done Done false
4278 1765 Acked Write 6 Sent Done Done false
4279 1766 Acked Write 10 Sent Done Done false
4280 1767 Acked Write 10 Sent Done Done false
4281 1768 Acked Read 7 Sent Done Done false
4282 1769 Acked Read 9 Sent Done Done false
4283 1770 Acked Read 1 Sent Done Done false
4284 1771 Acked Flush 0 Sent Done Done false
4285 1772 Acked Write 9 Sent Done Done false
4286 1773 Acked Write 7 Sent Done Done false
4287 1774 Acked Write 9 Sent Done Done false
4288 1775 Acked Write 4 Sent Done Done false
4289 1776 Acked Read 2 Sent Done Done false
4290 1777 Acked Write 1 Sent Done Done false
4291 1778 Acked Write 6 Sent Done Done false
4292 1779 Acked Write 6 Sent Done Done false
4293 1780 Acked Flush 0 Sent Done Done false
4294 1781 Acked Read 2 Sent Done Done false
4295 1782 Acked Read 3 Sent Done Done false
4296 1783 Acked Write 6 Sent Done Done false
4297 1784 Acked Read 8 Sent Done Done false
4298 1785 Acked Read 7 Sent Done Done false
4299 1786 Acked Write 3 Sent Done Done false
4300 1787 Acked Read 7 Sent Done Done false
4301 1788 Acked Write 2 Sent Done Done false
4302 1789 Acked Flush 0 Sent Done Done false
4303 1790 Acked Flush 0 Sent Done Done false
4304 1791 Acked Read 1 Sent Done Done false
4305 1792 Acked Write 2 Sent Done Done false
4306 1793 Acked Write 4 Sent Done Done false
4307 1794 Acked Read 10 Sent Done Done false
4308 1795 Acked Write 6 Sent Done Done false
4309 1796 Acked Read 2 Sent Done Done false
4310 1797 Acked Write 4 Sent Done Done false
4311 1798 Acked Read 1 Sent Done Done false
4312 1799 Acked Write 4 Sent Done Done false
4313 1800 Acked Flush 0 Sent Done Done false
4314 1801 Acked Read 5 Sent Done Done false
4315 1802 Acked Read 7 Sent Done Done false
4316 1803 Acked Flush 0 Sent Done Done false
4317 1804 Acked Read 1 Sent Done Done false
4318 1805 Acked Write 7 Sent Done Done false
4319 1806 Acked Write 2 Sent Done Done false
4320 1807 Acked Flush 0 Sent Done Done false
4321 1808 Acked Flush 0 Sent Done Done false
4322 1809 Acked Read 8 Sent Done Done false
4323 1810 Acked Read 7 Sent Done Done false
4324 1811 Acked Flush 0 Sent Done Done false
4325 1812 Acked Read 2 Sent Done Done false
4326 1813 Acked Read 1 Sent Done Done false
4327 1814 Acked Read 1 Sent Done Done false
4328 1815 Acked Read 7 Sent Done Done false
4329 1816 Acked Write 3 Sent Done Done false
4330 1817 Acked Write 10 Sent Done Done false
4331 1818 Acked Write 6 Sent Done Done false
4332 1819 Acked Write 2 Sent Done Done false
4333 1820 Acked Flush 0 Sent Done Done false
4334 1821 Acked Read 9 Sent Done Done false
4335 1822 Acked Write 1 Sent Done Done false
4336 1823 Acked Read 3 Sent Done Done false
4337 1824 Acked Write 10 Sent Done Done false
4338 1825 Acked Flush 0 Sent Done Done false
4339 1826 Acked Write 7 Sent Done Done false
4340 1827 Acked Read 4 Sent Done Done false
4341 1828 Acked Read 9 Sent Done Done false
4342 1829 Acked Read 6 Sent Done Done false
4343 1830 Acked Write 10 Sent Done Done false
4344 1831 Acked Read 6 Sent Done Done false
4345 1832 Acked Flush 0 Sent Done Done false
4346 1833 Acked Write 3 Sent Done Done false
4347 1834 Acked Read 3 Sent Done Done false
4348 1835 Acked Read 4 Sent Done Done false
4349 1836 Acked Write 6 Sent Done Done false
4350 1837 Acked Read 7 Sent Done Done false
4351 1838 Acked Read 10 Sent Done Done false
4352 1839 Acked Read 1 Sent Done Done false
4353 1840 Acked Write 9 Sent Done Done false
4354 1841 Acked Flush 0 Sent Done Done false
4355 1842 Acked Write 3 Sent Done Done false
4356 1843 Acked Write 8 Sent Done Done false
4357 1844 Acked Read 1 Sent Done Done false
4358 1845 Acked Read 2 Sent Done Done false
4359 1846 Acked Write 10 Sent Done Done false
4360 1847 Acked Write 9 Sent Done Done false
4361 1848 Acked Write 2 Sent Done Done false
4362 1849 Acked Write 8 Sent Done Done false
4363 1850 Acked Write 9 Sent Done Done false
4364 1851 Acked Write 5 Sent Done Done false
4365 1852 Acked Write 3 Sent Done Done false
4366 1853 Acked Read 9 Sent Done Done false
4367 1854 Acked Read 2 Sent Done Done false
4368 1855 Acked Read 1 Sent Done Done false
4369 1856 Acked Read 3 Sent Done Done false
4370 1857 Acked Read 4 Sent Done Done false
4371 1858 Acked Read 6 Sent Done Done false
4372 1859 Acked Read 3 Sent Done Done false
4373 1860 Acked Read 10 Sent Done Done false
4374 1861 Acked Read 10 Sent Done Done false
4375 1862 Acked Read 1 Sent Done Done false
4376 1863 Acked Write 8 Sent Done Done false
4377 1864 Acked Write 10 Sent Done Done false
4378 1865 Acked Write 6 Sent Done Done false
4379 1866 Acked Read 8 Sent Done Done false
4380 1867 Acked Flush 0 Sent Done Done false
4381 1868 Acked Write 8 Sent Done Done false
4382 1869 Acked Read 5 Sent Done Done false
4383 1870 Acked Flush 0 Sent Done Done false
4384 1871 Acked Read 10 Sent Done Done false
4385 1872 Acked Flush 0 Sent Done Done false
4386 1873 Acked Read 6 Sent Done Done false
4387 1874 Acked Write 4 Sent Done Done false
4388 1875 Acked Write 5 Sent Done Done false
4389 1876 Acked Read 6 Sent Done Done false
4390 1877 Acked Read 6 Sent Done Done false
4391 1878 Acked Read 10 Sent Done Done false
4392 1879 Acked Read 8 Sent Done Done false
4393 1880 Acked Read 2 Sent Done Done false
4394 1881 Acked Read 7 Sent Done Done false
4395 1882 Acked Read 9 Sent Done Done false
4396 1883 Acked Write 8 Sent Done Done false
4397 1884 Acked Read 2 Sent Done Done false
4398 1885 Acked Write 3 Sent Done Done false
4399 1886 Acked Write 4 Sent Done Done false
4400 1887 Acked Read 3 Sent Done Done false
4401 1888 Acked Read 6 Sent Done Done false
4402 1889 Acked Write 9 Sent Done Done false
4403 1890 Acked Read 8 Sent Done Done false
4404 1891 Acked Write 8 Sent Done Done false
4405 1892 Acked Write 3 Sent Done Done false
4406 1893 Acked Read 7 Sent Done Done false
4407 1894 Acked Write 8 Sent Done Done false
4408 1895 Acked Read 5 Sent Done Done false
4409 1896 Acked Write 7 Sent Done Done false
4410 1897 Acked Write 2 Sent Done Done false
4411 1898 Acked Write 7 Sent Done Done false
4412 1899 Acked Read 8 Sent Done Done false
4413 1900 Acked Read 5 Sent Done Done false
4414 1901 Acked Write 1 Sent Done Done false
4415 1902 Acked Flush 0 Sent Done Done false
4416 1903 Acked Read 1 Sent Done Done false
4417 STATES DS:0 DS:1 DS:2 TOTAL
4418 Sent 301 0 0 301
4419 Done 0 301 301 602
4420 Skipped 0 0 0 0
4421 Error 0 0 0 0
4422 Last Flush: Some(JobId(1602)) Some(JobId(1902)) Some(JobId(1902))
4423 Downstairs last five completed: 1602 1601 1600 1599 1598
4424 Upstairs last five completed: 1903 1902 1901 1900 1899
4425 CLIENT: Up:0 ds:301 act:2
44262025-03-07T18:13:30.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44272025-03-07T18:13:31.013ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44282025-03-07T18:13:31.013ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44292025-03-07T18:13:31.013ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44302025-03-07T18:13:31.013ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44312025-03-07T18:13:31.014ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
44322025-03-07T18:13:32.252ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
4433 [8810] got start action
4434 [8810] I am going to restart
4435 Starting downstairs at port 8810
4436 Make output file at "/tmp/dsc/downstairs-8810.txt"
4437 [8810][0] reports Starting
4438 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1211)
4439 [8810][0] reports Running
44402025-03-07T18:13:33.047ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
4441 ----------------------------------------------------------------
4442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4443 States: Offline Active Active
4444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4445 1603 Acked Read 8 Sent Done Done false
4446 1604 Acked Read 9 Sent Done Done false
4447 1605 Acked Read 4 Sent Done Done false
4448 1606 Acked Write 9 Sent Done Done false
4449 1607 Acked Flush 0 Sent Done Done false
4450 1608 Acked Read 7 Sent Done Done false
4451 1609 Acked Flush 0 Sent Done Done false
4452 1610 Acked Write 4 Sent Done Done false
4453 1611 Acked Write 5 Sent Done Done false
4454 1612 Acked Write 4 Sent Done Done false
4455 1613 Acked Read 9 Sent Done Done false
4456 1614 Acked Read 7 Sent Done Done false
4457 1615 Acked Read 2 Sent Done Done false
4458 1616 Acked Read 5 Sent Done Done false
4459 1617 Acked Write 2 Sent Done Done false
4460 1618 Acked Write 5 Sent Done Done false
4461 1619 Acked Read 2 Sent Done Done false
4462 1620 Acked Write 2 Sent Done Done false
4463 1621 Acked Read 3 Sent Done Done false
4464 1622 Acked Write 10 Sent Done Done false
4465 1623 Acked Read 10 Sent Done Done false
4466 1624 Acked Read 1 Sent Done Done false
4467 1625 Acked Write 1 Sent Done Done false
4468 1626 Acked Read 8 Sent Done Done false
4469 1627 Acked Read 8 Sent Done Done false
4470 1628 Acked Write 6 Sent Done Done false
4471 1629 Acked Read 6 Sent Done Done false
4472 1630 Acked Read 10 Sent Done Done false
4473 1631 Acked Read 2 Sent Done Done false
4474 1632 Acked Write 3 Sent Done Done false
4475 1633 Acked Write 7 Sent Done Done false
4476 1634 Acked Read 3 Sent Done Done false
4477 1635 Acked Read 5 Sent Done Done false
4478 1636 Acked Write 10 Sent Done Done false
4479 1637 Acked Read 3 Sent Done Done false
4480 1638 Acked Write 7 Sent Done Done false
4481 1639 Acked Write 9 Sent Done Done false
4482 1640 Acked Write 7 Sent Done Done false
4483 1641 Acked Read 9 Sent Done Done false
4484 1642 Acked Read 2 Sent Done Done false
4485 1643 Acked Write 9 Sent Done Done false
4486 1644 Acked Read 4 Sent Done Done false
4487 1645 Acked Read 1 Sent Done Done false
4488 1646 Acked Read 10 Sent Done Done false
4489 1647 Acked Read 1 Sent Done Done false
4490 1648 Acked Read 7 Sent Done Done false
4491 1649 Acked Read 10 Sent Done Done false
4492 1650 Acked Flush 0 Sent Done Done false
4493 1651 Acked Write 3 Sent Done Done false
4494 1652 Acked Write 1 Sent Done Done false
4495 1653 Acked Read 3 Sent Done Done false
4496 1654 Acked Write 3 Sent Done Done false
4497 1655 Acked Flush 0 Sent Done Done false
4498 1656 Acked Write 3 Sent Done Done false
4499 1657 Acked Read 4 Sent Done Done false
4500 1658 Acked Write 6 Sent Done Done false
4501 1659 Acked Write 6 Sent Done Done false
4502 1660 Acked Read 9 Sent Done Done false
4503 1661 Acked Read 1 Sent Done Done false
4504 1662 Acked Flush 0 Sent Done Done false
4505 1663 Acked Write 3 Sent Done Done false
4506 1664 Acked Write 10 Sent Done Done false
4507 1665 Acked Read 6 Sent Done Done false
4508 1666 Acked Flush 0 Sent Done Done false
4509 1667 Acked Write 3 Sent Done Done false
4510 1668 Acked Write 8 Sent Done Done false
4511 1669 Acked Read 9 Sent Done Done false
4512 1670 Acked Flush 0 Sent Done Done false
4513 1671 Acked Read 2 Sent Done Done false
4514 1672 Acked Flush 0 Sent Done Done false
4515 1673 Acked Read 9 Sent Done Done false
4516 1674 Acked Write 2 Sent Done Done false
4517 1675 Acked Write 6 Sent Done Done false
4518 1676 Acked Flush 0 Sent Done Done false
4519 1677 Acked Read 9 Sent Done Done false
4520 1678 Acked Write 5 Sent Done Done false
4521 1679 Acked Write 8 Sent Done Done false
4522 1680 Acked Write 1 Sent Done Done false
4523 1681 Acked Read 8 Sent Done Done false
4524 1682 Acked Read 9 Sent Done Done false
4525 1683 Acked Read 3 Sent Done Done false
4526 1684 Acked Write 9 Sent Done Done false
4527 1685 Acked Read 5 Sent Done Done false
4528 1686 Acked Write 9 Sent Done Done false
4529 1687 Acked Write 1 Sent Done Done false
4530 1688 Acked Flush 0 Sent Done Done false
4531 1689 Acked Flush 0 Sent Done Done false
4532 1690 Acked Write 8 Sent Done Done false
4533 1691 Acked Flush 0 Sent Done Done false
4534 1692 Acked Read 8 Sent Done Done false
4535 1693 Acked Write 5 Sent Done Done false
4536 1694 Acked Read 6 Sent Done Done false
4537 1695 Acked Write 5 Sent Done Done false
4538 1696 Acked Read 8 Sent Done Done false
4539 1697 Acked Read 4 Sent Done Done false
4540 1698 Acked Write 7 Sent Done Done false
4541 1699 Acked Write 3 Sent Done Done false
4542 1700 Acked Read 5 Sent Done Done false
4543 1701 Acked Read 7 Sent Done Done false
4544 1702 Acked Write 3 Sent Done Done false
4545 1703 Acked Read 8 Sent Done Done false
4546 1704 Acked Read 8 Sent Done Done false
4547 1705 Acked Read 6 Sent Done Done false
4548 1706 Acked Write 1 Sent Done Done false
4549 1707 Acked Read 4 Sent Done Done false
4550 1708 Acked Write 1 Sent Done Done false
4551 1709 Acked Read 6 Sent Done Done false
4552 1710 Acked Read 7 Sent Done Done false
4553 1711 Acked Read 6 Sent Done Done false
4554 1712 Acked Read 9 Sent Done Done false
4555 1713 Acked Read 6 Sent Done Done false
4556 1714 Acked Write 5 Sent Done Done false
4557 1715 Acked Write 8 Sent Done Done false
4558 1716 Acked Write 10 Sent Done Done false
4559 1717 Acked Read 6 Sent Done Done false
4560 1718 Acked Read 3 Sent Done Done false
4561 1719 Acked Read 2 Sent Done Done false
4562 1720 Acked Flush 0 Sent Done Done false
4563 1721 Acked Read 4 Sent Done Done false
4564 1722 Acked Read 3 Sent Done Done false
4565 1723 Acked Read 7 Sent Done Done false
4566 1724 Acked Read 8 Sent Done Done false
4567 1725 Acked Write 5 Sent Done Done false
4568 1726 Acked Read 4 Sent Done Done false
4569 1727 Acked Read 3 Sent Done Done false
4570 1728 Acked Read 7 Sent Done Done false
4571 1729 Acked Write 9 Sent Done Done false
4572 1730 Acked Write 10 Sent Done Done false
4573 1731 Acked Read 8 Sent Done Done false
4574 1732 Acked Write 4 Sent Done Done false
4575 1733 Acked Write 1 Sent Done Done false
4576 1734 Acked Read 9 Sent Done Done false
4577 1735 Acked Read 6 Sent Done Done false
4578 1736 Acked Read 4 Sent Done Done false
4579 1737 Acked Read 2 Sent Done Done false
4580 1738 Acked Read 1 Sent Done Done false
4581 1739 Acked Flush 0 Sent Done Done false
4582 1740 Acked Read 3 Sent Done Done false
4583 1741 Acked Write 3 Sent Done Done false
4584 1742 Acked Write 9 Sent Done Done false
4585 1743 Acked Read 4 Sent Done Done false
4586 1744 Acked Read 1 Sent Done Done false
4587 1745 Acked Write 1 Sent Done Done false
4588 1746 Acked Read 6 Sent Done Done false
4589 1747 Acked Write 3 Sent Done Done false
4590 1748 Acked Read 6 Sent Done Done false
4591 1749 Acked Write 3 Sent Done Done false
4592 1750 Acked Flush 0 Sent Done Done false
4593 1751 Acked Write 4 Sent Done Done false
4594 1752 Acked Read 9 Sent Done Done false
4595 1753 Acked Read 5 Sent Done Done false
4596 1754 Acked Write 4 Sent Done Done false
4597 1755 Acked Write 10 Sent Done Done false
4598 1756 Acked Read 8 Sent Done Done false
4599 1757 Acked Read 1 Sent Done Done false
4600 1758 Acked Read 7 Sent Done Done false
4601 1759 Acked Write 7 Sent Done Done false
4602 1760 Acked Write 1 Sent Done Done false
4603 1761 Acked Write 10 Sent Done Done false
4604 1762 Acked Write 2 Sent Done Done false
4605 1763 Acked Write 4 Sent Done Done false
4606 1764 Acked Write 1 Sent Done Done false
4607 1765 Acked Write 6 Sent Done Done false
4608 1766 Acked Write 10 Sent Done Done false
4609 1767 Acked Write 10 Sent Done Done false
4610 1768 Acked Read 7 Sent Done Done false
4611 1769 Acked Read 9 Sent Done Done false
4612 1770 Acked Read 1 Sent Done Done false
4613 1771 Acked Flush 0 Sent Done Done false
4614 1772 Acked Write 9 Sent Done Done false
4615 1773 Acked Write 7 Sent Done Done false
4616 1774 Acked Write 9 Sent Done Done false
4617 1775 Acked Write 4 Sent Done Done false
4618 1776 Acked Read 2 Sent Done Done false
4619 1777 Acked Write 1 Sent Done Done false
4620 1778 Acked Write 6 Sent Done Done false
4621 1779 Acked Write 6 Sent Done Done false
4622 1780 Acked Flush 0 Sent Done Done false
4623 1781 Acked Read 2 Sent Done Done false
4624 1782 Acked Read 3 Sent Done Done false
4625 1783 Acked Write 6 Sent Done Done false
4626 1784 Acked Read 8 Sent Done Done false
4627 1785 Acked Read 7 Sent Done Done false
4628 1786 Acked Write 3 Sent Done Done false
4629 1787 Acked Read 7 Sent Done Done false
4630 1788 Acked Write 2 Sent Done Done false
4631 1789 Acked Flush 0 Sent Done Done false
4632 1790 Acked Flush 0 Sent Done Done false
4633 1791 Acked Read 1 Sent Done Done false
4634 1792 Acked Write 2 Sent Done Done false
4635 1793 Acked Write 4 Sent Done Done false
4636 1794 Acked Read 10 Sent Done Done false
4637 1795 Acked Write 6 Sent Done Done false
4638 1796 Acked Read 2 Sent Done Done false
4639 1797 Acked Write 4 Sent Done Done false
4640 1798 Acked Read 1 Sent Done Done false
4641 1799 Acked Write 4 Sent Done Done false
4642 1800 Acked Flush 0 Sent Done Done false
4643 1801 Acked Read 5 Sent Done Done false
4644 1802 Acked Read 7 Sent Done Done false
4645 1803 Acked Flush 0 Sent Done Done false
4646 1804 Acked Read 1 Sent Done Done false
4647 1805 Acked Write 7 Sent Done Done false
4648 1806 Acked Write 2 Sent Done Done false
4649 1807 Acked Flush 0 Sent Done Done false
4650 1808 Acked Flush 0 Sent Done Done false
4651 1809 Acked Read 8 Sent Done Done false
4652 1810 Acked Read 7 Sent Done Done false
4653 1811 Acked Flush 0 Sent Done Done false
4654 1812 Acked Read 2 Sent Done Done false
4655 1813 Acked Read 1 Sent Done Done false
4656 1814 Acked Read 1 Sent Done Done false
4657 1815 Acked Read 7 Sent Done Done false
4658 1816 Acked Write 3 Sent Done Done false
4659 1817 Acked Write 10 Sent Done Done false
4660 1818 Acked Write 6 Sent Done Done false
4661 1819 Acked Write 2 Sent Done Done false
4662 1820 Acked Flush 0 Sent Done Done false
4663 1821 Acked Read 9 Sent Done Done false
4664 1822 Acked Write 1 Sent Done Done false
4665 1823 Acked Read 3 Sent Done Done false
4666 1824 Acked Write 10 Sent Done Done false
4667 1825 Acked Flush 0 Sent Done Done false
4668 1826 Acked Write 7 Sent Done Done false
4669 1827 Acked Read 4 Sent Done Done false
4670 1828 Acked Read 9 Sent Done Done false
4671 1829 Acked Read 6 Sent Done Done false
4672 1830 Acked Write 10 Sent Done Done false
4673 1831 Acked Read 6 Sent Done Done false
4674 1832 Acked Flush 0 Sent Done Done false
4675 1833 Acked Write 3 Sent Done Done false
4676 1834 Acked Read 3 Sent Done Done false
4677 1835 Acked Read 4 Sent Done Done false
4678 1836 Acked Write 6 Sent Done Done false
4679 1837 Acked Read 7 Sent Done Done false
4680 1838 Acked Read 10 Sent Done Done false
4681 1839 Acked Read 1 Sent Done Done false
4682 1840 Acked Write 9 Sent Done Done false
4683 1841 Acked Flush 0 Sent Done Done false
4684 1842 Acked Write 3 Sent Done Done false
4685 1843 Acked Write 8 Sent Done Done false
4686 1844 Acked Read 1 Sent Done Done false
4687 1845 Acked Read 2 Sent Done Done false
4688 1846 Acked Write 10 Sent Done Done false
4689 1847 Acked Write 9 Sent Done Done false
4690 1848 Acked Write 2 Sent Done Done false
4691 1849 Acked Write 8 Sent Done Done false
4692 1850 Acked Write 9 Sent Done Done false
4693 1851 Acked Write 5 Sent Done Done false
4694 1852 Acked Write 3 Sent Done Done false
4695 1853 Acked Read 9 Sent Done Done false
4696 1854 Acked Read 2 Sent Done Done false
4697 1855 Acked Read 1 Sent Done Done false
4698 1856 Acked Read 3 Sent Done Done false
4699 1857 Acked Read 4 Sent Done Done false
4700 1858 Acked Read 6 Sent Done Done false
4701 1859 Acked Read 3 Sent Done Done false
4702 1860 Acked Read 10 Sent Done Done false
4703 1861 Acked Read 10 Sent Done Done false
4704 1862 Acked Read 1 Sent Done Done false
4705 1863 Acked Write 8 Sent Done Done false
4706 1864 Acked Write 10 Sent Done Done false
4707 1865 Acked Write 6 Sent Done Done false
4708 1866 Acked Read 8 Sent Done Done false
4709 1867 Acked Flush 0 Sent Done Done false
4710 1868 Acked Write 8 Sent Done Done false
4711 1869 Acked Read 5 Sent Done Done false
4712 1870 Acked Flush 0 Sent Done Done false
4713 1871 Acked Read 10 Sent Done Done false
4714 1872 Acked Flush 0 Sent Done Done false
4715 1873 Acked Read 6 Sent Done Done false
4716 1874 Acked Write 4 Sent Done Done false
4717 1875 Acked Write 5 Sent Done Done false
4718 1876 Acked Read 6 Sent Done Done false
4719 1877 Acked Read 6 Sent Done Done false
4720 1878 Acked Read 10 Sent Done Done false
4721 1879 Acked Read 8 Sent Done Done false
4722 1880 Acked Read 2 Sent Done Done false
4723 1881 Acked Read 7 Sent Done Done false
4724 1882 Acked Read 9 Sent Done Done false
4725 1883 Acked Write 8 Sent Done Done false
4726 1884 Acked Read 2 Sent Done Done false
4727 1885 Acked Write 3 Sent Done Done false
4728 1886 Acked Write 4 Sent Done Done false
4729 1887 Acked Read 3 Sent Done Done false
4730 1888 Acked Read 6 Sent Done Done false
4731 1889 Acked Write 9 Sent Done Done false
4732 1890 Acked Read 8 Sent Done Done false
4733 1891 Acked Write 8 Sent Done Done false
4734 1892 Acked Write 3 Sent Done Done false
4735 1893 Acked Read 7 Sent Done Done false
4736 1894 Acked Write 8 Sent Done Done false
4737 1895 Acked Read 5 Sent Done Done false
4738 1896 Acked Write 7 Sent Done Done false
4739 1897 Acked Write 2 Sent Done Done false
4740 1898 Acked Write 7 Sent Done Done false
4741 1899 Acked Read 8 Sent Done Done false
4742 1900 Acked Read 5 Sent Done Done false
4743 1901 Acked Write 1 Sent Done Done false
4744 1902 Acked Flush 0 Sent Done Done false
4745 1903 Acked Read 1 Sent Done Done false
4746 STATES DS:0 DS:1 DS:2 TOTAL
4747 Sent 301 0 0 301
4748 Done 0 301 301 602
4749 Skipped 0 0 0 0
4750 Error 0 0 0 0
4751 Last Flush: Some(JobId(1602)) Some(JobId(1902)) Some(JobId(1902))
4752 Downstairs last five completed: 1602 1601 1600 1599 1598
4753 Upstairs last five completed: 1903 1902 1901 1900 1899
4754 CLIENT: Up:0 ds:301 act:2
47552025-03-07T18:13:34.048ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
47562025-03-07T18:13:35.048ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
47572025-03-07T18:13:36.050ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
47582025-03-07T18:13:37.253ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
4759 ----------------------------------------------------------------
4760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4761 States: Offline Active Active
4762 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4763 1603 Acked Read 8 Sent Done Done false
4764 1604 Acked Read 9 Sent Done Done false
4765 1605 Acked Read 4 Sent Done Done false
4766 1606 Acked Write 9 Sent Done Done false
4767 1607 Acked Flush 0 Sent Done Done false
4768 1608 Acked Read 7 Sent Done Done false
4769 1609 Acked Flush 0 Sent Done Done false
4770 1610 Acked Write 4 Sent Done Done false
4771 1611 Acked Write 5 Sent Done Done false
4772 1612 Acked Write 4 Sent Done Done false
4773 1613 Acked Read 9 Sent Done Done false
4774 1614 Acked Read 7 Sent Done Done false
4775 1615 Acked Read 2 Sent Done Done false
4776 1616 Acked Read 5 Sent Done Done false
4777 1617 Acked Write 2 Sent Done Done false
4778 1618 Acked Write 5 Sent Done Done false
4779 1619 Acked Read 2 Sent Done Done false
4780 1620 Acked Write 2 Sent Done Done false
4781 1621 Acked Read 3 Sent Done Done false
4782 1622 Acked Write 10 Sent Done Done false
4783 1623 Acked Read 10 Sent Done Done false
4784 1624 Acked Read 1 Sent Done Done false
4785 1625 Acked Write 1 Sent Done Done false
4786 1626 Acked Read 8 Sent Done Done false
4787 1627 Acked Read 8 Sent Done Done false
4788 1628 Acked Write 6 Sent Done Done false
4789 1629 Acked Read 6 Sent Done Done false
4790 1630 Acked Read 10 Sent Done Done false
4791 1631 Acked Read 2 Sent Done Done false
4792 1632 Acked Write 3 Sent Done Done false
4793 1633 Acked Write 7 Sent Done Done false
4794 1634 Acked Read 3 Sent Done Done false
4795 1635 Acked Read 5 Sent Done Done false
4796 1636 Acked Write 10 Sent Done Done false
4797 1637 Acked Read 3 Sent Done Done false
4798 1638 Acked Write 7 Sent Done Done false
4799 1639 Acked Write 9 Sent Done Done false
4800 1640 Acked Write 7 Sent Done Done false
4801 1641 Acked Read 9 Sent Done Done false
4802 1642 Acked Read 2 Sent Done Done false
4803 1643 Acked Write 9 Sent Done Done false
4804 1644 Acked Read 4 Sent Done Done false
4805 1645 Acked Read 1 Sent Done Done false
4806 1646 Acked Read 10 Sent Done Done false
4807 1647 Acked Read 1 Sent Done Done false
4808 1648 Acked Read 7 Sent Done Done false
4809 1649 Acked Read 10 Sent Done Done false
4810 1650 Acked Flush 0 Sent Done Done false
4811 1651 Acked Write 3 Sent Done Done false
4812 1652 Acked Write 1 Sent Done Done false
4813 1653 Acked Read 3 Sent Done Done false
4814 1654 Acked Write 3 Sent Done Done false
4815 1655 Acked Flush 0 Sent Done Done false
4816 1656 Acked Write 3 Sent Done Done false
4817 1657 Acked Read 4 Sent Done Done false
4818 1658 Acked Write 6 Sent Done Done false
4819 1659 Acked Write 6 Sent Done Done false
4820 1660 Acked Read 9 Sent Done Done false
4821 1661 Acked Read 1 Sent Done Done false
4822 1662 Acked Flush 0 Sent Done Done false
4823 1663 Acked Write 3 Sent Done Done false
4824 1664 Acked Write 10 Sent Done Done false
4825 1665 Acked Read 6 Sent Done Done false
4826 1666 Acked Flush 0 Sent Done Done false
4827 1667 Acked Write 3 Sent Done Done false
4828 1668 Acked Write 8 Sent Done Done false
4829 1669 Acked Read 9 Sent Done Done false
4830 1670 Acked Flush 0 Sent Done Done false
4831 1671 Acked Read 2 Sent Done Done false
4832 1672 Acked Flush 0 Sent Done Done false
4833 1673 Acked Read 9 Sent Done Done false
4834 1674 Acked Write 2 Sent Done Done false
4835 1675 Acked Write 6 Sent Done Done false
4836 1676 Acked Flush 0 Sent Done Done false
4837 1677 Acked Read 9 Sent Done Done false
4838 1678 Acked Write 5 Sent Done Done false
4839 1679 Acked Write 8 Sent Done Done false
4840 1680 Acked Write 1 Sent Done Done false
4841 1681 Acked Read 8 Sent Done Done false
4842 1682 Acked Read 9 Sent Done Done false
4843 1683 Acked Read 3 Sent Done Done false
4844 1684 Acked Write 9 Sent Done Done false
4845 1685 Acked Read 5 Sent Done Done false
4846 1686 Acked Write 9 Sent Done Done false
4847 1687 Acked Write 1 Sent Done Done false
4848 1688 Acked Flush 0 Sent Done Done false
4849 1689 Acked Flush 0 Sent Done Done false
4850 1690 Acked Write 8 Sent Done Done false
4851 1691 Acked Flush 0 Sent Done Done false
4852 1692 Acked Read 8 Sent Done Done false
4853 1693 Acked Write 5 Sent Done Done false
4854 1694 Acked Read 6 Sent Done Done false
4855 1695 Acked Write 5 Sent Done Done false
4856 1696 Acked Read 8 Sent Done Done false
4857 1697 Acked Read 4 Sent Done Done false
4858 1698 Acked Write 7 Sent Done Done false
4859 1699 Acked Write 3 Sent Done Done false
4860 1700 Acked Read 5 Sent Done Done false
4861 1701 Acked Read 7 Sent Done Done false
4862 1702 Acked Write 3 Sent Done Done false
4863 1703 Acked Read 8 Sent Done Done false
4864 1704 Acked Read 8 Sent Done Done false
4865 1705 Acked Read 6 Sent Done Done false
4866 1706 Acked Write 1 Sent Done Done false
4867 1707 Acked Read 4 Sent Done Done false
4868 1708 Acked Write 1 Sent Done Done false
4869 1709 Acked Read 6 Sent Done Done false
4870 1710 Acked Read 7 Sent Done Done false
4871 1711 Acked Read 6 Sent Done Done false
4872 1712 Acked Read 9 Sent Done Done false
4873 1713 Acked Read 6 Sent Done Done false
4874 1714 Acked Write 5 Sent Done Done false
4875 1715 Acked Write 8 Sent Done Done false
4876 1716 Acked Write 10 Sent Done Done false
4877 1717 Acked Read 6 Sent Done Done false
4878 1718 Acked Read 3 Sent Done Done false
4879 1719 Acked Read 2 Sent Done Done false
4880 1720 Acked Flush 0 Sent Done Done false
4881 1721 Acked Read 4 Sent Done Done false
4882 1722 Acked Read 3 Sent Done Done false
4883 1723 Acked Read 7 Sent Done Done false
4884 1724 Acked Read 8 Sent Done Done false
4885 1725 Acked Write 5 Sent Done Done false
4886 1726 Acked Read 4 Sent Done Done false
4887 1727 Acked Read 3 Sent Done Done false
4888 1728 Acked Read 7 Sent Done Done false
4889 1729 Acked Write 9 Sent Done Done false
4890 1730 Acked Write 10 Sent Done Done false
4891 1731 Acked Read 8 Sent Done Done false
4892 1732 Acked Write 4 Sent Done Done false
4893 1733 Acked Write 1 Sent Done Done false
4894 1734 Acked Read 9 Sent Done Done false
4895 1735 Acked Read 6 Sent Done Done false
4896 1736 Acked Read 4 Sent Done Done false
4897 1737 Acked Read 2 Sent Done Done false
4898 1738 Acked Read 1 Sent Done Done false
4899 1739 Acked Flush 0 Sent Done Done false
4900 1740 Acked Read 3 Sent Done Done false
4901 1741 Acked Write 3 Sent Done Done false
4902 1742 Acked Write 9 Sent Done Done false
4903 1743 Acked Read 4 Sent Done Done false
4904 1744 Acked Read 1 Sent Done Done false
4905 1745 Acked Write 1 Sent Done Done false
4906 1746 Acked Read 6 Sent Done Done false
4907 1747 Acked Write 3 Sent Done Done false
4908 1748 Acked Read 6 Sent Done Done false
4909 1749 Acked Write 3 Sent Done Done false
4910 1750 Acked Flush 0 Sent Done Done false
4911 1751 Acked Write 4 Sent Done Done false
4912 1752 Acked Read 9 Sent Done Done false
4913 1753 Acked Read 5 Sent Done Done false
4914 1754 Acked Write 4 Sent Done Done false
4915 1755 Acked Write 10 Sent Done Done false
4916 1756 Acked Read 8 Sent Done Done false
4917 1757 Acked Read 1 Sent Done Done false
4918 1758 Acked Read 7 Sent Done Done false
4919 1759 Acked Write 7 Sent Done Done false
4920 1760 Acked Write 1 Sent Done Done false
4921 1761 Acked Write 10 Sent Done Done false
4922 1762 Acked Write 2 Sent Done Done false
4923 1763 Acked Write 4 Sent Done Done false
4924 1764 Acked Write 1 Sent Done Done false
4925 1765 Acked Write 6 Sent Done Done false
4926 1766 Acked Write 10 Sent Done Done false
4927 1767 Acked Write 10 Sent Done Done false
4928 1768 Acked Read 7 Sent Done Done false
4929 1769 Acked Read 9 Sent Done Done false
4930 1770 Acked Read 1 Sent Done Done false
4931 1771 Acked Flush 0 Sent Done Done false
4932 1772 Acked Write 9 Sent Done Done false
4933 1773 Acked Write 7 Sent Done Done false
4934 1774 Acked Write 9 Sent Done Done false
4935 1775 Acked Write 4 Sent Done Done false
4936 1776 Acked Read 2 Sent Done Done false
4937 1777 Acked Write 1 Sent Done Done false
4938 1778 Acked Write 6 Sent Done Done false
4939 1779 Acked Write 6 Sent Done Done false
4940 1780 Acked Flush 0 Sent Done Done false
4941 1781 Acked Read 2 Sent Done Done false
4942 1782 Acked Read 3 Sent Done Done false
4943 1783 Acked Write 6 Sent Done Done false
4944 1784 Acked Read 8 Sent Done Done false
4945 1785 Acked Read 7 Sent Done Done false
4946 1786 Acked Write 3 Sent Done Done false
4947 1787 Acked Read 7 Sent Done Done false
4948 1788 Acked Write 2 Sent Done Done false
4949 1789 Acked Flush 0 Sent Done Done false
4950 1790 Acked Flush 0 Sent Done Done false
4951 1791 Acked Read 1 Sent Done Done false
4952 1792 Acked Write 2 Sent Done Done false
4953 1793 Acked Write 4 Sent Done Done false
4954 1794 Acked Read 10 Sent Done Done false
4955 1795 Acked Write 6 Sent Done Done false
4956 1796 Acked Read 2 Sent Done Done false
4957 1797 Acked Write 4 Sent Done Done false
4958 1798 Acked Read 1 Sent Done Done false
4959 1799 Acked Write 4 Sent Done Done false
4960 1800 Acked Flush 0 Sent Done Done false
4961 1801 Acked Read 5 Sent Done Done false
4962 1802 Acked Read 7 Sent Done Done false
4963 1803 Acked Flush 0 Sent Done Done false
4964 1804 Acked Read 1 Sent Done Done false
4965 1805 Acked Write 7 Sent Done Done false
4966 1806 Acked Write 2 Sent Done Done false
4967 1807 Acked Flush 0 Sent Done Done false
4968 1808 Acked Flush 0 Sent Done Done false
4969 1809 Acked Read 8 Sent Done Done false
4970 1810 Acked Read 7 Sent Done Done false
4971 1811 Acked Flush 0 Sent Done Done false
4972 1812 Acked Read 2 Sent Done Done false
4973 1813 Acked Read 1 Sent Done Done false
4974 1814 Acked Read 1 Sent Done Done false
4975 1815 Acked Read 7 Sent Done Done false
4976 1816 Acked Write 3 Sent Done Done false
4977 1817 Acked Write 10 Sent Done Done false
4978 1818 Acked Write 6 Sent Done Done false
4979 1819 Acked Write 2 Sent Done Done false
4980 1820 Acked Flush 0 Sent Done Done false
4981 1821 Acked Read 9 Sent Done Done false
4982 1822 Acked Write 1 Sent Done Done false
4983 1823 Acked Read 3 Sent Done Done false
4984 1824 Acked Write 10 Sent Done Done false
4985 1825 Acked Flush 0 Sent Done Done false
4986 1826 Acked Write 7 Sent Done Done false
4987 1827 Acked Read 4 Sent Done Done false
4988 1828 Acked Read 9 Sent Done Done false
4989 1829 Acked Read 6 Sent Done Done false
4990 1830 Acked Write 10 Sent Done Done false
4991 1831 Acked Read 6 Sent Done Done false
4992 1832 Acked Flush 0 Sent Done Done false
4993 1833 Acked Write 3 Sent Done Done false
4994 1834 Acked Read 3 Sent Done Done false
4995 1835 Acked Read 4 Sent Done Done false
4996 1836 Acked Write 6 Sent Done Done false
4997 1837 Acked Read 7 Sent Done Done false
4998 1838 Acked Read 10 Sent Done Done false
4999 1839 Acked Read 1 Sent Done Done false
5000 1840 Acked Write 9 Sent Done Done false
5001 1841 Acked Flush 0 Sent Done Done false
5002 1842 Acked Write 3 Sent Done Done false
5003 1843 Acked Write 8 Sent Done Done false
5004 1844 Acked Read 1 Sent Done Done false
5005 1845 Acked Read 2 Sent Done Done false
5006 1846 Acked Write 10 Sent Done Done false
5007 1847 Acked Write 9 Sent Done Done false
5008 1848 Acked Write 2 Sent Done Done false
5009 1849 Acked Write 8 Sent Done Done false
5010 1850 Acked Write 9 Sent Done Done false
5011 1851 Acked Write 5 Sent Done Done false
5012 1852 Acked Write 3 Sent Done Done false
5013 1853 Acked Read 9 Sent Done Done false
5014 1854 Acked Read 2 Sent Done Done false
5015 1855 Acked Read 1 Sent Done Done false
5016 1856 Acked Read 3 Sent Done Done false
5017 1857 Acked Read 4 Sent Done Done false
5018 1858 Acked Read 6 Sent Done Done false
5019 1859 Acked Read 3 Sent Done Done false
5020 1860 Acked Read 10 Sent Done Done false
5021 1861 Acked Read 10 Sent Done Done false
5022 1862 Acked Read 1 Sent Done Done false
5023 1863 Acked Write 8 Sent Done Done false
5024 1864 Acked Write 10 Sent Done Done false
5025 1865 Acked Write 6 Sent Done Done false
5026 1866 Acked Read 8 Sent Done Done false
5027 1867 Acked Flush 0 Sent Done Done false
5028 1868 Acked Write 8 Sent Done Done false
5029 1869 Acked Read 5 Sent Done Done false
5030 1870 Acked Flush 0 Sent Done Done false
5031 1871 Acked Read 10 Sent Done Done false
5032 1872 Acked Flush 0 Sent Done Done false
5033 1873 Acked Read 6 Sent Done Done false
5034 1874 Acked Write 4 Sent Done Done false
5035 1875 Acked Write 5 Sent Done Done false
5036 1876 Acked Read 6 Sent Done Done false
5037 1877 Acked Read 6 Sent Done Done false
5038 1878 Acked Read 10 Sent Done Done false
5039 1879 Acked Read 8 Sent Done Done false
5040 1880 Acked Read 2 Sent Done Done false
5041 1881 Acked Read 7 Sent Done Done false
5042 1882 Acked Read 9 Sent Done Done false
5043 1883 Acked Write 8 Sent Done Done false
5044 1884 Acked Read 2 Sent Done Done false
5045 1885 Acked Write 3 Sent Done Done false
5046 1886 Acked Write 4 Sent Done Done false
5047 1887 Acked Read 3 Sent Done Done false
5048 1888 Acked Read 6 Sent Done Done false
5049 1889 Acked Write 9 Sent Done Done false
5050 1890 Acked Read 8 Sent Done Done false
5051 1891 Acked Write 8 Sent Done Done false
5052 1892 Acked Write 3 Sent Done Done false
5053 1893 Acked Read 7 Sent Done Done false
5054 1894 Acked Write 8 Sent Done Done false
5055 1895 Acked Read 5 Sent Done Done false
5056 1896 Acked Write 7 Sent Done Done false
5057 1897 Acked Write 2 Sent Done Done false
5058 1898 Acked Write 7 Sent Done Done false
5059 1899 Acked Read 8 Sent Done Done false
5060 1900 Acked Read 5 Sent Done Done false
5061 1901 Acked Write 1 Sent Done Done false
5062 1902 Acked Flush 0 Sent Done Done false
5063 1903 Acked Read 1 Sent Done Done false
5064 STATES DS:0 DS:1 DS:2 TOTAL
5065 Sent 301 0 0 301
5066 Done 0 301 301 602
5067 Skipped 0 0 0 0
5068 Error 0 0 0 0
5069 Last Flush: Some(JobId(1602)) Some(JobId(1902)) Some(JobId(1902))
5070 Downstairs last five completed: 1602 1601 1600 1599 1598
5071 Upstairs last five completed: 1903 1902 1901 1900 1899
5072 CLIENT: Up:0 ds:301 act:2
50732025-03-07T18:13:38.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50742025-03-07T18:13:39.056ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50752025-03-07T18:13:40.058ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50762025-03-07T18:13:41.014ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50772025-03-07T18:13:41.015ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39922) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50782025-03-07T18:13:41.016ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50792025-03-07T18:13:41.016ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50802025-03-07T18:13:41.016ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50812025-03-07T18:13:41.016ZINFOcrucible: send last flush ID to this DS: Some(JobId(1602)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50822025-03-07T18:13:41.016ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50832025-03-07T18:13:41.016ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1602)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
50842025-03-07T18:13:41.017ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1602)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
5085 ----------------------------------------------------------------
5086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:1
5087 States: Active Active Active
5088 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5089 1903 Acked Read 1 Done Done Done true
5090 STATES DS:0 DS:1 DS:2 TOTAL
5091 Sent 0 0 0 0
5092 Done 1 1 1 3
5093 Skipped 0 0 0 0
5094 Error 0 0 0 0
5095 Last Flush: Some(JobId(1902)) Some(JobId(1902)) Some(JobId(1902))
5096 Downstairs last five completed: 1902 1901 1900 1899 1898
5097 Upstairs last five completed: 1903 1902 1901 1900 1899
5098 CLIENT: Up:0 ds:1 act:3
5099 ----------------------------------------------------------------
5100 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5101 Downstairs last five completed: 1904 1903 1902 1901 1900
5102 Upstairs last five completed: 1904 1903 1902 1901 1900
5103 CLIENT: Up:0 ds:0 act:3
5104 Replay: All jobs finished
5105 Main task has work to do, go find it
5106 got dsc Stop(0)
5107 stop 0
5108 Mar 07 18:13:45.266 INFO request completed, latency_us: 352, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1a6bb404-8768-4e1d-964e-598bc06b5539, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
5109 [8810] Got stop action so:false kr:false
5110 Mar 07 18:13:45.267 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/0, method: GET, req_id: d4a6d73b-aa4a-4937-91f3-02d4d662343d, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
51112025-03-07T18:13:45.267ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
51122025-03-07T18:13:45.267ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
51132025-03-07T18:13:45.267ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
51142025-03-07T18:13:45.267ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
5115 [8810] Exited with: ExitStatus(unix_wait_status(9))
5116 [8810][0] reports Exit
5117 Mar 07 18:13:49.270 INFO request completed, latency_us: 290, response_code: 200, uri: /state/cid/0, method: GET, req_id: 117a3490-d833-4c87-ad0f-5ef9c9841a09, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
5118 001/300 Read block 3418 len 36864
5119 002/300 Write block 618 len 40960 data: 2 3 3 3 3 3 3 3 3 3
5120 003/300 Write block 2767 len 8192 data: 3 3
5121 004/300 Write block 3052 len 8192 data: 2 3
5122 005/300 Write block 4765 len 36864 data: 4 4 4 4 3 3 3 3 2
5123 006/300 Read block 558 len 36864
5124 007/300 Flush
5125 008/300 Write block 4180 len 4096 data: 2
5126 009/300 Write block 3250 len 8192 data: 2 2
5127 010/300 Write block 3782 len 12288 data: 2 2 2
5128 011/300 Read block 669 len 28672
5129 012/300 Read block 1247 len 36864
5130 013/300 Read block 3735 len 32768
5131 014/300 Read block 544 len 28672
5132 015/300 Read block 1991 len 40960
5133 016/300 Read block 1427 len 36864
5134 017/300 Write block 4640 len 24576 data: 2 2 2 2 2 2
5135 018/300 Read block 3955 len 40960
5136 019/300 Write block 2787 len 32768 data: 2 2 2 2 2 2 2 2
5137 020/300 Read block 3141 len 40960
5138 021/300 Write block 2257 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5139 022/300 Write block 2110 len 16384 data: 2 2 2 2
5140 023/300 Write block 2335 len 32768 data: 2 2 2 2 2 2 2 2
5141 024/300 Read block 3525 len 16384
5142 025/300 Write block 1046 len 32768 data: 2 2 2 2 2 2 2 2
5143 026/300 Flush
5144 027/300 Write block 544 len 16384 data: 2 2 2 2
5145 028/300 Read block 3875 len 8192
5146 029/300 Write block 786 len 36864 data: 2 2 2 2 2 2 2 2 2
5147 030/300 Read block 2951 len 24576
5148 031/300 Read block 3092 len 40960
5149 032/300 Write block 2493 len 32768 data: 3 5 4 4 4 4 3 2
5150 033/300 Read block 4579 len 32768
5151 034/300 Read block 3931 len 8192
5152 035/300 Read block 4117 len 8192
5153 036/300 Write block 587 len 4096 data: 2
5154 037/300 Read block 3335 len 28672
5155 038/300 Flush
5156 039/300 Write block 3882 len 20480 data: 2 2 2 2 2
5157 040/300 Write block 180 len 4096 data: 3
5158 041/300 Write block 1870 len 4096 data: 3
5159 042/300 Write block 1983 len 32768 data: 3 3 2 2 2 2 2 2
5160 043/300 Write block 2314 len 4096 data: 3
5161 044/300 Read block 3341 len 16384
5162 045/300 Read block 4210 len 36864
5163 046/300 Write block 2226 len 12288 data: 2 2 2
5164 047/300 Write block 733 len 32768 data: 2 2 3 3 3 3 2 2
5165 048/300 Read block 2471 len 20480
5166 049/300 Flush
5167 050/300 Read block 1512 len 32768
5168 051/300 Flush
5169 052/300 Flush
5170 053/300 Write block 3269 len 8192 data: 3 2
5171 054/300 Write block 1694 len 20480 data: 3 3 3 3 2
5172 055/300 Write block 1922 len 36864 data: 3 3 2 2 2 2 2 2 2
5173 056/300 Write block 703 len 16384 data: 2 2 2 2
5174 057/300 Flush
5175 058/300 Flush
5176 059/300 Read block 1235 len 20480
5177 060/300 Read block 4121 len 8192
5178 061/300 Write block 2532 len 36864 data: 2 2 2 2 2 2 2 3 3
5179 062/300 Read block 1067 len 16384
5180 063/300 Read block 2529 len 40960
5181 064/300 Read block 3704 len 36864
5182 065/300 Read block 2378 len 20480
5183 066/300 Flush
5184 067/300 Write block 3566 len 20480 data: 3 4 3 3 2
5185 068/300 Read block 4157 len 16384
5186 069/300 Read block 4431 len 36864
5187 070/300 Read block 3053 len 24576
5188 071/300 Read block 3516 len 20480
5189 072/300 Write block 1178 len 36864 data: 2 2 2 2 2 2 2 2 2
5190 073/300 Read block 4190 len 16384
5191 074/300 Write block 3980 len 20480 data: 2 2 2 2 2
5192 075/300 Read block 1909 len 40960
5193 076/300 Write block 1619 len 36864 data: 3 2 2 2 2 2 2 2 2
5194 077/300 Write block 3143 len 20480 data: 2 2 2 2 2
5195 078/300 Write block 3797 len 32768 data: 4 3 2 2 2 2 2 2
5196 079/300 Write block 2671 len 32768 data: 2 2 2 2 2 2 2 2
5197 080/300 Read block 2145 len 12288
5198 081/300 Read block 4673 len 28672
5199 082/300 Read block 1195 len 32768
5200 083/300 Read block 2909 len 28672
5201 084/300 Read block 1506 len 20480
5202 085/300 Read block 2645 len 36864
5203 086/300 Read block 3088 len 32768
5204 087/300 Flush
5205 088/300 Write block 2875 len 28672 data: 2 2 2 2 2 2 2
5206 089/300 Write block 4791 len 36864 data: 3 3 3 3 2 2 2 2 2
5207 090/300 Write block 3683 len 28672 data: 2 2 2 2 2 2 2
5208 091/300 Write block 1643 len 8192 data: 2 2
5209 092/300 Read block 2128 len 28672
5210 093/300 Read block 3520 len 40960
5211 094/300 Read block 2235 len 40960
5212 095/300 Write block 2588 len 12288 data: 2 2 2
5213 096/300 Write block 3644 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5214 097/300 Read block 2849 len 36864
5215 098/300 Read block 3255 len 24576
5216 099/300 Read block 3069 len 36864
5217 100/300 Read block 4153 len 8192
5218 101/300 Read block 1172 len 20480
5219 102/300 Write block 4248 len 36864 data: 2 2 2 2 2 2 3 3 3
5220 103/300 Read block 1285 len 40960
5221 104/300 Write block 4602 len 8192 data: 2 2
5222 105/300 Write block 4313 len 24576 data: 2 2 2 2 2 2
5223 106/300 Read block 2856 len 32768
5224 107/300 Read block 3729 len 16384
5225 108/300 Read block 4469 len 20480
5226 109/300 Read block 4767 len 20480
5227 110/300 Read block 2591 len 8192
5228 111/300 Write block 2943 len 28672 data: 4 4 4 4 4 4 4
5229 112/300 Write block 768 len 32768 data: 2 2 2 2 2 2 2 2
5230 113/300 Read block 3955 len 24576
5231 114/300 Read block 3688 len 28672
5232 115/300 Read block 114 len 8192
5233 116/300 Write block 3937 len 20480 data: 2 3 3 3 4
5234 117/300 Read block 4711 len 36864
5235 118/300 Read block 3557 len 28672
5236 119/300 Read block 3380 len 8192
5237 120/300 Write block 282 len 36864 data: 2 3 3 2 2 2 2 2 2
5238 121/300 Read block 680 len 4096
5239 122/300 Write block 614 len 12288 data: 2 2 2
5240 123/300 Read block 3015 len 32768
5241 124/300 Write block 95 len 32768 data: 2 2 2 2 2 2 2 2
5242 125/300 Read block 439 len 36864
5243 126/300 Write block 1914 len 4096 data: 2
5244 127/300 Read block 1270 len 36864
5245 128/300 Write block 4636 len 12288 data: 2 2 2
5246 129/300 Write block 3428 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5247 130/300 Read block 2900 len 28672
5248 131/300 Read block 4814 len 20480
5249 132/300 Write block 1823 len 20480 data: 2 2 2 2 2
5250 133/300 Read block 124 len 8192
5251 134/300 Write block 2062 len 24576 data: 2 3 3 3 3 2
5252 135/300 Write block 2466 len 32768 data: 3 3 3 2 2 2 2 2
5253 136/300 Write block 271 len 8192 data: 2 2
5254 137/300 Write block 7 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5255 138/300 Flush
5256 139/300 Write block 1086 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5257 140/300 Flush
5258 141/300 Read block 1685 len 28672
5259 142/300 Write block 1004 len 20480 data: 2 2 2 3 3
5260 143/300 Read block 2650 len 36864
5261 144/300 Read block 4556 len 20480
5262 145/300 Read block 1128 len 12288
5263 146/300 Read block 2809 len 40960
5264 147/300 Flush
5265 148/300 Read block 4988 len 16384
5266 149/300 Read block 4813 len 12288
5267 150/300 Write block 322 len 8192 data: 2 2
5268 151/300 Write block 1291 len 16384 data: 2 2 2 3
5269 152/300 Read block 3953 len 20480
5270 153/300 Write block 3812 len 4096 data: 2
5271 154/300 Write block 1728 len 36864 data: 4 3 2 2 2 2 2 2 2
5272 155/300 Read block 4712 len 20480
5273 156/300 Read block 4225 len 28672
5274 157/300 Write block 2281 len 24576 data: 3 3 3 3 3 3
5275 158/300 Read block 4343 len 4096
5276 159/300 Read block 677 len 40960
5277 160/300 Read block 3115 len 28672
5278 161/300 Write block 1836 len 16384 data: 2 3 3 3
5279 162/300 Write block 3845 len 36864 data: 2 2 2 2 2 2 2 2 3
5280 163/300 Read block 4666 len 4096
5281 164/300 Write block 4542 len 32768 data: 3 3 3 4 4 4 4 3
5282 165/300 Read block 2535 len 12288
5283 166/300 Read block 12 len 12288
5284 167/300 Write block 4042 len 24576 data: 2 2 2 2 2 2
5285 168/300 Write block 4494 len 4096 data: 2
5286 169/300 Write block 305 len 24576 data: 3 3 5 6 6 5
5287 170/300 Write block 4786 len 28672 data: 2 3 3 3 3 4 4
5288 171/300 Read block 619 len 8192
5289 172/300 Read block 1420 len 24576
5290 173/300 Read block 1034 len 28672
5291 174/300 Write block 378 len 12288 data: 2 2 2
5292 175/300 Flush
5293 176/300 Read block 4688 len 8192
5294 177/300 Read block 3941 len 20480
5295 178/300 Read block 2133 len 16384
5296 179/300 Write block 4284 len 36864 data: 4 3 3 3 3 3 3 3 3
5297 180/300 Flush
5298 181/300 Write block 215 len 36864 data: 5 4 2 2 2 2 2 2 2
5299 182/300 Read block 188 len 32768
5300 183/300 Write block 4644 len 20480 data: 3 3 2 2 2
5301 184/300 Read block 2719 len 32768
5302 185/300 Read block 2335 len 16384
5303 186/300 Write block 664 len 40960 data: 3 3 2 2 2 2 2 2 2 3
5304 187/300 Read block 3622 len 20480
5305 188/300 Read block 2080 len 32768
5306 189/300 Read block 4979 len 20480
5307 190/300 Read block 4809 len 4096
5308 191/300 Write block 1649 len 24576 data: 2 2 2 2 2 2
5309 192/300 Read block 3278 len 40960
5310 193/300 Write block 663 len 40960 data: 4 4 4 3 3 3 3 3 3 3
5311 194/300 Read block 3819 len 12288
5312 195/300 Read block 117 len 20480
5313 196/300 Write block 1319 len 40960 data: 2 2 2 2 2 2 2 2 2 3
5314 197/300 Write block 70 len 40960 data: 2 2 3 3 3 3 3 3 3 3
5315 198/300 Write block 1009 len 36864 data: 3 3 3 3 3 3 3 3 2
5316 199/300 Read block 1824 len 32768
5317 200/300 Write block 4278 len 24576 data: 2 3 3 3 3 4
5318 201/300 Flush
5319 202/300 Write block 3679 len 16384 data: 2 2 2 2
5320 203/300 Read block 3939 len 36864
5321 204/300 Write block 326 len 12288 data: 4 4 4
5322 205/300 Read block 3707 len 36864
5323 206/300 Read block 2708 len 28672
5324 207/300 Read block 3686 len 16384
5325 208/300 Read block 3143 len 40960
5326 209/300 Read block 892 len 36864
5327 210/300 Read block 952 len 12288
5328 211/300 Write block 3233 len 16384 data: 3 2 2 2
5329 212/300 Write block 3988 len 20480 data: 2 2 3 3 3
5330 213/300 Write block 4524 len 20480 data: 3 2 2 2 2
5331 214/300 Flush
5332 215/300 Flush
5333 216/300 Write block 4958 len 12288 data: 2 2 2
5334 217/300 Flush
5335 218/300 Write block 2672 len 4096 data: 3
5336 219/300 Read block 1581 len 16384
5337 220/300 Write block 1798 len 4096 data: 2
5338 221/300 Read block 4026 len 28672
5339 222/300 Write block 2586 len 16384 data: 3 2 3 3
5340 223/300 Flush
5341 224/300 Read block 3991 len 12288
5342 225/300 Write block 4562 len 4096 data: 3
5343 226/300 Write block 1121 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5344 227/300 Flush
5345 228/300 Write block 1614 len 24576 data: 4 4 4 3 3 4
5346 229/300 Write block 3033 len 20480 data: 2 2 2 2 2
5347 230/300 Flush
5348 231/300 Read block 92 len 32768
5349 232/300 Read block 4404 len 8192
5350 233/300 Flush
5351 234/300 Read block 2643 len 8192
5352 235/300 Write block 3261 len 24576 data: 2 2 2 2 2 4
5353 236/300 Read block 4769 len 8192
5354 237/300 Flush
5355 238/300 Write block 2315 len 4096 data: 2
5356 239/300 Write block 4221 len 8192 data: 3 3
5357 240/300 Read block 4566 len 32768
5358 241/300 Write block 758 len 28672 data: 3 3 3 3 3 3 3
5359 242/300 Read block 1919 len 24576
5360 243/300 Read block 628 len 36864
5361 244/300 Read block 2358 len 12288
5362 245/300 Flush
5363 246/300 Read block 1196 len 8192
5364 247/300 Flush
5365 248/300 Write block 1922 len 40960 data: 4 4 3 3 3 3 3 3 3 2
5366 249/300 Write block 631 len 8192 data: 2 2
5367 250/300 Read block 3840 len 8192
5368 251/300 Write block 2581 len 12288 data: 3 3 3
5369 252/300 Read block 1408 len 16384
5370 253/300 Read block 938 len 12288
5371 254/300 Read block 2410 len 16384
5372 255/300 Write block 4696 len 16384 data: 2 2 2 2
5373 256/300 Read block 768 len 12288
5374 257/300 Read block 3269 len 4096
5375 258/300 Write block 2016 len 12288 data: 2 2 2
5376 259/300 Write block 1450 len 24576 data: 2 2 2 2 2 2
5377 260/300 Read block 4879 len 28672
5378 261/300 Read block 951 len 16384
5379 262/300 Read block 3331 len 16384
5380 263/300 Flush
5381 264/300 Write block 3524 len 32768 data: 2 2 2 2 2 2 3 3
5382 265/300 Write block 3582 len 8192 data: 2 2
5383 266/300 Flush
5384 267/300 Read block 57 len 8192
5385 268/300 Write block 3636 len 36864 data: 2 2 2 2 2 2 2 2 3
5386 269/300 Read block 2404 len 12288
5387 270/300 Read block 807 len 32768
5388 271/300 Flush
5389 272/300 Read block 1485 len 24576
5390 273/300 Read block 745 len 20480
5391 274/300 Read block 6 len 36864
5392 275/300 Flush
5393 276/300 Read block 4926 len 12288
5394 277/300 Write block 3057 len 20480 data: 4 4 4 4 4
5395 278/300 Read block 3306 len 40960
5396 279/300 Flush
5397 280/300 Read block 3318 len 4096
5398 281/300 Write block 4807 len 12288 data: 2 2 2
5399 282/300 Write block 4349 len 24576 data: 2 2 2 2 2 2
5400 283/300 Read block 3432 len 36864
5401 284/300 Flush
5402 285/300 Read block 3180 len 36864
5403 286/300 Write block 2147 len 8192 data: 3 3
5404 287/300 Flush
5405 288/300 Write block 1643 len 32768 data: 3 3 2 2 2 2 3 3
5406 289/300 Flush
5407 290/300 Write block 4125 len 20480 data: 3 3 3 3 3
5408 291/300 Read block 9 len 24576
5409 292/300 Read block 2870 len 8192
5410 293/300 Write block 52 len 20480 data: 2 2 2 2 2
5411 294/300 Read block 4577 len 24576
5412 295/300 Write block 3890 len 32768 data: 2 2 2 2 2 2 2 2
5413 296/300 Write block 3640 len 16384 data: 3 3 3 3
5414 297/300 Read block 3223 len 28672
5415 298/300 Read block 4592 len 16384
5416 299/300 Read block 1079 len 4096
5417 300/300 Write block 1095 len 40960 data: 3 2 2 2 2 2 2 2 2 2
5418 301/300 Read block 2724 len 36864
5419 Main task has work to do, go find it
5420 got dsc Start(0)
5421 start 0
5422 Mar 07 18:13:49.520 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/0, method: POST, req_id: 18ce9802-09a1-4c11-b64e-2381ec970710, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
5423 Replay: started 0, returned:Ok(())
5424 ----------------------------------------------------------------
5425 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5426 States: Offline Active Active
5427 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5428 1905 Acked Read 9 Sent Done Done false
5429 1906 Acked Write 10 Sent Done Done false
5430 1907 Acked Write 2 Sent Done Done false
5431 1908 Acked Write 2 Sent Done Done false
5432 1909 Acked Write 9 Sent Done Done false
5433 1910 Acked Read 9 Sent Done Done false
5434 1911 Acked Flush 0 Sent Done Done false
5435 1912 Acked Write 1 Sent Done Done false
5436 1913 Acked Write 2 Sent Done Done false
5437 1914 Acked Write 3 Sent Done Done false
5438 1915 Acked Read 7 Sent Done Done false
5439 1916 Acked Read 9 Sent Done Done false
5440 1917 Acked Read 8 Sent Done Done false
5441 1918 Acked Read 7 Sent Done Done false
5442 1919 Acked Read 10 Sent Done Done false
5443 1920 Acked Read 9 Sent Done Done false
5444 1921 Acked Write 6 Sent Done Done false
5445 1922 Acked Read 10 Sent Done Done false
5446 1923 Acked Write 8 Sent Done Done false
5447 1924 Acked Read 10 Sent Done Done false
5448 1925 Acked Write 10 Sent Done Done false
5449 1926 Acked Write 4 Sent Done Done false
5450 1927 Acked Write 8 Sent Done Done false
5451 1928 Acked Read 4 Sent Done Done false
5452 1929 Acked Write 8 Sent Done Done false
5453 1930 Acked Flush 0 Sent Done Done false
5454 1931 Acked Write 4 Sent Done Done false
5455 1932 Acked Read 2 Sent Done Done false
5456 1933 Acked Write 9 Sent Done Done false
5457 1934 Acked Read 6 Sent Done Done false
5458 1935 Acked Read 10 Sent Done Done false
5459 1936 Acked Write 8 Sent Done Done false
5460 1937 Acked Read 8 Sent Done Done false
5461 1938 Acked Read 2 Sent Done Done false
5462 1939 Acked Read 2 Sent Done Done false
5463 1940 Acked Write 1 Sent Done Done false
5464 1941 Acked Read 7 Sent Done Done false
5465 1942 Acked Flush 0 Sent Done Done false
5466 1943 Acked Write 5 Sent Done Done false
5467 1944 Acked Write 1 Sent Done Done false
5468 1945 Acked Write 1 Sent Done Done false
5469 1946 Acked Write 8 Sent Done Done false
5470 1947 Acked Write 1 Sent Done Done false
5471 1948 Acked Read 4 Sent Done Done false
5472 1949 Acked Read 9 Sent Done Done false
5473 1950 Acked Write 3 Sent Done Done false
5474 1951 Acked Write 8 Sent Done Done false
5475 1952 Acked Read 5 Sent Done Done false
5476 1953 Acked Flush 0 Sent Done Done false
5477 1954 Acked Read 8 Sent Done Done false
5478 1955 Acked Flush 0 Sent Done Done false
5479 1956 Acked Flush 0 Sent Done Done false
5480 1957 Acked Write 2 Sent Done Done false
5481 1958 Acked Write 5 Sent Done Done false
5482 1959 Acked Write 9 Sent Done Done false
5483 1960 Acked Write 4 Sent Done Done false
5484 1961 Acked Flush 0 Sent Done Done false
5485 1962 Acked Flush 0 Sent Done Done false
5486 1963 Acked Read 5 Sent Done Done false
5487 1964 Acked Read 2 Sent Done Done false
5488 1965 Acked Write 9 Sent Done Done false
5489 1966 Acked Read 4 Sent Done Done false
5490 1967 Acked Read 10 Sent Done Done false
5491 1968 Acked Read 9 Sent Done Done false
5492 1969 Acked Read 5 Sent Done Done false
5493 1970 Acked Flush 0 Sent Done Done false
5494 1971 Acked Write 5 Sent Done Done false
5495 1972 Acked Read 4 Sent Done Done false
5496 1973 Acked Read 9 Sent Done Done false
5497 1974 Acked Read 6 Sent Done Done false
5498 1975 Acked Read 5 Sent Done Done false
5499 1976 Acked Write 9 Sent Done Done false
5500 1977 Acked Read 4 Sent Done Done false
5501 1978 Acked Write 5 Sent Done Done false
5502 1979 Acked Read 10 Sent Done Done false
5503 1980 Acked Write 9 Sent Done Done false
5504 1981 Acked Write 5 Sent Done Done false
5505 1982 Acked Write 8 Sent Done Done false
5506 1983 Acked Write 8 Sent Done Done false
5507 1984 Acked Read 3 Sent Done Done false
5508 1985 Acked Read 7 Sent Done Done false
5509 1986 Acked Read 8 Sent Done Done false
5510 1987 Acked Read 7 Sent Done Done false
5511 1988 Acked Read 5 Sent Done Done false
5512 1989 Acked Read 9 Sent Done Done false
5513 1990 Acked Read 8 Sent Done Done false
5514 1991 Acked Flush 0 Sent Done Done false
5515 1992 Acked Write 7 Sent Done Done false
5516 1993 Acked Write 9 Sent Done Done false
5517 1994 Acked Write 7 Sent Done Done false
5518 1995 Acked Write 2 Sent Done Done false
5519 1996 Acked Read 7 Sent Done Done false
5520 1997 Acked Read 10 Sent Done Done false
5521 1998 Acked Read 10 Sent Done Done false
5522 1999 Acked Write 3 Sent Done Done false
5523 2000 Acked Write 10 Sent Done Done false
5524 2001 Acked Read 9 Sent Done Done false
5525 2002 Acked Read 6 Sent Done Done false
5526 2003 Acked Read 9 Sent Done Done false
5527 2004 Acked Read 2 Sent Done Done false
5528 2005 Acked Read 5 Sent Done Done false
5529 2006 Acked Write 9 Sent Done Done false
5530 2007 Acked Read 10 Sent Done Done false
5531 2008 Acked Write 2 Sent Done Done false
5532 2009 Acked Write 6 Sent Done Done false
5533 2010 Acked Read 8 Sent Done Done false
5534 2011 Acked Read 4 Sent Done Done false
5535 2012 Acked Read 5 Sent Done Done false
5536 2013 Acked Read 5 Sent Done Done false
5537 2014 Acked Read 2 Sent Done Done false
5538 2015 Acked Write 7 Sent Done Done false
5539 2016 Acked Write 8 Sent Done Done false
5540 2017 Acked Read 6 Sent Done Done false
5541 2018 Acked Read 7 Sent Done Done false
5542 2019 Acked Read 2 Sent Done Done false
5543 2020 Acked Write 5 Sent Done Done false
5544 2021 Acked Read 9 Sent Done Done false
5545 2022 Acked Read 7 Sent Done Done false
5546 2023 Acked Read 2 Sent Done Done false
5547 2024 Acked Write 9 Sent Done Done false
5548 2025 Acked Read 1 Sent Done Done false
5549 2026 Acked Write 3 Sent Done Done false
5550 2027 Acked Read 8 Sent Done Done false
5551 2028 Acked Write 8 Sent Done Done false
5552 2029 Acked Read 9 Sent Done Done false
5553 2030 Acked Write 1 Sent Done Done false
5554 2031 Acked Read 9 Sent Done Done false
5555 2032 Acked Write 3 Sent Done Done false
5556 2033 Acked Write 10 Sent Done Done false
5557 2034 Acked Read 7 Sent Done Done false
5558 2035 Acked Read 5 Sent Done Done false
5559 2036 Acked Write 5 Sent Done Done false
5560 2037 Acked Read 2 Sent Done Done false
5561 2038 Acked Write 6 Sent Done Done false
5562 2039 Acked Write 8 Sent Done Done false
5563 2040 Acked Write 2 Sent Done Done false
5564 2041 Acked Write 10 Sent Done Done false
5565 2042 Acked Flush 0 Sent Done Done false
5566 2043 Acked Write 10 Sent Done Done false
5567 2044 Acked Flush 0 Sent Done Done false
5568 2045 Acked Read 7 Sent Done Done false
5569 2046 Acked Write 5 Sent Done Done false
5570 2047 Acked Read 9 Sent Done Done false
5571 2048 Acked Read 5 Sent Done Done false
5572 2049 Acked Read 3 Sent Done Done false
5573 2050 Acked Read 10 Sent Done Done false
5574 2051 Acked Flush 0 Sent Done Done false
5575 2052 Acked Read 4 Sent Done Done false
5576 2053 Acked Read 3 Sent Done Done false
5577 2054 Acked Write 2 Sent Done Done false
5578 2055 Acked Write 4 Sent Done Done false
5579 2056 Acked Read 5 Sent Done Done false
5580 2057 Acked Write 1 Sent Done Done false
5581 2058 Acked Write 9 Sent Done Done false
5582 2059 Acked Read 5 Sent Done Done false
5583 2060 Acked Read 7 Sent Done Done false
5584 2061 Acked Write 6 Sent Done Done false
5585 2062 Acked Read 1 Sent Done Done false
5586 2063 Acked Read 10 Sent Done Done false
5587 2064 Acked Read 7 Sent Done Done false
5588 2065 Acked Write 4 Sent Done Done false
5589 2066 Acked Write 9 Sent Done Done false
5590 2067 Acked Read 1 Sent Done Done false
5591 2068 Acked Write 8 Sent Done Done false
5592 2069 Acked Read 3 Sent Done Done false
5593 2070 Acked Read 3 Sent Done Done false
5594 2071 Acked Write 6 Sent Done Done false
5595 2072 Acked Write 1 Sent Done Done false
5596 2073 Acked Write 6 Sent Done Done false
5597 2074 Acked Write 7 Sent Done Done false
5598 2075 Acked Read 2 Sent Done Done false
5599 2076 Acked Read 6 Sent Done Done false
5600 2077 Acked Read 7 Sent Done Done false
5601 2078 Acked Write 3 Sent Done Done false
5602 2079 Acked Flush 0 Sent Done Done false
5603 2080 Acked Read 2 Sent Done Done false
5604 2081 Acked Read 5 Sent Done Done false
5605 2082 Acked Read 4 Sent Done Done false
5606 2083 Acked Write 9 Sent Done Done false
5607 2084 Acked Flush 0 Sent Done Done false
5608 2085 Acked Write 9 Sent Done Done false
5609 2086 Acked Read 8 Sent Done Done false
5610 2087 Acked Write 5 Sent Done Done false
5611 2088 Acked Read 8 Sent Done Done false
5612 2089 Acked Read 4 Sent Done Done false
5613 2090 Acked Write 10 Sent Done Done false
5614 2091 Acked Read 5 Sent Done Done false
5615 2092 Acked Read 8 Sent Done Done false
5616 2093 Acked Read 5 Sent Done Done false
5617 2094 Acked Read 1 Sent Done Done false
5618 2095 Acked Write 6 Sent Done Done false
5619 2096 Acked Read 10 Sent Done Done false
5620 2097 Acked Write 10 Sent Done Done false
5621 2098 Acked Read 3 Sent Done Done false
5622 2099 Acked Read 5 Sent Done Done false
5623 2100 Acked Write 10 Sent Done Done false
5624 2101 Acked Write 10 Sent Done Done false
5625 2102 Acked Write 9 Sent Done Done false
5626 2103 Acked Read 8 Sent Done Done false
5627 2104 Acked Write 6 Sent Done Done false
5628 2105 Acked Flush 0 Sent Done Done false
5629 2106 Acked Write 4 Sent Done Done false
5630 2107 Acked Read 9 Sent Done Done false
5631 2108 Acked Write 3 Sent Done Done false
5632 2109 Acked Read 9 Sent Done Done false
5633 2110 Acked Read 7 Sent Done Done false
5634 2111 Acked Read 4 Sent Done Done false
5635 2112 Acked Read 10 Sent Done Done false
5636 2113 Acked Read 9 Sent Done Done false
5637 2114 Acked Read 3 Sent Done Done false
5638 2115 Acked Write 4 Sent Done Done false
5639 2116 Acked Write 5 Sent Done Done false
5640 2117 Acked Write 5 Sent Done Done false
5641 2118 Acked Flush 0 Sent Done Done false
5642 2119 Acked Flush 0 Sent Done Done false
5643 2120 Acked Write 3 Sent Done Done false
5644 2121 Acked Flush 0 Sent Done Done false
5645 2122 Acked Write 1 Sent Done Done false
5646 2123 Acked Read 4 Sent Done Done false
5647 2124 Acked Write 1 Sent Done Done false
5648 2125 Acked Read 7 Sent Done Done false
5649 2126 Acked Write 4 Sent Done Done false
5650 2127 Acked Flush 0 Sent Done Done false
5651 2128 Acked Read 3 Sent Done Done false
5652 2129 Acked Write 1 Sent Done Done false
5653 2130 Acked Write 10 Sent Done Done false
5654 2131 Acked Flush 0 Sent Done Done false
5655 2132 Acked Write 6 Sent Done Done false
5656 2133 Acked Write 5 Sent Done Done false
5657 2134 Acked Flush 0 Sent Done Done false
5658 2135 Acked Read 8 Sent Done Done false
5659 2136 Acked Read 2 Sent Done Done false
5660 2137 Acked Flush 0 Sent Done Done false
5661 2138 Acked Read 2 Sent Done Done false
5662 2139 Acked Write 6 Sent Done Done false
5663 2140 Acked Read 2 Sent Done Done false
5664 2141 Acked Flush 0 Sent Done Done false
5665 2142 Acked Write 1 Sent Done Done false
5666 2143 Acked Write 2 Sent Done Done false
5667 2144 Acked Read 8 Sent Done Done false
5668 2145 Acked Write 7 Sent Done Done false
5669 2146 Acked Read 6 Sent Done Done false
5670 2147 Acked Read 9 Sent Done Done false
5671 2148 Acked Read 3 Sent Done Done false
5672 2149 Acked Flush 0 Sent Done Done false
5673 2150 Acked Read 2 Sent Done Done false
5674 2151 Acked Flush 0 Sent Done Done false
5675 2152 Acked Write 10 Sent Done Done false
5676 2153 Acked Write 2 Sent Done Done false
5677 2154 Acked Read 2 Sent Done Done false
5678 2155 Acked Write 3 Sent Done Done false
5679 2156 Acked Read 4 Sent Done Done false
5680 2157 Acked Read 3 Sent Done Done false
5681 2158 Acked Read 4 Sent Done Done false
5682 2159 Acked Write 4 Sent Done Done false
5683 2160 Acked Read 3 Sent Done Done false
5684 2161 Acked Read 1 Sent Done Done false
5685 2162 Acked Write 3 Sent Done Done false
5686 2163 Acked Write 6 Sent Done Done false
5687 2164 Acked Read 7 Sent Done Done false
5688 2165 Acked Read 4 Sent Done Done false
5689 2166 Acked Read 4 Sent Done Done false
5690 2167 Acked Flush 0 Sent Done Done false
5691 2168 Acked Write 8 Sent Done Done false
5692 2169 Acked Write 2 Sent Done Done false
5693 2170 Acked Flush 0 Sent Done Done false
5694 2171 Acked Read 2 Sent Done Done false
5695 2172 Acked Write 9 Sent Done Done false
5696 2173 Acked Read 3 Sent Done Done false
5697 2174 Acked Read 8 Sent Done Done false
5698 2175 Acked Flush 0 Sent Done Done false
5699 2176 Acked Read 6 Sent Done Done false
5700 2177 Acked Read 5 Sent Done Done false
5701 2178 Acked Read 9 Sent Done Done false
5702 2179 Acked Flush 0 Sent Done Done false
5703 2180 Acked Read 3 Sent Done Done false
5704 2181 Acked Write 5 Sent Done Done false
5705 2182 Acked Read 10 Sent Done Done false
5706 2183 Acked Flush 0 Sent Done Done false
5707 2184 Acked Read 1 Sent Done Done false
5708 2185 Acked Write 3 Sent Done Done false
5709 2186 Acked Write 6 Sent Done Done false
5710 2187 Acked Read 9 Sent Done Done false
5711 2188 Acked Flush 0 Sent Done Done false
5712 2189 Acked Read 9 Sent Done Done false
5713 2190 Acked Write 2 Sent Done Done false
5714 2191 Acked Flush 0 Sent Done Done false
5715 2192 Acked Write 8 Sent Done Done false
5716 2193 Acked Flush 0 Sent Done Done false
5717 2194 Acked Write 5 Sent Done Done false
5718 2195 Acked Read 6 Sent Done Done false
5719 2196 Acked Read 2 Sent Done Done false
5720 2197 Acked Write 5 Sent Done Done false
5721 2198 Acked Read 6 Sent Done Done false
5722 2199 Acked Write 8 Sent Done Done false
5723 2200 Acked Write 4 Sent Done Done false
5724 2201 Acked Read 7 Sent Done Done false
5725 2202 Acked Read 4 Sent Done Done false
5726 2203 Acked Read 1 Sent Done Done false
5727 2204 Acked Write 10 Sent Done Done false
5728 2205 Acked Read 9 Sent Done Done false
5729 STATES DS:0 DS:1 DS:2 TOTAL
5730 Sent 301 0 0 301
5731 Done 0 301 301 602
5732 Skipped 0 0 0 0
5733 Error 0 0 0 0
5734 Last Flush: Some(JobId(1904)) Some(JobId(2193)) Some(JobId(2193))
5735 Downstairs last five completed: 1904 1903 1902 1901 1900
5736 Upstairs last five completed: 2205 2204 2203 2202 2201
5737 CLIENT: Up:0 ds:301 act:2
57382025-03-07T18:13:50.075ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
57392025-03-07T18:13:51.077ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
57402025-03-07T18:13:52.256ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
57412025-03-07T18:13:53.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
5742 [8810] got start action
5743 [8810] I am going to restart
5744 Starting downstairs at port 8810
5745 Make output file at "/tmp/dsc/downstairs-8810.txt"
5746 [8810][0] reports Starting
5747 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1212)
5748 [8810][0] reports Running
5749 ----------------------------------------------------------------
5750 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5751 States: Offline Active Active
5752 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5753 1905 Acked Read 9 Sent Done Done false
5754 1906 Acked Write 10 Sent Done Done false
5755 1907 Acked Write 2 Sent Done Done false
5756 1908 Acked Write 2 Sent Done Done false
5757 1909 Acked Write 9 Sent Done Done false
5758 1910 Acked Read 9 Sent Done Done false
5759 1911 Acked Flush 0 Sent Done Done false
5760 1912 Acked Write 1 Sent Done Done false
5761 1913 Acked Write 2 Sent Done Done false
5762 1914 Acked Write 3 Sent Done Done false
5763 1915 Acked Read 7 Sent Done Done false
5764 1916 Acked Read 9 Sent Done Done false
5765 1917 Acked Read 8 Sent Done Done false
5766 1918 Acked Read 7 Sent Done Done false
5767 1919 Acked Read 10 Sent Done Done false
5768 1920 Acked Read 9 Sent Done Done false
5769 1921 Acked Write 6 Sent Done Done false
5770 1922 Acked Read 10 Sent Done Done false
5771 1923 Acked Write 8 Sent Done Done false
5772 1924 Acked Read 10 Sent Done Done false
5773 1925 Acked Write 10 Sent Done Done false
5774 1926 Acked Write 4 Sent Done Done false
5775 1927 Acked Write 8 Sent Done Done false
5776 1928 Acked Read 4 Sent Done Done false
5777 1929 Acked Write 8 Sent Done Done false
5778 1930 Acked Flush 0 Sent Done Done false
5779 1931 Acked Write 4 Sent Done Done false
5780 1932 Acked Read 2 Sent Done Done false
5781 1933 Acked Write 9 Sent Done Done false
5782 1934 Acked Read 6 Sent Done Done false
5783 1935 Acked Read 10 Sent Done Done false
5784 1936 Acked Write 8 Sent Done Done false
5785 1937 Acked Read 8 Sent Done Done false
5786 1938 Acked Read 2 Sent Done Done false
5787 1939 Acked Read 2 Sent Done Done false
5788 1940 Acked Write 1 Sent Done Done false
5789 1941 Acked Read 7 Sent Done Done false
5790 1942 Acked Flush 0 Sent Done Done false
5791 1943 Acked Write 5 Sent Done Done false
5792 1944 Acked Write 1 Sent Done Done false
5793 1945 Acked Write 1 Sent Done Done false
5794 1946 Acked Write 8 Sent Done Done false
5795 1947 Acked Write 1 Sent Done Done false
5796 1948 Acked Read 4 Sent Done Done false
5797 1949 Acked Read 9 Sent Done Done false
5798 1950 Acked Write 3 Sent Done Done false
5799 1951 Acked Write 8 Sent Done Done false
5800 1952 Acked Read 5 Sent Done Done false
5801 1953 Acked Flush 0 Sent Done Done false
5802 1954 Acked Read 8 Sent Done Done false
5803 1955 Acked Flush 0 Sent Done Done false
5804 1956 Acked Flush 0 Sent Done Done false
5805 1957 Acked Write 2 Sent Done Done false
5806 1958 Acked Write 5 Sent Done Done false
5807 1959 Acked Write 9 Sent Done Done false
5808 1960 Acked Write 4 Sent Done Done false
5809 1961 Acked Flush 0 Sent Done Done false
5810 1962 Acked Flush 0 Sent Done Done false
5811 1963 Acked Read 5 Sent Done Done false
5812 1964 Acked Read 2 Sent Done Done false
5813 1965 Acked Write 9 Sent Done Done false
5814 1966 Acked Read 4 Sent Done Done false
5815 1967 Acked Read 10 Sent Done Done false
5816 1968 Acked Read 9 Sent Done Done false
5817 1969 Acked Read 5 Sent Done Done false
5818 1970 Acked Flush 0 Sent Done Done false
5819 1971 Acked Write 5 Sent Done Done false
5820 1972 Acked Read 4 Sent Done Done false
5821 1973 Acked Read 9 Sent Done Done false
5822 1974 Acked Read 6 Sent Done Done false
5823 1975 Acked Read 5 Sent Done Done false
5824 1976 Acked Write 9 Sent Done Done false
5825 1977 Acked Read 4 Sent Done Done false
5826 1978 Acked Write 5 Sent Done Done false
5827 1979 Acked Read 10 Sent Done Done false
5828 1980 Acked Write 9 Sent Done Done false
5829 1981 Acked Write 5 Sent Done Done false
5830 1982 Acked Write 8 Sent Done Done false
5831 1983 Acked Write 8 Sent Done Done false
5832 1984 Acked Read 3 Sent Done Done false
5833 1985 Acked Read 7 Sent Done Done false
5834 1986 Acked Read 8 Sent Done Done false
5835 1987 Acked Read 7 Sent Done Done false
5836 1988 Acked Read 5 Sent Done Done false
5837 1989 Acked Read 9 Sent Done Done false
5838 1990 Acked Read 8 Sent Done Done false
5839 1991 Acked Flush 0 Sent Done Done false
5840 1992 Acked Write 7 Sent Done Done false
5841 1993 Acked Write 9 Sent Done Done false
5842 1994 Acked Write 7 Sent Done Done false
5843 1995 Acked Write 2 Sent Done Done false
5844 1996 Acked Read 7 Sent Done Done false
5845 1997 Acked Read 10 Sent Done Done false
5846 1998 Acked Read 10 Sent Done Done false
5847 1999 Acked Write 3 Sent Done Done false
5848 2000 Acked Write 10 Sent Done Done false
5849 2001 Acked Read 9 Sent Done Done false
5850 2002 Acked Read 6 Sent Done Done false
5851 2003 Acked Read 9 Sent Done Done false
5852 2004 Acked Read 2 Sent Done Done false
5853 2005 Acked Read 5 Sent Done Done false
5854 2006 Acked Write 9 Sent Done Done false
5855 2007 Acked Read 10 Sent Done Done false
5856 2008 Acked Write 2 Sent Done Done false
5857 2009 Acked Write 6 Sent Done Done false
5858 2010 Acked Read 8 Sent Done Done false
5859 2011 Acked Read 4 Sent Done Done false
5860 2012 Acked Read 5 Sent Done Done false
5861 2013 Acked Read 5 Sent Done Done false
5862 2014 Acked Read 2 Sent Done Done false
5863 2015 Acked Write 7 Sent Done Done false
5864 2016 Acked Write 8 Sent Done Done false
5865 2017 Acked Read 6 Sent Done Done false
5866 2018 Acked Read 7 Sent Done Done false
5867 2019 Acked Read 2 Sent Done Done false
5868 2020 Acked Write 5 Sent Done Done false
5869 2021 Acked Read 9 Sent Done Done false
5870 2022 Acked Read 7 Sent Done Done false
5871 2023 Acked Read 2 Sent Done Done false
5872 2024 Acked Write 9 Sent Done Done false
5873 2025 Acked Read 1 Sent Done Done false
5874 2026 Acked Write 3 Sent Done Done false
5875 2027 Acked Read 8 Sent Done Done false
5876 2028 Acked Write 8 Sent Done Done false
5877 2029 Acked Read 9 Sent Done Done false
5878 2030 Acked Write 1 Sent Done Done false
5879 2031 Acked Read 9 Sent Done Done false
5880 2032 Acked Write 3 Sent Done Done false
5881 2033 Acked Write 10 Sent Done Done false
5882 2034 Acked Read 7 Sent Done Done false
5883 2035 Acked Read 5 Sent Done Done false
5884 2036 Acked Write 5 Sent Done Done false
5885 2037 Acked Read 2 Sent Done Done false
5886 2038 Acked Write 6 Sent Done Done false
5887 2039 Acked Write 8 Sent Done Done false
5888 2040 Acked Write 2 Sent Done Done false
5889 2041 Acked Write 10 Sent Done Done false
5890 2042 Acked Flush 0 Sent Done Done false
5891 2043 Acked Write 10 Sent Done Done false
5892 2044 Acked Flush 0 Sent Done Done false
5893 2045 Acked Read 7 Sent Done Done false
5894 2046 Acked Write 5 Sent Done Done false
5895 2047 Acked Read 9 Sent Done Done false
5896 2048 Acked Read 5 Sent Done Done false
5897 2049 Acked Read 3 Sent Done Done false
5898 2050 Acked Read 10 Sent Done Done false
5899 2051 Acked Flush 0 Sent Done Done false
5900 2052 Acked Read 4 Sent Done Done false
5901 2053 Acked Read 3 Sent Done Done false
5902 2054 Acked Write 2 Sent Done Done false
5903 2055 Acked Write 4 Sent Done Done false
5904 2056 Acked Read 5 Sent Done Done false
5905 2057 Acked Write 1 Sent Done Done false
5906 2058 Acked Write 9 Sent Done Done false
5907 2059 Acked Read 5 Sent Done Done false
5908 2060 Acked Read 7 Sent Done Done false
5909 2061 Acked Write 6 Sent Done Done false
5910 2062 Acked Read 1 Sent Done Done false
5911 2063 Acked Read 10 Sent Done Done false
5912 2064 Acked Read 7 Sent Done Done false
5913 2065 Acked Write 4 Sent Done Done false
5914 2066 Acked Write 9 Sent Done Done false
5915 2067 Acked Read 1 Sent Done Done false
5916 2068 Acked Write 8 Sent Done Done false
5917 2069 Acked Read 3 Sent Done Done false
5918 2070 Acked Read 3 Sent Done Done false
5919 2071 Acked Write 6 Sent Done Done false
5920 2072 Acked Write 1 Sent Done Done false
5921 2073 Acked Write 6 Sent Done Done false
5922 2074 Acked Write 7 Sent Done Done false
5923 2075 Acked Read 2 Sent Done Done false
5924 2076 Acked Read 6 Sent Done Done false
5925 2077 Acked Read 7 Sent Done Done false
5926 2078 Acked Write 3 Sent Done Done false
5927 2079 Acked Flush 0 Sent Done Done false
5928 2080 Acked Read 2 Sent Done Done false
5929 2081 Acked Read 5 Sent Done Done false
5930 2082 Acked Read 4 Sent Done Done false
5931 2083 Acked Write 9 Sent Done Done false
5932 2084 Acked Flush 0 Sent Done Done false
5933 2085 Acked Write 9 Sent Done Done false
5934 2086 Acked Read 8 Sent Done Done false
5935 2087 Acked Write 5 Sent Done Done false
5936 2088 Acked Read 8 Sent Done Done false
5937 2089 Acked Read 4 Sent Done Done false
5938 2090 Acked Write 10 Sent Done Done false
5939 2091 Acked Read 5 Sent Done Done false
5940 2092 Acked Read 8 Sent Done Done false
5941 2093 Acked Read 5 Sent Done Done false
5942 2094 Acked Read 1 Sent Done Done false
5943 2095 Acked Write 6 Sent Done Done false
5944 2096 Acked Read 10 Sent Done Done false
5945 2097 Acked Write 10 Sent Done Done false
5946 2098 Acked Read 3 Sent Done Done false
5947 2099 Acked Read 5 Sent Done Done false
5948 2100 Acked Write 10 Sent Done Done false
5949 2101 Acked Write 10 Sent Done Done false
5950 2102 Acked Write 9 Sent Done Done false
5951 2103 Acked Read 8 Sent Done Done false
5952 2104 Acked Write 6 Sent Done Done false
5953 2105 Acked Flush 0 Sent Done Done false
5954 2106 Acked Write 4 Sent Done Done false
5955 2107 Acked Read 9 Sent Done Done false
5956 2108 Acked Write 3 Sent Done Done false
5957 2109 Acked Read 9 Sent Done Done false
5958 2110 Acked Read 7 Sent Done Done false
5959 2111 Acked Read 4 Sent Done Done false
5960 2112 Acked Read 10 Sent Done Done false
5961 2113 Acked Read 9 Sent Done Done false
5962 2114 Acked Read 3 Sent Done Done false
5963 2115 Acked Write 4 Sent Done Done false
5964 2116 Acked Write 5 Sent Done Done false
5965 2117 Acked Write 5 Sent Done Done false
5966 2118 Acked Flush 0 Sent Done Done false
5967 2119 Acked Flush 0 Sent Done Done false
5968 2120 Acked Write 3 Sent Done Done false
5969 2121 Acked Flush 0 Sent Done Done false
5970 2122 Acked Write 1 Sent Done Done false
5971 2123 Acked Read 4 Sent Done Done false
5972 2124 Acked Write 1 Sent Done Done false
5973 2125 Acked Read 7 Sent Done Done false
5974 2126 Acked Write 4 Sent Done Done false
5975 2127 Acked Flush 0 Sent Done Done false
5976 2128 Acked Read 3 Sent Done Done false
5977 2129 Acked Write 1 Sent Done Done false
5978 2130 Acked Write 10 Sent Done Done false
5979 2131 Acked Flush 0 Sent Done Done false
5980 2132 Acked Write 6 Sent Done Done false
5981 2133 Acked Write 5 Sent Done Done false
5982 2134 Acked Flush 0 Sent Done Done false
5983 2135 Acked Read 8 Sent Done Done false
5984 2136 Acked Read 2 Sent Done Done false
5985 2137 Acked Flush 0 Sent Done Done false
5986 2138 Acked Read 2 Sent Done Done false
5987 2139 Acked Write 6 Sent Done Done false
5988 2140 Acked Read 2 Sent Done Done false
5989 2141 Acked Flush 0 Sent Done Done false
5990 2142 Acked Write 1 Sent Done Done false
5991 2143 Acked Write 2 Sent Done Done false
5992 2144 Acked Read 8 Sent Done Done false
5993 2145 Acked Write 7 Sent Done Done false
5994 2146 Acked Read 6 Sent Done Done false
5995 2147 Acked Read 9 Sent Done Done false
5996 2148 Acked Read 3 Sent Done Done false
5997 2149 Acked Flush 0 Sent Done Done false
5998 2150 Acked Read 2 Sent Done Done false
5999 2151 Acked Flush 0 Sent Done Done false
6000 2152 Acked Write 10 Sent Done Done false
6001 2153 Acked Write 2 Sent Done Done false
6002 2154 Acked Read 2 Sent Done Done false
6003 2155 Acked Write 3 Sent Done Done false
6004 2156 Acked Read 4 Sent Done Done false
6005 2157 Acked Read 3 Sent Done Done false
6006 2158 Acked Read 4 Sent Done Done false
6007 2159 Acked Write 4 Sent Done Done false
6008 2160 Acked Read 3 Sent Done Done false
6009 2161 Acked Read 1 Sent Done Done false
6010 2162 Acked Write 3 Sent Done Done false
6011 2163 Acked Write 6 Sent Done Done false
6012 2164 Acked Read 7 Sent Done Done false
6013 2165 Acked Read 4 Sent Done Done false
6014 2166 Acked Read 4 Sent Done Done false
6015 2167 Acked Flush 0 Sent Done Done false
6016 2168 Acked Write 8 Sent Done Done false
6017 2169 Acked Write 2 Sent Done Done false
6018 2170 Acked Flush 0 Sent Done Done false
6019 2171 Acked Read 2 Sent Done Done false
6020 2172 Acked Write 9 Sent Done Done false
6021 2173 Acked Read 3 Sent Done Done false
6022 2174 Acked Read 8 Sent Done Done false
6023 2175 Acked Flush 0 Sent Done Done false
6024 2176 Acked Read 6 Sent Done Done false
6025 2177 Acked Read 5 Sent Done Done false
6026 2178 Acked Read 9 Sent Done Done false
6027 2179 Acked Flush 0 Sent Done Done false
6028 2180 Acked Read 3 Sent Done Done false
6029 2181 Acked Write 5 Sent Done Done false
6030 2182 Acked Read 10 Sent Done Done false
6031 2183 Acked Flush 0 Sent Done Done false
6032 2184 Acked Read 1 Sent Done Done false
6033 2185 Acked Write 3 Sent Done Done false
6034 2186 Acked Write 6 Sent Done Done false
6035 2187 Acked Read 9 Sent Done Done false
6036 2188 Acked Flush 0 Sent Done Done false
6037 2189 Acked Read 9 Sent Done Done false
6038 2190 Acked Write 2 Sent Done Done false
6039 2191 Acked Flush 0 Sent Done Done false
6040 2192 Acked Write 8 Sent Done Done false
6041 2193 Acked Flush 0 Sent Done Done false
6042 2194 Acked Write 5 Sent Done Done false
6043 2195 Acked Read 6 Sent Done Done false
6044 2196 Acked Read 2 Sent Done Done false
6045 2197 Acked Write 5 Sent Done Done false
6046 2198 Acked Read 6 Sent Done Done false
6047 2199 Acked Write 8 Sent Done Done false
6048 2200 Acked Write 4 Sent Done Done false
6049 2201 Acked Read 7 Sent Done Done false
6050 2202 Acked Read 4 Sent Done Done false
6051 2203 Acked Read 1 Sent Done Done false
6052 2204 Acked Write 10 Sent Done Done false
6053 2205 Acked Read 9 Sent Done Done false
6054 STATES DS:0 DS:1 DS:2 TOTAL
6055 Sent 301 0 0 301
6056 Done 0 301 301 602
6057 Skipped 0 0 0 0
6058 Error 0 0 0 0
6059 Last Flush: Some(JobId(1904)) Some(JobId(2193)) Some(JobId(2193))
6060 Downstairs last five completed: 1904 1903 1902 1901 1900
6061 Upstairs last five completed: 2205 2204 2203 2202 2201
6062 CLIENT: Up:0 ds:301 act:2
60632025-03-07T18:13:54.083ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60642025-03-07T18:13:55.083ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60652025-03-07T18:13:55.269ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60662025-03-07T18:13:55.269ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34509) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60672025-03-07T18:13:55.270ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60682025-03-07T18:13:55.271ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60692025-03-07T18:13:55.271ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60702025-03-07T18:13:55.271ZINFOcrucible: send last flush ID to this DS: Some(JobId(1904)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60712025-03-07T18:13:55.271ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60722025-03-07T18:13:55.271ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1904)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60732025-03-07T18:13:55.271ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1904)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
6074 ----------------------------------------------------------------
6075 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6076 Downstairs last five completed: 2206 2205 2204 2203 2202
6077 Upstairs last five completed: 2206 2205 2204 2203 2202
6078 CLIENT: Up:0 ds:0 act:3
6079 Replay: All jobs finished
6080 Main task has work to do, go find it
6081 got dsc Stop(1)
6082 stop 1
6083 Mar 07 18:13:57.526 INFO[8820] Got stop action so:false kr:false
6084 request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4998ec0c-010b-49ee-983b-355b38f6cfa0, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
60852025-03-07T18:13:57.527ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60862025-03-07T18:13:57.527ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60872025-03-07T18:13:57.527ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
60882025-03-07T18:13:57.527ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
6089 Mar 07 18:13:57.527 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: e2540d14-0cc5-4a65-afd7-cc216521689b, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
6090 [8820] Exited with: ExitStatus(unix_wait_status(9))
6091 [8820][1] reports Exit
6092 Mar 07 18:14:01.529 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9ea8d07a-abb5-4c8a-af78-cb80dde62aaa, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
6093 001/300 Write block 3187 len 4096 data: 4
6094 002/300 Write block 589 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6095 003/300 Write block 672 len 32768 data: 4 4 2 2 3 3 3 3
6096 004/300 Read block 2103 len 36864
6097 005/300 Read block 3614 len 12288
6098 006/300 Read block 3461 len 36864
6099 007/300 Read block 2678 len 28672
6100 008/300 Write block 4582 len 36864 data: 2 2 2 2 2 2 2 2 2
6101 009/300 Read block 3066 len 36864
6102 010/300 Read block 2143 len 8192
6103 011/300 Read block 4423 len 40960
6104 012/300 Write block 1092 len 36864 data: 3 3 3 4 3 3 3 3 3
6105 013/300 Write block 3237 len 8192 data: 2 2
6106 014/300 Write block 119 len 4096 data: 2
6107 015/300 Read block 3624 len 24576
6108 016/300 Read block 3670 len 24576
6109 017/300 Read block 1871 len 24576
6110 018/300 Read block 4627 len 40960
6111 019/300 Read block 4048 len 28672
6112 020/300 Read block 133 len 20480
6113 021/300 Read block 569 len 36864
6114 022/300 Read block 3418 len 36864
6115 023/300 Write block 2655 len 12288 data: 2 2 3
6116 024/300 Write block 2840 len 32768 data: 2 2 2 2 2 2 2 2
6117 025/300 Read block 4429 len 40960
6118 026/300 Write block 382 len 8192 data: 2 2
6119 027/300 Read block 3800 len 40960
6120 028/300 Read block 4736 len 32768
6121 029/300 Write block 1650 len 28672 data: 4 3 3 3 3 2 2
6122 030/300 Read block 3807 len 8192
6123 031/300 Read block 2794 len 36864
6124 032/300 Write block 4907 len 28672 data: 2 2 3 3 3 3 3
6125 033/300 Write block 1964 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6126 034/300 Write block 42 len 36864 data: 2 2 3 3 3 3 3 2 2
6127 035/300 Write block 2726 len 16384 data: 2 2 2 2
6128 036/300 Write block 3670 len 28672 data: 2 2 2 2 2 2 2
6129 037/300 Read block 4245 len 20480
6130 038/300 Read block 2372 len 20480
6131 039/300 Write block 3740 len 20480 data: 2 2 2 2 2
6132 040/300 Read block 2476 len 16384
6133 041/300 Read block 538 len 12288
6134 042/300 Flush
6135 043/300 Read block 976 len 4096
6136 044/300 Write block 1617 len 36864 data: 4 4 5 3 3 3 3 3 3
6137 045/300 Flush
6138 046/300 Write block 1751 len 36864 data: 2 2 2 2 2 2 2 2 2
6139 047/300 Read block 3507 len 36864
6140 048/300 Read block 178 len 32768
6141 049/300 Flush
6142 050/300 Write block 366 len 8192 data: 2 2
6143 051/300 Write block 582 len 28672 data: 2 2 2 2 2 3 2
6144 052/300 Read block 1852 len 36864
6145 053/300 Read block 173 len 36864
6146 054/300 Read block 2549 len 4096
6147 055/300 Read block 1819 len 16384
6148 056/300 Read block 115 len 8192
6149 057/300 Flush
6150 058/300 Write block 276 len 28672 data: 3 3 3 2 2 2 3
6151 059/300 Read block 577 len 16384
6152 060/300 Read block 3128 len 12288
6153 061/300 Flush
6154 062/300 Read block 4948 len 8192
6155 063/300 Write block 410 len 24576 data: 2 2 2 2 2 2
6156 064/300 Read block 4944 len 36864
6157 065/300 Flush
6158 066/300 Flush
6159 067/300 Read block 2271 len 16384
6160 068/300 Write block 132 len 16384 data: 2 2 2 2
6161 069/300 Read block 2684 len 24576
6162 070/300 Write block 3900 len 16384 data: 2 2 2 2
6163 071/300 Read block 3281 len 32768
6164 072/300 Read block 1553 len 28672
6165 073/300 Write block 4036 len 16384 data: 2 2 2 2
6166 074/300 Read block 3981 len 8192
6167 075/300 Write block 4836 len 24576 data: 2 2 2 3 2 2
6168 076/300 Flush
6169 077/300 Write block 2042 len 28672 data: 2 2 2 2 2 3 3
6170 078/300 Write block 3492 len 4096 data: 2
6171 079/300 Read block 3817 len 36864
6172 080/300 Read block 3317 len 20480
6173 081/300 Write block 3071 len 32768 data: 2 2 3 3 2 3 3 3
6174 082/300 Read block 101 len 24576
6175 083/300 Write block 3063 len 40960 data: 4 3 3 3 3 2 2 2 3 3
6176 084/300 Read block 3887 len 20480
6177 085/300 Write block 2634 len 8192 data: 2 2
6178 086/300 Read block 3103 len 32768
6179 087/300 Read block 756 len 4096
6180 088/300 Read block 898 len 24576
6181 089/300 Read block 3250 len 32768
6182 090/300 Read block 1543 len 28672
6183 091/300 Write block 1905 len 32768 data: 3 3 3 3 4 4 4 3
6184 092/300 Write block 4353 len 16384 data: 3 3 2 3
6185 093/300 Write block 3004 len 28672 data: 4 4 3 3 3 3 4
6186 094/300 Write block 4236 len 40960 data: 2 2 2 2 3 3 3 2 3 3
6187 095/300 Read block 2285 len 20480
6188 096/300 Read block 3574 len 12288
6189 097/300 Read block 4110 len 28672
6190 098/300 Write block 3398 len 28672 data: 2 2 2 2 2 2 2
6191 099/300 Read block 4479 len 16384
6192 100/300 Write block 2365 len 4096 data: 3
6193 101/300 Flush
6194 102/300 Read block 2002 len 40960
6195 103/300 Write block 1484 len 16384 data: 3 3 3 3
6196 104/300 Read block 2003 len 32768
6197 105/300 Read block 4504 len 4096
6198 106/300 Read block 1691 len 4096
6199 107/300 Read block 4226 len 4096
6200 108/300 Read block 180 len 36864
6201 109/300 Read block 4654 len 36864
6202 110/300 Read block 1823 len 8192
6203 111/300 Write block 3781 len 24576 data: 2 3 3 3 2 2
6204 112/300 Write block 3665 len 28672 data: 2 2 2 2 2 3 3
6205 113/300 Read block 255 len 36864
6206 114/300 Write block 4087 len 16384 data: 2 2 2 2
6207 115/300 Write block 757 len 24576 data: 3 4 4 4 4 4
6208 116/300 Read block 4781 len 4096
6209 117/300 Read block 2463 len 32768
6210 118/300 Write block 4464 len 20480 data: 5 5 3 3 3
6211 119/300 Flush
6212 120/300 Read block 1763 len 4096
6213 121/300 Read block 1476 len 32768
6214 122/300 Write block 1801 len 8192 data: 2 2
6215 123/300 Flush
6216 124/300 Read block 4955 len 16384
6217 125/300 Flush
6218 126/300 Read block 1550 len 16384
6219 127/300 Write block 3880 len 28672 data: 2 2 3 3 3 3 3
6220 128/300 Read block 4650 len 32768
6221 129/300 Write block 2264 len 40960 data: 3 3 3 2 2 2 3 3 3 3
6222 130/300 Read block 1446 len 36864
6223 131/300 Write block 3063 len 36864 data: 5 4 4 4 4 3 3 3 4
6224 132/300 Read block 3742 len 40960
6225 133/300 Write block 3746 len 28672 data: 2 2 2 2 3 3 3
6226 134/300 Write block 1992 len 16384 data: 2 2 2 2
6227 135/300 Write block 1908 len 4096 data: 4
6228 136/300 Read block 2192 len 12288
6229 137/300 Read block 4293 len 8192
6230 138/300 Read block 1841 len 4096
6231 139/300 Write block 1786 len 8192 data: 2 2
6232 140/300 Write block 488 len 24576 data: 2 2 2 2 2 2
6233 141/300 Read block 4849 len 4096
6234 142/300 Read block 1359 len 8192
6235 143/300 Write block 1698 len 32768 data: 3 3 3 3 3 3 3 3
6236 144/300 Flush
6237 145/300 Flush
6238 146/300 Write block 3132 len 20480 data: 2 2 2 2 2
6239 147/300 Read block 1141 len 8192
6240 148/300 Read block 1152 len 40960
6241 149/300 Read block 4468 len 32768
6242 150/300 Read block 2021 len 24576
6243 151/300 Read block 4524 len 36864
6244 152/300 Flush
6245 153/300 Flush
6246 154/300 Read block 3044 len 4096
6247 155/300 Flush
6248 156/300 Read block 3604 len 40960
6249 157/300 Read block 4971 len 32768
6250 158/300 Write block 4962 len 16384 data: 2 2 2 2
6251 159/300 Read block 2600 len 8192
6252 160/300 Read block 3943 len 16384
6253 161/300 Read block 2201 len 4096
6254 162/300 Read block 812 len 24576
6255 163/300 Read block 4925 len 32768
6256 164/300 Write block 1822 len 4096 data: 2
6257 165/300 Write block 4222 len 16384 data: 4 2 2 2
6258 166/300 Read block 3406 len 32768
6259 167/300 Flush
6260 168/300 Write block 2861 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6261 169/300 Read block 1819 len 12288
6262 170/300 Write block 3007 len 36864 data: 4 4 4 5 4 3 3 3 3
6263 171/300 Read block 4101 len 24576
6264 172/300 Write block 337 len 28672 data: 2 3 3 3 3 3 3
6265 173/300 Read block 3682 len 20480
6266 174/300 Read block 1875 len 32768
6267 175/300 Read block 3117 len 40960
6268 176/300 Read block 2143 len 20480
6269 177/300 Read block 3214 len 40960
6270 178/300 Read block 2672 len 4096
6271 179/300 Read block 3425 len 12288
6272 180/300 Write block 4671 len 8192 data: 2 2
6273 181/300 Write block 3077 len 32768 data: 4 4 3 4 4 3 3 4
6274 182/300 Write block 2629 len 24576 data: 2 2 2 2 2 3
6275 183/300 Write block 1359 len 40960 data: 2 3 3 3 3 3 3 3 3 2
6276 184/300 Write block 472 len 24576 data: 2 2 2 2 2 2
6277 185/300 Flush
6278 186/300 Flush
6279 187/300 Read block 3792 len 12288
6280 188/300 Read block 4181 len 28672
6281 189/300 Read block 3741 len 28672
6282 190/300 Write block 2138 len 36864 data: 2 2 2 2 3 3 3 3 3
6283 191/300 Read block 2397 len 12288
6284 192/300 Write block 954 len 20480 data: 2 2 3 3 3
6285 193/300 Read block 1777 len 32768
6286 194/300 Read block 3425 len 8192
6287 195/300 Read block 3962 len 40960
6288 196/300 Read block 3211 len 24576
6289 197/300 Write block 4555 len 20480 data: 2 2 2 2 2
6290 198/300 Read block 3850 len 24576
6291 199/300 Write block 4433 len 28672 data: 2 2 2 2 2 2 2
6292 200/300 Read block 3935 len 4096
6293 201/300 Flush
6294 202/300 Read block 43 len 16384
6295 203/300 Write block 3725 len 4096 data: 4
6296 204/300 Read block 4652 len 32768
6297 205/300 Write block 4482 len 8192 data: 2 2
6298 206/300 Write block 2593 len 4096 data: 3
6299 207/300 Read block 2028 len 8192
6300 208/300 Read block 2466 len 28672
6301 209/300 Read block 3306 len 8192
6302 210/300 Read block 1317 len 4096
6303 211/300 Write block 4695 len 40960 data: 3 3 3 3 3 2 3 3 3 3
6304 212/300 Write block 2511 len 12288 data: 2 3 4
6305 213/300 Write block 801 len 36864 data: 2 2 2 2 2 2 2 2 2
6306 214/300 Read block 3868 len 32768
6307 215/300 Read block 4204 len 36864
6308 216/300 Write block 1846 len 28672 data: 3 3 3 3 3 3 3
6309 217/300 Flush
6310 218/300 Read block 435 len 16384
6311 219/300 Read block 4967 len 40960
6312 220/300 Read block 3656 len 12288
6313 221/300 Read block 249 len 28672
6314 222/300 Write block 1699 len 36864 data: 4 4 4 4 4 4 4 2 2
6315 223/300 Write block 687 len 12288 data: 2 2 2
6316 224/300 Flush
6317 225/300 Write block 1745 len 4096 data: 2
6318 226/300 Write block 1206 len 28672 data: 2 2 2 2 2 2 2
6319 227/300 Write block 165 len 12288 data: 3 3 2
6320 228/300 Write block 4223 len 32768 data: 3 3 3 2 2 2 3 2
6321 229/300 Write block 125 len 4096 data: 4
6322 230/300 Write block 4330 len 20480 data: 3 3 3 3 3
6323 231/300 Write block 3449 len 24576 data: 2 2 2 2 2 2
6324 232/300 Write block 190 len 16384 data: 3 3 3 3
6325 233/300 Read block 2732 len 32768
6326 234/300 Write block 350 len 12288 data: 2 2 2
6327 235/300 Write block 2819 len 12288 data: 3 3 2
6328 236/300 Flush
6329 237/300 Read block 4034 len 32768
6330 238/300 Read block 4573 len 12288
6331 239/300 Write block 2062 len 36864 data: 3 4 4 4 4 3 2 2 2
6332 240/300 Read block 1460 len 20480
6333 241/300 Flush
6334 242/300 Read block 828 len 16384
6335 243/300 Write block 4777 len 16384 data: 2 2 2 2
6336 244/300 Flush
6337 245/300 Write block 4837 len 32768 data: 3 3 4 3 3 2 2 2
6338 246/300 Flush
6339 247/300 Read block 1606 len 16384
6340 248/300 Write block 3633 len 16384 data: 2 2 2 3
6341 249/300 Read block 3187 len 20480
6342 250/300 Read block 3017 len 32768
6343 251/300 Write block 4694 len 28672 data: 3 4 4 4 4 4 3
6344 252/300 Read block 4104 len 12288
6345 253/300 Read block 4713 len 4096
6346 254/300 Read block 4555 len 8192
6347 255/300 Read block 2551 len 20480
6348 256/300 Read block 2065 len 8192
6349 257/300 Read block 74 len 20480
6350 258/300 Write block 3744 len 16384 data: 3 2 3 3
6351 259/300 Write block 4274 len 32768 data: 2 2 2 2 3 4 4 4
6352 260/300 Read block 3639 len 4096
6353 261/300 Write block 4339 len 32768 data: 2 2 2 3 3 3 2 2
6354 262/300 Read block 4911 len 24576
6355 263/300 Read block 2084 len 20480
6356 264/300 Write block 2725 len 32768 data: 2 3 3 3 3 2 2 2
6357 265/300 Write block 915 len 24576 data: 3 3 3 3 2 2
6358 266/300 Read block 4088 len 40960
6359 267/300 Write block 2059 len 4096 data: 2
6360 268/300 Read block 1450 len 12288
6361 269/300 Read block 4826 len 16384
6362 270/300 Flush
6363 271/300 Read block 4654 len 20480
6364 272/300 Write block 2511 len 24576 data: 3 4 5 4 4 4
6365 273/300 Write block 4533 len 20480 data: 2 2 2 2 2
6366 274/300 Read block 3608 len 12288
6367 275/300 Read block 3494 len 12288
6368 276/300 Read block 2914 len 24576
6369 277/300 Write block 2887 len 16384 data: 3 3 4 4
6370 278/300 Flush
6371 279/300 Write block 3905 len 12288 data: 2 2 2
6372 280/300 Read block 1093 len 8192
6373 281/300 Write block 3715 len 8192 data: 2 2
6374 282/300 Read block 734 len 24576
6375 283/300 Read block 371 len 36864
6376 284/300 Read block 2175 len 4096
6377 285/300 Flush
6378 286/300 Write block 1115 len 20480 data: 2 2 2 2 2
6379 287/300 Write block 797 len 24576 data: 2 2 2 2 3 3
6380 288/300 Read block 1922 len 32768
6381 289/300 Write block 3615 len 24576 data: 2 2 2 2 2 2
6382 290/300 Read block 4301 len 8192
6383 291/300 Read block 342 len 24576
6384 292/300 Read block 4422 len 4096
6385 293/300 Read block 136 len 20480
6386 294/300 Write block 3897 len 12288 data: 3 2 2
6387 295/300 Read block 4270 len 40960
6388 296/300 Read block 2583 len 36864
6389 297/300 Write block 1229 len 28672 data: 2 2 2 2 2 3 3
6390 298/300 Read block 2753 len 24576
6391 299/300 Read block 4795 len 28672
6392 300/300 Write block 2482 len 40960 data: 2 2 2 2 2 3 3 3 3 3
6393 301/300 Read block 676 len 24576
6394 Main task has work to do, go find it
6395 got dsc Start(1)
6396 start 1
6397 Mar 07 18:14:01.780 INFO request completed, latency_us: 167, response_code: 204, uri: /start/cid/1, method: POST, req_id: 84b3c290-b240-48dc-9a1b-cddf8e756f52, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
6398 Replay: started 1, returned:Ok(())
6399 ----------------------------------------------------------------
6400 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6401 States: Active Offline Active
6402 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6403 2207 Acked Write 1 Done Sent Done false
6404 2208 Acked Write 10 Done Sent Done false
6405 2209 Acked Write 8 Done Sent Done false
6406 2210 Acked Read 9 Done Sent Done false
6407 2211 Acked Read 3 Done Sent Done false
6408 2212 Acked Read 9 Done Sent Done false
6409 2213 Acked Read 7 Done Sent Done false
6410 2214 Acked Write 9 Done Sent Done false
6411 2215 Acked Read 9 Done Sent Done false
6412 2216 Acked Read 2 Done Sent Done false
6413 2217 Acked Read 10 Done Sent Done false
6414 2218 Acked Write 9 Done Sent Done false
6415 2219 Acked Write 2 Done Sent Done false
6416 2220 Acked Write 1 Done Sent Done false
6417 2221 Acked Read 6 Done Sent Done false
6418 2222 Acked Read 6 Done Sent Done false
6419 2223 Acked Read 6 Done Sent Done false
6420 2224 Acked Read 10 Done Sent Done false
6421 2225 Acked Read 7 Done Sent Done false
6422 2226 Acked Read 5 Done Sent Done false
6423 2227 Acked Read 9 Done Sent Done false
6424 2228 Acked Read 9 Done Sent Done false
6425 2229 Acked Write 3 Done Sent Done false
6426 2230 Acked Write 8 Done Sent Done false
6427 2231 Acked Read 10 Done Sent Done false
6428 2232 Acked Write 2 Done Sent Done false
6429 2233 Acked Read 10 Done Sent Done false
6430 2234 Acked Read 8 Done Sent Done false
6431 2235 Acked Write 7 Done Sent Done false
6432 2236 Acked Read 2 Done Sent Done false
6433 2237 Acked Read 9 Done Sent Done false
6434 2238 Acked Write 7 Done Sent Done false
6435 2239 Acked Write 10 Done Sent Done false
6436 2240 Acked Write 9 Done Sent Done false
6437 2241 Acked Write 4 Done Sent Done false
6438 2242 Acked Write 7 Done Sent Done false
6439 2243 Acked Read 5 Done Sent Done false
6440 2244 Acked Read 5 Done Sent Done false
6441 2245 Acked Write 5 Done Sent Done false
6442 2246 Acked Read 4 Done Sent Done false
6443 2247 Acked Read 3 Done Sent Done false
6444 2248 Acked Flush 0 Done Sent Done false
6445 2249 Acked Read 1 Done Sent Done false
6446 2250 Acked Write 9 Done Sent Done false
6447 2251 Acked Flush 0 Done Sent Done false
6448 2252 Acked Write 9 Done Sent Done false
6449 2253 Acked Read 9 Done Sent Done false
6450 2254 Acked Read 8 Done Sent Done false
6451 2255 Acked Flush 0 Done Sent Done false
6452 2256 Acked Write 2 Done Sent Done false
6453 2257 Acked Write 7 Done Sent Done false
6454 2258 Acked Read 9 Done Sent Done false
6455 2259 Acked Read 9 Done Sent Done false
6456 2260 Acked Read 1 Done Sent Done false
6457 2261 Acked Read 4 Done Sent Done false
6458 2262 Acked Read 2 Done Sent Done false
6459 2263 Acked Flush 0 Done Sent Done false
6460 2264 Acked Write 7 Done Sent Done false
6461 2265 Acked Read 4 Done Sent Done false
6462 2266 Acked Read 3 Done Sent Done false
6463 2267 Acked Flush 0 Done Sent Done false
6464 2268 Acked Read 2 Done Sent Done false
6465 2269 Acked Write 6 Done Sent Done false
6466 2270 Acked Read 9 Done Sent Done false
6467 2271 Acked Flush 0 Done Sent Done false
6468 2272 Acked Flush 0 Done Sent Done false
6469 2273 Acked Read 4 Done Sent Done false
6470 2274 Acked Write 4 Done Sent Done false
6471 2275 Acked Read 6 Done Sent Done false
6472 2276 Acked Write 4 Done Sent Done false
6473 2277 Acked Read 8 Done Sent Done false
6474 2278 Acked Read 7 Done Sent Done false
6475 2279 Acked Write 4 Done Sent Done false
6476 2280 Acked Read 2 Done Sent Done false
6477 2281 Acked Write 6 Done Sent Done false
6478 2282 Acked Flush 0 Done Sent Done false
6479 2283 Acked Write 7 Done Sent Done false
6480 2284 Acked Write 1 Done Sent Done false
6481 2285 Acked Read 9 Done Sent Done false
6482 2286 Acked Read 5 Done Sent Done false
6483 2287 Acked Write 8 Done Sent Done false
6484 2288 Acked Read 6 Done Sent Done false
6485 2289 Acked Write 10 Done Sent Done false
6486 2290 Acked Read 5 Done Sent Done false
6487 2291 Acked Write 2 Done Sent Done false
6488 2292 Acked Read 8 Done Sent Done false
6489 2293 Acked Read 1 Done Sent Done false
6490 2294 Acked Read 6 Done Sent Done false
6491 2295 Acked Read 8 Done Sent Done false
6492 2296 Acked Read 7 Done Sent Done false
6493 2297 Acked Write 8 Done Sent Done false
6494 2298 Acked Write 4 Done Sent Done false
6495 2299 Acked Write 7 Done Sent Done false
6496 2300 Acked Write 10 Done Sent Done false
6497 2301 Acked Read 5 Done Sent Done false
6498 2302 Acked Read 3 Done Sent Done false
6499 2303 Acked Read 7 Done Sent Done false
6500 2304 Acked Write 7 Done Sent Done false
6501 2305 Acked Read 4 Done Sent Done false
6502 2306 Acked Write 1 Done Sent Done false
6503 2307 Acked Flush 0 Done Sent Done false
6504 2308 Acked Read 10 Done Sent Done false
6505 2309 Acked Write 4 Done Sent Done false
6506 2310 Acked Read 8 Done Sent Done false
6507 2311 Acked Read 1 Done Sent Done false
6508 2312 Acked Read 1 Done Sent Done false
6509 2313 Acked Read 1 Done Sent Done false
6510 2314 Acked Read 9 Done Sent Done false
6511 2315 Acked Read 9 Done Sent Done false
6512 2316 Acked Read 2 Done Sent Done false
6513 2317 Acked Write 6 Done Sent Done false
6514 2318 Acked Write 7 Done Sent Done false
6515 2319 Acked Read 9 Done Sent Done false
6516 2320 Acked Write 4 Done Sent Done false
6517 2321 Acked Write 6 Done Sent Done false
6518 2322 Acked Read 1 Done Sent Done false
6519 2323 Acked Read 8 Done Sent Done false
6520 2324 Acked Write 5 Done Sent Done false
6521 2325 Acked Flush 0 Done Sent Done false
6522 2326 Acked Read 1 Done Sent Done false
6523 2327 Acked Read 8 Done Sent Done false
6524 2328 Acked Write 2 Done Sent Done false
6525 2329 Acked Flush 0 Done Sent Done false
6526 2330 Acked Read 4 Done Sent Done false
6527 2331 Acked Flush 0 Done Sent Done false
6528 2332 Acked Read 4 Done Sent Done false
6529 2333 Acked Write 7 Done Sent Done false
6530 2334 Acked Read 8 Done Sent Done false
6531 2335 Acked Write 10 Done Sent Done false
6532 2336 Acked Read 9 Done Sent Done false
6533 2337 Acked Write 9 Done Sent Done false
6534 2338 Acked Read 10 Done Sent Done false
6535 2339 Acked Write 7 Done Sent Done false
6536 2340 Acked Write 4 Done Sent Done false
6537 2341 Acked Write 1 Done Sent Done false
6538 2342 Acked Read 3 Done Sent Done false
6539 2343 Acked Read 2 Done Sent Done false
6540 2344 Acked Read 1 Done Sent Done false
6541 2345 Acked Write 2 Done Sent Done false
6542 2346 Acked Write 6 Done Sent Done false
6543 2347 Acked Read 1 Done Sent Done false
6544 2348 Acked Read 2 Done Sent Done false
6545 2349 Acked Write 8 Done Sent Done false
6546 2350 Acked Flush 0 Done Sent Done false
6547 2351 Acked Flush 0 Done Sent Done false
6548 2352 Acked Write 5 Done Sent Done false
6549 2353 Acked Read 2 Done Sent Done false
6550 2354 Acked Read 10 Done Sent Done false
6551 2355 Acked Read 8 Done Sent Done false
6552 2356 Acked Read 6 Done Sent Done false
6553 2357 Acked Read 9 Done Sent Done false
6554 2358 Acked Flush 0 Done Sent Done false
6555 2359 Acked Flush 0 Done Sent Done false
6556 2360 Acked Read 1 Done Sent Done false
6557 2361 Acked Flush 0 Done Sent Done false
6558 2362 Acked Read 10 Done Sent Done false
6559 2363 Acked Read 8 Done Sent Done false
6560 2364 Acked Write 4 Done Sent Done false
6561 2365 Acked Read 2 Done Sent Done false
6562 2366 Acked Read 4 Done Sent Done false
6563 2367 Acked Read 1 Done Sent Done false
6564 2368 Acked Read 6 Done Sent Done false
6565 2369 Acked Read 8 Done Sent Done false
6566 2370 Acked Write 1 Done Sent Done false
6567 2371 Acked Write 4 Done Sent Done false
6568 2372 Acked Read 8 Done Sent Done false
6569 2373 Acked Flush 0 Done Sent Done false
6570 2374 Acked Write 10 Done Sent Done false
6571 2375 Acked Read 3 Done Sent Done false
6572 2376 Acked Write 9 Done Sent Done false
6573 2377 Acked Read 6 Done Sent Done false
6574 2378 Acked Write 7 Done Sent Done false
6575 2379 Acked Read 5 Done Sent Done false
6576 2380 Acked Read 8 Done Sent Done false
6577 2381 Acked Read 10 Done Sent Done false
6578 2382 Acked Read 5 Done Sent Done false
6579 2383 Acked Read 10 Done Sent Done false
6580 2384 Acked Read 1 Done Sent Done false
6581 2385 Acked Read 3 Done Sent Done false
6582 2386 Acked Write 2 Done Sent Done false
6583 2387 Acked Write 8 Done Sent Done false
6584 2388 Acked Write 6 Done Sent Done false
6585 2389 Acked Write 10 Done Sent Done false
6586 2390 Acked Write 6 Done Sent Done false
6587 2391 Acked Flush 0 Done Sent Done false
6588 2392 Acked Flush 0 Done Sent Done false
6589 2393 Acked Read 3 Done Sent Done false
6590 2394 Acked Read 7 Done Sent Done false
6591 2395 Acked Read 7 Done Sent Done false
6592 2396 Acked Write 9 Done Sent Done false
6593 2397 Acked Read 3 Done Sent Done false
6594 2398 Acked Write 5 Done Sent Done false
6595 2399 Acked Read 8 Done Sent Done false
6596 2400 Acked Read 2 Done Sent Done false
6597 2401 Acked Read 10 Done Sent Done false
6598 2402 Acked Read 6 Done Sent Done false
6599 2403 Acked Write 5 Done Sent Done false
6600 2404 Acked Read 6 Done Sent Done false
6601 2405 Acked Write 7 Done Sent Done false
6602 2406 Acked Read 1 Done Sent Done false
6603 2407 Acked Flush 0 Done Sent Done false
6604 2408 Acked Read 4 Done Sent Done false
6605 2409 Acked Write 1 Done Sent Done false
6606 2410 Acked Read 8 Done Sent Done false
6607 2411 Acked Write 2 Done Sent Done false
6608 2412 Acked Write 1 Done Sent Done false
6609 2413 Acked Read 2 Done Sent Done false
6610 2414 Acked Read 7 Done Sent Done false
6611 2415 Acked Read 2 Done Sent Done false
6612 2416 Acked Read 1 Done Sent Done false
6613 2417 Acked Write 10 Done Sent Done false
6614 2418 Acked Write 3 Done Sent Done false
6615 2419 Acked Write 9 Done Sent Done false
6616 2420 Acked Read 8 Done Sent Done false
6617 2421 Acked Read 9 Done Sent Done false
6618 2422 Acked Write 7 Done Sent Done false
6619 2423 Acked Flush 0 Done Sent Done false
6620 2424 Acked Read 4 Done Sent Done false
6621 2425 Acked Read 10 Done Sent Done false
6622 2426 Acked Read 3 Done Sent Done false
6623 2427 Acked Read 7 Done Sent Done false
6624 2428 Acked Write 9 Done Sent Done false
6625 2429 Acked Write 3 Done Sent Done false
6626 2430 Acked Flush 0 Done Sent Done false
6627 2431 Acked Write 1 Done Sent Done false
6628 2432 Acked Write 7 Done Sent Done false
6629 2433 Acked Write 3 Done Sent Done false
6630 2434 Acked Write 8 Done Sent Done false
6631 2435 Acked Write 1 Done Sent Done false
6632 2436 Acked Write 5 Done Sent Done false
6633 2437 Acked Write 6 Done Sent Done false
6634 2438 Acked Write 4 Done Sent Done false
6635 2439 Acked Read 8 Done Sent Done false
6636 2440 Acked Write 3 Done Sent Done false
6637 2441 Acked Write 3 Done Sent Done false
6638 2442 Acked Flush 0 Done Sent Done false
6639 2443 Acked Read 8 Done Sent Done false
6640 2444 Acked Read 3 Done Sent Done false
6641 2445 Acked Write 9 Done Sent Done false
6642 2446 Acked Read 5 Done Sent Done false
6643 2447 Acked Flush 0 Done Sent Done false
6644 2448 Acked Read 4 Done Sent Done false
6645 2449 Acked Write 4 Done Sent Done false
6646 2450 Acked Flush 0 Done Sent Done false
6647 2451 Acked Write 8 Done Sent Done false
6648 2452 Acked Flush 0 Done Sent Done false
6649 2453 Acked Read 4 Done Sent Done false
6650 2454 Acked Write 4 Done Sent Done false
6651 2455 Acked Read 5 Done Sent Done false
6652 2456 Acked Read 8 Done Sent Done false
6653 2457 Acked Write 7 Done Sent Done false
6654 2458 Acked Read 3 Done Sent Done false
6655 2459 Acked Read 1 Done Sent Done false
6656 2460 Acked Read 2 Done Sent Done false
6657 2461 Acked Read 5 Done Sent Done false
6658 2462 Acked Read 2 Done Sent Done false
6659 2463 Acked Read 5 Done Sent Done false
6660 2464 Acked Write 4 Done Sent Done false
6661 2465 Acked Write 8 Done Sent Done false
6662 2466 Acked Read 1 Done Sent Done false
6663 2467 Acked Write 8 Done Sent Done false
6664 2468 Acked Read 6 Done Sent Done false
6665 2469 Acked Read 5 Done Sent Done false
6666 2470 Acked Write 8 Done Sent Done false
6667 2471 Acked Write 6 Done Sent Done false
6668 2472 Acked Read 10 Done Sent Done false
6669 2473 Acked Write 1 Done Sent Done false
6670 2474 Acked Read 3 Done Sent Done false
6671 2475 Acked Read 4 Done Sent Done false
6672 2476 Acked Flush 0 Done Sent Done false
6673 2477 Acked Read 5 Done Sent Done false
6674 2478 Acked Write 6 Done Sent Done false
6675 2479 Acked Write 5 Done Sent Done false
6676 2480 Acked Read 3 Done Sent Done false
6677 2481 Acked Read 3 Done Sent Done false
6678 2482 Acked Read 6 Done Sent Done false
6679 2483 Acked Write 4 Done Sent Done false
6680 2484 Acked Flush 0 Done Sent Done false
6681 2485 Acked Write 3 Done Sent Done false
6682 2486 Acked Read 2 Done Sent Done false
6683 2487 Acked Write 2 Done Sent Done false
6684 2488 Acked Read 6 Done Sent Done false
6685 2489 Acked Read 9 Done Sent Done false
6686 2490 Acked Read 1 Done Sent Done false
6687 2491 Acked Flush 0 Done Sent Done false
6688 2492 Acked Write 5 Done Sent Done false
6689 2493 Acked Write 6 Done Sent Done false
6690 2494 Acked Read 8 Done Sent Done false
6691 2495 Acked Write 6 Done Sent Done false
6692 2496 Acked Read 2 Done Sent Done false
6693 2497 Acked Read 6 Done Sent Done false
6694 2498 Acked Read 1 Done Sent Done false
6695 2499 Acked Read 5 Done Sent Done false
6696 2500 Acked Write 3 Done Sent Done false
6697 2501 Acked Read 10 Done Sent Done false
6698 2502 Acked Read 9 Done Sent Done false
6699 2503 Acked Write 7 Done Sent Done false
6700 2504 Acked Read 6 Done Sent Done false
6701 2505 Acked Read 7 Done Sent Done false
6702 2506 Acked Write 10 Done Sent Done false
6703 2507 Acked Read 6 Done Sent Done false
6704 STATES DS:0 DS:1 DS:2 TOTAL
6705 Sent 0 301 0 301
6706 Done 301 0 301 602
6707 Skipped 0 0 0 0
6708 Error 0 0 0 0
6709 Last Flush: Some(JobId(2491)) Some(JobId(2206)) Some(JobId(2491))
6710 Downstairs last five completed: 2206 2205 2204 2203 2202
6711 Upstairs last five completed: 2507 2506 2505 2504 2503
6712 CLIENT: Up:0 ds:301 act:2
67132025-03-07T18:14:02.282ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
67142025-03-07T18:14:03.093ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
6715 [8820] got start action
6716 [8820] I am going to restart
6717 Starting downstairs at port 8820
6718 Make output file at "/tmp/dsc/downstairs-8820.txt"
6719 [8820][1] reports Starting
6720 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1213)
6721 [8820][1] reports Running
67222025-03-07T18:14:04.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
67232025-03-07T18:14:05.096ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
6724 ----------------------------------------------------------------
6725 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6726 States: Active Offline Active
6727 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6728 2207 Acked Write 1 Done Sent Done false
6729 2208 Acked Write 10 Done Sent Done false
6730 2209 Acked Write 8 Done Sent Done false
6731 2210 Acked Read 9 Done Sent Done false
6732 2211 Acked Read 3 Done Sent Done false
6733 2212 Acked Read 9 Done Sent Done false
6734 2213 Acked Read 7 Done Sent Done false
6735 2214 Acked Write 9 Done Sent Done false
6736 2215 Acked Read 9 Done Sent Done false
6737 2216 Acked Read 2 Done Sent Done false
6738 2217 Acked Read 10 Done Sent Done false
6739 2218 Acked Write 9 Done Sent Done false
6740 2219 Acked Write 2 Done Sent Done false
6741 2220 Acked Write 1 Done Sent Done false
6742 2221 Acked Read 6 Done Sent Done false
6743 2222 Acked Read 6 Done Sent Done false
6744 2223 Acked Read 6 Done Sent Done false
6745 2224 Acked Read 10 Done Sent Done false
6746 2225 Acked Read 7 Done Sent Done false
6747 2226 Acked Read 5 Done Sent Done false
6748 2227 Acked Read 9 Done Sent Done false
6749 2228 Acked Read 9 Done Sent Done false
6750 2229 Acked Write 3 Done Sent Done false
6751 2230 Acked Write 8 Done Sent Done false
6752 2231 Acked Read 10 Done Sent Done false
6753 2232 Acked Write 2 Done Sent Done false
6754 2233 Acked Read 10 Done Sent Done false
6755 2234 Acked Read 8 Done Sent Done false
6756 2235 Acked Write 7 Done Sent Done false
6757 2236 Acked Read 2 Done Sent Done false
6758 2237 Acked Read 9 Done Sent Done false
6759 2238 Acked Write 7 Done Sent Done false
6760 2239 Acked Write 10 Done Sent Done false
6761 2240 Acked Write 9 Done Sent Done false
6762 2241 Acked Write 4 Done Sent Done false
6763 2242 Acked Write 7 Done Sent Done false
6764 2243 Acked Read 5 Done Sent Done false
6765 2244 Acked Read 5 Done Sent Done false
6766 2245 Acked Write 5 Done Sent Done false
6767 2246 Acked Read 4 Done Sent Done false
6768 2247 Acked Read 3 Done Sent Done false
6769 2248 Acked Flush 0 Done Sent Done false
6770 2249 Acked Read 1 Done Sent Done false
6771 2250 Acked Write 9 Done Sent Done false
6772 2251 Acked Flush 0 Done Sent Done false
6773 2252 Acked Write 9 Done Sent Done false
6774 2253 Acked Read 9 Done Sent Done false
6775 2254 Acked Read 8 Done Sent Done false
6776 2255 Acked Flush 0 Done Sent Done false
6777 2256 Acked Write 2 Done Sent Done false
6778 2257 Acked Write 7 Done Sent Done false
6779 2258 Acked Read 9 Done Sent Done false
6780 2259 Acked Read 9 Done Sent Done false
6781 2260 Acked Read 1 Done Sent Done false
6782 2261 Acked Read 4 Done Sent Done false
6783 2262 Acked Read 2 Done Sent Done false
6784 2263 Acked Flush 0 Done Sent Done false
6785 2264 Acked Write 7 Done Sent Done false
6786 2265 Acked Read 4 Done Sent Done false
6787 2266 Acked Read 3 Done Sent Done false
6788 2267 Acked Flush 0 Done Sent Done false
6789 2268 Acked Read 2 Done Sent Done false
6790 2269 Acked Write 6 Done Sent Done false
6791 2270 Acked Read 9 Done Sent Done false
6792 2271 Acked Flush 0 Done Sent Done false
6793 2272 Acked Flush 0 Done Sent Done false
6794 2273 Acked Read 4 Done Sent Done false
6795 2274 Acked Write 4 Done Sent Done false
6796 2275 Acked Read 6 Done Sent Done false
6797 2276 Acked Write 4 Done Sent Done false
6798 2277 Acked Read 8 Done Sent Done false
6799 2278 Acked Read 7 Done Sent Done false
6800 2279 Acked Write 4 Done Sent Done false
6801 2280 Acked Read 2 Done Sent Done false
6802 2281 Acked Write 6 Done Sent Done false
6803 2282 Acked Flush 0 Done Sent Done false
6804 2283 Acked Write 7 Done Sent Done false
6805 2284 Acked Write 1 Done Sent Done false
6806 2285 Acked Read 9 Done Sent Done false
6807 2286 Acked Read 5 Done Sent Done false
6808 2287 Acked Write 8 Done Sent Done false
6809 2288 Acked Read 6 Done Sent Done false
6810 2289 Acked Write 10 Done Sent Done false
6811 2290 Acked Read 5 Done Sent Done false
6812 2291 Acked Write 2 Done Sent Done false
6813 2292 Acked Read 8 Done Sent Done false
6814 2293 Acked Read 1 Done Sent Done false
6815 2294 Acked Read 6 Done Sent Done false
6816 2295 Acked Read 8 Done Sent Done false
6817 2296 Acked Read 7 Done Sent Done false
6818 2297 Acked Write 8 Done Sent Done false
6819 2298 Acked Write 4 Done Sent Done false
6820 2299 Acked Write 7 Done Sent Done false
6821 2300 Acked Write 10 Done Sent Done false
6822 2301 Acked Read 5 Done Sent Done false
6823 2302 Acked Read 3 Done Sent Done false
6824 2303 Acked Read 7 Done Sent Done false
6825 2304 Acked Write 7 Done Sent Done false
6826 2305 Acked Read 4 Done Sent Done false
6827 2306 Acked Write 1 Done Sent Done false
6828 2307 Acked Flush 0 Done Sent Done false
6829 2308 Acked Read 10 Done Sent Done false
6830 2309 Acked Write 4 Done Sent Done false
6831 2310 Acked Read 8 Done Sent Done false
6832 2311 Acked Read 1 Done Sent Done false
6833 2312 Acked Read 1 Done Sent Done false
6834 2313 Acked Read 1 Done Sent Done false
6835 2314 Acked Read 9 Done Sent Done false
6836 2315 Acked Read 9 Done Sent Done false
6837 2316 Acked Read 2 Done Sent Done false
6838 2317 Acked Write 6 Done Sent Done false
6839 2318 Acked Write 7 Done Sent Done false
6840 2319 Acked Read 9 Done Sent Done false
6841 2320 Acked Write 4 Done Sent Done false
6842 2321 Acked Write 6 Done Sent Done false
6843 2322 Acked Read 1 Done Sent Done false
6844 2323 Acked Read 8 Done Sent Done false
6845 2324 Acked Write 5 Done Sent Done false
6846 2325 Acked Flush 0 Done Sent Done false
6847 2326 Acked Read 1 Done Sent Done false
6848 2327 Acked Read 8 Done Sent Done false
6849 2328 Acked Write 2 Done Sent Done false
6850 2329 Acked Flush 0 Done Sent Done false
6851 2330 Acked Read 4 Done Sent Done false
6852 2331 Acked Flush 0 Done Sent Done false
6853 2332 Acked Read 4 Done Sent Done false
6854 2333 Acked Write 7 Done Sent Done false
6855 2334 Acked Read 8 Done Sent Done false
6856 2335 Acked Write 10 Done Sent Done false
6857 2336 Acked Read 9 Done Sent Done false
6858 2337 Acked Write 9 Done Sent Done false
6859 2338 Acked Read 10 Done Sent Done false
6860 2339 Acked Write 7 Done Sent Done false
6861 2340 Acked Write 4 Done Sent Done false
6862 2341 Acked Write 1 Done Sent Done false
6863 2342 Acked Read 3 Done Sent Done false
6864 2343 Acked Read 2 Done Sent Done false
6865 2344 Acked Read 1 Done Sent Done false
6866 2345 Acked Write 2 Done Sent Done false
6867 2346 Acked Write 6 Done Sent Done false
6868 2347 Acked Read 1 Done Sent Done false
6869 2348 Acked Read 2 Done Sent Done false
6870 2349 Acked Write 8 Done Sent Done false
6871 2350 Acked Flush 0 Done Sent Done false
6872 2351 Acked Flush 0 Done Sent Done false
6873 2352 Acked Write 5 Done Sent Done false
6874 2353 Acked Read 2 Done Sent Done false
6875 2354 Acked Read 10 Done Sent Done false
6876 2355 Acked Read 8 Done Sent Done false
6877 2356 Acked Read 6 Done Sent Done false
6878 2357 Acked Read 9 Done Sent Done false
6879 2358 Acked Flush 0 Done Sent Done false
6880 2359 Acked Flush 0 Done Sent Done false
6881 2360 Acked Read 1 Done Sent Done false
6882 2361 Acked Flush 0 Done Sent Done false
6883 2362 Acked Read 10 Done Sent Done false
6884 2363 Acked Read 8 Done Sent Done false
6885 2364 Acked Write 4 Done Sent Done false
6886 2365 Acked Read 2 Done Sent Done false
6887 2366 Acked Read 4 Done Sent Done false
6888 2367 Acked Read 1 Done Sent Done false
6889 2368 Acked Read 6 Done Sent Done false
6890 2369 Acked Read 8 Done Sent Done false
6891 2370 Acked Write 1 Done Sent Done false
6892 2371 Acked Write 4 Done Sent Done false
6893 2372 Acked Read 8 Done Sent Done false
6894 2373 Acked Flush 0 Done Sent Done false
6895 2374 Acked Write 10 Done Sent Done false
6896 2375 Acked Read 3 Done Sent Done false
6897 2376 Acked Write 9 Done Sent Done false
6898 2377 Acked Read 6 Done Sent Done false
6899 2378 Acked Write 7 Done Sent Done false
6900 2379 Acked Read 5 Done Sent Done false
6901 2380 Acked Read 8 Done Sent Done false
6902 2381 Acked Read 10 Done Sent Done false
6903 2382 Acked Read 5 Done Sent Done false
6904 2383 Acked Read 10 Done Sent Done false
6905 2384 Acked Read 1 Done Sent Done false
6906 2385 Acked Read 3 Done Sent Done false
6907 2386 Acked Write 2 Done Sent Done false
6908 2387 Acked Write 8 Done Sent Done false
6909 2388 Acked Write 6 Done Sent Done false
6910 2389 Acked Write 10 Done Sent Done false
6911 2390 Acked Write 6 Done Sent Done false
6912 2391 Acked Flush 0 Done Sent Done false
6913 2392 Acked Flush 0 Done Sent Done false
6914 2393 Acked Read 3 Done Sent Done false
6915 2394 Acked Read 7 Done Sent Done false
6916 2395 Acked Read 7 Done Sent Done false
6917 2396 Acked Write 9 Done Sent Done false
6918 2397 Acked Read 3 Done Sent Done false
6919 2398 Acked Write 5 Done Sent Done false
6920 2399 Acked Read 8 Done Sent Done false
6921 2400 Acked Read 2 Done Sent Done false
6922 2401 Acked Read 10 Done Sent Done false
6923 2402 Acked Read 6 Done Sent Done false
6924 2403 Acked Write 5 Done Sent Done false
6925 2404 Acked Read 6 Done Sent Done false
6926 2405 Acked Write 7 Done Sent Done false
6927 2406 Acked Read 1 Done Sent Done false
6928 2407 Acked Flush 0 Done Sent Done false
6929 2408 Acked Read 4 Done Sent Done false
6930 2409 Acked Write 1 Done Sent Done false
6931 2410 Acked Read 8 Done Sent Done false
6932 2411 Acked Write 2 Done Sent Done false
6933 2412 Acked Write 1 Done Sent Done false
6934 2413 Acked Read 2 Done Sent Done false
6935 2414 Acked Read 7 Done Sent Done false
6936 2415 Acked Read 2 Done Sent Done false
6937 2416 Acked Read 1 Done Sent Done false
6938 2417 Acked Write 10 Done Sent Done false
6939 2418 Acked Write 3 Done Sent Done false
6940 2419 Acked Write 9 Done Sent Done false
6941 2420 Acked Read 8 Done Sent Done false
6942 2421 Acked Read 9 Done Sent Done false
6943 2422 Acked Write 7 Done Sent Done false
6944 2423 Acked Flush 0 Done Sent Done false
6945 2424 Acked Read 4 Done Sent Done false
6946 2425 Acked Read 10 Done Sent Done false
6947 2426 Acked Read 3 Done Sent Done false
6948 2427 Acked Read 7 Done Sent Done false
6949 2428 Acked Write 9 Done Sent Done false
6950 2429 Acked Write 3 Done Sent Done false
6951 2430 Acked Flush 0 Done Sent Done false
6952 2431 Acked Write 1 Done Sent Done false
6953 2432 Acked Write 7 Done Sent Done false
6954 2433 Acked Write 3 Done Sent Done false
6955 2434 Acked Write 8 Done Sent Done false
6956 2435 Acked Write 1 Done Sent Done false
6957 2436 Acked Write 5 Done Sent Done false
6958 2437 Acked Write 6 Done Sent Done false
6959 2438 Acked Write 4 Done Sent Done false
6960 2439 Acked Read 8 Done Sent Done false
6961 2440 Acked Write 3 Done Sent Done false
6962 2441 Acked Write 3 Done Sent Done false
6963 2442 Acked Flush 0 Done Sent Done false
6964 2443 Acked Read 8 Done Sent Done false
6965 2444 Acked Read 3 Done Sent Done false
6966 2445 Acked Write 9 Done Sent Done false
6967 2446 Acked Read 5 Done Sent Done false
6968 2447 Acked Flush 0 Done Sent Done false
6969 2448 Acked Read 4 Done Sent Done false
6970 2449 Acked Write 4 Done Sent Done false
6971 2450 Acked Flush 0 Done Sent Done false
6972 2451 Acked Write 8 Done Sent Done false
6973 2452 Acked Flush 0 Done Sent Done false
6974 2453 Acked Read 4 Done Sent Done false
6975 2454 Acked Write 4 Done Sent Done false
6976 2455 Acked Read 5 Done Sent Done false
6977 2456 Acked Read 8 Done Sent Done false
6978 2457 Acked Write 7 Done Sent Done false
6979 2458 Acked Read 3 Done Sent Done false
6980 2459 Acked Read 1 Done Sent Done false
6981 2460 Acked Read 2 Done Sent Done false
6982 2461 Acked Read 5 Done Sent Done false
6983 2462 Acked Read 2 Done Sent Done false
6984 2463 Acked Read 5 Done Sent Done false
6985 2464 Acked Write 4 Done Sent Done false
6986 2465 Acked Write 8 Done Sent Done false
6987 2466 Acked Read 1 Done Sent Done false
6988 2467 Acked Write 8 Done Sent Done false
6989 2468 Acked Read 6 Done Sent Done false
6990 2469 Acked Read 5 Done Sent Done false
6991 2470 Acked Write 8 Done Sent Done false
6992 2471 Acked Write 6 Done Sent Done false
6993 2472 Acked Read 10 Done Sent Done false
6994 2473 Acked Write 1 Done Sent Done false
6995 2474 Acked Read 3 Done Sent Done false
6996 2475 Acked Read 4 Done Sent Done false
6997 2476 Acked Flush 0 Done Sent Done false
6998 2477 Acked Read 5 Done Sent Done false
6999 2478 Acked Write 6 Done Sent Done false
7000 2479 Acked Write 5 Done Sent Done false
7001 2480 Acked Read 3 Done Sent Done false
7002 2481 Acked Read 3 Done Sent Done false
7003 2482 Acked Read 6 Done Sent Done false
7004 2483 Acked Write 4 Done Sent Done false
7005 2484 Acked Flush 0 Done Sent Done false
7006 2485 Acked Write 3 Done Sent Done false
7007 2486 Acked Read 2 Done Sent Done false
7008 2487 Acked Write 2 Done Sent Done false
7009 2488 Acked Read 6 Done Sent Done false
7010 2489 Acked Read 9 Done Sent Done false
7011 2490 Acked Read 1 Done Sent Done false
7012 2491 Acked Flush 0 Done Sent Done false
7013 2492 Acked Write 5 Done Sent Done false
7014 2493 Acked Write 6 Done Sent Done false
7015 2494 Acked Read 8 Done Sent Done false
7016 2495 Acked Write 6 Done Sent Done false
7017 2496 Acked Read 2 Done Sent Done false
7018 2497 Acked Read 6 Done Sent Done false
7019 2498 Acked Read 1 Done Sent Done false
7020 2499 Acked Read 5 Done Sent Done false
7021 2500 Acked Write 3 Done Sent Done false
7022 2501 Acked Read 10 Done Sent Done false
7023 2502 Acked Read 9 Done Sent Done false
7024 2503 Acked Write 7 Done Sent Done false
7025 2504 Acked Read 6 Done Sent Done false
7026 2505 Acked Read 7 Done Sent Done false
7027 2506 Acked Write 10 Done Sent Done false
7028 2507 Acked Read 6 Done Sent Done false
7029 STATES DS:0 DS:1 DS:2 TOTAL
7030 Sent 0 301 0 301
7031 Done 301 0 301 602
7032 Skipped 0 0 0 0
7033 Error 0 0 0 0
7034 Last Flush: Some(JobId(2491)) Some(JobId(2206)) Some(JobId(2491))
7035 Downstairs last five completed: 2206 2205 2204 2203 2202
7036 Upstairs last five completed: 2507 2506 2505 2504 2503
7037 CLIENT: Up:0 ds:301 act:2
70382025-03-07T18:14:06.285ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70392025-03-07T18:14:07.259ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70402025-03-07T18:14:07.528ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70412025-03-07T18:14:07.528ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52381) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70422025-03-07T18:14:07.529ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70432025-03-07T18:14:07.529ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70442025-03-07T18:14:07.529ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70452025-03-07T18:14:07.529ZINFOcrucible: send last flush ID to this DS: Some(JobId(2206)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70462025-03-07T18:14:07.529ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70472025-03-07T18:14:07.530ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2206)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70482025-03-07T18:14:07.530ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2206)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
7049 ----------------------------------------------------------------
7050 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7051 Downstairs last five completed: 2508 2507 2506 2505 2504
7052 Upstairs last five completed: 2508 2507 2506 2505 2504
7053 CLIENT: Up:0 ds:0 act:3
7054 Replay: All jobs finished
7055 Main task has work to do, go find it
7056 got dsc Stop(0)
7057 stop 0
7058 Mar 07 18:14:09.786 INFO request completed, latency_us: 323, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0ffaa147-4eae-47d4-ad83-dd65c86cca86, remote_addr:[8810] Got stop action so:false kr:false
7059 127.0.0.1:64322, local_addr: 127.0.0.1:9998
70602025-03-07T18:14:09.787ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70612025-03-07T18:14:09.787ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70622025-03-07T18:14:09.787ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
70632025-03-07T18:14:09.787ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
7064 Mar 07 18:14:09.787 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: f58a2839-3e3b-4e35-8d6d-5901d565ae08, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
7065 [8810] Exited with: ExitStatus(unix_wait_status(9))
7066 [8810][0] reports Exit
7067 Mar 07 18:14:13.790 INFO request completed, latency_us: 305, response_code: 200, uri: /state/cid/0, method: GET, req_id: cd579879-7b4b-46b0-b408-706b82629f46, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
7068 001/300 Write block 2944 len 8192 data: 5 5
7069 002/300 Write block 796 len 16384 data: 2 3 3 3
7070 003/300 Flush
7071 004/300 Write block 26 len 4096 data: 2
7072 005/300 Read block 3950 len 40960
7073 006/300 Write block 4644 len 20480 data: 4 4 3 3 3
7074 007/300 Read block 2621 len 8192
7075 008/300 Write block 4739 len 40960 data: 2 2 2 2 3 3 3 3 3 3
7076 009/300 Read block 3616 len 40960
7077 010/300 Flush
7078 011/300 Read block 3640 len 4096
7079 012/300 Write block 1295 len 28672 data: 3 3 3 3 3 2 2
7080 013/300 Write block 4044 len 32768 data: 3 3 3 3 3 3 2 2
7081 014/300 Read block 4953 len 8192
7082 015/300 Write block 938 len 8192 data: 2 2
7083 016/300 Write block 403 len 36864 data: 2 2 2 2 2 2 2 3 3
7084 017/300 Write block 2884 len 16384 data: 3 4 4 4
7085 018/300 Write block 4779 len 32768 data: 3 3 2 2 2 2 2 3
7086 019/300 Read block 1455 len 4096
7087 020/300 Read block 3099 len 4096
7088 021/300 Write block 1461 len 36864 data: 3 4 4 4 3 3 3 3 3
7089 022/300 Write block 3906 len 16384 data: 3 3 2 2
7090 023/300 Read block 3561 len 24576
7091 024/300 Flush
7092 025/300 Read block 2671 len 40960
7093 026/300 Flush
7094 027/300 Write block 1012 len 16384 data: 4 4 4 4
7095 028/300 Write block 411 len 36864 data: 4 3 3 3 3 2 2 2 2
7096 029/300 Flush
7097 030/300 Flush
7098 031/300 Read block 2057 len 12288
7099 032/300 Write block 1332 len 4096 data: 2
7100 033/300 Write block 1996 len 4096 data: 2
7101 034/300 Read block 3284 len 12288
7102 035/300 Write block 459 len 4096 data: 2
7103 036/300 Write block 4758 len 4096 data: 3
7104 037/300 Write block 1936 len 20480 data: 2 2 2 2 2
7105 038/300 Write block 3914 len 36864 data: 2 2 2 2 2 3 3 3 3
7106 039/300 Read block 1028 len 36864
7107 040/300 Flush
7108 041/300 Flush
7109 042/300 Read block 2929 len 12288
7110 043/300 Write block 1113 len 4096 data: 2
7111 044/300 Read block 4681 len 28672
7112 045/300 Read block 1058 len 12288
7113 046/300 Read block 4349 len 24576
7114 047/300 Read block 4073 len 36864
7115 048/300 Read block 4883 len 36864
7116 049/300 Write block 1463 len 36864 data: 5 5 4 4 4 4 4 3 2
7117 050/300 Write block 1703 len 32768 data: 5 5 5 3 3 3 3 3
7118 051/300 Read block 2667 len 32768
7119 052/300 Write block 740 len 36864 data: 3 2 2 2 2 2 2 2 2
7120 053/300 Read block 4951 len 36864
7121 054/300 Write block 1853 len 36864 data: 2 2 2 2 2 2 2 2 2
7122 055/300 Read block 2169 len 12288
7123 056/300 Flush
7124 057/300 Read block 616 len 12288
7125 058/300 Write block 772 len 40960 data: 3 3 3 3 2 3 3 3 3 2
7126 059/300 Write block 2478 len 24576 data: 2 2 2 2 3 3
7127 060/300 Write block 19 len 20480 data: 3 3 3 3 3
7128 061/300 Read block 1982 len 36864
7129 062/300 Read block 811 len 8192
7130 063/300 Read block 3425 len 36864
7131 064/300 Write block 4543 len 16384 data: 4 4 5 5
7132 065/300 Read block 2427 len 24576
7133 066/300 Read block 3305 len 32768
7134 067/300 Flush
7135 068/300 Read block 2381 len 24576
7136 069/300 Write block 4667 len 8192 data: 2 2
7137 070/300 Write block 3781 len 40960 data: 3 4 4 4 3 3 2 2 2 2
7138 071/300 Read block 4156 len 28672
7139 072/300 Flush
7140 073/300 Read block 3718 len 32768
7141 074/300 Flush
7142 075/300 Flush
7143 076/300 Read block 3227 len 36864
7144 077/300 Read block 1457 len 4096
7145 078/300 Write block 2533 len 16384 data: 3 3 3 3
7146 079/300 Write block 1253 len 4096 data: 2
7147 080/300 Read block 782 len 8192
7148 081/300 Read block 3974 len 32768
7149 082/300 Write block 2552 len 20480 data: 2 3 3 3 2
7150 083/300 Read block 3771 len 40960
7151 084/300 Write block 1403 len 28672 data: 2 2 3 3 3 3 3
7152 085/300 Write block 44 len 12288 data: 4 4 4
7153 086/300 Write block 2463 len 20480 data: 3 3 3 4 4
7154 087/300 Read block 4288 len 16384
7155 088/300 Write block 1624 len 12288 data: 4 4 3
7156 089/300 Write block 2606 len 16384 data: 2 2 2 2
7157 090/300 Read block 4671 len 32768
7158 091/300 Write block 2008 len 36864 data: 2 2 2 2 2 2 2 2 3
7159 092/300 Write block 4621 len 16384 data: 2 2 2 3
7160 093/300 Flush
7161 094/300 Write block 3630 len 40960 data: 2 2 2 3 3 3 4 3 3 3
7162 095/300 Read block 3449 len 36864
7163 096/300 Write block 201 len 40960 data: 2 2 3 3 3 3 2 2 3 3
7164 097/300 Write block 1080 len 12288 data: 2 2 2
7165 098/300 Write block 1021 len 36864 data: 2 2 2 2 2 2 2 2 2
7166 099/300 Write block 2683 len 4096 data: 3
7167 100/300 Write block 3460 len 4096 data: 2
7168 101/300 Write block 3192 len 24576 data: 3 2 2 2 2 2
7169 102/300 Read block 430 len 24576
7170 103/300 Read block 4725 len 28672
7171 104/300 Read block 1492 len 12288
7172 105/300 Write block 2415 len 12288 data: 2 2 2
7173 106/300 Write block 1092 len 4096 data: 4
7174 107/300 Read block 467 len 40960
7175 108/300 Read block 2317 len 12288
7176 109/300 Write block 3695 len 8192 data: 2 2
7177 110/300 Read block 4010 len 8192
7178 111/300 Read block 2893 len 32768
7179 112/300 Read block 3812 len 12288
7180 113/300 Write block 4215 len 32768 data: 3 3 3 3 3 3 4 5
7181 114/300 Flush
7182 115/300 Read block 2497 len 24576
7183 116/300 Write block 1600 len 16384 data: 4 4 4 4
7184 117/300 Read block 4331 len 36864
7185 118/300 Write block 2741 len 24576 data: 2 2 2 2 2 2
7186 119/300 Read block 1333 len 32768
7187 120/300 Write block 1220 len 12288 data: 2 2 2
7188 121/300 Read block 3494 len 36864
7189 122/300 Write block 4270 len 32768 data: 2 2 2 2 3 3 3 3
7190 123/300 Write block 4624 len 24576 data: 4 3 3 3 3 3
7191 124/300 Read block 1710 len 36864
7192 125/300 Read block 3028 len 32768
7193 126/300 Read block 106 len 36864
7194 127/300 Write block 4493 len 8192 data: 2 3
7195 128/300 Read block 1752 len 36864
7196 129/300 Flush
7197 130/300 Flush
7198 131/300 Read block 4318 len 32768
7199 132/300 Write block 423 len 8192 data: 2 2
7200 133/300 Read block 3725 len 16384
7201 134/300 Read block 3544 len 20480
7202 135/300 Write block 4824 len 24576 data: 2 2 2 2 2 2
7203 136/300 Flush
7204 137/300 Write block 239 len 28672 data: 2 2 2 2 2 3 3
7205 138/300 Read block 2772 len 20480
7206 139/300 Write block 1669 len 24576 data: 2 2 2 2 2 2
7207 140/300 Flush
7208 141/300 Write block 1742 len 16384 data: 2 2 2 3
7209 142/300 Write block 3450 len 40960 data: 3 3 3 3 3 3 3 3 3 3
7210 143/300 Read block 470 len 20480
7211 144/300 Flush
7212 145/300 Read block 2637 len 4096
7213 146/300 Flush
7214 147/300 Write block 851 len 4096 data: 2
7215 148/300 Write block 475 len 4096 data: 3
7216 149/300 Read block 1859 len 24576
7217 150/300 Write block 1495 len 36864 data: 2 2 2 2 2 2 2 2 2
7218 151/300 Write block 3147 len 24576 data: 3 2 2 2 2 2
7219 152/300 Write block 3743 len 24576 data: 3 4 3 4 4 3
7220 153/300 Flush
7221 154/300 Read block 2942 len 12288
7222 155/300 Flush
7223 156/300 Read block 13 len 32768
7224 157/300 Read block 1321 len 8192
7225 158/300 Read block 648 len 16384
7226 159/300 Read block 117 len 36864
7227 160/300 Write block 820 len 40960 data: 3 2 2 2 3 4 4 4 3 3
7228 161/300 Read block 2578 len 12288
7229 162/300 Read block 2608 len 16384
7230 163/300 Write block 213 len 20480 data: 5 4 6 5 3
7231 164/300 Write block 4851 len 28672 data: 2 2 2 2 2 2 2
7232 165/300 Write block 1027 len 20480 data: 3 3 3 2 2
7233 166/300 Read block 4728 len 12288
7234 167/300 Read block 1784 len 20480
7235 168/300 Read block 1184 len 16384
7236 169/300 Read block 4064 len 36864
7237 170/300 Write block 2305 len 28672 data: 2 2 2 2 2 2 2
7238 171/300 Read block 4281 len 36864
7239 172/300 Read block 3609 len 32768
7240 173/300 Read block 824 len 20480
7241 174/300 Read block 4242 len 40960
7242 175/300 Write block 1052 len 20480 data: 3 3 2 2 2
7243 176/300 Read block 4639 len 32768
7244 177/300 Write block 3414 len 28672 data: 2 2 2 2 2 2 2
7245 178/300 Read block 2573 len 8192
7246 179/300 Read block 2852 len 24576
7247 180/300 Flush
7248 181/300 Read block 765 len 36864
7249 182/300 Read block 2083 len 24576
7250 183/300 Read block 2688 len 40960
7251 184/300 Flush
7252 185/300 Read block 3317 len 12288
7253 186/300 Write block 1316 len 12288 data: 2 2 2
7254 187/300 Read block 430 len 28672
7255 188/300 Write block 68 len 8192 data: 2 2
7256 189/300 Flush
7257 190/300 Read block 2372 len 28672
7258 191/300 Read block 1367 len 8192
7259 192/300 Write block 425 len 4096 data: 2
7260 193/300 Read block 1770 len 24576
7261 194/300 Read block 4388 len 16384
7262 195/300 Write block 4136 len 16384 data: 3 3 3 3
7263 196/300 Write block 193 len 24576 data: 4 3 3 3 3 3
7264 197/300 Write block 731 len 40960 data: 2 2 3 3 4 4 4 4 3 4
7265 198/300 Write block 3267 len 12288 data: 4 3 4
7266 199/300 Read block 3603 len 8192
7267 200/300 Write block 1559 len 12288 data: 2 2 2
7268 201/300 Flush
7269 202/300 Write block 2005 len 24576 data: 2 2 2 3 3 3
7270 203/300 Write block 3829 len 36864 data: 2 2 2 2 2 2 2 2 2
7271 204/300 Write block 627 len 40960 data: 4 2 2 2 3 3 2 2 2 2
7272 205/300 Read block 4617 len 4096
7273 206/300 Flush
7274 207/300 Read block 395 len 8192
7275 208/300 Read block 2143 len 40960
7276 209/300 Read block 2811 len 4096
7277 210/300 Write block 2650 len 36864 data: 3 3 2 2 2 3 3 4 3
7278 211/300 Read block 4422 len 12288
7279 212/300 Write block 4113 len 4096 data: 3
7280 213/300 Write block 4328 len 20480 data: 4 3 4 4 4
7281 214/300 Read block 629 len 4096
7282 215/300 Read block 550 len 4096
7283 216/300 Read block 2988 len 4096
7284 217/300 Read block 3703 len 20480
7285 218/300 Read block 935 len 16384
7286 219/300 Read block 215 len 12288
7287 220/300 Write block 369 len 4096 data: 3
7288 221/300 Write block 4277 len 36864 data: 4 4 5 5 5 4 5 5 4
7289 222/300 Write block 3509 len 8192 data: 3 3
7290 223/300 Write block 4720 len 4096 data: 2
7291 224/300 Write block 3715 len 16384 data: 3 3 2 2
7292 225/300 Write block 2062 len 24576 data: 4 5 5 5 5 4
7293 226/300 Read block 1657 len 16384
7294 227/300 Read block 2924 len 28672
7295 228/300 Write block 3694 len 36864 data: 2 3 3 2 2 3 3 3 3
7296 229/300 Read block 2735 len 16384
7297 230/300 Read block 566 len 20480
7298 231/300 Flush
7299 232/300 Read block 83 len 8192
7300 233/300 Read block 475 len 20480
7301 234/300 Write block 2222 len 28672 data: 2 2 2 2 3 3 3
7302 235/300 Write block 1008 len 16384 data: 4 4 4 4
7303 236/300 Read block 4553 len 32768
7304 237/300 Flush
7305 238/300 Read block 1720 len 4096
7306 239/300 Read block 4147 len 28672
7307 240/300 Write block 578 len 20480 data: 2 2 2 2 3
7308 241/300 Read block 472 len 4096
7309 242/300 Read block 1893 len 12288
7310 243/300 Read block 693 len 12288
7311 244/300 Read block 1900 len 36864
7312 245/300 Write block 2460 len 4096 data: 2
7313 246/300 Read block 4834 len 20480
7314 247/300 Read block 4191 len 24576
7315 248/300 Read block 4680 len 40960
7316 249/300 Write block 1619 len 40960 data: 6 4 4 4 4 5 5 4 3 2
7317 250/300 Write block 1032 len 4096 data: 2
7318 251/300 Write block 1516 len 4096 data: 3
7319 252/300 Read block 64 len 4096
7320 253/300 Read block 239 len 8192
7321 254/300 Write block 3754 len 16384 data: 2 2 2 2
7322 255/300 Flush
7323 256/300 Flush
7324 257/300 Write block 3663 len 4096 data: 2
7325 258/300 Flush
7326 259/300 Flush
7327 260/300 Write block 3026 len 12288 data: 3 3 3
7328 261/300 Write block 2339 len 20480 data: 3 3 3 3 2
7329 262/300 Read block 4899 len 24576
7330 263/300 Read block 1993 len 28672
7331 264/300 Write block 208 len 4096 data: 3
7332 265/300 Write block 4771 len 40960 data: 4 4 3 2 2 2 3 3 4 4
7333 266/300 Read block 272 len 36864
7334 267/300 Read block 2404 len 8192
7335 268/300 Write block 3906 len 16384 data: 4 4 3 3
7336 269/300 Read block 3509 len 24576
7337 270/300 Read block 4028 len 40960
7338 271/300 Write block 4938 len 12288 data: 2 2 2
7339 272/300 Read block 585 len 28672
7340 273/300 Read block 1967 len 12288
7341 274/300 Write block 347 len 16384 data: 2 2 2 3
7342 275/300 Write block 4203 len 24576 data: 3 3 3 3 3 3
7343 276/300 Write block 4894 len 8192 data: 3 4
7344 277/300 Read block 3588 len 8192
7345 278/300 Read block 3596 len 24576
7346 279/300 Read block 1461 len 20480
7347 280/300 Flush
7348 281/300 Read block 2849 len 8192
7349 282/300 Read block 4753 len 12288
7350 283/300 Read block 1672 len 12288
7351 284/300 Write block 1564 len 24576 data: 3 4 4 4 4 4
7352 285/300 Read block 1641 len 12288
7353 286/300 Write block 4054 len 36864 data: 2 2 2 2 2 2 2 2 2
7354 287/300 Read block 130 len 40960
7355 288/300 Flush
7356 289/300 Read block 3173 len 16384
7357 290/300 Write block 3650 len 40960 data: 3 3 3 3 2 2 2 2 2 2
7358 291/300 Read block 2789 len 20480
7359 292/300 Read block 681 len 36864
7360 293/300 Read block 2406 len 12288
7361 294/300 Read block 3820 len 16384
7362 295/300 Write block 3420 len 4096 data: 3
7363 296/300 Write block 3186 len 32768 data: 3 5 4 4 3 3 4 3
7364 297/300 Write block 3795 len 16384 data: 4 4 5 4
7365 298/300 Flush
7366 299/300 Flush
7367 300/300 Write block 2000 len 4096 data: 3
7368 301/300 Flush
7369 Main task has work to do, go find it
7370 got dsc Start(0)
7371 start 0
7372 Mar 07 18:14:14.024 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/0, method: POST, req_id: 60708e41-9459-46f6-aeba-72e571daf5ef, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
7373 Replay: started 0, returned:Ok(())
7374 ----------------------------------------------------------------
7375 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7376 States: Offline Active Active
7377 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7378 2509 Acked Write 2 Sent Done Done false
7379 2510 Acked Write 4 Sent Done Done false
7380 2511 Acked Flush 0 Sent Done Done false
7381 2512 Acked Write 1 Sent Done Done false
7382 2513 Acked Read 10 Sent Done Done false
7383 2514 Acked Write 5 Sent Done Done false
7384 2515 Acked Read 2 Sent Done Done false
7385 2516 Acked Write 10 Sent Done Done false
7386 2517 Acked Read 10 Sent Done Done false
7387 2518 Acked Flush 0 Sent Done Done false
7388 2519 Acked Read 1 Sent Done Done false
7389 2520 Acked Write 7 Sent Done Done false
7390 2521 Acked Write 8 Sent Done Done false
7391 2522 Acked Read 2 Sent Done Done false
7392 2523 Acked Write 2 Sent Done Done false
7393 2524 Acked Write 9 Sent Done Done false
7394 2525 Acked Write 4 Sent Done Done false
7395 2526 Acked Write 8 Sent Done Done false
7396 2527 Acked Read 1 Sent Done Done false
7397 2528 Acked Read 1 Sent Done Done false
7398 2529 Acked Write 9 Sent Done Done false
7399 2530 Acked Write 4 Sent Done Done false
7400 2531 Acked Read 6 Sent Done Done false
7401 2532 Acked Flush 0 Sent Done Done false
7402 2533 Acked Read 10 Sent Done Done false
7403 2534 Acked Flush 0 Sent Done Done false
7404 2535 Acked Write 4 Sent Done Done false
7405 2536 Acked Write 9 Sent Done Done false
7406 2537 Acked Flush 0 Sent Done Done false
7407 2538 Acked Flush 0 Sent Done Done false
7408 2539 Acked Read 3 Sent Done Done false
7409 2540 Acked Write 1 Sent Done Done false
7410 2541 Acked Write 1 Sent Done Done false
7411 2542 Acked Read 3 Sent Done Done false
7412 2543 Acked Write 1 Sent Done Done false
7413 2544 Acked Write 1 Sent Done Done false
7414 2545 Acked Write 5 Sent Done Done false
7415 2546 Acked Write 9 Sent Done Done false
7416 2547 Acked Read 9 Sent Done Done false
7417 2548 Acked Flush 0 Sent Done Done false
7418 2549 Acked Flush 0 Sent Done Done false
7419 2550 Acked Read 3 Sent Done Done false
7420 2551 Acked Write 1 Sent Done Done false
7421 2552 Acked Read 7 Sent Done Done false
7422 2553 Acked Read 3 Sent Done Done false
7423 2554 Acked Read 6 Sent Done Done false
7424 2555 Acked Read 9 Sent Done Done false
7425 2556 Acked Read 9 Sent Done Done false
7426 2557 Acked Write 9 Sent Done Done false
7427 2558 Acked Write 8 Sent Done Done false
7428 2559 Acked Read 8 Sent Done Done false
7429 2560 Acked Write 9 Sent Done Done false
7430 2561 Acked Read 9 Sent Done Done false
7431 2562 Acked Write 9 Sent Done Done false
7432 2563 Acked Read 3 Sent Done Done false
7433 2564 Acked Flush 0 Sent Done Done false
7434 2565 Acked Read 3 Sent Done Done false
7435 2566 Acked Write 10 Sent Done Done false
7436 2567 Acked Write 6 Sent Done Done false
7437 2568 Acked Write 5 Sent Done Done false
7438 2569 Acked Read 9 Sent Done Done false
7439 2570 Acked Read 2 Sent Done Done false
7440 2571 Acked Read 9 Sent Done Done false
7441 2572 Acked Write 4 Sent Done Done false
7442 2573 Acked Read 6 Sent Done Done false
7443 2574 Acked Read 8 Sent Done Done false
7444 2575 Acked Flush 0 Sent Done Done false
7445 2576 Acked Read 6 Sent Done Done false
7446 2577 Acked Write 2 Sent Done Done false
7447 2578 Acked Write 10 Sent Done Done false
7448 2579 Acked Read 7 Sent Done Done false
7449 2580 Acked Flush 0 Sent Done Done false
7450 2581 Acked Read 8 Sent Done Done false
7451 2582 Acked Flush 0 Sent Done Done false
7452 2583 Acked Flush 0 Sent Done Done false
7453 2584 Acked Read 9 Sent Done Done false
7454 2585 Acked Read 1 Sent Done Done false
7455 2586 Acked Write 4 Sent Done Done false
7456 2587 Acked Write 1 Sent Done Done false
7457 2588 Acked Read 2 Sent Done Done false
7458 2589 Acked Read 8 Sent Done Done false
7459 2590 Acked Write 5 Sent Done Done false
7460 2591 Acked Read 10 Sent Done Done false
7461 2592 Acked Write 7 Sent Done Done false
7462 2593 Acked Write 3 Sent Done Done false
7463 2594 Acked Write 5 Sent Done Done false
7464 2595 Acked Read 4 Sent Done Done false
7465 2596 Acked Write 3 Sent Done Done false
7466 2597 Acked Write 4 Sent Done Done false
7467 2598 Acked Read 8 Sent Done Done false
7468 2599 Acked Write 9 Sent Done Done false
7469 2600 Acked Write 4 Sent Done Done false
7470 2601 Acked Flush 0 Sent Done Done false
7471 2602 Acked Write 10 Sent Done Done false
7472 2603 Acked Read 9 Sent Done Done false
7473 2604 Acked Write 10 Sent Done Done false
7474 2605 Acked Write 3 Sent Done Done false
7475 2606 Acked Write 9 Sent Done Done false
7476 2607 Acked Write 1 Sent Done Done false
7477 2608 Acked Write 1 Sent Done Done false
7478 2609 Acked Write 6 Sent Done Done false
7479 2610 Acked Read 6 Sent Done Done false
7480 2611 Acked Read 7 Sent Done Done false
7481 2612 Acked Read 3 Sent Done Done false
7482 2613 Acked Write 3 Sent Done Done false
7483 2614 Acked Write 1 Sent Done Done false
7484 2615 Acked Read 10 Sent Done Done false
7485 2616 Acked Read 3 Sent Done Done false
7486 2617 Acked Write 2 Sent Done Done false
7487 2618 Acked Read 2 Sent Done Done false
7488 2619 Acked Read 8 Sent Done Done false
7489 2620 Acked Read 3 Sent Done Done false
7490 2621 Acked Write 8 Sent Done Done false
7491 2622 Acked Flush 0 Sent Done Done false
7492 2623 Acked Read 6 Sent Done Done false
7493 2624 Acked Write 4 Sent Done Done false
7494 2625 Acked Read 9 Sent Done Done false
7495 2626 Acked Write 6 Sent Done Done false
7496 2627 Acked Read 8 Sent Done Done false
7497 2628 Acked Write 3 Sent Done Done false
7498 2629 Acked Read 9 Sent Done Done false
7499 2630 Acked Write 8 Sent Done Done false
7500 2631 Acked Write 6 Sent Done Done false
7501 2632 Acked Read 9 Sent Done Done false
7502 2633 Acked Read 8 Sent Done Done false
7503 2634 Acked Read 9 Sent Done Done false
7504 2635 Acked Write 2 Sent Done Done false
7505 2636 Acked Read 9 Sent Done Done false
7506 2637 Acked Flush 0 Sent Done Done false
7507 2638 Acked Flush 0 Sent Done Done false
7508 2639 Acked Read 8 Sent Done Done false
7509 2640 Acked Write 2 Sent Done Done false
7510 2641 Acked Read 4 Sent Done Done false
7511 2642 Acked Read 5 Sent Done Done false
7512 2643 Acked Write 6 Sent Done Done false
7513 2644 Acked Flush 0 Sent Done Done false
7514 2645 Acked Write 7 Sent Done Done false
7515 2646 Acked Read 5 Sent Done Done false
7516 2647 Acked Write 6 Sent Done Done false
7517 2648 Acked Flush 0 Sent Done Done false
7518 2649 Acked Write 4 Sent Done Done false
7519 2650 Acked Write 10 Sent Done Done false
7520 2651 Acked Read 5 Sent Done Done false
7521 2652 Acked Flush 0 Sent Done Done false
7522 2653 Acked Read 1 Sent Done Done false
7523 2654 Acked Flush 0 Sent Done Done false
7524 2655 Acked Write 1 Sent Done Done false
7525 2656 Acked Write 1 Sent Done Done false
7526 2657 Acked Read 6 Sent Done Done false
7527 2658 Acked Write 9 Sent Done Done false
7528 2659 Acked Write 6 Sent Done Done false
7529 2660 Acked Write 6 Sent Done Done false
7530 2661 Acked Flush 0 Sent Done Done false
7531 2662 Acked Read 3 Sent Done Done false
7532 2663 Acked Flush 0 Sent Done Done false
7533 2664 Acked Read 8 Sent Done Done false
7534 2665 Acked Read 2 Sent Done Done false
7535 2666 Acked Read 4 Sent Done Done false
7536 2667 Acked Read 9 Sent Done Done false
7537 2668 Acked Write 10 Sent Done Done false
7538 2669 Acked Read 3 Sent Done Done false
7539 2670 Acked Read 4 Sent Done Done false
7540 2671 Acked Write 5 Sent Done Done false
7541 2672 Acked Write 7 Sent Done Done false
7542 2673 Acked Write 5 Sent Done Done false
7543 2674 Acked Read 3 Sent Done Done false
7544 2675 Acked Read 5 Sent Done Done false
7545 2676 Acked Read 4 Sent Done Done false
7546 2677 Acked Read 9 Sent Done Done false
7547 2678 Acked Write 7 Sent Done Done false
7548 2679 Acked Read 9 Sent Done Done false
7549 2680 Acked Read 8 Sent Done Done false
7550 2681 Acked Read 5 Sent Done Done false
7551 2682 Acked Read 10 Sent Done Done false
7552 2683 Acked Write 5 Sent Done Done false
7553 2684 Acked Read 8 Sent Done Done false
7554 2685 Acked Write 7 Sent Done Done false
7555 2686 Acked Read 2 Sent Done Done false
7556 2687 Acked Read 6 Sent Done Done false
7557 2688 Acked Flush 0 Sent Done Done false
7558 2689 Acked Read 9 Sent Done Done false
7559 2690 Acked Read 6 Sent Done Done false
7560 2691 Acked Read 10 Sent Done Done false
7561 2692 Acked Flush 0 Sent Done Done false
7562 2693 Acked Read 3 Sent Done Done false
7563 2694 Acked Write 3 Sent Done Done false
7564 2695 Acked Read 7 Sent Done Done false
7565 2696 Acked Write 2 Sent Done Done false
7566 2697 Acked Flush 0 Sent Done Done false
7567 2698 Acked Read 7 Sent Done Done false
7568 2699 Acked Read 2 Sent Done Done false
7569 2700 Acked Write 1 Sent Done Done false
7570 2701 Acked Read 6 Sent Done Done false
7571 2702 Acked Read 4 Sent Done Done false
7572 2703 Acked Write 4 Sent Done Done false
7573 2704 Acked Write 6 Sent Done Done false
7574 2705 Acked Write 10 Sent Done Done false
7575 2706 Acked Write 3 Sent Done Done false
7576 2707 Acked Read 2 Sent Done Done false
7577 2708 Acked Write 3 Sent Done Done false
7578 2709 Acked Flush 0 Sent Done Done false
7579 2710 Acked Write 6 Sent Done Done false
7580 2711 Acked Write 9 Sent Done Done false
7581 2712 Acked Write 10 Sent Done Done false
7582 2713 Acked Read 1 Sent Done Done false
7583 2714 Acked Flush 0 Sent Done Done false
7584 2715 Acked Read 2 Sent Done Done false
7585 2716 Acked Read 10 Sent Done Done false
7586 2717 Acked Read 1 Sent Done Done false
7587 2718 Acked Write 9 Sent Done Done false
7588 2719 Acked Read 3 Sent Done Done false
7589 2720 Acked Write 1 Sent Done Done false
7590 2721 Acked Write 5 Sent Done Done false
7591 2722 Acked Read 1 Sent Done Done false
7592 2723 Acked Read 1 Sent Done Done false
7593 2724 Acked Read 1 Sent Done Done false
7594 2725 Acked Read 5 Sent Done Done false
7595 2726 Acked Read 4 Sent Done Done false
7596 2727 Acked Read 3 Sent Done Done false
7597 2728 Acked Write 1 Sent Done Done false
7598 2729 Acked Write 9 Sent Done Done false
7599 2730 Acked Write 2 Sent Done Done false
7600 2731 Acked Write 1 Sent Done Done false
7601 2732 Acked Write 4 Sent Done Done false
7602 2733 Acked Write 6 Sent Done Done false
7603 2734 Acked Read 4 Sent Done Done false
7604 2735 Acked Read 7 Sent Done Done false
7605 2736 Acked Write 9 Sent Done Done false
7606 2737 Acked Read 4 Sent Done Done false
7607 2738 Acked Read 5 Sent Done Done false
7608 2739 Acked Flush 0 Sent Done Done false
7609 2740 Acked Read 2 Sent Done Done false
7610 2741 Acked Read 5 Sent Done Done false
7611 2742 Acked Write 7 Sent Done Done false
7612 2743 Acked Write 4 Sent Done Done false
7613 2744 Acked Read 8 Sent Done Done false
7614 2745 Acked Flush 0 Sent Done Done false
7615 2746 Acked Read 1 Sent Done Done false
7616 2747 Acked Read 7 Sent Done Done false
7617 2748 Acked Write 5 Sent Done Done false
7618 2749 Acked Read 1 Sent Done Done false
7619 2750 Acked Read 3 Sent Done Done false
7620 2751 Acked Read 3 Sent Done Done false
7621 2752 Acked Read 9 Sent Done Done false
7622 2753 Acked Write 1 Sent Done Done false
7623 2754 Acked Read 5 Sent Done Done false
7624 2755 Acked Read 6 Sent Done Done false
7625 2756 Acked Read 10 Sent Done Done false
7626 2757 Acked Write 10 Sent Done Done false
7627 2758 Acked Write 1 Sent Done Done false
7628 2759 Acked Write 1 Sent Done Done false
7629 2760 Acked Read 1 Sent Done Done false
7630 2761 Acked Read 2 Sent Done Done false
7631 2762 Acked Write 4 Sent Done Done false
7632 2763 Acked Flush 0 Sent Done Done false
7633 2764 Acked Flush 0 Sent Done Done false
7634 2765 Acked Write 1 Sent Done Done false
7635 2766 Acked Flush 0 Sent Done Done false
7636 2767 Acked Flush 0 Sent Done Done false
7637 2768 Acked Write 3 Sent Done Done false
7638 2769 Acked Write 5 Sent Done Done false
7639 2770 Acked Read 6 Sent Done Done false
7640 2771 Acked Read 7 Sent Done Done false
7641 2772 Acked Write 1 Sent Done Done false
7642 2773 Acked Write 10 Sent Done Done false
7643 2774 Acked Read 9 Sent Done Done false
7644 2775 Acked Read 2 Sent Done Done false
7645 2776 Acked Write 4 Sent Done Done false
7646 2777 Acked Read 6 Sent Done Done false
7647 2778 Acked Read 10 Sent Done Done false
7648 2779 Acked Write 3 Sent Done Done false
7649 2780 Acked Read 7 Sent Done Done false
7650 2781 Acked Read 3 Sent Done Done false
7651 2782 Acked Write 4 Sent Done Done false
7652 2783 Acked Write 6 Sent Done Done false
7653 2784 Acked Write 2 Sent Done Done false
7654 2785 Acked Read 2 Sent Done Done false
7655 2786 Acked Read 6 Sent Done Done false
7656 2787 Acked Read 5 Sent Done Done false
7657 2788 Acked Flush 0 Sent Done Done false
7658 2789 Acked Read 2 Sent Done Done false
7659 2790 Acked Read 3 Sent Done Done false
7660 2791 Acked Read 3 Sent Done Done false
7661 2792 Acked Write 6 Sent Done Done false
7662 2793 Acked Read 3 Sent Done Done false
7663 2794 Acked Write 9 Sent Done Done false
7664 2795 Acked Read 10 Sent Done Done false
7665 2796 Acked Flush 0 Sent Done Done false
7666 2797 Acked Read 4 Sent Done Done false
7667 2798 Acked Write 10 Sent Done Done false
7668 2799 Acked Read 5 Sent Done Done false
7669 2800 Acked Read 9 Sent Done Done false
7670 2801 Acked Read 3 Sent Done Done false
7671 2802 Acked Read 4 Sent Done Done false
7672 2803 Acked Write 1 Sent Done Done false
7673 2804 Acked Write 8 Sent Done Done false
7674 2805 Acked Write 4 Sent Done Done false
7675 2806 Acked Flush 0 Sent Done Done false
7676 2807 Acked Flush 0 Sent Done Done false
7677 2808 Acked Write 1 Sent Done Done false
7678 2809 Acked Flush 0 Sent Done Done false
7679 STATES DS:0 DS:1 DS:2 TOTAL
7680 Sent 301 0 0 301
7681 Done 0 301 301 602
7682 Skipped 0 0 0 0
7683 Error 0 0 0 0
7684 Last Flush: Some(JobId(2508)) Some(JobId(2809)) Some(JobId(2809))
7685 Downstairs last five completed: 2508 2507 2506 2505 2504
7686 Upstairs last five completed: 2809 2808 2807 2806 2805
7687 CLIENT: Up:0 ds:301 act:2
76882025-03-07T18:14:14.526ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
76892025-03-07T18:14:15.108ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
76902025-03-07T18:14:16.111ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
76912025-03-07T18:14:17.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
7692 ----------------------------------------------------------------
7693 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7694 States: Offline Active Active
7695 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7696 2509 Acked Write 2 Sent Done Done false
7697 2510 Acked Write 4 Sent Done Done false
7698 2511 Acked Flush 0 Sent Done Done false
7699 2512 Acked Write 1 Sent Done Done false
7700 2513 Acked Read 10 Sent Done Done false
7701 2514 Acked Write 5 Sent Done Done false
7702 2515 Acked Read 2 Sent Done Done false
7703 2516 Acked Write 10 Sent Done Done false
7704 2517 Acked Read 10 Sent Done Done false
7705 2518 Acked Flush 0 Sent Done Done false
7706 2519 Acked Read 1 Sent Done Done false
7707 2520 Acked Write 7 Sent Done Done false
7708 2521 Acked Write 8 Sent Done Done false
7709 2522 Acked Read 2 Sent Done Done false
7710 2523 Acked Write 2 Sent Done Done false
7711 2524 Acked Write 9 Sent Done Done false
7712 2525 Acked Write 4 Sent Done Done false
7713 2526 Acked Write 8 Sent Done Done false
7714 2527 Acked Read 1 Sent Done Done false
7715 2528 Acked Read 1 Sent Done Done false
7716 2529 Acked Write 9 Sent Done Done false
7717 2530 Acked Write 4 Sent Done Done false
7718 2531 Acked Read 6 Sent Done Done false
7719 2532 Acked Flush 0 Sent Done Done false
7720 2533 Acked Read 10 Sent Done Done false
7721 2534 Acked Flush 0 Sent Done Done false
7722 2535 Acked Write 4 Sent Done Done false
7723 2536 Acked Write 9 Sent Done Done false
7724 2537 Acked Flush 0 Sent Done Done false
7725 2538 Acked Flush 0 Sent Done Done false
7726 2539 Acked Read 3 Sent Done Done false
7727 2540 Acked Write 1 Sent Done Done false
7728 2541 Acked Write 1 Sent Done Done false
7729 2542 Acked Read 3 Sent Done Done false
7730 2543 Acked Write 1 Sent Done Done false
7731 2544 Acked Write 1 Sent Done Done false
7732 2545 Acked Write 5 Sent Done Done false
7733 2546 Acked Write 9 Sent Done Done false
7734 2547 Acked Read 9 Sent Done Done false
7735 2548 Acked Flush 0 Sent Done Done false
7736 2549 Acked Flush 0 Sent Done Done false
7737 2550 Acked Read 3 Sent Done Done false
7738 2551 Acked Write 1 Sent Done Done false
7739 2552 Acked Read 7 Sent Done Done false
7740 2553 Acked Read 3 Sent Done Done false
7741 2554 Acked Read 6 Sent Done Done false
7742 2555 Acked Read 9 Sent Done Done false
7743 2556 Acked Read 9 Sent Done Done false
7744 2557 Acked Write 9 Sent Done Done false
7745 2558 Acked Write 8 Sent Done Done false
7746 2559 Acked Read 8 Sent Done Done false
7747 2560 Acked Write 9 Sent Done Done false
7748 2561 Acked Read 9 Sent Done Done false
7749 2562 Acked Write 9 Sent Done Done false
7750 2563 Acked Read 3 Sent Done Done false
7751 2564 Acked Flush 0 Sent Done Done false
7752 2565 Acked Read 3 Sent Done Done false
7753 2566 Acked Write 10 Sent Done Done false
7754 2567 Acked Write 6 Sent Done Done false
7755 2568 Acked Write 5 Sent Done Done false
7756 2569 Acked Read 9 Sent Done Done false
7757 2570 Acked Read 2 Sent Done Done false
7758 2571 Acked Read 9 Sent Done Done false
7759 2572 Acked Write 4 Sent Done Done false
7760 2573 Acked Read 6 Sent Done Done false
7761 2574 Acked Read 8 Sent Done Done false
7762 2575 Acked Flush 0 Sent Done Done false
7763 2576 Acked Read 6 Sent Done Done false
7764 2577 Acked Write 2 Sent Done Done false
7765 2578 Acked Write 10 Sent Done Done false
7766 2579 Acked Read 7 Sent Done Done false
7767 2580 Acked Flush 0 Sent Done Done false
7768 2581 Acked Read 8 Sent Done Done false
7769 2582 Acked Flush 0 Sent Done Done false
7770 2583 Acked Flush 0 Sent Done Done false
7771 2584 Acked Read 9 Sent Done Done false
7772 2585 Acked Read 1 Sent Done Done false
7773 2586 Acked Write 4 Sent Done Done false
7774 2587 Acked Write 1 Sent Done Done false
7775 2588 Acked Read 2 Sent Done Done false
7776 2589 Acked Read 8 Sent Done Done false
7777 2590 Acked Write 5 Sent Done Done false
7778 2591 Acked Read 10 Sent Done Done false
7779 2592 Acked Write 7 Sent Done Done false
7780 2593 Acked Write 3 Sent Done Done false
7781 2594 Acked Write 5 Sent Done Done false
7782 2595 Acked Read 4 Sent Done Done false
7783 2596 Acked Write 3 Sent Done Done false
7784 2597 Acked Write 4 Sent Done Done false
7785 2598 Acked Read 8 Sent Done Done false
7786 2599 Acked Write 9 Sent Done Done false
7787 2600 Acked Write 4 Sent Done Done false
7788 2601 Acked Flush 0 Sent Done Done false
7789 2602 Acked Write 10 Sent Done Done false
7790 2603 Acked Read 9 Sent Done Done false
7791 2604 Acked Write 10 Sent Done Done false
7792 2605 Acked Write 3 Sent Done Done false
7793 2606 Acked Write 9 Sent Done Done false
7794 2607 Acked Write 1 Sent Done Done false
7795 2608 Acked Write 1 Sent Done Done false
7796 2609 Acked Write 6 Sent Done Done false
7797 2610 Acked Read 6 Sent Done Done false
7798 2611 Acked Read 7 Sent Done Done false
7799 2612 Acked Read 3 Sent Done Done false
7800 2613 Acked Write 3 Sent Done Done false
7801 2614 Acked Write 1 Sent Done Done false
7802 2615 Acked Read 10 Sent Done Done false
7803 2616 Acked Read 3 Sent Done Done false
7804 2617 Acked Write 2 Sent Done Done false
7805 2618 Acked Read 2 Sent Done Done false
7806 2619 Acked Read 8 Sent Done Done false
7807 2620 Acked Read 3 Sent Done Done false
7808 2621 Acked Write 8 Sent Done Done false
7809 2622 Acked Flush 0 Sent Done Done false
7810 2623 Acked Read 6 Sent Done Done false
7811 2624 Acked Write 4 Sent Done Done false
7812 2625 Acked Read 9 Sent Done Done false
7813 2626 Acked Write 6 Sent Done Done false
7814 2627 Acked Read 8 Sent Done Done false
7815 2628 Acked Write 3 Sent Done Done false
7816 2629 Acked Read 9 Sent Done Done false
7817 2630 Acked Write 8 Sent Done Done false
7818 2631 Acked Write 6 Sent Done Done false
7819 2632 Acked Read 9 Sent Done Done false
7820 2633 Acked Read 8 Sent Done Done false
7821 2634 Acked Read 9 Sent Done Done false
7822 2635 Acked Write 2 Sent Done Done false
7823 2636 Acked Read 9 Sent Done Done false
7824 2637 Acked Flush 0 Sent Done Done false
7825 2638 Acked Flush 0 Sent Done Done false
7826 2639 Acked Read 8 Sent Done Done false
7827 2640 Acked Write 2 Sent Done Done false
7828 2641 Acked Read 4 Sent Done Done false
7829 2642 Acked Read 5 Sent Done Done false
7830 2643 Acked Write 6 Sent Done Done false
7831 2644 Acked Flush 0 Sent Done Done false
7832 2645 Acked Write 7 Sent Done Done false
7833 2646 Acked Read 5 Sent Done Done false
7834 2647 Acked Write 6 Sent Done Done false
7835 2648 Acked Flush 0 Sent Done Done false
7836 2649 Acked Write 4 Sent Done Done false
7837 2650 Acked Write 10 Sent Done Done false
7838 2651 Acked Read 5 Sent Done Done false
7839 2652 Acked Flush 0 Sent Done Done false
7840 2653 Acked Read 1 Sent Done Done false
7841 2654 Acked Flush 0 Sent Done Done false
7842 2655 Acked Write 1 Sent Done Done false
7843 2656 Acked Write 1 Sent Done Done false
7844 2657 Acked Read 6 Sent Done Done false
7845 2658 Acked Write 9 Sent Done Done false
7846 2659 Acked Write 6 Sent Done Done false
7847 2660 Acked Write 6 Sent Done Done false
7848 2661 Acked Flush 0 Sent Done Done false
7849 2662 Acked Read 3 Sent Done Done false
7850 2663 Acked Flush 0 Sent Done Done false
7851 2664 Acked Read 8 Sent Done Done false
7852 2665 Acked Read 2 Sent Done Done false
7853 2666 Acked Read 4 Sent Done Done false
7854 2667 Acked Read 9 Sent Done Done false
7855 2668 Acked Write 10 Sent Done Done false
7856 2669 Acked Read 3 Sent Done Done false
7857 2670 Acked Read 4 Sent Done Done false
7858 2671 Acked Write 5 Sent Done Done false
7859 2672 Acked Write 7 Sent Done Done false
7860 2673 Acked Write 5 Sent Done Done false
7861 2674 Acked Read 3 Sent Done Done false
7862 2675 Acked Read 5 Sent Done Done false
7863 2676 Acked Read 4 Sent Done Done false
7864 2677 Acked Read 9 Sent Done Done false
7865 2678 Acked Write 7 Sent Done Done false
7866 2679 Acked Read 9 Sent Done Done false
7867 2680 Acked Read 8 Sent Done Done false
7868 2681 Acked Read 5 Sent Done Done false
7869 2682 Acked Read 10 Sent Done Done false
7870 2683 Acked Write 5 Sent Done Done false
7871 2684 Acked Read 8 Sent Done Done false
7872 2685 Acked Write 7 Sent Done Done false
7873 2686 Acked Read 2 Sent Done Done false
7874 2687 Acked Read 6 Sent Done Done false
7875 2688 Acked Flush 0 Sent Done Done false
7876 2689 Acked Read 9 Sent Done Done false
7877 2690 Acked Read 6 Sent Done Done false
7878 2691 Acked Read 10 Sent Done Done false
7879 2692 Acked Flush 0 Sent Done Done false
7880 2693 Acked Read 3 Sent Done Done false
7881 2694 Acked Write 3 Sent Done Done false
7882 2695 Acked Read 7 Sent Done Done false
7883 2696 Acked Write 2 Sent Done Done false
7884 2697 Acked Flush 0 Sent Done Done false
7885 2698 Acked Read 7 Sent Done Done false
7886 2699 Acked Read 2 Sent Done Done false
7887 2700 Acked Write 1 Sent Done Done false
7888 2701 Acked Read 6 Sent Done Done false
7889 2702 Acked Read 4 Sent Done Done false
7890 2703 Acked Write 4 Sent Done Done false
7891 2704 Acked Write 6 Sent Done Done false
7892 2705 Acked Write 10 Sent Done Done false
7893 2706 Acked Write 3 Sent Done Done false
7894 2707 Acked Read 2 Sent Done Done false
7895 2708 Acked Write 3 Sent Done Done false
7896 2709 Acked Flush 0 Sent Done Done false
7897 2710 Acked Write 6 Sent Done Done false
7898 2711 Acked Write 9 Sent Done Done false
7899 2712 Acked Write 10 Sent Done Done false
7900 2713 Acked Read 1 Sent Done Done false
7901 2714 Acked Flush 0 Sent Done Done false
7902 2715 Acked Read 2 Sent Done Done false
7903 2716 Acked Read 10 Sent Done Done false
7904 2717 Acked Read 1 Sent Done Done false
7905 2718 Acked Write 9 Sent Done Done false
7906 2719 Acked Read 3 Sent Done Done false
7907 2720 Acked Write 1 Sent Done Done false
7908 2721 Acked Write 5 Sent Done Done false
7909 2722 Acked Read 1 Sent Done Done false
7910 2723 Acked Read 1 Sent Done Done false
7911 2724 Acked Read 1 Sent Done Done false
7912 2725 Acked Read 5 Sent Done Done false
7913 2726 Acked Read 4 Sent Done Done false
7914 2727 Acked Read 3 Sent Done Done false
7915 2728 Acked Write 1 Sent Done Done false
7916 2729 Acked Write 9 Sent Done Done false
7917 2730 Acked Write 2 Sent Done Done false
7918 2731 Acked Write 1 Sent Done Done false
7919 2732 Acked Write 4 Sent Done Done false
7920 2733 Acked Write 6 Sent Done Done false
7921 2734 Acked Read 4 Sent Done Done false
7922 2735 Acked Read 7 Sent Done Done false
7923 2736 Acked Write 9 Sent Done Done false
7924 2737 Acked Read 4 Sent Done Done false
7925 2738 Acked Read 5 Sent Done Done false
7926 2739 Acked Flush 0 Sent Done Done false
7927 2740 Acked Read 2 Sent Done Done false
7928 2741 Acked Read 5 Sent Done Done false
7929 2742 Acked Write 7 Sent Done Done false
7930 2743 Acked Write 4 Sent Done Done false
7931 2744 Acked Read 8 Sent Done Done false
7932 2745 Acked Flush 0 Sent Done Done false
7933 2746 Acked Read 1 Sent Done Done false
7934 2747 Acked Read 7 Sent Done Done false
7935 2748 Acked Write 5 Sent Done Done false
7936 2749 Acked Read 1 Sent Done Done false
7937 2750 Acked Read 3 Sent Done Done false
7938 2751 Acked Read 3 Sent Done Done false
7939 2752 Acked Read 9 Sent Done Done false
7940 2753 Acked Write 1 Sent Done Done false
7941 2754 Acked Read 5 Sent Done Done false
7942 2755 Acked Read 6 Sent Done Done false
7943 2756 Acked Read 10 Sent Done Done false
7944 2757 Acked Write 10 Sent Done Done false
7945 2758 Acked Write 1 Sent Done Done false
7946 2759 Acked Write 1 Sent Done Done false
7947 2760 Acked Read 1 Sent Done Done false
7948 2761 Acked Read 2 Sent Done Done false
7949 2762 Acked Write 4 Sent Done Done false
7950 2763 Acked Flush 0 Sent Done Done false
7951 2764 Acked Flush 0 Sent Done Done false
7952 2765 Acked Write 1 Sent Done Done false
7953 2766 Acked Flush 0 Sent Done Done false
7954 2767 Acked Flush 0 Sent Done Done false
7955 2768 Acked Write 3 Sent Done Done false
7956 2769 Acked Write 5 Sent Done Done false
7957 2770 Acked Read 6 Sent Done Done false
7958 2771 Acked Read 7 Sent Done Done false
7959 2772 Acked Write 1 Sent Done Done false
7960 2773 Acked Write 10 Sent Done Done false
7961 2774 Acked Read 9 Sent Done Done false
7962 2775 Acked Read 2 Sent Done Done false
7963 2776 Acked Write 4 Sent Done Done false
7964 2777 Acked Read 6 Sent Done Done false
7965 2778 Acked Read 10 Sent Done Done false
7966 2779 Acked Write 3 Sent Done Done false
7967 2780 Acked Read 7 Sent Done Done false
7968 2781 Acked Read 3 Sent Done Done false
7969 2782 Acked Write 4 Sent Done Done false
7970 2783 Acked Write 6 Sent Done Done false
7971 2784 Acked Write 2 Sent Done Done false
7972 2785 Acked Read 2 Sent Done Done false
7973 2786 Acked Read 6 Sent Done Done false
7974 2787 Acked Read 5 Sent Done Done false
7975 2788 Acked Flush 0 Sent Done Done false
7976 2789 Acked Read 2 Sent Done Done false
7977 2790 Acked Read 3 Sent Done Done false
7978 2791 Acked Read 3 Sent Done Done false
7979 2792 Acked Write 6 Sent Done Done false
7980 2793 Acked Read 3 Sent Done Done false
7981 2794 Acked Write 9 Sent Done Done false
7982 2795 Acked Read 10 Sent Done Done false
7983 2796 Acked Flush 0 Sent Done Done false
7984 2797 Acked Read 4 Sent Done Done false
7985 2798 Acked Write 10 Sent Done Done false
7986 2799 Acked Read 5 Sent Done Done false
7987 2800 Acked Read 9 Sent Done Done false
7988 2801 Acked Read 3 Sent Done Done false
7989 2802 Acked Read 4 Sent Done Done false
7990 2803 Acked Write 1 Sent Done Done false
7991 2804 Acked Write 8 Sent Done Done false
7992 2805 Acked Write 4 Sent Done Done false
7993 2806 Acked Flush 0 Sent Done Done false
7994 2807 Acked Flush 0 Sent Done Done false
7995 2808 Acked Write 1 Sent Done Done false
7996 2809 Acked Flush 0 Sent Done Done false
7997 STATES DS:0 DS:1 DS:2 TOTAL
7998 Sent 301 0 0 301
7999 Done 0 301 301 602
8000 Skipped 0 0 0 0
8001 Error 0 0 0 0
8002 Last Flush: Some(JobId(2508)) Some(JobId(2809)) Some(JobId(2809))
8003 Downstairs last five completed: 2508 2507 2506 2505 2504
8004 Upstairs last five completed: 2809 2808 2807 2806 2805
8005 CLIENT: Up:0 ds:301 act:2
80062025-03-07T18:14:18.527ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80072025-03-07T18:14:19.116ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80082025-03-07T18:14:19.789ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80092025-03-07T18:14:19.789ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80102025-03-07T18:14:19.789ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80112025-03-07T18:14:19.789ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80122025-03-07T18:14:19.789ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8013 [8810] got start action
8014 [8810] I am going to restart
8015 Starting downstairs at port 8810
8016 Make output file at "/tmp/dsc/downstairs-8810.txt"
8017 [8810][0] reports Starting
8018 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1214)
8019 [8810][0] reports Running
80202025-03-07T18:14:20.291ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
80212025-03-07T18:14:21.120ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8022 ----------------------------------------------------------------
8023 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8024 States: Offline Active Active
8025 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8026 2509 Acked Write 2 Sent Done Done false
8027 2510 Acked Write 4 Sent Done Done false
8028 2511 Acked Flush 0 Sent Done Done false
8029 2512 Acked Write 1 Sent Done Done false
8030 2513 Acked Read 10 Sent Done Done false
8031 2514 Acked Write 5 Sent Done Done false
8032 2515 Acked Read 2 Sent Done Done false
8033 2516 Acked Write 10 Sent Done Done false
8034 2517 Acked Read 10 Sent Done Done false
8035 2518 Acked Flush 0 Sent Done Done false
8036 2519 Acked Read 1 Sent Done Done false
8037 2520 Acked Write 7 Sent Done Done false
8038 2521 Acked Write 8 Sent Done Done false
8039 2522 Acked Read 2 Sent Done Done false
8040 2523 Acked Write 2 Sent Done Done false
8041 2524 Acked Write 9 Sent Done Done false
8042 2525 Acked Write 4 Sent Done Done false
8043 2526 Acked Write 8 Sent Done Done false
8044 2527 Acked Read 1 Sent Done Done false
8045 2528 Acked Read 1 Sent Done Done false
8046 2529 Acked Write 9 Sent Done Done false
8047 2530 Acked Write 4 Sent Done Done false
8048 2531 Acked Read 6 Sent Done Done false
8049 2532 Acked Flush 0 Sent Done Done false
8050 2533 Acked Read 10 Sent Done Done false
8051 2534 Acked Flush 0 Sent Done Done false
8052 2535 Acked Write 4 Sent Done Done false
8053 2536 Acked Write 9 Sent Done Done false
8054 2537 Acked Flush 0 Sent Done Done false
8055 2538 Acked Flush 0 Sent Done Done false
8056 2539 Acked Read 3 Sent Done Done false
8057 2540 Acked Write 1 Sent Done Done false
8058 2541 Acked Write 1 Sent Done Done false
8059 2542 Acked Read 3 Sent Done Done false
8060 2543 Acked Write 1 Sent Done Done false
8061 2544 Acked Write 1 Sent Done Done false
8062 2545 Acked Write 5 Sent Done Done false
8063 2546 Acked Write 9 Sent Done Done false
8064 2547 Acked Read 9 Sent Done Done false
8065 2548 Acked Flush 0 Sent Done Done false
8066 2549 Acked Flush 0 Sent Done Done false
8067 2550 Acked Read 3 Sent Done Done false
8068 2551 Acked Write 1 Sent Done Done false
8069 2552 Acked Read 7 Sent Done Done false
8070 2553 Acked Read 3 Sent Done Done false
8071 2554 Acked Read 6 Sent Done Done false
8072 2555 Acked Read 9 Sent Done Done false
8073 2556 Acked Read 9 Sent Done Done false
8074 2557 Acked Write 9 Sent Done Done false
8075 2558 Acked Write 8 Sent Done Done false
8076 2559 Acked Read 8 Sent Done Done false
8077 2560 Acked Write 9 Sent Done Done false
8078 2561 Acked Read 9 Sent Done Done false
8079 2562 Acked Write 9 Sent Done Done false
8080 2563 Acked Read 3 Sent Done Done false
8081 2564 Acked Flush 0 Sent Done Done false
8082 2565 Acked Read 3 Sent Done Done false
8083 2566 Acked Write 10 Sent Done Done false
8084 2567 Acked Write 6 Sent Done Done false
8085 2568 Acked Write 5 Sent Done Done false
8086 2569 Acked Read 9 Sent Done Done false
8087 2570 Acked Read 2 Sent Done Done false
8088 2571 Acked Read 9 Sent Done Done false
8089 2572 Acked Write 4 Sent Done Done false
8090 2573 Acked Read 6 Sent Done Done false
8091 2574 Acked Read 8 Sent Done Done false
8092 2575 Acked Flush 0 Sent Done Done false
8093 2576 Acked Read 6 Sent Done Done false
8094 2577 Acked Write 2 Sent Done Done false
8095 2578 Acked Write 10 Sent Done Done false
8096 2579 Acked Read 7 Sent Done Done false
8097 2580 Acked Flush 0 Sent Done Done false
8098 2581 Acked Read 8 Sent Done Done false
8099 2582 Acked Flush 0 Sent Done Done false
8100 2583 Acked Flush 0 Sent Done Done false
8101 2584 Acked Read 9 Sent Done Done false
8102 2585 Acked Read 1 Sent Done Done false
8103 2586 Acked Write 4 Sent Done Done false
8104 2587 Acked Write 1 Sent Done Done false
8105 2588 Acked Read 2 Sent Done Done false
8106 2589 Acked Read 8 Sent Done Done false
8107 2590 Acked Write 5 Sent Done Done false
8108 2591 Acked Read 10 Sent Done Done false
8109 2592 Acked Write 7 Sent Done Done false
8110 2593 Acked Write 3 Sent Done Done false
8111 2594 Acked Write 5 Sent Done Done false
8112 2595 Acked Read 4 Sent Done Done false
8113 2596 Acked Write 3 Sent Done Done false
8114 2597 Acked Write 4 Sent Done Done false
8115 2598 Acked Read 8 Sent Done Done false
8116 2599 Acked Write 9 Sent Done Done false
8117 2600 Acked Write 4 Sent Done Done false
8118 2601 Acked Flush 0 Sent Done Done false
8119 2602 Acked Write 10 Sent Done Done false
8120 2603 Acked Read 9 Sent Done Done false
8121 2604 Acked Write 10 Sent Done Done false
8122 2605 Acked Write 3 Sent Done Done false
8123 2606 Acked Write 9 Sent Done Done false
8124 2607 Acked Write 1 Sent Done Done false
8125 2608 Acked Write 1 Sent Done Done false
8126 2609 Acked Write 6 Sent Done Done false
8127 2610 Acked Read 6 Sent Done Done false
8128 2611 Acked Read 7 Sent Done Done false
8129 2612 Acked Read 3 Sent Done Done false
8130 2613 Acked Write 3 Sent Done Done false
8131 2614 Acked Write 1 Sent Done Done false
8132 2615 Acked Read 10 Sent Done Done false
8133 2616 Acked Read 3 Sent Done Done false
8134 2617 Acked Write 2 Sent Done Done false
8135 2618 Acked Read 2 Sent Done Done false
8136 2619 Acked Read 8 Sent Done Done false
8137 2620 Acked Read 3 Sent Done Done false
8138 2621 Acked Write 8 Sent Done Done false
8139 2622 Acked Flush 0 Sent Done Done false
8140 2623 Acked Read 6 Sent Done Done false
8141 2624 Acked Write 4 Sent Done Done false
8142 2625 Acked Read 9 Sent Done Done false
8143 2626 Acked Write 6 Sent Done Done false
8144 2627 Acked Read 8 Sent Done Done false
8145 2628 Acked Write 3 Sent Done Done false
8146 2629 Acked Read 9 Sent Done Done false
8147 2630 Acked Write 8 Sent Done Done false
8148 2631 Acked Write 6 Sent Done Done false
8149 2632 Acked Read 9 Sent Done Done false
8150 2633 Acked Read 8 Sent Done Done false
8151 2634 Acked Read 9 Sent Done Done false
8152 2635 Acked Write 2 Sent Done Done false
8153 2636 Acked Read 9 Sent Done Done false
8154 2637 Acked Flush 0 Sent Done Done false
8155 2638 Acked Flush 0 Sent Done Done false
8156 2639 Acked Read 8 Sent Done Done false
8157 2640 Acked Write 2 Sent Done Done false
8158 2641 Acked Read 4 Sent Done Done false
8159 2642 Acked Read 5 Sent Done Done false
8160 2643 Acked Write 6 Sent Done Done false
8161 2644 Acked Flush 0 Sent Done Done false
8162 2645 Acked Write 7 Sent Done Done false
8163 2646 Acked Read 5 Sent Done Done false
8164 2647 Acked Write 6 Sent Done Done false
8165 2648 Acked Flush 0 Sent Done Done false
8166 2649 Acked Write 4 Sent Done Done false
8167 2650 Acked Write 10 Sent Done Done false
8168 2651 Acked Read 5 Sent Done Done false
8169 2652 Acked Flush 0 Sent Done Done false
8170 2653 Acked Read 1 Sent Done Done false
8171 2654 Acked Flush 0 Sent Done Done false
8172 2655 Acked Write 1 Sent Done Done false
8173 2656 Acked Write 1 Sent Done Done false
8174 2657 Acked Read 6 Sent Done Done false
8175 2658 Acked Write 9 Sent Done Done false
8176 2659 Acked Write 6 Sent Done Done false
8177 2660 Acked Write 6 Sent Done Done false
8178 2661 Acked Flush 0 Sent Done Done false
8179 2662 Acked Read 3 Sent Done Done false
8180 2663 Acked Flush 0 Sent Done Done false
8181 2664 Acked Read 8 Sent Done Done false
8182 2665 Acked Read 2 Sent Done Done false
8183 2666 Acked Read 4 Sent Done Done false
8184 2667 Acked Read 9 Sent Done Done false
8185 2668 Acked Write 10 Sent Done Done false
8186 2669 Acked Read 3 Sent Done Done false
8187 2670 Acked Read 4 Sent Done Done false
8188 2671 Acked Write 5 Sent Done Done false
8189 2672 Acked Write 7 Sent Done Done false
8190 2673 Acked Write 5 Sent Done Done false
8191 2674 Acked Read 3 Sent Done Done false
8192 2675 Acked Read 5 Sent Done Done false
8193 2676 Acked Read 4 Sent Done Done false
8194 2677 Acked Read 9 Sent Done Done false
8195 2678 Acked Write 7 Sent Done Done false
8196 2679 Acked Read 9 Sent Done Done false
8197 2680 Acked Read 8 Sent Done Done false
8198 2681 Acked Read 5 Sent Done Done false
8199 2682 Acked Read 10 Sent Done Done false
8200 2683 Acked Write 5 Sent Done Done false
8201 2684 Acked Read 8 Sent Done Done false
8202 2685 Acked Write 7 Sent Done Done false
8203 2686 Acked Read 2 Sent Done Done false
8204 2687 Acked Read 6 Sent Done Done false
8205 2688 Acked Flush 0 Sent Done Done false
8206 2689 Acked Read 9 Sent Done Done false
8207 2690 Acked Read 6 Sent Done Done false
8208 2691 Acked Read 10 Sent Done Done false
8209 2692 Acked Flush 0 Sent Done Done false
8210 2693 Acked Read 3 Sent Done Done false
8211 2694 Acked Write 3 Sent Done Done false
8212 2695 Acked Read 7 Sent Done Done false
8213 2696 Acked Write 2 Sent Done Done false
8214 2697 Acked Flush 0 Sent Done Done false
8215 2698 Acked Read 7 Sent Done Done false
8216 2699 Acked Read 2 Sent Done Done false
8217 2700 Acked Write 1 Sent Done Done false
8218 2701 Acked Read 6 Sent Done Done false
8219 2702 Acked Read 4 Sent Done Done false
8220 2703 Acked Write 4 Sent Done Done false
8221 2704 Acked Write 6 Sent Done Done false
8222 2705 Acked Write 10 Sent Done Done false
8223 2706 Acked Write 3 Sent Done Done false
8224 2707 Acked Read 2 Sent Done Done false
8225 2708 Acked Write 3 Sent Done Done false
8226 2709 Acked Flush 0 Sent Done Done false
8227 2710 Acked Write 6 Sent Done Done false
8228 2711 Acked Write 9 Sent Done Done false
8229 2712 Acked Write 10 Sent Done Done false
8230 2713 Acked Read 1 Sent Done Done false
8231 2714 Acked Flush 0 Sent Done Done false
8232 2715 Acked Read 2 Sent Done Done false
8233 2716 Acked Read 10 Sent Done Done false
8234 2717 Acked Read 1 Sent Done Done false
8235 2718 Acked Write 9 Sent Done Done false
8236 2719 Acked Read 3 Sent Done Done false
8237 2720 Acked Write 1 Sent Done Done false
8238 2721 Acked Write 5 Sent Done Done false
8239 2722 Acked Read 1 Sent Done Done false
8240 2723 Acked Read 1 Sent Done Done false
8241 2724 Acked Read 1 Sent Done Done false
8242 2725 Acked Read 5 Sent Done Done false
8243 2726 Acked Read 4 Sent Done Done false
8244 2727 Acked Read 3 Sent Done Done false
8245 2728 Acked Write 1 Sent Done Done false
8246 2729 Acked Write 9 Sent Done Done false
8247 2730 Acked Write 2 Sent Done Done false
8248 2731 Acked Write 1 Sent Done Done false
8249 2732 Acked Write 4 Sent Done Done false
8250 2733 Acked Write 6 Sent Done Done false
8251 2734 Acked Read 4 Sent Done Done false
8252 2735 Acked Read 7 Sent Done Done false
8253 2736 Acked Write 9 Sent Done Done false
8254 2737 Acked Read 4 Sent Done Done false
8255 2738 Acked Read 5 Sent Done Done false
8256 2739 Acked Flush 0 Sent Done Done false
8257 2740 Acked Read 2 Sent Done Done false
8258 2741 Acked Read 5 Sent Done Done false
8259 2742 Acked Write 7 Sent Done Done false
8260 2743 Acked Write 4 Sent Done Done false
8261 2744 Acked Read 8 Sent Done Done false
8262 2745 Acked Flush 0 Sent Done Done false
8263 2746 Acked Read 1 Sent Done Done false
8264 2747 Acked Read 7 Sent Done Done false
8265 2748 Acked Write 5 Sent Done Done false
8266 2749 Acked Read 1 Sent Done Done false
8267 2750 Acked Read 3 Sent Done Done false
8268 2751 Acked Read 3 Sent Done Done false
8269 2752 Acked Read 9 Sent Done Done false
8270 2753 Acked Write 1 Sent Done Done false
8271 2754 Acked Read 5 Sent Done Done false
8272 2755 Acked Read 6 Sent Done Done false
8273 2756 Acked Read 10 Sent Done Done false
8274 2757 Acked Write 10 Sent Done Done false
8275 2758 Acked Write 1 Sent Done Done false
8276 2759 Acked Write 1 Sent Done Done false
8277 2760 Acked Read 1 Sent Done Done false
8278 2761 Acked Read 2 Sent Done Done false
8279 2762 Acked Write 4 Sent Done Done false
8280 2763 Acked Flush 0 Sent Done Done false
8281 2764 Acked Flush 0 Sent Done Done false
8282 2765 Acked Write 1 Sent Done Done false
8283 2766 Acked Flush 0 Sent Done Done false
8284 2767 Acked Flush 0 Sent Done Done false
8285 2768 Acked Write 3 Sent Done Done false
8286 2769 Acked Write 5 Sent Done Done false
8287 2770 Acked Read 6 Sent Done Done false
8288 2771 Acked Read 7 Sent Done Done false
8289 2772 Acked Write 1 Sent Done Done false
8290 2773 Acked Write 10 Sent Done Done false
8291 2774 Acked Read 9 Sent Done Done false
8292 2775 Acked Read 2 Sent Done Done false
8293 2776 Acked Write 4 Sent Done Done false
8294 2777 Acked Read 6 Sent Done Done false
8295 2778 Acked Read 10 Sent Done Done false
8296 2779 Acked Write 3 Sent Done Done false
8297 2780 Acked Read 7 Sent Done Done false
8298 2781 Acked Read 3 Sent Done Done false
8299 2782 Acked Write 4 Sent Done Done false
8300 2783 Acked Write 6 Sent Done Done false
8301 2784 Acked Write 2 Sent Done Done false
8302 2785 Acked Read 2 Sent Done Done false
8303 2786 Acked Read 6 Sent Done Done false
8304 2787 Acked Read 5 Sent Done Done false
8305 2788 Acked Flush 0 Sent Done Done false
8306 2789 Acked Read 2 Sent Done Done false
8307 2790 Acked Read 3 Sent Done Done false
8308 2791 Acked Read 3 Sent Done Done false
8309 2792 Acked Write 6 Sent Done Done false
8310 2793 Acked Read 3 Sent Done Done false
8311 2794 Acked Write 9 Sent Done Done false
8312 2795 Acked Read 10 Sent Done Done false
8313 2796 Acked Flush 0 Sent Done Done false
8314 2797 Acked Read 4 Sent Done Done false
8315 2798 Acked Write 10 Sent Done Done false
8316 2799 Acked Read 5 Sent Done Done false
8317 2800 Acked Read 9 Sent Done Done false
8318 2801 Acked Read 3 Sent Done Done false
8319 2802 Acked Read 4 Sent Done Done false
8320 2803 Acked Write 1 Sent Done Done false
8321 2804 Acked Write 8 Sent Done Done false
8322 2805 Acked Write 4 Sent Done Done false
8323 2806 Acked Flush 0 Sent Done Done false
8324 2807 Acked Flush 0 Sent Done Done false
8325 2808 Acked Write 1 Sent Done Done false
8326 2809 Acked Flush 0 Sent Done Done false
8327 STATES DS:0 DS:1 DS:2 TOTAL
8328 Sent 301 0 0 301
8329 Done 0 301 301 602
8330 Skipped 0 0 0 0
8331 Error 0 0 0 0
8332 Last Flush: Some(JobId(2508)) Some(JobId(2809)) Some(JobId(2809))
8333 Downstairs last five completed: 2508 2507 2506 2505 2504
8334 Upstairs last five completed: 2809 2808 2807 2806 2805
8335 CLIENT: Up:0 ds:301 act:2
83362025-03-07T18:14:22.530ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
83372025-03-07T18:14:23.123ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
83382025-03-07T18:14:24.125ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
83392025-03-07T18:14:25.126ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8340 ----------------------------------------------------------------
8341 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8342 States: Offline Active Active
8343 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8344 2509 Acked Write 2 Sent Done Done false
8345 2510 Acked Write 4 Sent Done Done false
8346 2511 Acked Flush 0 Sent Done Done false
8347 2512 Acked Write 1 Sent Done Done false
8348 2513 Acked Read 10 Sent Done Done false
8349 2514 Acked Write 5 Sent Done Done false
8350 2515 Acked Read 2 Sent Done Done false
8351 2516 Acked Write 10 Sent Done Done false
8352 2517 Acked Read 10 Sent Done Done false
8353 2518 Acked Flush 0 Sent Done Done false
8354 2519 Acked Read 1 Sent Done Done false
8355 2520 Acked Write 7 Sent Done Done false
8356 2521 Acked Write 8 Sent Done Done false
8357 2522 Acked Read 2 Sent Done Done false
8358 2523 Acked Write 2 Sent Done Done false
8359 2524 Acked Write 9 Sent Done Done false
8360 2525 Acked Write 4 Sent Done Done false
8361 2526 Acked Write 8 Sent Done Done false
8362 2527 Acked Read 1 Sent Done Done false
8363 2528 Acked Read 1 Sent Done Done false
8364 2529 Acked Write 9 Sent Done Done false
8365 2530 Acked Write 4 Sent Done Done false
8366 2531 Acked Read 6 Sent Done Done false
8367 2532 Acked Flush 0 Sent Done Done false
8368 2533 Acked Read 10 Sent Done Done false
8369 2534 Acked Flush 0 Sent Done Done false
8370 2535 Acked Write 4 Sent Done Done false
8371 2536 Acked Write 9 Sent Done Done false
8372 2537 Acked Flush 0 Sent Done Done false
8373 2538 Acked Flush 0 Sent Done Done false
8374 2539 Acked Read 3 Sent Done Done false
8375 2540 Acked Write 1 Sent Done Done false
8376 2541 Acked Write 1 Sent Done Done false
8377 2542 Acked Read 3 Sent Done Done false
8378 2543 Acked Write 1 Sent Done Done false
8379 2544 Acked Write 1 Sent Done Done false
8380 2545 Acked Write 5 Sent Done Done false
8381 2546 Acked Write 9 Sent Done Done false
8382 2547 Acked Read 9 Sent Done Done false
8383 2548 Acked Flush 0 Sent Done Done false
8384 2549 Acked Flush 0 Sent Done Done false
8385 2550 Acked Read 3 Sent Done Done false
8386 2551 Acked Write 1 Sent Done Done false
8387 2552 Acked Read 7 Sent Done Done false
8388 2553 Acked Read 3 Sent Done Done false
8389 2554 Acked Read 6 Sent Done Done false
8390 2555 Acked Read 9 Sent Done Done false
8391 2556 Acked Read 9 Sent Done Done false
8392 2557 Acked Write 9 Sent Done Done false
8393 2558 Acked Write 8 Sent Done Done false
8394 2559 Acked Read 8 Sent Done Done false
8395 2560 Acked Write 9 Sent Done Done false
8396 2561 Acked Read 9 Sent Done Done false
8397 2562 Acked Write 9 Sent Done Done false
8398 2563 Acked Read 3 Sent Done Done false
8399 2564 Acked Flush 0 Sent Done Done false
8400 2565 Acked Read 3 Sent Done Done false
8401 2566 Acked Write 10 Sent Done Done false
8402 2567 Acked Write 6 Sent Done Done false
8403 2568 Acked Write 5 Sent Done Done false
8404 2569 Acked Read 9 Sent Done Done false
8405 2570 Acked Read 2 Sent Done Done false
8406 2571 Acked Read 9 Sent Done Done false
8407 2572 Acked Write 4 Sent Done Done false
8408 2573 Acked Read 6 Sent Done Done false
8409 2574 Acked Read 8 Sent Done Done false
8410 2575 Acked Flush 0 Sent Done Done false
8411 2576 Acked Read 6 Sent Done Done false
8412 2577 Acked Write 2 Sent Done Done false
8413 2578 Acked Write 10 Sent Done Done false
8414 2579 Acked Read 7 Sent Done Done false
8415 2580 Acked Flush 0 Sent Done Done false
8416 2581 Acked Read 8 Sent Done Done false
8417 2582 Acked Flush 0 Sent Done Done false
8418 2583 Acked Flush 0 Sent Done Done false
8419 2584 Acked Read 9 Sent Done Done false
8420 2585 Acked Read 1 Sent Done Done false
8421 2586 Acked Write 4 Sent Done Done false
8422 2587 Acked Write 1 Sent Done Done false
8423 2588 Acked Read 2 Sent Done Done false
8424 2589 Acked Read 8 Sent Done Done false
8425 2590 Acked Write 5 Sent Done Done false
8426 2591 Acked Read 10 Sent Done Done false
8427 2592 Acked Write 7 Sent Done Done false
8428 2593 Acked Write 3 Sent Done Done false
8429 2594 Acked Write 5 Sent Done Done false
8430 2595 Acked Read 4 Sent Done Done false
8431 2596 Acked Write 3 Sent Done Done false
8432 2597 Acked Write 4 Sent Done Done false
8433 2598 Acked Read 8 Sent Done Done false
8434 2599 Acked Write 9 Sent Done Done false
8435 2600 Acked Write 4 Sent Done Done false
8436 2601 Acked Flush 0 Sent Done Done false
8437 2602 Acked Write 10 Sent Done Done false
8438 2603 Acked Read 9 Sent Done Done false
8439 2604 Acked Write 10 Sent Done Done false
8440 2605 Acked Write 3 Sent Done Done false
8441 2606 Acked Write 9 Sent Done Done false
8442 2607 Acked Write 1 Sent Done Done false
8443 2608 Acked Write 1 Sent Done Done false
8444 2609 Acked Write 6 Sent Done Done false
8445 2610 Acked Read 6 Sent Done Done false
8446 2611 Acked Read 7 Sent Done Done false
8447 2612 Acked Read 3 Sent Done Done false
8448 2613 Acked Write 3 Sent Done Done false
8449 2614 Acked Write 1 Sent Done Done false
8450 2615 Acked Read 10 Sent Done Done false
8451 2616 Acked Read 3 Sent Done Done false
8452 2617 Acked Write 2 Sent Done Done false
8453 2618 Acked Read 2 Sent Done Done false
8454 2619 Acked Read 8 Sent Done Done false
8455 2620 Acked Read 3 Sent Done Done false
8456 2621 Acked Write 8 Sent Done Done false
8457 2622 Acked Flush 0 Sent Done Done false
8458 2623 Acked Read 6 Sent Done Done false
8459 2624 Acked Write 4 Sent Done Done false
8460 2625 Acked Read 9 Sent Done Done false
8461 2626 Acked Write 6 Sent Done Done false
8462 2627 Acked Read 8 Sent Done Done false
8463 2628 Acked Write 3 Sent Done Done false
8464 2629 Acked Read 9 Sent Done Done false
8465 2630 Acked Write 8 Sent Done Done false
8466 2631 Acked Write 6 Sent Done Done false
8467 2632 Acked Read 9 Sent Done Done false
8468 2633 Acked Read 8 Sent Done Done false
8469 2634 Acked Read 9 Sent Done Done false
8470 2635 Acked Write 2 Sent Done Done false
8471 2636 Acked Read 9 Sent Done Done false
8472 2637 Acked Flush 0 Sent Done Done false
8473 2638 Acked Flush 0 Sent Done Done false
8474 2639 Acked Read 8 Sent Done Done false
8475 2640 Acked Write 2 Sent Done Done false
8476 2641 Acked Read 4 Sent Done Done false
8477 2642 Acked Read 5 Sent Done Done false
8478 2643 Acked Write 6 Sent Done Done false
8479 2644 Acked Flush 0 Sent Done Done false
8480 2645 Acked Write 7 Sent Done Done false
8481 2646 Acked Read 5 Sent Done Done false
8482 2647 Acked Write 6 Sent Done Done false
8483 2648 Acked Flush 0 Sent Done Done false
8484 2649 Acked Write 4 Sent Done Done false
8485 2650 Acked Write 10 Sent Done Done false
8486 2651 Acked Read 5 Sent Done Done false
8487 2652 Acked Flush 0 Sent Done Done false
8488 2653 Acked Read 1 Sent Done Done false
8489 2654 Acked Flush 0 Sent Done Done false
8490 2655 Acked Write 1 Sent Done Done false
8491 2656 Acked Write 1 Sent Done Done false
8492 2657 Acked Read 6 Sent Done Done false
8493 2658 Acked Write 9 Sent Done Done false
8494 2659 Acked Write 6 Sent Done Done false
8495 2660 Acked Write 6 Sent Done Done false
8496 2661 Acked Flush 0 Sent Done Done false
8497 2662 Acked Read 3 Sent Done Done false
8498 2663 Acked Flush 0 Sent Done Done false
8499 2664 Acked Read 8 Sent Done Done false
8500 2665 Acked Read 2 Sent Done Done false
8501 2666 Acked Read 4 Sent Done Done false
8502 2667 Acked Read 9 Sent Done Done false
8503 2668 Acked Write 10 Sent Done Done false
8504 2669 Acked Read 3 Sent Done Done false
8505 2670 Acked Read 4 Sent Done Done false
8506 2671 Acked Write 5 Sent Done Done false
8507 2672 Acked Write 7 Sent Done Done false
8508 2673 Acked Write 5 Sent Done Done false
8509 2674 Acked Read 3 Sent Done Done false
8510 2675 Acked Read 5 Sent Done Done false
8511 2676 Acked Read 4 Sent Done Done false
8512 2677 Acked Read 9 Sent Done Done false
8513 2678 Acked Write 7 Sent Done Done false
8514 2679 Acked Read 9 Sent Done Done false
8515 2680 Acked Read 8 Sent Done Done false
8516 2681 Acked Read 5 Sent Done Done false
8517 2682 Acked Read 10 Sent Done Done false
8518 2683 Acked Write 5 Sent Done Done false
8519 2684 Acked Read 8 Sent Done Done false
8520 2685 Acked Write 7 Sent Done Done false
8521 2686 Acked Read 2 Sent Done Done false
8522 2687 Acked Read 6 Sent Done Done false
8523 2688 Acked Flush 0 Sent Done Done false
8524 2689 Acked Read 9 Sent Done Done false
8525 2690 Acked Read 6 Sent Done Done false
8526 2691 Acked Read 10 Sent Done Done false
8527 2692 Acked Flush 0 Sent Done Done false
8528 2693 Acked Read 3 Sent Done Done false
8529 2694 Acked Write 3 Sent Done Done false
8530 2695 Acked Read 7 Sent Done Done false
8531 2696 Acked Write 2 Sent Done Done false
8532 2697 Acked Flush 0 Sent Done Done false
8533 2698 Acked Read 7 Sent Done Done false
8534 2699 Acked Read 2 Sent Done Done false
8535 2700 Acked Write 1 Sent Done Done false
8536 2701 Acked Read 6 Sent Done Done false
8537 2702 Acked Read 4 Sent Done Done false
8538 2703 Acked Write 4 Sent Done Done false
8539 2704 Acked Write 6 Sent Done Done false
8540 2705 Acked Write 10 Sent Done Done false
8541 2706 Acked Write 3 Sent Done Done false
8542 2707 Acked Read 2 Sent Done Done false
8543 2708 Acked Write 3 Sent Done Done false
8544 2709 Acked Flush 0 Sent Done Done false
8545 2710 Acked Write 6 Sent Done Done false
8546 2711 Acked Write 9 Sent Done Done false
8547 2712 Acked Write 10 Sent Done Done false
8548 2713 Acked Read 1 Sent Done Done false
8549 2714 Acked Flush 0 Sent Done Done false
8550 2715 Acked Read 2 Sent Done Done false
8551 2716 Acked Read 10 Sent Done Done false
8552 2717 Acked Read 1 Sent Done Done false
8553 2718 Acked Write 9 Sent Done Done false
8554 2719 Acked Read 3 Sent Done Done false
8555 2720 Acked Write 1 Sent Done Done false
8556 2721 Acked Write 5 Sent Done Done false
8557 2722 Acked Read 1 Sent Done Done false
8558 2723 Acked Read 1 Sent Done Done false
8559 2724 Acked Read 1 Sent Done Done false
8560 2725 Acked Read 5 Sent Done Done false
8561 2726 Acked Read 4 Sent Done Done false
8562 2727 Acked Read 3 Sent Done Done false
8563 2728 Acked Write 1 Sent Done Done false
8564 2729 Acked Write 9 Sent Done Done false
8565 2730 Acked Write 2 Sent Done Done false
8566 2731 Acked Write 1 Sent Done Done false
8567 2732 Acked Write 4 Sent Done Done false
8568 2733 Acked Write 6 Sent Done Done false
8569 2734 Acked Read 4 Sent Done Done false
8570 2735 Acked Read 7 Sent Done Done false
8571 2736 Acked Write 9 Sent Done Done false
8572 2737 Acked Read 4 Sent Done Done false
8573 2738 Acked Read 5 Sent Done Done false
8574 2739 Acked Flush 0 Sent Done Done false
8575 2740 Acked Read 2 Sent Done Done false
8576 2741 Acked Read 5 Sent Done Done false
8577 2742 Acked Write 7 Sent Done Done false
8578 2743 Acked Write 4 Sent Done Done false
8579 2744 Acked Read 8 Sent Done Done false
8580 2745 Acked Flush 0 Sent Done Done false
8581 2746 Acked Read 1 Sent Done Done false
8582 2747 Acked Read 7 Sent Done Done false
8583 2748 Acked Write 5 Sent Done Done false
8584 2749 Acked Read 1 Sent Done Done false
8585 2750 Acked Read 3 Sent Done Done false
8586 2751 Acked Read 3 Sent Done Done false
8587 2752 Acked Read 9 Sent Done Done false
8588 2753 Acked Write 1 Sent Done Done false
8589 2754 Acked Read 5 Sent Done Done false
8590 2755 Acked Read 6 Sent Done Done false
8591 2756 Acked Read 10 Sent Done Done false
8592 2757 Acked Write 10 Sent Done Done false
8593 2758 Acked Write 1 Sent Done Done false
8594 2759 Acked Write 1 Sent Done Done false
8595 2760 Acked Read 1 Sent Done Done false
8596 2761 Acked Read 2 Sent Done Done false
8597 2762 Acked Write 4 Sent Done Done false
8598 2763 Acked Flush 0 Sent Done Done false
8599 2764 Acked Flush 0 Sent Done Done false
8600 2765 Acked Write 1 Sent Done Done false
8601 2766 Acked Flush 0 Sent Done Done false
8602 2767 Acked Flush 0 Sent Done Done false
8603 2768 Acked Write 3 Sent Done Done false
8604 2769 Acked Write 5 Sent Done Done false
8605 2770 Acked Read 6 Sent Done Done false
8606 2771 Acked Read 7 Sent Done Done false
8607 2772 Acked Write 1 Sent Done Done false
8608 2773 Acked Write 10 Sent Done Done false
8609 2774 Acked Read 9 Sent Done Done false
8610 2775 Acked Read 2 Sent Done Done false
8611 2776 Acked Write 4 Sent Done Done false
8612 2777 Acked Read 6 Sent Done Done false
8613 2778 Acked Read 10 Sent Done Done false
8614 2779 Acked Write 3 Sent Done Done false
8615 2780 Acked Read 7 Sent Done Done false
8616 2781 Acked Read 3 Sent Done Done false
8617 2782 Acked Write 4 Sent Done Done false
8618 2783 Acked Write 6 Sent Done Done false
8619 2784 Acked Write 2 Sent Done Done false
8620 2785 Acked Read 2 Sent Done Done false
8621 2786 Acked Read 6 Sent Done Done false
8622 2787 Acked Read 5 Sent Done Done false
8623 2788 Acked Flush 0 Sent Done Done false
8624 2789 Acked Read 2 Sent Done Done false
8625 2790 Acked Read 3 Sent Done Done false
8626 2791 Acked Read 3 Sent Done Done false
8627 2792 Acked Write 6 Sent Done Done false
8628 2793 Acked Read 3 Sent Done Done false
8629 2794 Acked Write 9 Sent Done Done false
8630 2795 Acked Read 10 Sent Done Done false
8631 2796 Acked Flush 0 Sent Done Done false
8632 2797 Acked Read 4 Sent Done Done false
8633 2798 Acked Write 10 Sent Done Done false
8634 2799 Acked Read 5 Sent Done Done false
8635 2800 Acked Read 9 Sent Done Done false
8636 2801 Acked Read 3 Sent Done Done false
8637 2802 Acked Read 4 Sent Done Done false
8638 2803 Acked Write 1 Sent Done Done false
8639 2804 Acked Write 8 Sent Done Done false
8640 2805 Acked Write 4 Sent Done Done false
8641 2806 Acked Flush 0 Sent Done Done false
8642 2807 Acked Flush 0 Sent Done Done false
8643 2808 Acked Write 1 Sent Done Done false
8644 2809 Acked Flush 0 Sent Done Done false
8645 STATES DS:0 DS:1 DS:2 TOTAL
8646 Sent 301 0 0 301
8647 Done 0 301 301 602
8648 Skipped 0 0 0 0
8649 Error 0 0 0 0
8650 Last Flush: Some(JobId(2508)) Some(JobId(2809)) Some(JobId(2809))
8651 Downstairs last five completed: 2508 2507 2506 2505 2504
8652 Upstairs last five completed: 2809 2808 2807 2806 2805
8653 CLIENT: Up:0 ds:301 act:2
86542025-03-07T18:14:26.532ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86552025-03-07T18:14:27.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86562025-03-07T18:14:28.131ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86572025-03-07T18:14:29.133ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86582025-03-07T18:14:29.791ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86592025-03-07T18:14:29.791ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:56758) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86602025-03-07T18:14:29.792ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86612025-03-07T18:14:29.792ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86622025-03-07T18:14:29.792ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86632025-03-07T18:14:29.792ZINFOcrucible: send last flush ID to this DS: Some(JobId(2508)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86642025-03-07T18:14:29.792ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86652025-03-07T18:14:29.792ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2508)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86662025-03-07T18:14:29.793ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2508)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8667 ----------------------------------------------------------------
8668 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8669 Downstairs last five completed: 2809 2808 2807 2806 2805
8670 Upstairs last five completed: 2809 2808 2807 2806 2805
8671 CLIENT: Up:0 ds:0 act:3
8672 Replay: All jobs finished
8673 Main task has work to do, go find it
8674 got dsc Stop(0)
8675 stop 0
8676 Mar 07 18:14:30.032[8810] Got stop action so:false kr:false
8677 INFO request completed, latency_us: 316, response_code: 204, uri: /stop/cid/0, method: POST, req_id: cf717f55-ac86-4a7f-b360-d1402a96b276, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
86782025-03-07T18:14:30.033ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86792025-03-07T18:14:30.033ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86802025-03-07T18:14:30.033ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
86812025-03-07T18:14:30.033ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
8682 Mar 07 18:14:30.033 INFO request completed, latency_us: 152, response_code: 200, uri: /state/cid/0, method: GET, req_id: 87628f59-2351-4bcc-a1f5-057642f99a2d, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
8683 [8810] Exited with: ExitStatus(unix_wait_status(9))
8684 [8810][0] reports Exit
8685 Mar 07 18:14:34.035 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/0, method: GET, req_id: c369b4e2-3558-4558-bf3a-5aafa95162b7, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
8686 001/300 Write block 2837 len 20480 data: 2 2 2 3 3
8687 002/300 Read block 3816 len 16384
8688 003/300 Read block 1746 len 16384
8689 004/300 Read block 5 len 4096
8690 005/300 Read block 429 len 20480
8691 006/300 Write block 1103 len 12288 data: 3 3 2
8692 007/300 Read block 724 len 20480
8693 008/300 Read block 1818 len 40960
8694 009/300 Read block 1891 len 12288
8695 010/300 Write block 3470 len 28672 data: 2 2 2 2 2 3 3
8696 011/300 Read block 4866 len 16384
8697 012/300 Read block 3161 len 32768
8698 013/300 Read block 1464 len 16384
8699 014/300 Write block 835 len 8192 data: 3 3
8700 015/300 Read block 1061 len 8192
8701 016/300 Read block 4622 len 28672
8702 017/300 Read block 2615 len 32768
8703 018/300 Read block 423 len 40960
8704 019/300 Write block 239 len 8192 data: 3 3
8705 020/300 Read block 2655 len 28672
8706 021/300 Read block 1488 len 20480
8707 022/300 Flush
8708 023/300 Flush
8709 024/300 Read block 873 len 28672
8710 025/300 Read block 4637 len 4096
8711 026/300 Flush
8712 027/300 Read block 3071 len 20480
8713 028/300 Write block 2041 len 32768 data: 2 3 3 3 3 3 4 4
8714 029/300 Write block 2331 len 32768 data: 2 2 2 2 3 3 3 3
8715 030/300 Read block 3754 len 32768
8716 031/300 Write block 3442 len 16384 data: 2 2 2 2
8717 032/300 Write block 4634 len 24576 data: 2 2 3 3 3 2
8718 033/300 Write block 521 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8719 034/300 Read block 2867 len 16384
8720 035/300 Read block 3672 len 20480
8721 036/300 Read block 156 len 40960
8722 037/300 Write block 1206 len 24576 data: 3 3 3 3 3 3
8723 038/300 Read block 3265 len 12288
8724 039/300 Read block 909 len 32768
8725 040/300 Flush
8726 041/300 Read block 20 len 12288
8727 042/300 Write block 2714 len 4096 data: 2
8728 043/300 Read block 4114 len 20480
8729 044/300 Write block 2118 len 16384 data: 2 2 2 2
8730 045/300 Write block 2186 len 32768 data: 3 3 3 3 2 2 2 2
8731 046/300 Write block 4046 len 24576 data: 4 4 4 4 3 3
8732 047/300 Write block 3775 len 12288 data: 2 2 2
8733 048/300 Write block 922 len 32768 data: 2 2 2 2 2 2 2 2
8734 049/300 Read block 4464 len 8192
8735 050/300 Write block 1068 len 8192 data: 3 3
8736 051/300 Read block 3904 len 36864
8737 052/300 Write block 2660 len 40960 data: 4 4 4 5 4 4 4 3 3 3
8738 053/300 Write block 1677 len 20480 data: 2 2 2 3 3
8739 054/300 Read block 3621 len 40960
8740 055/300 Read block 1507 len 20480
8741 056/300 Read block 590 len 4096
8742 057/300 Read block 732 len 4096
8743 058/300 Read block 960 len 8192
8744 059/300 Write block 4383 len 32768 data: 2 2 2 2 2 2 3 3
8745 060/300 Write block 3916 len 28672 data: 3 3 3 4 4 4 4
8746 061/300 Read block 126 len 4096
8747 062/300 Read block 4524 len 8192
8748 063/300 Read block 4778 len 40960
8749 064/300 Write block 4938 len 4096 data: 3
8750 065/300 Write block 2711 len 12288 data: 2 2 2
8751 066/300 Write block 2056 len 8192 data: 2 2
8752 067/300 Flush
8753 068/300 Read block 769 len 32768
8754 069/300 Write block 1839 len 20480 data: 4 3 3 2 2
8755 070/300 Read block 1735 len 36864
8756 071/300 Write block 802 len 28672 data: 4 3 3 3 3 3 3
8757 072/300 Write block 152 len 4096 data: 2
8758 073/300 Read block 4946 len 32768
8759 074/300 Write block 4169 len 16384 data: 2 2 2 2
8760 075/300 Read block 4361 len 28672
8761 076/300 Write block 13 len 32768 data: 3 3 3 3 2 3 4 4
8762 077/300 Read block 4990 len 24576
8763 078/300 Flush
8764 079/300 Read block 12 len 16384
8765 080/300 Write block 2425 len 16384 data: 2 2 2 2
8766 081/300 Flush
8767 082/300 Flush
8768 083/300 Write block 1866 len 8192 data: 2 2
8769 084/300 Write block 4097 len 32768 data: 2 2 2 2 2 2 3 3
8770 085/300 Write block 4340 len 32768 data: 3 3 4 4 4 3 3 2
8771 086/300 Read block 3535 len 32768
8772 087/300 Write block 867 len 20480 data: 3 3 3 3 3
8773 088/300 Read block 4361 len 36864
8774 089/300 Read block 4707 len 32768
8775 090/300 Read block 1152 len 40960
8776 091/300 Read block 948 len 36864
8777 092/300 Write block 4763 len 32768 data: 3 2 5 5 5 5 4 4
8778 093/300 Read block 1233 len 4096
8779 094/300 Flush
8780 095/300 Read block 2653 len 8192
8781 096/300 Read block 51 len 24576
8782 097/300 Read block 1634 len 24576
8783 098/300 Flush
8784 099/300 Read block 680 len 20480
8785 100/300 Read block 3902 len 20480
8786 101/300 Write block 4502 len 32768 data: 3 3 4 5 5 5 5 4
8787 102/300 Write block 2661 len 32768 data: 5 5 6 5 5 5 4 4
8788 103/300 Read block 620 len 8192
8789 104/300 Read block 533 len 40960
8790 105/300 Read block 828 len 40960
8791 106/300 Read block 2780 len 32768
8792 107/300 Write block 4071 len 16384 data: 3 2 2 2
8793 108/300 Flush
8794 109/300 Write block 4189 len 16384 data: 2 2 2 2
8795 110/300 Read block 1345 len 32768
8796 111/300 Read block 1771 len 8192
8797 112/300 Read block 2226 len 40960
8798 113/300 Read block 1118 len 28672
8799 114/300 Write block 4048 len 36864 data: 5 5 4 4 2 2 3 3 3
8800 115/300 Read block 3780 len 16384
8801 116/300 Write block 4075 len 8192 data: 3 3
8802 117/300 Write block 3521 len 20480 data: 2 2 3 3 3
8803 118/300 Write block 2901 len 24576 data: 2 2 2 2 2 2
8804 119/300 Read block 3650 len 24576
8805 120/300 Read block 853 len 40960
8806 121/300 Read block 4524 len 8192
8807 122/300 Flush
8808 123/300 Write block 2822 len 16384 data: 2 2 2 2
8809 124/300 Read block 3210 len 28672
8810 125/300 Read block 2864 len 28672
8811 126/300 Read block 799 len 20480
8812 127/300 Read block 3803 len 4096
8813 128/300 Write block 661 len 36864 data: 2 2 5 5 5 4 4 4 4
8814 129/300 Write block 3 len 28672 data: 2 2 2 2 3 3 3
8815 130/300 Read block 2363 len 8192
8816 131/300 Write block 4387 len 28672 data: 3 3 4 4 3 2 2
8817 132/300 Write block 2272 len 12288 data: 4 4 3
8818 133/300 Read block 2852 len 12288
8819 134/300 Read block 2123 len 12288
8820 135/300 Write block 1815 len 32768 data: 3 3 3 3 3 2 2 3
8821 136/300 Write block 1434 len 28672 data: 2 2 2 2 2 2 2
8822 137/300 Write block 3291 len 16384 data: 2 3 3 3
8823 138/300 Flush
8824 139/300 Read block 978 len 32768
8825 140/300 Flush
8826 141/300 Read block 1927 len 4096
8827 142/300 Read block 3225 len 36864
8828 143/300 Read block 452 len 16384
8829 144/300 Write block 5 len 8192 data: 3 3
8830 145/300 Write block 4774 len 20480 data: 3 3 3 4 4
8831 146/300 Read block 363 len 16384
8832 147/300 Write block 1422 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8833 148/300 Flush
8834 149/300 Read block 2066 len 28672
8835 150/300 Read block 2732 len 24576
8836 151/300 Read block 894 len 16384
8837 152/300 Write block 1549 len 20480 data: 2 2 2 2 2
8838 153/300 Read block 4101 len 24576
8839 154/300 Flush
8840 155/300 Read block 3389 len 12288
8841 156/300 Flush
8842 157/300 Read block 148 len 20480
8843 158/300 Write block 3454 len 20480 data: 4 4 4 4 4
8844 159/300 Read block 3254 len 40960
8845 160/300 Write block 3326 len 24576 data: 3 3 3 2 2 2
8846 161/300 Read block 2122 len 40960
8847 162/300 Read block 1487 len 24576
8848 163/300 Write block 1031 len 24576 data: 3 3 2 2 2 2
8849 164/300 Flush
8850 165/300 Write block 2504 len 12288 data: 3 3 3
8851 166/300 Write block 3231 len 28672 data: 2 3 4 3 3 3 3
8852 167/300 Read block 3778 len 16384
8853 168/300 Write block 716 len 8192 data: 3 3
8854 169/300 Write block 1920 len 20480 data: 3 3 5 5 4
8855 170/300 Write block 4538 len 40960 data: 2 2 3 3 4 5 5 6 6 5
8856 171/300 Write block 2265 len 8192 data: 4 4
8857 172/300 Read block 3546 len 16384
8858 173/300 Write block 3279 len 12288 data: 3 3 4
8859 174/300 Read block 536 len 36864
8860 175/300 Read block 11 len 32768
8861 176/300 Read block 680 len 36864
8862 177/300 Read block 4406 len 8192
8863 178/300 Read block 4084 len 40960
8864 179/300 Write block 3584 len 16384 data: 2 2 2 2
8865 180/300 Write block 112 len 8192 data: 2 2
8866 181/300 Read block 4939 len 12288
8867 182/300 Write block 592 len 32768 data: 3 3 3 3 3 3 3 2
8868 183/300 Read block 421 len 36864
8869 184/300 Read block 2811 len 32768
8870 185/300 Read block 999 len 8192
8871 186/300 Write block 3132 len 12288 data: 3 3 3
8872 187/300 Read block 4000 len 28672
8873 188/300 Flush
8874 189/300 Flush
8875 190/300 Read block 2152 len 16384
8876 191/300 Flush
8877 192/300 Write block 1405 len 36864 data: 4 4 4 4 4 3 3 3 3
8878 193/300 Read block 3128 len 12288
8879 194/300 Write block 10 len 36864 data: 3 3 3 4 4 4 4 3 4
8880 195/300 Write block 80 len 16384 data: 2 2 2 2
8881 196/300 Write block 857 len 12288 data: 2 2 2
8882 197/300 Write block 3638 len 20480 data: 4 4 4 4 4
8883 198/300 Read block 1020 len 12288
8884 199/300 Flush
8885 200/300 Read block 2578 len 36864
8886 201/300 Read block 835 len 4096
8887 202/300 Write block 3434 len 36864 data: 3 3 3 3 2 2 2 2 3
8888 203/300 Read block 4141 len 20480
8889 204/300 Write block 4617 len 12288 data: 2 2 2
8890 205/300 Read block 1062 len 8192
8891 206/300 Write block 3623 len 4096 data: 3
8892 207/300 Read block 4607 len 16384
8893 208/300 Write block 890 len 24576 data: 2 2 2 2 2 2
8894 209/300 Write block 159 len 40960 data: 3 4 4 4 3 3 4 4 3 2
8895 210/300 Flush
8896 211/300 Read block 2433 len 36864
8897 212/300 Write block 3285 len 36864 data: 2 2 2 2 2 2 3 4 4
8898 213/300 Read block 1414 len 8192
8899 214/300 Read block 4170 len 20480
8900 215/300 Read block 1160 len 8192
8901 216/300 Read block 1711 len 36864
8902 217/300 Flush
8903 218/300 Read block 2456 len 24576
8904 219/300 Read block 1303 len 20480
8905 220/300 Read block 1035 len 32768
8906 221/300 Read block 3386 len 12288
8907 222/300 Read block 2910 len 28672
8908 223/300 Read block 4017 len 28672
8909 224/300 Write block 4586 len 20480 data: 3 3 3 3 3
8910 225/300 Read block 3975 len 32768
8911 226/300 Write block 1564 len 32768 data: 4 5 5 5 5 5 4 4
8912 227/300 Read block 1714 len 24576
8913 228/300 Write block 426 len 40960 data: 2 2 2 3 3 3 3 3 3 3
8914 229/300 Flush
8915 230/300 Read block 2420 len 20480
8916 231/300 Read block 4072 len 20480
8917 232/300 Write block 952 len 12288 data: 2 2 3
8918 233/300 Flush
8919 234/300 Read block 781 len 4096
8920 235/300 Read block 2041 len 20480
8921 236/300 Write block 1196 len 40960 data: 2 2 3 3 3 3 2 2 2 2
8922 237/300 Read block 1899 len 16384
8923 238/300 Write block 4072 len 16384 data: 3 3 3 4
8924 239/300 Read block 1836 len 24576
8925 240/300 Read block 2591 len 28672
8926 241/300 Read block 1836 len 36864
8927 242/300 Flush
8928 243/300 Write block 3755 len 36864 data: 3 3 3 2 2 2 2 2 2
8929 244/300 Write block 1053 len 12288 data: 4 3 3
8930 245/300 Read block 1920 len 28672
8931 246/300 Write block 2013 len 28672 data: 3 3 3 4 3 3 2
8932 247/300 Read block 3263 len 28672
8933 248/300 Flush
8934 249/300 Write block 1698 len 8192 data: 4 5
8935 250/300 Flush
8936 251/300 Flush
8937 252/300 Read block 2825 len 8192
8938 253/300 Flush
8939 254/300 Write block 3684 len 16384 data: 3 3 3 3
8940 255/300 Read block 383 len 40960
8941 256/300 Write block 3956 len 20480 data: 2 3 3 3 3
8942 257/300 Write block 1803 len 12288 data: 2 2 3
8943 258/300 Write block 2220 len 36864 data: 2 2 3 3 3 3 4 4 4
8944 259/300 Write block 2404 len 4096 data: 2
8945 260/300 Read block 4068 len 36864
8946 261/300 Flush
8947 262/300 Write block 317 len 12288 data: 2 2 2
8948 263/300 Write block 4549 len 8192 data: 4 3
8949 264/300 Read block 4529 len 24576
8950 265/300 Read block 1842 len 28672
8951 266/300 Read block 4316 len 36864
8952 267/300 Read block 2985 len 4096
8953 268/300 Write block 2388 len 40960 data: 3 3 3 3 2 3 3 3 2 2
8954 269/300 Write block 4271 len 16384 data: 3 3 3 4
8955 270/300 Read block 1626 len 8192
8956 271/300 Read block 352 len 24576
8957 272/300 Read block 3825 len 8192
8958 273/300 Write block 4333 len 40960 data: 4 4 4 4 4 3 3 4 4 5
8959 274/300 Write block 4639 len 12288 data: 3 3 3
8960 275/300 Read block 4101 len 24576
8961 276/300 Read block 1695 len 16384
8962 277/300 Flush
8963 278/300 Write block 255 len 36864 data: 3 3 2 2 2 2 3 3 3
8964 279/300 Read block 1621 len 24576
8965 280/300 Read block 1388 len 4096
8966 281/300 Read block 2695 len 4096
8967 282/300 Write block 4945 len 12288 data: 3 3 2
8968 283/300 Read block 3537 len 8192
8969 284/300 Read block 4362 len 8192
8970 285/300 Read block 4543 len 16384
8971 286/300 Write block 2287 len 8192 data: 3 3
8972 287/300 Read block 3824 len 28672
8973 288/300 Read block 4202 len 36864
8974 289/300 Read block 4966 len 24576
8975 290/300 Flush
8976 291/300 Write block 4414 len 24576 data: 2 2 2 2 2 2
8977 292/300 Read block 1125 len 36864
8978 293/300 Flush
8979 294/300 Flush
8980 295/300 Write block 423 len 16384 data: 3 3 3 3
8981 296/300 Write block 3434 len 28672 data: 4 4 4 4 3 3 3
8982 297/300 Write block 1160 len 12288 data: 2 3 3
8983 298/300 Read block 1121 len 16384
8984 299/300 Write block 2732 len 8192 data: 3 2
8985 300/300 Read block 3221 len 24576
8986 301/300 Write block 1550 len 4096 data: 3
8987 Main task has work to do, go find it
8988 got dsc Start(0)
8989 start 0
8990 Mar 07 18:14:34.283 INFO request completed, latency_us: 189, response_code: 204, uri: /start/cid/0, method: POST, req_id: 6d2ad52d-9735-4042-b31d-466923c20be9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
8991 Replay: started 0, returned:Ok(())
8992 ----------------------------------------------------------------
8993 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8994 States: Offline Active Active
8995 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8996 2810 Acked Write 5 Sent Done Done false
8997 2811 Acked Read 4 Sent Done Done false
8998 2812 Acked Read 4 Sent Done Done false
8999 2813 Acked Read 1 Sent Done Done false
9000 2814 Acked Read 5 Sent Done Done false
9001 2815 Acked Write 3 Sent Done Done false
9002 2816 Acked Read 5 Sent Done Done false
9003 2817 Acked Read 10 Sent Done Done false
9004 2818 Acked Read 3 Sent Done Done false
9005 2819 Acked Write 7 Sent Done Done false
9006 2820 Acked Read 4 Sent Done Done false
9007 2821 Acked Read 8 Sent Done Done false
9008 2822 Acked Read 4 Sent Done Done false
9009 2823 Acked Write 2 Sent Done Done false
9010 2824 Acked Read 2 Sent Done Done false
9011 2825 Acked Read 7 Sent Done Done false
9012 2826 Acked Read 8 Sent Done Done false
9013 2827 Acked Read 10 Sent Done Done false
9014 2828 Acked Write 2 Sent Done Done false
9015 2829 Acked Read 7 Sent Done Done false
9016 2830 Acked Read 5 Sent Done Done false
9017 2831 Acked Flush 0 Sent Done Done false
9018 2832 Acked Flush 0 Sent Done Done false
9019 2833 Acked Read 7 Sent Done Done false
9020 2834 Acked Read 1 Sent Done Done false
9021 2835 Acked Flush 0 Sent Done Done false
9022 2836 Acked Read 5 Sent Done Done false
9023 2837 Acked Write 8 Sent Done Done false
9024 2838 Acked Write 8 Sent Done Done false
9025 2839 Acked Read 8 Sent Done Done false
9026 2840 Acked Write 4 Sent Done Done false
9027 2841 Acked Write 6 Sent Done Done false
9028 2842 Acked Write 10 Sent Done Done false
9029 2843 Acked Read 4 Sent Done Done false
9030 2844 Acked Read 5 Sent Done Done false
9031 2845 Acked Read 10 Sent Done Done false
9032 2846 Acked Write 6 Sent Done Done false
9033 2847 Acked Read 3 Sent Done Done false
9034 2848 Acked Read 8 Sent Done Done false
9035 2849 Acked Flush 0 Sent Done Done false
9036 2850 Acked Read 3 Sent Done Done false
9037 2851 Acked Write 1 Sent Done Done false
9038 2852 Acked Read 5 Sent Done Done false
9039 2853 Acked Write 4 Sent Done Done false
9040 2854 Acked Write 8 Sent Done Done false
9041 2855 Acked Write 6 Sent Done Done false
9042 2856 Acked Write 3 Sent Done Done false
9043 2857 Acked Write 8 Sent Done Done false
9044 2858 Acked Read 2 Sent Done Done false
9045 2859 Acked Write 2 Sent Done Done false
9046 2860 Acked Read 9 Sent Done Done false
9047 2861 Acked Write 10 Sent Done Done false
9048 2862 Acked Write 5 Sent Done Done false
9049 2863 Acked Read 10 Sent Done Done false
9050 2864 Acked Read 5 Sent Done Done false
9051 2865 Acked Read 1 Sent Done Done false
9052 2866 Acked Read 1 Sent Done Done false
9053 2867 Acked Read 2 Sent Done Done false
9054 2868 Acked Write 8 Sent Done Done false
9055 2869 Acked Write 7 Sent Done Done false
9056 2870 Acked Read 1 Sent Done Done false
9057 2871 Acked Read 2 Sent Done Done false
9058 2872 Acked Read 10 Sent Done Done false
9059 2873 Acked Write 1 Sent Done Done false
9060 2874 Acked Write 3 Sent Done Done false
9061 2875 Acked Write 2 Sent Done Done false
9062 2876 Acked Flush 0 Sent Done Done false
9063 2877 Acked Read 8 Sent Done Done false
9064 2878 Acked Write 5 Sent Done Done false
9065 2879 Acked Read 9 Sent Done Done false
9066 2880 Acked Write 7 Sent Done Done false
9067 2881 Acked Write 1 Sent Done Done false
9068 2882 Acked Read 8 Sent Done Done false
9069 2883 Acked Write 4 Sent Done Done false
9070 2884 Acked Read 7 Sent Done Done false
9071 2885 Acked Write 8 Sent Done Done false
9072 2886 Acked Read 6 Sent Done Done false
9073 2887 Acked Flush 0 Sent Done Done false
9074 2888 Acked Read 4 Sent Done Done false
9075 2889 Acked Write 4 Sent Done Done false
9076 2890 Acked Flush 0 Sent Done Done false
9077 2891 Acked Flush 0 Sent Done Done false
9078 2892 Acked Write 2 Sent Done Done false
9079 2893 Acked Write 8 Sent Done Done false
9080 2894 Acked Write 8 Sent Done Done false
9081 2895 Acked Read 8 Sent Done Done false
9082 2896 Acked Write 5 Sent Done Done false
9083 2897 Acked Read 9 Sent Done Done false
9084 2898 Acked Read 8 Sent Done Done false
9085 2899 Acked Read 10 Sent Done Done false
9086 2900 Acked Read 9 Sent Done Done false
9087 2901 Acked Write 8 Sent Done Done false
9088 2902 Acked Read 1 Sent Done Done false
9089 2903 Acked Flush 0 Sent Done Done false
9090 2904 Acked Read 2 Sent Done Done false
9091 2905 Acked Read 6 Sent Done Done false
9092 2906 Acked Read 6 Sent Done Done false
9093 2907 Acked Flush 0 Sent Done Done false
9094 2908 Acked Read 5 Sent Done Done false
9095 2909 Acked Read 5 Sent Done Done false
9096 2910 Acked Write 8 Sent Done Done false
9097 2911 Acked Write 8 Sent Done Done false
9098 2912 Acked Read 2 Sent Done Done false
9099 2913 Acked Read 10 Sent Done Done false
9100 2914 Acked Read 10 Sent Done Done false
9101 2915 Acked Read 8 Sent Done Done false
9102 2916 Acked Write 4 Sent Done Done false
9103 2917 Acked Flush 0 Sent Done Done false
9104 2918 Acked Write 4 Sent Done Done false
9105 2919 Acked Read 8 Sent Done Done false
9106 2920 Acked Read 2 Sent Done Done false
9107 2921 Acked Read 10 Sent Done Done false
9108 2922 Acked Read 7 Sent Done Done false
9109 2923 Acked Write 9 Sent Done Done false
9110 2924 Acked Read 4 Sent Done Done false
9111 2925 Acked Write 2 Sent Done Done false
9112 2926 Acked Write 5 Sent Done Done false
9113 2927 Acked Write 6 Sent Done Done false
9114 2928 Acked Read 6 Sent Done Done false
9115 2929 Acked Read 10 Sent Done Done false
9116 2930 Acked Read 2 Sent Done Done false
9117 2931 Acked Flush 0 Sent Done Done false
9118 2932 Acked Write 4 Sent Done Done false
9119 2933 Acked Read 7 Sent Done Done false
9120 2934 Acked Read 7 Sent Done Done false
9121 2935 Acked Read 5 Sent Done Done false
9122 2936 Acked Read 1 Sent Done Done false
9123 2937 Acked Write 9 Sent Done Done false
9124 2938 Acked Write 7 Sent Done Done false
9125 2939 Acked Read 2 Sent Done Done false
9126 2940 Acked Write 7 Sent Done Done false
9127 2941 Acked Write 3 Sent Done Done false
9128 2942 Acked Read 3 Sent Done Done false
9129 2943 Acked Read 3 Sent Done Done false
9130 2944 Acked Write 8 Sent Done Done false
9131 2945 Acked Write 7 Sent Done Done false
9132 2946 Acked Write 4 Sent Done Done false
9133 2947 Acked Flush 0 Sent Done Done false
9134 2948 Acked Read 8 Sent Done Done false
9135 2949 Acked Flush 0 Sent Done Done false
9136 2950 Acked Read 1 Sent Done Done false
9137 2951 Acked Read 9 Sent Done Done false
9138 2952 Acked Read 4 Sent Done Done false
9139 2953 Acked Write 2 Sent Done Done false
9140 2954 Acked Write 5 Sent Done Done false
9141 2955 Acked Read 4 Sent Done Done false
9142 2956 Acked Write 10 Sent Done Done false
9143 2957 Acked Flush 0 Sent Done Done false
9144 2958 Acked Read 7 Sent Done Done false
9145 2959 Acked Read 6 Sent Done Done false
9146 2960 Acked Read 4 Sent Done Done false
9147 2961 Acked Write 5 Sent Done Done false
9148 2962 Acked Read 6 Sent Done Done false
9149 2963 Acked Flush 0 Sent Done Done false
9150 2964 Acked Read 3 Sent Done Done false
9151 2965 Acked Flush 0 Sent Done Done false
9152 2966 Acked Read 5 Sent Done Done false
9153 2967 Acked Write 5 Sent Done Done false
9154 2968 Acked Read 10 Sent Done Done false
9155 2969 Acked Write 6 Sent Done Done false
9156 2970 Acked Read 10 Sent Done Done false
9157 2971 Acked Read 6 Sent Done Done false
9158 2972 Acked Write 6 Sent Done Done false
9159 2973 Acked Flush 0 Sent Done Done false
9160 2974 Acked Write 3 Sent Done Done false
9161 2975 Acked Write 7 Sent Done Done false
9162 2976 Acked Read 4 Sent Done Done false
9163 2977 Acked Write 2 Sent Done Done false
9164 2978 Acked Write 5 Sent Done Done false
9165 2979 Acked Write 10 Sent Done Done false
9166 2980 Acked Write 2 Sent Done Done false
9167 2981 Acked Read 4 Sent Done Done false
9168 2982 Acked Write 3 Sent Done Done false
9169 2983 Acked Read 9 Sent Done Done false
9170 2984 Acked Read 8 Sent Done Done false
9171 2985 Acked Read 9 Sent Done Done false
9172 2986 Acked Read 2 Sent Done Done false
9173 2987 Acked Read 10 Sent Done Done false
9174 2988 Acked Write 4 Sent Done Done false
9175 2989 Acked Write 2 Sent Done Done false
9176 2990 Acked Read 3 Sent Done Done false
9177 2991 Acked Write 8 Sent Done Done false
9178 2992 Acked Read 9 Sent Done Done false
9179 2993 Acked Read 8 Sent Done Done false
9180 2994 Acked Read 2 Sent Done Done false
9181 2995 Acked Write 3 Sent Done Done false
9182 2996 Acked Read 7 Sent Done Done false
9183 2997 Acked Flush 0 Sent Done Done false
9184 2998 Acked Flush 0 Sent Done Done false
9185 2999 Acked Read 4 Sent Done Done false
9186 3000 Acked Flush 0 Sent Done Done false
9187 3001 Acked Write 9 Sent Done Done false
9188 3002 Acked Read 3 Sent Done Done false
9189 3003 Acked Write 9 Sent Done Done false
9190 3004 Acked Write 4 Sent Done Done false
9191 3005 Acked Write 3 Sent Done Done false
9192 3006 Acked Write 5 Sent Done Done false
9193 3007 Acked Read 3 Sent Done Done false
9194 3008 Acked Flush 0 Sent Done Done false
9195 3009 Acked Read 9 Sent Done Done false
9196 3010 Acked Read 1 Sent Done Done false
9197 3011 Acked Write 9 Sent Done Done false
9198 3012 Acked Read 5 Sent Done Done false
9199 3013 Acked Write 3 Sent Done Done false
9200 3014 Acked Read 2 Sent Done Done false
9201 3015 Acked Write 1 Sent Done Done false
9202 3016 Acked Read 4 Sent Done Done false
9203 3017 Acked Write 6 Sent Done Done false
9204 3018 Acked Write 10 Sent Done Done false
9205 3019 Acked Flush 0 Sent Done Done false
9206 3020 Acked Read 9 Sent Done Done false
9207 3021 Acked Write 9 Sent Done Done false
9208 3022 Acked Read 2 Sent Done Done false
9209 3023 Acked Read 5 Sent Done Done false
9210 3024 Acked Read 2 Sent Done Done false
9211 3025 Acked Read 9 Sent Done Done false
9212 3026 Acked Flush 0 Sent Done Done false
9213 3027 Acked Read 6 Sent Done Done false
9214 3028 Acked Read 5 Sent Done Done false
9215 3029 Acked Read 8 Sent Done Done false
9216 3030 Acked Read 3 Sent Done Done false
9217 3031 Acked Read 7 Sent Done Done false
9218 3032 Acked Read 7 Sent Done Done false
9219 3033 Acked Write 5 Sent Done Done false
9220 3034 Acked Read 8 Sent Done Done false
9221 3035 Acked Write 8 Sent Done Done false
9222 3036 Acked Read 6 Sent Done Done false
9223 3037 Acked Write 10 Sent Done Done false
9224 3038 Acked Flush 0 Sent Done Done false
9225 3039 Acked Read 5 Sent Done Done false
9226 3040 Acked Read 5 Sent Done Done false
9227 3041 Acked Write 3 Sent Done Done false
9228 3042 Acked Flush 0 Sent Done Done false
9229 3043 Acked Read 1 Sent Done Done false
9230 3044 Acked Read 5 Sent Done Done false
9231 3045 Acked Write 10 Sent Done Done false
9232 3046 Acked Read 4 Sent Done Done false
9233 3047 Acked Write 4 Sent Done Done false
9234 3048 Acked Read 6 Sent Done Done false
9235 3049 Acked Read 7 Sent Done Done false
9236 3050 Acked Read 9 Sent Done Done false
9237 3051 Acked Flush 0 Sent Done Done false
9238 3052 Acked Write 9 Sent Done Done false
9239 3053 Acked Write 3 Sent Done Done false
9240 3054 Acked Read 7 Sent Done Done false
9241 3055 Acked Write 7 Sent Done Done false
9242 3056 Acked Read 7 Sent Done Done false
9243 3057 Acked Flush 0 Sent Done Done false
9244 3058 Acked Write 2 Sent Done Done false
9245 3059 Acked Flush 0 Sent Done Done false
9246 3060 Acked Flush 0 Sent Done Done false
9247 3061 Acked Read 2 Sent Done Done false
9248 3062 Acked Flush 0 Sent Done Done false
9249 3063 Acked Write 4 Sent Done Done false
9250 3064 Acked Read 10 Sent Done Done false
9251 3065 Acked Write 5 Sent Done Done false
9252 3066 Acked Write 3 Sent Done Done false
9253 3067 Acked Write 9 Sent Done Done false
9254 3068 Acked Write 1 Sent Done Done false
9255 3069 Acked Read 9 Sent Done Done false
9256 3070 Acked Flush 0 Sent Done Done false
9257 3071 Acked Write 3 Sent Done Done false
9258 3072 Acked Write 2 Sent Done Done false
9259 3073 Acked Read 6 Sent Done Done false
9260 3074 Acked Read 7 Sent Done Done false
9261 3075 Acked Read 9 Sent Done Done false
9262 3076 Acked Read 1 Sent Done Done false
9263 3077 Acked Write 10 Sent Done Done false
9264 3078 Acked Write 4 Sent Done Done false
9265 3079 Acked Read 2 Sent Done Done false
9266 3080 Acked Read 6 Sent Done Done false
9267 3081 Acked Read 2 Sent Done Done false
9268 3082 Acked Write 10 Sent Done Done false
9269 3083 Acked Write 3 Sent Done Done false
9270 3084 Acked Read 6 Sent Done Done false
9271 3085 Acked Read 4 Sent Done Done false
9272 3086 Acked Flush 0 Sent Done Done false
9273 3087 Acked Write 9 Sent Done Done false
9274 3088 Acked Read 6 Sent Done Done false
9275 3089 Acked Read 1 Sent Done Done false
9276 3090 Acked Read 1 Sent Done Done false
9277 3091 Acked Write 3 Sent Done Done false
9278 3092 Acked Read 2 Sent Done Done false
9279 3093 Acked Read 2 Sent Done Done false
9280 3094 Acked Read 4 Sent Done Done false
9281 3095 Acked Write 2 Sent Done Done false
9282 3096 Acked Read 7 Sent Done Done false
9283 3097 Acked Read 9 Sent Done Done false
9284 3098 Acked Read 6 Sent Done Done false
9285 3099 Acked Flush 0 Sent Done Done false
9286 3100 Acked Write 6 Sent Done Done false
9287 3101 Acked Read 9 Sent Done Done false
9288 3102 Acked Flush 0 Sent Done Done false
9289 3103 Acked Flush 0 Sent Done Done false
9290 3104 Acked Write 4 Sent Done Done false
9291 3105 Acked Write 7 Sent Done Done false
9292 3106 Acked Write 3 Sent Done Done false
9293 3107 Acked Read 4 Sent Done Done false
9294 3108 Acked Write 2 Sent Done Done false
9295 3109 Acked Read 6 Sent Done Done false
9296 3110 Acked Write 1 Sent Done Done false
9297 STATES DS:0 DS:1 DS:2 TOTAL
9298 Sent 301 0 0 301
9299 Done 0 301 301 602
9300 Skipped 0 0 0 0
9301 Error 0 0 0 0
9302 Last Flush: Some(JobId(2809)) Some(JobId(3103)) Some(JobId(3103))
9303 Downstairs last five completed: 2809 2808 2807 2806 2805
9304 Upstairs last five completed: 3110 3109 3108 3107 3106
9305 CLIENT: Up:0 ds:301 act:2
93062025-03-07T18:14:35.143ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
9307 [8810] got start action
9308 [8810] I am going to restart
9309 Starting downstairs at port 8810
9310 Make output file at "/tmp/dsc/downstairs-8810.txt"
9311 [8810][0] reports Starting
9312 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1215)
9313 [8810][0] reports Running
93142025-03-07T18:14:36.144ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
93152025-03-07T18:14:37.265ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
93162025-03-07T18:14:38.149ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
9317 ----------------------------------------------------------------
9318 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9319 States: Offline Active Active
9320 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9321 2810 Acked Write 5 Sent Done Done false
9322 2811 Acked Read 4 Sent Done Done false
9323 2812 Acked Read 4 Sent Done Done false
9324 2813 Acked Read 1 Sent Done Done false
9325 2814 Acked Read 5 Sent Done Done false
9326 2815 Acked Write 3 Sent Done Done false
9327 2816 Acked Read 5 Sent Done Done false
9328 2817 Acked Read 10 Sent Done Done false
9329 2818 Acked Read 3 Sent Done Done false
9330 2819 Acked Write 7 Sent Done Done false
9331 2820 Acked Read 4 Sent Done Done false
9332 2821 Acked Read 8 Sent Done Done false
9333 2822 Acked Read 4 Sent Done Done false
9334 2823 Acked Write 2 Sent Done Done false
9335 2824 Acked Read 2 Sent Done Done false
9336 2825 Acked Read 7 Sent Done Done false
9337 2826 Acked Read 8 Sent Done Done false
9338 2827 Acked Read 10 Sent Done Done false
9339 2828 Acked Write 2 Sent Done Done false
9340 2829 Acked Read 7 Sent Done Done false
9341 2830 Acked Read 5 Sent Done Done false
9342 2831 Acked Flush 0 Sent Done Done false
9343 2832 Acked Flush 0 Sent Done Done false
9344 2833 Acked Read 7 Sent Done Done false
9345 2834 Acked Read 1 Sent Done Done false
9346 2835 Acked Flush 0 Sent Done Done false
9347 2836 Acked Read 5 Sent Done Done false
9348 2837 Acked Write 8 Sent Done Done false
9349 2838 Acked Write 8 Sent Done Done false
9350 2839 Acked Read 8 Sent Done Done false
9351 2840 Acked Write 4 Sent Done Done false
9352 2841 Acked Write 6 Sent Done Done false
9353 2842 Acked Write 10 Sent Done Done false
9354 2843 Acked Read 4 Sent Done Done false
9355 2844 Acked Read 5 Sent Done Done false
9356 2845 Acked Read 10 Sent Done Done false
9357 2846 Acked Write 6 Sent Done Done false
9358 2847 Acked Read 3 Sent Done Done false
9359 2848 Acked Read 8 Sent Done Done false
9360 2849 Acked Flush 0 Sent Done Done false
9361 2850 Acked Read 3 Sent Done Done false
9362 2851 Acked Write 1 Sent Done Done false
9363 2852 Acked Read 5 Sent Done Done false
9364 2853 Acked Write 4 Sent Done Done false
9365 2854 Acked Write 8 Sent Done Done false
9366 2855 Acked Write 6 Sent Done Done false
9367 2856 Acked Write 3 Sent Done Done false
9368 2857 Acked Write 8 Sent Done Done false
9369 2858 Acked Read 2 Sent Done Done false
9370 2859 Acked Write 2 Sent Done Done false
9371 2860 Acked Read 9 Sent Done Done false
9372 2861 Acked Write 10 Sent Done Done false
9373 2862 Acked Write 5 Sent Done Done false
9374 2863 Acked Read 10 Sent Done Done false
9375 2864 Acked Read 5 Sent Done Done false
9376 2865 Acked Read 1 Sent Done Done false
9377 2866 Acked Read 1 Sent Done Done false
9378 2867 Acked Read 2 Sent Done Done false
9379 2868 Acked Write 8 Sent Done Done false
9380 2869 Acked Write 7 Sent Done Done false
9381 2870 Acked Read 1 Sent Done Done false
9382 2871 Acked Read 2 Sent Done Done false
9383 2872 Acked Read 10 Sent Done Done false
9384 2873 Acked Write 1 Sent Done Done false
9385 2874 Acked Write 3 Sent Done Done false
9386 2875 Acked Write 2 Sent Done Done false
9387 2876 Acked Flush 0 Sent Done Done false
9388 2877 Acked Read 8 Sent Done Done false
9389 2878 Acked Write 5 Sent Done Done false
9390 2879 Acked Read 9 Sent Done Done false
9391 2880 Acked Write 7 Sent Done Done false
9392 2881 Acked Write 1 Sent Done Done false
9393 2882 Acked Read 8 Sent Done Done false
9394 2883 Acked Write 4 Sent Done Done false
9395 2884 Acked Read 7 Sent Done Done false
9396 2885 Acked Write 8 Sent Done Done false
9397 2886 Acked Read 6 Sent Done Done false
9398 2887 Acked Flush 0 Sent Done Done false
9399 2888 Acked Read 4 Sent Done Done false
9400 2889 Acked Write 4 Sent Done Done false
9401 2890 Acked Flush 0 Sent Done Done false
9402 2891 Acked Flush 0 Sent Done Done false
9403 2892 Acked Write 2 Sent Done Done false
9404 2893 Acked Write 8 Sent Done Done false
9405 2894 Acked Write 8 Sent Done Done false
9406 2895 Acked Read 8 Sent Done Done false
9407 2896 Acked Write 5 Sent Done Done false
9408 2897 Acked Read 9 Sent Done Done false
9409 2898 Acked Read 8 Sent Done Done false
9410 2899 Acked Read 10 Sent Done Done false
9411 2900 Acked Read 9 Sent Done Done false
9412 2901 Acked Write 8 Sent Done Done false
9413 2902 Acked Read 1 Sent Done Done false
9414 2903 Acked Flush 0 Sent Done Done false
9415 2904 Acked Read 2 Sent Done Done false
9416 2905 Acked Read 6 Sent Done Done false
9417 2906 Acked Read 6 Sent Done Done false
9418 2907 Acked Flush 0 Sent Done Done false
9419 2908 Acked Read 5 Sent Done Done false
9420 2909 Acked Read 5 Sent Done Done false
9421 2910 Acked Write 8 Sent Done Done false
9422 2911 Acked Write 8 Sent Done Done false
9423 2912 Acked Read 2 Sent Done Done false
9424 2913 Acked Read 10 Sent Done Done false
9425 2914 Acked Read 10 Sent Done Done false
9426 2915 Acked Read 8 Sent Done Done false
9427 2916 Acked Write 4 Sent Done Done false
9428 2917 Acked Flush 0 Sent Done Done false
9429 2918 Acked Write 4 Sent Done Done false
9430 2919 Acked Read 8 Sent Done Done false
9431 2920 Acked Read 2 Sent Done Done false
9432 2921 Acked Read 10 Sent Done Done false
9433 2922 Acked Read 7 Sent Done Done false
9434 2923 Acked Write 9 Sent Done Done false
9435 2924 Acked Read 4 Sent Done Done false
9436 2925 Acked Write 2 Sent Done Done false
9437 2926 Acked Write 5 Sent Done Done false
9438 2927 Acked Write 6 Sent Done Done false
9439 2928 Acked Read 6 Sent Done Done false
9440 2929 Acked Read 10 Sent Done Done false
9441 2930 Acked Read 2 Sent Done Done false
9442 2931 Acked Flush 0 Sent Done Done false
9443 2932 Acked Write 4 Sent Done Done false
9444 2933 Acked Read 7 Sent Done Done false
9445 2934 Acked Read 7 Sent Done Done false
9446 2935 Acked Read 5 Sent Done Done false
9447 2936 Acked Read 1 Sent Done Done false
9448 2937 Acked Write 9 Sent Done Done false
9449 2938 Acked Write 7 Sent Done Done false
9450 2939 Acked Read 2 Sent Done Done false
9451 2940 Acked Write 7 Sent Done Done false
9452 2941 Acked Write 3 Sent Done Done false
9453 2942 Acked Read 3 Sent Done Done false
9454 2943 Acked Read 3 Sent Done Done false
9455 2944 Acked Write 8 Sent Done Done false
9456 2945 Acked Write 7 Sent Done Done false
9457 2946 Acked Write 4 Sent Done Done false
9458 2947 Acked Flush 0 Sent Done Done false
9459 2948 Acked Read 8 Sent Done Done false
9460 2949 Acked Flush 0 Sent Done Done false
9461 2950 Acked Read 1 Sent Done Done false
9462 2951 Acked Read 9 Sent Done Done false
9463 2952 Acked Read 4 Sent Done Done false
9464 2953 Acked Write 2 Sent Done Done false
9465 2954 Acked Write 5 Sent Done Done false
9466 2955 Acked Read 4 Sent Done Done false
9467 2956 Acked Write 10 Sent Done Done false
9468 2957 Acked Flush 0 Sent Done Done false
9469 2958 Acked Read 7 Sent Done Done false
9470 2959 Acked Read 6 Sent Done Done false
9471 2960 Acked Read 4 Sent Done Done false
9472 2961 Acked Write 5 Sent Done Done false
9473 2962 Acked Read 6 Sent Done Done false
9474 2963 Acked Flush 0 Sent Done Done false
9475 2964 Acked Read 3 Sent Done Done false
9476 2965 Acked Flush 0 Sent Done Done false
9477 2966 Acked Read 5 Sent Done Done false
9478 2967 Acked Write 5 Sent Done Done false
9479 2968 Acked Read 10 Sent Done Done false
9480 2969 Acked Write 6 Sent Done Done false
9481 2970 Acked Read 10 Sent Done Done false
9482 2971 Acked Read 6 Sent Done Done false
9483 2972 Acked Write 6 Sent Done Done false
9484 2973 Acked Flush 0 Sent Done Done false
9485 2974 Acked Write 3 Sent Done Done false
9486 2975 Acked Write 7 Sent Done Done false
9487 2976 Acked Read 4 Sent Done Done false
9488 2977 Acked Write 2 Sent Done Done false
9489 2978 Acked Write 5 Sent Done Done false
9490 2979 Acked Write 10 Sent Done Done false
9491 2980 Acked Write 2 Sent Done Done false
9492 2981 Acked Read 4 Sent Done Done false
9493 2982 Acked Write 3 Sent Done Done false
9494 2983 Acked Read 9 Sent Done Done false
9495 2984 Acked Read 8 Sent Done Done false
9496 2985 Acked Read 9 Sent Done Done false
9497 2986 Acked Read 2 Sent Done Done false
9498 2987 Acked Read 10 Sent Done Done false
9499 2988 Acked Write 4 Sent Done Done false
9500 2989 Acked Write 2 Sent Done Done false
9501 2990 Acked Read 3 Sent Done Done false
9502 2991 Acked Write 8 Sent Done Done false
9503 2992 Acked Read 9 Sent Done Done false
9504 2993 Acked Read 8 Sent Done Done false
9505 2994 Acked Read 2 Sent Done Done false
9506 2995 Acked Write 3 Sent Done Done false
9507 2996 Acked Read 7 Sent Done Done false
9508 2997 Acked Flush 0 Sent Done Done false
9509 2998 Acked Flush 0 Sent Done Done false
9510 2999 Acked Read 4 Sent Done Done false
9511 3000 Acked Flush 0 Sent Done Done false
9512 3001 Acked Write 9 Sent Done Done false
9513 3002 Acked Read 3 Sent Done Done false
9514 3003 Acked Write 9 Sent Done Done false
9515 3004 Acked Write 4 Sent Done Done false
9516 3005 Acked Write 3 Sent Done Done false
9517 3006 Acked Write 5 Sent Done Done false
9518 3007 Acked Read 3 Sent Done Done false
9519 3008 Acked Flush 0 Sent Done Done false
9520 3009 Acked Read 9 Sent Done Done false
9521 3010 Acked Read 1 Sent Done Done false
9522 3011 Acked Write 9 Sent Done Done false
9523 3012 Acked Read 5 Sent Done Done false
9524 3013 Acked Write 3 Sent Done Done false
9525 3014 Acked Read 2 Sent Done Done false
9526 3015 Acked Write 1 Sent Done Done false
9527 3016 Acked Read 4 Sent Done Done false
9528 3017 Acked Write 6 Sent Done Done false
9529 3018 Acked Write 10 Sent Done Done false
9530 3019 Acked Flush 0 Sent Done Done false
9531 3020 Acked Read 9 Sent Done Done false
9532 3021 Acked Write 9 Sent Done Done false
9533 3022 Acked Read 2 Sent Done Done false
9534 3023 Acked Read 5 Sent Done Done false
9535 3024 Acked Read 2 Sent Done Done false
9536 3025 Acked Read 9 Sent Done Done false
9537 3026 Acked Flush 0 Sent Done Done false
9538 3027 Acked Read 6 Sent Done Done false
9539 3028 Acked Read 5 Sent Done Done false
9540 3029 Acked Read 8 Sent Done Done false
9541 3030 Acked Read 3 Sent Done Done false
9542 3031 Acked Read 7 Sent Done Done false
9543 3032 Acked Read 7 Sent Done Done false
9544 3033 Acked Write 5 Sent Done Done false
9545 3034 Acked Read 8 Sent Done Done false
9546 3035 Acked Write 8 Sent Done Done false
9547 3036 Acked Read 6 Sent Done Done false
9548 3037 Acked Write 10 Sent Done Done false
9549 3038 Acked Flush 0 Sent Done Done false
9550 3039 Acked Read 5 Sent Done Done false
9551 3040 Acked Read 5 Sent Done Done false
9552 3041 Acked Write 3 Sent Done Done false
9553 3042 Acked Flush 0 Sent Done Done false
9554 3043 Acked Read 1 Sent Done Done false
9555 3044 Acked Read 5 Sent Done Done false
9556 3045 Acked Write 10 Sent Done Done false
9557 3046 Acked Read 4 Sent Done Done false
9558 3047 Acked Write 4 Sent Done Done false
9559 3048 Acked Read 6 Sent Done Done false
9560 3049 Acked Read 7 Sent Done Done false
9561 3050 Acked Read 9 Sent Done Done false
9562 3051 Acked Flush 0 Sent Done Done false
9563 3052 Acked Write 9 Sent Done Done false
9564 3053 Acked Write 3 Sent Done Done false
9565 3054 Acked Read 7 Sent Done Done false
9566 3055 Acked Write 7 Sent Done Done false
9567 3056 Acked Read 7 Sent Done Done false
9568 3057 Acked Flush 0 Sent Done Done false
9569 3058 Acked Write 2 Sent Done Done false
9570 3059 Acked Flush 0 Sent Done Done false
9571 3060 Acked Flush 0 Sent Done Done false
9572 3061 Acked Read 2 Sent Done Done false
9573 3062 Acked Flush 0 Sent Done Done false
9574 3063 Acked Write 4 Sent Done Done false
9575 3064 Acked Read 10 Sent Done Done false
9576 3065 Acked Write 5 Sent Done Done false
9577 3066 Acked Write 3 Sent Done Done false
9578 3067 Acked Write 9 Sent Done Done false
9579 3068 Acked Write 1 Sent Done Done false
9580 3069 Acked Read 9 Sent Done Done false
9581 3070 Acked Flush 0 Sent Done Done false
9582 3071 Acked Write 3 Sent Done Done false
9583 3072 Acked Write 2 Sent Done Done false
9584 3073 Acked Read 6 Sent Done Done false
9585 3074 Acked Read 7 Sent Done Done false
9586 3075 Acked Read 9 Sent Done Done false
9587 3076 Acked Read 1 Sent Done Done false
9588 3077 Acked Write 10 Sent Done Done false
9589 3078 Acked Write 4 Sent Done Done false
9590 3079 Acked Read 2 Sent Done Done false
9591 3080 Acked Read 6 Sent Done Done false
9592 3081 Acked Read 2 Sent Done Done false
9593 3082 Acked Write 10 Sent Done Done false
9594 3083 Acked Write 3 Sent Done Done false
9595 3084 Acked Read 6 Sent Done Done false
9596 3085 Acked Read 4 Sent Done Done false
9597 3086 Acked Flush 0 Sent Done Done false
9598 3087 Acked Write 9 Sent Done Done false
9599 3088 Acked Read 6 Sent Done Done false
9600 3089 Acked Read 1 Sent Done Done false
9601 3090 Acked Read 1 Sent Done Done false
9602 3091 Acked Write 3 Sent Done Done false
9603 3092 Acked Read 2 Sent Done Done false
9604 3093 Acked Read 2 Sent Done Done false
9605 3094 Acked Read 4 Sent Done Done false
9606 3095 Acked Write 2 Sent Done Done false
9607 3096 Acked Read 7 Sent Done Done false
9608 3097 Acked Read 9 Sent Done Done false
9609 3098 Acked Read 6 Sent Done Done false
9610 3099 Acked Flush 0 Sent Done Done false
9611 3100 Acked Write 6 Sent Done Done false
9612 3101 Acked Read 9 Sent Done Done false
9613 3102 Acked Flush 0 Sent Done Done false
9614 3103 Acked Flush 0 Sent Done Done false
9615 3104 Acked Write 4 Sent Done Done false
9616 3105 Acked Write 7 Sent Done Done false
9617 3106 Acked Write 3 Sent Done Done false
9618 3107 Acked Read 4 Sent Done Done false
9619 3108 Acked Write 2 Sent Done Done false
9620 3109 Acked Read 6 Sent Done Done false
9621 3110 Acked Write 1 Sent Done Done false
9622 STATES DS:0 DS:1 DS:2 TOTAL
9623 Sent 301 0 0 301
9624 Done 0 301 301 602
9625 Skipped 0 0 0 0
9626 Error 0 0 0 0
9627 Last Flush: Some(JobId(2809)) Some(JobId(3103)) Some(JobId(3103))
9628 Downstairs last five completed: 2809 2808 2807 2806 2805
9629 Upstairs last five completed: 3110 3109 3108 3107 3106
9630 CLIENT: Up:0 ds:301 act:2
96312025-03-07T18:14:39.150ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96322025-03-07T18:14:40.035ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96332025-03-07T18:14:40.035ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43204) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96342025-03-07T18:14:40.036ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96352025-03-07T18:14:40.037ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96362025-03-07T18:14:40.037ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96372025-03-07T18:14:40.037ZINFOcrucible: send last flush ID to this DS: Some(JobId(2809)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96382025-03-07T18:14:40.037ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96392025-03-07T18:14:40.037ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2809)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96402025-03-07T18:14:40.037ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2809)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
9641 ----------------------------------------------------------------
9642 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9643 Downstairs last five completed: 3111 3110 3109 3108 3107
9644 Upstairs last five completed: 3111 3110 3109 3108 3107
9645 CLIENT: Up:0 ds:0 act:3
9646 Replay: All jobs finished
9647 Main task has work to do, go find it
9648 got dsc Stop(1)
9649 stop 1
9650 Mar 07 18:14:42.288 INFO request completed, latency_us: 318[8820] Got stop action so:false kr:false
9651 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 705ca540-c3c8-4326-826a-1e790c701e2e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
96522025-03-07T18:14:42.289ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96532025-03-07T18:14:42.289ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96542025-03-07T18:14:42.289ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
96552025-03-07T18:14:42.289ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
9656 Mar 07 18:14:42.289 INFO request completed, latency_us: 161, response_code: 200, uri: /state/cid/1, method: GET, req_id: 563330db-f247-44dd-ab6c-a40e844a6f46, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
9657 [8820] Exited with: ExitStatus(unix_wait_status(9))
9658 [8820][1] reports Exit
9659 Mar 07 18:14:46.292 INFO request completed, latency_us: 304, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4c073d23-71b8-4b3a-a559-aa720d3793a8, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
9660 001/300 Read block 984 len 40960
9661 002/300 Read block 2176 len 40960
9662 003/300 Read block 2717 len 28672
9663 004/300 Read block 2453 len 4096
9664 005/300 Write block 230 len 16384 data: 2 2 2 2
9665 006/300 Read block 308 len 4096
9666 007/300 Write block 2921 len 40960 data: 4 4 4 5 4 3 3 3 2 2
9667 008/300 Read block 2248 len 20480
9668 009/300 Write block 1715 len 20480 data: 2 2 2 2 2
9669 010/300 Write block 2556 len 28672 data: 3 2 2 2 2 2 2
9670 011/300 Write block 195 len 4096 data: 4
9671 012/300 Read block 4438 len 12288
9672 013/300 Read block 2993 len 36864
9673 014/300 Write block 1206 len 36864 data: 4 4 4 4 4 4 3 2 2
9674 015/300 Read block 329 len 12288
9675 016/300 Read block 4278 len 36864
9676 017/300 Write block 51 len 32768 data: 2 3 3 3 3 3 2 2
9677 018/300 Read block 465 len 32768
9678 019/300 Read block 2992 len 24576
9679 020/300 Read block 2365 len 24576
9680 021/300 Read block 2692 len 20480
9681 022/300 Read block 4526 len 36864
9682 023/300 Write block 3550 len 28672 data: 2 2 3 3 3 3 3
9683 024/300 Write block 2087 len 20480 data: 2 2 2 2 2
9684 025/300 Read block 4427 len 40960
9685 026/300 Read block 621 len 4096
9686 027/300 Flush
9687 028/300 Read block 4706 len 16384
9688 029/300 Write block 2521 len 4096 data: 2
9689 030/300 Write block 3369 len 20480 data: 3 2 2 2 2
9690 031/300 Read block 4274 len 32768
9691 032/300 Flush
9692 033/300 Flush
9693 034/300 Read block 753 len 40960
9694 035/300 Write block 18 len 8192 data: 5 5
9695 036/300 Read block 4218 len 4096
9696 037/300 Write block 4518 len 12288 data: 2 3 3
9697 038/300 Read block 1102 len 16384
9698 039/300 Write block 1908 len 16384 data: 5 5 5 5
9699 040/300 Write block 4612 len 8192 data: 3 2
9700 041/300 Write block 1222 len 24576 data: 3 2 2 2 2 2
9701 042/300 Write block 2700 len 16384 data: 2 2 2 2
9702 043/300 Read block 1584 len 12288
9703 044/300 Write block 273 len 4096 data: 2
9704 045/300 Flush
9705 046/300 Write block 881 len 4096 data: 2
9706 047/300 Flush
9707 048/300 Read block 1793 len 40960
9708 049/300 Read block 3889 len 12288
9709 050/300 Write block 3885 len 12288 data: 4 4 2
9710 051/300 Read block 1971 len 32768
9711 052/300 Read block 3459 len 4096
9712 053/300 Write block 4172 len 4096 data: 3
9713 054/300 Write block 602 len 24576 data: 3 3 3 3 3 3
9714 055/300 Read block 689 len 24576
9715 056/300 Read block 3309 len 40960
9716 057/300 Write block 3877 len 12288 data: 2 2 2
9717 058/300 Read block 3300 len 8192
9718 059/300 Flush
9719 060/300 Read block 404 len 32768
9720 061/300 Read block 3057 len 36864
9721 062/300 Read block 3006 len 36864
9722 063/300 Read block 3299 len 20480
9723 064/300 Write block 3996 len 40960 data: 3 3 3 3 2 2 2 2 2 2
9724 065/300 Read block 1262 len 32768
9725 066/300 Read block 974 len 24576
9726 067/300 Read block 53 len 20480
9727 068/300 Write block 672 len 4096 data: 5
9728 069/300 Read block 723 len 8192
9729 070/300 Read block 1826 len 24576
9730 071/300 Write block 2699 len 16384 data: 2 3 3 3
9731 072/300 Read block 4630 len 16384
9732 073/300 Write block 2761 len 40960 data: 2 3 3 3 3 3 4 4 2 2
9733 074/300 Read block 2360 len 28672
9734 075/300 Write block 3808 len 28672 data: 2 2 2 2 3 2 2
9735 076/300 Read block 3785 len 20480
9736 077/300 Flush
9737 078/300 Write block 2899 len 36864 data: 2 2 3 3 3 3 3 3 2
9738 079/300 Read block 1025 len 28672
9739 080/300 Write block 4455 len 16384 data: 4 4 4 2
9740 081/300 Write block 3492 len 16384 data: 3 2 2 2
9741 082/300 Write block 1187 len 4096 data: 2
9742 083/300 Write block 2805 len 8192 data: 3 3
9743 084/300 Read block 4812 len 20480
9744 085/300 Read block 2078 len 20480
9745 086/300 Flush
9746 087/300 Read block 2674 len 8192
9747 088/300 Read block 4444 len 28672
9748 089/300 Flush
9749 090/300 Write block 4961 len 16384 data: 2 3 3 3
9750 091/300 Write block 1128 len 8192 data: 3 3
9751 092/300 Write block 3926 len 28672 data: 2 2 2 2 2 2 2
9752 093/300 Read block 1001 len 40960
9753 094/300 Flush
9754 095/300 Write block 4742 len 40960 data: 3 4 4 4 4 4 4 2 2 2
9755 096/300 Read block 574 len 28672
9756 097/300 Read block 2394 len 12288
9757 098/300 Flush
9758 099/300 Read block 4039 len 16384
9759 100/300 Write block 1244 len 12288 data: 2 2 2
9760 101/300 Write block 3275 len 16384 data: 2 2 2 3
9761 102/300 Flush
9762 103/300 Write block 4597 len 24576 data: 3 3 3 3 3 3
9763 104/300 Write block 955 len 40960 data: 3 4 4 4 3 3 2 2 2 2
9764 105/300 Write block 2555 len 28672 data: 4 4 3 3 3 3 3
9765 106/300 Read block 2800 len 24576
9766 107/300 Read block 4942 len 16384
9767 108/300 Write block 4902 len 16384 data: 4 4 4 3
9768 109/300 Read block 3052 len 24576
9769 110/300 Read block 322 len 8192
9770 111/300 Write block 3661 len 32768 data: 2 2 3 2 3 3 3 3
9771 112/300 Flush
9772 113/300 Read block 2766 len 32768
9773 114/300 Write block 434 len 4096 data: 4
9774 115/300 Write block 3203 len 32768 data: 3 2 2 2 2 2 2 2
9775 116/300 Read block 3711 len 32768
9776 117/300 Read block 2199 len 40960
9777 118/300 Flush
9778 119/300 Read block 3235 len 32768
9779 120/300 Read block 814 len 20480
9780 121/300 Read block 759 len 32768
9781 122/300 Read block 2429 len 40960
9782 123/300 Write block 3420 len 36864 data: 4 2 2 2 2 2 2 2 3
9783 124/300 Write block 1432 len 16384 data: 2 2 3 3
9784 125/300 Flush
9785 126/300 Write block 3412 len 16384 data: 2 2 3 3
9786 127/300 Read block 1121 len 28672
9787 128/300 Read block 1392 len 20480
9788 129/300 Read block 3982 len 4096
9789 130/300 Write block 2405 len 32768 data: 2 2 2 2 2 2 2 2
9790 131/300 Read block 2538 len 16384
9791 132/300 Read block 2989 len 8192
9792 133/300 Read block 4691 len 12288
9793 134/300 Read block 2929 len 28672
9794 135/300 Read block 2261 len 4096
9795 136/300 Read block 541 len 36864
9796 137/300 Read block 4646 len 20480
9797 138/300 Read block 291 len 40960
9798 139/300 Flush
9799 140/300 Write block 1585 len 32768 data: 2 2 2 2 2 3 3 3
9800 141/300 Write block 3084 len 32768 data: 5 4 3 3 3 3 3 3
9801 142/300 Write block 784 len 12288 data: 2 2 3
9802 143/300 Write block 1138 len 40960 data: 2 2 2 2 2 2 2 2 3 3
9803 144/300 Write block 3055 len 4096 data: 3
9804 145/300 Write block 2104 len 12288 data: 2 2 2
9805 146/300 Read block 816 len 8192
9806 147/300 Read block 156 len 16384
9807 148/300 Read block 1547 len 8192
9808 149/300 Read block 3790 len 28672
9809 150/300 Read block 696 len 28672
9810 151/300 Read block 2286 len 4096
9811 152/300 Read block 1224 len 40960
9812 153/300 Write block 3372 len 24576 data: 3 3 2 2 2 2
9813 154/300 Write block 4208 len 20480 data: 4 2 2 2 2
9814 155/300 Read block 4753 len 28672
9815 156/300 Write block 4410 len 28672 data: 2 2 2 2 3 3 3
9816 157/300 Read block 4888 len 12288
9817 158/300 Flush
9818 159/300 Read block 4538 len 40960
9819 160/300 Flush
9820 161/300 Write block 68 len 36864 data: 3 3 3 3 4 4 4 4 4
9821 162/300 Write block 1058 len 8192 data: 2 2
9822 163/300 Write block 4274 len 24576 data: 5 4 4 5 5 6
9823 164/300 Flush
9824 165/300 Read block 973 len 8192
9825 166/300 Flush
9826 167/300 Flush
9827 168/300 Read block 2566 len 8192
9828 169/300 Write block 2342 len 4096 data: 4
9829 170/300 Read block 3711 len 32768
9830 171/300 Read block 3651 len 4096
9831 172/300 Read block 3171 len 20480
9832 173/300 Read block 349 len 40960
9833 174/300 Read block 2417 len 16384
9834 175/300 Flush
9835 176/300 Read block 4911 len 12288
9836 177/300 Write block 4030 len 8192 data: 2 2
9837 178/300 Write block 2186 len 36864 data: 4 4 4 4 3 3 3 3 2
9838 179/300 Write block 348 len 24576 data: 3 3 4 3 3 2
9839 180/300 Write block 2778 len 4096 data: 3
9840 181/300 Read block 720 len 40960
9841 182/300 Read block 1466 len 8192
9842 183/300 Write block 4198 len 36864 data: 2 2 2 2 3 4 4 4 4
9843 184/300 Write block 4247 len 28672 data: 2 3 3 3 3 3 3
9844 185/300 Write block 2029 len 4096 data: 3
9845 186/300 Read block 4537 len 28672
9846 187/300 Read block 2535 len 28672
9847 188/300 Write block 4135 len 8192 data: 2 4
9848 189/300 Read block 4233 len 16384
9849 190/300 Read block 4543 len 36864
9850 191/300 Read block 3063 len 8192
9851 192/300 Read block 573 len 8192
9852 193/300 Write block 2509 len 36864 data: 3 3 4 5 6 5 5 5 4
9853 194/300 Read block 278 len 36864
9854 195/300 Read block 2571 len 28672
9855 196/300 Read block 1510 len 12288
9856 197/300 Read block 576 len 28672
9857 198/300 Read block 2171 len 12288
9858 199/300 Write block 3857 len 36864 data: 3 2 2 2 2 2 2 2 2
9859 200/300 Read block 3103 len 28672
9860 201/300 Write block 1818 len 32768 data: 4 4 3 3 4 3 3 3
9861 202/300 Read block 4100 len 4096
9862 203/300 Read block 3130 len 24576
9863 204/300 Read block 2636 len 24576
9864 205/300 Write block 3809 len 36864 data: 3 3 3 4 3 3 2 2 2
9865 206/300 Flush
9866 207/300 Read block 657 len 16384
9867 208/300 Write block 1598 len 4096 data: 3
9868 209/300 Read block 1351 len 24576
9869 210/300 Read block 2736 len 28672
9870 211/300 Read block 4058 len 4096
9871 212/300 Write block 4481 len 4096 data: 2
9872 213/300 Write block 4874 len 28672 data: 3 3 3 3 2 2 2
9873 214/300 Write block 3068 len 8192 data: 4 4
9874 215/300 Write block 1120 len 12288 data: 2 3 3
9875 216/300 Read block 1364 len 40960
9876 217/300 Write block 2209 len 8192 data: 3 2
9877 218/300 Write block 4832 len 28672 data: 2 2 2 2 3 4 4
9878 219/300 Read block 1464 len 28672
9879 220/300 Read block 4391 len 12288
9880 221/300 Read block 1037 len 16384
9881 222/300 Read block 2682 len 40960
9882 223/300 Read block 1950 len 24576
9883 224/300 Read block 3089 len 12288
9884 225/300 Read block 4010 len 16384
9885 226/300 Read block 3050 len 32768
9886 227/300 Read block 4583 len 36864
9887 228/300 Flush
9888 229/300 Flush
9889 230/300 Write block 2408 len 16384 data: 3 3 3 3
9890 231/300 Read block 3859 len 20480
9891 232/300 Flush
9892 233/300 Read block 1679 len 40960
9893 234/300 Write block 3927 len 40960 data: 3 3 3 3 3 3 2 2 2 2
9894 235/300 Write block 2730 len 40960 data: 3 3 4 3 2 2 2 2 2 2
9895 236/300 Read block 2296 len 4096
9896 237/300 Read block 3415 len 36864
9897 238/300 Read block 3407 len 12288
9898 239/300 Write block 4452 len 4096 data: 4
9899 240/300 Read block 513 len 36864
9900 241/300 Write block 2648 len 8192 data: 3 3
9901 242/300 Flush
9902 243/300 Write block 4369 len 20480 data: 2 2 3 3 4
9903 244/300 Read block 3448 len 12288
9904 245/300 Write block 395 len 32768 data: 3 3 3 3 3 2 2 2
9905 246/300 Flush
9906 247/300 Read block 4055 len 4096
9907 248/300 Write block 2544 len 16384 data: 4 3 3 3
9908 249/300 Read block 4650 len 36864
9909 250/300 Read block 457 len 8192
9910 251/300 Write block 2547 len 40960 data: 4 2 2 2 2 3 4 4 5 5
9911 252/300 Read block 1584 len 28672
9912 253/300 Write block 693 len 32768 data: 2 2 2 2 2 2 2 2
9913 254/300 Read block 2580 len 24576
9914 255/300 Write block 3779 len 24576 data: 2 2 4 5 5 5
9915 256/300 Write block 1667 len 40960 data: 2 2 3 3 3 3 3 3 2 2
9916 257/300 Write block 3763 len 12288 data: 3 2 2
9917 258/300 Read block 1479 len 24576
9918 259/300 Write block 347 len 40960 data: 3 4 4 5 4 4 3 2 2 2
9919 260/300 Write block 4363 len 28672 data: 3 2 4 4 3 3 3
9920 261/300 Read block 2021 len 32768
9921 262/300 Read block 1456 len 16384
9922 263/300 Write block 488 len 16384 data: 3 3 3 3
9923 264/300 Read block 1568 len 32768
9924 265/300 Read block 719 len 4096
9925 266/300 Read block 1 len 28672
9926 267/300 Write block 1748 len 12288 data: 2 2 2
9927 268/300 Write block 375 len 20480 data: 2 2 2 3 3
9928 269/300 Write block 3661 len 28672 data: 3 3 4 3 4 4 4
9929 270/300 Flush
9930 271/300 Read block 4071 len 4096
9931 272/300 Read block 2342 len 32768
9932 273/300 Read block 4589 len 4096
9933 274/300 Read block 1015 len 16384
9934 275/300 Read block 1079 len 28672
9935 276/300 Write block 384 len 40960 data: 2 2 2 2 2 2 2 2 2 3
9936 277/300 Read block 4895 len 40960
9937 278/300 Read block 1403 len 40960
9938 279/300 Read block 4349 len 16384
9939 280/300 Write block 2968 len 32768 data: 2 2 2 2 2 2 2 2
9940 281/300 Flush
9941 282/300 Write block 2339 len 36864 data: 4 4 4 5 3 2 2 2 2
9942 283/300 Read block 820 len 20480
9943 284/300 Write block 3896 len 36864 data: 3 4 3 3 3 3 3 3 2
9944 285/300 Write block 622 len 20480 data: 4 4 4 4 4
9945 286/300 Flush
9946 287/300 Write block 7 len 24576 data: 4 4 4 4 4 4
9947 288/300 Read block 4476 len 28672
9948 289/300 Flush
9949 290/300 Write block 2898 len 32768 data: 2 3 3 4 4 4 4 4
9950 291/300 Write block 2692 len 16384 data: 2 2 2 2
9951 292/300 Read block 835 len 28672
9952 293/300 Read block 2434 len 20480
9953 294/300 Write block 1949 len 20480 data: 4 4 4 2 2
9954 295/300 Read block 675 len 40960
9955 296/300 Read block 4565 len 36864
9956 297/300 Read block 2917 len 16384
9957 298/300 Read block 298 len 16384
9958 299/300 Read block 568 len 12288
9959 300/300 Read block 1167 len 4096
9960 301/300 Write block 4704 len 40960 data: 4 4 3 3 2 2 2 2 2 2
9961 Main task has work to do, go find it
9962 got dsc Start(1)
9963 start 1
9964 Mar 07 18:14:46.542 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: c9a50b1b-1466-4ad0-bdb8-9dc1116c381f, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
9965 Replay: started 1, returned:Ok(())
9966 ----------------------------------------------------------------
9967 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9968 States: Active Offline Active
9969 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9970 3112 Acked Read 10 Done Sent Done false
9971 3113 Acked Read 10 Done Sent Done false
9972 3114 Acked Read 7 Done Sent Done false
9973 3115 Acked Read 1 Done Sent Done false
9974 3116 Acked Write 4 Done Sent Done false
9975 3117 Acked Read 1 Done Sent Done false
9976 3118 Acked Write 10 Done Sent Done false
9977 3119 Acked Read 5 Done Sent Done false
9978 3120 Acked Write 5 Done Sent Done false
9979 3121 Acked Write 7 Done Sent Done false
9980 3122 Acked Write 1 Done Sent Done false
9981 3123 Acked Read 3 Done Sent Done false
9982 3124 Acked Read 9 Done Sent Done false
9983 3125 Acked Write 9 Done Sent Done false
9984 3126 Acked Read 3 Done Sent Done false
9985 3127 Acked Read 9 Done Sent Done false
9986 3128 Acked Write 8 Done Sent Done false
9987 3129 Acked Read 8 Done Sent Done false
9988 3130 Acked Read 6 Done Sent Done false
9989 3131 Acked Read 6 Done Sent Done false
9990 3132 Acked Read 5 Done Sent Done false
9991 3133 Acked Read 9 Done Sent Done false
9992 3134 Acked Write 7 Done Sent Done false
9993 3135 Acked Write 5 Done Sent Done false
9994 3136 Acked Read 10 Done Sent Done false
9995 3137 Acked Read 1 Done Sent Done false
9996 3138 Acked Flush 0 Done Sent Done false
9997 3139 Acked Read 4 Done Sent Done false
9998 3140 Acked Write 1 Done Sent Done false
9999 3141 Acked Write 5 Done Sent Done false
10000 3142 Acked Read 8 Done Sent Done false
10001 3143 Acked Flush 0 Done Sent Done false
10002 3144 Acked Flush 0 Done Sent Done false
10003 3145 Acked Read 10 Done Sent Done false
10004 3146 Acked Write 2 Done Sent Done false
10005 3147 Acked Read 1 Done Sent Done false
10006 3148 Acked Write 3 Done Sent Done false
10007 3149 Acked Read 4 Done Sent Done false
10008 3150 Acked Write 4 Done Sent Done false
10009 3151 Acked Write 2 Done Sent Done false
10010 3152 Acked Write 6 Done Sent Done false
10011 3153 Acked Write 4 Done Sent Done false
10012 3154 Acked Read 3 Done Sent Done false
10013 3155 Acked Write 1 Done Sent Done false
10014 3156 Acked Flush 0 Done Sent Done false
10015 3157 Acked Write 1 Done Sent Done false
10016 3158 Acked Flush 0 Done Sent Done false
10017 3159 Acked Read 10 Done Sent Done false
10018 3160 Acked Read 3 Done Sent Done false
10019 3161 Acked Write 3 Done Sent Done false
10020 3162 Acked Read 8 Done Sent Done false
10021 3163 Acked Read 1 Done Sent Done false
10022 3164 Acked Write 1 Done Sent Done false
10023 3165 Acked Write 6 Done Sent Done false
10024 3166 Acked Read 6 Done Sent Done false
10025 3167 Acked Read 10 Done Sent Done false
10026 3168 Acked Write 3 Done Sent Done false
10027 3169 Acked Read 2 Done Sent Done false
10028 3170 Acked Flush 0 Done Sent Done false
10029 3171 Acked Read 8 Done Sent Done false
10030 3172 Acked Read 9 Done Sent Done false
10031 3173 Acked Read 9 Done Sent Done false
10032 3174 Acked Read 5 Done Sent Done false
10033 3175 Acked Write 10 Done Sent Done false
10034 3176 Acked Read 8 Done Sent Done false
10035 3177 Acked Read 6 Done Sent Done false
10036 3178 Acked Read 5 Done Sent Done false
10037 3179 Acked Write 1 Done Sent Done false
10038 3180 Acked Read 2 Done Sent Done false
10039 3181 Acked Read 6 Done Sent Done false
10040 3182 Acked Write 4 Done Sent Done false
10041 3183 Acked Read 4 Done Sent Done false
10042 3184 Acked Write 10 Done Sent Done false
10043 3185 Acked Read 7 Done Sent Done false
10044 3186 Acked Write 7 Done Sent Done false
10045 3187 Acked Read 5 Done Sent Done false
10046 3188 Acked Flush 0 Done Sent Done false
10047 3189 Acked Write 9 Done Sent Done false
10048 3190 Acked Read 7 Done Sent Done false
10049 3191 Acked Write 4 Done Sent Done false
10050 3192 Acked Write 4 Done Sent Done false
10051 3193 Acked Write 1 Done Sent Done false
10052 3194 Acked Write 2 Done Sent Done false
10053 3195 Acked Read 5 Done Sent Done false
10054 3196 Acked Read 5 Done Sent Done false
10055 3197 Acked Flush 0 Done Sent Done false
10056 3198 Acked Read 2 Done Sent Done false
10057 3199 Acked Read 7 Done Sent Done false
10058 3200 Acked Flush 0 Done Sent Done false
10059 3201 Acked Write 4 Done Sent Done false
10060 3202 Acked Write 2 Done Sent Done false
10061 3203 Acked Write 7 Done Sent Done false
10062 3204 Acked Read 10 Done Sent Done false
10063 3205 Acked Flush 0 Done Sent Done false
10064 3206 Acked Write 10 Done Sent Done false
10065 3207 Acked Read 7 Done Sent Done false
10066 3208 Acked Read 3 Done Sent Done false
10067 3209 Acked Flush 0 Done Sent Done false
10068 3210 Acked Read 4 Done Sent Done false
10069 3211 Acked Write 3 Done Sent Done false
10070 3212 Acked Write 4 Done Sent Done false
10071 3213 Acked Flush 0 Done Sent Done false
10072 3214 Acked Write 6 Done Sent Done false
10073 3215 Acked Write 10 Done Sent Done false
10074 3216 Acked Write 7 Done Sent Done false
10075 3217 Acked Read 6 Done Sent Done false
10076 3218 Acked Read 4 Done Sent Done false
10077 3219 Acked Write 4 Done Sent Done false
10078 3220 Acked Read 6 Done Sent Done false
10079 3221 Acked Read 2 Done Sent Done false
10080 3222 Acked Write 8 Done Sent Done false
10081 3223 Acked Flush 0 Done Sent Done false
10082 3224 Acked Read 8 Done Sent Done false
10083 3225 Acked Write 1 Done Sent Done false
10084 3226 Acked Write 8 Done Sent Done false
10085 3227 Acked Read 8 Done Sent Done false
10086 3228 Acked Read 10 Done Sent Done false
10087 3229 Acked Flush 0 Done Sent Done false
10088 3230 Acked Read 8 Done Sent Done false
10089 3231 Acked Read 5 Done Sent Done false
10090 3232 Acked Read 8 Done Sent Done false
10091 3233 Acked Read 10 Done Sent Done false
10092 3234 Acked Write 9 Done Sent Done false
10093 3235 Acked Write 4 Done Sent Done false
10094 3236 Acked Flush 0 Done Sent Done false
10095 3237 Acked Write 4 Done Sent Done false
10096 3238 Acked Read 7 Done Sent Done false
10097 3239 Acked Read 5 Done Sent Done false
10098 3240 Acked Read 1 Done Sent Done false
10099 3241 Acked Write 8 Done Sent Done false
10100 3242 Acked Read 4 Done Sent Done false
10101 3243 Acked Read 2 Done Sent Done false
10102 3244 Acked Read 3 Done Sent Done false
10103 3245 Acked Read 7 Done Sent Done false
10104 3246 Acked Read 1 Done Sent Done false
10105 3247 Acked Read 9 Done Sent Done false
10106 3248 Acked Read 5 Done Sent Done false
10107 3249 Acked Read 10 Done Sent Done false
10108 3250 Acked Flush 0 Done Sent Done false
10109 3251 Acked Write 8 Done Sent Done false
10110 3252 Acked Write 8 Done Sent Done false
10111 3253 Acked Write 3 Done Sent Done false
10112 3254 Acked Write 10 Done Sent Done false
10113 3255 Acked Write 1 Done Sent Done false
10114 3256 Acked Write 3 Done Sent Done false
10115 3257 Acked Read 2 Done Sent Done false
10116 3258 Acked Read 4 Done Sent Done false
10117 3259 Acked Read 2 Done Sent Done false
10118 3260 Acked Read 7 Done Sent Done false
10119 3261 Acked Read 7 Done Sent Done false
10120 3262 Acked Read 1 Done Sent Done false
10121 3263 Acked Read 10 Done Sent Done false
10122 3264 Acked Write 6 Done Sent Done false
10123 3265 Acked Write 5 Done Sent Done false
10124 3266 Acked Read 7 Done Sent Done false
10125 3267 Acked Write 7 Done Sent Done false
10126 3268 Acked Read 3 Done Sent Done false
10127 3269 Acked Flush 0 Done Sent Done false
10128 3270 Acked Read 10 Done Sent Done false
10129 3271 Acked Flush 0 Done Sent Done false
10130 3272 Acked Write 9 Done Sent Done false
10131 3273 Acked Write 2 Done Sent Done false
10132 3274 Acked Write 6 Done Sent Done false
10133 3275 Acked Flush 0 Done Sent Done false
10134 3276 Acked Read 2 Done Sent Done false
10135 3277 Acked Flush 0 Done Sent Done false
10136 3278 Acked Flush 0 Done Sent Done false
10137 3279 Acked Read 2 Done Sent Done false
10138 3280 Acked Write 1 Done Sent Done false
10139 3281 Acked Read 8 Done Sent Done false
10140 3282 Acked Read 1 Done Sent Done false
10141 3283 Acked Read 5 Done Sent Done false
10142 3284 Acked Read 10 Done Sent Done false
10143 3285 Acked Read 4 Done Sent Done false
10144 3286 Acked Flush 0 Done Sent Done false
10145 3287 Acked Read 3 Done Sent Done false
10146 3288 Acked Write 2 Done Sent Done false
10147 3289 Acked Write 9 Done Sent Done false
10148 3290 Acked Write 6 Done Sent Done false
10149 3291 Acked Write 1 Done Sent Done false
10150 3292 Acked Read 10 Done Sent Done false
10151 3293 Acked Read 2 Done Sent Done false
10152 3294 Acked Write 9 Done Sent Done false
10153 3295 Acked Write 7 Done Sent Done false
10154 3296 Acked Write 1 Done Sent Done false
10155 3297 Acked Read 7 Done Sent Done false
10156 3298 Acked Read 7 Done Sent Done false
10157 3299 Acked Write 2 Done Sent Done false
10158 3300 Acked Read 4 Done Sent Done false
10159 3301 Acked Read 9 Done Sent Done false
10160 3302 Acked Read 2 Done Sent Done false
10161 3303 Acked Read 2 Done Sent Done false
10162 3304 Acked Write 9 Done Sent Done false
10163 3305 Acked Read 9 Done Sent Done false
10164 3306 Acked Read 7 Done Sent Done false
10165 3307 Acked Read 3 Done Sent Done false
10166 3308 Acked Read 7 Done Sent Done false
10167 3309 Acked Read 3 Done Sent Done false
10168 3310 Acked Write 9 Done Sent Done false
10169 3311 Acked Read 7 Done Sent Done false
10170 3312 Acked Write 8 Done Sent Done false
10171 3313 Acked Read 1 Done Sent Done false
10172 3314 Acked Read 6 Done Sent Done false
10173 3315 Acked Read 6 Done Sent Done false
10174 3316 Acked Write 9 Done Sent Done false
10175 3317 Acked Flush 0 Done Sent Done false
10176 3318 Acked Read 4 Done Sent Done false
10177 3319 Acked Write 1 Done Sent Done false
10178 3320 Acked Read 6 Done Sent Done false
10179 3321 Acked Read 7 Done Sent Done false
10180 3322 Acked Read 1 Done Sent Done false
10181 3323 Acked Write 1 Done Sent Done false
10182 3324 Acked Write 7 Done Sent Done false
10183 3325 Acked Write 2 Done Sent Done false
10184 3326 Acked Write 3 Done Sent Done false
10185 3327 Acked Read 10 Done Sent Done false
10186 3328 Acked Write 2 Done Sent Done false
10187 3329 Acked Write 7 Done Sent Done false
10188 3330 Acked Read 7 Done Sent Done false
10189 3331 Acked Read 3 Done Sent Done false
10190 3332 Acked Read 4 Done Sent Done false
10191 3333 Acked Read 10 Done Sent Done false
10192 3334 Acked Read 6 Done Sent Done false
10193 3335 Acked Read 3 Done Sent Done false
10194 3336 Acked Read 4 Done Sent Done false
10195 3337 Acked Read 8 Done Sent Done false
10196 3338 Acked Read 9 Done Sent Done false
10197 3339 Acked Flush 0 Done Sent Done false
10198 3340 Acked Flush 0 Done Sent Done false
10199 3341 Acked Write 4 Done Sent Done false
10200 3342 Acked Read 5 Done Sent Done false
10201 3343 Acked Flush 0 Done Sent Done false
10202 3344 Acked Read 10 Done Sent Done false
10203 3345 Acked Write 10 Done Sent Done false
10204 3346 Acked Write 10 Done Sent Done false
10205 3347 Acked Read 1 Done Sent Done false
10206 3348 Acked Read 9 Done Sent Done false
10207 3349 Acked Read 3 Done Sent Done false
10208 3350 Acked Write 1 Done Sent Done false
10209 3351 Acked Read 9 Done Sent Done false
10210 3352 Acked Write 2 Done Sent Done false
10211 3353 Acked Flush 0 Done Sent Done false
10212 3354 Acked Write 5 Done Sent Done false
10213 3355 Acked Read 3 Done Sent Done false
10214 3356 Acked Write 8 Done Sent Done false
10215 3357 Acked Flush 0 Done Sent Done false
10216 3358 Acked Read 1 Done Sent Done false
10217 3359 Acked Write 4 Done Sent Done false
10218 3360 Acked Read 9 Done Sent Done false
10219 3361 Acked Read 2 Done Sent Done false
10220 3362 Acked Write 10 Done Sent Done false
10221 3363 Acked Read 7 Done Sent Done false
10222 3364 Acked Write 8 Done Sent Done false
10223 3365 Acked Read 6 Done Sent Done false
10224 3366 Acked Write 6 Done Sent Done false
10225 3367 Acked Write 10 Done Sent Done false
10226 3368 Acked Write 3 Done Sent Done false
10227 3369 Acked Read 6 Done Sent Done false
10228 3370 Acked Write 10 Done Sent Done false
10229 3371 Acked Write 7 Done Sent Done false
10230 3372 Acked Read 8 Done Sent Done false
10231 3373 Acked Read 4 Done Sent Done false
10232 3374 Acked Write 4 Done Sent Done false
10233 3375 Acked Read 8 Done Sent Done false
10234 3376 Acked Read 1 Done Sent Done false
10235 3377 Acked Read 7 Done Sent Done false
10236 3378 Acked Write 3 Done Sent Done false
10237 3379 Acked Write 5 Done Sent Done false
10238 3380 Acked Write 7 Done Sent Done false
10239 3381 Acked Flush 0 Done Sent Done false
10240 3382 Acked Read 1 Done Sent Done false
10241 3383 Acked Read 8 Done Sent Done false
10242 3384 Acked Read 1 Done Sent Done false
10243 3385 Acked Read 4 Done Sent Done false
10244 3386 Acked Read 7 Done Sent Done false
10245 3387 Acked Write 10 Done Sent Done false
10246 3388 Acked Read 10 Done Sent Done false
10247 3389 Acked Read 10 Done Sent Done false
10248 3390 Acked Read 4 Done Sent Done false
10249 3391 Acked Write 8 Done Sent Done false
10250 3392 Acked Flush 0 Done Sent Done false
10251 3393 Acked Write 9 Done Sent Done false
10252 3394 Acked Read 5 Done Sent Done false
10253 3395 Acked Write 9 Done Sent Done false
10254 3396 Acked Write 5 Done Sent Done false
10255 3397 Acked Flush 0 Done Sent Done false
10256 3398 Acked Write 6 Done Sent Done false
10257 3399 Acked Read 7 Done Sent Done false
10258 3400 Acked Flush 0 Done Sent Done false
10259 3401 Acked Write 8 Done Sent Done false
10260 3402 Acked Write 4 Done Sent Done false
10261 3403 Acked Read 7 Done Sent Done false
10262 3404 Acked Read 5 Done Sent Done false
10263 3405 Acked Write 5 Done Sent Done false
10264 3406 Acked Read 10 Done Sent Done false
10265 3407 Acked Read 9 Done Sent Done false
10266 3408 Acked Read 4 Done Sent Done false
10267 3409 Acked Read 4 Done Sent Done false
10268 3410 Acked Read 3 Done Sent Done false
10269 3411 Acked Read 1 Done Sent Done false
10270 3412 Acked Write 10 Done Sent Sent false
10271 STATES DS:0 DS:1 DS:2 TOTAL
10272 Sent 0 301 1 302
10273 Done 301 0 300 601
10274 Skipped 0 0 0 0
10275 Error 0 0 0 0
10276 Last Flush: Some(JobId(3400)) Some(JobId(3111)) Some(JobId(3400))
10277 Downstairs last five completed: 3111 3110 3109 3108 3107
10278 Upstairs last five completed: 3412 3411 3410 3409 3408
10279 CLIENT: Up:0 ds:301 act:2
102802025-03-07T18:14:47.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
102812025-03-07T18:14:48.168ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
102822025-03-07T18:14:49.169ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
10283 [8820] got start action
10284 [8820] I am going to restart
10285 Starting downstairs at port 8820
10286 Make output file at "/tmp/dsc/downstairs-8820.txt"
10287 [8820][1] reports Starting
10288 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1216)
10289 [8820][1] reports Running
102902025-03-07T18:14:50.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
10291 ----------------------------------------------------------------
10292 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10293 States: Active Offline Active
10294 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10295 3112 Acked Read 10 Done Sent Done false
10296 3113 Acked Read 10 Done Sent Done false
10297 3114 Acked Read 7 Done Sent Done false
10298 3115 Acked Read 1 Done Sent Done false
10299 3116 Acked Write 4 Done Sent Done false
10300 3117 Acked Read 1 Done Sent Done false
10301 3118 Acked Write 10 Done Sent Done false
10302 3119 Acked Read 5 Done Sent Done false
10303 3120 Acked Write 5 Done Sent Done false
10304 3121 Acked Write 7 Done Sent Done false
10305 3122 Acked Write 1 Done Sent Done false
10306 3123 Acked Read 3 Done Sent Done false
10307 3124 Acked Read 9 Done Sent Done false
10308 3125 Acked Write 9 Done Sent Done false
10309 3126 Acked Read 3 Done Sent Done false
10310 3127 Acked Read 9 Done Sent Done false
10311 3128 Acked Write 8 Done Sent Done false
10312 3129 Acked Read 8 Done Sent Done false
10313 3130 Acked Read 6 Done Sent Done false
10314 3131 Acked Read 6 Done Sent Done false
10315 3132 Acked Read 5 Done Sent Done false
10316 3133 Acked Read 9 Done Sent Done false
10317 3134 Acked Write 7 Done Sent Done false
10318 3135 Acked Write 5 Done Sent Done false
10319 3136 Acked Read 10 Done Sent Done false
10320 3137 Acked Read 1 Done Sent Done false
10321 3138 Acked Flush 0 Done Sent Done false
10322 3139 Acked Read 4 Done Sent Done false
10323 3140 Acked Write 1 Done Sent Done false
10324 3141 Acked Write 5 Done Sent Done false
10325 3142 Acked Read 8 Done Sent Done false
10326 3143 Acked Flush 0 Done Sent Done false
10327 3144 Acked Flush 0 Done Sent Done false
10328 3145 Acked Read 10 Done Sent Done false
10329 3146 Acked Write 2 Done Sent Done false
10330 3147 Acked Read 1 Done Sent Done false
10331 3148 Acked Write 3 Done Sent Done false
10332 3149 Acked Read 4 Done Sent Done false
10333 3150 Acked Write 4 Done Sent Done false
10334 3151 Acked Write 2 Done Sent Done false
10335 3152 Acked Write 6 Done Sent Done false
10336 3153 Acked Write 4 Done Sent Done false
10337 3154 Acked Read 3 Done Sent Done false
10338 3155 Acked Write 1 Done Sent Done false
10339 3156 Acked Flush 0 Done Sent Done false
10340 3157 Acked Write 1 Done Sent Done false
10341 3158 Acked Flush 0 Done Sent Done false
10342 3159 Acked Read 10 Done Sent Done false
10343 3160 Acked Read 3 Done Sent Done false
10344 3161 Acked Write 3 Done Sent Done false
10345 3162 Acked Read 8 Done Sent Done false
10346 3163 Acked Read 1 Done Sent Done false
10347 3164 Acked Write 1 Done Sent Done false
10348 3165 Acked Write 6 Done Sent Done false
10349 3166 Acked Read 6 Done Sent Done false
10350 3167 Acked Read 10 Done Sent Done false
10351 3168 Acked Write 3 Done Sent Done false
10352 3169 Acked Read 2 Done Sent Done false
10353 3170 Acked Flush 0 Done Sent Done false
10354 3171 Acked Read 8 Done Sent Done false
10355 3172 Acked Read 9 Done Sent Done false
10356 3173 Acked Read 9 Done Sent Done false
10357 3174 Acked Read 5 Done Sent Done false
10358 3175 Acked Write 10 Done Sent Done false
10359 3176 Acked Read 8 Done Sent Done false
10360 3177 Acked Read 6 Done Sent Done false
10361 3178 Acked Read 5 Done Sent Done false
10362 3179 Acked Write 1 Done Sent Done false
10363 3180 Acked Read 2 Done Sent Done false
10364 3181 Acked Read 6 Done Sent Done false
10365 3182 Acked Write 4 Done Sent Done false
10366 3183 Acked Read 4 Done Sent Done false
10367 3184 Acked Write 10 Done Sent Done false
10368 3185 Acked Read 7 Done Sent Done false
10369 3186 Acked Write 7 Done Sent Done false
10370 3187 Acked Read 5 Done Sent Done false
10371 3188 Acked Flush 0 Done Sent Done false
10372 3189 Acked Write 9 Done Sent Done false
10373 3190 Acked Read 7 Done Sent Done false
10374 3191 Acked Write 4 Done Sent Done false
10375 3192 Acked Write 4 Done Sent Done false
10376 3193 Acked Write 1 Done Sent Done false
10377 3194 Acked Write 2 Done Sent Done false
10378 3195 Acked Read 5 Done Sent Done false
10379 3196 Acked Read 5 Done Sent Done false
10380 3197 Acked Flush 0 Done Sent Done false
10381 3198 Acked Read 2 Done Sent Done false
10382 3199 Acked Read 7 Done Sent Done false
10383 3200 Acked Flush 0 Done Sent Done false
10384 3201 Acked Write 4 Done Sent Done false
10385 3202 Acked Write 2 Done Sent Done false
10386 3203 Acked Write 7 Done Sent Done false
10387 3204 Acked Read 10 Done Sent Done false
10388 3205 Acked Flush 0 Done Sent Done false
10389 3206 Acked Write 10 Done Sent Done false
10390 3207 Acked Read 7 Done Sent Done false
10391 3208 Acked Read 3 Done Sent Done false
10392 3209 Acked Flush 0 Done Sent Done false
10393 3210 Acked Read 4 Done Sent Done false
10394 3211 Acked Write 3 Done Sent Done false
10395 3212 Acked Write 4 Done Sent Done false
10396 3213 Acked Flush 0 Done Sent Done false
10397 3214 Acked Write 6 Done Sent Done false
10398 3215 Acked Write 10 Done Sent Done false
10399 3216 Acked Write 7 Done Sent Done false
10400 3217 Acked Read 6 Done Sent Done false
10401 3218 Acked Read 4 Done Sent Done false
10402 3219 Acked Write 4 Done Sent Done false
10403 3220 Acked Read 6 Done Sent Done false
10404 3221 Acked Read 2 Done Sent Done false
10405 3222 Acked Write 8 Done Sent Done false
10406 3223 Acked Flush 0 Done Sent Done false
10407 3224 Acked Read 8 Done Sent Done false
10408 3225 Acked Write 1 Done Sent Done false
10409 3226 Acked Write 8 Done Sent Done false
10410 3227 Acked Read 8 Done Sent Done false
10411 3228 Acked Read 10 Done Sent Done false
10412 3229 Acked Flush 0 Done Sent Done false
10413 3230 Acked Read 8 Done Sent Done false
10414 3231 Acked Read 5 Done Sent Done false
10415 3232 Acked Read 8 Done Sent Done false
10416 3233 Acked Read 10 Done Sent Done false
10417 3234 Acked Write 9 Done Sent Done false
10418 3235 Acked Write 4 Done Sent Done false
10419 3236 Acked Flush 0 Done Sent Done false
10420 3237 Acked Write 4 Done Sent Done false
10421 3238 Acked Read 7 Done Sent Done false
10422 3239 Acked Read 5 Done Sent Done false
10423 3240 Acked Read 1 Done Sent Done false
10424 3241 Acked Write 8 Done Sent Done false
10425 3242 Acked Read 4 Done Sent Done false
10426 3243 Acked Read 2 Done Sent Done false
10427 3244 Acked Read 3 Done Sent Done false
10428 3245 Acked Read 7 Done Sent Done false
10429 3246 Acked Read 1 Done Sent Done false
10430 3247 Acked Read 9 Done Sent Done false
10431 3248 Acked Read 5 Done Sent Done false
10432 3249 Acked Read 10 Done Sent Done false
10433 3250 Acked Flush 0 Done Sent Done false
10434 3251 Acked Write 8 Done Sent Done false
10435 3252 Acked Write 8 Done Sent Done false
10436 3253 Acked Write 3 Done Sent Done false
10437 3254 Acked Write 10 Done Sent Done false
10438 3255 Acked Write 1 Done Sent Done false
10439 3256 Acked Write 3 Done Sent Done false
10440 3257 Acked Read 2 Done Sent Done false
10441 3258 Acked Read 4 Done Sent Done false
10442 3259 Acked Read 2 Done Sent Done false
10443 3260 Acked Read 7 Done Sent Done false
10444 3261 Acked Read 7 Done Sent Done false
10445 3262 Acked Read 1 Done Sent Done false
10446 3263 Acked Read 10 Done Sent Done false
10447 3264 Acked Write 6 Done Sent Done false
10448 3265 Acked Write 5 Done Sent Done false
10449 3266 Acked Read 7 Done Sent Done false
10450 3267 Acked Write 7 Done Sent Done false
10451 3268 Acked Read 3 Done Sent Done false
10452 3269 Acked Flush 0 Done Sent Done false
10453 3270 Acked Read 10 Done Sent Done false
10454 3271 Acked Flush 0 Done Sent Done false
10455 3272 Acked Write 9 Done Sent Done false
10456 3273 Acked Write 2 Done Sent Done false
10457 3274 Acked Write 6 Done Sent Done false
10458 3275 Acked Flush 0 Done Sent Done false
10459 3276 Acked Read 2 Done Sent Done false
10460 3277 Acked Flush 0 Done Sent Done false
10461 3278 Acked Flush 0 Done Sent Done false
10462 3279 Acked Read 2 Done Sent Done false
10463 3280 Acked Write 1 Done Sent Done false
10464 3281 Acked Read 8 Done Sent Done false
10465 3282 Acked Read 1 Done Sent Done false
10466 3283 Acked Read 5 Done Sent Done false
10467 3284 Acked Read 10 Done Sent Done false
10468 3285 Acked Read 4 Done Sent Done false
10469 3286 Acked Flush 0 Done Sent Done false
10470 3287 Acked Read 3 Done Sent Done false
10471 3288 Acked Write 2 Done Sent Done false
10472 3289 Acked Write 9 Done Sent Done false
10473 3290 Acked Write 6 Done Sent Done false
10474 3291 Acked Write 1 Done Sent Done false
10475 3292 Acked Read 10 Done Sent Done false
10476 3293 Acked Read 2 Done Sent Done false
10477 3294 Acked Write 9 Done Sent Done false
10478 3295 Acked Write 7 Done Sent Done false
10479 3296 Acked Write 1 Done Sent Done false
10480 3297 Acked Read 7 Done Sent Done false
10481 3298 Acked Read 7 Done Sent Done false
10482 3299 Acked Write 2 Done Sent Done false
10483 3300 Acked Read 4 Done Sent Done false
10484 3301 Acked Read 9 Done Sent Done false
10485 3302 Acked Read 2 Done Sent Done false
10486 3303 Acked Read 2 Done Sent Done false
10487 3304 Acked Write 9 Done Sent Done false
10488 3305 Acked Read 9 Done Sent Done false
10489 3306 Acked Read 7 Done Sent Done false
10490 3307 Acked Read 3 Done Sent Done false
10491 3308 Acked Read 7 Done Sent Done false
10492 3309 Acked Read 3 Done Sent Done false
10493 3310 Acked Write 9 Done Sent Done false
10494 3311 Acked Read 7 Done Sent Done false
10495 3312 Acked Write 8 Done Sent Done false
10496 3313 Acked Read 1 Done Sent Done false
10497 3314 Acked Read 6 Done Sent Done false
10498 3315 Acked Read 6 Done Sent Done false
10499 3316 Acked Write 9 Done Sent Done false
10500 3317 Acked Flush 0 Done Sent Done false
10501 3318 Acked Read 4 Done Sent Done false
10502 3319 Acked Write 1 Done Sent Done false
10503 3320 Acked Read 6 Done Sent Done false
10504 3321 Acked Read 7 Done Sent Done false
10505 3322 Acked Read 1 Done Sent Done false
10506 3323 Acked Write 1 Done Sent Done false
10507 3324 Acked Write 7 Done Sent Done false
10508 3325 Acked Write 2 Done Sent Done false
10509 3326 Acked Write 3 Done Sent Done false
10510 3327 Acked Read 10 Done Sent Done false
10511 3328 Acked Write 2 Done Sent Done false
10512 3329 Acked Write 7 Done Sent Done false
10513 3330 Acked Read 7 Done Sent Done false
10514 3331 Acked Read 3 Done Sent Done false
10515 3332 Acked Read 4 Done Sent Done false
10516 3333 Acked Read 10 Done Sent Done false
10517 3334 Acked Read 6 Done Sent Done false
10518 3335 Acked Read 3 Done Sent Done false
10519 3336 Acked Read 4 Done Sent Done false
10520 3337 Acked Read 8 Done Sent Done false
10521 3338 Acked Read 9 Done Sent Done false
10522 3339 Acked Flush 0 Done Sent Done false
10523 3340 Acked Flush 0 Done Sent Done false
10524 3341 Acked Write 4 Done Sent Done false
10525 3342 Acked Read 5 Done Sent Done false
10526 3343 Acked Flush 0 Done Sent Done false
10527 3344 Acked Read 10 Done Sent Done false
10528 3345 Acked Write 10 Done Sent Done false
10529 3346 Acked Write 10 Done Sent Done false
10530 3347 Acked Read 1 Done Sent Done false
10531 3348 Acked Read 9 Done Sent Done false
10532 3349 Acked Read 3 Done Sent Done false
10533 3350 Acked Write 1 Done Sent Done false
10534 3351 Acked Read 9 Done Sent Done false
10535 3352 Acked Write 2 Done Sent Done false
10536 3353 Acked Flush 0 Done Sent Done false
10537 3354 Acked Write 5 Done Sent Done false
10538 3355 Acked Read 3 Done Sent Done false
10539 3356 Acked Write 8 Done Sent Done false
10540 3357 Acked Flush 0 Done Sent Done false
10541 3358 Acked Read 1 Done Sent Done false
10542 3359 Acked Write 4 Done Sent Done false
10543 3360 Acked Read 9 Done Sent Done false
10544 3361 Acked Read 2 Done Sent Done false
10545 3362 Acked Write 10 Done Sent Done false
10546 3363 Acked Read 7 Done Sent Done false
10547 3364 Acked Write 8 Done Sent Done false
10548 3365 Acked Read 6 Done Sent Done false
10549 3366 Acked Write 6 Done Sent Done false
10550 3367 Acked Write 10 Done Sent Done false
10551 3368 Acked Write 3 Done Sent Done false
10552 3369 Acked Read 6 Done Sent Done false
10553 3370 Acked Write 10 Done Sent Done false
10554 3371 Acked Write 7 Done Sent Done false
10555 3372 Acked Read 8 Done Sent Done false
10556 3373 Acked Read 4 Done Sent Done false
10557 3374 Acked Write 4 Done Sent Done false
10558 3375 Acked Read 8 Done Sent Done false
10559 3376 Acked Read 1 Done Sent Done false
10560 3377 Acked Read 7 Done Sent Done false
10561 3378 Acked Write 3 Done Sent Done false
10562 3379 Acked Write 5 Done Sent Done false
10563 3380 Acked Write 7 Done Sent Done false
10564 3381 Acked Flush 0 Done Sent Done false
10565 3382 Acked Read 1 Done Sent Done false
10566 3383 Acked Read 8 Done Sent Done false
10567 3384 Acked Read 1 Done Sent Done false
10568 3385 Acked Read 4 Done Sent Done false
10569 3386 Acked Read 7 Done Sent Done false
10570 3387 Acked Write 10 Done Sent Done false
10571 3388 Acked Read 10 Done Sent Done false
10572 3389 Acked Read 10 Done Sent Done false
10573 3390 Acked Read 4 Done Sent Done false
10574 3391 Acked Write 8 Done Sent Done false
10575 3392 Acked Flush 0 Done Sent Done false
10576 3393 Acked Write 9 Done Sent Done false
10577 3394 Acked Read 5 Done Sent Done false
10578 3395 Acked Write 9 Done Sent Done false
10579 3396 Acked Write 5 Done Sent Done false
10580 3397 Acked Flush 0 Done Sent Done false
10581 3398 Acked Write 6 Done Sent Done false
10582 3399 Acked Read 7 Done Sent Done false
10583 3400 Acked Flush 0 Done Sent Done false
10584 3401 Acked Write 8 Done Sent Done false
10585 3402 Acked Write 4 Done Sent Done false
10586 3403 Acked Read 7 Done Sent Done false
10587 3404 Acked Read 5 Done Sent Done false
10588 3405 Acked Write 5 Done Sent Done false
10589 3406 Acked Read 10 Done Sent Done false
10590 3407 Acked Read 9 Done Sent Done false
10591 3408 Acked Read 4 Done Sent Done false
10592 3409 Acked Read 4 Done Sent Done false
10593 3410 Acked Read 3 Done Sent Done false
10594 3411 Acked Read 1 Done Sent Done false
10595 3412 Acked Write 10 Done Sent Done false
10596 STATES DS:0 DS:1 DS:2 TOTAL
10597 Sent 0 301 0 301
10598 Done 301 0 301 602
10599 Skipped 0 0 0 0
10600 Error 0 0 0 0
10601 Last Flush: Some(JobId(3400)) Some(JobId(3111)) Some(JobId(3400))
10602 Downstairs last five completed: 3111 3110 3109 3108 3107
10603 Upstairs last five completed: 3412 3411 3410 3409 3408
10604 CLIENT: Up:0 ds:301 act:2
106052025-03-07T18:14:51.173ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106062025-03-07T18:14:52.269ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106072025-03-07T18:14:52.290ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106082025-03-07T18:14:52.290ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65021) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106092025-03-07T18:14:52.291ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106102025-03-07T18:14:52.292ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106112025-03-07T18:14:52.292ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106122025-03-07T18:14:52.292ZINFOcrucible: send last flush ID to this DS: Some(JobId(3111)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106132025-03-07T18:14:52.292ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106142025-03-07T18:14:52.292ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3111)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106152025-03-07T18:14:52.292ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3111)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
10616 ----------------------------------------------------------------
10617 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10618 Downstairs last five completed: 3413 3412 3411 3410 3409
10619 Upstairs last five completed: 3413 3412 3411 3410 3409
10620 CLIENT: Up:0 ds:0 act:3
10621 Replay: All jobs finished
10622 Main task has work to do, go find it
10623 got dsc Stop(0)
10624 stop 0
10625 Mar 07 18:14:54.547 INFO request completed, [8810] Got stop action so:false kr:false
10626 latency_us: 322, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7832ceb6-3d83-4ee3-ac63-24e6be663bc4, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
106272025-03-07T18:14:54.547ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106282025-03-07T18:14:54.548ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106292025-03-07T18:14:54.548ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
106302025-03-07T18:14:54.548ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
10631 Mar 07 18:14:54.548 INFO request completed, latency_us: 261, response_code: 200, uri: /state/cid/0, method: GET, req_id: 447a9907-dffd-42e5-b551-5fbc73397b3e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
10632 [8810] Exited with: ExitStatus(unix_wait_status(9))
10633 [8810][0] reports Exit
10634 Mar 07 18:14:58.551 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4787e33e-9c6b-4507-92d3-e2bc15558e63, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
10635 001/300 Read block 2086 len 36864
10636 002/300 Write block 4248 len 32768 data: 4 4 4 4 4 4 4 4
10637 003/300 Read block 2130 len 24576
10638 004/300 Read block 1418 len 40960
10639 005/300 Read block 2650 len 36864
10640 006/300 Read block 3649 len 16384
10641 007/300 Flush
10642 008/300 Read block 4317 len 20480
10643 009/300 Read block 2394 len 16384
10644 010/300 Flush
10645 011/300 Flush
10646 012/300 Read block 3030 len 20480
10647 013/300 Write block 3947 len 28672 data: 3 3 3 2 2 2 2
10648 014/300 Write block 2534 len 36864 data: 4 4 4 3 3 4 4 4 4
10649 015/300 Write block 3733 len 8192 data: 2 2
10650 016/300 Read block 4203 len 32768
10651 017/300 Write block 4576 len 4096 data: 2
10652 018/300 Read block 905 len 24576
10653 019/300 Write block 4871 len 20480 data: 3 3 3 4 4
10654 020/300 Read block 64 len 28672
10655 021/300 Write block 3849 len 20480 data: 3 3 3 3 4
10656 022/300 Write block 3385 len 20480 data: 3 3 3 3 3
10657 023/300 Write block 1153 len 36864 data: 3 3 3 2 2 2 2 3 4
10658 024/300 Read block 4054 len 4096
10659 025/300 Read block 3384 len 28672
10660 026/300 Read block 512 len 8192
10661 027/300 Write block 2474 len 36864 data: 2 2 2 2 3 3 3 3 4
10662 028/300 Write block 696 len 28672 data: 3 3 3 3 3 2 2
10663 029/300 Write block 933 len 28672 data: 2 2 2 2 2 3 3
10664 030/300 Write block 4794 len 36864 data: 4 3 3 3 3 3 3 3 2
10665 031/300 Write block 2295 len 40960 data: 2 2 2 2 2 2 2 2 3 2
10666 032/300 Write block 4305 len 12288 data: 3 3 3
10667 033/300 Flush
10668 034/300 Write block 1209 len 32768 data: 5 5 5 4 3 3 2 2
10669 035/300 Write block 878 len 20480 data: 2 2 2 3 2
10670 036/300 Read block 3850 len 8192
10671 037/300 Write block 3131 len 20480 data: 2 4 4 4 3
10672 038/300 Write block 1244 len 32768 data: 3 3 3 2 2 2 2 3
10673 039/300 Write block 21 len 24576 data: 4 4 4 3 3 3
10674 040/300 Read block 1425 len 28672
10675 041/300 Write block 730 len 8192 data: 2 3
10676 042/300 Flush
10677 043/300 Write block 1450 len 32768 data: 3 3 3 3 3 3 2 2
10678 044/300 Read block 1642 len 28672
10679 045/300 Read block 60 len 8192
10680 046/300 Read block 513 len 20480
10681 047/300 Write block 1111 len 20480 data: 2 2 3 2 3
10682 048/300 Read block 3499 len 16384
10683 049/300 Write block 3645 len 20480 data: 3 3 3 3 3
10684 050/300 Read block 3177 len 8192
10685 051/300 Read block 677 len 4096
10686 052/300 Flush
10687 053/300 Write block 2119 len 20480 data: 3 3 3 2 2
10688 054/300 Read block 2152 len 24576
10689 055/300 Write block 1163 len 28672 data: 3 3 3 3 3 3 3
10690 056/300 Write block 3149 len 28672 data: 3 3 3 3 2 2 2
10691 057/300 Write block 417 len 28672 data: 3 3 3 2 2 2 4
10692 058/300 Write block 3989 len 28672 data: 3 4 4 4 3 3 3
10693 059/300 Write block 3125 len 32768 data: 2 2 2 2 2 2 3 5
10694 060/300 Flush
10695 061/300 Write block 1017 len 36864 data: 3 2 2 2 3 3 3 3 3
10696 062/300 Write block 4122 len 40960 data: 3 3 3 4 4 4 4 4 3 3
10697 063/300 Write block 4370 len 40960 data: 3 4 4 5 4 4 4 5 3 3
10698 064/300 Flush
10699 065/300 Read block 4212 len 4096
10700 066/300 Read block 1387 len 4096
10701 067/300 Read block 2443 len 20480
10702 068/300 Read block 3910 len 20480
10703 069/300 Write block 4694 len 32768 data: 4 5 5 5 5 5 4 4
10704 070/300 Write block 84 len 4096 data: 2
10705 071/300 Flush
10706 072/300 Write block 1361 len 8192 data: 4 4
10707 073/300 Write block 1597 len 4096 data: 4
10708 074/300 Read block 4056 len 36864
10709 075/300 Read block 4987 len 4096
10710 076/300 Write block 1181 len 4096 data: 3
10711 077/300 Write block 3654 len 24576 data: 3 3 3 3 3 3
10712 078/300 Read block 4209 len 24576
10713 079/300 Write block 1411 len 24576 data: 4 4 4 3 2 2
10714 080/300 Write block 1187 len 12288 data: 3 2 3
10715 081/300 Read block 3015 len 4096
10716 082/300 Write block 1802 len 12288 data: 3 3 3
10717 083/300 Read block 1205 len 12288
10718 084/300 Flush
10719 085/300 Read block 4398 len 16384
10720 086/300 Write block 1703 len 8192 data: 6 6
10721 087/300 Read block 1586 len 20480
10722 088/300 Write block 1634 len 28672 data: 2 2 2 2 2 2 2
10723 089/300 Write block 1587 len 16384 data: 3 3 3 4
10724 090/300 Write block 3165 len 12288 data: 2 2 2
10725 091/300 Flush
10726 092/300 Write block 3140 len 32768 data: 2 2 2 3 3 3 3 4
10727 093/300 Read block 3738 len 4096
10728 094/300 Write block 3063 len 36864 data: 6 5 5 5 5 5 5 4 5
10729 095/300 Write block 1487 len 20480 data: 4 3 2 2 2
10730 096/300 Read block 2208 len 4096
10731 097/300 Read block 1467 len 8192
10732 098/300 Read block 2828 len 16384
10733 099/300 Read block 2566 len 36864
10734 100/300 Write block 2918 len 36864 data: 3 2 3 5 5 5 6 5 4
10735 101/300 Read block 3154 len 16384
10736 102/300 Flush
10737 103/300 Read block 468 len 28672
10738 104/300 Write block 3520 len 40960 data: 2 3 3 4 4 4 3 3 3 3
10739 105/300 Write block 1290 len 12288 data: 2 3 3
10740 106/300 Write block 594 len 40960 data: 4 4 4 4 4 3 2 3 4 4
10741 107/300 Flush
10742 108/300 Write block 2816 len 8192 data: 3 3
10743 109/300 Write block 3729 len 16384 data: 2 2 2 2
10744 110/300 Read block 4766 len 16384
10745 111/300 Read block 2297 len 20480
10746 112/300 Write block 3305 len 36864 data: 2 2 2 2 2 2 2 2 2
10747 113/300 Read block 4583 len 32768
10748 114/300 Read block 492 len 28672
10749 115/300 Write block 2658 len 4096 data: 4
10750 116/300 Write block 623 len 32768 data: 5 5 5 5 5 3 3 3
10751 117/300 Read block 1933 len 24576
10752 118/300 Read block 437 len 20480
10753 119/300 Read block 1153 len 40960
10754 120/300 Read block 4826 len 12288
10755 121/300 Write block 933 len 4096 data: 3
10756 122/300 Read block 2940 len 8192
10757 123/300 Read block 1922 len 20480
10758 124/300 Read block 3845 len 40960
10759 125/300 Read block 3321 len 40960
10760 126/300 Write block 1119 len 20480 data: 3 3 4 4 3
10761 127/300 Read block 716 len 24576
10762 128/300 Write block 1491 len 32768 data: 3 2 2 2 3 3 3 3
10763 129/300 Read block 4076 len 20480
10764 130/300 Write block 4196 len 8192 data: 2 2
10765 131/300 Write block 1263 len 24576 data: 2 2 2 2 3 3
10766 132/300 Read block 2018 len 12288
10767 133/300 Write block 4919 len 32768 data: 2 2 2 2 2 2 2 2
10768 134/300 Read block 1926 len 20480
10769 135/300 Write block 3320 len 36864 data: 2 2 3 3 3 3 4 4 4
10770 136/300 Read block 265 len 12288
10771 137/300 Write block 459 len 8192 data: 3 2
10772 138/300 Write block 3649 len 4096 data: 4
10773 139/300 Flush
10774 140/300 Write block 4914 len 20480 data: 3 3 3 3 3
10775 141/300 Write block 787 len 20480 data: 3 3 3 3 3
10776 142/300 Write block 1268 len 20480 data: 4 3 2 2 2
10777 143/300 Write block 2619 len 20480 data: 3 3 3 2 2
10778 144/300 Read block 1749 len 16384
10779 145/300 Write block 780 len 20480 data: 4 3 2 2 3
10780 146/300 Write block 3117 len 8192 data: 3 2
10781 147/300 Write block 1104 len 16384 data: 4 3 2 2
10782 148/300 Flush
10783 149/300 Read block 1999 len 32768
10784 150/300 Read block 3540 len 40960
10785 151/300 Read block 4776 len 8192
10786 152/300 Write block 827 len 20480 data: 5 4 4 3 3
10787 153/300 Write block 1944 len 20480 data: 2 2 2 2 2
10788 154/300 Read block 4708 len 12288
10789 155/300 Read block 2237 len 32768
10790 156/300 Write block 601 len 40960 data: 4 5 5 4 4 4 4 2 2 2
10791 157/300 Read block 1877 len 20480
10792 158/300 Read block 786 len 24576
10793 159/300 Write block 3994 len 8192 data: 4 4
10794 160/300 Write block 2304 len 24576 data: 3 3 3 3 3 3
10795 161/300 Read block 4668 len 12288
10796 162/300 Read block 2205 len 12288
10797 163/300 Read block 1618 len 20480
10798 164/300 Write block 3474 len 16384 data: 3 4 4 3
10799 165/300 Read block 3509 len 36864
10800 166/300 Write block 2409 len 24576 data: 4 4 4 3 2 2
10801 167/300 Flush
10802 168/300 Write block 2877 len 40960 data: 3 3 3 3 3 2 3 4 5 5
10803 169/300 Read block 2797 len 36864
10804 170/300 Flush
10805 171/300 Read block 4320 len 28672
10806 172/300 Write block 2283 len 40960 data: 4 4 4 4 4 4 2 2 2 2
10807 173/300 Write block 739 len 40960 data: 4 5 3 3 3 3 3 3 3 3
10808 174/300 Read block 1060 len 20480
10809 175/300 Read block 2954 len 12288
10810 176/300 Read block 3318 len 12288
10811 177/300 Flush
10812 178/300 Flush
10813 179/300 Read block 3087 len 4096
10814 180/300 Write block 3600 len 8192 data: 3 2
10815 181/300 Read block 1487 len 12288
10816 182/300 Write block 193 len 24576 data: 5 4 5 4 4 4
10817 183/300 Read block 3886 len 24576
10818 184/300 Read block 2698 len 12288
10819 185/300 Read block 3664 len 16384
10820 186/300 Write block 956 len 28672 data: 5 5 5 4 4 3 3
10821 187/300 Write block 3964 len 4096 data: 3
10822 188/300 Read block 3269 len 4096
10823 189/300 Write block 2749 len 24576 data: 2 2 2 3 2 3
10824 190/300 Write block 1344 len 12288 data: 2 2 2
10825 191/300 Read block 4430 len 20480
10826 192/300 Read block 4035 len 28672
10827 193/300 Read block 491 len 12288
10828 194/300 Read block 1357 len 28672
10829 195/300 Write block 723 len 40960 data: 2 2 2 3 3 3 3 3 4 3
10830 196/300 Read block 2639 len 20480
10831 197/300 Read block 1734 len 28672
10832 198/300 Write block 3699 len 8192 data: 4 4
10833 199/300 Write block 809 len 36864 data: 3 2 2 2 2 2 2 2 4
10834 200/300 Write block 4496 len 12288 data: 2 2 2
10835 201/300 Write block 526 len 32768 data: 3 3 3 3 3 2 2 2
10836 202/300 Read block 3240 len 8192
10837 203/300 Flush
10838 204/300 Write block 1531 len 20480 data: 3 3 3 3 3
10839 205/300 Read block 532 len 40960
10840 206/300 Write block 2807 len 28672 data: 4 4 4 3 3 3 2
10841 207/300 Flush
10842 208/300 Read block 4987 len 16384
10843 209/300 Write block 1434 len 40960 data: 4 4 3 3 3 3 3 2 2 2
10844 210/300 Write block 731 len 28672 data: 5 4 4 4 5 5 5
10845 211/300 Read block 2754 len 20480
10846 212/300 Flush
10847 213/300 Write block 2052 len 16384 data: 3 2 2 2
10848 214/300 Flush
10849 215/300 Read block 1367 len 12288
10850 216/300 Read block 363 len 24576
10851 217/300 Write block 2129 len 20480 data: 2 2 2 2 2
10852 218/300 Write block 4546 len 40960 data: 7 6 5 5 4 3 3 2 2 3
10853 219/300 Write block 777 len 24576 data: 4 4 4 5 4 3
10854 220/300 Read block 3195 len 40960
10855 221/300 Write block 3865 len 28672 data: 3 2 2 2 2 2 2
10856 222/300 Read block 2130 len 20480
10857 223/300 Write block 4449 len 40960 data: 3 3 4 5 4 5 5 5 5 3
10858 224/300 Write block 3748 len 24576 data: 4 3 4 4 4 2
10859 225/300 Read block 3772 len 8192
10860 226/300 Write block 3988 len 32768 data: 3 4 5 5 5 4 5 5
10861 227/300 Read block 2713 len 24576
10862 228/300 Write block 424 len 28672 data: 4 4 4 3 3 4 4
10863 229/300 Read block 1551 len 36864
10864 230/300 Write block 4036 len 12288 data: 3 3 3
10865 231/300 Flush
10866 232/300 Read block 2815 len 4096
10867 233/300 Write block 4541 len 36864 data: 4 5 6 6 7 8 7 6 6
10868 234/300 Read block 88 len 12288
10869 235/300 Write block 3151 len 36864 data: 4 4 3 3 3 2 2 2 3
10870 236/300 Read block 645 len 4096
10871 237/300 Read block 430 len 40960
10872 238/300 Read block 680 len 36864
10873 239/300 Flush
10874 240/300 Write block 3811 len 8192 data: 4 5
10875 241/300 Write block 2458 len 32768 data: 3 3 3 2 2 4 4 4
10876 242/300 Write block 2436 len 20480 data: 3 4 4 4 4
10877 243/300 Flush
10878 244/300 Write block 3261 len 4096 data: 3
10879 245/300 Write block 1766 len 8192 data: 2 2
10880 246/300 Read block 4176 len 40960
10881 247/300 Read block 1425 len 8192
10882 248/300 Read block 4320 len 32768
10883 249/300 Read block 3043 len 32768
10884 250/300 Write block 4774 len 4096 data: 4
10885 251/300 Write block 206 len 20480 data: 4 3 4 4 4
10886 252/300 Read block 2610 len 4096
10887 253/300 Read block 3320 len 28672
10888 254/300 Read block 3806 len 8192
10889 255/300 Read block 1528 len 4096
10890 256/300 Write block 2016 len 20480 data: 5 4 4 3 2
10891 257/300 Flush
10892 258/300 Write block 4166 len 36864 data: 2 2 2 3 3 3 4 2 2
10893 259/300 Read block 361 len 28672
10894 260/300 Read block 304 len 8192
10895 261/300 Write block 1799 len 32768 data: 2 2 3 4 4 4 4 2
10896 262/300 Write block 1486 len 36864 data: 4 5 4 3 3 4 3 3 3
10897 263/300 Write block 1991 len 20480 data: 2 3 3 3 3
10898 264/300 Read block 4762 len 20480
10899 265/300 Read block 2482 len 36864
10900 266/300 Write block 2716 len 32768 data: 2 2 2 2 2 2 2 2
10901 267/300 Read block 2101 len 12288
10902 268/300 Write block 937 len 40960 data: 3 4 4 2 2 2 2 2 2 2
10903 269/300 Read block 4554 len 16384
10904 270/300 Read block 2996 len 28672
10905 271/300 Write block 471 len 4096 data: 2
10906 272/300 Write block 2900 len 20480 data: 4 5 5 5 5
10907 273/300 Write block 1450 len 16384 data: 4 4 4 4
10908 274/300 Read block 4783 len 12288
10909 275/300 Write block 3471 len 32768 data: 3 3 3 4 5 5 4 2
10910 276/300 Read block 4876 len 28672
10911 277/300 Read block 4092 len 4096
10912 278/300 Write block 93 len 16384 data: 2 2 3 3
10913 279/300 Read block 461 len 12288
10914 280/300 Read block 1776 len 16384
10915 281/300 Read block 3123 len 16384
10916 282/300 Read block 21 len 4096
10917 283/300 Read block 3701 len 32768
10918 284/300 Read block 2673 len 32768
10919 285/300 Read block 2238 len 36864
10920 286/300 Write block 3834 len 12288 data: 3 3 3
10921 287/300 Read block 753 len 8192
10922 288/300 Write block 2930 len 16384 data: 3 3 3 3
10923 289/300 Write block 3297 len 8192 data: 3 3
10924 290/300 Write block 2 len 4096 data: 2
10925 291/300 Read block 2932 len 8192
10926 292/300 Write block 4629 len 28672 data: 4 2 2 2 2 3 3
10927 293/300 Read block 141 len 36864
10928 294/300 Flush
10929 295/300 Read block 4513 len 16384
10930 296/300 Write block 1567 len 40960 data: 6 6 6 5 5 3 2 2 2 3
10931 297/300 Write block 1745 len 32768 data: 4 2 2 3 3 3 3 3
10932 298/300 Write block 3893 len 16384 data: 3 3 3 4
10933 299/300 Write block 2328 len 36864 data: 2 2 2 3 3 3 3 4 4
10934 300/300 Read block 919 len 20480
10935 301/300 Read block 3133 len 24576
10936 Main task has work to do, go find it
10937 got dsc Start(0)
10938 start 0
10939 Mar 07 18:14:58.787 INFO request completed, latency_us: 199, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5725c08f-ca0e-4fe6-8bfb-3ba1afcd4059, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
10940 Replay: started 0, returned:Ok(())
10941 ----------------------------------------------------------------
10942 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10943 States: Offline Active Active
10944 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10945 3414 Acked Read 9 Sent Done Done false
10946 3415 Acked Write 8 Sent Done Done false
10947 3416 Acked Read 6 Sent Done Done false
10948 3417 Acked Read 10 Sent Done Done false
10949 3418 Acked Read 9 Sent Done Done false
10950 3419 Acked Read 4 Sent Done Done false
10951 3420 Acked Flush 0 Sent Done Done false
10952 3421 Acked Read 5 Sent Done Done false
10953 3422 Acked Read 4 Sent Done Done false
10954 3423 Acked Flush 0 Sent Done Done false
10955 3424 Acked Flush 0 Sent Done Done false
10956 3425 Acked Read 5 Sent Done Done false
10957 3426 Acked Write 7 Sent Done Done false
10958 3427 Acked Write 9 Sent Done Done false
10959 3428 Acked Write 2 Sent Done Done false
10960 3429 Acked Read 8 Sent Done Done false
10961 3430 Acked Write 1 Sent Done Done false
10962 3431 Acked Read 6 Sent Done Done false
10963 3432 Acked Write 5 Sent Done Done false
10964 3433 Acked Read 7 Sent Done Done false
10965 3434 Acked Write 5 Sent Done Done false
10966 3435 Acked Write 5 Sent Done Done false
10967 3436 Acked Write 9 Sent Done Done false
10968 3437 Acked Read 1 Sent Done Done false
10969 3438 Acked Read 7 Sent Done Done false
10970 3439 Acked Read 2 Sent Done Done false
10971 3440 Acked Write 9 Sent Done Done false
10972 3441 Acked Write 7 Sent Done Done false
10973 3442 Acked Write 7 Sent Done Done false
10974 3443 Acked Write 9 Sent Done Done false
10975 3444 Acked Write 10 Sent Done Done false
10976 3445 Acked Write 3 Sent Done Done false
10977 3446 Acked Flush 0 Sent Done Done false
10978 3447 Acked Write 8 Sent Done Done false
10979 3448 Acked Write 5 Sent Done Done false
10980 3449 Acked Read 2 Sent Done Done false
10981 3450 Acked Write 5 Sent Done Done false
10982 3451 Acked Write 8 Sent Done Done false
10983 3452 Acked Write 6 Sent Done Done false
10984 3453 Acked Read 7 Sent Done Done false
10985 3454 Acked Write 2 Sent Done Done false
10986 3455 Acked Flush 0 Sent Done Done false
10987 3456 Acked Write 8 Sent Done Done false
10988 3457 Acked Read 7 Sent Done Done false
10989 3458 Acked Read 2 Sent Done Done false
10990 3459 Acked Read 5 Sent Done Done false
10991 3460 Acked Write 5 Sent Done Done false
10992 3461 Acked Read 4 Sent Done Done false
10993 3462 Acked Write 5 Sent Done Done false
10994 3463 Acked Read 2 Sent Done Done false
10995 3464 Acked Read 1 Sent Done Done false
10996 3465 Acked Flush 0 Sent Done Done false
10997 3466 Acked Write 5 Sent Done Done false
10998 3467 Acked Read 6 Sent Done Done false
10999 3468 Acked Write 7 Sent Done Done false
11000 3469 Acked Write 7 Sent Done Done false
11001 3470 Acked Write 7 Sent Done Done false
11002 3471 Acked Write 7 Sent Done Done false
11003 3472 Acked Write 8 Sent Done Done false
11004 3473 Acked Flush 0 Sent Done Done false
11005 3474 Acked Write 9 Sent Done Done false
11006 3475 Acked Write 10 Sent Done Done false
11007 3476 Acked Write 10 Sent Done Done false
11008 3477 Acked Flush 0 Sent Done Done false
11009 3478 Acked Read 1 Sent Done Done false
11010 3479 Acked Read 1 Sent Done Done false
11011 3480 Acked Read 5 Sent Done Done false
11012 3481 Acked Read 5 Sent Done Done false
11013 3482 Acked Write 8 Sent Done Done false
11014 3483 Acked Write 1 Sent Done Done false
11015 3484 Acked Flush 0 Sent Done Done false
11016 3485 Acked Write 2 Sent Done Done false
11017 3486 Acked Write 1 Sent Done Done false
11018 3487 Acked Read 9 Sent Done Done false
11019 3488 Acked Read 1 Sent Done Done false
11020 3489 Acked Write 1 Sent Done Done false
11021 3490 Acked Write 6 Sent Done Done false
11022 3491 Acked Read 6 Sent Done Done false
11023 3492 Acked Write 6 Sent Done Done false
11024 3493 Acked Write 3 Sent Done Done false
11025 3494 Acked Read 1 Sent Done Done false
11026 3495 Acked Write 3 Sent Done Done false
11027 3496 Acked Read 3 Sent Done Done false
11028 3497 Acked Flush 0 Sent Done Done false
11029 3498 Acked Read 4 Sent Done Done false
11030 3499 Acked Write 2 Sent Done Done false
11031 3500 Acked Read 5 Sent Done Done false
11032 3501 Acked Write 7 Sent Done Done false
11033 3502 Acked Write 4 Sent Done Done false
11034 3503 Acked Write 3 Sent Done Done false
11035 3504 Acked Flush 0 Sent Done Done false
11036 3505 Acked Write 8 Sent Done Done false
11037 3506 Acked Read 1 Sent Done Done false
11038 3507 Acked Write 9 Sent Done Done false
11039 3508 Acked Write 5 Sent Done Done false
11040 3509 Acked Read 1 Sent Done Done false
11041 3510 Acked Read 2 Sent Done Done false
11042 3511 Acked Read 4 Sent Done Done false
11043 3512 Acked Read 9 Sent Done Done false
11044 3513 Acked Write 9 Sent Done Done false
11045 3514 Acked Read 4 Sent Done Done false
11046 3515 Acked Flush 0 Sent Done Done false
11047 3516 Acked Read 7 Sent Done Done false
11048 3517 Acked Write 10 Sent Done Done false
11049 3518 Acked Write 3 Sent Done Done false
11050 3519 Acked Write 10 Sent Done Done false
11051 3520 Acked Flush 0 Sent Done Done false
11052 3521 Acked Write 2 Sent Done Done false
11053 3522 Acked Write 4 Sent Done Done false
11054 3523 Acked Read 4 Sent Done Done false
11055 3524 Acked Read 5 Sent Done Done false
11056 3525 Acked Write 9 Sent Done Done false
11057 3526 Acked Read 8 Sent Done Done false
11058 3527 Acked Read 7 Sent Done Done false
11059 3528 Acked Write 1 Sent Done Done false
11060 3529 Acked Write 8 Sent Done Done false
11061 3530 Acked Read 6 Sent Done Done false
11062 3531 Acked Read 5 Sent Done Done false
11063 3532 Acked Read 10 Sent Done Done false
11064 3533 Acked Read 3 Sent Done Done false
11065 3534 Acked Write 1 Sent Done Done false
11066 3535 Acked Read 2 Sent Done Done false
11067 3536 Acked Read 5 Sent Done Done false
11068 3537 Acked Read 10 Sent Done Done false
11069 3538 Acked Read 10 Sent Done Done false
11070 3539 Acked Write 5 Sent Done Done false
11071 3540 Acked Read 6 Sent Done Done false
11072 3541 Acked Write 8 Sent Done Done false
11073 3542 Acked Read 5 Sent Done Done false
11074 3543 Acked Write 2 Sent Done Done false
11075 3544 Acked Write 6 Sent Done Done false
11076 3545 Acked Read 3 Sent Done Done false
11077 3546 Acked Write 8 Sent Done Done false
11078 3547 Acked Read 5 Sent Done Done false
11079 3548 Acked Write 9 Sent Done Done false
11080 3549 Acked Read 3 Sent Done Done false
11081 3550 Acked Write 2 Sent Done Done false
11082 3551 Acked Write 1 Sent Done Done false
11083 3552 Acked Flush 0 Sent Done Done false
11084 3553 Acked Write 5 Sent Done Done false
11085 3554 Acked Write 5 Sent Done Done false
11086 3555 Acked Write 5 Sent Done Done false
11087 3556 Acked Write 5 Sent Done Done false
11088 3557 Acked Read 4 Sent Done Done false
11089 3558 Acked Write 5 Sent Done Done false
11090 3559 Acked Write 2 Sent Done Done false
11091 3560 Acked Write 4 Sent Done Done false
11092 3561 Acked Flush 0 Sent Done Done false
11093 3562 Acked Read 8 Sent Done Done false
11094 3563 Acked Read 10 Sent Done Done false
11095 3564 Acked Read 2 Sent Done Done false
11096 3565 Acked Write 5 Sent Done Done false
11097 3566 Acked Write 5 Sent Done Done false
11098 3567 Acked Read 3 Sent Done Done false
11099 3568 Acked Read 8 Sent Done Done false
11100 3569 Acked Write 10 Sent Done Done false
11101 3570 Acked Read 5 Sent Done Done false
11102 3571 Acked Read 6 Sent Done Done false
11103 3572 Acked Write 2 Sent Done Done false
11104 3573 Acked Write 6 Sent Done Done false
11105 3574 Acked Read 3 Sent Done Done false
11106 3575 Acked Read 3 Sent Done Done false
11107 3576 Acked Read 5 Sent Done Done false
11108 3577 Acked Write 4 Sent Done Done false
11109 3578 Acked Read 9 Sent Done Done false
11110 3579 Acked Write 6 Sent Done Done false
11111 3580 Acked Flush 0 Sent Done Done false
11112 3581 Acked Write 10 Sent Done Done false
11113 3582 Acked Read 9 Sent Done Done false
11114 3583 Acked Flush 0 Sent Done Done false
11115 3584 Acked Read 7 Sent Done Done false
11116 3585 Acked Write 10 Sent Done Done false
11117 3586 Acked Write 10 Sent Done Done false
11118 3587 Acked Read 5 Sent Done Done false
11119 3588 Acked Read 3 Sent Done Done false
11120 3589 Acked Read 3 Sent Done Done false
11121 3590 Acked Flush 0 Sent Done Done false
11122 3591 Acked Flush 0 Sent Done Done false
11123 3592 Acked Read 1 Sent Done Done false
11124 3593 Acked Write 2 Sent Done Done false
11125 3594 Acked Read 3 Sent Done Done false
11126 3595 Acked Write 6 Sent Done Done false
11127 3596 Acked Read 6 Sent Done Done false
11128 3597 Acked Read 3 Sent Done Done false
11129 3598 Acked Read 4 Sent Done Done false
11130 3599 Acked Write 7 Sent Done Done false
11131 3600 Acked Write 1 Sent Done Done false
11132 3601 Acked Read 1 Sent Done Done false
11133 3602 Acked Write 6 Sent Done Done false
11134 3603 Acked Write 3 Sent Done Done false
11135 3604 Acked Read 5 Sent Done Done false
11136 3605 Acked Read 7 Sent Done Done false
11137 3606 Acked Read 3 Sent Done Done false
11138 3607 Acked Read 7 Sent Done Done false
11139 3608 Acked Write 10 Sent Done Done false
11140 3609 Acked Read 5 Sent Done Done false
11141 3610 Acked Read 7 Sent Done Done false
11142 3611 Acked Write 2 Sent Done Done false
11143 3612 Acked Write 9 Sent Done Done false
11144 3613 Acked Write 3 Sent Done Done false
11145 3614 Acked Write 8 Sent Done Done false
11146 3615 Acked Read 2 Sent Done Done false
11147 3616 Acked Flush 0 Sent Done Done false
11148 3617 Acked Write 5 Sent Done Done false
11149 3618 Acked Read 10 Sent Done Done false
11150 3619 Acked Write 7 Sent Done Done false
11151 3620 Acked Flush 0 Sent Done Done false
11152 3621 Acked Read 4 Sent Done Done false
11153 3622 Acked Write 10 Sent Done Done false
11154 3623 Acked Write 7 Sent Done Done false
11155 3624 Acked Read 5 Sent Done Done false
11156 3625 Acked Flush 0 Sent Done Done false
11157 3626 Acked Write 4 Sent Done Done false
11158 3627 Acked Flush 0 Sent Done Done false
11159 3628 Acked Read 3 Sent Done Done false
11160 3629 Acked Read 6 Sent Done Done false
11161 3630 Acked Write 5 Sent Done Done false
11162 3631 Acked Write 10 Sent Done Done false
11163 3632 Acked Write 6 Sent Done Done false
11164 3633 Acked Read 10 Sent Done Done false
11165 3634 Acked Write 7 Sent Done Done false
11166 3635 Acked Read 5 Sent Done Done false
11167 3636 Acked Write 10 Sent Done Done false
11168 3637 Acked Write 6 Sent Done Done false
11169 3638 Acked Read 2 Sent Done Done false
11170 3639 Acked Write 8 Sent Done Done false
11171 3640 Acked Read 6 Sent Done Done false
11172 3641 Acked Write 7 Sent Done Done false
11173 3642 Acked Read 9 Sent Done Done false
11174 3643 Acked Write 3 Sent Done Done false
11175 3644 Acked Flush 0 Sent Done Done false
11176 3645 Acked Read 1 Sent Done Done false
11177 3646 Acked Write 9 Sent Done Done false
11178 3647 Acked Read 3 Sent Done Done false
11179 3648 Acked Write 9 Sent Done Done false
11180 3649 Acked Read 1 Sent Done Done false
11181 3650 Acked Read 10 Sent Done Done false
11182 3651 Acked Read 9 Sent Done Done false
11183 3652 Acked Flush 0 Sent Done Done false
11184 3653 Acked Write 2 Sent Done Done false
11185 3654 Acked Write 8 Sent Done Done false
11186 3655 Acked Write 5 Sent Done Done false
11187 3656 Acked Flush 0 Sent Done Done false
11188 3657 Acked Write 1 Sent Done Done false
11189 3658 Acked Write 2 Sent Done Done false
11190 3659 Acked Read 10 Sent Done Done false
11191 3660 Acked Read 2 Sent Done Done false
11192 3661 Acked Read 8 Sent Done Done false
11193 3662 Acked Read 8 Sent Done Done false
11194 3663 Acked Write 1 Sent Done Done false
11195 3664 Acked Write 5 Sent Done Done false
11196 3665 Acked Read 1 Sent Done Done false
11197 3666 Acked Read 7 Sent Done Done false
11198 3667 Acked Read 2 Sent Done Done false
11199 3668 Acked Read 1 Sent Done Done false
11200 3669 Acked Write 5 Sent Done Done false
11201 3670 Acked Flush 0 Sent Done Done false
11202 3671 Acked Write 9 Sent Done Done false
11203 3672 Acked Read 7 Sent Done Done false
11204 3673 Acked Read 2 Sent Done Done false
11205 3674 Acked Write 8 Sent Done Done false
11206 3675 Acked Write 9 Sent Done Done false
11207 3676 Acked Write 5 Sent Done Done false
11208 3677 Acked Read 5 Sent Done Done false
11209 3678 Acked Read 9 Sent Done Done false
11210 3679 Acked Write 8 Sent Done Done false
11211 3680 Acked Read 3 Sent Done Done false
11212 3681 Acked Write 10 Sent Done Done false
11213 3682 Acked Read 4 Sent Done Done false
11214 3683 Acked Read 7 Sent Done Done false
11215 3684 Acked Write 1 Sent Done Done false
11216 3685 Acked Write 5 Sent Done Done false
11217 3686 Acked Write 4 Sent Done Done false
11218 3687 Acked Read 3 Sent Done Done false
11219 3688 Acked Write 8 Sent Done Done false
11220 3689 Acked Read 7 Sent Done Done false
11221 3690 Acked Read 1 Sent Done Done false
11222 3691 Acked Write 4 Sent Done Done false
11223 3692 Acked Read 3 Sent Done Done false
11224 3693 Acked Read 4 Sent Done Done false
11225 3694 Acked Read 4 Sent Done Done false
11226 3695 Acked Read 1 Sent Done Done false
11227 3696 Acked Read 8 Sent Done Done false
11228 3697 Acked Read 8 Sent Done Done false
11229 3698 Acked Read 9 Sent Done Done false
11230 3699 Acked Write 3 Sent Done Done false
11231 3700 Acked Read 2 Sent Done Done false
11232 3701 Acked Write 4 Sent Done Done false
11233 3702 Acked Write 2 Sent Done Done false
11234 3703 Acked Write 1 Sent Done Done false
11235 3704 Acked Read 2 Sent Done Done false
11236 3705 Acked Write 7 Sent Done Done false
11237 3706 Acked Read 9 Sent Done Done false
11238 3707 Acked Flush 0 Sent Done Done false
11239 3708 Acked Read 4 Sent Done Done false
11240 3709 Acked Write 10 Sent Done Done false
11241 3710 Acked Write 8 Sent Done Done false
11242 3711 Acked Write 4 Sent Done Done false
11243 3712 Acked Write 9 Sent Done Done false
11244 3713 Acked Read 5 Sent Done Done false
11245 3714 Acked Read 6 Sent Done Done false
11246 STATES DS:0 DS:1 DS:2 TOTAL
11247 Sent 301 0 0 301
11248 Done 0 301 301 602
11249 Skipped 0 0 0 0
11250 Error 0 0 0 0
11251 Last Flush: Some(JobId(3413)) Some(JobId(3707)) Some(JobId(3707))
11252 Downstairs last five completed: 3413 3412 3411 3410 3409
11253 Upstairs last five completed: 3714 3713 3712 3711 3710
11254 CLIENT: Up:0 ds:301 act:2
112552025-03-07T18:14:59.289ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
112562025-03-07T18:15:00.185ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
112572025-03-07T18:15:01.188ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
112582025-03-07T18:15:02.272ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11259 [8810] got start action
11260 [8810] I am going to restart
11261 Starting downstairs at port 8810
11262 Make output file at "/tmp/dsc/downstairs-8810.txt"
11263 [8810][0] reports Starting
11264 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1217)
11265 [8810][0] reports Running
11266 ----------------------------------------------------------------
11267 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11268 States: Offline Active Active
11269 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11270 3414 Acked Read 9 Sent Done Done false
11271 3415 Acked Write 8 Sent Done Done false
11272 3416 Acked Read 6 Sent Done Done false
11273 3417 Acked Read 10 Sent Done Done false
11274 3418 Acked Read 9 Sent Done Done false
11275 3419 Acked Read 4 Sent Done Done false
11276 3420 Acked Flush 0 Sent Done Done false
11277 3421 Acked Read 5 Sent Done Done false
11278 3422 Acked Read 4 Sent Done Done false
11279 3423 Acked Flush 0 Sent Done Done false
11280 3424 Acked Flush 0 Sent Done Done false
11281 3425 Acked Read 5 Sent Done Done false
11282 3426 Acked Write 7 Sent Done Done false
11283 3427 Acked Write 9 Sent Done Done false
11284 3428 Acked Write 2 Sent Done Done false
11285 3429 Acked Read 8 Sent Done Done false
11286 3430 Acked Write 1 Sent Done Done false
11287 3431 Acked Read 6 Sent Done Done false
11288 3432 Acked Write 5 Sent Done Done false
11289 3433 Acked Read 7 Sent Done Done false
11290 3434 Acked Write 5 Sent Done Done false
11291 3435 Acked Write 5 Sent Done Done false
11292 3436 Acked Write 9 Sent Done Done false
11293 3437 Acked Read 1 Sent Done Done false
11294 3438 Acked Read 7 Sent Done Done false
11295 3439 Acked Read 2 Sent Done Done false
11296 3440 Acked Write 9 Sent Done Done false
11297 3441 Acked Write 7 Sent Done Done false
11298 3442 Acked Write 7 Sent Done Done false
11299 3443 Acked Write 9 Sent Done Done false
11300 3444 Acked Write 10 Sent Done Done false
11301 3445 Acked Write 3 Sent Done Done false
11302 3446 Acked Flush 0 Sent Done Done false
11303 3447 Acked Write 8 Sent Done Done false
11304 3448 Acked Write 5 Sent Done Done false
11305 3449 Acked Read 2 Sent Done Done false
11306 3450 Acked Write 5 Sent Done Done false
11307 3451 Acked Write 8 Sent Done Done false
11308 3452 Acked Write 6 Sent Done Done false
11309 3453 Acked Read 7 Sent Done Done false
11310 3454 Acked Write 2 Sent Done Done false
11311 3455 Acked Flush 0 Sent Done Done false
11312 3456 Acked Write 8 Sent Done Done false
11313 3457 Acked Read 7 Sent Done Done false
11314 3458 Acked Read 2 Sent Done Done false
11315 3459 Acked Read 5 Sent Done Done false
11316 3460 Acked Write 5 Sent Done Done false
11317 3461 Acked Read 4 Sent Done Done false
11318 3462 Acked Write 5 Sent Done Done false
11319 3463 Acked Read 2 Sent Done Done false
11320 3464 Acked Read 1 Sent Done Done false
11321 3465 Acked Flush 0 Sent Done Done false
11322 3466 Acked Write 5 Sent Done Done false
11323 3467 Acked Read 6 Sent Done Done false
11324 3468 Acked Write 7 Sent Done Done false
11325 3469 Acked Write 7 Sent Done Done false
11326 3470 Acked Write 7 Sent Done Done false
11327 3471 Acked Write 7 Sent Done Done false
11328 3472 Acked Write 8 Sent Done Done false
11329 3473 Acked Flush 0 Sent Done Done false
11330 3474 Acked Write 9 Sent Done Done false
11331 3475 Acked Write 10 Sent Done Done false
11332 3476 Acked Write 10 Sent Done Done false
11333 3477 Acked Flush 0 Sent Done Done false
11334 3478 Acked Read 1 Sent Done Done false
11335 3479 Acked Read 1 Sent Done Done false
11336 3480 Acked Read 5 Sent Done Done false
11337 3481 Acked Read 5 Sent Done Done false
11338 3482 Acked Write 8 Sent Done Done false
11339 3483 Acked Write 1 Sent Done Done false
11340 3484 Acked Flush 0 Sent Done Done false
11341 3485 Acked Write 2 Sent Done Done false
11342 3486 Acked Write 1 Sent Done Done false
11343 3487 Acked Read 9 Sent Done Done false
11344 3488 Acked Read 1 Sent Done Done false
11345 3489 Acked Write 1 Sent Done Done false
11346 3490 Acked Write 6 Sent Done Done false
11347 3491 Acked Read 6 Sent Done Done false
11348 3492 Acked Write 6 Sent Done Done false
11349 3493 Acked Write 3 Sent Done Done false
11350 3494 Acked Read 1 Sent Done Done false
11351 3495 Acked Write 3 Sent Done Done false
11352 3496 Acked Read 3 Sent Done Done false
11353 3497 Acked Flush 0 Sent Done Done false
11354 3498 Acked Read 4 Sent Done Done false
11355 3499 Acked Write 2 Sent Done Done false
11356 3500 Acked Read 5 Sent Done Done false
11357 3501 Acked Write 7 Sent Done Done false
11358 3502 Acked Write 4 Sent Done Done false
11359 3503 Acked Write 3 Sent Done Done false
11360 3504 Acked Flush 0 Sent Done Done false
11361 3505 Acked Write 8 Sent Done Done false
11362 3506 Acked Read 1 Sent Done Done false
11363 3507 Acked Write 9 Sent Done Done false
11364 3508 Acked Write 5 Sent Done Done false
11365 3509 Acked Read 1 Sent Done Done false
11366 3510 Acked Read 2 Sent Done Done false
11367 3511 Acked Read 4 Sent Done Done false
11368 3512 Acked Read 9 Sent Done Done false
11369 3513 Acked Write 9 Sent Done Done false
11370 3514 Acked Read 4 Sent Done Done false
11371 3515 Acked Flush 0 Sent Done Done false
11372 3516 Acked Read 7 Sent Done Done false
11373 3517 Acked Write 10 Sent Done Done false
11374 3518 Acked Write 3 Sent Done Done false
11375 3519 Acked Write 10 Sent Done Done false
11376 3520 Acked Flush 0 Sent Done Done false
11377 3521 Acked Write 2 Sent Done Done false
11378 3522 Acked Write 4 Sent Done Done false
11379 3523 Acked Read 4 Sent Done Done false
11380 3524 Acked Read 5 Sent Done Done false
11381 3525 Acked Write 9 Sent Done Done false
11382 3526 Acked Read 8 Sent Done Done false
11383 3527 Acked Read 7 Sent Done Done false
11384 3528 Acked Write 1 Sent Done Done false
11385 3529 Acked Write 8 Sent Done Done false
11386 3530 Acked Read 6 Sent Done Done false
11387 3531 Acked Read 5 Sent Done Done false
11388 3532 Acked Read 10 Sent Done Done false
11389 3533 Acked Read 3 Sent Done Done false
11390 3534 Acked Write 1 Sent Done Done false
11391 3535 Acked Read 2 Sent Done Done false
11392 3536 Acked Read 5 Sent Done Done false
11393 3537 Acked Read 10 Sent Done Done false
11394 3538 Acked Read 10 Sent Done Done false
11395 3539 Acked Write 5 Sent Done Done false
11396 3540 Acked Read 6 Sent Done Done false
11397 3541 Acked Write 8 Sent Done Done false
11398 3542 Acked Read 5 Sent Done Done false
11399 3543 Acked Write 2 Sent Done Done false
11400 3544 Acked Write 6 Sent Done Done false
11401 3545 Acked Read 3 Sent Done Done false
11402 3546 Acked Write 8 Sent Done Done false
11403 3547 Acked Read 5 Sent Done Done false
11404 3548 Acked Write 9 Sent Done Done false
11405 3549 Acked Read 3 Sent Done Done false
11406 3550 Acked Write 2 Sent Done Done false
11407 3551 Acked Write 1 Sent Done Done false
11408 3552 Acked Flush 0 Sent Done Done false
11409 3553 Acked Write 5 Sent Done Done false
11410 3554 Acked Write 5 Sent Done Done false
11411 3555 Acked Write 5 Sent Done Done false
11412 3556 Acked Write 5 Sent Done Done false
11413 3557 Acked Read 4 Sent Done Done false
11414 3558 Acked Write 5 Sent Done Done false
11415 3559 Acked Write 2 Sent Done Done false
11416 3560 Acked Write 4 Sent Done Done false
11417 3561 Acked Flush 0 Sent Done Done false
11418 3562 Acked Read 8 Sent Done Done false
11419 3563 Acked Read 10 Sent Done Done false
11420 3564 Acked Read 2 Sent Done Done false
11421 3565 Acked Write 5 Sent Done Done false
11422 3566 Acked Write 5 Sent Done Done false
11423 3567 Acked Read 3 Sent Done Done false
11424 3568 Acked Read 8 Sent Done Done false
11425 3569 Acked Write 10 Sent Done Done false
11426 3570 Acked Read 5 Sent Done Done false
11427 3571 Acked Read 6 Sent Done Done false
11428 3572 Acked Write 2 Sent Done Done false
11429 3573 Acked Write 6 Sent Done Done false
11430 3574 Acked Read 3 Sent Done Done false
11431 3575 Acked Read 3 Sent Done Done false
11432 3576 Acked Read 5 Sent Done Done false
11433 3577 Acked Write 4 Sent Done Done false
11434 3578 Acked Read 9 Sent Done Done false
11435 3579 Acked Write 6 Sent Done Done false
11436 3580 Acked Flush 0 Sent Done Done false
11437 3581 Acked Write 10 Sent Done Done false
11438 3582 Acked Read 9 Sent Done Done false
11439 3583 Acked Flush 0 Sent Done Done false
11440 3584 Acked Read 7 Sent Done Done false
11441 3585 Acked Write 10 Sent Done Done false
11442 3586 Acked Write 10 Sent Done Done false
11443 3587 Acked Read 5 Sent Done Done false
11444 3588 Acked Read 3 Sent Done Done false
11445 3589 Acked Read 3 Sent Done Done false
11446 3590 Acked Flush 0 Sent Done Done false
11447 3591 Acked Flush 0 Sent Done Done false
11448 3592 Acked Read 1 Sent Done Done false
11449 3593 Acked Write 2 Sent Done Done false
11450 3594 Acked Read 3 Sent Done Done false
11451 3595 Acked Write 6 Sent Done Done false
11452 3596 Acked Read 6 Sent Done Done false
11453 3597 Acked Read 3 Sent Done Done false
11454 3598 Acked Read 4 Sent Done Done false
11455 3599 Acked Write 7 Sent Done Done false
11456 3600 Acked Write 1 Sent Done Done false
11457 3601 Acked Read 1 Sent Done Done false
11458 3602 Acked Write 6 Sent Done Done false
11459 3603 Acked Write 3 Sent Done Done false
11460 3604 Acked Read 5 Sent Done Done false
11461 3605 Acked Read 7 Sent Done Done false
11462 3606 Acked Read 3 Sent Done Done false
11463 3607 Acked Read 7 Sent Done Done false
11464 3608 Acked Write 10 Sent Done Done false
11465 3609 Acked Read 5 Sent Done Done false
11466 3610 Acked Read 7 Sent Done Done false
11467 3611 Acked Write 2 Sent Done Done false
11468 3612 Acked Write 9 Sent Done Done false
11469 3613 Acked Write 3 Sent Done Done false
11470 3614 Acked Write 8 Sent Done Done false
11471 3615 Acked Read 2 Sent Done Done false
11472 3616 Acked Flush 0 Sent Done Done false
11473 3617 Acked Write 5 Sent Done Done false
11474 3618 Acked Read 10 Sent Done Done false
11475 3619 Acked Write 7 Sent Done Done false
11476 3620 Acked Flush 0 Sent Done Done false
11477 3621 Acked Read 4 Sent Done Done false
11478 3622 Acked Write 10 Sent Done Done false
11479 3623 Acked Write 7 Sent Done Done false
11480 3624 Acked Read 5 Sent Done Done false
11481 3625 Acked Flush 0 Sent Done Done false
11482 3626 Acked Write 4 Sent Done Done false
11483 3627 Acked Flush 0 Sent Done Done false
11484 3628 Acked Read 3 Sent Done Done false
11485 3629 Acked Read 6 Sent Done Done false
11486 3630 Acked Write 5 Sent Done Done false
11487 3631 Acked Write 10 Sent Done Done false
11488 3632 Acked Write 6 Sent Done Done false
11489 3633 Acked Read 10 Sent Done Done false
11490 3634 Acked Write 7 Sent Done Done false
11491 3635 Acked Read 5 Sent Done Done false
11492 3636 Acked Write 10 Sent Done Done false
11493 3637 Acked Write 6 Sent Done Done false
11494 3638 Acked Read 2 Sent Done Done false
11495 3639 Acked Write 8 Sent Done Done false
11496 3640 Acked Read 6 Sent Done Done false
11497 3641 Acked Write 7 Sent Done Done false
11498 3642 Acked Read 9 Sent Done Done false
11499 3643 Acked Write 3 Sent Done Done false
11500 3644 Acked Flush 0 Sent Done Done false
11501 3645 Acked Read 1 Sent Done Done false
11502 3646 Acked Write 9 Sent Done Done false
11503 3647 Acked Read 3 Sent Done Done false
11504 3648 Acked Write 9 Sent Done Done false
11505 3649 Acked Read 1 Sent Done Done false
11506 3650 Acked Read 10 Sent Done Done false
11507 3651 Acked Read 9 Sent Done Done false
11508 3652 Acked Flush 0 Sent Done Done false
11509 3653 Acked Write 2 Sent Done Done false
11510 3654 Acked Write 8 Sent Done Done false
11511 3655 Acked Write 5 Sent Done Done false
11512 3656 Acked Flush 0 Sent Done Done false
11513 3657 Acked Write 1 Sent Done Done false
11514 3658 Acked Write 2 Sent Done Done false
11515 3659 Acked Read 10 Sent Done Done false
11516 3660 Acked Read 2 Sent Done Done false
11517 3661 Acked Read 8 Sent Done Done false
11518 3662 Acked Read 8 Sent Done Done false
11519 3663 Acked Write 1 Sent Done Done false
11520 3664 Acked Write 5 Sent Done Done false
11521 3665 Acked Read 1 Sent Done Done false
11522 3666 Acked Read 7 Sent Done Done false
11523 3667 Acked Read 2 Sent Done Done false
11524 3668 Acked Read 1 Sent Done Done false
11525 3669 Acked Write 5 Sent Done Done false
11526 3670 Acked Flush 0 Sent Done Done false
11527 3671 Acked Write 9 Sent Done Done false
11528 3672 Acked Read 7 Sent Done Done false
11529 3673 Acked Read 2 Sent Done Done false
11530 3674 Acked Write 8 Sent Done Done false
11531 3675 Acked Write 9 Sent Done Done false
11532 3676 Acked Write 5 Sent Done Done false
11533 3677 Acked Read 5 Sent Done Done false
11534 3678 Acked Read 9 Sent Done Done false
11535 3679 Acked Write 8 Sent Done Done false
11536 3680 Acked Read 3 Sent Done Done false
11537 3681 Acked Write 10 Sent Done Done false
11538 3682 Acked Read 4 Sent Done Done false
11539 3683 Acked Read 7 Sent Done Done false
11540 3684 Acked Write 1 Sent Done Done false
11541 3685 Acked Write 5 Sent Done Done false
11542 3686 Acked Write 4 Sent Done Done false
11543 3687 Acked Read 3 Sent Done Done false
11544 3688 Acked Write 8 Sent Done Done false
11545 3689 Acked Read 7 Sent Done Done false
11546 3690 Acked Read 1 Sent Done Done false
11547 3691 Acked Write 4 Sent Done Done false
11548 3692 Acked Read 3 Sent Done Done false
11549 3693 Acked Read 4 Sent Done Done false
11550 3694 Acked Read 4 Sent Done Done false
11551 3695 Acked Read 1 Sent Done Done false
11552 3696 Acked Read 8 Sent Done Done false
11553 3697 Acked Read 8 Sent Done Done false
11554 3698 Acked Read 9 Sent Done Done false
11555 3699 Acked Write 3 Sent Done Done false
11556 3700 Acked Read 2 Sent Done Done false
11557 3701 Acked Write 4 Sent Done Done false
11558 3702 Acked Write 2 Sent Done Done false
11559 3703 Acked Write 1 Sent Done Done false
11560 3704 Acked Read 2 Sent Done Done false
11561 3705 Acked Write 7 Sent Done Done false
11562 3706 Acked Read 9 Sent Done Done false
11563 3707 Acked Flush 0 Sent Done Done false
11564 3708 Acked Read 4 Sent Done Done false
11565 3709 Acked Write 10 Sent Done Done false
11566 3710 Acked Write 8 Sent Done Done false
11567 3711 Acked Write 4 Sent Done Done false
11568 3712 Acked Write 9 Sent Done Done false
11569 3713 Acked Read 5 Sent Done Done false
11570 3714 Acked Read 6 Sent Done Done false
11571 STATES DS:0 DS:1 DS:2 TOTAL
11572 Sent 301 0 0 301
11573 Done 0 301 301 602
11574 Skipped 0 0 0 0
11575 Error 0 0 0 0
11576 Last Flush: Some(JobId(3413)) Some(JobId(3707)) Some(JobId(3707))
11577 Downstairs last five completed: 3413 3412 3411 3410 3409
11578 Upstairs last five completed: 3714 3713 3712 3711 3710
11579 CLIENT: Up:0 ds:301 act:2
115802025-03-07T18:15:03.291ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115812025-03-07T18:15:04.192ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115822025-03-07T18:15:04.549ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115832025-03-07T18:15:04.549ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44787) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115842025-03-07T18:15:04.550ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115852025-03-07T18:15:04.551ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115862025-03-07T18:15:04.551ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115872025-03-07T18:15:04.551ZINFOcrucible: send last flush ID to this DS: Some(JobId(3413)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115882025-03-07T18:15:04.551ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115892025-03-07T18:15:04.551ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3413)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
115902025-03-07T18:15:04.551ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3413)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11591 ----------------------------------------------------------------
11592 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11593 Downstairs last five completed: 3715 3714 3713 3712 3711
11594 Upstairs last five completed: 3715 3714 3713 3712 3711
11595 CLIENT: Up:0 ds:0 act:3
11596 Replay: All jobs finished
11597 Main task has work to do, go find it
11598 got dsc Stop(1)
11599 stop 1
11600 Mar 07 18:15:06.792 INFO request completed, latency_us: 348, response_code:[8820] Got stop action so:false kr:false
11601 204, uri: /stop/cid/1, method: POST, req_id: da2d64e1-2000-4f97-b7a8-25289eac5f7c, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
11602 Mar 07 18:15:06.793 INFO request completed, latency_us: 145, response_code: 200, uri: /state/cid/1, method: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:06.793434985Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
11603 GET, req_id: ec8982b6-b4e9-4699-805e-d20b6165f681, remote_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:06.793489479Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
11604 0.1:64322, local_addr: 127.0.0.1{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:06.793527554Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
11605 :9998
116062025-03-07T18:15:06.793ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
11607 [8820] Exited with: ExitStatus(unix_wait_status(9))
11608 [8820][1] reports Exit
11609 Mar 07 18:15:10.796 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: 85e7fa26-5b38-4110-b3a1-a14f4f5f46db, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
11610 001/300 Read block 2056 len 4096
11611 002/300 Read block 3179 len 28672
11612 003/300 Read block 3981 len 20480
11613 004/300 Read block 189 len 32768
11614 005/300 Read block 1886 len 20480
11615 006/300 Read block 642 len 8192
11616 007/300 Read block 4381 len 36864
11617 008/300 Write block 701 len 20480 data: 3 3 3 3 3
11618 009/300 Read block 1291 len 32768
11619 010/300 Flush
11620 011/300 Write block 2376 len 24576 data: 3 3 3 3 2 2
11621 012/300 Read block 3096 len 12288
11622 013/300 Write block 387 len 36864 data: 3 3 3 3 3 3 4 3 4
11623 014/300 Write block 91 len 16384 data: 2 2 3 3
11624 015/300 Read block 1423 len 20480
11625 016/300 Write block 1860 len 16384 data: 3 3 2 2
11626 017/300 Flush
11627 018/300 Read block 2508 len 4096
11628 019/300 Write block 1454 len 4096 data: 4
11629 020/300 Write block 2045 len 28672 data: 4 4 5 5 3 3 3
11630 021/300 Flush
11631 022/300 Write block 4271 len 8192 data: 4 4
11632 023/300 Write block 4624 len 8192 data: 5 4
11633 024/300 Read block 1920 len 20480
11634 025/300 Read block 865 len 28672
11635 026/300 Write block 220 len 8192 data: 3 3
11636 027/300 Write block 3600 len 20480 data: 4 3 2 2 2
11637 028/300 Read block 2328 len 24576
11638 029/300 Write block 4660 len 12288 data: 2 2 2
11639 030/300 Write block 2780 len 24576 data: 2 3 2 2 2 2
11640 031/300 Flush
11641 032/300 Write block 2739 len 12288 data: 3 2 3
11642 033/300 Read block 1377 len 32768
11643 034/300 Read block 1470 len 32768
11644 035/300 Read block 306 len 12288
11645 036/300 Read block 1641 len 24576
11646 037/300 Write block 4719 len 40960 data: 2 3 2 2 2 2 2 2 2 2
11647 038/300 Write block 127 len 24576 data: 4 4 3 3 3 3
11648 039/300 Read block 2081 len 16384
11649 040/300 Read block 1023 len 20480
11650 041/300 Read block 2954 len 28672
11651 042/300 Read block 2684 len 20480
11652 043/300 Read block 3376 len 40960
11653 044/300 Write block 3990 len 4096 data: 6
11654 045/300 Read block 1603 len 32768
11655 046/300 Flush
11656 047/300 Read block 3163 len 28672
11657 048/300 Write block 418 len 32768 data: 4 4 3 3 3 5 5 5
11658 049/300 Read block 846 len 4096
11659 050/300 Read block 2285 len 20480
11660 051/300 Write block 1772 len 24576 data: 2 3 3 3 3 3
11661 052/300 Write block 2781 len 8192 data: 4 3
11662 053/300 Read block 3358 len 24576
11663 054/300 Read block 2417 len 16384
11664 055/300 Write block 2104 len 32768 data: 3 3 3 2 2 2 3 3
11665 056/300 Write block 1190 len 28672 data: 3 3 3 3 2 2 3
11666 057/300 Flush
11667 058/300 Write block 528 len 24576 data: 4 4 4 3 3 3
11668 059/300 Write block 3754 len 20480 data: 3 4 4 4 3
11669 060/300 Read block 4266 len 4096
11670 061/300 Read block 4962 len 20480
11671 062/300 Read block 1842 len 36864
11672 063/300 Read block 4608 len 12288
11673 064/300 Write block 3080 len 28672 data: 5 5 4 4 6 5 4
11674 065/300 Read block 2155 len 8192
11675 066/300 Read block 4593 len 20480
11676 067/300 Flush
11677 068/300 Write block 3377 len 12288 data: 3 2 2
11678 069/300 Read block 1396 len 36864
11679 070/300 Write block 3191 len 8192 data: 4 5
11680 071/300 Read block 2429 len 8192
11681 072/300 Write block 3705 len 20480 data: 3 3 3 2 2
11682 073/300 Write block 2939 len 20480 data: 2 2 2 3 5
11683 074/300 Write block 1326 len 28672 data: 3 3 4 3 3 3 3
11684 075/300 Read block 1467 len 36864
11685 076/300 Write block 1292 len 36864 data: 4 3 4 4 4 4 4 4 3
11686 077/300 Read block 3173 len 40960
11687 078/300 Read block 476 len 32768
11688 079/300 Write block 4811 len 32768 data: 2 2 2 2 2 2 2 2
11689 080/300 Read block 3376 len 36864
11690 081/300 Read block 1003 len 12288
11691 082/300 Write block 1178 len 40960 data: 3 3 3 4 3 3 3 3 3 4
11692 083/300 Read block 1752 len 12288
11693 084/300 Read block 1299 len 36864
11694 085/300 Read block 2866 len 32768
11695 086/300 Write block 670 len 12288 data: 4 4 6
11696 087/300 Read block 1552 len 8192
11697 088/300 Read block 2723 len 28672
11698 089/300 Write block 2214 len 4096 data: 2
11699 090/300 Flush
11700 091/300 Write block 2044 len 40960 data: 4 5 5 6 6 4 4 4 4 3
11701 092/300 Read block 288 len 40960
11702 093/300 Read block 4501 len 24576
11703 094/300 Read block 502 len 40960
11704 095/300 Write block 741 len 36864 data: 4 4 4 4 4 4 4 4 2
11705 096/300 Write block 2732 len 40960 data: 5 4 3 3 3 3 3 4 3 4
11706 097/300 Read block 1174 len 4096
11707 098/300 Read block 1742 len 20480
11708 099/300 Write block 3030 len 12288 data: 2 2 2
11709 100/300 Write block 3276 len 24576 data: 3 3 4 4 4 5
11710 101/300 Write block 1695 len 4096 data: 4
11711 102/300 Write block 2975 len 12288 data: 3 2 2
11712 103/300 Write block 2122 len 40960 data: 3 3 2 2 2 2 2 3 3 3
11713 104/300 Read block 2989 len 20480
11714 105/300 Flush
11715 106/300 Write block 1685 len 40960 data: 2 2 2 2 2 2 2 2 3 4
11716 107/300 Write block 514 len 24576 data: 3 3 3 2 3 2
11717 108/300 Read block 1412 len 20480
11718 109/300 Read block 1347 len 8192
11719 110/300 Read block 3243 len 40960
11720 111/300 Read block 4462 len 8192
11721 112/300 Write block 1903 len 36864 data: 4 4 4 4 4 6 6 6 6
11722 113/300 Read block 1375 len 12288
11723 114/300 Write block 1810 len 8192 data: 3 3
11724 115/300 Read block 4751 len 12288
11725 116/300 Read block 1436 len 24576
11726 117/300 Write block 2198 len 24576 data: 3 3 3 3 3 2
11727 118/300 Flush
11728 119/300 Read block 1856 len 28672
11729 120/300 Read block 3058 len 4096
11730 121/300 Write block 4106 len 40960 data: 3 3 3 3 4 3 3 4 3 2
11731 122/300 Write block 2910 len 36864 data: 4 3 3 3 4 4 3 3 4
11732 123/300 Write block 4106 len 12288 data: 4 4 4
11733 124/300 Read block 113 len 8192
11734 125/300 Write block 196 len 36864 data: 5 5 5 2 2 3 3 4 4
11735 126/300 Read block 708 len 24576
11736 127/300 Flush
11737 128/300 Read block 4329 len 12288
11738 129/300 Read block 4298 len 40960
11739 130/300 Write block 1553 len 4096 data: 3
11740 131/300 Write block 4302 len 24576 data: 2 2 2 4 4 4
11741 132/300 Write block 1778 len 36864 data: 3 3 3 2 2 2 2 2 3
11742 133/300 Write block 1227 len 4096 data: 3
11743 134/300 Read block 4283 len 28672
11744 135/300 Read block 4940 len 16384
11745 136/300 Read block 1530 len 16384
11746 137/300 Write block 4038 len 8192 data: 4 3
11747 138/300 Read block 681 len 8192
11748 139/300 Flush
11749 140/300 Write block 1693 len 4096 data: 4
11750 141/300 Write block 3083 len 4096 data: 5
11751 142/300 Read block 4588 len 28672
11752 143/300 Write block 1134 len 12288 data: 2 2 2
11753 144/300 Write block 3407 len 28672 data: 2 2 2 2 2 3 3
11754 145/300 Write block 1973 len 16384 data: 3 2 2 2
11755 146/300 Read block 2432 len 28672
11756 147/300 Read block 2135 len 24576
11757 148/300 Read block 1267 len 36864
11758 149/300 Flush
11759 150/300 Read block 1772 len 16384
11760 151/300 Write block 1430 len 40960 data: 3 3 3 3 5 5 4 4 4 4
11761 152/300 Write block 3426 len 12288 data: 3 3 4
11762 153/300 Flush
11763 154/300 Write block 3603 len 16384 data: 3 3 3 3
11764 155/300 Read block 1428 len 24576
11765 156/300 Write block 2707 len 32768 data: 2 2 2 2 3 3 3 3
11766 157/300 Read block 2487 len 16384
11767 158/300 Read block 4657 len 28672
11768 159/300 Read block 1308 len 32768
11769 160/300 Read block 3041 len 12288
11770 161/300 Write block 488 len 4096 data: 4
11771 162/300 Read block 1313 len 20480
11772 163/300 Read block 2043 len 32768
11773 164/300 Read block 952 len 28672
11774 165/300 Read block 3952 len 16384
11775 166/300 Write block 1390 len 8192 data: 2 2
11776 167/300 Read block 1281 len 16384
11777 168/300 Write block 3934 len 28672 data: 3 3 3 3 4 4 4
11778 169/300 Write block 4211 len 12288 data: 3 3 3
11779 170/300 Read block 679 len 4096
11780 171/300 Read block 189 len 24576
11781 172/300 Write block 1769 len 32768 data: 2 2 2 3 4 4 4 4
11782 173/300 Write block 4361 len 36864 data: 3 3 4 3 5 5 4 4 4
11783 174/300 Read block 2853 len 20480
11784 175/300 Read block 269 len 16384
11785 176/300 Write block 4950 len 4096 data: 2
11786 177/300 Read block 75 len 40960
11787 178/300 Write block 940 len 28672 data: 3 3 3 3 3 3 3
11788 179/300 Read block 190 len 4096
11789 180/300 Read block 4181 len 8192
11790 181/300 Read block 2652 len 12288
11791 182/300 Read block 2399 len 36864
11792 183/300 Flush
11793 184/300 Read block 3785 len 24576
11794 185/300 Write block 1007 len 36864 data: 4 5 5 5 5 5 5 5 5
11795 186/300 Flush
11796 187/300 Flush
11797 188/300 Write block 3082 len 24576 data: 5 6 7 6 5 4
11798 189/300 Read block 298 len 16384
11799 190/300 Write block 1234 len 8192 data: 4 4
11800 191/300 Write block 4688 len 28672 data: 4 3 2 4 4 3 5
11801 192/300 Read block 4597 len 16384
11802 193/300 Write block 4506 len 4096 data: 6
11803 194/300 Read block 3770 len 4096
11804 195/300 Flush
11805 196/300 Read block 3402 len 20480
11806 197/300 Read block 2112 len 40960
11807 198/300 Read block 2358 len 28672
11808 199/300 Flush
11809 200/300 Write block 3634 len 32768 data: 4 4 5 4 5 5 5 5
11810 201/300 Flush
11811 202/300 Flush
11812 203/300 Flush
11813 204/300 Read block 1771 len 20480
11814 205/300 Read block 2850 len 32768
11815 206/300 Read block 3888 len 40960
11816 207/300 Read block 1289 len 24576
11817 208/300 Write block 747 len 40960 data: 5 5 3 2 2 2 2 2 2 2
11818 209/300 Read block 4255 len 4096
11819 210/300 Read block 2197 len 36864
11820 211/300 Write block 3425 len 8192 data: 3 4
11821 212/300 Write block 1318 len 28672 data: 3 3 3 3 3 3 3
11822 213/300 Write block 4643 len 24576 data: 3 5 5 4 4 4
11823 214/300 Read block 1710 len 36864
11824 215/300 Write block 28 len 12288 data: 2 2 2
11825 216/300 Write block 2495 len 4096 data: 5
11826 217/300 Write block 1214 len 12288 data: 4 3 3
11827 218/300 Write block 1801 len 36864 data: 4 5 5 5 5 3 2 2 2
11828 219/300 Write block 4406 len 40960 data: 2 2 2 2 3 3 3 3 4 4
11829 220/300 Write block 3637 len 8192 data: 5 6
11830 221/300 Read block 3517 len 32768
11831 222/300 Read block 1480 len 16384
11832 223/300 Write block 4954 len 36864 data: 2 2 2 2 3 3 3 3 4
11833 224/300 Read block 1864 len 4096
11834 225/300 Read block 4454 len 4096
11835 226/300 Read block 2524 len 24576
11836 227/300 Flush
11837 228/300 Read block 816 len 32768
11838 229/300 Write block 1616 len 8192 data: 5 5
11839 230/300 Write block 1008 len 16384 data: 6 6 6 6
11840 231/300 Read block 2178 len 28672
11841 232/300 Read block 2547 len 12288
11842 233/300 Read block 599 len 16384
11843 234/300 Read block 4940 len 28672
11844 235/300 Read block 4360 len 4096
11845 236/300 Write block 3124 len 20480 data: 2 3 3 3 3
11846 237/300 Read block 1471 len 32768
11847 238/300 Read block 1289 len 20480
11848 239/300 Flush
11849 240/300 Read block 1201 len 20480
11850 241/300 Read block 2504 len 20480
11851 242/300 Read block 1674 len 4096
11852 243/300 Write block 2389 len 28672 data: 4 4 4 3 4 4 4
11853 244/300 Write block 134 len 32768 data: 3 3 2 2 2 2 2 2
11854 245/300 Read block 2084 len 24576
11855 246/300 Read block 3186 len 28672
11856 247/300 Read block 2148 len 8192
11857 248/300 Read block 2946 len 24576
11858 249/300 Read block 2910 len 28672
11859 250/300 Write block 2818 len 32768 data: 3 4 4 3 3 3 3 3
11860 251/300 Write block 3167 len 20480 data: 3 2 2 2 2
11861 252/300 Read block 4391 len 20480
11862 253/300 Write block 4004 len 8192 data: 3 3
11863 254/300 Read block 1444 len 16384
11864 255/300 Read block 3664 len 40960
11865 256/300 Read block 4106 len 24576
11866 257/300 Write block 2216 len 24576 data: 2 2 2 2 3 3
11867 258/300 Read block 1504 len 40960
11868 259/300 Flush
11869 260/300 Read block 2190 len 40960
11870 261/300 Read block 2023 len 32768
11871 262/300 Flush
11872 263/300 Read block 1437 len 8192
11873 264/300 Write block 1251 len 20480 data: 4 2 3 2 2
11874 265/300 Write block 4080 len 32768 data: 3 3 3 3 2 2 2 3
11875 266/300 Read block 4189 len 20480
11876 267/300 Write block 263 len 36864 data: 4 3 3 3 3 3 2 2 3
11877 268/300 Write block 2598 len 40960 data: 3 3 3 3 3 3 3 2 3 3
11878 269/300 Read block 2630 len 36864
11879 270/300 Read block 620 len 20480
11880 271/300 Write block 2531 len 16384 data: 2 3 4 5
11881 272/300 Write block 4618 len 28672 data: 3 3 2 3 3 3 6
11882 273/300 Read block 2957 len 36864
11883 274/300 Read block 2176 len 36864
11884 275/300 Read block 4015 len 40960
11885 276/300 Read block 410 len 12288
11886 277/300 Write block 3773 len 12288 data: 2 2 3
11887 278/300 Write block 3441 len 20480 data: 3 4 3 3 3
11888 279/300 Write block 1123 len 20480 data: 4 3 3 3 3
11889 280/300 Write block 207 len 8192 data: 4 5
11890 281/300 Read block 3892 len 36864
11891 282/300 Write block 3494 len 40960 data: 3 3 2 2 2 2 2 2 2 2
11892 283/300 Read block 3066 len 16384
11893 284/300 Flush
11894 285/300 Write block 4574 len 24576 data: 2 2 3 2 2 2
11895 286/300 Read block 1165 len 32768
11896 287/300 Read block 2100 len 36864
11897 288/300 Read block 645 len 36864
11898 289/300 Read block 4283 len 36864
11899 290/300 Flush
11900 291/300 Write block 2780 len 16384 data: 3 5 4 3
11901 292/300 Read block 1604 len 12288
11902 293/300 Flush
11903 294/300 Write block 109 len 40960 data: 2 2 2 3 3 2 2 2 2 2
11904 295/300 Read block 456 len 20480
11905 296/300 Write block 1 len 32768 data: 2 3 3 3 4 4 5 5
11906 297/300 Read block 3202 len 40960
11907 298/300 Read block 4574 len 12288
11908 299/300 Read block 3527 len 40960
11909 300/300 Read block 4112 len 28672
11910 301/300 Read block 293 len 4096
11911 Main task has work to do, go find it
11912 got dsc Start(1)
11913 start 1
11914 Mar 07 18:15:11.049 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: fa043186-93a3-498c-95a0-265df0db29f1, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
11915 Replay: started 1, returned:Ok(())
11916 ----------------------------------------------------------------
11917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11918 States: Active Offline Active
11919 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11920 3716 Acked Read 1 Done Sent Done false
11921 3717 Acked Read 7 Done Sent Done false
11922 3718 Acked Read 5 Done Sent Done false
11923 3719 Acked Read 8 Done Sent Done false
11924 3720 Acked Read 5 Done Sent Done false
11925 3721 Acked Read 2 Done Sent Done false
11926 3722 Acked Read 9 Done Sent Done false
11927 3723 Acked Write 5 Done Sent Done false
11928 3724 Acked Read 8 Done Sent Done false
11929 3725 Acked Flush 0 Done Sent Done false
11930 3726 Acked Write 6 Done Sent Done false
11931 3727 Acked Read 3 Done Sent Done false
11932 3728 Acked Write 9 Done Sent Done false
11933 3729 Acked Write 4 Done Sent Done false
11934 3730 Acked Read 5 Done Sent Done false
11935 3731 Acked Write 4 Done Sent Done false
11936 3732 Acked Flush 0 Done Sent Done false
11937 3733 Acked Read 1 Done Sent Done false
11938 3734 Acked Write 1 Done Sent Done false
11939 3735 Acked Write 7 Done Sent Done false
11940 3736 Acked Flush 0 Done Sent Done false
11941 3737 Acked Write 2 Done Sent Done false
11942 3738 Acked Write 2 Done Sent Done false
11943 3739 Acked Read 5 Done Sent Done false
11944 3740 Acked Read 7 Done Sent Done false
11945 3741 Acked Write 2 Done Sent Done false
11946 3742 Acked Write 5 Done Sent Done false
11947 3743 Acked Read 6 Done Sent Done false
11948 3744 Acked Write 3 Done Sent Done false
11949 3745 Acked Write 6 Done Sent Done false
11950 3746 Acked Flush 0 Done Sent Done false
11951 3747 Acked Write 3 Done Sent Done false
11952 3748 Acked Read 8 Done Sent Done false
11953 3749 Acked Read 8 Done Sent Done false
11954 3750 Acked Read 3 Done Sent Done false
11955 3751 Acked Read 6 Done Sent Done false
11956 3752 Acked Write 10 Done Sent Done false
11957 3753 Acked Write 6 Done Sent Done false
11958 3754 Acked Read 4 Done Sent Done false
11959 3755 Acked Read 5 Done Sent Done false
11960 3756 Acked Read 7 Done Sent Done false
11961 3757 Acked Read 5 Done Sent Done false
11962 3758 Acked Read 10 Done Sent Done false
11963 3759 Acked Write 1 Done Sent Done false
11964 3760 Acked Read 8 Done Sent Done false
11965 3761 Acked Flush 0 Done Sent Done false
11966 3762 Acked Read 7 Done Sent Done false
11967 3763 Acked Write 8 Done Sent Done false
11968 3764 Acked Read 1 Done Sent Done false
11969 3765 Acked Read 5 Done Sent Done false
11970 3766 Acked Write 6 Done Sent Done false
11971 3767 Acked Write 2 Done Sent Done false
11972 3768 Acked Read 6 Done Sent Done false
11973 3769 Acked Read 4 Done Sent Done false
11974 3770 Acked Write 8 Done Sent Done false
11975 3771 Acked Write 7 Done Sent Done false
11976 3772 Acked Flush 0 Done Sent Done false
11977 3773 Acked Write 6 Done Sent Done false
11978 3774 Acked Write 5 Done Sent Done false
11979 3775 Acked Read 1 Done Sent Done false
11980 3776 Acked Read 5 Done Sent Done false
11981 3777 Acked Read 9 Done Sent Done false
11982 3778 Acked Read 3 Done Sent Done false
11983 3779 Acked Write 7 Done Sent Done false
11984 3780 Acked Read 2 Done Sent Done false
11985 3781 Acked Read 5 Done Sent Done false
11986 3782 Acked Flush 0 Done Sent Done false
11987 3783 Acked Write 3 Done Sent Done false
11988 3784 Acked Read 9 Done Sent Done false
11989 3785 Acked Write 2 Done Sent Done false
11990 3786 Acked Read 2 Done Sent Done false
11991 3787 Acked Write 5 Done Sent Done false
11992 3788 Acked Write 5 Done Sent Done false
11993 3789 Acked Write 7 Done Sent Done false
11994 3790 Acked Read 9 Done Sent Done false
11995 3791 Acked Write 9 Done Sent Done false
11996 3792 Acked Read 10 Done Sent Done false
11997 3793 Acked Read 8 Done Sent Done false
11998 3794 Acked Write 8 Done Sent Done false
11999 3795 Acked Read 9 Done Sent Done false
12000 3796 Acked Read 3 Done Sent Done false
12001 3797 Acked Write 10 Done Sent Done false
12002 3798 Acked Read 3 Done Sent Done false
12003 3799 Acked Read 9 Done Sent Done false
12004 3800 Acked Read 8 Done Sent Done false
12005 3801 Acked Write 3 Done Sent Done false
12006 3802 Acked Read 2 Done Sent Done false
12007 3803 Acked Read 7 Done Sent Done false
12008 3804 Acked Write 1 Done Sent Done false
12009 3805 Acked Flush 0 Done Sent Done false
12010 3806 Acked Write 10 Done Sent Done false
12011 3807 Acked Read 10 Done Sent Done false
12012 3808 Acked Read 6 Done Sent Done false
12013 3809 Acked Read 10 Done Sent Done false
12014 3810 Acked Write 9 Done Sent Done false
12015 3811 Acked Write 10 Done Sent Done false
12016 3812 Acked Read 1 Done Sent Done false
12017 3813 Acked Read 5 Done Sent Done false
12018 3814 Acked Write 3 Done Sent Done false
12019 3815 Acked Write 6 Done Sent Done false
12020 3816 Acked Write 1 Done Sent Done false
12021 3817 Acked Write 3 Done Sent Done false
12022 3818 Acked Write 10 Done Sent Done false
12023 3819 Acked Read 5 Done Sent Done false
12024 3820 Acked Flush 0 Done Sent Done false
12025 3821 Acked Write 10 Done Sent Done false
12026 3822 Acked Write 6 Done Sent Done false
12027 3823 Acked Read 5 Done Sent Done false
12028 3824 Acked Read 2 Done Sent Done false
12029 3825 Acked Read 10 Done Sent Done false
12030 3826 Acked Read 2 Done Sent Done false
12031 3827 Acked Write 9 Done Sent Done false
12032 3828 Acked Read 3 Done Sent Done false
12033 3829 Acked Write 2 Done Sent Done false
12034 3830 Acked Read 3 Done Sent Done false
12035 3831 Acked Read 6 Done Sent Done false
12036 3832 Acked Write 6 Done Sent Done false
12037 3833 Acked Flush 0 Done Sent Done false
12038 3834 Acked Read 7 Done Sent Done false
12039 3835 Acked Read 1 Done Sent Done false
12040 3836 Acked Write 10 Done Sent Done false
12041 3837 Acked Write 9 Done Sent Done false
12042 3838 Acked Write 3 Done Sent Done false
12043 3839 Acked Read 2 Done Sent Done false
12044 3840 Acked Write 9 Done Sent Done false
12045 3841 Acked Read 6 Done Sent Done false
12046 3842 Acked Flush 0 Done Sent Done false
12047 3843 Acked Read 3 Done Sent Done false
12048 3844 Acked Read 10 Done Sent Done false
12049 3845 Acked Write 1 Done Sent Done false
12050 3846 Acked Write 6 Done Sent Done false
12051 3847 Acked Write 9 Done Sent Done false
12052 3848 Acked Write 1 Done Sent Done false
12053 3849 Acked Read 7 Done Sent Done false
12054 3850 Acked Read 4 Done Sent Done false
12055 3851 Acked Read 4 Done Sent Done false
12056 3852 Acked Write 2 Done Sent Done false
12057 3853 Acked Read 2 Done Sent Done false
12058 3854 Acked Flush 0 Done Sent Done false
12059 3855 Acked Write 1 Done Sent Done false
12060 3856 Acked Write 1 Done Sent Done false
12061 3857 Acked Read 7 Done Sent Done false
12062 3858 Acked Write 3 Done Sent Done false
12063 3859 Acked Write 7 Done Sent Done false
12064 3860 Acked Write 4 Done Sent Done false
12065 3861 Acked Read 7 Done Sent Done false
12066 3862 Acked Read 6 Done Sent Done false
12067 3863 Acked Read 9 Done Sent Done false
12068 3864 Acked Flush 0 Done Sent Done false
12069 3865 Acked Read 4 Done Sent Done false
12070 3866 Acked Write 10 Done Sent Done false
12071 3867 Acked Write 3 Done Sent Done false
12072 3868 Acked Flush 0 Done Sent Done false
12073 3869 Acked Write 4 Done Sent Done false
12074 3870 Acked Read 6 Done Sent Done false
12075 3871 Acked Write 8 Done Sent Done false
12076 3872 Acked Read 4 Done Sent Done false
12077 3873 Acked Read 7 Done Sent Done false
12078 3874 Acked Read 8 Done Sent Done false
12079 3875 Acked Read 3 Done Sent Done false
12080 3876 Acked Write 1 Done Sent Done false
12081 3877 Acked Read 5 Done Sent Done false
12082 3878 Acked Read 8 Done Sent Done false
12083 3879 Acked Read 7 Done Sent Done false
12084 3880 Acked Read 4 Done Sent Done false
12085 3881 Acked Write 2 Done Sent Done false
12086 3882 Acked Read 4 Done Sent Done false
12087 3883 Acked Write 7 Done Sent Done false
12088 3884 Acked Write 3 Done Sent Done false
12089 3885 Acked Read 1 Done Sent Done false
12090 3886 Acked Read 6 Done Sent Done false
12091 3887 Acked Write 8 Done Sent Done false
12092 3888 Acked Write 9 Done Sent Done false
12093 3889 Acked Read 5 Done Sent Done false
12094 3890 Acked Read 4 Done Sent Done false
12095 3891 Acked Write 1 Done Sent Done false
12096 3892 Acked Read 10 Done Sent Done false
12097 3893 Acked Write 7 Done Sent Done false
12098 3894 Acked Read 1 Done Sent Done false
12099 3895 Acked Read 2 Done Sent Done false
12100 3896 Acked Read 3 Done Sent Done false
12101 3897 Acked Read 9 Done Sent Done false
12102 3898 Acked Flush 0 Done Sent Done false
12103 3899 Acked Read 6 Done Sent Done false
12104 3900 Acked Write 9 Done Sent Done false
12105 3901 Acked Flush 0 Done Sent Done false
12106 3902 Acked Flush 0 Done Sent Done false
12107 3903 Acked Write 6 Done Sent Done false
12108 3904 Acked Read 4 Done Sent Done false
12109 3905 Acked Write 2 Done Sent Done false
12110 3906 Acked Write 7 Done Sent Done false
12111 3907 Acked Read 4 Done Sent Done false
12112 3908 Acked Write 1 Done Sent Done false
12113 3909 Acked Read 1 Done Sent Done false
12114 3910 Acked Flush 0 Done Sent Done false
12115 3911 Acked Read 5 Done Sent Done false
12116 3912 Acked Read 10 Done Sent Done false
12117 3913 Acked Read 7 Done Sent Done false
12118 3914 Acked Flush 0 Done Sent Done false
12119 3915 Acked Write 8 Done Sent Done false
12120 3916 Acked Flush 0 Done Sent Done false
12121 3917 Acked Flush 0 Done Sent Done false
12122 3918 Acked Flush 0 Done Sent Done false
12123 3919 Acked Read 5 Done Sent Done false
12124 3920 Acked Read 8 Done Sent Done false
12125 3921 Acked Read 10 Done Sent Done false
12126 3922 Acked Read 6 Done Sent Done false
12127 3923 Acked Write 10 Done Sent Done false
12128 3924 Acked Read 1 Done Sent Done false
12129 3925 Acked Read 9 Done Sent Done false
12130 3926 Acked Write 2 Done Sent Done false
12131 3927 Acked Write 7 Done Sent Done false
12132 3928 Acked Write 6 Done Sent Done false
12133 3929 Acked Read 9 Done Sent Done false
12134 3930 Acked Write 3 Done Sent Done false
12135 3931 Acked Write 1 Done Sent Done false
12136 3932 Acked Write 3 Done Sent Done false
12137 3933 Acked Write 9 Done Sent Done false
12138 3934 Acked Write 10 Done Sent Done false
12139 3935 Acked Write 2 Done Sent Done false
12140 3936 Acked Read 8 Done Sent Done false
12141 3937 Acked Read 4 Done Sent Done false
12142 3938 Acked Write 9 Done Sent Done false
12143 3939 Acked Read 1 Done Sent Done false
12144 3940 Acked Read 1 Done Sent Done false
12145 3941 Acked Read 6 Done Sent Done false
12146 3942 Acked Flush 0 Done Sent Done false
12147 3943 Acked Read 8 Done Sent Done false
12148 3944 Acked Write 2 Done Sent Done false
12149 3945 Acked Write 4 Done Sent Done false
12150 3946 Acked Read 7 Done Sent Done false
12151 3947 Acked Read 3 Done Sent Done false
12152 3948 Acked Read 4 Done Sent Done false
12153 3949 Acked Read 7 Done Sent Done false
12154 3950 Acked Read 1 Done Sent Done false
12155 3951 Acked Write 5 Done Sent Done false
12156 3952 Acked Read 8 Done Sent Done false
12157 3953 Acked Read 5 Done Sent Done false
12158 3954 Acked Flush 0 Done Sent Done false
12159 3955 Acked Read 5 Done Sent Done false
12160 3956 Acked Read 5 Done Sent Done false
12161 3957 Acked Read 1 Done Sent Done false
12162 3958 Acked Write 7 Done Sent Done false
12163 3959 Acked Write 8 Done Sent Done false
12164 3960 Acked Read 6 Done Sent Done false
12165 3961 Acked Read 7 Done Sent Done false
12166 3962 Acked Read 2 Done Sent Done false
12167 3963 Acked Read 6 Done Sent Done false
12168 3964 Acked Read 7 Done Sent Done false
12169 3965 Acked Write 8 Done Sent Done false
12170 3966 Acked Write 5 Done Sent Done false
12171 3967 Acked Read 5 Done Sent Done false
12172 3968 Acked Write 2 Done Sent Done false
12173 3969 Acked Read 4 Done Sent Done false
12174 3970 Acked Read 10 Done Sent Done false
12175 3971 Acked Read 6 Done Sent Done false
12176 3972 Acked Write 6 Done Sent Done false
12177 3973 Acked Read 10 Done Sent Done false
12178 3974 Acked Flush 0 Done Sent Done false
12179 3975 Acked Read 10 Done Sent Done false
12180 3976 Acked Read 8 Done Sent Done false
12181 3977 Acked Flush 0 Done Sent Done false
12182 3978 Acked Read 2 Done Sent Done false
12183 3979 Acked Write 5 Done Sent Done false
12184 3980 Acked Write 8 Done Sent Done false
12185 3981 Acked Read 5 Done Sent Done false
12186 3982 Acked Write 9 Done Sent Done false
12187 3983 Acked Write 10 Done Sent Done false
12188 3984 Acked Read 9 Done Sent Done false
12189 3985 Acked Read 5 Done Sent Done false
12190 3986 Acked Write 4 Done Sent Done false
12191 3987 Acked Write 7 Done Sent Done false
12192 3988 Acked Read 9 Done Sent Done false
12193 3989 Acked Read 9 Done Sent Done false
12194 3990 Acked Read 10 Done Sent Done false
12195 3991 Acked Read 3 Done Sent Done false
12196 3992 Acked Write 3 Done Sent Done false
12197 3993 Acked Write 5 Done Sent Done false
12198 3994 Acked Write 5 Done Sent Done false
12199 3995 Acked Write 2 Done Sent Done false
12200 3996 Acked Read 9 Done Sent Done false
12201 3997 Acked Write 10 Done Sent Done false
12202 3998 Acked Read 4 Done Sent Done false
12203 3999 Acked Flush 0 Done Sent Done false
12204 4000 Acked Write 6 Done Sent Done false
12205 4001 Acked Read 8 Done Sent Done false
12206 4002 Acked Read 9 Done Sent Done false
12207 4003 Acked Read 9 Done Sent Done false
12208 4004 Acked Read 9 Done Sent Done false
12209 4005 Acked Flush 0 Done Sent Done false
12210 4006 Acked Write 4 Done Sent Done false
12211 4007 Acked Read 3 Done Sent Done false
12212 4008 Acked Flush 0 Done Sent Done false
12213 4009 Acked Write 10 Done Sent Done false
12214 4010 Acked Read 5 Done Sent Done false
12215 4011 Acked Write 8 Done Sent Done false
12216 4012 Acked Read 10 Done Sent Done false
12217 4013 Acked Read 3 Done Sent Done false
12218 4014 Acked Read 10 Done Sent Done false
12219 4015 Acked Read 7 Done Sent Done false
12220 4016 Acked Read 1 Done Sent Done false
12221 STATES DS:0 DS:1 DS:2 TOTAL
12222 Sent 0 301 0 301
12223 Done 301 0 301 602
12224 Skipped 0 0 0 0
12225 Error 0 0 0 0
12226 Last Flush: Some(JobId(4008)) Some(JobId(3715)) Some(JobId(4008))
12227 Downstairs last five completed: 3715 3714 3713 3712 3711
12228 Upstairs last five completed: 4016 4015 4014 4013 4012
12229 CLIENT: Up:0 ds:301 act:2
122302025-03-07T18:15:11.552ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
122312025-03-07T18:15:12.273ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
12232 [8820] got start action
12233 [8820] I am going to restart
12234 Starting downstairs at port 8820
12235 Make output file at "/tmp/dsc/downstairs-8820.txt"
12236 [8820][1] reports Starting
12237 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
12238 [8820][1] reports Running
122392025-03-07T18:15:13.206ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
122402025-03-07T18:15:14.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
12241 ----------------------------------------------------------------
12242 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12243 States: Active Offline Active
12244 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12245 3716 Acked Read 1 Done Sent Done false
12246 3717 Acked Read 7 Done Sent Done false
12247 3718 Acked Read 5 Done Sent Done false
12248 3719 Acked Read 8 Done Sent Done false
12249 3720 Acked Read 5 Done Sent Done false
12250 3721 Acked Read 2 Done Sent Done false
12251 3722 Acked Read 9 Done Sent Done false
12252 3723 Acked Write 5 Done Sent Done false
12253 3724 Acked Read 8 Done Sent Done false
12254 3725 Acked Flush 0 Done Sent Done false
12255 3726 Acked Write 6 Done Sent Done false
12256 3727 Acked Read 3 Done Sent Done false
12257 3728 Acked Write 9 Done Sent Done false
12258 3729 Acked Write 4 Done Sent Done false
12259 3730 Acked Read 5 Done Sent Done false
12260 3731 Acked Write 4 Done Sent Done false
12261 3732 Acked Flush 0 Done Sent Done false
12262 3733 Acked Read 1 Done Sent Done false
12263 3734 Acked Write 1 Done Sent Done false
12264 3735 Acked Write 7 Done Sent Done false
12265 3736 Acked Flush 0 Done Sent Done false
12266 3737 Acked Write 2 Done Sent Done false
12267 3738 Acked Write 2 Done Sent Done false
12268 3739 Acked Read 5 Done Sent Done false
12269 3740 Acked Read 7 Done Sent Done false
12270 3741 Acked Write 2 Done Sent Done false
12271 3742 Acked Write 5 Done Sent Done false
12272 3743 Acked Read 6 Done Sent Done false
12273 3744 Acked Write 3 Done Sent Done false
12274 3745 Acked Write 6 Done Sent Done false
12275 3746 Acked Flush 0 Done Sent Done false
12276 3747 Acked Write 3 Done Sent Done false
12277 3748 Acked Read 8 Done Sent Done false
12278 3749 Acked Read 8 Done Sent Done false
12279 3750 Acked Read 3 Done Sent Done false
12280 3751 Acked Read 6 Done Sent Done false
12281 3752 Acked Write 10 Done Sent Done false
12282 3753 Acked Write 6 Done Sent Done false
12283 3754 Acked Read 4 Done Sent Done false
12284 3755 Acked Read 5 Done Sent Done false
12285 3756 Acked Read 7 Done Sent Done false
12286 3757 Acked Read 5 Done Sent Done false
12287 3758 Acked Read 10 Done Sent Done false
12288 3759 Acked Write 1 Done Sent Done false
12289 3760 Acked Read 8 Done Sent Done false
12290 3761 Acked Flush 0 Done Sent Done false
12291 3762 Acked Read 7 Done Sent Done false
12292 3763 Acked Write 8 Done Sent Done false
12293 3764 Acked Read 1 Done Sent Done false
12294 3765 Acked Read 5 Done Sent Done false
12295 3766 Acked Write 6 Done Sent Done false
12296 3767 Acked Write 2 Done Sent Done false
12297 3768 Acked Read 6 Done Sent Done false
12298 3769 Acked Read 4 Done Sent Done false
12299 3770 Acked Write 8 Done Sent Done false
12300 3771 Acked Write 7 Done Sent Done false
12301 3772 Acked Flush 0 Done Sent Done false
12302 3773 Acked Write 6 Done Sent Done false
12303 3774 Acked Write 5 Done Sent Done false
12304 3775 Acked Read 1 Done Sent Done false
12305 3776 Acked Read 5 Done Sent Done false
12306 3777 Acked Read 9 Done Sent Done false
12307 3778 Acked Read 3 Done Sent Done false
12308 3779 Acked Write 7 Done Sent Done false
12309 3780 Acked Read 2 Done Sent Done false
12310 3781 Acked Read 5 Done Sent Done false
12311 3782 Acked Flush 0 Done Sent Done false
12312 3783 Acked Write 3 Done Sent Done false
12313 3784 Acked Read 9 Done Sent Done false
12314 3785 Acked Write 2 Done Sent Done false
12315 3786 Acked Read 2 Done Sent Done false
12316 3787 Acked Write 5 Done Sent Done false
12317 3788 Acked Write 5 Done Sent Done false
12318 3789 Acked Write 7 Done Sent Done false
12319 3790 Acked Read 9 Done Sent Done false
12320 3791 Acked Write 9 Done Sent Done false
12321 3792 Acked Read 10 Done Sent Done false
12322 3793 Acked Read 8 Done Sent Done false
12323 3794 Acked Write 8 Done Sent Done false
12324 3795 Acked Read 9 Done Sent Done false
12325 3796 Acked Read 3 Done Sent Done false
12326 3797 Acked Write 10 Done Sent Done false
12327 3798 Acked Read 3 Done Sent Done false
12328 3799 Acked Read 9 Done Sent Done false
12329 3800 Acked Read 8 Done Sent Done false
12330 3801 Acked Write 3 Done Sent Done false
12331 3802 Acked Read 2 Done Sent Done false
12332 3803 Acked Read 7 Done Sent Done false
12333 3804 Acked Write 1 Done Sent Done false
12334 3805 Acked Flush 0 Done Sent Done false
12335 3806 Acked Write 10 Done Sent Done false
12336 3807 Acked Read 10 Done Sent Done false
12337 3808 Acked Read 6 Done Sent Done false
12338 3809 Acked Read 10 Done Sent Done false
12339 3810 Acked Write 9 Done Sent Done false
12340 3811 Acked Write 10 Done Sent Done false
12341 3812 Acked Read 1 Done Sent Done false
12342 3813 Acked Read 5 Done Sent Done false
12343 3814 Acked Write 3 Done Sent Done false
12344 3815 Acked Write 6 Done Sent Done false
12345 3816 Acked Write 1 Done Sent Done false
12346 3817 Acked Write 3 Done Sent Done false
12347 3818 Acked Write 10 Done Sent Done false
12348 3819 Acked Read 5 Done Sent Done false
12349 3820 Acked Flush 0 Done Sent Done false
12350 3821 Acked Write 10 Done Sent Done false
12351 3822 Acked Write 6 Done Sent Done false
12352 3823 Acked Read 5 Done Sent Done false
12353 3824 Acked Read 2 Done Sent Done false
12354 3825 Acked Read 10 Done Sent Done false
12355 3826 Acked Read 2 Done Sent Done false
12356 3827 Acked Write 9 Done Sent Done false
12357 3828 Acked Read 3 Done Sent Done false
12358 3829 Acked Write 2 Done Sent Done false
12359 3830 Acked Read 3 Done Sent Done false
12360 3831 Acked Read 6 Done Sent Done false
12361 3832 Acked Write 6 Done Sent Done false
12362 3833 Acked Flush 0 Done Sent Done false
12363 3834 Acked Read 7 Done Sent Done false
12364 3835 Acked Read 1 Done Sent Done false
12365 3836 Acked Write 10 Done Sent Done false
12366 3837 Acked Write 9 Done Sent Done false
12367 3838 Acked Write 3 Done Sent Done false
12368 3839 Acked Read 2 Done Sent Done false
12369 3840 Acked Write 9 Done Sent Done false
12370 3841 Acked Read 6 Done Sent Done false
12371 3842 Acked Flush 0 Done Sent Done false
12372 3843 Acked Read 3 Done Sent Done false
12373 3844 Acked Read 10 Done Sent Done false
12374 3845 Acked Write 1 Done Sent Done false
12375 3846 Acked Write 6 Done Sent Done false
12376 3847 Acked Write 9 Done Sent Done false
12377 3848 Acked Write 1 Done Sent Done false
12378 3849 Acked Read 7 Done Sent Done false
12379 3850 Acked Read 4 Done Sent Done false
12380 3851 Acked Read 4 Done Sent Done false
12381 3852 Acked Write 2 Done Sent Done false
12382 3853 Acked Read 2 Done Sent Done false
12383 3854 Acked Flush 0 Done Sent Done false
12384 3855 Acked Write 1 Done Sent Done false
12385 3856 Acked Write 1 Done Sent Done false
12386 3857 Acked Read 7 Done Sent Done false
12387 3858 Acked Write 3 Done Sent Done false
12388 3859 Acked Write 7 Done Sent Done false
12389 3860 Acked Write 4 Done Sent Done false
12390 3861 Acked Read 7 Done Sent Done false
12391 3862 Acked Read 6 Done Sent Done false
12392 3863 Acked Read 9 Done Sent Done false
12393 3864 Acked Flush 0 Done Sent Done false
12394 3865 Acked Read 4 Done Sent Done false
12395 3866 Acked Write 10 Done Sent Done false
12396 3867 Acked Write 3 Done Sent Done false
12397 3868 Acked Flush 0 Done Sent Done false
12398 3869 Acked Write 4 Done Sent Done false
12399 3870 Acked Read 6 Done Sent Done false
12400 3871 Acked Write 8 Done Sent Done false
12401 3872 Acked Read 4 Done Sent Done false
12402 3873 Acked Read 7 Done Sent Done false
12403 3874 Acked Read 8 Done Sent Done false
12404 3875 Acked Read 3 Done Sent Done false
12405 3876 Acked Write 1 Done Sent Done false
12406 3877 Acked Read 5 Done Sent Done false
12407 3878 Acked Read 8 Done Sent Done false
12408 3879 Acked Read 7 Done Sent Done false
12409 3880 Acked Read 4 Done Sent Done false
12410 3881 Acked Write 2 Done Sent Done false
12411 3882 Acked Read 4 Done Sent Done false
12412 3883 Acked Write 7 Done Sent Done false
12413 3884 Acked Write 3 Done Sent Done false
12414 3885 Acked Read 1 Done Sent Done false
12415 3886 Acked Read 6 Done Sent Done false
12416 3887 Acked Write 8 Done Sent Done false
12417 3888 Acked Write 9 Done Sent Done false
12418 3889 Acked Read 5 Done Sent Done false
12419 3890 Acked Read 4 Done Sent Done false
12420 3891 Acked Write 1 Done Sent Done false
12421 3892 Acked Read 10 Done Sent Done false
12422 3893 Acked Write 7 Done Sent Done false
12423 3894 Acked Read 1 Done Sent Done false
12424 3895 Acked Read 2 Done Sent Done false
12425 3896 Acked Read 3 Done Sent Done false
12426 3897 Acked Read 9 Done Sent Done false
12427 3898 Acked Flush 0 Done Sent Done false
12428 3899 Acked Read 6 Done Sent Done false
12429 3900 Acked Write 9 Done Sent Done false
12430 3901 Acked Flush 0 Done Sent Done false
12431 3902 Acked Flush 0 Done Sent Done false
12432 3903 Acked Write 6 Done Sent Done false
12433 3904 Acked Read 4 Done Sent Done false
12434 3905 Acked Write 2 Done Sent Done false
12435 3906 Acked Write 7 Done Sent Done false
12436 3907 Acked Read 4 Done Sent Done false
12437 3908 Acked Write 1 Done Sent Done false
12438 3909 Acked Read 1 Done Sent Done false
12439 3910 Acked Flush 0 Done Sent Done false
12440 3911 Acked Read 5 Done Sent Done false
12441 3912 Acked Read 10 Done Sent Done false
12442 3913 Acked Read 7 Done Sent Done false
12443 3914 Acked Flush 0 Done Sent Done false
12444 3915 Acked Write 8 Done Sent Done false
12445 3916 Acked Flush 0 Done Sent Done false
12446 3917 Acked Flush 0 Done Sent Done false
12447 3918 Acked Flush 0 Done Sent Done false
12448 3919 Acked Read 5 Done Sent Done false
12449 3920 Acked Read 8 Done Sent Done false
12450 3921 Acked Read 10 Done Sent Done false
12451 3922 Acked Read 6 Done Sent Done false
12452 3923 Acked Write 10 Done Sent Done false
12453 3924 Acked Read 1 Done Sent Done false
12454 3925 Acked Read 9 Done Sent Done false
12455 3926 Acked Write 2 Done Sent Done false
12456 3927 Acked Write 7 Done Sent Done false
12457 3928 Acked Write 6 Done Sent Done false
12458 3929 Acked Read 9 Done Sent Done false
12459 3930 Acked Write 3 Done Sent Done false
12460 3931 Acked Write 1 Done Sent Done false
12461 3932 Acked Write 3 Done Sent Done false
12462 3933 Acked Write 9 Done Sent Done false
12463 3934 Acked Write 10 Done Sent Done false
12464 3935 Acked Write 2 Done Sent Done false
12465 3936 Acked Read 8 Done Sent Done false
12466 3937 Acked Read 4 Done Sent Done false
12467 3938 Acked Write 9 Done Sent Done false
12468 3939 Acked Read 1 Done Sent Done false
12469 3940 Acked Read 1 Done Sent Done false
12470 3941 Acked Read 6 Done Sent Done false
12471 3942 Acked Flush 0 Done Sent Done false
12472 3943 Acked Read 8 Done Sent Done false
12473 3944 Acked Write 2 Done Sent Done false
12474 3945 Acked Write 4 Done Sent Done false
12475 3946 Acked Read 7 Done Sent Done false
12476 3947 Acked Read 3 Done Sent Done false
12477 3948 Acked Read 4 Done Sent Done false
12478 3949 Acked Read 7 Done Sent Done false
12479 3950 Acked Read 1 Done Sent Done false
12480 3951 Acked Write 5 Done Sent Done false
12481 3952 Acked Read 8 Done Sent Done false
12482 3953 Acked Read 5 Done Sent Done false
12483 3954 Acked Flush 0 Done Sent Done false
12484 3955 Acked Read 5 Done Sent Done false
12485 3956 Acked Read 5 Done Sent Done false
12486 3957 Acked Read 1 Done Sent Done false
12487 3958 Acked Write 7 Done Sent Done false
12488 3959 Acked Write 8 Done Sent Done false
12489 3960 Acked Read 6 Done Sent Done false
12490 3961 Acked Read 7 Done Sent Done false
12491 3962 Acked Read 2 Done Sent Done false
12492 3963 Acked Read 6 Done Sent Done false
12493 3964 Acked Read 7 Done Sent Done false
12494 3965 Acked Write 8 Done Sent Done false
12495 3966 Acked Write 5 Done Sent Done false
12496 3967 Acked Read 5 Done Sent Done false
12497 3968 Acked Write 2 Done Sent Done false
12498 3969 Acked Read 4 Done Sent Done false
12499 3970 Acked Read 10 Done Sent Done false
12500 3971 Acked Read 6 Done Sent Done false
12501 3972 Acked Write 6 Done Sent Done false
12502 3973 Acked Read 10 Done Sent Done false
12503 3974 Acked Flush 0 Done Sent Done false
12504 3975 Acked Read 10 Done Sent Done false
12505 3976 Acked Read 8 Done Sent Done false
12506 3977 Acked Flush 0 Done Sent Done false
12507 3978 Acked Read 2 Done Sent Done false
12508 3979 Acked Write 5 Done Sent Done false
12509 3980 Acked Write 8 Done Sent Done false
12510 3981 Acked Read 5 Done Sent Done false
12511 3982 Acked Write 9 Done Sent Done false
12512 3983 Acked Write 10 Done Sent Done false
12513 3984 Acked Read 9 Done Sent Done false
12514 3985 Acked Read 5 Done Sent Done false
12515 3986 Acked Write 4 Done Sent Done false
12516 3987 Acked Write 7 Done Sent Done false
12517 3988 Acked Read 9 Done Sent Done false
12518 3989 Acked Read 9 Done Sent Done false
12519 3990 Acked Read 10 Done Sent Done false
12520 3991 Acked Read 3 Done Sent Done false
12521 3992 Acked Write 3 Done Sent Done false
12522 3993 Acked Write 5 Done Sent Done false
12523 3994 Acked Write 5 Done Sent Done false
12524 3995 Acked Write 2 Done Sent Done false
12525 3996 Acked Read 9 Done Sent Done false
12526 3997 Acked Write 10 Done Sent Done false
12527 3998 Acked Read 4 Done Sent Done false
12528 3999 Acked Flush 0 Done Sent Done false
12529 4000 Acked Write 6 Done Sent Done false
12530 4001 Acked Read 8 Done Sent Done false
12531 4002 Acked Read 9 Done Sent Done false
12532 4003 Acked Read 9 Done Sent Done false
12533 4004 Acked Read 9 Done Sent Done false
12534 4005 Acked Flush 0 Done Sent Done false
12535 4006 Acked Write 4 Done Sent Done false
12536 4007 Acked Read 3 Done Sent Done false
12537 4008 Acked Flush 0 Done Sent Done false
12538 4009 Acked Write 10 Done Sent Done false
12539 4010 Acked Read 5 Done Sent Done false
12540 4011 Acked Write 8 Done Sent Done false
12541 4012 Acked Read 10 Done Sent Done false
12542 4013 Acked Read 3 Done Sent Done false
12543 4014 Acked Read 10 Done Sent Done false
12544 4015 Acked Read 7 Done Sent Done false
12545 4016 Acked Read 1 Done Sent Done false
12546 STATES DS:0 DS:1 DS:2 TOTAL
12547 Sent 0 301 0 301
12548 Done 301 0 301 602
12549 Skipped 0 0 0 0
12550 Error 0 0 0 0
12551 Last Flush: Some(JobId(4008)) Some(JobId(3715)) Some(JobId(4008))
12552 Downstairs last five completed: 3715 3714 3713 3712 3711
12553 Upstairs last five completed: 4016 4015 4014 4013 4012
12554 CLIENT: Up:0 ds:301 act:2
125552025-03-07T18:15:15.554ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125562025-03-07T18:15:16.212ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125572025-03-07T18:15:16.795ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125582025-03-07T18:15:16.795ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59251) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125592025-03-07T18:15:16.796ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125602025-03-07T18:15:16.797ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125612025-03-07T18:15:16.797ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125622025-03-07T18:15:16.797ZINFOcrucible: send last flush ID to this DS: Some(JobId(3715)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125632025-03-07T18:15:16.797ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125642025-03-07T18:15:16.797ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3715)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
125652025-03-07T18:15:16.797ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3715)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
12566 ----------------------------------------------------------------
12567 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12568 Downstairs last five completed: 4017 4016 4015 4014 4013
12569 Upstairs last five completed: 4017 4016 4015 4014 4013
12570 CLIENT: Up:0 ds:0 act:3
12571 Replay: All jobs finished
12572 Main task has work to do, go find it
12573 got dsc Stop(0)
12574 stop 0
12575 Mar 07 18:15:19.056 [8810] Got stop action so:false kr:false
12576 INFO request completed, latency_us: 325, response_code: 204, uri: /stop/cid/0, method: POST, req_id: c43c6d9a-b806-4846-94ab-264bc86a6b28, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
125772025-03-07T18:15:19.057ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
12578 Mar 07 18:15:19.057 INFO request completed, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:19.057108529Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
12579 latency_us: 156, response_code: 200{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:19.057131747Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
12580 , uri: /state/cid/0, method: GET, req_id: 71866155-168d-47d8-8d6b-64319c591f36, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
125812025-03-07T18:15:19.057ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
12582 [8810] Exited with: ExitStatus(unix_wait_status(9))
12583 [8810][0] reports Exit
12584 Mar 07 18:15:23.058 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/0, method: GET, req_id: 590572e6-ff53-497a-af8a-bd862f877f55, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
12585 001/300 Read block 2357 len 32768
12586 002/300 Flush
12587 003/300 Read block 2897 len 40960
12588 004/300 Write block 825 len 4096 data: 5
12589 005/300 Read block 3472 len 4096
12590 006/300 Read block 4156 len 40960
12591 007/300 Read block 1883 len 20480
12592 008/300 Read block 4530 len 20480
12593 009/300 Flush
12594 010/300 Write block 2113 len 36864 data: 3 3 3 2 2 3 4 4 4
12595 011/300 Read block 3791 len 4096
12596 012/300 Flush
12597 013/300 Write block 783 len 40960 data: 3 4 3 4 4 4 4 4 4 3
12598 014/300 Read block 38 len 32768
12599 015/300 Write block 108 len 4096 data: 2
12600 016/300 Write block 4589 len 16384 data: 4 4 2 3
12601 017/300 Write block 1981 len 4096 data: 3
12602 018/300 Flush
12603 019/300 Read block 1404 len 40960
12604 020/300 Write block 1155 len 16384 data: 4 3 3 3
12605 021/300 Write block 3948 len 36864 data: 4 4 3 3 3 3 2 2 3
12606 022/300 Read block 4751 len 24576
12607 023/300 Read block 2073 len 12288
12608 024/300 Write block 3263 len 40960 data: 3 3 3 5 5 4 5 3 2 2
12609 025/300 Write block 1629 len 8192 data: 3 2
12610 026/300 Read block 3965 len 24576
12611 027/300 Write block 2462 len 28672 data: 3 5 5 5 5 5 4
12612 028/300 Read block 554 len 8192
12613 029/300 Read block 3510 len 28672
12614 030/300 Read block 152 len 36864
12615 031/300 Write block 4207 len 36864 data: 4 5 3 3 4 4 4 3 4
12616 032/300 Read block 54 len 40960
12617 033/300 Write block 4128 len 12288 data: 5 5 4
12618 034/300 Write block 4304 len 8192 data: 3 5
12619 035/300 Write block 4241 len 40960 data: 4 4 3 4 4 2 3 5 5 5
12620 036/300 Read block 1936 len 12288
12621 037/300 Read block 4763 len 4096
12622 038/300 Write block 1837 len 40960 data: 4 4 5 4 4 3 3 2 2 4
12623 039/300 Read block 4971 len 32768
12624 040/300 Read block 3775 len 16384
12625 041/300 Write block 2407 len 28672 data: 3 4 5 5 5 4 3
12626 042/300 Flush
12627 043/300 Write block 2229 len 12288 data: 2 2 2
12628 044/300 Read block 359 len 8192
12629 045/300 Write block 4055 len 12288 data: 4 4 3
12630 046/300 Read block 944 len 32768
12631 047/300 Write block 3948 len 28672 data: 5 5 4 4 4 4 3
12632 048/300 Read block 3938 len 40960
12633 049/300 Read block 3063 len 20480
12634 050/300 Write block 3921 len 28672 data: 5 5 3 3 3 3 4
12635 051/300 Write block 692 len 4096 data: 2
12636 052/300 Read block 562 len 8192
12637 053/300 Write block 560 len 20480 data: 2 2 2 2 2
12638 054/300 Write block 3040 len 12288 data: 2 3 3
12639 055/300 Write block 3918 len 32768 data: 4 5 5 6 6 4 4 4
12640 056/300 Read block 3870 len 20480
12641 057/300 Write block 4653 len 12288 data: 3 3 3
12642 058/300 Write block 3566 len 4096 data: 4
12643 059/300 Write block 3850 len 36864 data: 4 4 4 5 3 3 3 4 3
12644 060/300 Read block 4087 len 24576
12645 061/300 Write block 1422 len 12288 data: 3 3 3
12646 062/300 Read block 1968 len 32768
12647 063/300 Write block 2000 len 40960 data: 4 3 2 2 2 3 3 3 4 4
12648 064/300 Write block 859 len 16384 data: 3 2 2 2
12649 065/300 Write block 4669 len 24576 data: 2 2 3 3 2 2
12650 066/300 Read block 3605 len 40960
12651 067/300 Read block 4840 len 4096
12652 068/300 Flush
12653 069/300 Read block 1212 len 40960
12654 070/300 Read block 3011 len 4096
12655 071/300 Write block 4510 len 28672 data: 4 4 3 3 2 2 2
12656 072/300 Read block 621 len 4096
12657 073/300 Read block 4743 len 32768
12658 074/300 Read block 922 len 8192
12659 075/300 Write block 2050 len 40960 data: 5 5 5 4 3 3 3 3 2 3
12660 076/300 Read block 2828 len 24576
12661 077/300 Read block 3208 len 16384
12662 078/300 Read block 1019 len 8192
12663 079/300 Write block 4384 len 12288 data: 3 3 3
12664 080/300 Read block 725 len 4096
12665 081/300 Write block 3061 len 24576 data: 5 4 7 6 6 6
12666 082/300 Read block 3314 len 16384
12667 083/300 Write block 889 len 8192 data: 2 3
12668 084/300 Flush
12669 085/300 Write block 347 len 12288 data: 4 5 5
12670 086/300 Read block 1417 len 24576
12671 087/300 Read block 2440 len 28672
12672 088/300 Read block 991 len 24576
12673 089/300 Read block 2390 len 8192
12674 090/300 Write block 4261 len 8192 data: 3 3
12675 091/300 Read block 4258 len 28672
12676 092/300 Read block 880 len 12288
12677 093/300 Read block 4811 len 8192
12678 094/300 Read block 3748 len 12288
12679 095/300 Read block 63 len 32768
12680 096/300 Read block 1730 len 40960
12681 097/300 Write block 2893 len 32768 data: 3 3 2 2 2 3 4 5
12682 098/300 Write block 3252 len 36864 data: 2 2 2 2 2 2 2 2 2
12683 099/300 Write block 3770 len 24576 data: 2 2 2 3 3 4
12684 100/300 Read block 1583 len 12288
12685 101/300 Write block 4965 len 28672 data: 3 2 2 2 2 2 2
12686 102/300 Read block 1448 len 36864
12687 103/300 Read block 463 len 8192
12688 104/300 Read block 4212 len 40960
12689 105/300 Read block 388 len 36864
12690 106/300 Write block 4388 len 40960 data: 4 5 5 4 3 3 2 2 2 2
12691 107/300 Read block 577 len 28672
12692 108/300 Flush
12693 109/300 Flush
12694 110/300 Read block 1686 len 4096
12695 111/300 Read block 3206 len 8192
12696 112/300 Write block 4091 len 4096 data: 2
12697 113/300 Read block 3551 len 16384
12698 114/300 Write block 4963 len 16384 data: 4 4 4 3
12699 115/300 Write block 3303 len 24576 data: 2 2 3 3 3 3
12700 116/300 Flush
12701 117/300 Read block 4821 len 36864
12702 118/300 Read block 1535 len 40960
12703 119/300 Write block 1266 len 28672 data: 3 4 5 4 3 3 3
12704 120/300 Read block 1134 len 20480
12705 121/300 Read block 2862 len 32768
12706 122/300 Read block 1257 len 28672
12707 123/300 Read block 2585 len 12288
12708 124/300 Write block 4923 len 20480 data: 3 3 3 3 2
12709 125/300 Write block 272 len 20480 data: 3 3 3 3 4
12710 126/300 Read block 1957 len 4096
12711 127/300 Write block 1991 len 20480 data: 3 4 4 4 4
12712 128/300 Read block 2607 len 28672
12713 129/300 Write block 1330 len 24576 data: 4 4 4 2 3 3
12714 130/300 Write block 2269 len 40960 data: 3 4 4 5 5 4 2 2 2 2
12715 131/300 Write block 3332 len 24576 data: 3 3 3 3 4 4
12716 132/300 Read block 1108 len 4096
12717 133/300 Read block 2696 len 16384
12718 134/300 Read block 3524 len 4096
12719 135/300 Write block 2563 len 12288 data: 2 2 2
12720 136/300 Flush
12721 137/300 Read block 4670 len 8192
12722 138/300 Write block 2634 len 20480 data: 4 3 2 2 2
12723 139/300 Write block 4425 len 40960 data: 4 4 4 2 2 2 2 2 3 3
12724 140/300 Read block 4178 len 28672
12725 141/300 Flush
12726 142/300 Read block 4110 len 12288
12727 143/300 Write block 656 len 8192 data: 2 2
12728 144/300 Read block 4071 len 4096
12729 145/300 Read block 2148 len 8192
12730 146/300 Write block 3854 len 28672 data: 4 4 4 5 4 3 3
12731 147/300 Read block 2128 len 36864
12732 148/300 Read block 4231 len 4096
12733 149/300 Read block 3418 len 4096
12734 150/300 Flush
12735 151/300 Write block 1314 len 32768 data: 2 2 3 3 4 4 4 4
12736 152/300 Read block 3132 len 32768
12737 153/300 Read block 146 len 8192
12738 154/300 Read block 2014 len 36864
12739 155/300 Read block 1989 len 8192
12740 156/300 Read block 2543 len 28672
12741 157/300 Read block 185 len 20480
12742 158/300 Write block 4257 len 32768 data: 3 4 4 3 4 4 3 3
12743 159/300 Read block 1591 len 40960
12744 160/300 Write block 2063 len 8192 data: 6 6
12745 161/300 Write block 3420 len 32768 data: 5 3 3 3 3 4 5 4
12746 162/300 Write block 2173 len 40960 data: 2 2 2 2 2 2 2 2 2 2
12747 163/300 Read block 3418 len 32768
12748 164/300 Read block 3886 len 12288
12749 165/300 Read block 4560 len 8192
12750 166/300 Write block 4431 len 40960 data: 3 3 4 4 3 3 3 3 3 2
12751 167/300 Write block 1310 len 12288 data: 2 2 2
12752 168/300 Write block 1073 len 20480 data: 3 2 2 2 2
12753 169/300 Write block 758 len 40960 data: 5 5 5 5 5 4 4 3 2 2
12754 170/300 Read block 1074 len 8192
12755 171/300 Read block 874 len 40960
12756 172/300 Write block 3265 len 8192 data: 4 6
12757 173/300 Write block 2730 len 40960 data: 4 4 6 5 4 4 4 4 4 5
12758 174/300 Write block 561 len 32768 data: 3 3 3 3 2 2 2 2
12759 175/300 Flush
12760 176/300 Flush
12761 177/300 Write block 290 len 24576 data: 3 2 2 2 2 2
12762 178/300 Read block 1741 len 12288
12763 179/300 Read block 3486 len 24576
12764 180/300 Read block 3514 len 20480
12765 181/300 Read block 782 len 16384
12766 182/300 Write block 70 len 28672 data: 4 4 5 5 5 5 5
12767 183/300 Write block 4351 len 24576 data: 3 3 4 4 3 4
12768 184/300 Read block 4415 len 4096
12769 185/300 Read block 371 len 32768
12770 186/300 Read block 3256 len 36864
12771 187/300 Write block 2018 len 8192 data: 5 4
12772 188/300 Read block 4447 len 36864
12773 189/300 Read block 2015 len 24576
12774 190/300 Write block 2683 len 12288 data: 4 3 2
12775 191/300 Read block 3057 len 8192
12776 192/300 Read block 1344 len 20480
12777 193/300 Read block 217 len 16384
12778 194/300 Read block 4855 len 40960
12779 195/300 Write block 4603 len 12288 data: 3 2 2
12780 196/300 Write block 2296 len 24576 data: 3 3 3 3 3 3
12781 197/300 Read block 1123 len 40960
12782 198/300 Read block 1857 len 36864
12783 199/300 Read block 204 len 28672
12784 200/300 Write block 3368 len 16384 data: 3 4 3 3
12785 201/300 Write block 3457 len 28672 data: 5 5 4 3 2 2 2
12786 202/300 Read block 1376 len 32768
12787 203/300 Write block 3932 len 4096 data: 4
12788 204/300 Write block 1523 len 8192 data: 3 3
12789 205/300 Write block 2052 len 24576 data: 6 5 4 4 4 4
12790 206/300 Flush
12791 207/300 Write block 737 len 8192 data: 6 5
12792 208/300 Flush
12793 209/300 Write block 792 len 28672 data: 4 3 3 2 3 4 4
12794 210/300 Read block 2904 len 12288
12795 211/300 Write block 1306 len 8192 data: 2 2
12796 212/300 Write block 4695 len 16384 data: 6 6 6 6
12797 213/300 Read block 2450 len 20480
12798 214/300 Read block 1892 len 12288
12799 215/300 Write block 230 len 36864 data: 3 3 3 3 2 2 2 2 2
12800 216/300 Read block 3510 len 32768
12801 217/300 Write block 2802 len 8192 data: 4 3
12802 218/300 Read block 4051 len 40960
12803 219/300 Write block 1512 len 20480 data: 3 3 3 3 4
12804 220/300 Write block 426 len 12288 data: 5 4 4
12805 221/300 Read block 1728 len 8192
12806 222/300 Read block 2635 len 20480
12807 223/300 Read block 2443 len 28672
12808 224/300 Flush
12809 225/300 Write block 144 len 24576 data: 2 2 2 2 3 3
12810 226/300 Write block 1890 len 20480 data: 3 2 2 3 3
12811 227/300 Read block 1428 len 20480
12812 228/300 Write block 4325 len 40960 data: 3 3 3 5 4 5 5 5 5 5
12813 229/300 Write block 4671 len 24576 data: 4 4 3 3 2 2
12814 230/300 Write block 2319 len 8192 data: 2 2
12815 231/300 Write block 1743 len 20480 data: 3 3 5 3 3
12816 232/300 Read block 3021 len 8192
12817 233/300 Write block 3949 len 8192 data: 6 5
12818 234/300 Read block 492 len 32768
12819 235/300 Read block 1169 len 36864
12820 236/300 Write block 1988 len 8192 data: 3 3
12821 237/300 Read block 3458 len 8192
12822 238/300 Write block 4862 len 20480 data: 2 2 2 2 2
12823 239/300 Read block 750 len 20480
12824 240/300 Write block 153 len 28672 data: 2 2 2 3 3 2 4
12825 241/300 Read block 3077 len 16384
12826 242/300 Write block 2502 len 8192 data: 2 3
12827 243/300 Read block 4917 len 28672
12828 244/300 Flush
12829 245/300 Write block 4690 len 28672 data: 3 5 5 4 6 7 7
12830 246/300 Read block 1658 len 20480
12831 247/300 Write block 1440 len 8192 data: 4 3
12832 248/300 Read block 3344 len 40960
12833 249/300 Read block 2480 len 16384
12834 250/300 Write block 2586 len 20480 data: 4 3 4 4 3
12835 251/300 Read block 1461 len 36864
12836 252/300 Read block 808 len 36864
12837 253/300 Write block 2182 len 28672 data: 3 3 3 3 5 5 5
12838 254/300 Write block 968 len 28672 data: 2 2 2 2 2 2 2
12839 255/300 Read block 1460 len 32768
12840 256/300 Read block 2693 len 12288
12841 257/300 Read block 4584 len 40960
12842 258/300 Flush
12843 259/300 Read block 3426 len 24576
12844 260/300 Read block 3425 len 8192
12845 261/300 Write block 1719 len 16384 data: 3 3 3 3
12846 262/300 Read block 1061 len 24576
12847 263/300 Read block 1880 len 20480
12848 264/300 Read block 3455 len 32768
12849 265/300 Write block 1889 len 16384 data: 4 4 3 3
12850 266/300 Read block 3307 len 36864
12851 267/300 Write block 3124 len 32768 data: 3 4 4 4 4 3 3 4
12852 268/300 Read block 2130 len 24576
12853 269/300 Read block 308 len 28672
12854 270/300 Flush
12855 271/300 Write block 3724 len 20480 data: 4 5 3 2 2
12856 272/300 Read block 780 len 4096
12857 273/300 Read block 964 len 40960
12858 274/300 Read block 2673 len 32768
12859 275/300 Flush
12860 276/300 Read block 2174 len 12288
12861 277/300 Read block 4587 len 36864
12862 278/300 Write block 3169 len 20480 data: 3 3 3 2 2
12863 279/300 Read block 416 len 36864
12864 280/300 Write block 4208 len 32768 data: 6 4 4 5 5 5 4 5
12865 281/300 Write block 3981 len 16384 data: 3 3 3 3
12866 282/300 Read block 2078 len 16384
12867 283/300 Write block 1276 len 20480 data: 2 2 2 2 2
12868 284/300 Read block 4902 len 36864
12869 285/300 Write block 1653 len 24576 data: 4 4 3 3 2 2
12870 286/300 Write block 1969 len 28672 data: 3 3 3 3 4 3 3
12871 287/300 Read block 1390 len 12288
12872 288/300 Write block 4070 len 24576 data: 3 4 4 4 4 5
12873 289/300 Write block 3378 len 40960 data: 3 3 2 2 2 3 3 4 4 4
12874 290/300 Read block 4591 len 36864
12875 291/300 Write block 4669 len 32768 data: 3 3 5 5 4 4 3 3
12876 292/300 Read block 2733 len 20480
12877 293/300 Write block 4011 len 32768 data: 2 2 2 2 2 2 3 3
12878 294/300 Read block 4201 len 12288
12879 295/300 Read block 4266 len 8192
12880 296/300 Write block 4911 len 40960 data: 4 4 4 4 4 4 4 4 3 3
12881 297/300 Read block 2733 len 28672
12882 298/300 Read block 4316 len 24576
12883 299/300 Read block 2810 len 28672
12884 300/300 Write block 4031 len 40960 data: 3 2 2 2 2 4 4 5 4 2
12885 301/300 Read block 2785 len 24576
12886 Main task has work to do, go find it
12887 got dsc Start(0)
12888 start 0
12889 Mar 07 18:15:23.311 INFO request completed, latency_us: 168, response_code: 204, uri: /start/cid/0, method: POST, req_id: acea31d7-7ccb-4873-8099-dca906cd2417, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
12890 Replay: started 0, returned:Ok(())
12891 ----------------------------------------------------------------
12892 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12893 States: Offline Active Active
12894 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12895 4018 Acked Read 8 Sent Done Done false
12896 4019 Acked Flush 0 Sent Done Done false
12897 4020 Acked Read 10 Sent Done Done false
12898 4021 Acked Write 1 Sent Done Done false
12899 4022 Acked Read 1 Sent Done Done false
12900 4023 Acked Read 10 Sent Done Done false
12901 4024 Acked Read 5 Sent Done Done false
12902 4025 Acked Read 5 Sent Done Done false
12903 4026 Acked Flush 0 Sent Done Done false
12904 4027 Acked Write 9 Sent Done Done false
12905 4028 Acked Read 1 Sent Done Done false
12906 4029 Acked Flush 0 Sent Done Done false
12907 4030 Acked Write 10 Sent Done Done false
12908 4031 Acked Read 8 Sent Done Done false
12909 4032 Acked Write 1 Sent Done Done false
12910 4033 Acked Write 4 Sent Done Done false
12911 4034 Acked Write 1 Sent Done Done false
12912 4035 Acked Flush 0 Sent Done Done false
12913 4036 Acked Read 10 Sent Done Done false
12914 4037 Acked Write 4 Sent Done Done false
12915 4038 Acked Write 9 Sent Done Done false
12916 4039 Acked Read 6 Sent Done Done false
12917 4040 Acked Read 3 Sent Done Done false
12918 4041 Acked Write 10 Sent Done Done false
12919 4042 Acked Write 2 Sent Done Done false
12920 4043 Acked Read 6 Sent Done Done false
12921 4044 Acked Write 7 Sent Done Done false
12922 4045 Acked Read 2 Sent Done Done false
12923 4046 Acked Read 7 Sent Done Done false
12924 4047 Acked Read 9 Sent Done Done false
12925 4048 Acked Write 9 Sent Done Done false
12926 4049 Acked Read 10 Sent Done Done false
12927 4050 Acked Write 3 Sent Done Done false
12928 4051 Acked Write 2 Sent Done Done false
12929 4052 Acked Write 10 Sent Done Done false
12930 4053 Acked Read 3 Sent Done Done false
12931 4054 Acked Read 1 Sent Done Done false
12932 4055 Acked Write 10 Sent Done Done false
12933 4056 Acked Read 8 Sent Done Done false
12934 4057 Acked Read 4 Sent Done Done false
12935 4058 Acked Write 7 Sent Done Done false
12936 4059 Acked Flush 0 Sent Done Done false
12937 4060 Acked Write 3 Sent Done Done false
12938 4061 Acked Read 2 Sent Done Done false
12939 4062 Acked Write 3 Sent Done Done false
12940 4063 Acked Read 8 Sent Done Done false
12941 4064 Acked Write 7 Sent Done Done false
12942 4065 Acked Read 10 Sent Done Done false
12943 4066 Acked Read 5 Sent Done Done false
12944 4067 Acked Write 7 Sent Done Done false
12945 4068 Acked Write 1 Sent Done Done false
12946 4069 Acked Read 2 Sent Done Done false
12947 4070 Acked Write 5 Sent Done Done false
12948 4071 Acked Write 3 Sent Done Done false
12949 4072 Acked Write 8 Sent Done Done false
12950 4073 Acked Read 5 Sent Done Done false
12951 4074 Acked Write 3 Sent Done Done false
12952 4075 Acked Write 1 Sent Done Done false
12953 4076 Acked Write 9 Sent Done Done false
12954 4077 Acked Read 6 Sent Done Done false
12955 4078 Acked Write 3 Sent Done Done false
12956 4079 Acked Read 8 Sent Done Done false
12957 4080 Acked Write 10 Sent Done Done false
12958 4081 Acked Write 4 Sent Done Done false
12959 4082 Acked Write 6 Sent Done Done false
12960 4083 Acked Read 10 Sent Done Done false
12961 4084 Acked Read 1 Sent Done Done false
12962 4085 Acked Flush 0 Sent Done Done false
12963 4086 Acked Read 10 Sent Done Done false
12964 4087 Acked Read 1 Sent Done Done false
12965 4088 Acked Write 7 Sent Done Done false
12966 4089 Acked Read 1 Sent Done Done false
12967 4090 Acked Read 8 Sent Done Done false
12968 4091 Acked Read 2 Sent Done Done false
12969 4092 Acked Write 10 Sent Done Done false
12970 4093 Acked Read 6 Sent Done Done false
12971 4094 Acked Read 4 Sent Done Done false
12972 4095 Acked Read 2 Sent Done Done false
12973 4096 Acked Write 3 Sent Done Done false
12974 4097 Acked Read 1 Sent Done Done false
12975 4098 Acked Write 6 Sent Done Done false
12976 4099 Acked Read 4 Sent Done Done false
12977 4100 Acked Write 2 Sent Done Done false
12978 4101 Acked Flush 0 Sent Done Done false
12979 4102 Acked Write 3 Sent Done Done false
12980 4103 Acked Read 6 Sent Done Done false
12981 4104 Acked Read 7 Sent Done Done false
12982 4105 Acked Read 6 Sent Done Done false
12983 4106 Acked Read 2 Sent Done Done false
12984 4107 Acked Write 2 Sent Done Done false
12985 4108 Acked Read 7 Sent Done Done false
12986 4109 Acked Read 3 Sent Done Done false
12987 4110 Acked Read 2 Sent Done Done false
12988 4111 Acked Read 3 Sent Done Done false
12989 4112 Acked Read 8 Sent Done Done false
12990 4113 Acked Read 10 Sent Done Done false
12991 4114 Acked Write 8 Sent Done Done false
12992 4115 Acked Write 9 Sent Done Done false
12993 4116 Acked Write 6 Sent Done Done false
12994 4117 Acked Read 3 Sent Done Done false
12995 4118 Acked Write 7 Sent Done Done false
12996 4119 Acked Read 9 Sent Done Done false
12997 4120 Acked Read 2 Sent Done Done false
12998 4121 Acked Read 10 Sent Done Done false
12999 4122 Acked Read 9 Sent Done Done false
13000 4123 Acked Write 10 Sent Done Done false
13001 4124 Acked Read 7 Sent Done Done false
13002 4125 Acked Flush 0 Sent Done Done false
13003 4126 Acked Flush 0 Sent Done Done false
13004 4127 Acked Read 1 Sent Done Done false
13005 4128 Acked Read 2 Sent Done Done false
13006 4129 Acked Write 1 Sent Done Done false
13007 4130 Acked Read 4 Sent Done Done false
13008 4131 Acked Write 4 Sent Done Done false
13009 4132 Acked Write 6 Sent Done Done false
13010 4133 Acked Flush 0 Sent Done Done false
13011 4134 Acked Read 9 Sent Done Done false
13012 4135 Acked Read 10 Sent Done Done false
13013 4136 Acked Write 7 Sent Done Done false
13014 4137 Acked Read 5 Sent Done Done false
13015 4138 Acked Read 8 Sent Done Done false
13016 4139 Acked Read 7 Sent Done Done false
13017 4140 Acked Read 3 Sent Done Done false
13018 4141 Acked Write 5 Sent Done Done false
13019 4142 Acked Write 5 Sent Done Done false
13020 4143 Acked Read 1 Sent Done Done false
13021 4144 Acked Write 5 Sent Done Done false
13022 4145 Acked Read 7 Sent Done Done false
13023 4146 Acked Write 6 Sent Done Done false
13024 4147 Acked Write 10 Sent Done Done false
13025 4148 Acked Write 6 Sent Done Done false
13026 4149 Acked Read 1 Sent Done Done false
13027 4150 Acked Read 4 Sent Done Done false
13028 4151 Acked Read 1 Sent Done Done false
13029 4152 Acked Write 3 Sent Done Done false
13030 4153 Acked Flush 0 Sent Done Done false
13031 4154 Acked Read 2 Sent Done Done false
13032 4155 Acked Write 5 Sent Done Done false
13033 4156 Acked Write 10 Sent Done Done false
13034 4157 Acked Read 7 Sent Done Done false
13035 4158 Acked Flush 0 Sent Done Done false
13036 4159 Acked Read 3 Sent Done Done false
13037 4160 Acked Write 2 Sent Done Done false
13038 4161 Acked Read 1 Sent Done Done false
13039 4162 Acked Read 2 Sent Done Done false
13040 4163 Acked Write 7 Sent Done Done false
13041 4164 Acked Read 9 Sent Done Done false
13042 4165 Acked Read 1 Sent Done Done false
13043 4166 Acked Read 1 Sent Done Done false
13044 4167 Acked Flush 0 Sent Done Done false
13045 4168 Acked Write 8 Sent Done Done false
13046 4169 Acked Read 8 Sent Done Done false
13047 4170 Acked Read 2 Sent Done Done false
13048 4171 Acked Read 9 Sent Done Done false
13049 4172 Acked Read 2 Sent Done Done false
13050 4173 Acked Read 7 Sent Done Done false
13051 4174 Acked Read 5 Sent Done Done false
13052 4175 Acked Write 8 Sent Done Done false
13053 4176 Acked Read 10 Sent Done Done false
13054 4177 Acked Write 2 Sent Done Done false
13055 4178 Acked Write 8 Sent Done Done false
13056 4179 Acked Write 10 Sent Done Done false
13057 4180 Acked Read 8 Sent Done Done false
13058 4181 Acked Read 3 Sent Done Done false
13059 4182 Acked Read 2 Sent Done Done false
13060 4183 Acked Write 10 Sent Done Done false
13061 4184 Acked Write 3 Sent Done Done false
13062 4185 Acked Write 5 Sent Done Done false
13063 4186 Acked Write 10 Sent Done Done false
13064 4187 Acked Read 2 Sent Done Done false
13065 4188 Acked Read 10 Sent Done Done false
13066 4189 Acked Write 2 Sent Done Done false
13067 4190 Acked Write 10 Sent Done Done false
13068 4191 Acked Write 8 Sent Done Done false
13069 4192 Acked Flush 0 Sent Done Done false
13070 4193 Acked Flush 0 Sent Done Done false
13071 4194 Acked Write 6 Sent Done Done false
13072 4195 Acked Read 3 Sent Done Done false
13073 4196 Acked Read 6 Sent Done Done false
13074 4197 Acked Read 5 Sent Done Done false
13075 4198 Acked Read 4 Sent Done Done false
13076 4199 Acked Write 7 Sent Done Done false
13077 4200 Acked Write 6 Sent Done Done false
13078 4201 Acked Read 1 Sent Done Done false
13079 4202 Acked Read 8 Sent Done Done false
13080 4203 Acked Read 9 Sent Done Done false
13081 4204 Acked Write 2 Sent Done Done false
13082 4205 Acked Read 9 Sent Done Done false
13083 4206 Acked Read 6 Sent Done Done false
13084 4207 Acked Write 3 Sent Done Done false
13085 4208 Acked Read 2 Sent Done Done false
13086 4209 Acked Read 5 Sent Done Done false
13087 4210 Acked Read 4 Sent Done Done false
13088 4211 Acked Read 10 Sent Done Done false
13089 4212 Acked Write 3 Sent Done Done false
13090 4213 Acked Write 6 Sent Done Done false
13091 4214 Acked Read 10 Sent Done Done false
13092 4215 Acked Read 9 Sent Done Done false
13093 4216 Acked Read 7 Sent Done Done false
13094 4217 Acked Write 4 Sent Done Done false
13095 4218 Acked Write 7 Sent Done Done false
13096 4219 Acked Read 8 Sent Done Done false
13097 4220 Acked Write 1 Sent Done Done false
13098 4221 Acked Write 2 Sent Done Done false
13099 4222 Acked Write 6 Sent Done Done false
13100 4223 Acked Flush 0 Sent Done Done false
13101 4224 Acked Write 2 Sent Done Done false
13102 4225 Acked Flush 0 Sent Done Done false
13103 4226 Acked Write 7 Sent Done Done false
13104 4227 Acked Read 3 Sent Done Done false
13105 4228 Acked Write 2 Sent Done Done false
13106 4229 Acked Write 4 Sent Done Done false
13107 4230 Acked Read 5 Sent Done Done false
13108 4231 Acked Read 3 Sent Done Done false
13109 4232 Acked Write 9 Sent Done Done false
13110 4233 Acked Read 8 Sent Done Done false
13111 4234 Acked Write 2 Sent Done Done false
13112 4235 Acked Read 10 Sent Done Done false
13113 4236 Acked Write 5 Sent Done Done false
13114 4237 Acked Write 3 Sent Done Done false
13115 4238 Acked Read 2 Sent Done Done false
13116 4239 Acked Read 5 Sent Done Done false
13117 4240 Acked Read 7 Sent Done Done false
13118 4241 Acked Flush 0 Sent Done Done false
13119 4242 Acked Write 6 Sent Done Done false
13120 4243 Acked Write 5 Sent Done Done false
13121 4244 Acked Read 5 Sent Done Done false
13122 4245 Acked Write 10 Sent Done Done false
13123 4246 Acked Write 6 Sent Done Done false
13124 4247 Acked Write 2 Sent Done Done false
13125 4248 Acked Write 5 Sent Done Done false
13126 4249 Acked Read 2 Sent Done Done false
13127 4250 Acked Write 2 Sent Done Done false
13128 4251 Acked Read 8 Sent Done Done false
13129 4252 Acked Read 9 Sent Done Done false
13130 4253 Acked Write 2 Sent Done Done false
13131 4254 Acked Read 2 Sent Done Done false
13132 4255 Acked Write 5 Sent Done Done false
13133 4256 Acked Read 5 Sent Done Done false
13134 4257 Acked Write 7 Sent Done Done false
13135 4258 Acked Read 4 Sent Done Done false
13136 4259 Acked Write 2 Sent Done Done false
13137 4260 Acked Read 7 Sent Done Done false
13138 4261 Acked Flush 0 Sent Done Done false
13139 4262 Acked Write 7 Sent Done Done false
13140 4263 Acked Read 5 Sent Done Done false
13141 4264 Acked Write 2 Sent Done Done false
13142 4265 Acked Read 10 Sent Done Done false
13143 4266 Acked Read 4 Sent Done Done false
13144 4267 Acked Write 5 Sent Done Done false
13145 4268 Acked Read 9 Sent Done Done false
13146 4269 Acked Read 9 Sent Done Done false
13147 4270 Acked Write 7 Sent Done Done false
13148 4271 Acked Write 7 Sent Done Done false
13149 4272 Acked Read 8 Sent Done Done false
13150 4273 Acked Read 3 Sent Done Done false
13151 4274 Acked Read 10 Sent Done Done false
13152 4275 Acked Flush 0 Sent Done Done false
13153 4276 Acked Read 6 Sent Done Done false
13154 4277 Acked Read 2 Sent Done Done false
13155 4278 Acked Write 4 Sent Done Done false
13156 4279 Acked Read 6 Sent Done Done false
13157 4280 Acked Read 5 Sent Done Done false
13158 4281 Acked Read 8 Sent Done Done false
13159 4282 Acked Write 4 Sent Done Done false
13160 4283 Acked Read 9 Sent Done Done false
13161 4284 Acked Write 8 Sent Done Done false
13162 4285 Acked Read 6 Sent Done Done false
13163 4286 Acked Read 7 Sent Done Done false
13164 4287 Acked Flush 0 Sent Done Done false
13165 4288 Acked Write 5 Sent Done Done false
13166 4289 Acked Read 1 Sent Done Done false
13167 4290 Acked Read 10 Sent Done Done false
13168 4291 Acked Read 8 Sent Done Done false
13169 4292 Acked Flush 0 Sent Done Done false
13170 4293 Acked Read 3 Sent Done Done false
13171 4294 Acked Read 9 Sent Done Done false
13172 4295 Acked Write 5 Sent Done Done false
13173 4296 Acked Read 9 Sent Done Done false
13174 4297 Acked Write 8 Sent Done Done false
13175 4298 Acked Write 4 Sent Done Done false
13176 4299 Acked Read 4 Sent Done Done false
13177 4300 Acked Write 5 Sent Done Done false
13178 4301 Acked Read 9 Sent Done Done false
13179 4302 Acked Write 6 Sent Done Done false
13180 4303 Acked Write 7 Sent Done Done false
13181 4304 Acked Read 3 Sent Done Done false
13182 4305 Acked Write 6 Sent Done Done false
13183 4306 Acked Write 10 Sent Done Done false
13184 4307 Acked Read 9 Sent Done Done false
13185 4308 Acked Write 8 Sent Done Done false
13186 4309 Acked Read 5 Sent Done Done false
13187 4310 Acked Write 8 Sent Done Done false
13188 4311 Acked Read 3 Sent Done Done false
13189 4312 Acked Read 2 Sent Done Done false
13190 4313 Acked Write 10 Sent Done Done false
13191 4314 Acked Read 7 Sent Done Done false
13192 4315 Acked Read 6 Sent Done Done false
13193 4316 Acked Read 7 Sent Done Done false
13194 4317 Acked Write 10 Sent Done Done false
13195 4318 Acked Read 6 Sent Done Done false
13196 STATES DS:0 DS:1 DS:2 TOTAL
13197 Sent 301 0 0 301
13198 Done 0 301 301 602
13199 Skipped 0 0 0 0
13200 Error 0 0 0 0
13201 Last Flush: Some(JobId(4017)) Some(JobId(4292)) Some(JobId(4292))
13202 Downstairs last five completed: 4017 4016 4015 4014 4013
13203 Upstairs last five completed: 4318 4317 4316 4315 4314
13204 CLIENT: Up:0 ds:301 act:2
132052025-03-07T18:15:24.225ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
132062025-03-07T18:15:25.226ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
132072025-03-07T18:15:26.229ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
13208 [8810] got start action
13209 [8810] I am going to restart
13210 Starting downstairs at port 8810
13211 Make output file at "/tmp/dsc/downstairs-8810.txt"
13212 [8810][0] reports Starting
13213 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219)
13214 [8810][0] reports Running
132152025-03-07T18:15:27.299ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
13216 ----------------------------------------------------------------
13217 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13218 States: Offline Active Active
13219 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13220 4018 Acked Read 8 Sent Done Done false
13221 4019 Acked Flush 0 Sent Done Done false
13222 4020 Acked Read 10 Sent Done Done false
13223 4021 Acked Write 1 Sent Done Done false
13224 4022 Acked Read 1 Sent Done Done false
13225 4023 Acked Read 10 Sent Done Done false
13226 4024 Acked Read 5 Sent Done Done false
13227 4025 Acked Read 5 Sent Done Done false
13228 4026 Acked Flush 0 Sent Done Done false
13229 4027 Acked Write 9 Sent Done Done false
13230 4028 Acked Read 1 Sent Done Done false
13231 4029 Acked Flush 0 Sent Done Done false
13232 4030 Acked Write 10 Sent Done Done false
13233 4031 Acked Read 8 Sent Done Done false
13234 4032 Acked Write 1 Sent Done Done false
13235 4033 Acked Write 4 Sent Done Done false
13236 4034 Acked Write 1 Sent Done Done false
13237 4035 Acked Flush 0 Sent Done Done false
13238 4036 Acked Read 10 Sent Done Done false
13239 4037 Acked Write 4 Sent Done Done false
13240 4038 Acked Write 9 Sent Done Done false
13241 4039 Acked Read 6 Sent Done Done false
13242 4040 Acked Read 3 Sent Done Done false
13243 4041 Acked Write 10 Sent Done Done false
13244 4042 Acked Write 2 Sent Done Done false
13245 4043 Acked Read 6 Sent Done Done false
13246 4044 Acked Write 7 Sent Done Done false
13247 4045 Acked Read 2 Sent Done Done false
13248 4046 Acked Read 7 Sent Done Done false
13249 4047 Acked Read 9 Sent Done Done false
13250 4048 Acked Write 9 Sent Done Done false
13251 4049 Acked Read 10 Sent Done Done false
13252 4050 Acked Write 3 Sent Done Done false
13253 4051 Acked Write 2 Sent Done Done false
13254 4052 Acked Write 10 Sent Done Done false
13255 4053 Acked Read 3 Sent Done Done false
13256 4054 Acked Read 1 Sent Done Done false
13257 4055 Acked Write 10 Sent Done Done false
13258 4056 Acked Read 8 Sent Done Done false
13259 4057 Acked Read 4 Sent Done Done false
13260 4058 Acked Write 7 Sent Done Done false
13261 4059 Acked Flush 0 Sent Done Done false
13262 4060 Acked Write 3 Sent Done Done false
13263 4061 Acked Read 2 Sent Done Done false
13264 4062 Acked Write 3 Sent Done Done false
13265 4063 Acked Read 8 Sent Done Done false
13266 4064 Acked Write 7 Sent Done Done false
13267 4065 Acked Read 10 Sent Done Done false
13268 4066 Acked Read 5 Sent Done Done false
13269 4067 Acked Write 7 Sent Done Done false
13270 4068 Acked Write 1 Sent Done Done false
13271 4069 Acked Read 2 Sent Done Done false
13272 4070 Acked Write 5 Sent Done Done false
13273 4071 Acked Write 3 Sent Done Done false
13274 4072 Acked Write 8 Sent Done Done false
13275 4073 Acked Read 5 Sent Done Done false
13276 4074 Acked Write 3 Sent Done Done false
13277 4075 Acked Write 1 Sent Done Done false
13278 4076 Acked Write 9 Sent Done Done false
13279 4077 Acked Read 6 Sent Done Done false
13280 4078 Acked Write 3 Sent Done Done false
13281 4079 Acked Read 8 Sent Done Done false
13282 4080 Acked Write 10 Sent Done Done false
13283 4081 Acked Write 4 Sent Done Done false
13284 4082 Acked Write 6 Sent Done Done false
13285 4083 Acked Read 10 Sent Done Done false
13286 4084 Acked Read 1 Sent Done Done false
13287 4085 Acked Flush 0 Sent Done Done false
13288 4086 Acked Read 10 Sent Done Done false
13289 4087 Acked Read 1 Sent Done Done false
13290 4088 Acked Write 7 Sent Done Done false
13291 4089 Acked Read 1 Sent Done Done false
13292 4090 Acked Read 8 Sent Done Done false
13293 4091 Acked Read 2 Sent Done Done false
13294 4092 Acked Write 10 Sent Done Done false
13295 4093 Acked Read 6 Sent Done Done false
13296 4094 Acked Read 4 Sent Done Done false
13297 4095 Acked Read 2 Sent Done Done false
13298 4096 Acked Write 3 Sent Done Done false
13299 4097 Acked Read 1 Sent Done Done false
13300 4098 Acked Write 6 Sent Done Done false
13301 4099 Acked Read 4 Sent Done Done false
13302 4100 Acked Write 2 Sent Done Done false
13303 4101 Acked Flush 0 Sent Done Done false
13304 4102 Acked Write 3 Sent Done Done false
13305 4103 Acked Read 6 Sent Done Done false
13306 4104 Acked Read 7 Sent Done Done false
13307 4105 Acked Read 6 Sent Done Done false
13308 4106 Acked Read 2 Sent Done Done false
13309 4107 Acked Write 2 Sent Done Done false
13310 4108 Acked Read 7 Sent Done Done false
13311 4109 Acked Read 3 Sent Done Done false
13312 4110 Acked Read 2 Sent Done Done false
13313 4111 Acked Read 3 Sent Done Done false
13314 4112 Acked Read 8 Sent Done Done false
13315 4113 Acked Read 10 Sent Done Done false
13316 4114 Acked Write 8 Sent Done Done false
13317 4115 Acked Write 9 Sent Done Done false
13318 4116 Acked Write 6 Sent Done Done false
13319 4117 Acked Read 3 Sent Done Done false
13320 4118 Acked Write 7 Sent Done Done false
13321 4119 Acked Read 9 Sent Done Done false
13322 4120 Acked Read 2 Sent Done Done false
13323 4121 Acked Read 10 Sent Done Done false
13324 4122 Acked Read 9 Sent Done Done false
13325 4123 Acked Write 10 Sent Done Done false
13326 4124 Acked Read 7 Sent Done Done false
13327 4125 Acked Flush 0 Sent Done Done false
13328 4126 Acked Flush 0 Sent Done Done false
13329 4127 Acked Read 1 Sent Done Done false
13330 4128 Acked Read 2 Sent Done Done false
13331 4129 Acked Write 1 Sent Done Done false
13332 4130 Acked Read 4 Sent Done Done false
13333 4131 Acked Write 4 Sent Done Done false
13334 4132 Acked Write 6 Sent Done Done false
13335 4133 Acked Flush 0 Sent Done Done false
13336 4134 Acked Read 9 Sent Done Done false
13337 4135 Acked Read 10 Sent Done Done false
13338 4136 Acked Write 7 Sent Done Done false
13339 4137 Acked Read 5 Sent Done Done false
13340 4138 Acked Read 8 Sent Done Done false
13341 4139 Acked Read 7 Sent Done Done false
13342 4140 Acked Read 3 Sent Done Done false
13343 4141 Acked Write 5 Sent Done Done false
13344 4142 Acked Write 5 Sent Done Done false
13345 4143 Acked Read 1 Sent Done Done false
13346 4144 Acked Write 5 Sent Done Done false
13347 4145 Acked Read 7 Sent Done Done false
13348 4146 Acked Write 6 Sent Done Done false
13349 4147 Acked Write 10 Sent Done Done false
13350 4148 Acked Write 6 Sent Done Done false
13351 4149 Acked Read 1 Sent Done Done false
13352 4150 Acked Read 4 Sent Done Done false
13353 4151 Acked Read 1 Sent Done Done false
13354 4152 Acked Write 3 Sent Done Done false
13355 4153 Acked Flush 0 Sent Done Done false
13356 4154 Acked Read 2 Sent Done Done false
13357 4155 Acked Write 5 Sent Done Done false
13358 4156 Acked Write 10 Sent Done Done false
13359 4157 Acked Read 7 Sent Done Done false
13360 4158 Acked Flush 0 Sent Done Done false
13361 4159 Acked Read 3 Sent Done Done false
13362 4160 Acked Write 2 Sent Done Done false
13363 4161 Acked Read 1 Sent Done Done false
13364 4162 Acked Read 2 Sent Done Done false
13365 4163 Acked Write 7 Sent Done Done false
13366 4164 Acked Read 9 Sent Done Done false
13367 4165 Acked Read 1 Sent Done Done false
13368 4166 Acked Read 1 Sent Done Done false
13369 4167 Acked Flush 0 Sent Done Done false
13370 4168 Acked Write 8 Sent Done Done false
13371 4169 Acked Read 8 Sent Done Done false
13372 4170 Acked Read 2 Sent Done Done false
13373 4171 Acked Read 9 Sent Done Done false
13374 4172 Acked Read 2 Sent Done Done false
13375 4173 Acked Read 7 Sent Done Done false
13376 4174 Acked Read 5 Sent Done Done false
13377 4175 Acked Write 8 Sent Done Done false
13378 4176 Acked Read 10 Sent Done Done false
13379 4177 Acked Write 2 Sent Done Done false
13380 4178 Acked Write 8 Sent Done Done false
13381 4179 Acked Write 10 Sent Done Done false
13382 4180 Acked Read 8 Sent Done Done false
13383 4181 Acked Read 3 Sent Done Done false
13384 4182 Acked Read 2 Sent Done Done false
13385 4183 Acked Write 10 Sent Done Done false
13386 4184 Acked Write 3 Sent Done Done false
13387 4185 Acked Write 5 Sent Done Done false
13388 4186 Acked Write 10 Sent Done Done false
13389 4187 Acked Read 2 Sent Done Done false
13390 4188 Acked Read 10 Sent Done Done false
13391 4189 Acked Write 2 Sent Done Done false
13392 4190 Acked Write 10 Sent Done Done false
13393 4191 Acked Write 8 Sent Done Done false
13394 4192 Acked Flush 0 Sent Done Done false
13395 4193 Acked Flush 0 Sent Done Done false
13396 4194 Acked Write 6 Sent Done Done false
13397 4195 Acked Read 3 Sent Done Done false
13398 4196 Acked Read 6 Sent Done Done false
13399 4197 Acked Read 5 Sent Done Done false
13400 4198 Acked Read 4 Sent Done Done false
13401 4199 Acked Write 7 Sent Done Done false
13402 4200 Acked Write 6 Sent Done Done false
13403 4201 Acked Read 1 Sent Done Done false
13404 4202 Acked Read 8 Sent Done Done false
13405 4203 Acked Read 9 Sent Done Done false
13406 4204 Acked Write 2 Sent Done Done false
13407 4205 Acked Read 9 Sent Done Done false
13408 4206 Acked Read 6 Sent Done Done false
13409 4207 Acked Write 3 Sent Done Done false
13410 4208 Acked Read 2 Sent Done Done false
13411 4209 Acked Read 5 Sent Done Done false
13412 4210 Acked Read 4 Sent Done Done false
13413 4211 Acked Read 10 Sent Done Done false
13414 4212 Acked Write 3 Sent Done Done false
13415 4213 Acked Write 6 Sent Done Done false
13416 4214 Acked Read 10 Sent Done Done false
13417 4215 Acked Read 9 Sent Done Done false
13418 4216 Acked Read 7 Sent Done Done false
13419 4217 Acked Write 4 Sent Done Done false
13420 4218 Acked Write 7 Sent Done Done false
13421 4219 Acked Read 8 Sent Done Done false
13422 4220 Acked Write 1 Sent Done Done false
13423 4221 Acked Write 2 Sent Done Done false
13424 4222 Acked Write 6 Sent Done Done false
13425 4223 Acked Flush 0 Sent Done Done false
13426 4224 Acked Write 2 Sent Done Done false
13427 4225 Acked Flush 0 Sent Done Done false
13428 4226 Acked Write 7 Sent Done Done false
13429 4227 Acked Read 3 Sent Done Done false
13430 4228 Acked Write 2 Sent Done Done false
13431 4229 Acked Write 4 Sent Done Done false
13432 4230 Acked Read 5 Sent Done Done false
13433 4231 Acked Read 3 Sent Done Done false
13434 4232 Acked Write 9 Sent Done Done false
13435 4233 Acked Read 8 Sent Done Done false
13436 4234 Acked Write 2 Sent Done Done false
13437 4235 Acked Read 10 Sent Done Done false
13438 4236 Acked Write 5 Sent Done Done false
13439 4237 Acked Write 3 Sent Done Done false
13440 4238 Acked Read 2 Sent Done Done false
13441 4239 Acked Read 5 Sent Done Done false
13442 4240 Acked Read 7 Sent Done Done false
13443 4241 Acked Flush 0 Sent Done Done false
13444 4242 Acked Write 6 Sent Done Done false
13445 4243 Acked Write 5 Sent Done Done false
13446 4244 Acked Read 5 Sent Done Done false
13447 4245 Acked Write 10 Sent Done Done false
13448 4246 Acked Write 6 Sent Done Done false
13449 4247 Acked Write 2 Sent Done Done false
13450 4248 Acked Write 5 Sent Done Done false
13451 4249 Acked Read 2 Sent Done Done false
13452 4250 Acked Write 2 Sent Done Done false
13453 4251 Acked Read 8 Sent Done Done false
13454 4252 Acked Read 9 Sent Done Done false
13455 4253 Acked Write 2 Sent Done Done false
13456 4254 Acked Read 2 Sent Done Done false
13457 4255 Acked Write 5 Sent Done Done false
13458 4256 Acked Read 5 Sent Done Done false
13459 4257 Acked Write 7 Sent Done Done false
13460 4258 Acked Read 4 Sent Done Done false
13461 4259 Acked Write 2 Sent Done Done false
13462 4260 Acked Read 7 Sent Done Done false
13463 4261 Acked Flush 0 Sent Done Done false
13464 4262 Acked Write 7 Sent Done Done false
13465 4263 Acked Read 5 Sent Done Done false
13466 4264 Acked Write 2 Sent Done Done false
13467 4265 Acked Read 10 Sent Done Done false
13468 4266 Acked Read 4 Sent Done Done false
13469 4267 Acked Write 5 Sent Done Done false
13470 4268 Acked Read 9 Sent Done Done false
13471 4269 Acked Read 9 Sent Done Done false
13472 4270 Acked Write 7 Sent Done Done false
13473 4271 Acked Write 7 Sent Done Done false
13474 4272 Acked Read 8 Sent Done Done false
13475 4273 Acked Read 3 Sent Done Done false
13476 4274 Acked Read 10 Sent Done Done false
13477 4275 Acked Flush 0 Sent Done Done false
13478 4276 Acked Read 6 Sent Done Done false
13479 4277 Acked Read 2 Sent Done Done false
13480 4278 Acked Write 4 Sent Done Done false
13481 4279 Acked Read 6 Sent Done Done false
13482 4280 Acked Read 5 Sent Done Done false
13483 4281 Acked Read 8 Sent Done Done false
13484 4282 Acked Write 4 Sent Done Done false
13485 4283 Acked Read 9 Sent Done Done false
13486 4284 Acked Write 8 Sent Done Done false
13487 4285 Acked Read 6 Sent Done Done false
13488 4286 Acked Read 7 Sent Done Done false
13489 4287 Acked Flush 0 Sent Done Done false
13490 4288 Acked Write 5 Sent Done Done false
13491 4289 Acked Read 1 Sent Done Done false
13492 4290 Acked Read 10 Sent Done Done false
13493 4291 Acked Read 8 Sent Done Done false
13494 4292 Acked Flush 0 Sent Done Done false
13495 4293 Acked Read 3 Sent Done Done false
13496 4294 Acked Read 9 Sent Done Done false
13497 4295 Acked Write 5 Sent Done Done false
13498 4296 Acked Read 9 Sent Done Done false
13499 4297 Acked Write 8 Sent Done Done false
13500 4298 Acked Write 4 Sent Done Done false
13501 4299 Acked Read 4 Sent Done Done false
13502 4300 Acked Write 5 Sent Done Done false
13503 4301 Acked Read 9 Sent Done Done false
13504 4302 Acked Write 6 Sent Done Done false
13505 4303 Acked Write 7 Sent Done Done false
13506 4304 Acked Read 3 Sent Done Done false
13507 4305 Acked Write 6 Sent Done Done false
13508 4306 Acked Write 10 Sent Done Done false
13509 4307 Acked Read 9 Sent Done Done false
13510 4308 Acked Write 8 Sent Done Done false
13511 4309 Acked Read 5 Sent Done Done false
13512 4310 Acked Write 8 Sent Done Done false
13513 4311 Acked Read 3 Sent Done Done false
13514 4312 Acked Read 2 Sent Done Done false
13515 4313 Acked Write 10 Sent Done Done false
13516 4314 Acked Read 7 Sent Done Done false
13517 4315 Acked Read 6 Sent Done Done false
13518 4316 Acked Read 7 Sent Done Done false
13519 4317 Acked Write 10 Sent Done Done false
13520 4318 Acked Read 6 Sent Done Done false
13521 STATES DS:0 DS:1 DS:2 TOTAL
13522 Sent 301 0 0 301
13523 Done 0 301 301 602
13524 Skipped 0 0 0 0
13525 Error 0 0 0 0
13526 Last Flush: Some(JobId(4017)) Some(JobId(4292)) Some(JobId(4292))
13527 Downstairs last five completed: 4017 4016 4015 4014 4013
13528 Upstairs last five completed: 4318 4317 4316 4315 4314
13529 CLIENT: Up:0 ds:301 act:2
135302025-03-07T18:15:28.233ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135312025-03-07T18:15:29.058ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135322025-03-07T18:15:29.058ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34460) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135332025-03-07T18:15:29.059ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135342025-03-07T18:15:29.060ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135352025-03-07T18:15:29.060ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135362025-03-07T18:15:29.060ZINFOcrucible: send last flush ID to this DS: Some(JobId(4017)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135372025-03-07T18:15:29.060ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135382025-03-07T18:15:29.060ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4017)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
135392025-03-07T18:15:29.060ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4017)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
13540 ----------------------------------------------------------------
13541 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13542 Downstairs last five completed: 4319 4318 4317 4316 4315
13543 Upstairs last five completed: 4319 4318 4317 4316 4315
13544 CLIENT: Up:0 ds:0 act:3
13545 Replay: All jobs finished
13546 Main task has work to do, go find it
13547 got dsc Stop(1)
13548 stop 1
13549 Mar 07 18:15:31.316 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/1, method: POST[8820] Got stop action so:false kr:false
13550 , req_id: 396d25af-3659-492f-8554-581b57f4a93b, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
13551 Mar 07 18:15:31.317{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:31.316987128Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
13552 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/1, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:31.317081097Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
13553 method: GET, req_id: e57661de-0874-49bd-96b4-b909b599ac34, remote_addr:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:31.317154478Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
13554 127.0.0.1:64322, local_addr: 127.0.0.1:9998
135552025-03-07T18:15:31.317ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
13556 [8820] Exited with: ExitStatus(unix_wait_status(9))
13557 [8820][1] reports Exit
13558 Mar 07 18:15:35.319 INFO request completed, latency_us: 304, response_code: 200, uri: /state/cid/1, method: GET, req_id: 00e14884-a800-461f-b2a6-ed0890b995c0, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
13559 001/300 Flush
13560 002/300 Read block 4647 len 12288
13561 003/300 Write block 1844 len 24576 data: 3 3 5 4 4 4
13562 004/300 Read block 4781 len 28672
13563 005/300 Write block 1601 len 20480 data: 5 5 5 4 3
13564 006/300 Write block 320 len 24576 data: 2 2 3 3 3 4
13565 007/300 Read block 4209 len 8192
13566 008/300 Read block 2753 len 40960
13567 009/300 Read block 1315 len 24576
13568 010/300 Write block 4313 len 40960 data: 3 3 3 3 3 3 3 3 3 3
13569 011/300 Read block 4275 len 12288
13570 012/300 Write block 2114 len 28672 data: 4 4 3 3 4 5 5
13571 013/300 Flush
13572 014/300 Write block 1815 len 32768 data: 4 4 4 5 5 4 4 5
13573 015/300 Write block 4637 len 20480 data: 4 4 4 4 4
13574 016/300 Write block 2737 len 36864 data: 5 5 6 4 5 3 3 3 3
13575 017/300 Write block 2821 len 36864 data: 4 4 4 4 4 2 2 2 2
13576 018/300 Read block 790 len 28672
13577 019/300 Flush
13578 020/300 Write block 4296 len 16384 data: 2 2 2 2
13579 021/300 Write block 1715 len 40960 data: 3 3 3 3 4 4 4 4 3 4
13580 022/300 Write block 2962 len 4096 data: 2
13581 023/300 Write block 3929 len 28672 data: 4 4 4 5 3 4 4
13582 024/300 Flush
13583 025/300 Flush
13584 026/300 Read block 1033 len 32768
13585 027/300 Read block 4709 len 36864
13586 028/300 Write block 348 len 40960 data: 6 6 6 5 5 4 3 3 3 2
13587 029/300 Write block 2005 len 16384 data: 4 4 4 5
13588 030/300 Read block 1187 len 28672
13589 031/300 Write block 4574 len 36864 data: 3 3 4 3 3 3 2 2 3
13590 032/300 Write block 1164 len 4096 data: 4
13591 033/300 Flush
13592 034/300 Write block 3113 len 32768 data: 4 4 4 4 4 3 2 2
13593 035/300 Read block 566 len 24576
13594 036/300 Read block 4961 len 28672
13595 037/300 Flush
13596 038/300 Read block 4570 len 32768
13597 039/300 Read block 98 len 16384
13598 040/300 Read block 107 len 16384
13599 041/300 Write block 3421 len 20480 data: 4 4 4 4 5
13600 042/300 Read block 3179 len 16384
13601 043/300 Flush
13602 044/300 Read block 2333 len 40960
13603 045/300 Read block 2294 len 24576
13604 046/300 Read block 518 len 32768
13605 047/300 Read block 1425 len 24576
13606 048/300 Read block 4733 len 20480
13607 049/300 Read block 4085 len 8192
13608 050/300 Read block 1542 len 32768
13609 051/300 Write block 2119 len 36864 data: 6 6 5 4 4 3 3 3 3
13610 052/300 Write block 3368 len 16384 data: 4 5 4 4
13611 053/300 Read block 1698 len 32768
13612 054/300 Read block 3428 len 16384
13613 055/300 Write block 2950 len 4096 data: 5
13614 056/300 Flush
13615 057/300 Flush
13616 058/300 Write block 2531 len 4096 data: 3
13617 059/300 Write block 453 len 20480 data: 2 2 2 2 2
13618 060/300 Read block 3912 len 8192
13619 061/300 Read block 3538 len 32768
13620 062/300 Read block 2211 len 20480
13621 063/300 Write block 3236 len 8192 data: 4 4
13622 064/300 Write block 4293 len 32768 data: 2 2 2 3 3 3 3 2
13623 065/300 Read block 2074 len 32768
13624 066/300 Read block 4594 len 16384
13625 067/300 Read block 1260 len 24576
13626 068/300 Read block 4283 len 4096
13627 069/300 Read block 4095 len 32768
13628 070/300 Read block 1927 len 28672
13629 071/300 Write block 847 len 28672 data: 2 2 2 2 3 2 2
13630 072/300 Write block 3201 len 40960 data: 3 3 4 3 3 3 3 3 3 3
13631 073/300 Flush
13632 074/300 Read block 2143 len 36864
13633 075/300 Write block 2175 len 28672 data: 3 3 3 3 3 3 3
13634 076/300 Write block 1546 len 8192 data: 3 3
13635 077/300 Read block 4349 len 40960
13636 078/300 Read block 4670 len 40960
13637 079/300 Flush
13638 080/300 Write block 1541 len 36864 data: 4 3 3 3 3 4 4 3 3
13639 081/300 Read block 1926 len 40960
13640 082/300 Write block 1912 len 24576 data: 4 3 3 3 3 3
13641 083/300 Write block 742 len 16384 data: 5 5 5 5
13642 084/300 Read block 1841 len 32768
13643 085/300 Flush
13644 086/300 Read block 3610 len 32768
13645 087/300 Write block 1017 len 20480 data: 4 3 3 3 4
13646 088/300 Read block 2832 len 4096
13647 089/300 Write block 757 len 20480 data: 4 6 6 6 6
13648 090/300 Write block 276 len 24576 data: 5 4 4 3 3 3
13649 091/300 Write block 752 len 36864 data: 3 3 3 3 3 5 7 7 7
13650 092/300 Write block 3482 len 36864 data: 2 2 2 2 2 2 2 2 2
13651 093/300 Read block 1422 len 32768
13652 094/300 Write block 634 len 32768 data: 3 3 3 2 2 2 2 2
13653 095/300 Flush
13654 096/300 Read block 4358 len 12288
13655 097/300 Read block 4173 len 36864
13656 098/300 Read block 3427 len 20480
13657 099/300 Read block 4776 len 24576
13658 100/300 Flush
13659 101/300 Write block 4048 len 40960 data: 6 6 5 5 3 3 4 5 5 4
13660 102/300 Write block 2914 len 4096 data: 5
13661 103/300 Write block 2415 len 24576 data: 3 3 3 2 2 2
13662 104/300 Write block 615 len 20480 data: 3 3 2 3 4
13663 105/300 Read block 1762 len 32768
13664 106/300 Write block 3869 len 32768 data: 3 3 3 2 2 2 2 2
13665 107/300 Write block 968 len 24576 data: 3 3 3 3 3 3
13666 108/300 Read block 1908 len 12288
13667 109/300 Write block 983 len 8192 data: 3 3
13668 110/300 Flush
13669 111/300 Write block 786 len 28672 data: 5 5 5 5 5 5 5
13670 112/300 Read block 3654 len 16384
13671 113/300 Write block 3659 len 40960 data: 4 2 4 4 5 4 5 5 5 4
13672 114/300 Read block 1488 len 16384
13673 115/300 Read block 4069 len 4096
13674 116/300 Write block 3079 len 4096 data: 4
13675 117/300 Flush
13676 118/300 Write block 4127 len 16384 data: 5 6 6 5
13677 119/300 Flush
13678 120/300 Read block 1407 len 40960
13679 121/300 Write block 3362 len 32768 data: 2 2 2 2 2 2 5 6
13680 122/300 Write block 4944 len 20480 data: 3 4 4 3 2
13681 123/300 Flush
13682 124/300 Write block 120 len 28672 data: 3 3 3 3 3 5 4
13683 125/300 Write block 2163 len 4096 data: 2
13684 126/300 Write block 4514 len 28672 data: 3 3 3 2 3 4 4
13685 127/300 Read block 527 len 4096
13686 128/300 Read block 1937 len 32768
13687 129/300 Flush
13688 130/300 Write block 4402 len 8192 data: 2 2
13689 131/300 Write block 3442 len 40960 data: 5 4 4 4 2 2 2 3 4 4
13690 132/300 Read block 3898 len 36864
13691 133/300 Flush
13692 134/300 Write block 3086 len 12288 data: 6 5 4
13693 135/300 Read block 1114 len 8192
13694 136/300 Read block 4326 len 28672
13695 137/300 Write block 319 len 24576 data: 3 3 3 4 4 4
13696 138/300 Flush
13697 139/300 Flush
13698 140/300 Read block 200 len 4096
13699 141/300 Write block 3009 len 4096 data: 5
13700 142/300 Flush
13701 143/300 Write block 4927 len 8192 data: 3 2
13702 144/300 Write block 1434 len 8192 data: 6 6
13703 145/300 Read block 305 len 16384
13704 146/300 Write block 2626 len 8192 data: 2 2
13705 147/300 Read block 2897 len 40960
13706 148/300 Read block 1853 len 40960
13707 149/300 Write block 4860 len 36864 data: 2 2 3 3 3 3 3 2 2
13708 150/300 Flush
13709 151/300 Write block 1286 len 40960 data: 3 3 2 2 3 4 5 4 5 5
13710 152/300 Flush
13711 153/300 Write block 4519 len 20480 data: 5 5 3 3 3
13712 154/300 Read block 4340 len 28672
13713 155/300 Read block 4242 len 40960
13714 156/300 Read block 524 len 24576
13715 157/300 Write block 820 len 24576 data: 4 3 3 3 4 6
13716 158/300 Read block 4398 len 28672
13717 159/300 Write block 4042 len 24576 data: 3 3 4 4 5 5
13718 160/300 Write block 3767 len 36864 data: 2 2 2 3 3 3 4 4 5
13719 161/300 Read block 55 len 4096
13720 162/300 Write block 4194 len 40960 data: 2 2 3 3 3 3 3 3 4 5
13721 163/300 Read block 1247 len 16384
13722 164/300 Read block 4537 len 20480
13723 165/300 Flush
13724 166/300 Flush
13725 167/300 Write block 4199 len 40960 data: 4 4 4 5 6 5 5 5 5 7
13726 168/300 Read block 2782 len 32768
13727 169/300 Read block 4012 len 32768
13728 170/300 Read block 2008 len 36864
13729 171/300 Read block 488 len 12288
13730 172/300 Flush
13731 173/300 Write block 4876 len 24576 data: 4 4 3 3 3 2
13732 174/300 Write block 4016 len 24576 data: 3 4 4 3 3 3
13733 175/300 Write block 4601 len 28672 data: 4 4 4 3 3 3 3
13734 176/300 Read block 3853 len 4096
13735 177/300 Read block 2493 len 36864
13736 178/300 Write block 1093 len 28672 data: 4 4 5 4 4 4 4
13737 179/300 Write block 2391 len 28672 data: 5 4 5 5 5 3 3
13738 180/300 Read block 4775 len 24576
13739 181/300 Write block 4311 len 32768 data: 3 3 4 4 4 4 4 4
13740 182/300 Write block 15 len 4096 data: 5
13741 183/300 Read block 4486 len 36864
13742 184/300 Read block 2084 len 16384
13743 185/300 Write block 3938 len 36864 data: 5 5 5 5 4 4 4 4 4
13744 186/300 Read block 2279 len 8192
13745 187/300 Write block 3096 len 40960 data: 3 3 3 3 4 3 3 3 3 3
13746 188/300 Flush
13747 189/300 Read block 2688 len 32768
13748 190/300 Write block 767 len 40960 data: 3 3 3 3 3 4 4 4 4 3
13749 191/300 Flush
13750 192/300 Write block 4432 len 32768 data: 4 5 5 4 4 4 4 4
13751 193/300 Read block 3959 len 20480
13752 194/300 Read block 1659 len 16384
13753 195/300 Read block 2249 len 40960
13754 196/300 Flush
13755 197/300 Read block 4399 len 12288
13756 198/300 Write block 2196 len 8192 data: 2 2
13757 199/300 Read block 2433 len 28672
13758 200/300 Flush
13759 201/300 Write block 2241 len 16384 data: 5 4 4 4
13760 202/300 Write block 4957 len 20480 data: 3 4 4 4 4
13761 203/300 Read block 1359 len 36864
13762 204/300 Read block 1337 len 36864
13763 205/300 Read block 3337 len 12288
13764 206/300 Read block 3189 len 24576
13765 207/300 Read block 1402 len 32768
13766 208/300 Write block 2679 len 8192 data: 3 3
13767 209/300 Write block 1776 len 4096 data: 5
13768 210/300 Read block 4244 len 28672
13769 211/300 Read block 4837 len 20480
13770 212/300 Read block 3351 len 12288
13771 213/300 Write block 678 len 4096 data: 4
13772 214/300 Write block 3800 len 36864 data: 3 3 3 3 3 2 2 2 3
13773 215/300 Read block 1207 len 32768
13774 216/300 Write block 1718 len 36864 data: 4 5 5 5 5 4 5 4 4
13775 217/300 Flush
13776 218/300 Write block 4699 len 40960 data: 6 5 5 4 4 5 5 4 4 3
13777 219/300 Read block 1904 len 4096
13778 220/300 Write block 1821 len 8192 data: 5 6
13779 221/300 Read block 3379 len 16384
13780 222/300 Read block 1925 len 40960
13781 223/300 Read block 4453 len 12288
13782 224/300 Read block 1785 len 36864
13783 225/300 Write block 3758 len 12288 data: 4 3 3
13784 226/300 Read block 4029 len 8192
13785 227/300 Read block 381 len 4096
13786 228/300 Write block 2527 len 24576 data: 2 2 2 2 4 4
13787 229/300 Read block 1742 len 36864
13788 230/300 Read block 68 len 16384
13789 231/300 Write block 4079 len 32768 data: 3 4 4 4 4 3 3 3
13790 232/300 Read block 3999 len 36864
13791 233/300 Write block 3407 len 20480 data: 3 3 3 3 3
13792 234/300 Read block 943 len 12288
13793 235/300 Read block 2225 len 20480
13794 236/300 Write block 1258 len 40960 data: 2 2 2 2 2 3 3 3 4 5
13795 237/300 Write block 2160 len 24576 data: 2 2 2 3 3 3
13796 238/300 Read block 4890 len 12288
13797 239/300 Read block 4396 len 12288
13798 240/300 Write block 3485 len 4096 data: 3
13799 241/300 Flush
13800 242/300 Write block 421 len 24576 data: 4 4 6 6 6 6
13801 243/300 Read block 4278 len 36864
13802 244/300 Read block 3756 len 32768
13803 245/300 Read block 1465 len 32768
13804 246/300 Flush
13805 247/300 Read block 536 len 32768
13806 248/300 Read block 2718 len 40960
13807 249/300 Read block 4435 len 36864
13808 250/300 Read block 3957 len 12288
13809 251/300 Flush
13810 252/300 Read block 1728 len 16384
13811 253/300 Read block 2366 len 12288
13812 254/300 Read block 4345 len 8192
13813 255/300 Read block 3900 len 20480
13814 256/300 Read block 768 len 4096
13815 257/300 Write block 4831 len 36864 data: 2 3 3 3 3 4 5 5 5
13816 258/300 Read block 948 len 12288
13817 259/300 Write block 4608 len 20480 data: 3 3 3 3 4
13818 260/300 Flush
13819 261/300 Write block 2719 len 32768 data: 3 3 3 3 3 2 3 4
13820 262/300 Read block 1576 len 32768
13821 263/300 Write block 2107 len 24576 data: 3 3 3 4 4 3
13822 264/300 Flush
13823 265/300 Read block 2901 len 24576
13824 266/300 Read block 4182 len 20480
13825 267/300 Read block 4256 len 16384
13826 268/300 Read block 324 len 24576
13827 269/300 Write block 3498 len 16384 data: 3 3 3 3
13828 270/300 Read block 4654 len 8192
13829 271/300 Read block 2626 len 32768
13830 272/300 Write block 761 len 8192 data: 7 6
13831 273/300 Flush
13832 274/300 Read block 2588 len 4096
13833 275/300 Write block 1299 len 28672 data: 5 4 3 2 2 2 2
13834 276/300 Flush
13835 277/300 Write block 3587 len 32768 data: 3 2 2 2 2 2 2 2
13836 278/300 Read block 1655 len 28672
13837 279/300 Read block 1013 len 32768
13838 280/300 Write block 729 len 4096 data: 4
13839 281/300 Read block 493 len 4096
13840 282/300 Read block 3645 len 40960
13841 283/300 Flush
13842 284/300 Read block 874 len 8192
13843 285/300 Read block 3334 len 32768
13844 286/300 Write block 3567 len 16384 data: 5 4 4 3
13845 287/300 Flush
13846 288/300 Read block 755 len 36864
13847 289/300 Read block 898 len 4096
13848 290/300 Flush
13849 291/300 Read block 1803 len 16384
13850 292/300 Write block 3444 len 8192 data: 5 5
13851 293/300 Write block 1899 len 8192 data: 2 2
13852 294/300 Read block 941 len 40960
13853 295/300 Read block 2581 len 20480
13854 296/300 Write block 3294 len 20480 data: 4 3 3 4 4
13855 297/300 Write block 686 len 8192 data: 2 3
13856 298/300 Write block 4917 len 24576 data: 5 5 4 4 3 3
13857 299/300 Write block 2683 len 24576 data: 5 4 3 3 2 2
13858 300/300 Write block 1078 len 32768 data: 2 2 3 3 3 2 2 2
13859 301/300 Write block 2018 len 36864 data: 6 5 3 2 2 2 3 3 3
13860 Main task has work to do, go find it
13861 got dsc Start(1)
13862 start 1
13863 Mar 07 18:15:35.569 INFO request completed, latency_us: 192, response_code: 204, uri: /start/cid/1, method: POST, req_id: 53b62d30-4ab8-4224-af8f-0c467eef930e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
13864 Replay: started 1, returned:Ok(())
13865 ----------------------------------------------------------------
13866 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13867 States: Active Offline Active
13868 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13869 4320 Acked Flush 0 Done Sent Done false
13870 4321 Acked Read 3 Done Sent Done false
13871 4322 Acked Write 6 Done Sent Done false
13872 4323 Acked Read 7 Done Sent Done false
13873 4324 Acked Write 5 Done Sent Done false
13874 4325 Acked Write 6 Done Sent Done false
13875 4326 Acked Read 2 Done Sent Done false
13876 4327 Acked Read 10 Done Sent Done false
13877 4328 Acked Read 6 Done Sent Done false
13878 4329 Acked Write 10 Done Sent Done false
13879 4330 Acked Read 3 Done Sent Done false
13880 4331 Acked Write 7 Done Sent Done false
13881 4332 Acked Flush 0 Done Sent Done false
13882 4333 Acked Write 8 Done Sent Done false
13883 4334 Acked Write 5 Done Sent Done false
13884 4335 Acked Write 9 Done Sent Done false
13885 4336 Acked Write 9 Done Sent Done false
13886 4337 Acked Read 7 Done Sent Done false
13887 4338 Acked Flush 0 Done Sent Done false
13888 4339 Acked Write 4 Done Sent Done false
13889 4340 Acked Write 10 Done Sent Done false
13890 4341 Acked Write 1 Done Sent Done false
13891 4342 Acked Write 7 Done Sent Done false
13892 4343 Acked Flush 0 Done Sent Done false
13893 4344 Acked Flush 0 Done Sent Done false
13894 4345 Acked Read 8 Done Sent Done false
13895 4346 Acked Read 9 Done Sent Done false
13896 4347 Acked Write 10 Done Sent Done false
13897 4348 Acked Write 4 Done Sent Done false
13898 4349 Acked Read 7 Done Sent Done false
13899 4350 Acked Write 9 Done Sent Done false
13900 4351 Acked Write 1 Done Sent Done false
13901 4352 Acked Flush 0 Done Sent Done false
13902 4353 Acked Write 8 Done Sent Done false
13903 4354 Acked Read 6 Done Sent Done false
13904 4355 Acked Read 7 Done Sent Done false
13905 4356 Acked Flush 0 Done Sent Done false
13906 4357 Acked Read 8 Done Sent Done false
13907 4358 Acked Read 4 Done Sent Done false
13908 4359 Acked Read 4 Done Sent Done false
13909 4360 Acked Write 5 Done Sent Done false
13910 4361 Acked Read 4 Done Sent Done false
13911 4362 Acked Flush 0 Done Sent Done false
13912 4363 Acked Read 10 Done Sent Done false
13913 4364 Acked Read 6 Done Sent Done false
13914 4365 Acked Read 8 Done Sent Done false
13915 4366 Acked Read 6 Done Sent Done false
13916 4367 Acked Read 5 Done Sent Done false
13917 4368 Acked Read 2 Done Sent Done false
13918 4369 Acked Read 8 Done Sent Done false
13919 4370 Acked Write 9 Done Sent Done false
13920 4371 Acked Write 4 Done Sent Done false
13921 4372 Acked Read 8 Done Sent Done false
13922 4373 Acked Read 4 Done Sent Done false
13923 4374 Acked Write 1 Done Sent Done false
13924 4375 Acked Flush 0 Done Sent Done false
13925 4376 Acked Flush 0 Done Sent Done false
13926 4377 Acked Write 1 Done Sent Done false
13927 4378 Acked Write 5 Done Sent Done false
13928 4379 Acked Read 2 Done Sent Done false
13929 4380 Acked Read 8 Done Sent Done false
13930 4381 Acked Read 5 Done Sent Done false
13931 4382 Acked Write 2 Done Sent Done false
13932 4383 Acked Write 8 Done Sent Done false
13933 4384 Acked Read 8 Done Sent Done false
13934 4385 Acked Read 4 Done Sent Done false
13935 4386 Acked Read 6 Done Sent Done false
13936 4387 Acked Read 1 Done Sent Done false
13937 4388 Acked Read 8 Done Sent Done false
13938 4389 Acked Read 7 Done Sent Done false
13939 4390 Acked Write 7 Done Sent Done false
13940 4391 Acked Write 10 Done Sent Done false
13941 4392 Acked Flush 0 Done Sent Done false
13942 4393 Acked Read 9 Done Sent Done false
13943 4394 Acked Write 7 Done Sent Done false
13944 4395 Acked Write 2 Done Sent Done false
13945 4396 Acked Read 10 Done Sent Done false
13946 4397 Acked Read 10 Done Sent Done false
13947 4398 Acked Flush 0 Done Sent Done false
13948 4399 Acked Write 9 Done Sent Done false
13949 4400 Acked Read 10 Done Sent Done false
13950 4401 Acked Write 6 Done Sent Done false
13951 4402 Acked Write 4 Done Sent Done false
13952 4403 Acked Read 8 Done Sent Done false
13953 4404 Acked Flush 0 Done Sent Done false
13954 4405 Acked Read 8 Done Sent Done false
13955 4406 Acked Write 5 Done Sent Done false
13956 4407 Acked Read 1 Done Sent Done false
13957 4408 Acked Write 5 Done Sent Done false
13958 4409 Acked Write 6 Done Sent Done false
13959 4410 Acked Write 9 Done Sent Done false
13960 4411 Acked Write 9 Done Sent Done false
13961 4412 Acked Read 8 Done Sent Done false
13962 4413 Acked Write 8 Done Sent Done false
13963 4414 Acked Flush 0 Done Sent Done false
13964 4415 Acked Read 3 Done Sent Done false
13965 4416 Acked Read 9 Done Sent Done false
13966 4417 Acked Read 5 Done Sent Done false
13967 4418 Acked Read 6 Done Sent Done false
13968 4419 Acked Flush 0 Done Sent Done false
13969 4420 Acked Write 10 Done Sent Done false
13970 4421 Acked Write 1 Done Sent Done false
13971 4422 Acked Write 6 Done Sent Done false
13972 4423 Acked Write 5 Done Sent Done false
13973 4424 Acked Read 8 Done Sent Done false
13974 4425 Acked Write 8 Done Sent Done false
13975 4426 Acked Write 6 Done Sent Done false
13976 4427 Acked Read 3 Done Sent Done false
13977 4428 Acked Write 2 Done Sent Done false
13978 4429 Acked Flush 0 Done Sent Done false
13979 4430 Acked Write 7 Done Sent Done false
13980 4431 Acked Read 4 Done Sent Done false
13981 4432 Acked Write 10 Done Sent Done false
13982 4433 Acked Read 4 Done Sent Done false
13983 4434 Acked Read 1 Done Sent Done false
13984 4435 Acked Write 1 Done Sent Done false
13985 4436 Acked Flush 0 Done Sent Done false
13986 4437 Acked Write 4 Done Sent Done false
13987 4438 Acked Flush 0 Done Sent Done false
13988 4439 Acked Read 10 Done Sent Done false
13989 4440 Acked Write 8 Done Sent Done false
13990 4441 Acked Write 5 Done Sent Done false
13991 4442 Acked Flush 0 Done Sent Done false
13992 4443 Acked Write 7 Done Sent Done false
13993 4444 Acked Write 1 Done Sent Done false
13994 4445 Acked Write 7 Done Sent Done false
13995 4446 Acked Read 1 Done Sent Done false
13996 4447 Acked Read 8 Done Sent Done false
13997 4448 Acked Flush 0 Done Sent Done false
13998 4449 Acked Write 2 Done Sent Done false
13999 4450 Acked Write 10 Done Sent Done false
14000 4451 Acked Read 9 Done Sent Done false
14001 4452 Acked Flush 0 Done Sent Done false
14002 4453 Acked Write 3 Done Sent Done false
14003 4454 Acked Read 2 Done Sent Done false
14004 4455 Acked Read 7 Done Sent Done false
14005 4456 Acked Write 6 Done Sent Done false
14006 4457 Acked Flush 0 Done Sent Done false
14007 4458 Acked Flush 0 Done Sent Done false
14008 4459 Acked Read 1 Done Sent Done false
14009 4460 Acked Write 1 Done Sent Done false
14010 4461 Acked Flush 0 Done Sent Done false
14011 4462 Acked Write 2 Done Sent Done false
14012 4463 Acked Write 2 Done Sent Done false
14013 4464 Acked Read 4 Done Sent Done false
14014 4465 Acked Write 2 Done Sent Done false
14015 4466 Acked Read 10 Done Sent Done false
14016 4467 Acked Read 10 Done Sent Done false
14017 4468 Acked Write 9 Done Sent Done false
14018 4469 Acked Flush 0 Done Sent Done false
14019 4470 Acked Write 10 Done Sent Done false
14020 4471 Acked Flush 0 Done Sent Done false
14021 4472 Acked Write 5 Done Sent Done false
14022 4473 Acked Read 7 Done Sent Done false
14023 4474 Acked Read 10 Done Sent Done false
14024 4475 Acked Read 6 Done Sent Done false
14025 4476 Acked Write 6 Done Sent Done false
14026 4477 Acked Read 7 Done Sent Done false
14027 4478 Acked Write 6 Done Sent Done false
14028 4479 Acked Write 9 Done Sent Done false
14029 4480 Acked Read 1 Done Sent Done false
14030 4481 Acked Write 10 Done Sent Done false
14031 4482 Acked Read 4 Done Sent Done false
14032 4483 Acked Read 5 Done Sent Done false
14033 4484 Acked Flush 0 Done Sent Done false
14034 4485 Acked Flush 0 Done Sent Done false
14035 4486 Acked Write 10 Done Sent Done false
14036 4487 Acked Read 8 Done Sent Done false
14037 4488 Acked Read 8 Done Sent Done false
14038 4489 Acked Read 9 Done Sent Done false
14039 4490 Acked Read 3 Done Sent Done false
14040 4491 Acked Flush 0 Done Sent Done false
14041 4492 Acked Write 6 Done Sent Done false
14042 4493 Acked Write 6 Done Sent Done false
14043 4494 Acked Write 7 Done Sent Done false
14044 4495 Acked Read 1 Done Sent Done false
14045 4496 Acked Read 9 Done Sent Done false
14046 4497 Acked Write 7 Done Sent Done false
14047 4498 Acked Write 7 Done Sent Done false
14048 4499 Acked Read 6 Done Sent Done false
14049 4500 Acked Write 8 Done Sent Done false
14050 4501 Acked Write 1 Done Sent Done false
14051 4502 Acked Read 9 Done Sent Done false
14052 4503 Acked Read 4 Done Sent Done false
14053 4504 Acked Write 9 Done Sent Done false
14054 4505 Acked Read 2 Done Sent Done false
14055 4506 Acked Write 10 Done Sent Done false
14056 4507 Acked Flush 0 Done Sent Done false
14057 4508 Acked Read 8 Done Sent Done false
14058 4509 Acked Write 10 Done Sent Done false
14059 4510 Acked Flush 0 Done Sent Done false
14060 4511 Acked Write 8 Done Sent Done false
14061 4512 Acked Read 5 Done Sent Done false
14062 4513 Acked Read 4 Done Sent Done false
14063 4514 Acked Read 10 Done Sent Done false
14064 4515 Acked Flush 0 Done Sent Done false
14065 4516 Acked Read 3 Done Sent Done false
14066 4517 Acked Write 2 Done Sent Done false
14067 4518 Acked Read 7 Done Sent Done false
14068 4519 Acked Flush 0 Done Sent Done false
14069 4520 Acked Write 4 Done Sent Done false
14070 4521 Acked Write 5 Done Sent Done false
14071 4522 Acked Read 9 Done Sent Done false
14072 4523 Acked Read 9 Done Sent Done false
14073 4524 Acked Read 3 Done Sent Done false
14074 4525 Acked Read 6 Done Sent Done false
14075 4526 Acked Read 8 Done Sent Done false
14076 4527 Acked Write 2 Done Sent Done false
14077 4528 Acked Write 1 Done Sent Done false
14078 4529 Acked Read 7 Done Sent Done false
14079 4530 Acked Read 5 Done Sent Done false
14080 4531 Acked Read 3 Done Sent Done false
14081 4532 Acked Write 1 Done Sent Done false
14082 4533 Acked Write 9 Done Sent Done false
14083 4534 Acked Read 8 Done Sent Done false
14084 4535 Acked Write 9 Done Sent Done false
14085 4536 Acked Flush 0 Done Sent Done false
14086 4537 Acked Write 10 Done Sent Done false
14087 4538 Acked Read 1 Done Sent Done false
14088 4539 Acked Write 2 Done Sent Done false
14089 4540 Acked Read 4 Done Sent Done false
14090 4541 Acked Read 10 Done Sent Done false
14091 4542 Acked Read 3 Done Sent Done false
14092 4543 Acked Read 9 Done Sent Done false
14093 4544 Acked Write 3 Done Sent Done false
14094 4545 Acked Read 2 Done Sent Done false
14095 4546 Acked Read 1 Done Sent Done false
14096 4547 Acked Write 6 Done Sent Done false
14097 4548 Acked Read 9 Done Sent Done false
14098 4549 Acked Read 4 Done Sent Done false
14099 4550 Acked Write 8 Done Sent Done false
14100 4551 Acked Read 9 Done Sent Done false
14101 4552 Acked Write 5 Done Sent Done false
14102 4553 Acked Read 3 Done Sent Done false
14103 4554 Acked Read 5 Done Sent Done false
14104 4555 Acked Write 10 Done Sent Done false
14105 4556 Acked Write 6 Done Sent Done false
14106 4557 Acked Read 3 Done Sent Done false
14107 4558 Acked Read 3 Done Sent Done false
14108 4559 Acked Write 1 Done Sent Done false
14109 4560 Acked Flush 0 Done Sent Done false
14110 4561 Acked Write 6 Done Sent Done false
14111 4562 Acked Read 9 Done Sent Done false
14112 4563 Acked Read 8 Done Sent Done false
14113 4564 Acked Read 8 Done Sent Done false
14114 4565 Acked Flush 0 Done Sent Done false
14115 4566 Acked Read 8 Done Sent Done false
14116 4567 Acked Read 10 Done Sent Done false
14117 4568 Acked Read 9 Done Sent Done false
14118 4569 Acked Read 3 Done Sent Done false
14119 4570 Acked Flush 0 Done Sent Done false
14120 4571 Acked Read 4 Done Sent Done false
14121 4572 Acked Read 3 Done Sent Done false
14122 4573 Acked Read 2 Done Sent Done false
14123 4574 Acked Read 5 Done Sent Done false
14124 4575 Acked Read 1 Done Sent Done false
14125 4576 Acked Write 9 Done Sent Done false
14126 4577 Acked Read 3 Done Sent Done false
14127 4578 Acked Write 5 Done Sent Done false
14128 4579 Acked Flush 0 Done Sent Done false
14129 4580 Acked Write 8 Done Sent Done false
14130 4581 Acked Read 8 Done Sent Done false
14131 4582 Acked Write 6 Done Sent Done false
14132 4583 Acked Flush 0 Done Sent Done false
14133 4584 Acked Read 6 Done Sent Done false
14134 4585 Acked Read 5 Done Sent Done false
14135 4586 Acked Read 4 Done Sent Done false
14136 4587 Acked Read 6 Done Sent Done false
14137 4588 Acked Write 4 Done Sent Done false
14138 4589 Acked Read 2 Done Sent Done false
14139 4590 Acked Read 8 Done Sent Done false
14140 4591 Acked Write 2 Done Sent Done false
14141 4592 Acked Flush 0 Done Sent Done false
14142 4593 Acked Read 1 Done Sent Done false
14143 4594 Acked Write 7 Done Sent Done false
14144 4595 Acked Flush 0 Done Sent Done false
14145 4596 Acked Write 8 Done Sent Done false
14146 4597 Acked Read 7 Done Sent Done false
14147 4598 Acked Read 8 Done Sent Done false
14148 4599 Acked Write 1 Done Sent Done false
14149 4600 Acked Read 1 Done Sent Done false
14150 4601 Acked Read 10 Done Sent Done false
14151 4602 Acked Flush 0 Done Sent Done false
14152 4603 Acked Read 2 Done Sent Done false
14153 4604 Acked Read 8 Done Sent Done false
14154 4605 Acked Write 4 Done Sent Done false
14155 4606 Acked Flush 0 Done Sent Done false
14156 4607 Acked Read 9 Done Sent Done false
14157 4608 Acked Read 1 Done Sent Done false
14158 4609 Acked Flush 0 Done Sent Done false
14159 4610 Acked Read 4 Done Sent Done false
14160 4611 Acked Write 2 Done Sent Done false
14161 4612 Acked Write 2 Done Sent Done false
14162 4613 Acked Read 10 Done Sent Done false
14163 4614 Acked Read 5 Done Sent Done false
14164 4615 Acked Write 5 Done Sent Done false
14165 4616 Acked Write 2 Done Sent Done false
14166 4617 Acked Write 6 Done Sent Done false
14167 4618 Acked Write 6 Done Sent Done false
14168 4619 Acked Write 8 Done Sent Done false
14169 4620 Acked Write 9 Done Sent Sent false
14170 STATES DS:0 DS:1 DS:2 TOTAL
14171 Sent 0 301 1 302
14172 Done 301 0 300 601
14173 Skipped 0 0 0 0
14174 Error 0 0 0 0
14175 Last Flush: Some(JobId(4609)) Some(JobId(4319)) Some(JobId(4609))
14176 Downstairs last five completed: 4319 4318 4317 4316 4315
14177 Upstairs last five completed: 4620 4619 4618 4617 4616
14178 CLIENT: Up:0 ds:301 act:2
141792025-03-07T18:15:36.246ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
141802025-03-07T18:15:37.278ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
141812025-03-07T18:15:38.249ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
141822025-03-07T18:15:39.562ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14183 ----------------------------------------------------------------
14184 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14185 States: Active Offline Active
14186 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14187 4320 Acked Flush 0 Done Sent Done false
14188 4321 Acked Read 3 Done Sent Done false
14189 4322 Acked Write 6 Done Sent Done false
14190 4323 Acked Read 7 Done Sent Done false
14191 4324 Acked Write 5 Done Sent Done false
14192 4325 Acked Write 6 Done Sent Done false
14193 4326 Acked Read 2 Done Sent Done false
14194 4327 Acked Read 10 Done Sent Done false
14195 4328 Acked Read 6 Done Sent Done false
14196 4329 Acked Write 10 Done Sent Done false
14197 4330 Acked Read 3 Done Sent Done false
14198 4331 Acked Write 7 Done Sent Done false
14199 4332 Acked Flush 0 Done Sent Done false
14200 4333 Acked Write 8 Done Sent Done false
14201 4334 Acked Write 5 Done Sent Done false
14202 4335 Acked Write 9 Done Sent Done false
14203 4336 Acked Write 9 Done Sent Done false
14204 4337 Acked Read 7 Done Sent Done false
14205 4338 Acked Flush 0 Done Sent Done false
14206 4339 Acked Write 4 Done Sent Done false
14207 4340 Acked Write 10 Done Sent Done false
14208 4341 Acked Write 1 Done Sent Done false
14209 4342 Acked Write 7 Done Sent Done false
14210 4343 Acked Flush 0 Done Sent Done false
14211 4344 Acked Flush 0 Done Sent Done false
14212 4345 Acked Read 8 Done Sent Done false
14213 4346 Acked Read 9 Done Sent Done false
14214 4347 Acked Write 10 Done Sent Done false
14215 4348 Acked Write 4 Done Sent Done false
14216 4349 Acked Read 7 Done Sent Done false
14217 4350 Acked Write 9 Done Sent Done false
14218 4351 Acked Write 1 Done Sent Done false
14219 4352 Acked Flush 0 Done Sent Done false
14220 4353 Acked Write 8 Done Sent Done false
14221 4354 Acked Read 6 Done Sent Done false
14222 4355 Acked Read 7 Done Sent Done false
14223 4356 Acked Flush 0 Done Sent Done false
14224 4357 Acked Read 8 Done Sent Done false
14225 4358 Acked Read 4 Done Sent Done false
14226 4359 Acked Read 4 Done Sent Done false
14227 4360 Acked Write 5 Done Sent Done false
14228 4361 Acked Read 4 Done Sent Done false
14229 4362 Acked Flush 0 Done Sent Done false
14230 4363 Acked Read 10 Done Sent Done false
14231 4364 Acked Read 6 Done Sent Done false
14232 4365 Acked Read 8 Done Sent Done false
14233 4366 Acked Read 6 Done Sent Done false
14234 4367 Acked Read 5 Done Sent Done false
14235 4368 Acked Read 2 Done Sent Done false
14236 4369 Acked Read 8 Done Sent Done false
14237 4370 Acked Write 9 Done Sent Done false
14238 4371 Acked Write 4 Done Sent Done false
14239 4372 Acked Read 8 Done Sent Done false
14240 4373 Acked Read 4 Done Sent Done false
14241 4374 Acked Write 1 Done Sent Done false
14242 4375 Acked Flush 0 Done Sent Done false
14243 4376 Acked Flush 0 Done Sent Done false
14244 4377 Acked Write 1 Done Sent Done false
14245 4378 Acked Write 5 Done Sent Done false
14246 4379 Acked Read 2 Done Sent Done false
14247 4380 Acked Read 8 Done Sent Done false
14248 4381 Acked Read 5 Done Sent Done false
14249 4382 Acked Write 2 Done Sent Done false
14250 4383 Acked Write 8 Done Sent Done false
14251 4384 Acked Read 8 Done Sent Done false
14252 4385 Acked Read 4 Done Sent Done false
14253 4386 Acked Read 6 Done Sent Done false
14254 4387 Acked Read 1 Done Sent Done false
14255 4388 Acked Read 8 Done Sent Done false
14256 4389 Acked Read 7 Done Sent Done false
14257 4390 Acked Write 7 Done Sent Done false
14258 4391 Acked Write 10 Done Sent Done false
14259 4392 Acked Flush 0 Done Sent Done false
14260 4393 Acked Read 9 Done Sent Done false
14261 4394 Acked Write 7 Done Sent Done false
14262 4395 Acked Write 2 Done Sent Done false
14263 4396 Acked Read 10 Done Sent Done false
14264 4397 Acked Read 10 Done Sent Done false
14265 4398 Acked Flush 0 Done Sent Done false
14266 4399 Acked Write 9 Done Sent Done false
14267 4400 Acked Read 10 Done Sent Done false
14268 4401 Acked Write 6 Done Sent Done false
14269 4402 Acked Write 4 Done Sent Done false
14270 4403 Acked Read 8 Done Sent Done false
14271 4404 Acked Flush 0 Done Sent Done false
14272 4405 Acked Read 8 Done Sent Done false
14273 4406 Acked Write 5 Done Sent Done false
14274 4407 Acked Read 1 Done Sent Done false
14275 4408 Acked Write 5 Done Sent Done false
14276 4409 Acked Write 6 Done Sent Done false
14277 4410 Acked Write 9 Done Sent Done false
14278 4411 Acked Write 9 Done Sent Done false
14279 4412 Acked Read 8 Done Sent Done false
14280 4413 Acked Write 8 Done Sent Done false
14281 4414 Acked Flush 0 Done Sent Done false
14282 4415 Acked Read 3 Done Sent Done false
14283 4416 Acked Read 9 Done Sent Done false
14284 4417 Acked Read 5 Done Sent Done false
14285 4418 Acked Read 6 Done Sent Done false
14286 4419 Acked Flush 0 Done Sent Done false
14287 4420 Acked Write 10 Done Sent Done false
14288 4421 Acked Write 1 Done Sent Done false
14289 4422 Acked Write 6 Done Sent Done false
14290 4423 Acked Write 5 Done Sent Done false
14291 4424 Acked Read 8 Done Sent Done false
14292 4425 Acked Write 8 Done Sent Done false
14293 4426 Acked Write 6 Done Sent Done false
14294 4427 Acked Read 3 Done Sent Done false
14295 4428 Acked Write 2 Done Sent Done false
14296 4429 Acked Flush 0 Done Sent Done false
14297 4430 Acked Write 7 Done Sent Done false
14298 4431 Acked Read 4 Done Sent Done false
14299 4432 Acked Write 10 Done Sent Done false
14300 4433 Acked Read 4 Done Sent Done false
14301 4434 Acked Read 1 Done Sent Done false
14302 4435 Acked Write 1 Done Sent Done false
14303 4436 Acked Flush 0 Done Sent Done false
14304 4437 Acked Write 4 Done Sent Done false
14305 4438 Acked Flush 0 Done Sent Done false
14306 4439 Acked Read 10 Done Sent Done false
14307 4440 Acked Write 8 Done Sent Done false
14308 4441 Acked Write 5 Done Sent Done false
14309 4442 Acked Flush 0 Done Sent Done false
14310 4443 Acked Write 7 Done Sent Done false
14311 4444 Acked Write 1 Done Sent Done false
14312 4445 Acked Write 7 Done Sent Done false
14313 4446 Acked Read 1 Done Sent Done false
14314 4447 Acked Read 8 Done Sent Done false
14315 4448 Acked Flush 0 Done Sent Done false
14316 4449 Acked Write 2 Done Sent Done false
14317 4450 Acked Write 10 Done Sent Done false
14318 4451 Acked Read 9 Done Sent Done false
14319 4452 Acked Flush 0 Done Sent Done false
14320 4453 Acked Write 3 Done Sent Done false
14321 4454 Acked Read 2 Done Sent Done false
14322 4455 Acked Read 7 Done Sent Done false
14323 4456 Acked Write 6 Done Sent Done false
14324 4457 Acked Flush 0 Done Sent Done false
14325 4458 Acked Flush 0 Done Sent Done false
14326 4459 Acked Read 1 Done Sent Done false
14327 4460 Acked Write 1 Done Sent Done false
14328 4461 Acked Flush 0 Done Sent Done false
14329 4462 Acked Write 2 Done Sent Done false
14330 4463 Acked Write 2 Done Sent Done false
14331 4464 Acked Read 4 Done Sent Done false
14332 4465 Acked Write 2 Done Sent Done false
14333 4466 Acked Read 10 Done Sent Done false
14334 4467 Acked Read 10 Done Sent Done false
14335 4468 Acked Write 9 Done Sent Done false
14336 4469 Acked Flush 0 Done Sent Done false
14337 4470 Acked Write 10 Done Sent Done false
14338 4471 Acked Flush 0 Done Sent Done false
14339 4472 Acked Write 5 Done Sent Done false
14340 4473 Acked Read 7 Done Sent Done false
14341 4474 Acked Read 10 Done Sent Done false
14342 4475 Acked Read 6 Done Sent Done false
14343 4476 Acked Write 6 Done Sent Done false
14344 4477 Acked Read 7 Done Sent Done false
14345 4478 Acked Write 6 Done Sent Done false
14346 4479 Acked Write 9 Done Sent Done false
14347 4480 Acked Read 1 Done Sent Done false
14348 4481 Acked Write 10 Done Sent Done false
14349 4482 Acked Read 4 Done Sent Done false
14350 4483 Acked Read 5 Done Sent Done false
14351 4484 Acked Flush 0 Done Sent Done false
14352 4485 Acked Flush 0 Done Sent Done false
14353 4486 Acked Write 10 Done Sent Done false
14354 4487 Acked Read 8 Done Sent Done false
14355 4488 Acked Read 8 Done Sent Done false
14356 4489 Acked Read 9 Done Sent Done false
14357 4490 Acked Read 3 Done Sent Done false
14358 4491 Acked Flush 0 Done Sent Done false
14359 4492 Acked Write 6 Done Sent Done false
14360 4493 Acked Write 6 Done Sent Done false
14361 4494 Acked Write 7 Done Sent Done false
14362 4495 Acked Read 1 Done Sent Done false
14363 4496 Acked Read 9 Done Sent Done false
14364 4497 Acked Write 7 Done Sent Done false
14365 4498 Acked Write 7 Done Sent Done false
14366 4499 Acked Read 6 Done Sent Done false
14367 4500 Acked Write 8 Done Sent Done false
14368 4501 Acked Write 1 Done Sent Done false
14369 4502 Acked Read 9 Done Sent Done false
14370 4503 Acked Read 4 Done Sent Done false
14371 4504 Acked Write 9 Done Sent Done false
14372 4505 Acked Read 2 Done Sent Done false
14373 4506 Acked Write 10 Done Sent Done false
14374 4507 Acked Flush 0 Done Sent Done false
14375 4508 Acked Read 8 Done Sent Done false
14376 4509 Acked Write 10 Done Sent Done false
14377 4510 Acked Flush 0 Done Sent Done false
14378 4511 Acked Write 8 Done Sent Done false
14379 4512 Acked Read 5 Done Sent Done false
14380 4513 Acked Read 4 Done Sent Done false
14381 4514 Acked Read 10 Done Sent Done false
14382 4515 Acked Flush 0 Done Sent Done false
14383 4516 Acked Read 3 Done Sent Done false
14384 4517 Acked Write 2 Done Sent Done false
14385 4518 Acked Read 7 Done Sent Done false
14386 4519 Acked Flush 0 Done Sent Done false
14387 4520 Acked Write 4 Done Sent Done false
14388 4521 Acked Write 5 Done Sent Done false
14389 4522 Acked Read 9 Done Sent Done false
14390 4523 Acked Read 9 Done Sent Done false
14391 4524 Acked Read 3 Done Sent Done false
14392 4525 Acked Read 6 Done Sent Done false
14393 4526 Acked Read 8 Done Sent Done false
14394 4527 Acked Write 2 Done Sent Done false
14395 4528 Acked Write 1 Done Sent Done false
14396 4529 Acked Read 7 Done Sent Done false
14397 4530 Acked Read 5 Done Sent Done false
14398 4531 Acked Read 3 Done Sent Done false
14399 4532 Acked Write 1 Done Sent Done false
14400 4533 Acked Write 9 Done Sent Done false
14401 4534 Acked Read 8 Done Sent Done false
14402 4535 Acked Write 9 Done Sent Done false
14403 4536 Acked Flush 0 Done Sent Done false
14404 4537 Acked Write 10 Done Sent Done false
14405 4538 Acked Read 1 Done Sent Done false
14406 4539 Acked Write 2 Done Sent Done false
14407 4540 Acked Read 4 Done Sent Done false
14408 4541 Acked Read 10 Done Sent Done false
14409 4542 Acked Read 3 Done Sent Done false
14410 4543 Acked Read 9 Done Sent Done false
14411 4544 Acked Write 3 Done Sent Done false
14412 4545 Acked Read 2 Done Sent Done false
14413 4546 Acked Read 1 Done Sent Done false
14414 4547 Acked Write 6 Done Sent Done false
14415 4548 Acked Read 9 Done Sent Done false
14416 4549 Acked Read 4 Done Sent Done false
14417 4550 Acked Write 8 Done Sent Done false
14418 4551 Acked Read 9 Done Sent Done false
14419 4552 Acked Write 5 Done Sent Done false
14420 4553 Acked Read 3 Done Sent Done false
14421 4554 Acked Read 5 Done Sent Done false
14422 4555 Acked Write 10 Done Sent Done false
14423 4556 Acked Write 6 Done Sent Done false
14424 4557 Acked Read 3 Done Sent Done false
14425 4558 Acked Read 3 Done Sent Done false
14426 4559 Acked Write 1 Done Sent Done false
14427 4560 Acked Flush 0 Done Sent Done false
14428 4561 Acked Write 6 Done Sent Done false
14429 4562 Acked Read 9 Done Sent Done false
14430 4563 Acked Read 8 Done Sent Done false
14431 4564 Acked Read 8 Done Sent Done false
14432 4565 Acked Flush 0 Done Sent Done false
14433 4566 Acked Read 8 Done Sent Done false
14434 4567 Acked Read 10 Done Sent Done false
14435 4568 Acked Read 9 Done Sent Done false
14436 4569 Acked Read 3 Done Sent Done false
14437 4570 Acked Flush 0 Done Sent Done false
14438 4571 Acked Read 4 Done Sent Done false
14439 4572 Acked Read 3 Done Sent Done false
14440 4573 Acked Read 2 Done Sent Done false
14441 4574 Acked Read 5 Done Sent Done false
14442 4575 Acked Read 1 Done Sent Done false
14443 4576 Acked Write 9 Done Sent Done false
14444 4577 Acked Read 3 Done Sent Done false
14445 4578 Acked Write 5 Done Sent Done false
14446 4579 Acked Flush 0 Done Sent Done false
14447 4580 Acked Write 8 Done Sent Done false
14448 4581 Acked Read 8 Done Sent Done false
14449 4582 Acked Write 6 Done Sent Done false
14450 4583 Acked Flush 0 Done Sent Done false
14451 4584 Acked Read 6 Done Sent Done false
14452 4585 Acked Read 5 Done Sent Done false
14453 4586 Acked Read 4 Done Sent Done false
14454 4587 Acked Read 6 Done Sent Done false
14455 4588 Acked Write 4 Done Sent Done false
14456 4589 Acked Read 2 Done Sent Done false
14457 4590 Acked Read 8 Done Sent Done false
14458 4591 Acked Write 2 Done Sent Done false
14459 4592 Acked Flush 0 Done Sent Done false
14460 4593 Acked Read 1 Done Sent Done false
14461 4594 Acked Write 7 Done Sent Done false
14462 4595 Acked Flush 0 Done Sent Done false
14463 4596 Acked Write 8 Done Sent Done false
14464 4597 Acked Read 7 Done Sent Done false
14465 4598 Acked Read 8 Done Sent Done false
14466 4599 Acked Write 1 Done Sent Done false
14467 4600 Acked Read 1 Done Sent Done false
14468 4601 Acked Read 10 Done Sent Done false
14469 4602 Acked Flush 0 Done Sent Done false
14470 4603 Acked Read 2 Done Sent Done false
14471 4604 Acked Read 8 Done Sent Done false
14472 4605 Acked Write 4 Done Sent Done false
14473 4606 Acked Flush 0 Done Sent Done false
14474 4607 Acked Read 9 Done Sent Done false
14475 4608 Acked Read 1 Done Sent Done false
14476 4609 Acked Flush 0 Done Sent Done false
14477 4610 Acked Read 4 Done Sent Done false
14478 4611 Acked Write 2 Done Sent Done false
14479 4612 Acked Write 2 Done Sent Done false
14480 4613 Acked Read 10 Done Sent Done false
14481 4614 Acked Read 5 Done Sent Done false
14482 4615 Acked Write 5 Done Sent Done false
14483 4616 Acked Write 2 Done Sent Done false
14484 4617 Acked Write 6 Done Sent Done false
14485 4618 Acked Write 6 Done Sent Done false
14486 4619 Acked Write 8 Done Sent Done false
14487 4620 Acked Write 9 Done Sent Done false
14488 STATES DS:0 DS:1 DS:2 TOTAL
14489 Sent 0 301 0 301
14490 Done 301 0 301 602
14491 Skipped 0 0 0 0
14492 Error 0 0 0 0
14493 Last Flush: Some(JobId(4609)) Some(JobId(4319)) Some(JobId(4609))
14494 Downstairs last five completed: 4319 4318 4317 4316 4315
14495 Upstairs last five completed: 4620 4619 4618 4617 4616
14496 CLIENT: Up:0 ds:301 act:2
144972025-03-07T18:15:40.251ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
144982025-03-07T18:15:41.253ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
144992025-03-07T18:15:41.318ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
145002025-03-07T18:15:41.318ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
145012025-03-07T18:15:41.318ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
145022025-03-07T18:15:41.318ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
145032025-03-07T18:15:41.318ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14504 [8820] got start action
14505 [8820] I am going to restart
14506 Starting downstairs at port 8820
14507 Make output file at "/tmp/dsc/downstairs-8820.txt"
14508 [8820][1] reports Starting
14509 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1220)
14510 [8820][1] reports Running
145112025-03-07T18:15:42.280ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
145122025-03-07T18:15:43.258ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14513 ----------------------------------------------------------------
14514 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14515 States: Active Offline Active
14516 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14517 4320 Acked Flush 0 Done Sent Done false
14518 4321 Acked Read 3 Done Sent Done false
14519 4322 Acked Write 6 Done Sent Done false
14520 4323 Acked Read 7 Done Sent Done false
14521 4324 Acked Write 5 Done Sent Done false
14522 4325 Acked Write 6 Done Sent Done false
14523 4326 Acked Read 2 Done Sent Done false
14524 4327 Acked Read 10 Done Sent Done false
14525 4328 Acked Read 6 Done Sent Done false
14526 4329 Acked Write 10 Done Sent Done false
14527 4330 Acked Read 3 Done Sent Done false
14528 4331 Acked Write 7 Done Sent Done false
14529 4332 Acked Flush 0 Done Sent Done false
14530 4333 Acked Write 8 Done Sent Done false
14531 4334 Acked Write 5 Done Sent Done false
14532 4335 Acked Write 9 Done Sent Done false
14533 4336 Acked Write 9 Done Sent Done false
14534 4337 Acked Read 7 Done Sent Done false
14535 4338 Acked Flush 0 Done Sent Done false
14536 4339 Acked Write 4 Done Sent Done false
14537 4340 Acked Write 10 Done Sent Done false
14538 4341 Acked Write 1 Done Sent Done false
14539 4342 Acked Write 7 Done Sent Done false
14540 4343 Acked Flush 0 Done Sent Done false
14541 4344 Acked Flush 0 Done Sent Done false
14542 4345 Acked Read 8 Done Sent Done false
14543 4346 Acked Read 9 Done Sent Done false
14544 4347 Acked Write 10 Done Sent Done false
14545 4348 Acked Write 4 Done Sent Done false
14546 4349 Acked Read 7 Done Sent Done false
14547 4350 Acked Write 9 Done Sent Done false
14548 4351 Acked Write 1 Done Sent Done false
14549 4352 Acked Flush 0 Done Sent Done false
14550 4353 Acked Write 8 Done Sent Done false
14551 4354 Acked Read 6 Done Sent Done false
14552 4355 Acked Read 7 Done Sent Done false
14553 4356 Acked Flush 0 Done Sent Done false
14554 4357 Acked Read 8 Done Sent Done false
14555 4358 Acked Read 4 Done Sent Done false
14556 4359 Acked Read 4 Done Sent Done false
14557 4360 Acked Write 5 Done Sent Done false
14558 4361 Acked Read 4 Done Sent Done false
14559 4362 Acked Flush 0 Done Sent Done false
14560 4363 Acked Read 10 Done Sent Done false
14561 4364 Acked Read 6 Done Sent Done false
14562 4365 Acked Read 8 Done Sent Done false
14563 4366 Acked Read 6 Done Sent Done false
14564 4367 Acked Read 5 Done Sent Done false
14565 4368 Acked Read 2 Done Sent Done false
14566 4369 Acked Read 8 Done Sent Done false
14567 4370 Acked Write 9 Done Sent Done false
14568 4371 Acked Write 4 Done Sent Done false
14569 4372 Acked Read 8 Done Sent Done false
14570 4373 Acked Read 4 Done Sent Done false
14571 4374 Acked Write 1 Done Sent Done false
14572 4375 Acked Flush 0 Done Sent Done false
14573 4376 Acked Flush 0 Done Sent Done false
14574 4377 Acked Write 1 Done Sent Done false
14575 4378 Acked Write 5 Done Sent Done false
14576 4379 Acked Read 2 Done Sent Done false
14577 4380 Acked Read 8 Done Sent Done false
14578 4381 Acked Read 5 Done Sent Done false
14579 4382 Acked Write 2 Done Sent Done false
14580 4383 Acked Write 8 Done Sent Done false
14581 4384 Acked Read 8 Done Sent Done false
14582 4385 Acked Read 4 Done Sent Done false
14583 4386 Acked Read 6 Done Sent Done false
14584 4387 Acked Read 1 Done Sent Done false
14585 4388 Acked Read 8 Done Sent Done false
14586 4389 Acked Read 7 Done Sent Done false
14587 4390 Acked Write 7 Done Sent Done false
14588 4391 Acked Write 10 Done Sent Done false
14589 4392 Acked Flush 0 Done Sent Done false
14590 4393 Acked Read 9 Done Sent Done false
14591 4394 Acked Write 7 Done Sent Done false
14592 4395 Acked Write 2 Done Sent Done false
14593 4396 Acked Read 10 Done Sent Done false
14594 4397 Acked Read 10 Done Sent Done false
14595 4398 Acked Flush 0 Done Sent Done false
14596 4399 Acked Write 9 Done Sent Done false
14597 4400 Acked Read 10 Done Sent Done false
14598 4401 Acked Write 6 Done Sent Done false
14599 4402 Acked Write 4 Done Sent Done false
14600 4403 Acked Read 8 Done Sent Done false
14601 4404 Acked Flush 0 Done Sent Done false
14602 4405 Acked Read 8 Done Sent Done false
14603 4406 Acked Write 5 Done Sent Done false
14604 4407 Acked Read 1 Done Sent Done false
14605 4408 Acked Write 5 Done Sent Done false
14606 4409 Acked Write 6 Done Sent Done false
14607 4410 Acked Write 9 Done Sent Done false
14608 4411 Acked Write 9 Done Sent Done false
14609 4412 Acked Read 8 Done Sent Done false
14610 4413 Acked Write 8 Done Sent Done false
14611 4414 Acked Flush 0 Done Sent Done false
14612 4415 Acked Read 3 Done Sent Done false
14613 4416 Acked Read 9 Done Sent Done false
14614 4417 Acked Read 5 Done Sent Done false
14615 4418 Acked Read 6 Done Sent Done false
14616 4419 Acked Flush 0 Done Sent Done false
14617 4420 Acked Write 10 Done Sent Done false
14618 4421 Acked Write 1 Done Sent Done false
14619 4422 Acked Write 6 Done Sent Done false
14620 4423 Acked Write 5 Done Sent Done false
14621 4424 Acked Read 8 Done Sent Done false
14622 4425 Acked Write 8 Done Sent Done false
14623 4426 Acked Write 6 Done Sent Done false
14624 4427 Acked Read 3 Done Sent Done false
14625 4428 Acked Write 2 Done Sent Done false
14626 4429 Acked Flush 0 Done Sent Done false
14627 4430 Acked Write 7 Done Sent Done false
14628 4431 Acked Read 4 Done Sent Done false
14629 4432 Acked Write 10 Done Sent Done false
14630 4433 Acked Read 4 Done Sent Done false
14631 4434 Acked Read 1 Done Sent Done false
14632 4435 Acked Write 1 Done Sent Done false
14633 4436 Acked Flush 0 Done Sent Done false
14634 4437 Acked Write 4 Done Sent Done false
14635 4438 Acked Flush 0 Done Sent Done false
14636 4439 Acked Read 10 Done Sent Done false
14637 4440 Acked Write 8 Done Sent Done false
14638 4441 Acked Write 5 Done Sent Done false
14639 4442 Acked Flush 0 Done Sent Done false
14640 4443 Acked Write 7 Done Sent Done false
14641 4444 Acked Write 1 Done Sent Done false
14642 4445 Acked Write 7 Done Sent Done false
14643 4446 Acked Read 1 Done Sent Done false
14644 4447 Acked Read 8 Done Sent Done false
14645 4448 Acked Flush 0 Done Sent Done false
14646 4449 Acked Write 2 Done Sent Done false
14647 4450 Acked Write 10 Done Sent Done false
14648 4451 Acked Read 9 Done Sent Done false
14649 4452 Acked Flush 0 Done Sent Done false
14650 4453 Acked Write 3 Done Sent Done false
14651 4454 Acked Read 2 Done Sent Done false
14652 4455 Acked Read 7 Done Sent Done false
14653 4456 Acked Write 6 Done Sent Done false
14654 4457 Acked Flush 0 Done Sent Done false
14655 4458 Acked Flush 0 Done Sent Done false
14656 4459 Acked Read 1 Done Sent Done false
14657 4460 Acked Write 1 Done Sent Done false
14658 4461 Acked Flush 0 Done Sent Done false
14659 4462 Acked Write 2 Done Sent Done false
14660 4463 Acked Write 2 Done Sent Done false
14661 4464 Acked Read 4 Done Sent Done false
14662 4465 Acked Write 2 Done Sent Done false
14663 4466 Acked Read 10 Done Sent Done false
14664 4467 Acked Read 10 Done Sent Done false
14665 4468 Acked Write 9 Done Sent Done false
14666 4469 Acked Flush 0 Done Sent Done false
14667 4470 Acked Write 10 Done Sent Done false
14668 4471 Acked Flush 0 Done Sent Done false
14669 4472 Acked Write 5 Done Sent Done false
14670 4473 Acked Read 7 Done Sent Done false
14671 4474 Acked Read 10 Done Sent Done false
14672 4475 Acked Read 6 Done Sent Done false
14673 4476 Acked Write 6 Done Sent Done false
14674 4477 Acked Read 7 Done Sent Done false
14675 4478 Acked Write 6 Done Sent Done false
14676 4479 Acked Write 9 Done Sent Done false
14677 4480 Acked Read 1 Done Sent Done false
14678 4481 Acked Write 10 Done Sent Done false
14679 4482 Acked Read 4 Done Sent Done false
14680 4483 Acked Read 5 Done Sent Done false
14681 4484 Acked Flush 0 Done Sent Done false
14682 4485 Acked Flush 0 Done Sent Done false
14683 4486 Acked Write 10 Done Sent Done false
14684 4487 Acked Read 8 Done Sent Done false
14685 4488 Acked Read 8 Done Sent Done false
14686 4489 Acked Read 9 Done Sent Done false
14687 4490 Acked Read 3 Done Sent Done false
14688 4491 Acked Flush 0 Done Sent Done false
14689 4492 Acked Write 6 Done Sent Done false
14690 4493 Acked Write 6 Done Sent Done false
14691 4494 Acked Write 7 Done Sent Done false
14692 4495 Acked Read 1 Done Sent Done false
14693 4496 Acked Read 9 Done Sent Done false
14694 4497 Acked Write 7 Done Sent Done false
14695 4498 Acked Write 7 Done Sent Done false
14696 4499 Acked Read 6 Done Sent Done false
14697 4500 Acked Write 8 Done Sent Done false
14698 4501 Acked Write 1 Done Sent Done false
14699 4502 Acked Read 9 Done Sent Done false
14700 4503 Acked Read 4 Done Sent Done false
14701 4504 Acked Write 9 Done Sent Done false
14702 4505 Acked Read 2 Done Sent Done false
14703 4506 Acked Write 10 Done Sent Done false
14704 4507 Acked Flush 0 Done Sent Done false
14705 4508 Acked Read 8 Done Sent Done false
14706 4509 Acked Write 10 Done Sent Done false
14707 4510 Acked Flush 0 Done Sent Done false
14708 4511 Acked Write 8 Done Sent Done false
14709 4512 Acked Read 5 Done Sent Done false
14710 4513 Acked Read 4 Done Sent Done false
14711 4514 Acked Read 10 Done Sent Done false
14712 4515 Acked Flush 0 Done Sent Done false
14713 4516 Acked Read 3 Done Sent Done false
14714 4517 Acked Write 2 Done Sent Done false
14715 4518 Acked Read 7 Done Sent Done false
14716 4519 Acked Flush 0 Done Sent Done false
14717 4520 Acked Write 4 Done Sent Done false
14718 4521 Acked Write 5 Done Sent Done false
14719 4522 Acked Read 9 Done Sent Done false
14720 4523 Acked Read 9 Done Sent Done false
14721 4524 Acked Read 3 Done Sent Done false
14722 4525 Acked Read 6 Done Sent Done false
14723 4526 Acked Read 8 Done Sent Done false
14724 4527 Acked Write 2 Done Sent Done false
14725 4528 Acked Write 1 Done Sent Done false
14726 4529 Acked Read 7 Done Sent Done false
14727 4530 Acked Read 5 Done Sent Done false
14728 4531 Acked Read 3 Done Sent Done false
14729 4532 Acked Write 1 Done Sent Done false
14730 4533 Acked Write 9 Done Sent Done false
14731 4534 Acked Read 8 Done Sent Done false
14732 4535 Acked Write 9 Done Sent Done false
14733 4536 Acked Flush 0 Done Sent Done false
14734 4537 Acked Write 10 Done Sent Done false
14735 4538 Acked Read 1 Done Sent Done false
14736 4539 Acked Write 2 Done Sent Done false
14737 4540 Acked Read 4 Done Sent Done false
14738 4541 Acked Read 10 Done Sent Done false
14739 4542 Acked Read 3 Done Sent Done false
14740 4543 Acked Read 9 Done Sent Done false
14741 4544 Acked Write 3 Done Sent Done false
14742 4545 Acked Read 2 Done Sent Done false
14743 4546 Acked Read 1 Done Sent Done false
14744 4547 Acked Write 6 Done Sent Done false
14745 4548 Acked Read 9 Done Sent Done false
14746 4549 Acked Read 4 Done Sent Done false
14747 4550 Acked Write 8 Done Sent Done false
14748 4551 Acked Read 9 Done Sent Done false
14749 4552 Acked Write 5 Done Sent Done false
14750 4553 Acked Read 3 Done Sent Done false
14751 4554 Acked Read 5 Done Sent Done false
14752 4555 Acked Write 10 Done Sent Done false
14753 4556 Acked Write 6 Done Sent Done false
14754 4557 Acked Read 3 Done Sent Done false
14755 4558 Acked Read 3 Done Sent Done false
14756 4559 Acked Write 1 Done Sent Done false
14757 4560 Acked Flush 0 Done Sent Done false
14758 4561 Acked Write 6 Done Sent Done false
14759 4562 Acked Read 9 Done Sent Done false
14760 4563 Acked Read 8 Done Sent Done false
14761 4564 Acked Read 8 Done Sent Done false
14762 4565 Acked Flush 0 Done Sent Done false
14763 4566 Acked Read 8 Done Sent Done false
14764 4567 Acked Read 10 Done Sent Done false
14765 4568 Acked Read 9 Done Sent Done false
14766 4569 Acked Read 3 Done Sent Done false
14767 4570 Acked Flush 0 Done Sent Done false
14768 4571 Acked Read 4 Done Sent Done false
14769 4572 Acked Read 3 Done Sent Done false
14770 4573 Acked Read 2 Done Sent Done false
14771 4574 Acked Read 5 Done Sent Done false
14772 4575 Acked Read 1 Done Sent Done false
14773 4576 Acked Write 9 Done Sent Done false
14774 4577 Acked Read 3 Done Sent Done false
14775 4578 Acked Write 5 Done Sent Done false
14776 4579 Acked Flush 0 Done Sent Done false
14777 4580 Acked Write 8 Done Sent Done false
14778 4581 Acked Read 8 Done Sent Done false
14779 4582 Acked Write 6 Done Sent Done false
14780 4583 Acked Flush 0 Done Sent Done false
14781 4584 Acked Read 6 Done Sent Done false
14782 4585 Acked Read 5 Done Sent Done false
14783 4586 Acked Read 4 Done Sent Done false
14784 4587 Acked Read 6 Done Sent Done false
14785 4588 Acked Write 4 Done Sent Done false
14786 4589 Acked Read 2 Done Sent Done false
14787 4590 Acked Read 8 Done Sent Done false
14788 4591 Acked Write 2 Done Sent Done false
14789 4592 Acked Flush 0 Done Sent Done false
14790 4593 Acked Read 1 Done Sent Done false
14791 4594 Acked Write 7 Done Sent Done false
14792 4595 Acked Flush 0 Done Sent Done false
14793 4596 Acked Write 8 Done Sent Done false
14794 4597 Acked Read 7 Done Sent Done false
14795 4598 Acked Read 8 Done Sent Done false
14796 4599 Acked Write 1 Done Sent Done false
14797 4600 Acked Read 1 Done Sent Done false
14798 4601 Acked Read 10 Done Sent Done false
14799 4602 Acked Flush 0 Done Sent Done false
14800 4603 Acked Read 2 Done Sent Done false
14801 4604 Acked Read 8 Done Sent Done false
14802 4605 Acked Write 4 Done Sent Done false
14803 4606 Acked Flush 0 Done Sent Done false
14804 4607 Acked Read 9 Done Sent Done false
14805 4608 Acked Read 1 Done Sent Done false
14806 4609 Acked Flush 0 Done Sent Done false
14807 4610 Acked Read 4 Done Sent Done false
14808 4611 Acked Write 2 Done Sent Done false
14809 4612 Acked Write 2 Done Sent Done false
14810 4613 Acked Read 10 Done Sent Done false
14811 4614 Acked Read 5 Done Sent Done false
14812 4615 Acked Write 5 Done Sent Done false
14813 4616 Acked Write 2 Done Sent Done false
14814 4617 Acked Write 6 Done Sent Done false
14815 4618 Acked Write 6 Done Sent Done false
14816 4619 Acked Write 8 Done Sent Done false
14817 4620 Acked Write 9 Done Sent Done false
14818 STATES DS:0 DS:1 DS:2 TOTAL
14819 Sent 0 301 0 301
14820 Done 301 0 301 602
14821 Skipped 0 0 0 0
14822 Error 0 0 0 0
14823 Last Flush: Some(JobId(4609)) Some(JobId(4319)) Some(JobId(4609))
14824 Downstairs last five completed: 4319 4318 4317 4316 4315
14825 Upstairs last five completed: 4620 4619 4618 4617 4616
14826 CLIENT: Up:0 ds:301 act:2
148272025-03-07T18:15:44.564ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
148282025-03-07T18:15:45.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
148292025-03-07T18:15:46.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
148302025-03-07T18:15:47.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
14831 ----------------------------------------------------------------
14832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14833 States: Active Offline Active
14834 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14835 4320 Acked Flush 0 Done Sent Done false
14836 4321 Acked Read 3 Done Sent Done false
14837 4322 Acked Write 6 Done Sent Done false
14838 4323 Acked Read 7 Done Sent Done false
14839 4324 Acked Write 5 Done Sent Done false
14840 4325 Acked Write 6 Done Sent Done false
14841 4326 Acked Read 2 Done Sent Done false
14842 4327 Acked Read 10 Done Sent Done false
14843 4328 Acked Read 6 Done Sent Done false
14844 4329 Acked Write 10 Done Sent Done false
14845 4330 Acked Read 3 Done Sent Done false
14846 4331 Acked Write 7 Done Sent Done false
14847 4332 Acked Flush 0 Done Sent Done false
14848 4333 Acked Write 8 Done Sent Done false
14849 4334 Acked Write 5 Done Sent Done false
14850 4335 Acked Write 9 Done Sent Done false
14851 4336 Acked Write 9 Done Sent Done false
14852 4337 Acked Read 7 Done Sent Done false
14853 4338 Acked Flush 0 Done Sent Done false
14854 4339 Acked Write 4 Done Sent Done false
14855 4340 Acked Write 10 Done Sent Done false
14856 4341 Acked Write 1 Done Sent Done false
14857 4342 Acked Write 7 Done Sent Done false
14858 4343 Acked Flush 0 Done Sent Done false
14859 4344 Acked Flush 0 Done Sent Done false
14860 4345 Acked Read 8 Done Sent Done false
14861 4346 Acked Read 9 Done Sent Done false
14862 4347 Acked Write 10 Done Sent Done false
14863 4348 Acked Write 4 Done Sent Done false
14864 4349 Acked Read 7 Done Sent Done false
14865 4350 Acked Write 9 Done Sent Done false
14866 4351 Acked Write 1 Done Sent Done false
14867 4352 Acked Flush 0 Done Sent Done false
14868 4353 Acked Write 8 Done Sent Done false
14869 4354 Acked Read 6 Done Sent Done false
14870 4355 Acked Read 7 Done Sent Done false
14871 4356 Acked Flush 0 Done Sent Done false
14872 4357 Acked Read 8 Done Sent Done false
14873 4358 Acked Read 4 Done Sent Done false
14874 4359 Acked Read 4 Done Sent Done false
14875 4360 Acked Write 5 Done Sent Done false
14876 4361 Acked Read 4 Done Sent Done false
14877 4362 Acked Flush 0 Done Sent Done false
14878 4363 Acked Read 10 Done Sent Done false
14879 4364 Acked Read 6 Done Sent Done false
14880 4365 Acked Read 8 Done Sent Done false
14881 4366 Acked Read 6 Done Sent Done false
14882 4367 Acked Read 5 Done Sent Done false
14883 4368 Acked Read 2 Done Sent Done false
14884 4369 Acked Read 8 Done Sent Done false
14885 4370 Acked Write 9 Done Sent Done false
14886 4371 Acked Write 4 Done Sent Done false
14887 4372 Acked Read 8 Done Sent Done false
14888 4373 Acked Read 4 Done Sent Done false
14889 4374 Acked Write 1 Done Sent Done false
14890 4375 Acked Flush 0 Done Sent Done false
14891 4376 Acked Flush 0 Done Sent Done false
14892 4377 Acked Write 1 Done Sent Done false
14893 4378 Acked Write 5 Done Sent Done false
14894 4379 Acked Read 2 Done Sent Done false
14895 4380 Acked Read 8 Done Sent Done false
14896 4381 Acked Read 5 Done Sent Done false
14897 4382 Acked Write 2 Done Sent Done false
14898 4383 Acked Write 8 Done Sent Done false
14899 4384 Acked Read 8 Done Sent Done false
14900 4385 Acked Read 4 Done Sent Done false
14901 4386 Acked Read 6 Done Sent Done false
14902 4387 Acked Read 1 Done Sent Done false
14903 4388 Acked Read 8 Done Sent Done false
14904 4389 Acked Read 7 Done Sent Done false
14905 4390 Acked Write 7 Done Sent Done false
14906 4391 Acked Write 10 Done Sent Done false
14907 4392 Acked Flush 0 Done Sent Done false
14908 4393 Acked Read 9 Done Sent Done false
14909 4394 Acked Write 7 Done Sent Done false
14910 4395 Acked Write 2 Done Sent Done false
14911 4396 Acked Read 10 Done Sent Done false
14912 4397 Acked Read 10 Done Sent Done false
14913 4398 Acked Flush 0 Done Sent Done false
14914 4399 Acked Write 9 Done Sent Done false
14915 4400 Acked Read 10 Done Sent Done false
14916 4401 Acked Write 6 Done Sent Done false
14917 4402 Acked Write 4 Done Sent Done false
14918 4403 Acked Read 8 Done Sent Done false
14919 4404 Acked Flush 0 Done Sent Done false
14920 4405 Acked Read 8 Done Sent Done false
14921 4406 Acked Write 5 Done Sent Done false
14922 4407 Acked Read 1 Done Sent Done false
14923 4408 Acked Write 5 Done Sent Done false
14924 4409 Acked Write 6 Done Sent Done false
14925 4410 Acked Write 9 Done Sent Done false
14926 4411 Acked Write 9 Done Sent Done false
14927 4412 Acked Read 8 Done Sent Done false
14928 4413 Acked Write 8 Done Sent Done false
14929 4414 Acked Flush 0 Done Sent Done false
14930 4415 Acked Read 3 Done Sent Done false
14931 4416 Acked Read 9 Done Sent Done false
14932 4417 Acked Read 5 Done Sent Done false
14933 4418 Acked Read 6 Done Sent Done false
14934 4419 Acked Flush 0 Done Sent Done false
14935 4420 Acked Write 10 Done Sent Done false
14936 4421 Acked Write 1 Done Sent Done false
14937 4422 Acked Write 6 Done Sent Done false
14938 4423 Acked Write 5 Done Sent Done false
14939 4424 Acked Read 8 Done Sent Done false
14940 4425 Acked Write 8 Done Sent Done false
14941 4426 Acked Write 6 Done Sent Done false
14942 4427 Acked Read 3 Done Sent Done false
14943 4428 Acked Write 2 Done Sent Done false
14944 4429 Acked Flush 0 Done Sent Done false
14945 4430 Acked Write 7 Done Sent Done false
14946 4431 Acked Read 4 Done Sent Done false
14947 4432 Acked Write 10 Done Sent Done false
14948 4433 Acked Read 4 Done Sent Done false
14949 4434 Acked Read 1 Done Sent Done false
14950 4435 Acked Write 1 Done Sent Done false
14951 4436 Acked Flush 0 Done Sent Done false
14952 4437 Acked Write 4 Done Sent Done false
14953 4438 Acked Flush 0 Done Sent Done false
14954 4439 Acked Read 10 Done Sent Done false
14955 4440 Acked Write 8 Done Sent Done false
14956 4441 Acked Write 5 Done Sent Done false
14957 4442 Acked Flush 0 Done Sent Done false
14958 4443 Acked Write 7 Done Sent Done false
14959 4444 Acked Write 1 Done Sent Done false
14960 4445 Acked Write 7 Done Sent Done false
14961 4446 Acked Read 1 Done Sent Done false
14962 4447 Acked Read 8 Done Sent Done false
14963 4448 Acked Flush 0 Done Sent Done false
14964 4449 Acked Write 2 Done Sent Done false
14965 4450 Acked Write 10 Done Sent Done false
14966 4451 Acked Read 9 Done Sent Done false
14967 4452 Acked Flush 0 Done Sent Done false
14968 4453 Acked Write 3 Done Sent Done false
14969 4454 Acked Read 2 Done Sent Done false
14970 4455 Acked Read 7 Done Sent Done false
14971 4456 Acked Write 6 Done Sent Done false
14972 4457 Acked Flush 0 Done Sent Done false
14973 4458 Acked Flush 0 Done Sent Done false
14974 4459 Acked Read 1 Done Sent Done false
14975 4460 Acked Write 1 Done Sent Done false
14976 4461 Acked Flush 0 Done Sent Done false
14977 4462 Acked Write 2 Done Sent Done false
14978 4463 Acked Write 2 Done Sent Done false
14979 4464 Acked Read 4 Done Sent Done false
14980 4465 Acked Write 2 Done Sent Done false
14981 4466 Acked Read 10 Done Sent Done false
14982 4467 Acked Read 10 Done Sent Done false
14983 4468 Acked Write 9 Done Sent Done false
14984 4469 Acked Flush 0 Done Sent Done false
14985 4470 Acked Write 10 Done Sent Done false
14986 4471 Acked Flush 0 Done Sent Done false
14987 4472 Acked Write 5 Done Sent Done false
14988 4473 Acked Read 7 Done Sent Done false
14989 4474 Acked Read 10 Done Sent Done false
14990 4475 Acked Read 6 Done Sent Done false
14991 4476 Acked Write 6 Done Sent Done false
14992 4477 Acked Read 7 Done Sent Done false
14993 4478 Acked Write 6 Done Sent Done false
14994 4479 Acked Write 9 Done Sent Done false
14995 4480 Acked Read 1 Done Sent Done false
14996 4481 Acked Write 10 Done Sent Done false
14997 4482 Acked Read 4 Done Sent Done false
14998 4483 Acked Read 5 Done Sent Done false
14999 4484 Acked Flush 0 Done Sent Done false
15000 4485 Acked Flush 0 Done Sent Done false
15001 4486 Acked Write 10 Done Sent Done false
15002 4487 Acked Read 8 Done Sent Done false
15003 4488 Acked Read 8 Done Sent Done false
15004 4489 Acked Read 9 Done Sent Done false
15005 4490 Acked Read 3 Done Sent Done false
15006 4491 Acked Flush 0 Done Sent Done false
15007 4492 Acked Write 6 Done Sent Done false
15008 4493 Acked Write 6 Done Sent Done false
15009 4494 Acked Write 7 Done Sent Done false
15010 4495 Acked Read 1 Done Sent Done false
15011 4496 Acked Read 9 Done Sent Done false
15012 4497 Acked Write 7 Done Sent Done false
15013 4498 Acked Write 7 Done Sent Done false
15014 4499 Acked Read 6 Done Sent Done false
15015 4500 Acked Write 8 Done Sent Done false
15016 4501 Acked Write 1 Done Sent Done false
15017 4502 Acked Read 9 Done Sent Done false
15018 4503 Acked Read 4 Done Sent Done false
15019 4504 Acked Write 9 Done Sent Done false
15020 4505 Acked Read 2 Done Sent Done false
15021 4506 Acked Write 10 Done Sent Done false
15022 4507 Acked Flush 0 Done Sent Done false
15023 4508 Acked Read 8 Done Sent Done false
15024 4509 Acked Write 10 Done Sent Done false
15025 4510 Acked Flush 0 Done Sent Done false
15026 4511 Acked Write 8 Done Sent Done false
15027 4512 Acked Read 5 Done Sent Done false
15028 4513 Acked Read 4 Done Sent Done false
15029 4514 Acked Read 10 Done Sent Done false
15030 4515 Acked Flush 0 Done Sent Done false
15031 4516 Acked Read 3 Done Sent Done false
15032 4517 Acked Write 2 Done Sent Done false
15033 4518 Acked Read 7 Done Sent Done false
15034 4519 Acked Flush 0 Done Sent Done false
15035 4520 Acked Write 4 Done Sent Done false
15036 4521 Acked Write 5 Done Sent Done false
15037 4522 Acked Read 9 Done Sent Done false
15038 4523 Acked Read 9 Done Sent Done false
15039 4524 Acked Read 3 Done Sent Done false
15040 4525 Acked Read 6 Done Sent Done false
15041 4526 Acked Read 8 Done Sent Done false
15042 4527 Acked Write 2 Done Sent Done false
15043 4528 Acked Write 1 Done Sent Done false
15044 4529 Acked Read 7 Done Sent Done false
15045 4530 Acked Read 5 Done Sent Done false
15046 4531 Acked Read 3 Done Sent Done false
15047 4532 Acked Write 1 Done Sent Done false
15048 4533 Acked Write 9 Done Sent Done false
15049 4534 Acked Read 8 Done Sent Done false
15050 4535 Acked Write 9 Done Sent Done false
15051 4536 Acked Flush 0 Done Sent Done false
15052 4537 Acked Write 10 Done Sent Done false
15053 4538 Acked Read 1 Done Sent Done false
15054 4539 Acked Write 2 Done Sent Done false
15055 4540 Acked Read 4 Done Sent Done false
15056 4541 Acked Read 10 Done Sent Done false
15057 4542 Acked Read 3 Done Sent Done false
15058 4543 Acked Read 9 Done Sent Done false
15059 4544 Acked Write 3 Done Sent Done false
15060 4545 Acked Read 2 Done Sent Done false
15061 4546 Acked Read 1 Done Sent Done false
15062 4547 Acked Write 6 Done Sent Done false
15063 4548 Acked Read 9 Done Sent Done false
15064 4549 Acked Read 4 Done Sent Done false
15065 4550 Acked Write 8 Done Sent Done false
15066 4551 Acked Read 9 Done Sent Done false
15067 4552 Acked Write 5 Done Sent Done false
15068 4553 Acked Read 3 Done Sent Done false
15069 4554 Acked Read 5 Done Sent Done false
15070 4555 Acked Write 10 Done Sent Done false
15071 4556 Acked Write 6 Done Sent Done false
15072 4557 Acked Read 3 Done Sent Done false
15073 4558 Acked Read 3 Done Sent Done false
15074 4559 Acked Write 1 Done Sent Done false
15075 4560 Acked Flush 0 Done Sent Done false
15076 4561 Acked Write 6 Done Sent Done false
15077 4562 Acked Read 9 Done Sent Done false
15078 4563 Acked Read 8 Done Sent Done false
15079 4564 Acked Read 8 Done Sent Done false
15080 4565 Acked Flush 0 Done Sent Done false
15081 4566 Acked Read 8 Done Sent Done false
15082 4567 Acked Read 10 Done Sent Done false
15083 4568 Acked Read 9 Done Sent Done false
15084 4569 Acked Read 3 Done Sent Done false
15085 4570 Acked Flush 0 Done Sent Done false
15086 4571 Acked Read 4 Done Sent Done false
15087 4572 Acked Read 3 Done Sent Done false
15088 4573 Acked Read 2 Done Sent Done false
15089 4574 Acked Read 5 Done Sent Done false
15090 4575 Acked Read 1 Done Sent Done false
15091 4576 Acked Write 9 Done Sent Done false
15092 4577 Acked Read 3 Done Sent Done false
15093 4578 Acked Write 5 Done Sent Done false
15094 4579 Acked Flush 0 Done Sent Done false
15095 4580 Acked Write 8 Done Sent Done false
15096 4581 Acked Read 8 Done Sent Done false
15097 4582 Acked Write 6 Done Sent Done false
15098 4583 Acked Flush 0 Done Sent Done false
15099 4584 Acked Read 6 Done Sent Done false
15100 4585 Acked Read 5 Done Sent Done false
15101 4586 Acked Read 4 Done Sent Done false
15102 4587 Acked Read 6 Done Sent Done false
15103 4588 Acked Write 4 Done Sent Done false
15104 4589 Acked Read 2 Done Sent Done false
15105 4590 Acked Read 8 Done Sent Done false
15106 4591 Acked Write 2 Done Sent Done false
15107 4592 Acked Flush 0 Done Sent Done false
15108 4593 Acked Read 1 Done Sent Done false
15109 4594 Acked Write 7 Done Sent Done false
15110 4595 Acked Flush 0 Done Sent Done false
15111 4596 Acked Write 8 Done Sent Done false
15112 4597 Acked Read 7 Done Sent Done false
15113 4598 Acked Read 8 Done Sent Done false
15114 4599 Acked Write 1 Done Sent Done false
15115 4600 Acked Read 1 Done Sent Done false
15116 4601 Acked Read 10 Done Sent Done false
15117 4602 Acked Flush 0 Done Sent Done false
15118 4603 Acked Read 2 Done Sent Done false
15119 4604 Acked Read 8 Done Sent Done false
15120 4605 Acked Write 4 Done Sent Done false
15121 4606 Acked Flush 0 Done Sent Done false
15122 4607 Acked Read 9 Done Sent Done false
15123 4608 Acked Read 1 Done Sent Done false
15124 4609 Acked Flush 0 Done Sent Done false
15125 4610 Acked Read 4 Done Sent Done false
15126 4611 Acked Write 2 Done Sent Done false
15127 4612 Acked Write 2 Done Sent Done false
15128 4613 Acked Read 10 Done Sent Done false
15129 4614 Acked Read 5 Done Sent Done false
15130 4615 Acked Write 5 Done Sent Done false
15131 4616 Acked Write 2 Done Sent Done false
15132 4617 Acked Write 6 Done Sent Done false
15133 4618 Acked Write 6 Done Sent Done false
15134 4619 Acked Write 8 Done Sent Done false
15135 4620 Acked Write 9 Done Sent Done false
15136 STATES DS:0 DS:1 DS:2 TOTAL
15137 Sent 0 301 0 301
15138 Done 301 0 301 602
15139 Skipped 0 0 0 0
15140 Error 0 0 0 0
15141 Last Flush: Some(JobId(4609)) Some(JobId(4319)) Some(JobId(4609))
15142 Downstairs last five completed: 4319 4318 4317 4316 4315
15143 Upstairs last five completed: 4620 4619 4618 4617 4616
15144 CLIENT: Up:0 ds:301 act:2
151452025-03-07T18:15:48.265ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151462025-03-07T18:15:49.564ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151472025-03-07T18:15:50.269ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151482025-03-07T18:15:51.271ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151492025-03-07T18:15:51.319ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151502025-03-07T18:15:51.319ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36025) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151512025-03-07T18:15:51.320ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151522025-03-07T18:15:51.320ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151532025-03-07T18:15:51.320ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151542025-03-07T18:15:51.320ZINFOcrucible: send last flush ID to this DS: Some(JobId(4319)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151552025-03-07T18:15:51.320ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151562025-03-07T18:15:51.320ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4319)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151572025-03-07T18:15:51.320ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4319)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
15158 ----------------------------------------------------------------
15159 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:11
15160 States: Active Active Active
15161 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15162 4610 Acked Read 4 Done Done Done true
15163 4611 Acked Write 2 Done Done Done true
15164 4612 Acked Write 2 Done Done Done true
15165 4613 Acked Read 10 Done Done Done true
15166 4614 Acked Read 5 Done Done Done true
15167 4615 Acked Write 5 Done Done Done true
15168 4616 Acked Write 2 Done Done Done true
15169 4617 Acked Write 6 Done Done Done true
15170 4618 Acked Write 6 Done Done Done true
15171 4619 Acked Write 8 Done Done Done true
15172 4620 Acked Write 9 Done Done Done true
15173 STATES DS:0 DS:1 DS:2 TOTAL
15174 Sent 0 0 0 0
15175 Done 11 11 11 33
15176 Skipped 0 0 0 0
15177 Error 0 0 0 0
15178 Last Flush: Some(JobId(4609)) Some(JobId(4609)) Some(JobId(4609))
15179 Downstairs last five completed: 4609 4608 4607 4606 4605
15180 Upstairs last five completed: 4620 4619 4618 4617 4616
15181 CLIENT: Up:0 ds:11 act:3
15182 ----------------------------------------------------------------
15183 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15184 Downstairs last five completed: 4621 4620 4619 4618 4617
15185 Upstairs last five completed: 4621 4620 4619 4618 4617
15186 CLIENT: Up:0 ds:0 act:3
15187 Replay: All jobs finished
15188 Main task has work to do, go find it
15189 got dsc Stop(0)
15190 stop 0
15191 Mar 07 18:15:55.580[8810] Got stop action so:false kr:false
15192 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b8a4254c-ccd2-4f16-898a-1d9fdaa03bfa, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
151932025-03-07T18:15:55.581ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
15194 Mar 07 18:15:55.581 INFO request completed, latency_us: 124, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4f35bcb7-fe2c-49fb-a082-33db3759c058, remote_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:15:55.581122653Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
15195 0.1:64322, local_addr: 127.0.0.1:9998
151962025-03-07T18:15:55.581ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
151972025-03-07T18:15:55.581ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
15198 [8810] Exited with: ExitStatus(unix_wait_status(9))
15199 [8810][0] reports Exit
15200 Mar 07 18:15:59.583 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: bf8ac478-3b1b-4991-8640-d2808b9d28b9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
15201 001/300 Write block 523 len 16384 data: 3 3 3 4
15202 002/300 Read block 4545 len 32768
15203 003/300 Write block 3377 len 12288 data: 4 4 4
15204 004/300 Flush
15205 005/300 Flush
15206 006/300 Read block 2611 len 8192
15207 007/300 Read block 1950 len 40960
15208 008/300 Write block 265 len 12288 data: 4 4 4
15209 009/300 Write block 645 len 28672 data: 3 3 3 3 3 3 2
15210 010/300 Read block 3283 len 20480
15211 011/300 Read block 4783 len 36864
15212 012/300 Write block 182 len 4096 data: 2
15213 013/300 Read block 2736 len 32768
15214 014/300 Write block 286 len 36864 data: 3 3 3 3 4 3 3 3 3
15215 015/300 Read block 1093 len 12288
15216 016/300 Write block 3576 len 32768 data: 2 2 2 2 2 2 3 3
15217 017/300 Write block 1459 len 20480 data: 3 3 4 5 6
15218 018/300 Write block 1871 len 28672 data: 3 3 3 2 2 2 2
15219 019/300 Read block 2360 len 32768
15220 020/300 Write block 4092 len 4096 data: 2
15221 021/300 Read block 1298 len 20480
15222 022/300 Read block 1288 len 32768
15223 023/300 Write block 1613 len 24576 data: 3 5 5 6 6 5
15224 024/300 Read block 1767 len 36864
15225 025/300 Write block 785 len 12288 data: 4 6 6
15226 026/300 Read block 2683 len 20480
15227 027/300 Write block 4851 len 32768 data: 3 3 3 3 3 3 3 2
15228 028/300 Write block 243 len 28672 data: 3 4 4 3 3 3 3
15229 029/300 Write block 1159 len 32768 data: 3 4 5 4 4 5 4 4
15230 030/300 Read block 1142 len 36864
15231 031/300 Read block 3682 len 16384
15232 032/300 Write block 2026 len 32768 data: 4 3 3 4 3 2 3 3
15233 033/300 Read block 1191 len 32768
15234 034/300 Write block 881 len 16384 data: 4 3 2 2
15235 035/300 Flush
15236 036/300 Read block 1801 len 20480
15237 037/300 Write block 4918 len 4096 data: 6
15238 038/300 Read block 4467 len 8192
15239 039/300 Read block 3904 len 32768
15240 040/300 Write block 4396 len 28672 data: 3 3 2 2 2 2 3
15241 041/300 Read block 1909 len 28672
15242 042/300 Flush
15243 043/300 Write block 4830 len 24576 data: 2 3 4 4 4 4
15244 044/300 Read block 3681 len 20480
15245 045/300 Read block 926 len 20480
15246 046/300 Read block 147 len 40960
15247 047/300 Write block 2864 len 4096 data: 3
15248 048/300 Read block 2499 len 4096
15249 049/300 Write block 3526 len 16384 data: 4 4 4 4
15250 050/300 Write block 960 len 40960 data: 5 4 4 3 3 2 2 2 4 4
15251 051/300 Read block 4518 len 20480
15252 052/300 Write block 991 len 28672 data: 2 2 2 2 2 2 2
15253 053/300 Read block 3281 len 4096
15254 054/300 Write block 166 len 40960 data: 5 4 3 2 3 3 3 3 3 3
15255 055/300 Write block 4427 len 32768 data: 5 3 3 3 4 5 6 6
15256 056/300 Write block 3995 len 36864 data: 6 4 4 4 4 3 3 3 3
15257 057/300 Write block 4261 len 20480 data: 5 5 4 4 3
15258 058/300 Write block 1103 len 36864 data: 4 5 4 3 3 2 2 2 3
15259 059/300 Read block 2271 len 36864
15260 060/300 Read block 3165 len 16384
15261 061/300 Write block 1834 len 4096 data: 2
15262 062/300 Write block 2021 len 36864 data: 3 3 3 4 4 5 4 4 5
15263 063/300 Read block 4930 len 20480
15264 064/300 Read block 4914 len 8192
15265 065/300 Write block 274 len 12288 data: 4 4 6
15266 066/300 Write block 3139 len 8192 data: 2 3
15267 067/300 Flush
15268 068/300 Read block 564 len 24576
15269 069/300 Write block 2071 len 28672 data: 2 2 2 2 2 2 2
15270 070/300 Flush
15271 071/300 Read block 399 len 4096
15272 072/300 Read block 3197 len 36864
15273 073/300 Write block 80 len 24576 data: 3 3 3 3 3 2
15274 074/300 Read block 2291 len 8192
15275 075/300 Read block 1086 len 36864
15276 076/300 Write block 2699 len 20480 data: 3 4 4 4 3
15277 077/300 Write block 2198 len 8192 data: 4 4
15278 078/300 Flush
15279 079/300 Read block 4655 len 36864
15280 080/300 Read block 3574 len 32768
15281 081/300 Read block 2467 len 36864
15282 082/300 Write block 4429 len 40960 data: 4 4 5 6 7 7 5 5 5 5
15283 083/300 Read block 1162 len 24576
15284 084/300 Read block 1643 len 4096
15285 085/300 Flush
15286 086/300 Read block 2081 len 28672
15287 087/300 Read block 1894 len 40960
15288 088/300 Read block 3284 len 4096
15289 089/300 Write block 4797 len 12288 data: 4 4 4
15290 090/300 Read block 2461 len 8192
15291 091/300 Read block 774 len 4096
15292 092/300 Read block 2228 len 32768
15293 093/300 Write block 325 len 12288 data: 5 5 5
15294 094/300 Read block 4234 len 16384
15295 095/300 Read block 714 len 32768
15296 096/300 Write block 288 len 24576 data: 4 4 5 4 4 4
15297 097/300 Read block 3900 len 16384
15298 098/300 Read block 264 len 12288
15299 099/300 Write block 2590 len 8192 data: 4 3
15300 100/300 Write block 616 len 36864 data: 4 3 4 5 4 4 5 6 6
15301 101/300 Flush
15302 102/300 Write block 491 len 4096 data: 4
15303 103/300 Read block 226 len 40960
15304 104/300 Read block 4698 len 28672
15305 105/300 Read block 833 len 36864
15306 106/300 Write block 1765 len 32768 data: 2 3 3 2 3 3 3 4
15307 107/300 Write block 4277 len 12288 data: 6 6 7
15308 108/300 Read block 4799 len 20480
15309 109/300 Read block 3114 len 4096
15310 110/300 Write block 3097 len 24576 data: 4 4 4 5 4 4
15311 111/300 Write block 2444 len 16384 data: 2 2 2 2
15312 112/300 Write block 3753 len 8192 data: 3 4
15313 113/300 Write block 4597 len 32768 data: 4 4 4 4 5 5 5 4
15314 114/300 Write block 83 len 32768 data: 4 4 3 2 2 2 2 2
15315 115/300 Flush
15316 116/300 Read block 2749 len 32768
15317 117/300 Read block 254 len 40960
15318 118/300 Flush
15319 119/300 Flush
15320 120/300 Read block 604 len 16384
15321 121/300 Read block 2809 len 24576
15322 122/300 Flush
15323 123/300 Write block 3475 len 12288 data: 6 6 5
15324 124/300 Read block 441 len 16384
15325 125/300 Read block 2838 len 36864
15326 126/300 Write block 4042 len 20480 data: 4 4 5 5 6
15327 127/300 Read block 4031 len 16384
15328 128/300 Flush
15329 129/300 Read block 3878 len 16384
15330 130/300 Write block 464 len 8192 data: 2 2
15331 131/300 Read block 2361 len 12288
15332 132/300 Read block 521 len 4096
15333 133/300 Read block 4535 len 12288
15334 134/300 Read block 1572 len 20480
15335 135/300 Read block 2011 len 36864
15336 136/300 Write block 381 len 28672 data: 2 3 3 3 3 3 4
15337 137/300 Read block 2000 len 4096
15338 138/300 Write block 580 len 4096 data: 3
15339 139/300 Write block 1023 len 32768 data: 4 4 4 3 4 4 4 3
15340 140/300 Read block 2401 len 36864
15341 141/300 Read block 3777 len 8192
15342 142/300 Read block 4968 len 36864
15343 143/300 Flush
15344 144/300 Flush
15345 145/300 Write block 2975 len 4096 data: 4
15346 146/300 Write block 1115 len 40960 data: 4 3 3 3 4 4 5 5 5 4
15347 147/300 Read block 3502 len 4096
15348 148/300 Read block 3485 len 4096
15349 149/300 Flush
15350 150/300 Flush
15351 151/300 Read block 2754 len 24576
15352 152/300 Read block 3284 len 32768
15353 153/300 Write block 3352 len 28672 data: 3 2 2 2 2 2 2
15354 154/300 Read block 4326 len 4096
15355 155/300 Read block 2851 len 4096
15356 156/300 Read block 4808 len 12288
15357 157/300 Write block 1623 len 8192 data: 5 6
15358 158/300 Read block 4115 len 36864
15359 159/300 Read block 3332 len 4096
15360 160/300 Read block 3402 len 4096
15361 161/300 Read block 4765 len 36864
15362 162/300 Write block 740 len 28672 data: 6 5 6 6 6 6 5
15363 163/300 Write block 1152 len 36864 data: 3 4 4 5 4 4 4 4 5
15364 164/300 Write block 1807 len 12288 data: 3 3 3
15365 165/300 Read block 2382 len 24576
15366 166/300 Read block 469 len 20480
15367 167/300 Read block 2929 len 20480
15368 168/300 Read block 2799 len 32768
15369 169/300 Read block 4682 len 8192
15370 170/300 Read block 4115 len 4096
15371 171/300 Flush
15372 172/300 Read block 1878 len 28672
15373 173/300 Read block 3509 len 12288
15374 174/300 Read block 3716 len 8192
15375 175/300 Read block 2096 len 36864
15376 176/300 Read block 1431 len 28672
15377 177/300 Read block 2176 len 32768
15378 178/300 Write block 532 len 16384 data: 4 4 2 2
15379 179/300 Flush
15380 180/300 Read block 2335 len 8192
15381 181/300 Read block 3437 len 20480
15382 182/300 Read block 1357 len 40960
15383 183/300 Read block 4180 len 32768
15384 184/300 Write block 2405 len 4096 data: 3
15385 185/300 Read block 1408 len 12288
15386 186/300 Read block 2283 len 28672
15387 187/300 Read block 3089 len 32768
15388 188/300 Write block 746 len 24576 data: 6 6 6 4 3 3
15389 189/300 Write block 2209 len 28672 data: 4 3 2 2 2 3 2
15390 190/300 Write block 4411 len 36864 data: 4 4 4 5 5 4 3 3 3
15391 191/300 Write block 2887 len 20480 data: 5 4 5 5 4
15392 192/300 Write block 4251 len 16384 data: 5 5 5 5
15393 193/300 Read block 4432 len 16384
15394 194/300 Read block 2723 len 28672
15395 195/300 Read block 3349 len 16384
15396 196/300 Read block 937 len 16384
15397 197/300 Read block 2145 len 24576
15398 198/300 Flush
15399 199/300 Read block 2409 len 20480
15400 200/300 Write block 1742 len 36864 data: 3 4 4 6 4 4 4 4 4
15401 201/300 Read block 714 len 8192
15402 202/300 Read block 258 len 12288
15403 203/300 Write block 3371 len 24576 data: 5 4 4 3 3 3
15404 204/300 Read block 3512 len 36864
15405 205/300 Write block 2102 len 20480 data: 2 2 4 4 4
15406 206/300 Write block 3480 len 4096 data: 2
15407 207/300 Read block 3618 len 4096
15408 208/300 Read block 2303 len 28672
15409 209/300 Write block 1847 len 40960 data: 5 5 5 4 4 4 3 3 3 3
15410 210/300 Write block 4979 len 32768 data: 2 2 2 2 2 2 2 2
15411 211/300 Write block 345 len 40960 data: 3 3 5 7 7 7 6 6 5 4
15412 212/300 Write block 1850 len 12288 data: 5 5 5
15413 213/300 Flush
15414 214/300 Write block 4172 len 40960 data: 5 3 3 2 2 2 2 2 3 2
15415 215/300 Flush
15416 216/300 Write block 1397 len 36864 data: 3 2 2 2 2 2 3 3 5
15417 217/300 Read block 1321 len 12288
15418 218/300 Write block 3281 len 36864 data: 6 3 3 2 3 3 3 3 3
15419 219/300 Read block 4409 len 20480
15420 220/300 Read block 934 len 40960
15421 221/300 Read block 393 len 12288
15422 222/300 Read block 4978 len 12288
15423 223/300 Read block 3215 len 12288
15424 224/300 Write block 144 len 32768 data: 3 3 3 3 4 4 3 2
15425 225/300 Read block 3066 len 4096
15426 226/300 Read block 1556 len 40960
15427 227/300 Write block 1739 len 16384 data: 2 2 2 4
15428 228/300 Read block 2515 len 20480
15429 229/300 Read block 1584 len 4096
15430 230/300 Write block 1586 len 8192 data: 3 4
15431 231/300 Write block 1018 len 4096 data: 4
15432 232/300 Read block 3377 len 24576
15433 233/300 Read block 1393 len 16384
15434 234/300 Write block 1247 len 28672 data: 3 3 3 3 5 3 4
15435 235/300 Write block 128 len 40960 data: 5 4 4 4 4 3 4 4 3 3
15436 236/300 Write block 400 len 20480 data: 3 3 3 3 3
15437 237/300 Write block 3301 len 28672 data: 2 2 3 3 4 4 4
15438 238/300 Read block 3029 len 24576
15439 239/300 Read block 3186 len 24576
15440 240/300 Write block 449 len 16384 data: 2 2 2 2
15441 241/300 Write block 4584 len 36864 data: 3 3 4 4 4 5 5 3 4
15442 242/300 Read block 4045 len 24576
15443 243/300 Write block 2802 len 12288 data: 5 4 4
15444 244/300 Flush
15445 245/300 Read block 3753 len 36864
15446 246/300 Read block 1363 len 40960
15447 247/300 Write block 1461 len 28672 data: 5 6 7 6 5 5 5
15448 248/300 Write block 4075 len 16384 data: 6 4 3 3
15449 249/300 Write block 3175 len 24576 data: 2 2 2 2 2 3
15450 250/300 Write block 2516 len 40960 data: 6 5 3 3 3 3 2 2 2 2
15451 251/300 Read block 686 len 40960
15452 252/300 Read block 3709 len 24576
15453 253/300 Read block 741 len 40960
15454 254/300 Write block 3199 len 12288 data: 3 3 4
15455 255/300 Read block 3782 len 32768
15456 256/300 Read block 1056 len 4096
15457 257/300 Write block 1711 len 16384 data: 2 2 2 2
15458 258/300 Write block 2854 len 4096 data: 2
15459 259/300 Write block 201 len 16384 data: 4 4 5 5
15460 260/300 Read block 4026 len 16384
15461 261/300 Read block 2838 len 40960
15462 262/300 Read block 3776 len 8192
15463 263/300 Write block 3868 len 16384 data: 3 4 4 4
15464 264/300 Write block 4389 len 4096 data: 6
15465 265/300 Flush
15466 266/300 Read block 396 len 40960
15467 267/300 Write block 3774 len 36864 data: 5 6 3 3 2 3 3 5 6
15468 268/300 Read block 300 len 12288
15469 269/300 Write block 3448 len 8192 data: 3 4
15470 270/300 Write block 2389 len 16384 data: 5 5 6 5
15471 271/300 Read block 450 len 20480
15472 272/300 Write block 4466 len 40960 data: 4 4 4 3 4 3 3 3 3 3
15473 273/300 Read block 2309 len 32768
15474 274/300 Read block 2001 len 36864
15475 275/300 Write block 3251 len 8192 data: 3 3
15476 276/300 Read block 1419 len 20480
15477 277/300 Write block 1186 len 36864 data: 4 5 3 4 4 4 4 4 3
15478 278/300 Read block 3115 len 8192
15479 279/300 Write block 4270 len 4096 data: 3
15480 280/300 Read block 2681 len 20480
15481 281/300 Read block 3656 len 16384
15482 282/300 Read block 4636 len 28672
15483 283/300 Read block 1781 len 36864
15484 284/300 Read block 1214 len 20480
15485 285/300 Flush
15486 286/300 Read block 852 len 12288
15487 287/300 Flush
15488 288/300 Write block 2548 len 8192 data: 3 3
15489 289/300 Write block 2561 len 8192 data: 4 3
15490 290/300 Flush
15491 291/300 Read block 525 len 16384
15492 292/300 Write block 681 len 28672 data: 2 2 2 2 2 3 4
15493 293/300 Read block 2331 len 32768
15494 294/300 Write block 731 len 24576 data: 6 5 5 5 6 6
15495 295/300 Read block 2681 len 36864
15496 296/300 Write block 3873 len 40960 data: 3 3 3 3 3 3 3 3 3 4
15497 297/300 Write block 3505 len 16384 data: 2 2 2 3
15498 298/300 Read block 228 len 12288
15499 299/300 Read block 4366 len 24576
15500 300/300 Read block 3565 len 12288
15501 301/300 Read block 3807 len 40960
15502 Main task has work to do, go find it
15503 got dsc Start(0)
15504 start 0
15505 Mar 07 18:15:59.833 INFO request completed, latency_us: 166, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4e0716cd-097b-426c-bde7-b302dd5ffa20, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
15506 Replay: started 0, returned:Ok(())
15507 ----------------------------------------------------------------
15508 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15509 States: Offline Active Active
15510 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15511 4622 Acked Write 4 Sent Done Done false
15512 4623 Acked Read 8 Sent Done Done false
15513 4624 Acked Write 3 Sent Done Done false
15514 4625 Acked Flush 0 Sent Done Done false
15515 4626 Acked Flush 0 Sent Done Done false
15516 4627 Acked Read 2 Sent Done Done false
15517 4628 Acked Read 10 Sent Done Done false
15518 4629 Acked Write 3 Sent Done Done false
15519 4630 Acked Write 7 Sent Done Done false
15520 4631 Acked Read 5 Sent Done Done false
15521 4632 Acked Read 9 Sent Done Done false
15522 4633 Acked Write 1 Sent Done Done false
15523 4634 Acked Read 8 Sent Done Done false
15524 4635 Acked Write 9 Sent Done Done false
15525 4636 Acked Read 3 Sent Done Done false
15526 4637 Acked Write 8 Sent Done Done false
15527 4638 Acked Write 5 Sent Done Done false
15528 4639 Acked Write 7 Sent Done Done false
15529 4640 Acked Read 8 Sent Done Done false
15530 4641 Acked Write 1 Sent Done Done false
15531 4642 Acked Read 5 Sent Done Done false
15532 4643 Acked Read 8 Sent Done Done false
15533 4644 Acked Write 6 Sent Done Done false
15534 4645 Acked Read 9 Sent Done Done false
15535 4646 Acked Write 3 Sent Done Done false
15536 4647 Acked Read 5 Sent Done Done false
15537 4648 Acked Write 8 Sent Done Done false
15538 4649 Acked Write 7 Sent Done Done false
15539 4650 Acked Write 8 Sent Done Done false
15540 4651 Acked Read 9 Sent Done Done false
15541 4652 Acked Read 4 Sent Done Done false
15542 4653 Acked Write 8 Sent Done Done false
15543 4654 Acked Read 8 Sent Done Done false
15544 4655 Acked Write 4 Sent Done Done false
15545 4656 Acked Flush 0 Sent Done Done false
15546 4657 Acked Read 5 Sent Done Done false
15547 4658 Acked Write 1 Sent Done Done false
15548 4659 Acked Read 2 Sent Done Done false
15549 4660 Acked Read 8 Sent Done Done false
15550 4661 Acked Write 7 Sent Done Done false
15551 4662 Acked Read 7 Sent Done Done false
15552 4663 Acked Flush 0 Sent Done Done false
15553 4664 Acked Write 6 Sent Done Done false
15554 4665 Acked Read 5 Sent Done Done false
15555 4666 Acked Read 5 Sent Done Done false
15556 4667 Acked Read 10 Sent Done Done false
15557 4668 Acked Write 1 Sent Done Done false
15558 4669 Acked Read 1 Sent Done Done false
15559 4670 Acked Write 4 Sent Done Done false
15560 4671 Acked Write 10 Sent Done Done false
15561 4672 Acked Read 5 Sent Done Done false
15562 4673 Acked Write 7 Sent Done Done false
15563 4674 Acked Read 1 Sent Done Done false
15564 4675 Acked Write 10 Sent Done Done false
15565 4676 Acked Write 8 Sent Done Done false
15566 4677 Acked Write 9 Sent Done Done false
15567 4678 Acked Write 5 Sent Done Done false
15568 4679 Acked Write 9 Sent Done Done false
15569 4680 Acked Read 9 Sent Done Done false
15570 4681 Acked Read 4 Sent Done Done false
15571 4682 Acked Write 1 Sent Done Done false
15572 4683 Acked Write 9 Sent Done Done false
15573 4684 Acked Read 5 Sent Done Done false
15574 4685 Acked Read 2 Sent Done Done false
15575 4686 Acked Write 3 Sent Done Done false
15576 4687 Acked Write 2 Sent Done Done false
15577 4688 Acked Flush 0 Sent Done Done false
15578 4689 Acked Read 6 Sent Done Done false
15579 4690 Acked Write 7 Sent Done Done false
15580 4691 Acked Flush 0 Sent Done Done false
15581 4692 Acked Read 1 Sent Done Done false
15582 4693 Acked Read 9 Sent Done Done false
15583 4694 Acked Write 6 Sent Done Done false
15584 4695 Acked Read 2 Sent Done Done false
15585 4696 Acked Read 9 Sent Done Done false
15586 4697 Acked Write 5 Sent Done Done false
15587 4698 Acked Write 2 Sent Done Done false
15588 4699 Acked Flush 0 Sent Done Done false
15589 4700 Acked Read 9 Sent Done Done false
15590 4701 Acked Read 8 Sent Done Done false
15591 4702 Acked Read 9 Sent Done Done false
15592 4703 Acked Write 10 Sent Done Done false
15593 4704 Acked Read 6 Sent Done Done false
15594 4705 Acked Read 1 Sent Done Done false
15595 4706 Acked Flush 0 Sent Done Done false
15596 4707 Acked Read 7 Sent Done Done false
15597 4708 Acked Read 10 Sent Done Done false
15598 4709 Acked Read 1 Sent Done Done false
15599 4710 Acked Write 3 Sent Done Done false
15600 4711 Acked Read 2 Sent Done Done false
15601 4712 Acked Read 1 Sent Done Done false
15602 4713 Acked Read 8 Sent Done Done false
15603 4714 Acked Write 3 Sent Done Done false
15604 4715 Acked Read 4 Sent Done Done false
15605 4716 Acked Read 8 Sent Done Done false
15606 4717 Acked Write 6 Sent Done Done false
15607 4718 Acked Read 4 Sent Done Done false
15608 4719 Acked Read 3 Sent Done Done false
15609 4720 Acked Write 2 Sent Done Done false
15610 4721 Acked Write 9 Sent Done Done false
15611 4722 Acked Flush 0 Sent Done Done false
15612 4723 Acked Write 1 Sent Done Done false
15613 4724 Acked Read 10 Sent Done Done false
15614 4725 Acked Read 7 Sent Done Done false
15615 4726 Acked Read 9 Sent Done Done false
15616 4727 Acked Write 8 Sent Done Done false
15617 4728 Acked Write 3 Sent Done Done false
15618 4729 Acked Read 5 Sent Done Done false
15619 4730 Acked Read 1 Sent Done Done false
15620 4731 Acked Write 6 Sent Done Done false
15621 4732 Acked Write 4 Sent Done Done false
15622 4733 Acked Write 2 Sent Done Done false
15623 4734 Acked Write 8 Sent Done Done false
15624 4735 Acked Write 8 Sent Done Done false
15625 4736 Acked Flush 0 Sent Done Done false
15626 4737 Acked Read 8 Sent Done Done false
15627 4738 Acked Read 10 Sent Done Done false
15628 4739 Acked Flush 0 Sent Done Done false
15629 4740 Acked Flush 0 Sent Done Done false
15630 4741 Acked Read 4 Sent Done Done false
15631 4742 Acked Read 6 Sent Done Done false
15632 4743 Acked Flush 0 Sent Done Done false
15633 4744 Acked Write 3 Sent Done Done false
15634 4745 Acked Read 4 Sent Done Done false
15635 4746 Acked Read 9 Sent Done Done false
15636 4747 Acked Write 5 Sent Done Done false
15637 4748 Acked Read 4 Sent Done Done false
15638 4749 Acked Flush 0 Sent Done Done false
15639 4750 Acked Read 4 Sent Done Done false
15640 4751 Acked Write 2 Sent Done Done false
15641 4752 Acked Read 3 Sent Done Done false
15642 4753 Acked Read 1 Sent Done Done false
15643 4754 Acked Read 3 Sent Done Done false
15644 4755 Acked Read 5 Sent Done Done false
15645 4756 Acked Read 9 Sent Done Done false
15646 4757 Acked Write 7 Sent Done Done false
15647 4758 Acked Read 1 Sent Done Done false
15648 4759 Acked Write 1 Sent Done Done false
15649 4760 Acked Write 8 Sent Done Done false
15650 4761 Acked Read 9 Sent Done Done false
15651 4762 Acked Read 2 Sent Done Done false
15652 4763 Acked Read 9 Sent Done Done false
15653 4764 Acked Flush 0 Sent Done Done false
15654 4765 Acked Flush 0 Sent Done Done false
15655 4766 Acked Write 1 Sent Done Done false
15656 4767 Acked Write 10 Sent Done Done false
15657 4768 Acked Read 1 Sent Done Done false
15658 4769 Acked Read 1 Sent Done Done false
15659 4770 Acked Flush 0 Sent Done Done false
15660 4771 Acked Flush 0 Sent Done Done false
15661 4772 Acked Read 6 Sent Done Done false
15662 4773 Acked Read 8 Sent Done Done false
15663 4774 Acked Write 7 Sent Done Done false
15664 4775 Acked Read 1 Sent Done Done false
15665 4776 Acked Read 1 Sent Done Done false
15666 4777 Acked Read 3 Sent Done Done false
15667 4778 Acked Write 2 Sent Done Done false
15668 4779 Acked Read 9 Sent Done Done false
15669 4780 Acked Read 1 Sent Done Done false
15670 4781 Acked Read 1 Sent Done Done false
15671 4782 Acked Read 9 Sent Done Done false
15672 4783 Acked Write 7 Sent Done Done false
15673 4784 Acked Write 9 Sent Done Done false
15674 4785 Acked Write 3 Sent Done Done false
15675 4786 Acked Read 6 Sent Done Done false
15676 4787 Acked Read 5 Sent Done Done false
15677 4788 Acked Read 5 Sent Done Done false
15678 4789 Acked Read 8 Sent Done Done false
15679 4790 Acked Read 2 Sent Done Done false
15680 4791 Acked Read 1 Sent Done Done false
15681 4792 Acked Flush 0 Sent Done Done false
15682 4793 Acked Read 7 Sent Done Done false
15683 4794 Acked Read 3 Sent Done Done false
15684 4795 Acked Read 2 Sent Done Done false
15685 4796 Acked Read 9 Sent Done Done false
15686 4797 Acked Read 7 Sent Done Done false
15687 4798 Acked Read 8 Sent Done Done false
15688 4799 Acked Write 4 Sent Done Done false
15689 4800 Acked Flush 0 Sent Done Done false
15690 4801 Acked Read 2 Sent Done Done false
15691 4802 Acked Read 5 Sent Done Done false
15692 4803 Acked Read 10 Sent Done Done false
15693 4804 Acked Read 8 Sent Done Done false
15694 4805 Acked Write 1 Sent Done Done false
15695 4806 Acked Read 3 Sent Done Done false
15696 4807 Acked Read 7 Sent Done Done false
15697 4808 Acked Read 8 Sent Done Done false
15698 4809 Acked Write 6 Sent Done Done false
15699 4810 Acked Write 7 Sent Done Done false
15700 4811 Acked Write 9 Sent Done Done false
15701 4812 Acked Write 5 Sent Done Done false
15702 4813 Acked Write 4 Sent Done Done false
15703 4814 Acked Read 4 Sent Done Done false
15704 4815 Acked Read 7 Sent Done Done false
15705 4816 Acked Read 4 Sent Done Done false
15706 4817 Acked Read 4 Sent Done Done false
15707 4818 Acked Read 6 Sent Done Done false
15708 4819 Acked Flush 0 Sent Done Done false
15709 4820 Acked Read 5 Sent Done Done false
15710 4821 Acked Write 9 Sent Done Done false
15711 4822 Acked Read 2 Sent Done Done false
15712 4823 Acked Read 3 Sent Done Done false
15713 4824 Acked Write 6 Sent Done Done false
15714 4825 Acked Read 9 Sent Done Done false
15715 4826 Acked Write 5 Sent Done Done false
15716 4827 Acked Write 1 Sent Done Done false
15717 4828 Acked Read 1 Sent Done Done false
15718 4829 Acked Read 7 Sent Done Done false
15719 4830 Acked Write 10 Sent Done Done false
15720 4831 Acked Write 8 Sent Done Done false
15721 4832 Acked Write 10 Sent Done Done false
15722 4833 Acked Write 3 Sent Done Done false
15723 4834 Acked Flush 0 Sent Done Done false
15724 4835 Acked Write 10 Sent Done Done false
15725 4836 Acked Flush 0 Sent Done Done false
15726 4837 Acked Write 9 Sent Done Done false
15727 4838 Acked Read 3 Sent Done Done false
15728 4839 Acked Write 9 Sent Done Done false
15729 4840 Acked Read 5 Sent Done Done false
15730 4841 Acked Read 10 Sent Done Done false
15731 4842 Acked Read 3 Sent Done Done false
15732 4843 Acked Read 3 Sent Done Done false
15733 4844 Acked Read 3 Sent Done Done false
15734 4845 Acked Write 8 Sent Done Done false
15735 4846 Acked Read 1 Sent Done Done false
15736 4847 Acked Read 10 Sent Done Done false
15737 4848 Acked Write 4 Sent Done Done false
15738 4849 Acked Read 5 Sent Done Done false
15739 4850 Acked Read 1 Sent Done Done false
15740 4851 Acked Write 2 Sent Done Done false
15741 4852 Acked Write 1 Sent Done Done false
15742 4853 Acked Read 6 Sent Done Done false
15743 4854 Acked Read 4 Sent Done Done false
15744 4855 Acked Write 7 Sent Done Done false
15745 4856 Acked Write 10 Sent Done Done false
15746 4857 Acked Write 5 Sent Done Done false
15747 4858 Acked Write 7 Sent Done Done false
15748 4859 Acked Read 6 Sent Done Done false
15749 4860 Acked Read 6 Sent Done Done false
15750 4861 Acked Write 4 Sent Done Done false
15751 4862 Acked Write 9 Sent Done Done false
15752 4863 Acked Read 6 Sent Done Done false
15753 4864 Acked Write 3 Sent Done Done false
15754 4865 Acked Flush 0 Sent Done Done false
15755 4866 Acked Read 9 Sent Done Done false
15756 4867 Acked Read 10 Sent Done Done false
15757 4868 Acked Write 7 Sent Done Done false
15758 4869 Acked Write 4 Sent Done Done false
15759 4870 Acked Write 6 Sent Done Done false
15760 4871 Acked Write 10 Sent Done Done false
15761 4872 Acked Read 10 Sent Done Done false
15762 4873 Acked Read 6 Sent Done Done false
15763 4874 Acked Read 10 Sent Done Done false
15764 4875 Acked Write 3 Sent Done Done false
15765 4876 Acked Read 8 Sent Done Done false
15766 4877 Acked Read 1 Sent Done Done false
15767 4878 Acked Write 4 Sent Done Done false
15768 4879 Acked Write 1 Sent Done Done false
15769 4880 Acked Write 4 Sent Done Done false
15770 4881 Acked Read 4 Sent Done Done false
15771 4882 Acked Read 10 Sent Done Done false
15772 4883 Acked Read 2 Sent Done Done false
15773 4884 Acked Write 4 Sent Done Done false
15774 4885 Acked Write 1 Sent Done Done false
15775 4886 Acked Flush 0 Sent Done Done false
15776 4887 Acked Read 10 Sent Done Done false
15777 4888 Acked Write 9 Sent Done Done false
15778 4889 Acked Read 3 Sent Done Done false
15779 4890 Acked Write 2 Sent Done Done false
15780 4891 Acked Write 4 Sent Done Done false
15781 4892 Acked Read 5 Sent Done Done false
15782 4893 Acked Write 10 Sent Done Done false
15783 4894 Acked Read 8 Sent Done Done false
15784 4895 Acked Read 9 Sent Done Done false
15785 4896 Acked Write 2 Sent Done Done false
15786 4897 Acked Read 5 Sent Done Done false
15787 4898 Acked Write 9 Sent Done Done false
15788 4899 Acked Read 2 Sent Done Done false
15789 4900 Acked Write 1 Sent Done Done false
15790 4901 Acked Read 5 Sent Done Done false
15791 4902 Acked Read 4 Sent Done Done false
15792 4903 Acked Read 7 Sent Done Done false
15793 4904 Acked Read 9 Sent Done Done false
15794 4905 Acked Read 5 Sent Done Done false
15795 4906 Acked Flush 0 Sent Done Done false
15796 4907 Acked Read 3 Sent Done Done false
15797 4908 Acked Flush 0 Sent Done Done false
15798 4909 Acked Write 2 Sent Done Done false
15799 4910 Acked Write 2 Sent Done Done false
15800 4911 Acked Flush 0 Sent Done Done false
15801 4912 Acked Read 4 Sent Done Done false
15802 4913 Acked Write 7 Sent Done Done false
15803 4914 Acked Read 8 Sent Done Done false
15804 4915 Acked Write 6 Sent Done Done false
15805 4916 Acked Read 9 Sent Done Done false
15806 4917 Acked Write 10 Sent Done Done false
15807 4918 Acked Write 4 Sent Done Done false
15808 4919 Acked Read 3 Sent Done Done false
15809 4920 Acked Read 6 Sent Done Done false
15810 4921 Acked Read 3 Sent Done Done false
15811 4922 Acked Read 10 Sent Done Done false
15812 STATES DS:0 DS:1 DS:2 TOTAL
15813 Sent 301 0 0 301
15814 Done 0 301 301 602
15815 Skipped 0 0 0 0
15816 Error 0 0 0 0
15817 Last Flush: Some(JobId(4621)) Some(JobId(4911)) Some(JobId(4911))
15818 Downstairs last five completed: 4621 4620 4619 4618 4617
15819 Upstairs last five completed: 4922 4921 4920 4919 4918
15820 CLIENT: Up:0 ds:301 act:2
158212025-03-07T18:16:00.335ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
158222025-03-07T18:16:01.287ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
15823 [8810] got start action
15824 [8810] I am going to restart
15825 Starting downstairs at port 8810
15826 Make output file at "/tmp/dsc/downstairs-8810.txt"
15827 [8810][0] reports Starting
15828 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1221)
15829 [8810][0] reports Running
158302025-03-07T18:16:02.290ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
158312025-03-07T18:16:03.292ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
15832 ----------------------------------------------------------------
15833 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15834 States: Offline Active Active
15835 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15836 4622 Acked Write 4 Sent Done Done false
15837 4623 Acked Read 8 Sent Done Done false
15838 4624 Acked Write 3 Sent Done Done false
15839 4625 Acked Flush 0 Sent Done Done false
15840 4626 Acked Flush 0 Sent Done Done false
15841 4627 Acked Read 2 Sent Done Done false
15842 4628 Acked Read 10 Sent Done Done false
15843 4629 Acked Write 3 Sent Done Done false
15844 4630 Acked Write 7 Sent Done Done false
15845 4631 Acked Read 5 Sent Done Done false
15846 4632 Acked Read 9 Sent Done Done false
15847 4633 Acked Write 1 Sent Done Done false
15848 4634 Acked Read 8 Sent Done Done false
15849 4635 Acked Write 9 Sent Done Done false
15850 4636 Acked Read 3 Sent Done Done false
15851 4637 Acked Write 8 Sent Done Done false
15852 4638 Acked Write 5 Sent Done Done false
15853 4639 Acked Write 7 Sent Done Done false
15854 4640 Acked Read 8 Sent Done Done false
15855 4641 Acked Write 1 Sent Done Done false
15856 4642 Acked Read 5 Sent Done Done false
15857 4643 Acked Read 8 Sent Done Done false
15858 4644 Acked Write 6 Sent Done Done false
15859 4645 Acked Read 9 Sent Done Done false
15860 4646 Acked Write 3 Sent Done Done false
15861 4647 Acked Read 5 Sent Done Done false
15862 4648 Acked Write 8 Sent Done Done false
15863 4649 Acked Write 7 Sent Done Done false
15864 4650 Acked Write 8 Sent Done Done false
15865 4651 Acked Read 9 Sent Done Done false
15866 4652 Acked Read 4 Sent Done Done false
15867 4653 Acked Write 8 Sent Done Done false
15868 4654 Acked Read 8 Sent Done Done false
15869 4655 Acked Write 4 Sent Done Done false
15870 4656 Acked Flush 0 Sent Done Done false
15871 4657 Acked Read 5 Sent Done Done false
15872 4658 Acked Write 1 Sent Done Done false
15873 4659 Acked Read 2 Sent Done Done false
15874 4660 Acked Read 8 Sent Done Done false
15875 4661 Acked Write 7 Sent Done Done false
15876 4662 Acked Read 7 Sent Done Done false
15877 4663 Acked Flush 0 Sent Done Done false
15878 4664 Acked Write 6 Sent Done Done false
15879 4665 Acked Read 5 Sent Done Done false
15880 4666 Acked Read 5 Sent Done Done false
15881 4667 Acked Read 10 Sent Done Done false
15882 4668 Acked Write 1 Sent Done Done false
15883 4669 Acked Read 1 Sent Done Done false
15884 4670 Acked Write 4 Sent Done Done false
15885 4671 Acked Write 10 Sent Done Done false
15886 4672 Acked Read 5 Sent Done Done false
15887 4673 Acked Write 7 Sent Done Done false
15888 4674 Acked Read 1 Sent Done Done false
15889 4675 Acked Write 10 Sent Done Done false
15890 4676 Acked Write 8 Sent Done Done false
15891 4677 Acked Write 9 Sent Done Done false
15892 4678 Acked Write 5 Sent Done Done false
15893 4679 Acked Write 9 Sent Done Done false
15894 4680 Acked Read 9 Sent Done Done false
15895 4681 Acked Read 4 Sent Done Done false
15896 4682 Acked Write 1 Sent Done Done false
15897 4683 Acked Write 9 Sent Done Done false
15898 4684 Acked Read 5 Sent Done Done false
15899 4685 Acked Read 2 Sent Done Done false
15900 4686 Acked Write 3 Sent Done Done false
15901 4687 Acked Write 2 Sent Done Done false
15902 4688 Acked Flush 0 Sent Done Done false
15903 4689 Acked Read 6 Sent Done Done false
15904 4690 Acked Write 7 Sent Done Done false
15905 4691 Acked Flush 0 Sent Done Done false
15906 4692 Acked Read 1 Sent Done Done false
15907 4693 Acked Read 9 Sent Done Done false
15908 4694 Acked Write 6 Sent Done Done false
15909 4695 Acked Read 2 Sent Done Done false
15910 4696 Acked Read 9 Sent Done Done false
15911 4697 Acked Write 5 Sent Done Done false
15912 4698 Acked Write 2 Sent Done Done false
15913 4699 Acked Flush 0 Sent Done Done false
15914 4700 Acked Read 9 Sent Done Done false
15915 4701 Acked Read 8 Sent Done Done false
15916 4702 Acked Read 9 Sent Done Done false
15917 4703 Acked Write 10 Sent Done Done false
15918 4704 Acked Read 6 Sent Done Done false
15919 4705 Acked Read 1 Sent Done Done false
15920 4706 Acked Flush 0 Sent Done Done false
15921 4707 Acked Read 7 Sent Done Done false
15922 4708 Acked Read 10 Sent Done Done false
15923 4709 Acked Read 1 Sent Done Done false
15924 4710 Acked Write 3 Sent Done Done false
15925 4711 Acked Read 2 Sent Done Done false
15926 4712 Acked Read 1 Sent Done Done false
15927 4713 Acked Read 8 Sent Done Done false
15928 4714 Acked Write 3 Sent Done Done false
15929 4715 Acked Read 4 Sent Done Done false
15930 4716 Acked Read 8 Sent Done Done false
15931 4717 Acked Write 6 Sent Done Done false
15932 4718 Acked Read 4 Sent Done Done false
15933 4719 Acked Read 3 Sent Done Done false
15934 4720 Acked Write 2 Sent Done Done false
15935 4721 Acked Write 9 Sent Done Done false
15936 4722 Acked Flush 0 Sent Done Done false
15937 4723 Acked Write 1 Sent Done Done false
15938 4724 Acked Read 10 Sent Done Done false
15939 4725 Acked Read 7 Sent Done Done false
15940 4726 Acked Read 9 Sent Done Done false
15941 4727 Acked Write 8 Sent Done Done false
15942 4728 Acked Write 3 Sent Done Done false
15943 4729 Acked Read 5 Sent Done Done false
15944 4730 Acked Read 1 Sent Done Done false
15945 4731 Acked Write 6 Sent Done Done false
15946 4732 Acked Write 4 Sent Done Done false
15947 4733 Acked Write 2 Sent Done Done false
15948 4734 Acked Write 8 Sent Done Done false
15949 4735 Acked Write 8 Sent Done Done false
15950 4736 Acked Flush 0 Sent Done Done false
15951 4737 Acked Read 8 Sent Done Done false
15952 4738 Acked Read 10 Sent Done Done false
15953 4739 Acked Flush 0 Sent Done Done false
15954 4740 Acked Flush 0 Sent Done Done false
15955 4741 Acked Read 4 Sent Done Done false
15956 4742 Acked Read 6 Sent Done Done false
15957 4743 Acked Flush 0 Sent Done Done false
15958 4744 Acked Write 3 Sent Done Done false
15959 4745 Acked Read 4 Sent Done Done false
15960 4746 Acked Read 9 Sent Done Done false
15961 4747 Acked Write 5 Sent Done Done false
15962 4748 Acked Read 4 Sent Done Done false
15963 4749 Acked Flush 0 Sent Done Done false
15964 4750 Acked Read 4 Sent Done Done false
15965 4751 Acked Write 2 Sent Done Done false
15966 4752 Acked Read 3 Sent Done Done false
15967 4753 Acked Read 1 Sent Done Done false
15968 4754 Acked Read 3 Sent Done Done false
15969 4755 Acked Read 5 Sent Done Done false
15970 4756 Acked Read 9 Sent Done Done false
15971 4757 Acked Write 7 Sent Done Done false
15972 4758 Acked Read 1 Sent Done Done false
15973 4759 Acked Write 1 Sent Done Done false
15974 4760 Acked Write 8 Sent Done Done false
15975 4761 Acked Read 9 Sent Done Done false
15976 4762 Acked Read 2 Sent Done Done false
15977 4763 Acked Read 9 Sent Done Done false
15978 4764 Acked Flush 0 Sent Done Done false
15979 4765 Acked Flush 0 Sent Done Done false
15980 4766 Acked Write 1 Sent Done Done false
15981 4767 Acked Write 10 Sent Done Done false
15982 4768 Acked Read 1 Sent Done Done false
15983 4769 Acked Read 1 Sent Done Done false
15984 4770 Acked Flush 0 Sent Done Done false
15985 4771 Acked Flush 0 Sent Done Done false
15986 4772 Acked Read 6 Sent Done Done false
15987 4773 Acked Read 8 Sent Done Done false
15988 4774 Acked Write 7 Sent Done Done false
15989 4775 Acked Read 1 Sent Done Done false
15990 4776 Acked Read 1 Sent Done Done false
15991 4777 Acked Read 3 Sent Done Done false
15992 4778 Acked Write 2 Sent Done Done false
15993 4779 Acked Read 9 Sent Done Done false
15994 4780 Acked Read 1 Sent Done Done false
15995 4781 Acked Read 1 Sent Done Done false
15996 4782 Acked Read 9 Sent Done Done false
15997 4783 Acked Write 7 Sent Done Done false
15998 4784 Acked Write 9 Sent Done Done false
15999 4785 Acked Write 3 Sent Done Done false
16000 4786 Acked Read 6 Sent Done Done false
16001 4787 Acked Read 5 Sent Done Done false
16002 4788 Acked Read 5 Sent Done Done false
16003 4789 Acked Read 8 Sent Done Done false
16004 4790 Acked Read 2 Sent Done Done false
16005 4791 Acked Read 1 Sent Done Done false
16006 4792 Acked Flush 0 Sent Done Done false
16007 4793 Acked Read 7 Sent Done Done false
16008 4794 Acked Read 3 Sent Done Done false
16009 4795 Acked Read 2 Sent Done Done false
16010 4796 Acked Read 9 Sent Done Done false
16011 4797 Acked Read 7 Sent Done Done false
16012 4798 Acked Read 8 Sent Done Done false
16013 4799 Acked Write 4 Sent Done Done false
16014 4800 Acked Flush 0 Sent Done Done false
16015 4801 Acked Read 2 Sent Done Done false
16016 4802 Acked Read 5 Sent Done Done false
16017 4803 Acked Read 10 Sent Done Done false
16018 4804 Acked Read 8 Sent Done Done false
16019 4805 Acked Write 1 Sent Done Done false
16020 4806 Acked Read 3 Sent Done Done false
16021 4807 Acked Read 7 Sent Done Done false
16022 4808 Acked Read 8 Sent Done Done false
16023 4809 Acked Write 6 Sent Done Done false
16024 4810 Acked Write 7 Sent Done Done false
16025 4811 Acked Write 9 Sent Done Done false
16026 4812 Acked Write 5 Sent Done Done false
16027 4813 Acked Write 4 Sent Done Done false
16028 4814 Acked Read 4 Sent Done Done false
16029 4815 Acked Read 7 Sent Done Done false
16030 4816 Acked Read 4 Sent Done Done false
16031 4817 Acked Read 4 Sent Done Done false
16032 4818 Acked Read 6 Sent Done Done false
16033 4819 Acked Flush 0 Sent Done Done false
16034 4820 Acked Read 5 Sent Done Done false
16035 4821 Acked Write 9 Sent Done Done false
16036 4822 Acked Read 2 Sent Done Done false
16037 4823 Acked Read 3 Sent Done Done false
16038 4824 Acked Write 6 Sent Done Done false
16039 4825 Acked Read 9 Sent Done Done false
16040 4826 Acked Write 5 Sent Done Done false
16041 4827 Acked Write 1 Sent Done Done false
16042 4828 Acked Read 1 Sent Done Done false
16043 4829 Acked Read 7 Sent Done Done false
16044 4830 Acked Write 10 Sent Done Done false
16045 4831 Acked Write 8 Sent Done Done false
16046 4832 Acked Write 10 Sent Done Done false
16047 4833 Acked Write 3 Sent Done Done false
16048 4834 Acked Flush 0 Sent Done Done false
16049 4835 Acked Write 10 Sent Done Done false
16050 4836 Acked Flush 0 Sent Done Done false
16051 4837 Acked Write 9 Sent Done Done false
16052 4838 Acked Read 3 Sent Done Done false
16053 4839 Acked Write 9 Sent Done Done false
16054 4840 Acked Read 5 Sent Done Done false
16055 4841 Acked Read 10 Sent Done Done false
16056 4842 Acked Read 3 Sent Done Done false
16057 4843 Acked Read 3 Sent Done Done false
16058 4844 Acked Read 3 Sent Done Done false
16059 4845 Acked Write 8 Sent Done Done false
16060 4846 Acked Read 1 Sent Done Done false
16061 4847 Acked Read 10 Sent Done Done false
16062 4848 Acked Write 4 Sent Done Done false
16063 4849 Acked Read 5 Sent Done Done false
16064 4850 Acked Read 1 Sent Done Done false
16065 4851 Acked Write 2 Sent Done Done false
16066 4852 Acked Write 1 Sent Done Done false
16067 4853 Acked Read 6 Sent Done Done false
16068 4854 Acked Read 4 Sent Done Done false
16069 4855 Acked Write 7 Sent Done Done false
16070 4856 Acked Write 10 Sent Done Done false
16071 4857 Acked Write 5 Sent Done Done false
16072 4858 Acked Write 7 Sent Done Done false
16073 4859 Acked Read 6 Sent Done Done false
16074 4860 Acked Read 6 Sent Done Done false
16075 4861 Acked Write 4 Sent Done Done false
16076 4862 Acked Write 9 Sent Done Done false
16077 4863 Acked Read 6 Sent Done Done false
16078 4864 Acked Write 3 Sent Done Done false
16079 4865 Acked Flush 0 Sent Done Done false
16080 4866 Acked Read 9 Sent Done Done false
16081 4867 Acked Read 10 Sent Done Done false
16082 4868 Acked Write 7 Sent Done Done false
16083 4869 Acked Write 4 Sent Done Done false
16084 4870 Acked Write 6 Sent Done Done false
16085 4871 Acked Write 10 Sent Done Done false
16086 4872 Acked Read 10 Sent Done Done false
16087 4873 Acked Read 6 Sent Done Done false
16088 4874 Acked Read 10 Sent Done Done false
16089 4875 Acked Write 3 Sent Done Done false
16090 4876 Acked Read 8 Sent Done Done false
16091 4877 Acked Read 1 Sent Done Done false
16092 4878 Acked Write 4 Sent Done Done false
16093 4879 Acked Write 1 Sent Done Done false
16094 4880 Acked Write 4 Sent Done Done false
16095 4881 Acked Read 4 Sent Done Done false
16096 4882 Acked Read 10 Sent Done Done false
16097 4883 Acked Read 2 Sent Done Done false
16098 4884 Acked Write 4 Sent Done Done false
16099 4885 Acked Write 1 Sent Done Done false
16100 4886 Acked Flush 0 Sent Done Done false
16101 4887 Acked Read 10 Sent Done Done false
16102 4888 Acked Write 9 Sent Done Done false
16103 4889 Acked Read 3 Sent Done Done false
16104 4890 Acked Write 2 Sent Done Done false
16105 4891 Acked Write 4 Sent Done Done false
16106 4892 Acked Read 5 Sent Done Done false
16107 4893 Acked Write 10 Sent Done Done false
16108 4894 Acked Read 8 Sent Done Done false
16109 4895 Acked Read 9 Sent Done Done false
16110 4896 Acked Write 2 Sent Done Done false
16111 4897 Acked Read 5 Sent Done Done false
16112 4898 Acked Write 9 Sent Done Done false
16113 4899 Acked Read 2 Sent Done Done false
16114 4900 Acked Write 1 Sent Done Done false
16115 4901 Acked Read 5 Sent Done Done false
16116 4902 Acked Read 4 Sent Done Done false
16117 4903 Acked Read 7 Sent Done Done false
16118 4904 Acked Read 9 Sent Done Done false
16119 4905 Acked Read 5 Sent Done Done false
16120 4906 Acked Flush 0 Sent Done Done false
16121 4907 Acked Read 3 Sent Done Done false
16122 4908 Acked Flush 0 Sent Done Done false
16123 4909 Acked Write 2 Sent Done Done false
16124 4910 Acked Write 2 Sent Done Done false
16125 4911 Acked Flush 0 Sent Done Done false
16126 4912 Acked Read 4 Sent Done Done false
16127 4913 Acked Write 7 Sent Done Done false
16128 4914 Acked Read 8 Sent Done Done false
16129 4915 Acked Write 6 Sent Done Done false
16130 4916 Acked Read 9 Sent Done Done false
16131 4917 Acked Write 10 Sent Done Done false
16132 4918 Acked Write 4 Sent Done Done false
16133 4919 Acked Read 3 Sent Done Done false
16134 4920 Acked Read 6 Sent Done Done false
16135 4921 Acked Read 3 Sent Done Done false
16136 4922 Acked Read 10 Sent Done Done false
16137 STATES DS:0 DS:1 DS:2 TOTAL
16138 Sent 301 0 0 301
16139 Done 0 301 301 602
16140 Skipped 0 0 0 0
16141 Error 0 0 0 0
16142 Last Flush: Some(JobId(4621)) Some(JobId(4911)) Some(JobId(4911))
16143 Downstairs last five completed: 4621 4620 4619 4618 4617
16144 Upstairs last five completed: 4922 4921 4920 4919 4918
16145 CLIENT: Up:0 ds:301 act:2
161462025-03-07T18:16:04.336ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161472025-03-07T18:16:05.295ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161482025-03-07T18:16:05.582ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161492025-03-07T18:16:05.582ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50873) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161502025-03-07T18:16:05.583ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161512025-03-07T18:16:05.584ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161522025-03-07T18:16:05.584ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161532025-03-07T18:16:05.584ZINFOcrucible: send last flush ID to this DS: Some(JobId(4621)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161542025-03-07T18:16:05.584ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161552025-03-07T18:16:05.584ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4621)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161562025-03-07T18:16:05.584ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4621)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
16157 ----------------------------------------------------------------
16158 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16159 Downstairs last five completed: 4923 4922 4921 4920 4919
16160 Upstairs last five completed: 4923 4922 4921 4920 4919
16161 CLIENT: Up:0 ds:0 act:3
16162 Replay: All jobs finished
16163 Main task has work to do, go find it
16164 got dsc Stop(0)
16165 stop 0
16166 Mar 07 18:16:07.838 INFO request completed, latency_us: 376, response_code: 204, uri: /stop/cid/0, [8810] Got stop action so:false kr:false
16167 method: POST, req_id: 8f8476d6-ff20-4a03-9148-7f7387edbb31, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
16168 Mar 07 18:16:07.838 INFO request completed, latency_us: 100, response_code: 200, uri: /state/cid/0, method: GET, req_id: 10012e5e-beca-4ae1-9785-73f4b0675dab, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
161692025-03-07T18:16:07.838ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161702025-03-07T18:16:07.838ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161712025-03-07T18:16:07.838ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
161722025-03-07T18:16:07.838ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
16173 [8810] Exited with: ExitStatus(unix_wait_status(9))
16174 [8810][0] reports Exit
16175 Mar 07 18:16:11.841 INFO request completed, latency_us: 297, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3cc6e139-8c1a-45b1-b9e4-edfb3f8e13d3, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
16176 001/300 Write block 4029 len 40960 data: 2 3 4 3 3 3 3 5 5 6
16177 002/300 Flush
16178 003/300 Read block 1621 len 40960
16179 004/300 Read block 1723 len 16384
16180 005/300 Write block 645 len 40960 data: 4 4 4 4 4 4 3 2 2 2
16181 006/300 Flush
16182 007/300 Write block 2630 len 4096 data: 3
16183 008/300 Write block 4634 len 8192 data: 4 4
16184 009/300 Flush
16185 010/300 Write block 2644 len 28672 data: 2 2 2 2 4 4 4
16186 011/300 Write block 1310 len 32768 data: 3 3 3 2 3 3 4 4
16187 012/300 Flush
16188 013/300 Read block 1099 len 24576
16189 014/300 Write block 870 len 8192 data: 4 4
16190 015/300 Write block 931 len 36864 data: 2 2 4 3 3 3 4 5 5
16191 016/300 Read block 3663 len 32768
16192 017/300 Read block 4141 len 12288
16193 018/300 Write block 614 len 36864 data: 3 4 5 4 5 6 5 5 6
16194 019/300 Read block 2453 len 4096
16195 020/300 Write block 1867 len 20480 data: 3 2 2 4 4
16196 021/300 Write block 2651 len 36864 data: 4 3 3 3 4 4 5 5 4
16197 022/300 Read block 846 len 20480
16198 023/300 Write block 3241 len 32768 data: 3 3 3 2 2 2 2 2
16199 024/300 Flush
16200 025/300 Write block 2555 len 8192 data: 6 6
16201 026/300 Read block 4679 len 12288
16202 027/300 Write block 638 len 28672 data: 3 3 3 3 2 3 3
16203 028/300 Read block 4563 len 4096
16204 029/300 Read block 1846 len 20480
16205 030/300 Write block 4446 len 20480 data: 2 2 2 4 4
16206 031/300 Read block 976 len 28672
16207 032/300 Flush
16208 033/300 Read block 4144 len 32768
16209 034/300 Write block 693 len 36864 data: 3 3 3 4 4 4 4 4 4
16210 035/300 Write block 1699 len 4096 data: 6
16211 036/300 Read block 1576 len 32768
16212 037/300 Write block 2583 len 4096 data: 4
16213 038/300 Read block 4880 len 4096
16214 039/300 Write block 4379 len 8192 data: 4 3
16215 040/300 Read block 846 len 16384
16216 041/300 Write block 4732 len 32768 data: 2 2 2 2 2 3 3 3
16217 042/300 Write block 2243 len 12288 data: 5 5 3
16218 043/300 Write block 2004 len 4096 data: 3
16219 044/300 Read block 537 len 20480
16220 045/300 Write block 620 len 4096 data: 6
16221 046/300 Read block 4147 len 4096
16222 047/300 Read block 973 len 32768
16223 048/300 Read block 4354 len 36864
16224 049/300 Read block 2025 len 8192
16225 050/300 Write block 2638 len 36864 data: 3 2 2 2 2 2 3 3 3
16226 051/300 Read block 1714 len 4096
16227 052/300 Flush
16228 053/300 Read block 3289 len 32768
16229 054/300 Read block 2629 len 8192
16230 055/300 Write block 3101 len 8192 data: 5 5
16231 056/300 Write block 4120 len 36864 data: 2 2 4 4 4 5 5 6 7
16232 057/300 Write block 170 len 16384 data: 4 4 4 4
16233 058/300 Read block 1267 len 40960
16234 059/300 Write block 3145 len 40960 data: 4 4 5 3 4 4 5 5 4 4
16235 060/300 Read block 429 len 12288
16236 061/300 Write block 4198 len 28672 data: 4 5 5 5 6 7 6
16237 062/300 Read block 4758 len 12288
16238 063/300 Read block 1824 len 4096
16239 064/300 Write block 1000 len 28672 data: 2 2 2 2 3 3 3
16240 065/300 Read block 3243 len 28672
16241 066/300 Write block 3878 len 40960 data: 4 4 4 4 5 4 4 5 5 3
16242 067/300 Read block 4378 len 8192
16243 068/300 Write block 1934 len 4096 data: 2
16244 069/300 Read block 1734 len 4096
16245 070/300 Read block 1758 len 32768
16246 071/300 Write block 2353 len 24576 data: 2 2 2 2 2 2
16247 072/300 Read block 3914 len 36864
16248 073/300 Write block 4914 len 12288 data: 5 5 5
16249 074/300 Write block 3655 len 4096 data: 4
16250 075/300 Read block 2274 len 20480
16251 076/300 Write block 1515 len 16384 data: 4 5 3 3
16252 077/300 Write block 429 len 8192 data: 5 5
16253 078/300 Read block 2965 len 16384
16254 079/300 Write block 2206 len 24576 data: 2 3 3 5 4 3
16255 080/300 Read block 1673 len 40960
16256 081/300 Flush
16257 082/300 Write block 4563 len 12288 data: 3 3 2
16258 083/300 Flush
16259 084/300 Write block 1464 len 4096 data: 7
16260 085/300 Write block 1507 len 36864 data: 2 2 2 2 2 4 4 4 5
16261 086/300 Flush
16262 087/300 Read block 3048 len 12288
16263 088/300 Read block 4478 len 24576
16264 089/300 Read block 3886 len 20480
16265 090/300 Write block 254 len 4096 data: 3
16266 091/300 Write block 3772 len 28672 data: 4 5 6 7 4 4 3
16267 092/300 Read block 3815 len 8192
16268 093/300 Write block 3202 len 36864 data: 4 5 4 4 4 4 4 4 4
16269 094/300 Read block 121 len 28672
16270 095/300 Read block 3933 len 20480
16271 096/300 Read block 4200 len 40960
16272 097/300 Read block 1523 len 32768
16273 098/300 Flush
16274 099/300 Read block 2852 len 4096
16275 100/300 Read block 4557 len 8192
16276 101/300 Write block 2104 len 28672 data: 5 5 5 4 4 4 5
16277 102/300 Read block 3350 len 8192
16278 103/300 Read block 1127 len 16384
16279 104/300 Read block 483 len 28672
16280 105/300 Read block 1512 len 28672
16281 106/300 Read block 2706 len 36864
16282 107/300 Flush
16283 108/300 Flush
16284 109/300 Read block 1199 len 16384
16285 110/300 Write block 2428 len 32768 data: 3 2 2 2 3 2 2 2
16286 111/300 Read block 4886 len 32768
16287 112/300 Write block 3967 len 28672 data: 2 2 2 2 2 2 2
16288 113/300 Write block 2689 len 32768 data: 2 2 2 3 3 3 3 2
16289 114/300 Write block 310 len 16384 data: 6 5 3 3
16290 115/300 Write block 3922 len 28672 data: 7 5 5 5 4 5 4
16291 116/300 Read block 395 len 32768
16292 117/300 Write block 188 len 40960 data: 2 3 4 4 4 6 5 6 6 6
16293 118/300 Read block 535 len 28672
16294 119/300 Write block 30 len 40960 data: 3 2 2 2 2 2 2 2 2 2
16295 120/300 Read block 3519 len 8192
16296 121/300 Read block 2276 len 16384
16297 122/300 Flush
16298 123/300 Read block 111 len 36864
16299 124/300 Write block 626 len 20480 data: 6 6 4 4 4
16300 125/300 Read block 1524 len 32768
16301 126/300 Write block 1524 len 36864 data: 4 3 2 2 2 2 2 4 4
16302 127/300 Read block 4224 len 36864
16303 128/300 Read block 1303 len 20480
16304 129/300 Write block 4921 len 36864 data: 4 4 4 4 4 4 4 3 2
16305 130/300 Read block 3500 len 36864
16306 131/300 Read block 1812 len 40960
16307 132/300 Read block 122 len 12288
16308 133/300 Read block 1089 len 36864
16309 134/300 Read block 3287 len 32768
16310 135/300 Read block 2744 len 20480
16311 136/300 Write block 901 len 40960 data: 2 2 3 3 3 2 2 2 3 3
16312 137/300 Read block 1524 len 4096
16313 138/300 Write block 3576 len 28672 data: 3 3 3 3 3 3 4
16314 139/300 Read block 1260 len 40960
16315 140/300 Read block 1461 len 36864
16316 141/300 Write block 3167 len 4096 data: 4
16317 142/300 Write block 1884 len 40960 data: 2 2 2 2 3 5 5 4 4 4
16318 143/300 Flush
16319 144/300 Flush
16320 145/300 Read block 243 len 24576
16321 146/300 Read block 62 len 20480
16322 147/300 Flush
16323 148/300 Read block 1303 len 4096
16324 149/300 Read block 1351 len 12288
16325 150/300 Read block 2269 len 36864
16326 151/300 Read block 1958 len 4096
16327 152/300 Write block 1235 len 4096 data: 5
16328 153/300 Flush
16329 154/300 Write block 3649 len 12288 data: 5 4 4
16330 155/300 Flush
16331 156/300 Read block 3087 len 20480
16332 157/300 Write block 3125 len 36864 data: 5 5 5 5 4 4 5 6 5
16333 158/300 Write block 1626 len 24576 data: 5 4 3 4 3 2
16334 159/300 Read block 852 len 24576
16335 160/300 Read block 2913 len 24576
16336 161/300 Flush
16337 162/300 Flush
16338 163/300 Read block 1697 len 28672
16339 164/300 Write block 3016 len 20480 data: 3 2 2 2 2
16340 165/300 Write block 168 len 12288 data: 4 3 5
16341 166/300 Flush
16342 167/300 Read block 945 len 24576
16343 168/300 Read block 3368 len 28672
16344 169/300 Read block 268 len 12288
16345 170/300 Write block 1162 len 12288 data: 5 5 6
16346 171/300 Read block 3432 len 28672
16347 172/300 Write block 2839 len 20480 data: 3 4 4 3 3
16348 173/300 Flush
16349 174/300 Write block 4376 len 4096 data: 5
16350 175/300 Read block 3539 len 8192
16351 176/300 Write block 394 len 4096 data: 4
16352 177/300 Write block 4523 len 4096 data: 4
16353 178/300 Read block 4509 len 24576
16354 179/300 Write block 3819 len 12288 data: 2 2 2
16355 180/300 Write block 1656 len 32768 data: 4 3 3 2 2 2 2 2
16356 181/300 Read block 1350 len 24576
16357 182/300 Flush
16358 183/300 Write block 2756 len 16384 data: 3 3 2 2
16359 184/300 Write block 3613 len 24576 data: 2 2 3 3 3 3
16360 185/300 Write block 3191 len 36864 data: 5 6 4 3 3 3 3 3 4
16361 186/300 Write block 383 len 4096 data: 4
16362 187/300 Read block 1021 len 4096
16363 188/300 Read block 4796 len 20480
16364 189/300 Read block 1319 len 16384
16365 190/300 Write block 264 len 32768 data: 4 5 5 5 4 3 3 4
16366 191/300 Write block 2461 len 24576 data: 3 4 6 6 6 6
16367 192/300 Write block 1617 len 12288 data: 7 6 7
16368 193/300 Read block 2079 len 28672
16369 194/300 Write block 393 len 8192 data: 5 5
16370 195/300 Read block 1737 len 40960
16371 196/300 Read block 975 len 16384
16372 197/300 Write block 1016 len 40960 data: 4 5 5 4 4 5 4 5 5 5
16373 198/300 Write block 4957 len 8192 data: 4 5
16374 199/300 Read block 2976 len 36864
16375 200/300 Write block 281 len 4096 data: 4
16376 201/300 Read block 4567 len 32768
16377 202/300 Write block 3315 len 8192 data: 2 2
16378 203/300 Read block 4626 len 8192
16379 204/300 Write block 4940 len 12288 data: 3 2 2
16380 205/300 Write block 68 len 36864 data: 4 4 5 5 6 6 6 6 6
16381 206/300 Write block 1437 len 20480 data: 5 5 5 5 4
16382 207/300 Read block 4441 len 40960
16383 208/300 Read block 1059 len 36864
16384 209/300 Read block 3530 len 16384
16385 210/300 Write block 2482 len 16384 data: 5 4 3 3
16386 211/300 Read block 4681 len 28672
16387 212/300 Write block 3357 len 40960 data: 3 3 2 2 2 3 3 3 3 3
16388 213/300 Read block 2103 len 28672
16389 214/300 Write block 4827 len 12288 data: 3 3 3
16390 215/300 Read block 4290 len 32768
16391 216/300 Read block 3563 len 32768
16392 217/300 Write block 177 len 20480 data: 3 3 4 4 3
16393 218/300 Read block 2061 len 16384
16394 219/300 Read block 1483 len 36864
16395 220/300 Read block 3627 len 20480
16396 221/300 Write block 686 len 36864 data: 4 5 3 3 2 2 3 4 4
16397 222/300 Read block 4691 len 8192
16398 223/300 Read block 2249 len 20480
16399 224/300 Write block 3883 len 28672 data: 5 5 6 6 4 2 2
16400 225/300 Write block 4051 len 32768 data: 6 4 4 5 6 6 5 3
16401 226/300 Write block 359 len 4096 data: 2
16402 227/300 Read block 2546 len 36864
16403 228/300 Read block 1695 len 28672
16404 229/300 Read block 4279 len 36864
16405 230/300 Read block 602 len 40960
16406 231/300 Flush
16407 232/300 Write block 2449 len 8192 data: 2 3
16408 233/300 Flush
16409 234/300 Read block 500 len 36864
16410 235/300 Write block 4947 len 40960 data: 4 3 2 3 2 2 2 3 3 3
16411 236/300 Flush
16412 237/300 Write block 1932 len 4096 data: 2
16413 238/300 Read block 1386 len 40960
16414 239/300 Write block 604 len 32768 data: 5 5 5 5 3 3 3 2
16415 240/300 Read block 3224 len 24576
16416 241/300 Write block 3532 len 8192 data: 3 3
16417 242/300 Write block 3445 len 36864 data: 6 3 3 4 5 5 5 4 4
16418 243/300 Write block 2733 len 32768 data: 6 5 5 5 6 6 7 5
16419 244/300 Flush
16420 245/300 Write block 745 len 28672 data: 7 7 7 7 5 4 4
16421 246/300 Read block 1951 len 4096
16422 247/300 Read block 581 len 12288
16423 248/300 Read block 1268 len 32768
16424 249/300 Write block 2487 len 4096 data: 4
16425 250/300 Read block 1012 len 24576
16426 251/300 Write block 3024 len 12288 data: 3 3 4
16427 252/300 Read block 46 len 28672
16428 253/300 Read block 442 len 4096
16429 254/300 Read block 759 len 16384
16430 255/300 Write block 483 len 40960 data: 2 2 2 2 2 5 4 4 5 3
16431 256/300 Write block 3200 len 28672 data: 4 5 5 6 5 5 5
16432 257/300 Write block 4748 len 36864 data: 5 3 3 3 2 2 2 3 3
16433 258/300 Read block 3076 len 28672
16434 259/300 Read block 2198 len 12288
16435 260/300 Read block 3213 len 24576
16436 261/300 Read block 3738 len 8192
16437 262/300 Write block 4965 len 4096 data: 5
16438 263/300 Write block 3722 len 4096 data: 4
16439 264/300 Write block 382 len 12288 data: 4 5 4
16440 265/300 Write block 255 len 8192 data: 4 4
16441 266/300 Read block 2012 len 16384
16442 267/300 Write block 684 len 28672 data: 3 3 5 6 4 4 3
16443 268/300 Read block 3311 len 32768
16444 269/300 Write block 2032 len 36864 data: 4 4 3 3 3 3 2 2 2
16445 270/300 Write block 4056 len 36864 data: 7 6 4 3 3 3 3 2 3
16446 271/300 Write block 4916 len 36864 data: 6 6 7 5 5 5 5 5 5
16447 272/300 Write block 2640 len 16384 data: 3 3 3 3
16448 273/300 Write block 4843 len 40960 data: 3 3 2 2 2 2 2 2 4 4
16449 274/300 Read block 3446 len 20480
16450 275/300 Read block 4277 len 36864
16451 276/300 Write block 4112 len 20480 data: 4 5 4 3 2
16452 277/300 Flush
16453 278/300 Write block 2474 len 12288 data: 3 3 3
16454 279/300 Write block 4205 len 32768 data: 6 6 6 8 5 5 6 6
16455 280/300 Read block 580 len 32768
16456 281/300 Read block 4464 len 24576
16457 282/300 Read block 1699 len 24576
16458 283/300 Flush
16459 284/300 Read block 2747 len 28672
16460 285/300 Read block 1664 len 28672
16461 286/300 Read block 691 len 32768
16462 287/300 Read block 3481 len 12288
16463 288/300 Write block 3676 len 20480 data: 3 2 2 3 3
16464 289/300 Flush
16465 290/300 Write block 3824 len 12288 data: 2 2 2
16466 291/300 Read block 4651 len 8192
16467 292/300 Read block 1325 len 36864
16468 293/300 Write block 744 len 32768 data: 7 8 8 8 8 6 5 5
16469 294/300 Write block 875 len 40960 data: 2 2 2 3 3 3 5 4 3 3
16470 295/300 Flush
16471 296/300 Read block 1951 len 8192
16472 297/300 Read block 4773 len 28672
16473 298/300 Write block 2108 len 36864 data: 5 5 6 5 4 4 5 5 4
16474 299/300 Read block 2621 len 8192
16475 300/300 Write block 378 len 12288 data: 4 4 3
16476 301/300 Write block 4919 len 4096 data: 6
16477 Main task has work to do, go find it
16478 got dsc Start(0)
16479 start 0
16480 Mar 07 18:16:12.087 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/0, method: POST, req_id: 933cce4e-95ea-4e2e-b25b-0f8bc86019df, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
16481 Replay: started 0, returned:Ok(())
16482 ----------------------------------------------------------------
16483 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16484 States: Offline Active Active
16485 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16486 4924 Acked Write 10 Sent Done Done false
16487 4925 Acked Flush 0 Sent Done Done false
16488 4926 Acked Read 10 Sent Done Done false
16489 4927 Acked Read 4 Sent Done Done false
16490 4928 Acked Write 10 Sent Done Done false
16491 4929 Acked Flush 0 Sent Done Done false
16492 4930 Acked Write 1 Sent Done Done false
16493 4931 Acked Write 2 Sent Done Done false
16494 4932 Acked Flush 0 Sent Done Done false
16495 4933 Acked Write 7 Sent Done Done false
16496 4934 Acked Write 8 Sent Done Done false
16497 4935 Acked Flush 0 Sent Done Done false
16498 4936 Acked Read 6 Sent Done Done false
16499 4937 Acked Write 2 Sent Done Done false
16500 4938 Acked Write 9 Sent Done Done false
16501 4939 Acked Read 8 Sent Done Done false
16502 4940 Acked Read 3 Sent Done Done false
16503 4941 Acked Write 9 Sent Done Done false
16504 4942 Acked Read 1 Sent Done Done false
16505 4943 Acked Write 5 Sent Done Done false
16506 4944 Acked Write 9 Sent Done Done false
16507 4945 Acked Read 5 Sent Done Done false
16508 4946 Acked Write 8 Sent Done Done false
16509 4947 Acked Flush 0 Sent Done Done false
16510 4948 Acked Write 2 Sent Done Done false
16511 4949 Acked Read 3 Sent Done Done false
16512 4950 Acked Write 7 Sent Done Done false
16513 4951 Acked Read 1 Sent Done Done false
16514 4952 Acked Read 5 Sent Done Done false
16515 4953 Acked Write 5 Sent Done Done false
16516 4954 Acked Read 7 Sent Done Done false
16517 4955 Acked Flush 0 Sent Done Done false
16518 4956 Acked Read 8 Sent Done Done false
16519 4957 Acked Write 9 Sent Done Done false
16520 4958 Acked Write 1 Sent Done Done false
16521 4959 Acked Read 8 Sent Done Done false
16522 4960 Acked Write 1 Sent Done Done false
16523 4961 Acked Read 1 Sent Done Done false
16524 4962 Acked Write 2 Sent Done Done false
16525 4963 Acked Read 4 Sent Done Done false
16526 4964 Acked Write 8 Sent Done Done false
16527 4965 Acked Write 3 Sent Done Done false
16528 4966 Acked Write 1 Sent Done Done false
16529 4967 Acked Read 5 Sent Done Done false
16530 4968 Acked Write 1 Sent Done Done false
16531 4969 Acked Read 1 Sent Done Done false
16532 4970 Acked Read 8 Sent Done Done false
16533 4971 Acked Read 9 Sent Done Done false
16534 4972 Acked Read 2 Sent Done Done false
16535 4973 Acked Write 9 Sent Done Done false
16536 4974 Acked Read 1 Sent Done Done false
16537 4975 Acked Flush 0 Sent Done Done false
16538 4976 Acked Read 8 Sent Done Done false
16539 4977 Acked Read 2 Sent Done Done false
16540 4978 Acked Write 2 Sent Done Done false
16541 4979 Acked Write 9 Sent Done Done false
16542 4980 Acked Write 4 Sent Done Done false
16543 4981 Acked Read 10 Sent Done Done false
16544 4982 Acked Write 10 Sent Done Done false
16545 4983 Acked Read 3 Sent Done Done false
16546 4984 Acked Write 7 Sent Done Done false
16547 4985 Acked Read 3 Sent Done Done false
16548 4986 Acked Read 1 Sent Done Done false
16549 4987 Acked Write 7 Sent Done Done false
16550 4988 Acked Read 7 Sent Done Done false
16551 4989 Acked Write 10 Sent Done Done false
16552 4990 Acked Read 2 Sent Done Done false
16553 4991 Acked Write 1 Sent Done Done false
16554 4992 Acked Read 1 Sent Done Done false
16555 4993 Acked Read 8 Sent Done Done false
16556 4994 Acked Write 6 Sent Done Done false
16557 4995 Acked Read 9 Sent Done Done false
16558 4996 Acked Write 3 Sent Done Done false
16559 4997 Acked Write 1 Sent Done Done false
16560 4998 Acked Read 5 Sent Done Done false
16561 4999 Acked Write 4 Sent Done Done false
16562 5000 Acked Write 2 Sent Done Done false
16563 5001 Acked Read 4 Sent Done Done false
16564 5002 Acked Write 6 Sent Done Done false
16565 5003 Acked Read 10 Sent Done Done false
16566 5004 Acked Flush 0 Sent Done Done false
16567 5005 Acked Write 3 Sent Done Done false
16568 5006 Acked Flush 0 Sent Done Done false
16569 5007 Acked Write 1 Sent Done Done false
16570 5008 Acked Write 9 Sent Done Done false
16571 5009 Acked Flush 0 Sent Done Done false
16572 5010 Acked Read 3 Sent Done Done false
16573 5011 Acked Read 6 Sent Done Done false
16574 5012 Acked Read 5 Sent Done Done false
16575 5013 Acked Write 1 Sent Done Done false
16576 5014 Acked Write 7 Sent Done Done false
16577 5015 Acked Read 2 Sent Done Done false
16578 5016 Acked Write 9 Sent Done Done false
16579 5017 Acked Read 7 Sent Done Done false
16580 5018 Acked Read 5 Sent Done Done false
16581 5019 Acked Read 10 Sent Done Done false
16582 5020 Acked Read 8 Sent Done Done false
16583 5021 Acked Flush 0 Sent Done Done false
16584 5022 Acked Read 1 Sent Done Done false
16585 5023 Acked Read 2 Sent Done Done false
16586 5024 Acked Write 7 Sent Done Done false
16587 5025 Acked Read 2 Sent Done Done false
16588 5026 Acked Read 4 Sent Done Done false
16589 5027 Acked Read 7 Sent Done Done false
16590 5028 Acked Read 7 Sent Done Done false
16591 5029 Acked Read 9 Sent Done Done false
16592 5030 Acked Flush 0 Sent Done Done false
16593 5031 Acked Flush 0 Sent Done Done false
16594 5032 Acked Read 4 Sent Done Done false
16595 5033 Acked Write 8 Sent Done Done false
16596 5034 Acked Read 8 Sent Done Done false
16597 5035 Acked Write 7 Sent Done Done false
16598 5036 Acked Write 8 Sent Done Done false
16599 5037 Acked Write 4 Sent Done Done false
16600 5038 Acked Write 7 Sent Done Done false
16601 5039 Acked Read 8 Sent Done Done false
16602 5040 Acked Write 10 Sent Done Done false
16603 5041 Acked Read 7 Sent Done Done false
16604 5042 Acked Write 10 Sent Done Done false
16605 5043 Acked Read 2 Sent Done Done false
16606 5044 Acked Read 4 Sent Done Done false
16607 5045 Acked Flush 0 Sent Done Done false
16608 5046 Acked Read 9 Sent Done Done false
16609 5047 Acked Write 5 Sent Done Done false
16610 5048 Acked Read 8 Sent Done Done false
16611 5049 Acked Write 9 Sent Done Done false
16612 5050 Acked Read 9 Sent Done Done false
16613 5051 Acked Read 5 Sent Done Done false
16614 5052 Acked Write 9 Sent Done Done false
16615 5053 Acked Read 9 Sent Done Done false
16616 5054 Acked Read 10 Sent Done Done false
16617 5055 Acked Read 3 Sent Done Done false
16618 5056 Acked Read 9 Sent Done Done false
16619 5057 Acked Read 8 Sent Done Done false
16620 5058 Acked Read 5 Sent Done Done false
16621 5059 Acked Write 10 Sent Done Done false
16622 5060 Acked Read 1 Sent Done Done false
16623 5061 Acked Write 7 Sent Done Done false
16624 5062 Acked Read 10 Sent Done Done false
16625 5063 Acked Read 9 Sent Done Done false
16626 5064 Acked Write 1 Sent Done Done false
16627 5065 Acked Write 10 Sent Done Done false
16628 5066 Acked Flush 0 Sent Done Done false
16629 5067 Acked Flush 0 Sent Done Done false
16630 5068 Acked Read 6 Sent Done Done false
16631 5069 Acked Read 5 Sent Done Done false
16632 5070 Acked Flush 0 Sent Done Done false
16633 5071 Acked Read 1 Sent Done Done false
16634 5072 Acked Read 3 Sent Done Done false
16635 5073 Acked Read 9 Sent Done Done false
16636 5074 Acked Read 1 Sent Done Done false
16637 5075 Acked Write 1 Sent Done Done false
16638 5076 Acked Flush 0 Sent Done Done false
16639 5077 Acked Write 3 Sent Done Done false
16640 5078 Acked Flush 0 Sent Done Done false
16641 5079 Acked Read 5 Sent Done Done false
16642 5080 Acked Write 9 Sent Done Done false
16643 5081 Acked Write 6 Sent Done Done false
16644 5082 Acked Read 6 Sent Done Done false
16645 5083 Acked Read 6 Sent Done Done false
16646 5084 Acked Flush 0 Sent Done Done false
16647 5085 Acked Flush 0 Sent Done Done false
16648 5086 Acked Read 7 Sent Done Done false
16649 5087 Acked Write 5 Sent Done Done false
16650 5088 Acked Write 3 Sent Done Done false
16651 5089 Acked Flush 0 Sent Done Done false
16652 5090 Acked Read 6 Sent Done Done false
16653 5091 Acked Read 7 Sent Done Done false
16654 5092 Acked Read 3 Sent Done Done false
16655 5093 Acked Write 3 Sent Done Done false
16656 5094 Acked Read 7 Sent Done Done false
16657 5095 Acked Write 5 Sent Done Done false
16658 5096 Acked Flush 0 Sent Done Done false
16659 5097 Acked Write 1 Sent Done Done false
16660 5098 Acked Read 2 Sent Done Done false
16661 5099 Acked Write 1 Sent Done Done false
16662 5100 Acked Write 1 Sent Done Done false
16663 5101 Acked Read 6 Sent Done Done false
16664 5102 Acked Write 3 Sent Done Done false
16665 5103 Acked Write 8 Sent Done Done false
16666 5104 Acked Read 6 Sent Done Done false
16667 5105 Acked Flush 0 Sent Done Done false
16668 5106 Acked Write 4 Sent Done Done false
16669 5107 Acked Write 6 Sent Done Done false
16670 5108 Acked Write 9 Sent Done Done false
16671 5109 Acked Write 1 Sent Done Done false
16672 5110 Acked Read 1 Sent Done Done false
16673 5111 Acked Read 5 Sent Done Done false
16674 5112 Acked Read 4 Sent Done Done false
16675 5113 Acked Write 8 Sent Done Done false
16676 5114 Acked Write 6 Sent Done Done false
16677 5115 Acked Write 3 Sent Done Done false
16678 5116 Acked Read 7 Sent Done Done false
16679 5117 Acked Write 2 Sent Done Done false
16680 5118 Acked Read 10 Sent Done Done false
16681 5119 Acked Read 4 Sent Done Done false
16682 5120 Acked Write 10 Sent Done Done false
16683 5121 Acked Write 2 Sent Done Done false
16684 5122 Acked Read 9 Sent Done Done false
16685 5123 Acked Write 1 Sent Done Done false
16686 5124 Acked Read 8 Sent Done Done false
16687 5125 Acked Write 2 Sent Done Done false
16688 5126 Acked Read 2 Sent Done Done false
16689 5127 Acked Write 3 Sent Done Done false
16690 5128 Acked Write 9 Sent Done Done false
16691 5129 Acked Write 5 Sent Done Done false
16692 5130 Acked Read 10 Sent Done Done false
16693 5131 Acked Read 9 Sent Done Done false
16694 5132 Acked Read 4 Sent Done Done false
16695 5133 Acked Write 4 Sent Done Done false
16696 5134 Acked Read 7 Sent Done Done false
16697 5135 Acked Write 10 Sent Done Done false
16698 5136 Acked Read 7 Sent Done Done false
16699 5137 Acked Write 3 Sent Done Done false
16700 5138 Acked Read 8 Sent Done Done false
16701 5139 Acked Read 8 Sent Done Done false
16702 5140 Acked Write 5 Sent Done Done false
16703 5141 Acked Read 4 Sent Done Done false
16704 5142 Acked Read 9 Sent Done Done false
16705 5143 Acked Read 5 Sent Done Done false
16706 5144 Acked Write 9 Sent Done Done false
16707 5145 Acked Read 2 Sent Done Done false
16708 5146 Acked Read 5 Sent Done Done false
16709 5147 Acked Write 7 Sent Done Done false
16710 5148 Acked Write 8 Sent Done Done false
16711 5149 Acked Write 1 Sent Done Done false
16712 5150 Acked Read 9 Sent Done Done false
16713 5151 Acked Read 7 Sent Done Done false
16714 5152 Acked Read 9 Sent Done Done false
16715 5153 Acked Read 10 Sent Done Done false
16716 5154 Acked Flush 0 Sent Done Done false
16717 5155 Acked Write 2 Sent Done Done false
16718 5156 Acked Flush 0 Sent Done Done false
16719 5157 Acked Read 9 Sent Done Done false
16720 5158 Acked Write 10 Sent Done Done false
16721 5159 Acked Flush 0 Sent Done Done false
16722 5160 Acked Write 1 Sent Done Done false
16723 5161 Acked Read 10 Sent Done Done false
16724 5162 Acked Write 8 Sent Done Done false
16725 5163 Acked Read 6 Sent Done Done false
16726 5164 Acked Write 2 Sent Done Done false
16727 5165 Acked Write 9 Sent Done Done false
16728 5166 Acked Write 8 Sent Done Done false
16729 5167 Acked Flush 0 Sent Done Done false
16730 5168 Acked Write 7 Sent Done Done false
16731 5169 Acked Read 1 Sent Done Done false
16732 5170 Acked Read 3 Sent Done Done false
16733 5171 Acked Read 8 Sent Done Done false
16734 5172 Acked Write 1 Sent Done Done false
16735 5173 Acked Read 6 Sent Done Done false
16736 5174 Acked Write 3 Sent Done Done false
16737 5175 Acked Read 7 Sent Done Done false
16738 5176 Acked Read 1 Sent Done Done false
16739 5177 Acked Read 4 Sent Done Done false
16740 5178 Acked Write 10 Sent Done Done false
16741 5179 Acked Write 7 Sent Done Done false
16742 5180 Acked Write 9 Sent Done Done false
16743 5181 Acked Read 7 Sent Done Done false
16744 5182 Acked Read 3 Sent Done Done false
16745 5183 Acked Read 6 Sent Done Done false
16746 5184 Acked Read 2 Sent Done Done false
16747 5185 Acked Write 1 Sent Done Done false
16748 5186 Acked Write 1 Sent Done Done false
16749 5187 Acked Write 3 Sent Done Done false
16750 5188 Acked Write 2 Sent Done Done false
16751 5189 Acked Read 4 Sent Done Done false
16752 5190 Acked Write 7 Sent Done Done false
16753 5191 Acked Read 8 Sent Done Done false
16754 5192 Acked Write 9 Sent Done Done false
16755 5193 Acked Write 9 Sent Done Done false
16756 5194 Acked Write 9 Sent Done Done false
16757 5195 Acked Write 4 Sent Done Done false
16758 5196 Acked Write 10 Sent Done Done false
16759 5197 Acked Read 5 Sent Done Done false
16760 5198 Acked Read 9 Sent Done Done false
16761 5199 Acked Write 5 Sent Done Done false
16762 5200 Acked Flush 0 Sent Done Done false
16763 5201 Acked Write 3 Sent Done Done false
16764 5202 Acked Write 8 Sent Done Done false
16765 5203 Acked Read 8 Sent Done Done false
16766 5204 Acked Read 6 Sent Done Done false
16767 5205 Acked Read 6 Sent Done Done false
16768 5206 Acked Flush 0 Sent Done Done false
16769 5207 Acked Read 7 Sent Done Done false
16770 5208 Acked Read 7 Sent Done Done false
16771 5209 Acked Read 8 Sent Done Done false
16772 5210 Acked Read 3 Sent Done Done false
16773 5211 Acked Write 5 Sent Done Done false
16774 5212 Acked Flush 0 Sent Done Done false
16775 5213 Acked Write 3 Sent Done Done false
16776 5214 Acked Read 2 Sent Done Done false
16777 5215 Acked Read 9 Sent Done Done false
16778 5216 Acked Write 8 Sent Done Done false
16779 5217 Acked Write 10 Sent Done Done false
16780 5218 Acked Flush 0 Sent Done Done false
16781 5219 Acked Read 2 Sent Done Done false
16782 5220 Acked Read 7 Sent Done Done false
16783 5221 Acked Write 9 Sent Done Done false
16784 5222 Acked Read 2 Sent Done Done false
16785 5223 Acked Write 3 Sent Done Done false
16786 5224 Acked Write 1 Sent Done Done false
16787 STATES DS:0 DS:1 DS:2 TOTAL
16788 Sent 301 0 0 301
16789 Done 0 301 301 602
16790 Skipped 0 0 0 0
16791 Error 0 0 0 0
16792 Last Flush: Some(JobId(4923)) Some(JobId(5218)) Some(JobId(5218))
16793 Downstairs last five completed: 4923 4922 4921 4920 4919
16794 Upstairs last five completed: 5224 5223 5222 5221 5220
16795 CLIENT: Up:0 ds:301 act:2
167962025-03-07T18:16:12.590ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
167972025-03-07T18:16:13.309ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
167982025-03-07T18:16:14.309ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
167992025-03-07T18:16:15.312ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
16800 [8810] got start action
16801 [8810] I am going to restart
16802 Starting downstairs at port 8810
16803 Make output file at "/tmp/dsc/downstairs-8810.txt"
16804 [8810][0] reports Starting
16805 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1222)
16806 [8810][0] reports Running
16807 ----------------------------------------------------------------
16808 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16809 States: Offline Active Active
16810 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16811 4924 Acked Write 10 Sent Done Done false
16812 4925 Acked Flush 0 Sent Done Done false
16813 4926 Acked Read 10 Sent Done Done false
16814 4927 Acked Read 4 Sent Done Done false
16815 4928 Acked Write 10 Sent Done Done false
16816 4929 Acked Flush 0 Sent Done Done false
16817 4930 Acked Write 1 Sent Done Done false
16818 4931 Acked Write 2 Sent Done Done false
16819 4932 Acked Flush 0 Sent Done Done false
16820 4933 Acked Write 7 Sent Done Done false
16821 4934 Acked Write 8 Sent Done Done false
16822 4935 Acked Flush 0 Sent Done Done false
16823 4936 Acked Read 6 Sent Done Done false
16824 4937 Acked Write 2 Sent Done Done false
16825 4938 Acked Write 9 Sent Done Done false
16826 4939 Acked Read 8 Sent Done Done false
16827 4940 Acked Read 3 Sent Done Done false
16828 4941 Acked Write 9 Sent Done Done false
16829 4942 Acked Read 1 Sent Done Done false
16830 4943 Acked Write 5 Sent Done Done false
16831 4944 Acked Write 9 Sent Done Done false
16832 4945 Acked Read 5 Sent Done Done false
16833 4946 Acked Write 8 Sent Done Done false
16834 4947 Acked Flush 0 Sent Done Done false
16835 4948 Acked Write 2 Sent Done Done false
16836 4949 Acked Read 3 Sent Done Done false
16837 4950 Acked Write 7 Sent Done Done false
16838 4951 Acked Read 1 Sent Done Done false
16839 4952 Acked Read 5 Sent Done Done false
16840 4953 Acked Write 5 Sent Done Done false
16841 4954 Acked Read 7 Sent Done Done false
16842 4955 Acked Flush 0 Sent Done Done false
16843 4956 Acked Read 8 Sent Done Done false
16844 4957 Acked Write 9 Sent Done Done false
16845 4958 Acked Write 1 Sent Done Done false
16846 4959 Acked Read 8 Sent Done Done false
16847 4960 Acked Write 1 Sent Done Done false
16848 4961 Acked Read 1 Sent Done Done false
16849 4962 Acked Write 2 Sent Done Done false
16850 4963 Acked Read 4 Sent Done Done false
16851 4964 Acked Write 8 Sent Done Done false
16852 4965 Acked Write 3 Sent Done Done false
16853 4966 Acked Write 1 Sent Done Done false
16854 4967 Acked Read 5 Sent Done Done false
16855 4968 Acked Write 1 Sent Done Done false
16856 4969 Acked Read 1 Sent Done Done false
16857 4970 Acked Read 8 Sent Done Done false
16858 4971 Acked Read 9 Sent Done Done false
16859 4972 Acked Read 2 Sent Done Done false
16860 4973 Acked Write 9 Sent Done Done false
16861 4974 Acked Read 1 Sent Done Done false
16862 4975 Acked Flush 0 Sent Done Done false
16863 4976 Acked Read 8 Sent Done Done false
16864 4977 Acked Read 2 Sent Done Done false
16865 4978 Acked Write 2 Sent Done Done false
16866 4979 Acked Write 9 Sent Done Done false
16867 4980 Acked Write 4 Sent Done Done false
16868 4981 Acked Read 10 Sent Done Done false
16869 4982 Acked Write 10 Sent Done Done false
16870 4983 Acked Read 3 Sent Done Done false
16871 4984 Acked Write 7 Sent Done Done false
16872 4985 Acked Read 3 Sent Done Done false
16873 4986 Acked Read 1 Sent Done Done false
16874 4987 Acked Write 7 Sent Done Done false
16875 4988 Acked Read 7 Sent Done Done false
16876 4989 Acked Write 10 Sent Done Done false
16877 4990 Acked Read 2 Sent Done Done false
16878 4991 Acked Write 1 Sent Done Done false
16879 4992 Acked Read 1 Sent Done Done false
16880 4993 Acked Read 8 Sent Done Done false
16881 4994 Acked Write 6 Sent Done Done false
16882 4995 Acked Read 9 Sent Done Done false
16883 4996 Acked Write 3 Sent Done Done false
16884 4997 Acked Write 1 Sent Done Done false
16885 4998 Acked Read 5 Sent Done Done false
16886 4999 Acked Write 4 Sent Done Done false
16887 5000 Acked Write 2 Sent Done Done false
16888 5001 Acked Read 4 Sent Done Done false
16889 5002 Acked Write 6 Sent Done Done false
16890 5003 Acked Read 10 Sent Done Done false
16891 5004 Acked Flush 0 Sent Done Done false
16892 5005 Acked Write 3 Sent Done Done false
16893 5006 Acked Flush 0 Sent Done Done false
16894 5007 Acked Write 1 Sent Done Done false
16895 5008 Acked Write 9 Sent Done Done false
16896 5009 Acked Flush 0 Sent Done Done false
16897 5010 Acked Read 3 Sent Done Done false
16898 5011 Acked Read 6 Sent Done Done false
16899 5012 Acked Read 5 Sent Done Done false
16900 5013 Acked Write 1 Sent Done Done false
16901 5014 Acked Write 7 Sent Done Done false
16902 5015 Acked Read 2 Sent Done Done false
16903 5016 Acked Write 9 Sent Done Done false
16904 5017 Acked Read 7 Sent Done Done false
16905 5018 Acked Read 5 Sent Done Done false
16906 5019 Acked Read 10 Sent Done Done false
16907 5020 Acked Read 8 Sent Done Done false
16908 5021 Acked Flush 0 Sent Done Done false
16909 5022 Acked Read 1 Sent Done Done false
16910 5023 Acked Read 2 Sent Done Done false
16911 5024 Acked Write 7 Sent Done Done false
16912 5025 Acked Read 2 Sent Done Done false
16913 5026 Acked Read 4 Sent Done Done false
16914 5027 Acked Read 7 Sent Done Done false
16915 5028 Acked Read 7 Sent Done Done false
16916 5029 Acked Read 9 Sent Done Done false
16917 5030 Acked Flush 0 Sent Done Done false
16918 5031 Acked Flush 0 Sent Done Done false
16919 5032 Acked Read 4 Sent Done Done false
16920 5033 Acked Write 8 Sent Done Done false
16921 5034 Acked Read 8 Sent Done Done false
16922 5035 Acked Write 7 Sent Done Done false
16923 5036 Acked Write 8 Sent Done Done false
16924 5037 Acked Write 4 Sent Done Done false
16925 5038 Acked Write 7 Sent Done Done false
16926 5039 Acked Read 8 Sent Done Done false
16927 5040 Acked Write 10 Sent Done Done false
16928 5041 Acked Read 7 Sent Done Done false
16929 5042 Acked Write 10 Sent Done Done false
16930 5043 Acked Read 2 Sent Done Done false
16931 5044 Acked Read 4 Sent Done Done false
16932 5045 Acked Flush 0 Sent Done Done false
16933 5046 Acked Read 9 Sent Done Done false
16934 5047 Acked Write 5 Sent Done Done false
16935 5048 Acked Read 8 Sent Done Done false
16936 5049 Acked Write 9 Sent Done Done false
16937 5050 Acked Read 9 Sent Done Done false
16938 5051 Acked Read 5 Sent Done Done false
16939 5052 Acked Write 9 Sent Done Done false
16940 5053 Acked Read 9 Sent Done Done false
16941 5054 Acked Read 10 Sent Done Done false
16942 5055 Acked Read 3 Sent Done Done false
16943 5056 Acked Read 9 Sent Done Done false
16944 5057 Acked Read 8 Sent Done Done false
16945 5058 Acked Read 5 Sent Done Done false
16946 5059 Acked Write 10 Sent Done Done false
16947 5060 Acked Read 1 Sent Done Done false
16948 5061 Acked Write 7 Sent Done Done false
16949 5062 Acked Read 10 Sent Done Done false
16950 5063 Acked Read 9 Sent Done Done false
16951 5064 Acked Write 1 Sent Done Done false
16952 5065 Acked Write 10 Sent Done Done false
16953 5066 Acked Flush 0 Sent Done Done false
16954 5067 Acked Flush 0 Sent Done Done false
16955 5068 Acked Read 6 Sent Done Done false
16956 5069 Acked Read 5 Sent Done Done false
16957 5070 Acked Flush 0 Sent Done Done false
16958 5071 Acked Read 1 Sent Done Done false
16959 5072 Acked Read 3 Sent Done Done false
16960 5073 Acked Read 9 Sent Done Done false
16961 5074 Acked Read 1 Sent Done Done false
16962 5075 Acked Write 1 Sent Done Done false
16963 5076 Acked Flush 0 Sent Done Done false
16964 5077 Acked Write 3 Sent Done Done false
16965 5078 Acked Flush 0 Sent Done Done false
16966 5079 Acked Read 5 Sent Done Done false
16967 5080 Acked Write 9 Sent Done Done false
16968 5081 Acked Write 6 Sent Done Done false
16969 5082 Acked Read 6 Sent Done Done false
16970 5083 Acked Read 6 Sent Done Done false
16971 5084 Acked Flush 0 Sent Done Done false
16972 5085 Acked Flush 0 Sent Done Done false
16973 5086 Acked Read 7 Sent Done Done false
16974 5087 Acked Write 5 Sent Done Done false
16975 5088 Acked Write 3 Sent Done Done false
16976 5089 Acked Flush 0 Sent Done Done false
16977 5090 Acked Read 6 Sent Done Done false
16978 5091 Acked Read 7 Sent Done Done false
16979 5092 Acked Read 3 Sent Done Done false
16980 5093 Acked Write 3 Sent Done Done false
16981 5094 Acked Read 7 Sent Done Done false
16982 5095 Acked Write 5 Sent Done Done false
16983 5096 Acked Flush 0 Sent Done Done false
16984 5097 Acked Write 1 Sent Done Done false
16985 5098 Acked Read 2 Sent Done Done false
16986 5099 Acked Write 1 Sent Done Done false
16987 5100 Acked Write 1 Sent Done Done false
16988 5101 Acked Read 6 Sent Done Done false
16989 5102 Acked Write 3 Sent Done Done false
16990 5103 Acked Write 8 Sent Done Done false
16991 5104 Acked Read 6 Sent Done Done false
16992 5105 Acked Flush 0 Sent Done Done false
16993 5106 Acked Write 4 Sent Done Done false
16994 5107 Acked Write 6 Sent Done Done false
16995 5108 Acked Write 9 Sent Done Done false
16996 5109 Acked Write 1 Sent Done Done false
16997 5110 Acked Read 1 Sent Done Done false
16998 5111 Acked Read 5 Sent Done Done false
16999 5112 Acked Read 4 Sent Done Done false
17000 5113 Acked Write 8 Sent Done Done false
17001 5114 Acked Write 6 Sent Done Done false
17002 5115 Acked Write 3 Sent Done Done false
17003 5116 Acked Read 7 Sent Done Done false
17004 5117 Acked Write 2 Sent Done Done false
17005 5118 Acked Read 10 Sent Done Done false
17006 5119 Acked Read 4 Sent Done Done false
17007 5120 Acked Write 10 Sent Done Done false
17008 5121 Acked Write 2 Sent Done Done false
17009 5122 Acked Read 9 Sent Done Done false
17010 5123 Acked Write 1 Sent Done Done false
17011 5124 Acked Read 8 Sent Done Done false
17012 5125 Acked Write 2 Sent Done Done false
17013 5126 Acked Read 2 Sent Done Done false
17014 5127 Acked Write 3 Sent Done Done false
17015 5128 Acked Write 9 Sent Done Done false
17016 5129 Acked Write 5 Sent Done Done false
17017 5130 Acked Read 10 Sent Done Done false
17018 5131 Acked Read 9 Sent Done Done false
17019 5132 Acked Read 4 Sent Done Done false
17020 5133 Acked Write 4 Sent Done Done false
17021 5134 Acked Read 7 Sent Done Done false
17022 5135 Acked Write 10 Sent Done Done false
17023 5136 Acked Read 7 Sent Done Done false
17024 5137 Acked Write 3 Sent Done Done false
17025 5138 Acked Read 8 Sent Done Done false
17026 5139 Acked Read 8 Sent Done Done false
17027 5140 Acked Write 5 Sent Done Done false
17028 5141 Acked Read 4 Sent Done Done false
17029 5142 Acked Read 9 Sent Done Done false
17030 5143 Acked Read 5 Sent Done Done false
17031 5144 Acked Write 9 Sent Done Done false
17032 5145 Acked Read 2 Sent Done Done false
17033 5146 Acked Read 5 Sent Done Done false
17034 5147 Acked Write 7 Sent Done Done false
17035 5148 Acked Write 8 Sent Done Done false
17036 5149 Acked Write 1 Sent Done Done false
17037 5150 Acked Read 9 Sent Done Done false
17038 5151 Acked Read 7 Sent Done Done false
17039 5152 Acked Read 9 Sent Done Done false
17040 5153 Acked Read 10 Sent Done Done false
17041 5154 Acked Flush 0 Sent Done Done false
17042 5155 Acked Write 2 Sent Done Done false
17043 5156 Acked Flush 0 Sent Done Done false
17044 5157 Acked Read 9 Sent Done Done false
17045 5158 Acked Write 10 Sent Done Done false
17046 5159 Acked Flush 0 Sent Done Done false
17047 5160 Acked Write 1 Sent Done Done false
17048 5161 Acked Read 10 Sent Done Done false
17049 5162 Acked Write 8 Sent Done Done false
17050 5163 Acked Read 6 Sent Done Done false
17051 5164 Acked Write 2 Sent Done Done false
17052 5165 Acked Write 9 Sent Done Done false
17053 5166 Acked Write 8 Sent Done Done false
17054 5167 Acked Flush 0 Sent Done Done false
17055 5168 Acked Write 7 Sent Done Done false
17056 5169 Acked Read 1 Sent Done Done false
17057 5170 Acked Read 3 Sent Done Done false
17058 5171 Acked Read 8 Sent Done Done false
17059 5172 Acked Write 1 Sent Done Done false
17060 5173 Acked Read 6 Sent Done Done false
17061 5174 Acked Write 3 Sent Done Done false
17062 5175 Acked Read 7 Sent Done Done false
17063 5176 Acked Read 1 Sent Done Done false
17064 5177 Acked Read 4 Sent Done Done false
17065 5178 Acked Write 10 Sent Done Done false
17066 5179 Acked Write 7 Sent Done Done false
17067 5180 Acked Write 9 Sent Done Done false
17068 5181 Acked Read 7 Sent Done Done false
17069 5182 Acked Read 3 Sent Done Done false
17070 5183 Acked Read 6 Sent Done Done false
17071 5184 Acked Read 2 Sent Done Done false
17072 5185 Acked Write 1 Sent Done Done false
17073 5186 Acked Write 1 Sent Done Done false
17074 5187 Acked Write 3 Sent Done Done false
17075 5188 Acked Write 2 Sent Done Done false
17076 5189 Acked Read 4 Sent Done Done false
17077 5190 Acked Write 7 Sent Done Done false
17078 5191 Acked Read 8 Sent Done Done false
17079 5192 Acked Write 9 Sent Done Done false
17080 5193 Acked Write 9 Sent Done Done false
17081 5194 Acked Write 9 Sent Done Done false
17082 5195 Acked Write 4 Sent Done Done false
17083 5196 Acked Write 10 Sent Done Done false
17084 5197 Acked Read 5 Sent Done Done false
17085 5198 Acked Read 9 Sent Done Done false
17086 5199 Acked Write 5 Sent Done Done false
17087 5200 Acked Flush 0 Sent Done Done false
17088 5201 Acked Write 3 Sent Done Done false
17089 5202 Acked Write 8 Sent Done Done false
17090 5203 Acked Read 8 Sent Done Done false
17091 5204 Acked Read 6 Sent Done Done false
17092 5205 Acked Read 6 Sent Done Done false
17093 5206 Acked Flush 0 Sent Done Done false
17094 5207 Acked Read 7 Sent Done Done false
17095 5208 Acked Read 7 Sent Done Done false
17096 5209 Acked Read 8 Sent Done Done false
17097 5210 Acked Read 3 Sent Done Done false
17098 5211 Acked Write 5 Sent Done Done false
17099 5212 Acked Flush 0 Sent Done Done false
17100 5213 Acked Write 3 Sent Done Done false
17101 5214 Acked Read 2 Sent Done Done false
17102 5215 Acked Read 9 Sent Done Done false
17103 5216 Acked Write 8 Sent Done Done false
17104 5217 Acked Write 10 Sent Done Done false
17105 5218 Acked Flush 0 Sent Done Done false
17106 5219 Acked Read 2 Sent Done Done false
17107 5220 Acked Read 7 Sent Done Done false
17108 5221 Acked Write 9 Sent Done Done false
17109 5222 Acked Read 2 Sent Done Done false
17110 5223 Acked Write 3 Sent Done Done false
17111 5224 Acked Write 1 Sent Done Done false
17112 STATES DS:0 DS:1 DS:2 TOTAL
17113 Sent 301 0 0 301
17114 Done 0 301 301 602
17115 Skipped 0 0 0 0
17116 Error 0 0 0 0
17117 Last Flush: Some(JobId(4923)) Some(JobId(5218)) Some(JobId(5218))
17118 Downstairs last five completed: 4923 4922 4921 4920 4919
17119 Upstairs last five completed: 5224 5223 5222 5221 5220
17120 CLIENT: Up:0 ds:301 act:2
171212025-03-07T18:16:17.315ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171222025-03-07T18:16:17.839ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171232025-03-07T18:16:17.839ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47549) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171242025-03-07T18:16:17.840ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171252025-03-07T18:16:17.841ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171262025-03-07T18:16:17.841ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171272025-03-07T18:16:17.841ZINFOcrucible: send last flush ID to this DS: Some(JobId(4923)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171282025-03-07T18:16:17.841ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171292025-03-07T18:16:17.841ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4923)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171302025-03-07T18:16:17.841ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4923)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
17131 ----------------------------------------------------------------
17132 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17133 Downstairs last five completed: 5225 5224 5223 5222 5221
17134 Upstairs last five completed: 5225 5224 5223 5222 5221
17135 CLIENT: Up:0 ds:0 act:3
17136 Replay: All jobs finished
17137 Main task has work to do, go find it
17138 got dsc Stop(0)
17139 stop 0
17140 Mar 07 18:16:20.093 INFO request completed, latency_us: 325, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 2d174307-86ea-431b-8a45-2731fb58e059, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
17141 [8810] Got stop action so:false kr:false
17142 Mar 07 18:16:20.093 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8e52d7e0-a189-4603-a2b8-7d5de10d6bf7, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
171432025-03-07T18:16:20.094ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171442025-03-07T18:16:20.094ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171452025-03-07T18:16:20.094ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
171462025-03-07T18:16:20.094ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
17147 [8810] Exited with: ExitStatus(unix_wait_status(9))
17148 [8810][0] reports Exit
17149 Mar 07 18:16:24.096 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/0, method: GET, req_id: 88a91acc-7457-4924-ab59-fa30a101ccc6, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
17150 001/300 Read block 3015 len 24576
17151 002/300 Read block 4807 len 36864
17152 003/300 Read block 3926 len 40960
17153 004/300 Read block 1864 len 32768
17154 005/300 Read block 568 len 40960
17155 006/300 Write block 4291 len 12288 data: 4 4 3
17156 007/300 Write block 4255 len 36864 data: 5 4 4 5 5 4 6 6 5
17157 008/300 Read block 2241 len 24576
17158 009/300 Read block 1641 len 12288
17159 010/300 Write block 1065 len 4096 data: 2
17160 011/300 Read block 4973 len 4096
17161 012/300 Write block 1522 len 16384 data: 3 4 5 4
17162 013/300 Read block 3032 len 24576
17163 014/300 Read block 2259 len 24576
17164 015/300 Write block 2382 len 4096 data: 3
17165 016/300 Write block 1566 len 36864 data: 6 7 7 7 6 6 4 3 3
17166 017/300 Write block 1375 len 20480 data: 2 2 2 2 2
17167 018/300 Read block 3497 len 4096
17168 019/300 Write block 287 len 24576 data: 4 5 5 6 5 5
17169 020/300 Write block 3071 len 4096 data: 6
17170 021/300 Read block 2004 len 16384
17171 022/300 Read block 2977 len 8192
17172 023/300 Flush
17173 024/300 Write block 2617 len 12288 data: 2 3 4
17174 025/300 Write block 3323 len 32768 data: 4 4 4 5 5 5 3 3
17175 026/300 Write block 3744 len 8192 data: 5 4
17176 027/300 Write block 1481 len 12288 data: 3 3 3
17177 028/300 Write block 219 len 40960 data: 3 4 4 3 3 2 2 2 2 2
17178 029/300 Read block 2045 len 28672
17179 030/300 Write block 4515 len 16384 data: 4 4 3 4
17180 031/300 Read block 368 len 36864
17181 032/300 Read block 2844 len 28672
17182 033/300 Write block 2276 len 28672 data: 3 3 3 3 3 4 4
17183 034/300 Read block 638 len 36864
17184 035/300 Read block 2727 len 4096
17185 036/300 Read block 1192 len 4096
17186 037/300 Read block 708 len 28672
17187 038/300 Write block 4530 len 8192 data: 2 2
17188 039/300 Read block 2634 len 20480
17189 040/300 Read block 2473 len 28672
17190 041/300 Write block 352 len 20480 data: 7 6 5 4 4
17191 042/300 Write block 126 len 36864 data: 5 5 6 5 5 5 5 4 5
17192 043/300 Flush
17193 044/300 Write block 2827 len 4096 data: 3
17194 045/300 Write block 2506 len 20480 data: 4 3 3 4 4
17195 046/300 Read block 4699 len 28672
17196 047/300 Read block 4121 len 20480
17197 048/300 Write block 3244 len 16384 data: 3 3 3 3
17198 049/300 Read block 1827 len 8192
17199 050/300 Write block 2921 len 28672 data: 6 6 6 7 6 5 4
17200 051/300 Read block 2987 len 24576
17201 052/300 Write block 1157 len 8192 data: 5 5
17202 053/300 Write block 2202 len 8192 data: 4 3
17203 054/300 Read block 2626 len 24576
17204 055/300 Read block 3230 len 4096
17205 056/300 Read block 4461 len 12288
17206 057/300 Write block 1935 len 20480 data: 2 3 3 3 3
17207 058/300 Read block 1413 len 36864
17208 059/300 Write block 3764 len 12288 data: 3 3 2
17209 060/300 Write block 3212 len 36864 data: 2 2 2 2 2 2 2 3 3
17210 061/300 Read block 3561 len 32768
17211 062/300 Read block 2404 len 32768
17212 063/300 Write block 33 len 12288 data: 3 3 3
17213 064/300 Write block 4667 len 40960 data: 3 3 4 4 6 6 5 5 4 4
17214 065/300 Write block 2594 len 4096 data: 2
17215 066/300 Read block 912 len 8192
17216 067/300 Read block 974 len 8192
17217 068/300 Write block 2916 len 16384 data: 4 4 5 3
17218 069/300 Read block 1330 len 28672
17219 070/300 Read block 693 len 8192
17220 071/300 Write block 3782 len 4096 data: 7
17221 072/300 Read block 2210 len 24576
17222 073/300 Write block 3918 len 36864 data: 5 6 6 7 8 6 6 6 5
17223 074/300 Write block 2181 len 20480 data: 4 4 4 4 4
17224 075/300 Read block 1026 len 40960
17225 076/300 Write block 66 len 40960 data: 2 2 5 5 6 6 7 7 7 7
17226 077/300 Read block 4293 len 20480
17227 078/300 Write block 295 len 32768 data: 3 2 2 2 2 2 2 3
17228 079/300 Read block 2796 len 32768
17229 080/300 Read block 1101 len 24576
17230 081/300 Read block 4440 len 32768
17231 082/300 Write block 1548 len 28672 data: 4 4 4 3 3 4 2
17232 083/300 Flush
17233 084/300 Write block 942 len 16384 data: 4 4 4 4
17234 085/300 Read block 4214 len 24576
17235 086/300 Write block 2633 len 20480 data: 3 5 4 3 3
17236 087/300 Read block 4238 len 28672
17237 088/300 Write block 270 len 28672 data: 4 5 4 4 5 5 7
17238 089/300 Read block 2789 len 20480
17239 090/300 Flush
17240 091/300 Flush
17241 092/300 Read block 2595 len 24576
17242 093/300 Write block 1544 len 20480 data: 4 4 5 5 5
17243 094/300 Flush
17244 095/300 Read block 1433 len 12288
17245 096/300 Write block 2568 len 12288 data: 2 2 2
17246 097/300 Write block 3309 len 28672 data: 3 3 3 3 3 2 3
17247 098/300 Write block 22 len 28672 data: 5 5 4 4 4 2 3
17248 099/300 Read block 3991 len 24576
17249 100/300 Read block 2486 len 24576
17250 101/300 Read block 225 len 12288
17251 102/300 Read block 2054 len 24576
17252 103/300 Read block 2023 len 28672
17253 104/300 Write block 2006 len 40960 data: 5 5 6 5 4 3 3 4 4 4
17254 105/300 Write block 1877 len 36864 data: 3 2 2 2 2 2 2 3 3
17255 106/300 Write block 723 len 32768 data: 3 3 3 4 4 4 5 4
17256 107/300 Write block 3080 len 32768 data: 6 6 6 7 8 7 7 6
17257 108/300 Read block 4291 len 24576
17258 109/300 Flush
17259 110/300 Flush
17260 111/300 Read block 1530 len 32768
17261 112/300 Write block 2499 len 8192 data: 4 3
17262 113/300 Read block 4718 len 24576
17263 114/300 Flush
17264 115/300 Read block 4497 len 40960
17265 116/300 Read block 2242 len 40960
17266 117/300 Read block 4893 len 12288
17267 118/300 Read block 50 len 36864
17268 119/300 Write block 146 len 32768 data: 4 4 5 5 4 3 3 3
17269 120/300 Flush
17270 121/300 Read block 2875 len 8192
17271 122/300 Read block 3654 len 12288
17272 123/300 Read block 3258 len 40960
17273 124/300 Read block 4426 len 16384
17274 125/300 Write block 4786 len 4096 data: 4
17275 126/300 Read block 956 len 12288
17276 127/300 Flush
17277 128/300 Read block 3436 len 40960
17278 129/300 Read block 3378 len 4096
17279 130/300 Write block 1138 len 40960 data: 3 3 3 3 3 3 3 3 4 4
17280 131/300 Flush
17281 132/300 Write block 503 len 36864 data: 3 3 3 3 2 2 3 3 3
17282 133/300 Read block 2825 len 20480
17283 134/300 Flush
17284 135/300 Read block 987 len 4096
17285 136/300 Read block 3413 len 24576
17286 137/300 Write block 2512 len 4096 data: 6
17287 138/300 Read block 1111 len 12288
17288 139/300 Read block 219 len 16384
17289 140/300 Write block 3844 len 4096 data: 2
17290 141/300 Write block 4857 len 12288 data: 4 3 2
17291 142/300 Write block 3395 len 16384 data: 2 2 2 3
17292 143/300 Read block 3676 len 20480
17293 144/300 Read block 3205 len 40960
17294 145/300 Flush
17295 146/300 Flush
17296 147/300 Read block 3920 len 36864
17297 148/300 Write block 473 len 24576 data: 3 3 4 3 3 2
17298 149/300 Read block 3860 len 24576
17299 150/300 Read block 271 len 4096
17300 151/300 Read block 3690 len 4096
17301 152/300 Read block 1769 len 24576
17302 153/300 Read block 4989 len 24576
17303 154/300 Read block 75 len 24576
17304 155/300 Read block 554 len 20480
17305 156/300 Read block 3122 len 4096
17306 157/300 Write block 1268 len 28672 data: 6 5 4 4 4 2 2
17307 158/300 Write block 3147 len 8192 data: 6 4
17308 159/300 Read block 1375 len 40960
17309 160/300 Write block 4209 len 36864 data: 6 6 7 7 6 5 6 4 4
17310 161/300 Read block 1586 len 16384
17311 162/300 Read block 408 len 28672
17312 163/300 Write block 103 len 12288 data: 2 2 2
17313 164/300 Read block 687 len 40960
17314 165/300 Read block 1263 len 32768
17315 166/300 Flush
17316 167/300 Read block 3303 len 4096
17317 168/300 Read block 1408 len 8192
17318 169/300 Read block 2962 len 16384
17319 170/300 Flush
17320 171/300 Flush
17321 172/300 Write block 3230 len 28672 data: 2 3 4 5 4 4 5
17322 173/300 Flush
17323 174/300 Flush
17324 175/300 Write block 3018 len 24576 data: 3 3 3 2 2 2
17325 176/300 Write block 4327 len 36864 data: 4 6 5 6 6 6 6 6 5
17326 177/300 Flush
17327 178/300 Write block 399 len 28672 data: 4 4 4 4 4 4 3
17328 179/300 Write block 4273 len 24576 data: 4 6 5 5 7 7
17329 180/300 Flush
17330 181/300 Write block 2863 len 8192 data: 3 4
17331 182/300 Write block 1062 len 4096 data: 2
17332 183/300 Write block 4831 len 32768 data: 4 5 5 5 5 5 6 6
17333 184/300 Read block 2670 len 12288
17334 185/300 Write block 3105 len 40960 data: 4 3 3 3 3 2 3 3 5 5
17335 186/300 Write block 1573 len 16384 data: 4 4 3 4
17336 187/300 Read block 762 len 24576
17337 188/300 Read block 2756 len 12288
17338 189/300 Write block 1677 len 4096 data: 3
17339 190/300 Write block 3386 len 8192 data: 5 5
17340 191/300 Read block 4768 len 16384
17341 192/300 Flush
17342 193/300 Read block 3408 len 28672
17343 194/300 Write block 4548 len 28672 data: 7 7 5 4 4 3 3
17344 195/300 Write block 3757 len 36864 data: 5 5 4 4 3 3 4 4 4
17345 196/300 Write block 1692 len 24576 data: 3 5 5 5 4 4
17346 197/300 Write block 3293 len 20480 data: 5 5 4 4 5
17347 198/300 Write block 3349 len 36864 data: 2 3 3 4 3 3 3 3 4
17348 199/300 Write block 933 len 8192 data: 5 4
17349 200/300 Write block 1392 len 36864 data: 2 2 2 2 2 4 3 3 3
17350 201/300 Write block 4227 len 20480 data: 3 3 4 3 2
17351 202/300 Read block 3850 len 28672
17352 203/300 Write block 4252 len 24576 data: 6 6 6 6 5 5
17353 204/300 Read block 2360 len 20480
17354 205/300 Write block 3807 len 28672 data: 3 4 4 4 5 6 4
17355 206/300 Read block 1644 len 36864
17356 207/300 Read block 1501 len 4096
17357 208/300 Read block 3677 len 20480
17358 209/300 Read block 108 len 12288
17359 210/300 Read block 2250 len 32768
17360 211/300 Flush
17361 212/300 Write block 683 len 4096 data: 3
17362 213/300 Write block 2429 len 36864 data: 3 3 3 4 3 3 3 4 5
17363 214/300 Write block 4117 len 36864 data: 2 2 2 3 3 5 5 5 6
17364 215/300 Write block 4530 len 32768 data: 3 3 2 3 3 3 3 3
17365 216/300 Write block 1915 len 16384 data: 4 4 4 3
17366 217/300 Read block 157 len 8192
17367 218/300 Read block 1065 len 24576
17368 219/300 Write block 3140 len 28672 data: 4 3 3 4 4 5 5
17369 220/300 Write block 2133 len 20480 data: 3 2 2 2 2
17370 221/300 Read block 3841 len 36864
17371 222/300 Write block 474 len 40960 data: 4 5 4 4 3 2 2 2 2 3
17372 223/300 Write block 3535 len 12288 data: 2 2 2
17373 224/300 Write block 484 len 36864 data: 3 3 3 3 6 5 5 6 4
17374 225/300 Read block 1568 len 40960
17375 226/300 Read block 2206 len 24576
17376 227/300 Read block 146 len 20480
17377 228/300 Flush
17378 229/300 Read block 4022 len 16384
17379 230/300 Read block 939 len 20480
17380 231/300 Write block 1576 len 24576 data: 5 3 3 3 3 3
17381 232/300 Read block 304 len 36864
17382 233/300 Read block 3684 len 12288
17383 234/300 Read block 4557 len 16384
17384 235/300 Read block 1555 len 40960
17385 236/300 Read block 4403 len 40960
17386 237/300 Read block 3428 len 8192
17387 238/300 Read block 3572 len 36864
17388 239/300 Read block 4558 len 24576
17389 240/300 Write block 2257 len 12288 data: 3 3 3
17390 241/300 Read block 2023 len 32768
17391 242/300 Flush
17392 243/300 Read block 4902 len 32768
17393 244/300 Read block 1226 len 4096
17394 245/300 Read block 4391 len 40960
17395 246/300 Read block 4740 len 8192
17396 247/300 Read block 4249 len 36864
17397 248/300 Flush
17398 249/300 Read block 3264 len 12288
17399 250/300 Flush
17400 251/300 Flush
17401 252/300 Flush
17402 253/300 Write block 4977 len 16384 data: 2 2 3 3
17403 254/300 Write block 162 len 36864 data: 5 4 4 5 6 5 5 4 6
17404 255/300 Flush
17405 256/300 Write block 1457 len 28672 data: 3 3 4 4 6 7 8
17406 257/300 Read block 4099 len 32768
17407 258/300 Read block 2711 len 20480
17408 259/300 Read block 4023 len 16384
17409 260/300 Read block 3412 len 24576
17410 261/300 Read block 3514 len 28672
17411 262/300 Write block 1360 len 32768 data: 4 5 5 4 4 4 4 4
17412 263/300 Read block 1270 len 8192
17413 264/300 Write block 2770 len 8192 data: 3 2
17414 265/300 Write block 2978 len 36864 data: 2 2 2 2 2 2 2 2 2
17415 266/300 Read block 1319 len 4096
17416 267/300 Read block 2977 len 32768
17417 268/300 Write block 2802 len 8192 data: 6 5
17418 269/300 Write block 4958 len 16384 data: 6 5 5 5
17419 270/300 Write block 131 len 36864 data: 6 6 5 6 5 4 4 3 3
17420 271/300 Read block 2947 len 36864
17421 272/300 Read block 332 len 40960
17422 273/300 Write block 1963 len 12288 data: 2 3 3
17423 274/300 Read block 4333 len 4096
17424 275/300 Read block 4806 len 8192
17425 276/300 Read block 3157 len 40960
17426 277/300 Write block 2501 len 4096 data: 2
17427 278/300 Write block 572 len 16384 data: 2 2 2 2
17428 279/300 Read block 1032 len 40960
17429 280/300 Write block 1176 len 4096 data: 2
17430 281/300 Read block 593 len 12288
17431 282/300 Write block 3384 len 8192 data: 4 5
17432 283/300 Read block 711 len 4096
17433 284/300 Read block 4514 len 40960
17434 285/300 Read block 1908 len 4096
17435 286/300 Read block 138 len 24576
17436 287/300 Read block 4561 len 28672
17437 288/300 Write block 339 len 12288 data: 4 4 4
17438 289/300 Read block 3818 len 8192
17439 290/300 Flush
17440 291/300 Flush
17441 292/300 Read block 3312 len 4096
17442 293/300 Read block 2545 len 16384
17443 294/300 Read block 196 len 12288
17444 295/300 Write block 1307 len 24576 data: 3 2 2 4 4 4
17445 296/300 Read block 4787 len 28672
17446 297/300 Flush
17447 298/300 Write block 629 len 40960 data: 5 5 4 4 3 4 4 4 3 4
17448 299/300 Write block 2749 len 8192 data: 3 3
17449 300/300 Read block 3399 len 28672
17450 301/300 Read block 4402 len 16384
17451 Main task has work to do, go find it
17452 got dsc Start(0)
17453 start 0
17454 Mar 07 18:16:24.343 INFO request completed, latency_us: 183, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4efb5d47-d1ed-4c46-84db-907c00aaaafc, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
17455 Replay: started 0, returned:Ok(())
17456 ----------------------------------------------------------------
17457 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17458 States: Offline Active Active
17459 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17460 5226 Acked Read 6 Sent Done Done false
17461 5227 Acked Read 9 Sent Done Done false
17462 5228 Acked Read 10 Sent Done Done false
17463 5229 Acked Read 8 Sent Done Done false
17464 5230 Acked Read 10 Sent Done Done false
17465 5231 Acked Write 3 Sent Done Done false
17466 5232 Acked Write 9 Sent Done Done false
17467 5233 Acked Read 6 Sent Done Done false
17468 5234 Acked Read 3 Sent Done Done false
17469 5235 Acked Write 1 Sent Done Done false
17470 5236 Acked Read 1 Sent Done Done false
17471 5237 Acked Write 4 Sent Done Done false
17472 5238 Acked Read 6 Sent Done Done false
17473 5239 Acked Read 6 Sent Done Done false
17474 5240 Acked Write 1 Sent Done Done false
17475 5241 Acked Write 9 Sent Done Done false
17476 5242 Acked Write 5 Sent Done Done false
17477 5243 Acked Read 1 Sent Done Done false
17478 5244 Acked Write 6 Sent Done Done false
17479 5245 Acked Write 1 Sent Done Done false
17480 5246 Acked Read 4 Sent Done Done false
17481 5247 Acked Read 2 Sent Done Done false
17482 5248 Acked Flush 0 Sent Done Done false
17483 5249 Acked Write 3 Sent Done Done false
17484 5250 Acked Write 8 Sent Done Done false
17485 5251 Acked Write 2 Sent Done Done false
17486 5252 Acked Write 3 Sent Done Done false
17487 5253 Acked Write 10 Sent Done Done false
17488 5254 Acked Read 7 Sent Done Done false
17489 5255 Acked Write 4 Sent Done Done false
17490 5256 Acked Read 9 Sent Done Done false
17491 5257 Acked Read 7 Sent Done Done false
17492 5258 Acked Write 7 Sent Done Done false
17493 5259 Acked Read 9 Sent Done Done false
17494 5260 Acked Read 1 Sent Done Done false
17495 5261 Acked Read 1 Sent Done Done false
17496 5262 Acked Read 7 Sent Done Done false
17497 5263 Acked Write 2 Sent Done Done false
17498 5264 Acked Read 5 Sent Done Done false
17499 5265 Acked Read 7 Sent Done Done false
17500 5266 Acked Write 5 Sent Done Done false
17501 5267 Acked Write 9 Sent Done Done false
17502 5268 Acked Flush 0 Sent Done Done false
17503 5269 Acked Write 1 Sent Done Done false
17504 5270 Acked Write 5 Sent Done Done false
17505 5271 Acked Read 7 Sent Done Done false
17506 5272 Acked Read 5 Sent Done Done false
17507 5273 Acked Write 4 Sent Done Done false
17508 5274 Acked Read 2 Sent Done Done false
17509 5275 Acked Write 7 Sent Done Done false
17510 5276 Acked Read 6 Sent Done Done false
17511 5277 Acked Write 2 Sent Done Done false
17512 5278 Acked Write 2 Sent Done Done false
17513 5279 Acked Read 6 Sent Done Done false
17514 5280 Acked Read 1 Sent Done Done false
17515 5281 Acked Read 3 Sent Done Done false
17516 5282 Acked Write 5 Sent Done Done false
17517 5283 Acked Read 9 Sent Done Done false
17518 5284 Acked Write 3 Sent Done Done false
17519 5285 Acked Write 9 Sent Done Done false
17520 5286 Acked Read 8 Sent Done Done false
17521 5287 Acked Read 8 Sent Done Done false
17522 5288 Acked Write 3 Sent Done Done false
17523 5289 Acked Write 10 Sent Done Done false
17524 5290 Acked Write 1 Sent Done Done false
17525 5291 Acked Read 2 Sent Done Done false
17526 5292 Acked Read 2 Sent Done Done false
17527 5293 Acked Write 4 Sent Done Done false
17528 5294 Acked Read 7 Sent Done Done false
17529 5295 Acked Read 2 Sent Done Done false
17530 5296 Acked Write 1 Sent Done Done false
17531 5297 Acked Read 6 Sent Done Done false
17532 5298 Acked Write 9 Sent Done Done false
17533 5299 Acked Write 5 Sent Done Done false
17534 5300 Acked Read 10 Sent Done Done false
17535 5301 Acked Write 10 Sent Done Done false
17536 5302 Acked Read 5 Sent Done Done false
17537 5303 Acked Write 8 Sent Done Done false
17538 5304 Acked Read 8 Sent Done Done false
17539 5305 Acked Read 6 Sent Done Done false
17540 5306 Acked Read 8 Sent Done Done false
17541 5307 Acked Write 7 Sent Done Done false
17542 5308 Acked Flush 0 Sent Done Done false
17543 5309 Acked Write 4 Sent Done Done false
17544 5310 Acked Read 6 Sent Done Done false
17545 5311 Acked Write 5 Sent Done Done false
17546 5312 Acked Read 7 Sent Done Done false
17547 5313 Acked Write 7 Sent Done Done false
17548 5314 Acked Read 5 Sent Done Done false
17549 5315 Acked Flush 0 Sent Done Done false
17550 5316 Acked Flush 0 Sent Done Done false
17551 5317 Acked Read 6 Sent Done Done false
17552 5318 Acked Write 5 Sent Done Done false
17553 5319 Acked Flush 0 Sent Done Done false
17554 5320 Acked Read 3 Sent Done Done false
17555 5321 Acked Write 3 Sent Done Done false
17556 5322 Acked Write 7 Sent Done Done false
17557 5323 Acked Write 7 Sent Done Done false
17558 5324 Acked Read 6 Sent Done Done false
17559 5325 Acked Read 6 Sent Done Done false
17560 5326 Acked Read 3 Sent Done Done false
17561 5327 Acked Read 6 Sent Done Done false
17562 5328 Acked Read 7 Sent Done Done false
17563 5329 Acked Write 10 Sent Done Done false
17564 5330 Acked Write 9 Sent Done Done false
17565 5331 Acked Write 8 Sent Done Done false
17566 5332 Acked Write 8 Sent Done Done false
17567 5333 Acked Read 6 Sent Done Done false
17568 5334 Acked Flush 0 Sent Done Done false
17569 5335 Acked Flush 0 Sent Done Done false
17570 5336 Acked Read 8 Sent Done Done false
17571 5337 Acked Write 2 Sent Done Done false
17572 5338 Acked Read 6 Sent Done Done false
17573 5339 Acked Flush 0 Sent Done Done false
17574 5340 Acked Read 10 Sent Done Done false
17575 5341 Acked Read 10 Sent Done Done false
17576 5342 Acked Read 3 Sent Done Done false
17577 5343 Acked Read 9 Sent Done Done false
17578 5344 Acked Write 8 Sent Done Done false
17579 5345 Acked Flush 0 Sent Done Done false
17580 5346 Acked Read 2 Sent Done Done false
17581 5347 Acked Read 3 Sent Done Done false
17582 5348 Acked Read 10 Sent Done Done false
17583 5349 Acked Read 4 Sent Done Done false
17584 5350 Acked Write 1 Sent Done Done false
17585 5351 Acked Read 3 Sent Done Done false
17586 5352 Acked Flush 0 Sent Done Done false
17587 5353 Acked Read 10 Sent Done Done false
17588 5354 Acked Read 1 Sent Done Done false
17589 5355 Acked Write 10 Sent Done Done false
17590 5356 Acked Flush 0 Sent Done Done false
17591 5357 Acked Write 9 Sent Done Done false
17592 5358 Acked Read 5 Sent Done Done false
17593 5359 Acked Flush 0 Sent Done Done false
17594 5360 Acked Read 1 Sent Done Done false
17595 5361 Acked Read 6 Sent Done Done false
17596 5362 Acked Write 1 Sent Done Done false
17597 5363 Acked Read 3 Sent Done Done false
17598 5364 Acked Read 4 Sent Done Done false
17599 5365 Acked Write 1 Sent Done Done false
17600 5366 Acked Write 3 Sent Done Done false
17601 5367 Acked Write 4 Sent Done Done false
17602 5368 Acked Read 5 Sent Done Done false
17603 5369 Acked Read 10 Sent Done Done false
17604 5370 Acked Flush 0 Sent Done Done false
17605 5371 Acked Flush 0 Sent Done Done false
17606 5372 Acked Read 9 Sent Done Done false
17607 5373 Acked Write 6 Sent Done Done false
17608 5374 Acked Read 6 Sent Done Done false
17609 5375 Acked Read 1 Sent Done Done false
17610 5376 Acked Read 1 Sent Done Done false
17611 5377 Acked Read 6 Sent Done Done false
17612 5378 Acked Read 6 Sent Done Done false
17613 5379 Acked Read 6 Sent Done Done false
17614 5380 Acked Read 5 Sent Done Done false
17615 5381 Acked Read 1 Sent Done Done false
17616 5382 Acked Write 7 Sent Done Done false
17617 5383 Acked Write 2 Sent Done Done false
17618 5384 Acked Read 10 Sent Done Done false
17619 5385 Acked Write 9 Sent Done Done false
17620 5386 Acked Read 4 Sent Done Done false
17621 5387 Acked Read 7 Sent Done Done false
17622 5388 Acked Write 3 Sent Done Done false
17623 5389 Acked Read 10 Sent Done Done false
17624 5390 Acked Read 8 Sent Done Done false
17625 5391 Acked Flush 0 Sent Done Done false
17626 5392 Acked Read 1 Sent Done Done false
17627 5393 Acked Read 2 Sent Done Done false
17628 5394 Acked Read 4 Sent Done Done false
17629 5395 Acked Flush 0 Sent Done Done false
17630 5396 Acked Flush 0 Sent Done Done false
17631 5397 Acked Write 7 Sent Done Done false
17632 5398 Acked Flush 0 Sent Done Done false
17633 5399 Acked Flush 0 Sent Done Done false
17634 5400 Acked Write 6 Sent Done Done false
17635 5401 Acked Write 9 Sent Done Done false
17636 5402 Acked Flush 0 Sent Done Done false
17637 5403 Acked Write 7 Sent Done Done false
17638 5404 Acked Write 6 Sent Done Done false
17639 5405 Acked Flush 0 Sent Done Done false
17640 5406 Acked Write 2 Sent Done Done false
17641 5407 Acked Write 1 Sent Done Done false
17642 5408 Acked Write 8 Sent Done Done false
17643 5409 Acked Read 3 Sent Done Done false
17644 5410 Acked Write 10 Sent Done Done false
17645 5411 Acked Write 4 Sent Done Done false
17646 5412 Acked Read 6 Sent Done Done false
17647 5413 Acked Read 3 Sent Done Done false
17648 5414 Acked Write 1 Sent Done Done false
17649 5415 Acked Write 2 Sent Done Done false
17650 5416 Acked Read 4 Sent Done Done false
17651 5417 Acked Flush 0 Sent Done Done false
17652 5418 Acked Read 7 Sent Done Done false
17653 5419 Acked Write 7 Sent Done Done false
17654 5420 Acked Write 9 Sent Done Done false
17655 5421 Acked Write 6 Sent Done Done false
17656 5422 Acked Write 5 Sent Done Done false
17657 5423 Acked Write 9 Sent Done Done false
17658 5424 Acked Write 2 Sent Done Done false
17659 5425 Acked Write 9 Sent Done Done false
17660 5426 Acked Write 5 Sent Done Done false
17661 5427 Acked Read 7 Sent Done Done false
17662 5428 Acked Write 6 Sent Done Done false
17663 5429 Acked Read 5 Sent Done Done false
17664 5430 Acked Write 7 Sent Done Done false
17665 5431 Acked Read 9 Sent Done Done false
17666 5432 Acked Read 1 Sent Done Done false
17667 5433 Acked Read 5 Sent Done Done false
17668 5434 Acked Read 3 Sent Done Done false
17669 5435 Acked Read 8 Sent Done Done false
17670 5436 Acked Flush 0 Sent Done Done false
17671 5437 Acked Write 1 Sent Done Done false
17672 5438 Acked Write 9 Sent Done Done false
17673 5439 Acked Write 9 Sent Done Done false
17674 5440 Acked Write 8 Sent Done Done false
17675 5441 Acked Write 4 Sent Done Done false
17676 5442 Acked Read 2 Sent Done Done false
17677 5443 Acked Read 6 Sent Done Done false
17678 5444 Acked Write 7 Sent Done Done false
17679 5445 Acked Write 5 Sent Done Done false
17680 5446 Acked Read 9 Sent Done Done false
17681 5447 Acked Write 10 Sent Done Done false
17682 5448 Acked Write 3 Sent Done Done false
17683 5449 Acked Write 9 Sent Done Done false
17684 5450 Acked Read 10 Sent Done Done false
17685 5451 Acked Read 6 Sent Done Done false
17686 5452 Acked Read 5 Sent Done Done false
17687 5453 Acked Flush 0 Sent Done Done false
17688 5454 Acked Read 4 Sent Done Done false
17689 5455 Acked Read 5 Sent Done Done false
17690 5456 Acked Write 6 Sent Done Done false
17691 5457 Acked Read 9 Sent Done Done false
17692 5458 Acked Read 3 Sent Done Done false
17693 5459 Acked Read 4 Sent Done Done false
17694 5460 Acked Read 10 Sent Done Done false
17695 5461 Acked Read 10 Sent Done Done false
17696 5462 Acked Read 2 Sent Done Done false
17697 5463 Acked Read 9 Sent Done Done false
17698 5464 Acked Read 6 Sent Done Done false
17699 5465 Acked Write 3 Sent Done Done false
17700 5466 Acked Read 8 Sent Done Done false
17701 5467 Acked Flush 0 Sent Done Done false
17702 5468 Acked Read 8 Sent Done Done false
17703 5469 Acked Read 1 Sent Done Done false
17704 5470 Acked Read 10 Sent Done Done false
17705 5471 Acked Read 2 Sent Done Done false
17706 5472 Acked Read 9 Sent Done Done false
17707 5473 Acked Flush 0 Sent Done Done false
17708 5474 Acked Read 3 Sent Done Done false
17709 5475 Acked Flush 0 Sent Done Done false
17710 5476 Acked Flush 0 Sent Done Done false
17711 5477 Acked Flush 0 Sent Done Done false
17712 5478 Acked Write 4 Sent Done Done false
17713 5479 Acked Write 9 Sent Done Done false
17714 5480 Acked Flush 0 Sent Done Done false
17715 5481 Acked Write 7 Sent Done Done false
17716 5482 Acked Read 8 Sent Done Done false
17717 5483 Acked Read 5 Sent Done Done false
17718 5484 Acked Read 4 Sent Done Done false
17719 5485 Acked Read 6 Sent Done Done false
17720 5486 Acked Read 7 Sent Done Done false
17721 5487 Acked Write 8 Sent Done Done false
17722 5488 Acked Read 2 Sent Done Done false
17723 5489 Acked Write 2 Sent Done Done false
17724 5490 Acked Write 9 Sent Done Done false
17725 5491 Acked Read 1 Sent Done Done false
17726 5492 Acked Read 8 Sent Done Done false
17727 5493 Acked Write 2 Sent Done Done false
17728 5494 Acked Write 4 Sent Done Done false
17729 5495 Acked Write 9 Sent Done Done false
17730 5496 Acked Read 9 Sent Done Done false
17731 5497 Acked Read 10 Sent Done Done false
17732 5498 Acked Write 3 Sent Done Done false
17733 5499 Acked Read 1 Sent Done Done false
17734 5500 Acked Read 2 Sent Done Done false
17735 5501 Acked Read 10 Sent Done Done false
17736 5502 Acked Write 1 Sent Done Done false
17737 5503 Acked Write 4 Sent Done Done false
17738 5504 Acked Read 10 Sent Done Done false
17739 5505 Acked Write 1 Sent Done Done false
17740 5506 Acked Read 3 Sent Done Done false
17741 5507 Acked Write 2 Sent Done Done false
17742 5508 Acked Read 1 Sent Done Done false
17743 5509 Acked Read 10 Sent Done Done false
17744 5510 Acked Read 1 Sent Done Done false
17745 5511 Acked Read 6 Sent Done Done false
17746 5512 Acked Read 7 Sent Done Done false
17747 5513 Acked Write 3 Sent Done Done false
17748 5514 Acked Read 2 Sent Done Done false
17749 5515 Acked Flush 0 Sent Done Done false
17750 5516 Acked Flush 0 Sent Done Done false
17751 5517 Acked Read 1 Sent Done Done false
17752 5518 Acked Read 4 Sent Done Done false
17753 5519 Acked Read 3 Sent Done Done false
17754 5520 Acked Write 6 Sent Done Done false
17755 5521 Acked Read 7 Sent Done Done false
17756 5522 Acked Flush 0 Sent Done Done false
17757 5523 Acked Write 10 Sent Done Done false
17758 5524 Acked Write 2 Sent Done Done false
17759 5525 Acked Read 7 Sent Done Done false
17760 5526 Acked Read 4 Sent Done Done false
17761 STATES DS:0 DS:1 DS:2 TOTAL
17762 Sent 301 0 0 301
17763 Done 0 301 301 602
17764 Skipped 0 0 0 0
17765 Error 0 0 0 0
17766 Last Flush: Some(JobId(5225)) Some(JobId(5522)) Some(JobId(5522))
17767 Downstairs last five completed: 5225 5224 5223 5222 5221
17768 Upstairs last five completed: 5526 5525 5524 5523 5522
17769 CLIENT: Up:0 ds:301 act:2
177702025-03-07T18:16:25.330ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
177712025-03-07T18:16:27.334ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
177722025-03-07T18:16:28.335ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
17773 ----------------------------------------------------------------
17774 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17775 States: Offline Active Active
17776 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17777 5226 Acked Read 6 Sent Done Done false
17778 5227 Acked Read 9 Sent Done Done false
17779 5228 Acked Read 10 Sent Done Done false
17780 5229 Acked Read 8 Sent Done Done false
17781 5230 Acked Read 10 Sent Done Done false
17782 5231 Acked Write 3 Sent Done Done false
17783 5232 Acked Write 9 Sent Done Done false
17784 5233 Acked Read 6 Sent Done Done false
17785 5234 Acked Read 3 Sent Done Done false
17786 5235 Acked Write 1 Sent Done Done false
17787 5236 Acked Read 1 Sent Done Done false
17788 5237 Acked Write 4 Sent Done Done false
17789 5238 Acked Read 6 Sent Done Done false
17790 5239 Acked Read 6 Sent Done Done false
17791 5240 Acked Write 1 Sent Done Done false
17792 5241 Acked Write 9 Sent Done Done false
17793 5242 Acked Write 5 Sent Done Done false
17794 5243 Acked Read 1 Sent Done Done false
17795 5244 Acked Write 6 Sent Done Done false
17796 5245 Acked Write 1 Sent Done Done false
17797 5246 Acked Read 4 Sent Done Done false
17798 5247 Acked Read 2 Sent Done Done false
17799 5248 Acked Flush 0 Sent Done Done false
17800 5249 Acked Write 3 Sent Done Done false
17801 5250 Acked Write 8 Sent Done Done false
17802 5251 Acked Write 2 Sent Done Done false
17803 5252 Acked Write 3 Sent Done Done false
17804 5253 Acked Write 10 Sent Done Done false
17805 5254 Acked Read 7 Sent Done Done false
17806 5255 Acked Write 4 Sent Done Done false
17807 5256 Acked Read 9 Sent Done Done false
17808 5257 Acked Read 7 Sent Done Done false
17809 5258 Acked Write 7 Sent Done Done false
17810 5259 Acked Read 9 Sent Done Done false
17811 5260 Acked Read 1 Sent Done Done false
17812 5261 Acked Read 1 Sent Done Done false
17813 5262 Acked Read 7 Sent Done Done false
17814 5263 Acked Write 2 Sent Done Done false
17815 5264 Acked Read 5 Sent Done Done false
17816 5265 Acked Read 7 Sent Done Done false
17817 5266 Acked Write 5 Sent Done Done false
17818 5267 Acked Write 9 Sent Done Done false
17819 5268 Acked Flush 0 Sent Done Done false
17820 5269 Acked Write 1 Sent Done Done false
17821 5270 Acked Write 5 Sent Done Done false
17822 5271 Acked Read 7 Sent Done Done false
17823 5272 Acked Read 5 Sent Done Done false
17824 5273 Acked Write 4 Sent Done Done false
17825 5274 Acked Read 2 Sent Done Done false
17826 5275 Acked Write 7 Sent Done Done false
17827 5276 Acked Read 6 Sent Done Done false
17828 5277 Acked Write 2 Sent Done Done false
17829 5278 Acked Write 2 Sent Done Done false
17830 5279 Acked Read 6 Sent Done Done false
17831 5280 Acked Read 1 Sent Done Done false
17832 5281 Acked Read 3 Sent Done Done false
17833 5282 Acked Write 5 Sent Done Done false
17834 5283 Acked Read 9 Sent Done Done false
17835 5284 Acked Write 3 Sent Done Done false
17836 5285 Acked Write 9 Sent Done Done false
17837 5286 Acked Read 8 Sent Done Done false
17838 5287 Acked Read 8 Sent Done Done false
17839 5288 Acked Write 3 Sent Done Done false
17840 5289 Acked Write 10 Sent Done Done false
17841 5290 Acked Write 1 Sent Done Done false
17842 5291 Acked Read 2 Sent Done Done false
17843 5292 Acked Read 2 Sent Done Done false
17844 5293 Acked Write 4 Sent Done Done false
17845 5294 Acked Read 7 Sent Done Done false
17846 5295 Acked Read 2 Sent Done Done false
17847 5296 Acked Write 1 Sent Done Done false
17848 5297 Acked Read 6 Sent Done Done false
17849 5298 Acked Write 9 Sent Done Done false
17850 5299 Acked Write 5 Sent Done Done false
17851 5300 Acked Read 10 Sent Done Done false
17852 5301 Acked Write 10 Sent Done Done false
17853 5302 Acked Read 5 Sent Done Done false
17854 5303 Acked Write 8 Sent Done Done false
17855 5304 Acked Read 8 Sent Done Done false
17856 5305 Acked Read 6 Sent Done Done false
17857 5306 Acked Read 8 Sent Done Done false
17858 5307 Acked Write 7 Sent Done Done false
17859 5308 Acked Flush 0 Sent Done Done false
17860 5309 Acked Write 4 Sent Done Done false
17861 5310 Acked Read 6 Sent Done Done false
17862 5311 Acked Write 5 Sent Done Done false
17863 5312 Acked Read 7 Sent Done Done false
17864 5313 Acked Write 7 Sent Done Done false
17865 5314 Acked Read 5 Sent Done Done false
17866 5315 Acked Flush 0 Sent Done Done false
17867 5316 Acked Flush 0 Sent Done Done false
17868 5317 Acked Read 6 Sent Done Done false
17869 5318 Acked Write 5 Sent Done Done false
17870 5319 Acked Flush 0 Sent Done Done false
17871 5320 Acked Read 3 Sent Done Done false
17872 5321 Acked Write 3 Sent Done Done false
17873 5322 Acked Write 7 Sent Done Done false
17874 5323 Acked Write 7 Sent Done Done false
17875 5324 Acked Read 6 Sent Done Done false
17876 5325 Acked Read 6 Sent Done Done false
17877 5326 Acked Read 3 Sent Done Done false
17878 5327 Acked Read 6 Sent Done Done false
17879 5328 Acked Read 7 Sent Done Done false
17880 5329 Acked Write 10 Sent Done Done false
17881 5330 Acked Write 9 Sent Done Done false
17882 5331 Acked Write 8 Sent Done Done false
17883 5332 Acked Write 8 Sent Done Done false
17884 5333 Acked Read 6 Sent Done Done false
17885 5334 Acked Flush 0 Sent Done Done false
17886 5335 Acked Flush 0 Sent Done Done false
17887 5336 Acked Read 8 Sent Done Done false
17888 5337 Acked Write 2 Sent Done Done false
17889 5338 Acked Read 6 Sent Done Done false
17890 5339 Acked Flush 0 Sent Done Done false
17891 5340 Acked Read 10 Sent Done Done false
17892 5341 Acked Read 10 Sent Done Done false
17893 5342 Acked Read 3 Sent Done Done false
17894 5343 Acked Read 9 Sent Done Done false
17895 5344 Acked Write 8 Sent Done Done false
17896 5345 Acked Flush 0 Sent Done Done false
17897 5346 Acked Read 2 Sent Done Done false
17898 5347 Acked Read 3 Sent Done Done false
17899 5348 Acked Read 10 Sent Done Done false
17900 5349 Acked Read 4 Sent Done Done false
17901 5350 Acked Write 1 Sent Done Done false
17902 5351 Acked Read 3 Sent Done Done false
17903 5352 Acked Flush 0 Sent Done Done false
17904 5353 Acked Read 10 Sent Done Done false
17905 5354 Acked Read 1 Sent Done Done false
17906 5355 Acked Write 10 Sent Done Done false
17907 5356 Acked Flush 0 Sent Done Done false
17908 5357 Acked Write 9 Sent Done Done false
17909 5358 Acked Read 5 Sent Done Done false
17910 5359 Acked Flush 0 Sent Done Done false
17911 5360 Acked Read 1 Sent Done Done false
17912 5361 Acked Read 6 Sent Done Done false
17913 5362 Acked Write 1 Sent Done Done false
17914 5363 Acked Read 3 Sent Done Done false
17915 5364 Acked Read 4 Sent Done Done false
17916 5365 Acked Write 1 Sent Done Done false
17917 5366 Acked Write 3 Sent Done Done false
17918 5367 Acked Write 4 Sent Done Done false
17919 5368 Acked Read 5 Sent Done Done false
17920 5369 Acked Read 10 Sent Done Done false
17921 5370 Acked Flush 0 Sent Done Done false
17922 5371 Acked Flush 0 Sent Done Done false
17923 5372 Acked Read 9 Sent Done Done false
17924 5373 Acked Write 6 Sent Done Done false
17925 5374 Acked Read 6 Sent Done Done false
17926 5375 Acked Read 1 Sent Done Done false
17927 5376 Acked Read 1 Sent Done Done false
17928 5377 Acked Read 6 Sent Done Done false
17929 5378 Acked Read 6 Sent Done Done false
17930 5379 Acked Read 6 Sent Done Done false
17931 5380 Acked Read 5 Sent Done Done false
17932 5381 Acked Read 1 Sent Done Done false
17933 5382 Acked Write 7 Sent Done Done false
17934 5383 Acked Write 2 Sent Done Done false
17935 5384 Acked Read 10 Sent Done Done false
17936 5385 Acked Write 9 Sent Done Done false
17937 5386 Acked Read 4 Sent Done Done false
17938 5387 Acked Read 7 Sent Done Done false
17939 5388 Acked Write 3 Sent Done Done false
17940 5389 Acked Read 10 Sent Done Done false
17941 5390 Acked Read 8 Sent Done Done false
17942 5391 Acked Flush 0 Sent Done Done false
17943 5392 Acked Read 1 Sent Done Done false
17944 5393 Acked Read 2 Sent Done Done false
17945 5394 Acked Read 4 Sent Done Done false
17946 5395 Acked Flush 0 Sent Done Done false
17947 5396 Acked Flush 0 Sent Done Done false
17948 5397 Acked Write 7 Sent Done Done false
17949 5398 Acked Flush 0 Sent Done Done false
17950 5399 Acked Flush 0 Sent Done Done false
17951 5400 Acked Write 6 Sent Done Done false
17952 5401 Acked Write 9 Sent Done Done false
17953 5402 Acked Flush 0 Sent Done Done false
17954 5403 Acked Write 7 Sent Done Done false
17955 5404 Acked Write 6 Sent Done Done false
17956 5405 Acked Flush 0 Sent Done Done false
17957 5406 Acked Write 2 Sent Done Done false
17958 5407 Acked Write 1 Sent Done Done false
17959 5408 Acked Write 8 Sent Done Done false
17960 5409 Acked Read 3 Sent Done Done false
17961 5410 Acked Write 10 Sent Done Done false
17962 5411 Acked Write 4 Sent Done Done false
17963 5412 Acked Read 6 Sent Done Done false
17964 5413 Acked Read 3 Sent Done Done false
17965 5414 Acked Write 1 Sent Done Done false
17966 5415 Acked Write 2 Sent Done Done false
17967 5416 Acked Read 4 Sent Done Done false
17968 5417 Acked Flush 0 Sent Done Done false
17969 5418 Acked Read 7 Sent Done Done false
17970 5419 Acked Write 7 Sent Done Done false
17971 5420 Acked Write 9 Sent Done Done false
17972 5421 Acked Write 6 Sent Done Done false
17973 5422 Acked Write 5 Sent Done Done false
17974 5423 Acked Write 9 Sent Done Done false
17975 5424 Acked Write 2 Sent Done Done false
17976 5425 Acked Write 9 Sent Done Done false
17977 5426 Acked Write 5 Sent Done Done false
17978 5427 Acked Read 7 Sent Done Done false
17979 5428 Acked Write 6 Sent Done Done false
17980 5429 Acked Read 5 Sent Done Done false
17981 5430 Acked Write 7 Sent Done Done false
17982 5431 Acked Read 9 Sent Done Done false
17983 5432 Acked Read 1 Sent Done Done false
17984 5433 Acked Read 5 Sent Done Done false
17985 5434 Acked Read 3 Sent Done Done false
17986 5435 Acked Read 8 Sent Done Done false
17987 5436 Acked Flush 0 Sent Done Done false
17988 5437 Acked Write 1 Sent Done Done false
17989 5438 Acked Write 9 Sent Done Done false
17990 5439 Acked Write 9 Sent Done Done false
17991 5440 Acked Write 8 Sent Done Done false
17992 5441 Acked Write 4 Sent Done Done false
17993 5442 Acked Read 2 Sent Done Done false
17994 5443 Acked Read 6 Sent Done Done false
17995 5444 Acked Write 7 Sent Done Done false
17996 5445 Acked Write 5 Sent Done Done false
17997 5446 Acked Read 9 Sent Done Done false
17998 5447 Acked Write 10 Sent Done Done false
17999 5448 Acked Write 3 Sent Done Done false
18000 5449 Acked Write 9 Sent Done Done false
18001 5450 Acked Read 10 Sent Done Done false
18002 5451 Acked Read 6 Sent Done Done false
18003 5452 Acked Read 5 Sent Done Done false
18004 5453 Acked Flush 0 Sent Done Done false
18005 5454 Acked Read 4 Sent Done Done false
18006 5455 Acked Read 5 Sent Done Done false
18007 5456 Acked Write 6 Sent Done Done false
18008 5457 Acked Read 9 Sent Done Done false
18009 5458 Acked Read 3 Sent Done Done false
18010 5459 Acked Read 4 Sent Done Done false
18011 5460 Acked Read 10 Sent Done Done false
18012 5461 Acked Read 10 Sent Done Done false
18013 5462 Acked Read 2 Sent Done Done false
18014 5463 Acked Read 9 Sent Done Done false
18015 5464 Acked Read 6 Sent Done Done false
18016 5465 Acked Write 3 Sent Done Done false
18017 5466 Acked Read 8 Sent Done Done false
18018 5467 Acked Flush 0 Sent Done Done false
18019 5468 Acked Read 8 Sent Done Done false
18020 5469 Acked Read 1 Sent Done Done false
18021 5470 Acked Read 10 Sent Done Done false
18022 5471 Acked Read 2 Sent Done Done false
18023 5472 Acked Read 9 Sent Done Done false
18024 5473 Acked Flush 0 Sent Done Done false
18025 5474 Acked Read 3 Sent Done Done false
18026 5475 Acked Flush 0 Sent Done Done false
18027 5476 Acked Flush 0 Sent Done Done false
18028 5477 Acked Flush 0 Sent Done Done false
18029 5478 Acked Write 4 Sent Done Done false
18030 5479 Acked Write 9 Sent Done Done false
18031 5480 Acked Flush 0 Sent Done Done false
18032 5481 Acked Write 7 Sent Done Done false
18033 5482 Acked Read 8 Sent Done Done false
18034 5483 Acked Read 5 Sent Done Done false
18035 5484 Acked Read 4 Sent Done Done false
18036 5485 Acked Read 6 Sent Done Done false
18037 5486 Acked Read 7 Sent Done Done false
18038 5487 Acked Write 8 Sent Done Done false
18039 5488 Acked Read 2 Sent Done Done false
18040 5489 Acked Write 2 Sent Done Done false
18041 5490 Acked Write 9 Sent Done Done false
18042 5491 Acked Read 1 Sent Done Done false
18043 5492 Acked Read 8 Sent Done Done false
18044 5493 Acked Write 2 Sent Done Done false
18045 5494 Acked Write 4 Sent Done Done false
18046 5495 Acked Write 9 Sent Done Done false
18047 5496 Acked Read 9 Sent Done Done false
18048 5497 Acked Read 10 Sent Done Done false
18049 5498 Acked Write 3 Sent Done Done false
18050 5499 Acked Read 1 Sent Done Done false
18051 5500 Acked Read 2 Sent Done Done false
18052 5501 Acked Read 10 Sent Done Done false
18053 5502 Acked Write 1 Sent Done Done false
18054 5503 Acked Write 4 Sent Done Done false
18055 5504 Acked Read 10 Sent Done Done false
18056 5505 Acked Write 1 Sent Done Done false
18057 5506 Acked Read 3 Sent Done Done false
18058 5507 Acked Write 2 Sent Done Done false
18059 5508 Acked Read 1 Sent Done Done false
18060 5509 Acked Read 10 Sent Done Done false
18061 5510 Acked Read 1 Sent Done Done false
18062 5511 Acked Read 6 Sent Done Done false
18063 5512 Acked Read 7 Sent Done Done false
18064 5513 Acked Write 3 Sent Done Done false
18065 5514 Acked Read 2 Sent Done Done false
18066 5515 Acked Flush 0 Sent Done Done false
18067 5516 Acked Flush 0 Sent Done Done false
18068 5517 Acked Read 1 Sent Done Done false
18069 5518 Acked Read 4 Sent Done Done false
18070 5519 Acked Read 3 Sent Done Done false
18071 5520 Acked Write 6 Sent Done Done false
18072 5521 Acked Read 7 Sent Done Done false
18073 5522 Acked Flush 0 Sent Done Done false
18074 5523 Acked Write 10 Sent Done Done false
18075 5524 Acked Write 2 Sent Done Done false
18076 5525 Acked Read 7 Sent Done Done false
18077 5526 Acked Read 4 Sent Done Done false
18078 STATES DS:0 DS:1 DS:2 TOTAL
18079 Sent 301 0 0 301
18080 Done 0 301 301 602
18081 Skipped 0 0 0 0
18082 Error 0 0 0 0
18083 Last Flush: Some(JobId(5225)) Some(JobId(5522)) Some(JobId(5522))
18084 Downstairs last five completed: 5225 5224 5223 5222 5221
18085 Upstairs last five completed: 5526 5525 5524 5523 5522
18086 CLIENT: Up:0 ds:301 act:2
180872025-03-07T18:16:29.336ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180882025-03-07T18:16:30.094ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180892025-03-07T18:16:30.094ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180902025-03-07T18:16:30.094ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180912025-03-07T18:16:30.094ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180922025-03-07T18:16:30.095ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180932025-03-07T18:16:30.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
180942025-03-07T18:16:32.339ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18095 ----------------------------------------------------------------
18096 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18097 States: Offline Active Active
18098 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18099 5226 Acked Read 6 Sent Done Done false
18100 5227 Acked Read 9 Sent Done Done false
18101 5228 Acked Read 10 Sent Done Done false
18102 5229 Acked Read 8 Sent Done Done false
18103 5230 Acked Read 10 Sent Done Done false
18104 5231 Acked Write 3 Sent Done Done false
18105 5232 Acked Write 9 Sent Done Done false
18106 5233 Acked Read 6 Sent Done Done false
18107 5234 Acked Read 3 Sent Done Done false
18108 5235 Acked Write 1 Sent Done Done false
18109 5236 Acked Read 1 Sent Done Done false
18110 5237 Acked Write 4 Sent Done Done false
18111 5238 Acked Read 6 Sent Done Done false
18112 5239 Acked Read 6 Sent Done Done false
18113 5240 Acked Write 1 Sent Done Done false
18114 5241 Acked Write 9 Sent Done Done false
18115 5242 Acked Write 5 Sent Done Done false
18116 5243 Acked Read 1 Sent Done Done false
18117 5244 Acked Write 6 Sent Done Done false
18118 5245 Acked Write 1 Sent Done Done false
18119 5246 Acked Read 4 Sent Done Done false
18120 5247 Acked Read 2 Sent Done Done false
18121 5248 Acked Flush 0 Sent Done Done false
18122 5249 Acked Write 3 Sent Done Done false
18123 5250 Acked Write 8 Sent Done Done false
18124 5251 Acked Write 2 Sent Done Done false
18125 5252 Acked Write 3 Sent Done Done false
18126 5253 Acked Write 10 Sent Done Done false
18127 5254 Acked Read 7 Sent Done Done false
18128 5255 Acked Write 4 Sent Done Done false
18129 5256 Acked Read 9 Sent Done Done false
18130 5257 Acked Read 7 Sent Done Done false
18131 5258 Acked Write 7 Sent Done Done false
18132 5259 Acked Read 9 Sent Done Done false
18133 5260 Acked Read 1 Sent Done Done false
18134 5261 Acked Read 1 Sent Done Done false
18135 5262 Acked Read 7 Sent Done Done false
18136 5263 Acked Write 2 Sent Done Done false
18137 5264 Acked Read 5 Sent Done Done false
18138 5265 Acked Read 7 Sent Done Done false
18139 5266 Acked Write 5 Sent Done Done false
18140 5267 Acked Write 9 Sent Done Done false
18141 5268 Acked Flush 0 Sent Done Done false
18142 5269 Acked Write 1 Sent Done Done false
18143 5270 Acked Write 5 Sent Done Done false
18144 5271 Acked Read 7 Sent Done Done false
18145 5272 Acked Read 5 Sent Done Done false
18146 5273 Acked Write 4 Sent Done Done false
18147 5274 Acked Read 2 Sent Done Done false
18148 5275 Acked Write 7 Sent Done Done false
18149 5276 Acked Read 6 Sent Done Done false
18150 5277 Acked Write 2 Sent Done Done false
18151 5278 Acked Write 2 Sent Done Done false
18152 5279 Acked Read 6 Sent Done Done false
18153 5280 Acked Read 1 Sent Done Done false
18154 5281 Acked Read 3 Sent Done Done false
18155 5282 Acked Write 5 Sent Done Done false
18156 5283 Acked Read 9 Sent Done Done false
18157 5284 Acked Write 3 Sent Done Done false
18158 5285 Acked Write 9 Sent Done Done false
18159 5286 Acked Read 8 Sent Done Done false
18160 5287 Acked Read 8 Sent Done Done false
18161 5288 Acked Write 3 Sent Done Done false
18162 5289 Acked Write 10 Sent Done Done false
18163 5290 Acked Write 1 Sent Done Done false
18164 5291 Acked Read 2 Sent Done Done false
18165 5292 Acked Read 2 Sent Done Done false
18166 5293 Acked Write 4 Sent Done Done false
18167 5294 Acked Read 7 Sent Done Done false
18168 5295 Acked Read 2 Sent Done Done false
18169 5296 Acked Write 1 Sent Done Done false
18170 5297 Acked Read 6 Sent Done Done false
18171 5298 Acked Write 9 Sent Done Done false
18172 5299 Acked Write 5 Sent Done Done false
18173 5300 Acked Read 10 Sent Done Done false
18174 5301 Acked Write 10 Sent Done Done false
18175 5302 Acked Read 5 Sent Done Done false
18176 5303 Acked Write 8 Sent Done Done false
18177 5304 Acked Read 8 Sent Done Done false
18178 5305 Acked Read 6 Sent Done Done false
18179 5306 Acked Read 8 Sent Done Done false
18180 5307 Acked Write 7 Sent Done Done false
18181 5308 Acked Flush 0 Sent Done Done false
18182 5309 Acked Write 4 Sent Done Done false
18183 5310 Acked Read 6 Sent Done Done false
18184 5311 Acked Write 5 Sent Done Done false
18185 5312 Acked Read 7 Sent Done Done false
18186 5313 Acked Write 7 Sent Done Done false
18187 5314 Acked Read 5 Sent Done Done false
18188 5315 Acked Flush 0 Sent Done Done false
18189 5316 Acked Flush 0 Sent Done Done false
18190 5317 Acked Read 6 Sent Done Done false
18191 5318 Acked Write 5 Sent Done Done false
18192 5319 Acked Flush 0 Sent Done Done false
18193 5320 Acked Read 3 Sent Done Done false
18194 5321 Acked Write 3 Sent Done Done false
18195 5322 Acked Write 7 Sent Done Done false
18196 5323 Acked Write 7 Sent Done Done false
18197 5324 Acked Read 6 Sent Done Done false
18198 5325 Acked Read 6 Sent Done Done false
18199 5326 Acked Read 3 Sent Done Done false
18200 5327 Acked Read 6 Sent Done Done false
18201 5328 Acked Read 7 Sent Done Done false
18202 5329 Acked Write 10 Sent Done Done false
18203 5330 Acked Write 9 Sent Done Done false
18204 5331 Acked Write 8 Sent Done Done false
18205 5332 Acked Write 8 Sent Done Done false
18206 5333 Acked Read 6 Sent Done Done false
18207 5334 Acked Flush 0 Sent Done Done false
18208 5335 Acked Flush 0 Sent Done Done false
18209 5336 Acked Read 8 Sent Done Done false
18210 5337 Acked Write 2 Sent Done Done false
18211 5338 Acked Read 6 Sent Done Done false
18212 5339 Acked Flush 0 Sent Done Done false
18213 5340 Acked Read 10 Sent Done Done false
18214 5341 Acked Read 10 Sent Done Done false
18215 5342 Acked Read 3 Sent Done Done false
18216 5343 Acked Read 9 Sent Done Done false
18217 5344 Acked Write 8 Sent Done Done false
18218 5345 Acked Flush 0 Sent Done Done false
18219 5346 Acked Read 2 Sent Done Done false
18220 5347 Acked Read 3 Sent Done Done false
18221 5348 Acked Read 10 Sent Done Done false
18222 5349 Acked Read 4 Sent Done Done false
18223 5350 Acked Write 1 Sent Done Done false
18224 5351 Acked Read 3 Sent Done Done false
18225 5352 Acked Flush 0 Sent Done Done false
18226 5353 Acked Read 10 Sent Done Done false
18227 5354 Acked Read 1 Sent Done Done false
18228 5355 Acked Write 10 Sent Done Done false
18229 5356 Acked Flush 0 Sent Done Done false
18230 5357 Acked Write 9 Sent Done Done false
18231 5358 Acked Read 5 Sent Done Done false
18232 5359 Acked Flush 0 Sent Done Done false
18233 5360 Acked Read 1 Sent Done Done false
18234 5361 Acked Read 6 Sent Done Done false
18235 5362 Acked Write 1 Sent Done Done false
18236 5363 Acked Read 3 Sent Done Done false
18237 5364 Acked Read 4 Sent Done Done false
18238 5365 Acked Write 1 Sent Done Done false
18239 5366 Acked Write 3 Sent Done Done false
18240 5367 Acked Write 4 Sent Done Done false
18241 5368 Acked Read 5 Sent Done Done false
18242 5369 Acked Read 10 Sent Done Done false
18243 5370 Acked Flush 0 Sent Done Done false
18244 5371 Acked Flush 0 Sent Done Done false
18245 5372 Acked Read 9 Sent Done Done false
18246 5373 Acked Write 6 Sent Done Done false
18247 5374 Acked Read 6 Sent Done Done false
18248 5375 Acked Read 1 Sent Done Done false
18249 5376 Acked Read 1 Sent Done Done false
18250 5377 Acked Read 6 Sent Done Done false
18251 5378 Acked Read 6 Sent Done Done false
18252 5379 Acked Read 6 Sent Done Done false
18253 5380 Acked Read 5 Sent Done Done false
18254 5381 Acked Read 1 Sent Done Done false
18255 5382 Acked Write 7 Sent Done Done false
18256 5383 Acked Write 2 Sent Done Done false
18257 5384 Acked Read 10 Sent Done Done false
18258 5385 Acked Write 9 Sent Done Done false
18259 5386 Acked Read 4 Sent Done Done false
18260 5387 Acked Read 7 Sent Done Done false
18261 5388 Acked Write 3 Sent Done Done false
18262 5389 Acked Read 10 Sent Done Done false
18263 5390 Acked Read 8 Sent Done Done false
18264 5391 Acked Flush 0 Sent Done Done false
18265 5392 Acked Read 1 Sent Done Done false
18266 5393 Acked Read 2 Sent Done Done false
18267 5394 Acked Read 4 Sent Done Done false
18268 5395 Acked Flush 0 Sent Done Done false
18269 5396 Acked Flush 0 Sent Done Done false
18270 5397 Acked Write 7 Sent Done Done false
18271 5398 Acked Flush 0 Sent Done Done false
18272 5399 Acked Flush 0 Sent Done Done false
18273 5400 Acked Write 6 Sent Done Done false
18274 5401 Acked Write 9 Sent Done Done false
18275 5402 Acked Flush 0 Sent Done Done false
18276 5403 Acked Write 7 Sent Done Done false
18277 5404 Acked Write 6 Sent Done Done false
18278 5405 Acked Flush 0 Sent Done Done false
18279 5406 Acked Write 2 Sent Done Done false
18280 5407 Acked Write 1 Sent Done Done false
18281 5408 Acked Write 8 Sent Done Done false
18282 5409 Acked Read 3 Sent Done Done false
18283 5410 Acked Write 10 Sent Done Done false
18284 5411 Acked Write 4 Sent Done Done false
18285 5412 Acked Read 6 Sent Done Done false
18286 5413 Acked Read 3 Sent Done Done false
18287 5414 Acked Write 1 Sent Done Done false
18288 5415 Acked Write 2 Sent Done Done false
18289 5416 Acked Read 4 Sent Done Done false
18290 5417 Acked Flush 0 Sent Done Done false
18291 5418 Acked Read 7 Sent Done Done false
18292 5419 Acked Write 7 Sent Done Done false
18293 5420 Acked Write 9 Sent Done Done false
18294 5421 Acked Write 6 Sent Done Done false
18295 5422 Acked Write 5 Sent Done Done false
18296 5423 Acked Write 9 Sent Done Done false
18297 5424 Acked Write 2 Sent Done Done false
18298 5425 Acked Write 9 Sent Done Done false
18299 5426 Acked Write 5 Sent Done Done false
18300 5427 Acked Read 7 Sent Done Done false
18301 5428 Acked Write 6 Sent Done Done false
18302 5429 Acked Read 5 Sent Done Done false
18303 5430 Acked Write 7 Sent Done Done false
18304 5431 Acked Read 9 Sent Done Done false
18305 5432 Acked Read 1 Sent Done Done false
18306 5433 Acked Read 5 Sent Done Done false
18307 5434 Acked Read 3 Sent Done Done false
18308 5435 Acked Read 8 Sent Done Done false
18309 5436 Acked Flush 0 Sent Done Done false
18310 5437 Acked Write 1 Sent Done Done false
18311 5438 Acked Write 9 Sent Done Done false
18312 5439 Acked Write 9 Sent Done Done false
18313 5440 Acked Write 8 Sent Done Done false
18314 5441 Acked Write 4 Sent Done Done false
18315 5442 Acked Read 2 Sent Done Done false
18316 5443 Acked Read 6 Sent Done Done false
18317 5444 Acked Write 7 Sent Done Done false
18318 5445 Acked Write 5 Sent Done Done false
18319 5446 Acked Read 9 Sent Done Done false
18320 5447 Acked Write 10 Sent Done Done false
18321 5448 Acked Write 3 Sent Done Done false
18322 5449 Acked Write 9 Sent Done Done false
18323 5450 Acked Read 10 Sent Done Done false
18324 5451 Acked Read 6 Sent Done Done false
18325 5452 Acked Read 5 Sent Done Done false
18326 5453 Acked Flush 0 Sent Done Done false
18327 5454 Acked Read 4 Sent Done Done false
18328 5455 Acked Read 5 Sent Done Done false
18329 5456 Acked Write 6 Sent Done Done false
18330 5457 Acked Read 9 Sent Done Done false
18331 5458 Acked Read 3 Sent Done Done false
18332 5459 Acked Read 4 Sent Done Done false
18333 5460 Acked Read 10 Sent Done Done false
18334 5461 Acked Read 10 Sent Done Done false
18335 5462 Acked Read 2 Sent Done Done false
18336 5463 Acked Read 9 Sent Done Done false
18337 5464 Acked Read 6 Sent Done Done false
18338 5465 Acked Write 3 Sent Done Done false
18339 5466 Acked Read 8 Sent Done Done false
18340 5467 Acked Flush 0 Sent Done Done false
18341 5468 Acked Read 8 Sent Done Done false
18342 5469 Acked Read 1 Sent Done Done false
18343 5470 Acked Read 10 Sent Done Done false
18344 5471 Acked Read 2 Sent Done Done false
18345 5472 Acked Read 9 Sent Done Done false
18346 5473 Acked Flush 0 Sent Done Done false
18347 5474 Acked Read 3 Sent Done Done false
18348 5475 Acked Flush 0 Sent Done Done false
18349 5476 Acked Flush 0 Sent Done Done false
18350 5477 Acked Flush 0 Sent Done Done false
18351 5478 Acked Write 4 Sent Done Done false
18352 5479 Acked Write 9 Sent Done Done false
18353 5480 Acked Flush 0 Sent Done Done false
18354 5481 Acked Write 7 Sent Done Done false
18355 5482 Acked Read 8 Sent Done Done false
18356 5483 Acked Read 5 Sent Done Done false
18357 5484 Acked Read 4 Sent Done Done false
18358 5485 Acked Read 6 Sent Done Done false
18359 5486 Acked Read 7 Sent Done Done false
18360 5487 Acked Write 8 Sent Done Done false
18361 5488 Acked Read 2 Sent Done Done false
18362 5489 Acked Write 2 Sent Done Done false
18363 5490 Acked Write 9 Sent Done Done false
18364 5491 Acked Read 1 Sent Done Done false
18365 5492 Acked Read 8 Sent Done Done false
18366 5493 Acked Write 2 Sent Done Done false
18367 5494 Acked Write 4 Sent Done Done false
18368 5495 Acked Write 9 Sent Done Done false
18369 5496 Acked Read 9 Sent Done Done false
18370 5497 Acked Read 10 Sent Done Done false
18371 5498 Acked Write 3 Sent Done Done false
18372 5499 Acked Read 1 Sent Done Done false
18373 5500 Acked Read 2 Sent Done Done false
18374 5501 Acked Read 10 Sent Done Done false
18375 5502 Acked Write 1 Sent Done Done false
18376 5503 Acked Write 4 Sent Done Done false
18377 5504 Acked Read 10 Sent Done Done false
18378 5505 Acked Write 1 Sent Done Done false
18379 5506 Acked Read 3 Sent Done Done false
18380 5507 Acked Write 2 Sent Done Done false
18381 5508 Acked Read 1 Sent Done Done false
18382 5509 Acked Read 10 Sent Done Done false
18383 5510 Acked Read 1 Sent Done Done false
18384 5511 Acked Read 6 Sent Done Done false
18385 5512 Acked Read 7 Sent Done Done false
18386 5513 Acked Write 3 Sent Done Done false
18387 5514 Acked Read 2 Sent Done Done false
18388 5515 Acked Flush 0 Sent Done Done false
18389 5516 Acked Flush 0 Sent Done Done false
18390 5517 Acked Read 1 Sent Done Done false
18391 5518 Acked Read 4 Sent Done Done false
18392 5519 Acked Read 3 Sent Done Done false
18393 5520 Acked Write 6 Sent Done Done false
18394 5521 Acked Read 7 Sent Done Done false
18395 5522 Acked Flush 0 Sent Done Done false
18396 5523 Acked Write 10 Sent Done Done false
18397 5524 Acked Write 2 Sent Done Done false
18398 5525 Acked Read 7 Sent Done Done false
18399 5526 Acked Read 4 Sent Done Done false
18400 STATES DS:0 DS:1 DS:2 TOTAL
18401 Sent 301 0 0 301
18402 Done 0 301 301 602
18403 Skipped 0 0 0 0
18404 Error 0 0 0 0
18405 Last Flush: Some(JobId(5225)) Some(JobId(5522)) Some(JobId(5522))
18406 Downstairs last five completed: 5225 5224 5223 5222 5221
18407 Upstairs last five completed: 5526 5525 5524 5523 5522
18408 CLIENT: Up:0 ds:301 act:2
184092025-03-07T18:16:33.340ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
184102025-03-07T18:16:34.341ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
184112025-03-07T18:16:35.344ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18412 [8810] got start action
18413 [8810] I am going to restart
18414 Starting downstairs at port 8810
18415 Make output file at "/tmp/dsc/downstairs-8810.txt"
18416 [8810][0] reports Starting
18417 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1223)
18418 [8810][0] reports Running
18419 ----------------------------------------------------------------
18420 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18421 States: Offline Active Active
18422 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18423 5226 Acked Read 6 Sent Done Done false
18424 5227 Acked Read 9 Sent Done Done false
18425 5228 Acked Read 10 Sent Done Done false
18426 5229 Acked Read 8 Sent Done Done false
18427 5230 Acked Read 10 Sent Done Done false
18428 5231 Acked Write 3 Sent Done Done false
18429 5232 Acked Write 9 Sent Done Done false
18430 5233 Acked Read 6 Sent Done Done false
18431 5234 Acked Read 3 Sent Done Done false
18432 5235 Acked Write 1 Sent Done Done false
18433 5236 Acked Read 1 Sent Done Done false
18434 5237 Acked Write 4 Sent Done Done false
18435 5238 Acked Read 6 Sent Done Done false
18436 5239 Acked Read 6 Sent Done Done false
18437 5240 Acked Write 1 Sent Done Done false
18438 5241 Acked Write 9 Sent Done Done false
18439 5242 Acked Write 5 Sent Done Done false
18440 5243 Acked Read 1 Sent Done Done false
18441 5244 Acked Write 6 Sent Done Done false
18442 5245 Acked Write 1 Sent Done Done false
18443 5246 Acked Read 4 Sent Done Done false
18444 5247 Acked Read 2 Sent Done Done false
18445 5248 Acked Flush 0 Sent Done Done false
18446 5249 Acked Write 3 Sent Done Done false
18447 5250 Acked Write 8 Sent Done Done false
18448 5251 Acked Write 2 Sent Done Done false
18449 5252 Acked Write 3 Sent Done Done false
18450 5253 Acked Write 10 Sent Done Done false
18451 5254 Acked Read 7 Sent Done Done false
18452 5255 Acked Write 4 Sent Done Done false
18453 5256 Acked Read 9 Sent Done Done false
18454 5257 Acked Read 7 Sent Done Done false
18455 5258 Acked Write 7 Sent Done Done false
18456 5259 Acked Read 9 Sent Done Done false
18457 5260 Acked Read 1 Sent Done Done false
18458 5261 Acked Read 1 Sent Done Done false
18459 5262 Acked Read 7 Sent Done Done false
18460 5263 Acked Write 2 Sent Done Done false
18461 5264 Acked Read 5 Sent Done Done false
18462 5265 Acked Read 7 Sent Done Done false
18463 5266 Acked Write 5 Sent Done Done false
18464 5267 Acked Write 9 Sent Done Done false
18465 5268 Acked Flush 0 Sent Done Done false
18466 5269 Acked Write 1 Sent Done Done false
18467 5270 Acked Write 5 Sent Done Done false
18468 5271 Acked Read 7 Sent Done Done false
18469 5272 Acked Read 5 Sent Done Done false
18470 5273 Acked Write 4 Sent Done Done false
18471 5274 Acked Read 2 Sent Done Done false
18472 5275 Acked Write 7 Sent Done Done false
18473 5276 Acked Read 6 Sent Done Done false
18474 5277 Acked Write 2 Sent Done Done false
18475 5278 Acked Write 2 Sent Done Done false
18476 5279 Acked Read 6 Sent Done Done false
18477 5280 Acked Read 1 Sent Done Done false
18478 5281 Acked Read 3 Sent Done Done false
18479 5282 Acked Write 5 Sent Done Done false
18480 5283 Acked Read 9 Sent Done Done false
18481 5284 Acked Write 3 Sent Done Done false
18482 5285 Acked Write 9 Sent Done Done false
18483 5286 Acked Read 8 Sent Done Done false
18484 5287 Acked Read 8 Sent Done Done false
18485 5288 Acked Write 3 Sent Done Done false
18486 5289 Acked Write 10 Sent Done Done false
18487 5290 Acked Write 1 Sent Done Done false
18488 5291 Acked Read 2 Sent Done Done false
18489 5292 Acked Read 2 Sent Done Done false
18490 5293 Acked Write 4 Sent Done Done false
18491 5294 Acked Read 7 Sent Done Done false
18492 5295 Acked Read 2 Sent Done Done false
18493 5296 Acked Write 1 Sent Done Done false
18494 5297 Acked Read 6 Sent Done Done false
18495 5298 Acked Write 9 Sent Done Done false
18496 5299 Acked Write 5 Sent Done Done false
18497 5300 Acked Read 10 Sent Done Done false
18498 5301 Acked Write 10 Sent Done Done false
18499 5302 Acked Read 5 Sent Done Done false
18500 5303 Acked Write 8 Sent Done Done false
18501 5304 Acked Read 8 Sent Done Done false
18502 5305 Acked Read 6 Sent Done Done false
18503 5306 Acked Read 8 Sent Done Done false
18504 5307 Acked Write 7 Sent Done Done false
18505 5308 Acked Flush 0 Sent Done Done false
18506 5309 Acked Write 4 Sent Done Done false
18507 5310 Acked Read 6 Sent Done Done false
18508 5311 Acked Write 5 Sent Done Done false
18509 5312 Acked Read 7 Sent Done Done false
18510 5313 Acked Write 7 Sent Done Done false
18511 5314 Acked Read 5 Sent Done Done false
18512 5315 Acked Flush 0 Sent Done Done false
18513 5316 Acked Flush 0 Sent Done Done false
18514 5317 Acked Read 6 Sent Done Done false
18515 5318 Acked Write 5 Sent Done Done false
18516 5319 Acked Flush 0 Sent Done Done false
18517 5320 Acked Read 3 Sent Done Done false
18518 5321 Acked Write 3 Sent Done Done false
18519 5322 Acked Write 7 Sent Done Done false
18520 5323 Acked Write 7 Sent Done Done false
18521 5324 Acked Read 6 Sent Done Done false
18522 5325 Acked Read 6 Sent Done Done false
18523 5326 Acked Read 3 Sent Done Done false
18524 5327 Acked Read 6 Sent Done Done false
18525 5328 Acked Read 7 Sent Done Done false
18526 5329 Acked Write 10 Sent Done Done false
18527 5330 Acked Write 9 Sent Done Done false
18528 5331 Acked Write 8 Sent Done Done false
18529 5332 Acked Write 8 Sent Done Done false
18530 5333 Acked Read 6 Sent Done Done false
18531 5334 Acked Flush 0 Sent Done Done false
18532 5335 Acked Flush 0 Sent Done Done false
18533 5336 Acked Read 8 Sent Done Done false
18534 5337 Acked Write 2 Sent Done Done false
18535 5338 Acked Read 6 Sent Done Done false
18536 5339 Acked Flush 0 Sent Done Done false
18537 5340 Acked Read 10 Sent Done Done false
18538 5341 Acked Read 10 Sent Done Done false
18539 5342 Acked Read 3 Sent Done Done false
18540 5343 Acked Read 9 Sent Done Done false
18541 5344 Acked Write 8 Sent Done Done false
18542 5345 Acked Flush 0 Sent Done Done false
18543 5346 Acked Read 2 Sent Done Done false
18544 5347 Acked Read 3 Sent Done Done false
18545 5348 Acked Read 10 Sent Done Done false
18546 5349 Acked Read 4 Sent Done Done false
18547 5350 Acked Write 1 Sent Done Done false
18548 5351 Acked Read 3 Sent Done Done false
18549 5352 Acked Flush 0 Sent Done Done false
18550 5353 Acked Read 10 Sent Done Done false
18551 5354 Acked Read 1 Sent Done Done false
18552 5355 Acked Write 10 Sent Done Done false
18553 5356 Acked Flush 0 Sent Done Done false
18554 5357 Acked Write 9 Sent Done Done false
18555 5358 Acked Read 5 Sent Done Done false
18556 5359 Acked Flush 0 Sent Done Done false
18557 5360 Acked Read 1 Sent Done Done false
18558 5361 Acked Read 6 Sent Done Done false
18559 5362 Acked Write 1 Sent Done Done false
18560 5363 Acked Read 3 Sent Done Done false
18561 5364 Acked Read 4 Sent Done Done false
18562 5365 Acked Write 1 Sent Done Done false
18563 5366 Acked Write 3 Sent Done Done false
18564 5367 Acked Write 4 Sent Done Done false
18565 5368 Acked Read 5 Sent Done Done false
18566 5369 Acked Read 10 Sent Done Done false
18567 5370 Acked Flush 0 Sent Done Done false
18568 5371 Acked Flush 0 Sent Done Done false
18569 5372 Acked Read 9 Sent Done Done false
18570 5373 Acked Write 6 Sent Done Done false
18571 5374 Acked Read 6 Sent Done Done false
18572 5375 Acked Read 1 Sent Done Done false
18573 5376 Acked Read 1 Sent Done Done false
18574 5377 Acked Read 6 Sent Done Done false
18575 5378 Acked Read 6 Sent Done Done false
18576 5379 Acked Read 6 Sent Done Done false
18577 5380 Acked Read 5 Sent Done Done false
18578 5381 Acked Read 1 Sent Done Done false
18579 5382 Acked Write 7 Sent Done Done false
18580 5383 Acked Write 2 Sent Done Done false
18581 5384 Acked Read 10 Sent Done Done false
18582 5385 Acked Write 9 Sent Done Done false
18583 5386 Acked Read 4 Sent Done Done false
18584 5387 Acked Read 7 Sent Done Done false
18585 5388 Acked Write 3 Sent Done Done false
18586 5389 Acked Read 10 Sent Done Done false
18587 5390 Acked Read 8 Sent Done Done false
18588 5391 Acked Flush 0 Sent Done Done false
18589 5392 Acked Read 1 Sent Done Done false
18590 5393 Acked Read 2 Sent Done Done false
18591 5394 Acked Read 4 Sent Done Done false
18592 5395 Acked Flush 0 Sent Done Done false
18593 5396 Acked Flush 0 Sent Done Done false
18594 5397 Acked Write 7 Sent Done Done false
18595 5398 Acked Flush 0 Sent Done Done false
18596 5399 Acked Flush 0 Sent Done Done false
18597 5400 Acked Write 6 Sent Done Done false
18598 5401 Acked Write 9 Sent Done Done false
18599 5402 Acked Flush 0 Sent Done Done false
18600 5403 Acked Write 7 Sent Done Done false
18601 5404 Acked Write 6 Sent Done Done false
18602 5405 Acked Flush 0 Sent Done Done false
18603 5406 Acked Write 2 Sent Done Done false
18604 5407 Acked Write 1 Sent Done Done false
18605 5408 Acked Write 8 Sent Done Done false
18606 5409 Acked Read 3 Sent Done Done false
18607 5410 Acked Write 10 Sent Done Done false
18608 5411 Acked Write 4 Sent Done Done false
18609 5412 Acked Read 6 Sent Done Done false
18610 5413 Acked Read 3 Sent Done Done false
18611 5414 Acked Write 1 Sent Done Done false
18612 5415 Acked Write 2 Sent Done Done false
18613 5416 Acked Read 4 Sent Done Done false
18614 5417 Acked Flush 0 Sent Done Done false
18615 5418 Acked Read 7 Sent Done Done false
18616 5419 Acked Write 7 Sent Done Done false
18617 5420 Acked Write 9 Sent Done Done false
18618 5421 Acked Write 6 Sent Done Done false
18619 5422 Acked Write 5 Sent Done Done false
18620 5423 Acked Write 9 Sent Done Done false
18621 5424 Acked Write 2 Sent Done Done false
18622 5425 Acked Write 9 Sent Done Done false
18623 5426 Acked Write 5 Sent Done Done false
18624 5427 Acked Read 7 Sent Done Done false
18625 5428 Acked Write 6 Sent Done Done false
18626 5429 Acked Read 5 Sent Done Done false
18627 5430 Acked Write 7 Sent Done Done false
18628 5431 Acked Read 9 Sent Done Done false
18629 5432 Acked Read 1 Sent Done Done false
18630 5433 Acked Read 5 Sent Done Done false
18631 5434 Acked Read 3 Sent Done Done false
18632 5435 Acked Read 8 Sent Done Done false
18633 5436 Acked Flush 0 Sent Done Done false
18634 5437 Acked Write 1 Sent Done Done false
18635 5438 Acked Write 9 Sent Done Done false
18636 5439 Acked Write 9 Sent Done Done false
18637 5440 Acked Write 8 Sent Done Done false
18638 5441 Acked Write 4 Sent Done Done false
18639 5442 Acked Read 2 Sent Done Done false
18640 5443 Acked Read 6 Sent Done Done false
18641 5444 Acked Write 7 Sent Done Done false
18642 5445 Acked Write 5 Sent Done Done false
18643 5446 Acked Read 9 Sent Done Done false
18644 5447 Acked Write 10 Sent Done Done false
18645 5448 Acked Write 3 Sent Done Done false
18646 5449 Acked Write 9 Sent Done Done false
18647 5450 Acked Read 10 Sent Done Done false
18648 5451 Acked Read 6 Sent Done Done false
18649 5452 Acked Read 5 Sent Done Done false
18650 5453 Acked Flush 0 Sent Done Done false
18651 5454 Acked Read 4 Sent Done Done false
18652 5455 Acked Read 5 Sent Done Done false
18653 5456 Acked Write 6 Sent Done Done false
18654 5457 Acked Read 9 Sent Done Done false
18655 5458 Acked Read 3 Sent Done Done false
18656 5459 Acked Read 4 Sent Done Done false
18657 5460 Acked Read 10 Sent Done Done false
18658 5461 Acked Read 10 Sent Done Done false
18659 5462 Acked Read 2 Sent Done Done false
18660 5463 Acked Read 9 Sent Done Done false
18661 5464 Acked Read 6 Sent Done Done false
18662 5465 Acked Write 3 Sent Done Done false
18663 5466 Acked Read 8 Sent Done Done false
18664 5467 Acked Flush 0 Sent Done Done false
18665 5468 Acked Read 8 Sent Done Done false
18666 5469 Acked Read 1 Sent Done Done false
18667 5470 Acked Read 10 Sent Done Done false
18668 5471 Acked Read 2 Sent Done Done false
18669 5472 Acked Read 9 Sent Done Done false
18670 5473 Acked Flush 0 Sent Done Done false
18671 5474 Acked Read 3 Sent Done Done false
18672 5475 Acked Flush 0 Sent Done Done false
18673 5476 Acked Flush 0 Sent Done Done false
18674 5477 Acked Flush 0 Sent Done Done false
18675 5478 Acked Write 4 Sent Done Done false
18676 5479 Acked Write 9 Sent Done Done false
18677 5480 Acked Flush 0 Sent Done Done false
18678 5481 Acked Write 7 Sent Done Done false
18679 5482 Acked Read 8 Sent Done Done false
18680 5483 Acked Read 5 Sent Done Done false
18681 5484 Acked Read 4 Sent Done Done false
18682 5485 Acked Read 6 Sent Done Done false
18683 5486 Acked Read 7 Sent Done Done false
18684 5487 Acked Write 8 Sent Done Done false
18685 5488 Acked Read 2 Sent Done Done false
18686 5489 Acked Write 2 Sent Done Done false
18687 5490 Acked Write 9 Sent Done Done false
18688 5491 Acked Read 1 Sent Done Done false
18689 5492 Acked Read 8 Sent Done Done false
18690 5493 Acked Write 2 Sent Done Done false
18691 5494 Acked Write 4 Sent Done Done false
18692 5495 Acked Write 9 Sent Done Done false
18693 5496 Acked Read 9 Sent Done Done false
18694 5497 Acked Read 10 Sent Done Done false
18695 5498 Acked Write 3 Sent Done Done false
18696 5499 Acked Read 1 Sent Done Done false
18697 5500 Acked Read 2 Sent Done Done false
18698 5501 Acked Read 10 Sent Done Done false
18699 5502 Acked Write 1 Sent Done Done false
18700 5503 Acked Write 4 Sent Done Done false
18701 5504 Acked Read 10 Sent Done Done false
18702 5505 Acked Write 1 Sent Done Done false
18703 5506 Acked Read 3 Sent Done Done false
18704 5507 Acked Write 2 Sent Done Done false
18705 5508 Acked Read 1 Sent Done Done false
18706 5509 Acked Read 10 Sent Done Done false
18707 5510 Acked Read 1 Sent Done Done false
18708 5511 Acked Read 6 Sent Done Done false
18709 5512 Acked Read 7 Sent Done Done false
18710 5513 Acked Write 3 Sent Done Done false
18711 5514 Acked Read 2 Sent Done Done false
18712 5515 Acked Flush 0 Sent Done Done false
18713 5516 Acked Flush 0 Sent Done Done false
18714 5517 Acked Read 1 Sent Done Done false
18715 5518 Acked Read 4 Sent Done Done false
18716 5519 Acked Read 3 Sent Done Done false
18717 5520 Acked Write 6 Sent Done Done false
18718 5521 Acked Read 7 Sent Done Done false
18719 5522 Acked Flush 0 Sent Done Done false
18720 5523 Acked Write 10 Sent Done Done false
18721 5524 Acked Write 2 Sent Done Done false
18722 5525 Acked Read 7 Sent Done Done false
18723 5526 Acked Read 4 Sent Done Done false
18724 STATES DS:0 DS:1 DS:2 TOTAL
18725 Sent 301 0 0 301
18726 Done 0 301 301 602
18727 Skipped 0 0 0 0
18728 Error 0 0 0 0
18729 Last Flush: Some(JobId(5225)) Some(JobId(5522)) Some(JobId(5522))
18730 Downstairs last five completed: 5225 5224 5223 5222 5221
18731 Upstairs last five completed: 5526 5525 5524 5523 5522
18732 CLIENT: Up:0 ds:301 act:2
187332025-03-07T18:16:37.346ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187342025-03-07T18:16:38.348ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187352025-03-07T18:16:39.351ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187362025-03-07T18:16:40.096ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187372025-03-07T18:16:40.096ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52013) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187382025-03-07T18:16:40.097ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187392025-03-07T18:16:40.098ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187402025-03-07T18:16:40.098ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187412025-03-07T18:16:40.098ZINFOcrucible: send last flush ID to this DS: Some(JobId(5225)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187422025-03-07T18:16:40.098ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187432025-03-07T18:16:40.098ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5225)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187442025-03-07T18:16:40.098ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5225)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18745 ----------------------------------------------------------------
18746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
18747 States: Active Active Active
18748 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18749 5523 Acked Write 10 Done Done Done true
18750 5524 Acked Write 2 Done Done Done true
18751 5525 Acked Read 7 Done Done Done true
18752 5526 Acked Read 4 Done Done Done true
18753 STATES DS:0 DS:1 DS:2 TOTAL
18754 Sent 0 0 0 0
18755 Done 4 4 4 12
18756 Skipped 0 0 0 0
18757 Error 0 0 0 0
18758 Last Flush: Some(JobId(5522)) Some(JobId(5522)) Some(JobId(5522))
18759 Downstairs last five completed: 5522 5521 5520 5519 5518
18760 Upstairs last five completed: 5526 5525 5524 5523 5522
18761 CLIENT: Up:0 ds:4 act:3
18762 ----------------------------------------------------------------
18763 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18764 Downstairs last five completed: 5527 5526 5525 5524 5523
18765 Upstairs last five completed: 5527 5526 5525 5524 5523
18766 CLIENT: Up:0 ds:0 act:3
18767 Replay: All jobs finished
18768 Main task has work to do, go find it
18769 got dsc Stop(0)
18770 stop 0
18771 Mar 07 18:16:44.353 INFO request completed, latency_us: 349, response_code: 204, uri: [8810] Got stop action so:false kr:false
18772 /stop/cid/0, method: POST, req_id: 2c66091f-af8c-4e1e-bb90-130bb8256595, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
187732025-03-07T18:16:44.354ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187742025-03-07T18:16:44.354ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
187752025-03-07T18:16:44.354ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
18776 Mar 07 18:16:44.354 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/0, method: GET, req_id: ccf90d08-47ae-46b6-a7e0-082c8395147f, remote_addr: 127.0.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:16:44.354408572Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
18777 .1:64322, local_addr: 127.0.0.1:9998
18778 [8810] Exited with: ExitStatus(unix_wait_status(9))
18779 [8810][0] reports Exit
18780 Mar 07 18:16:48.356 INFO request completed, latency_us: 198, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8162ff82-01ff-4e66-a1c4-ff8774605330, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
18781 001/300 Read block 367 len 20480
18782 002/300 Read block 4012 len 28672
18783 003/300 Write block 2003 len 36864 data: 3 4 5 6 6 7 6 5 4
18784 004/300 Write block 163 len 32768 data: 5 5 6 7 6 6 5 7
18785 005/300 Read block 4771 len 16384
18786 006/300 Write block 2892 len 24576 data: 4 4 4 3 3 3
18787 007/300 Write block 2799 len 16384 data: 3 4 4 7
18788 008/300 Write block 3984 len 40960 data: 4 2 2 2 4 5 7 6 6 5
18789 009/300 Read block 1662 len 12288
18790 010/300 Read block 2257 len 12288
18791 011/300 Read block 136 len 4096
18792 012/300 Read block 1811 len 12288
18793 013/300 Write block 1957 len 40960 data: 2 4 4 3 2 2 3 4 4 3
18794 014/300 Flush
18795 015/300 Write block 3123 len 20480 data: 2 4 6 6 6
18796 016/300 Write block 1164 len 4096 data: 7
18797 017/300 Read block 2553 len 4096
18798 018/300 Write block 4535 len 12288 data: 4 4 4
18799 019/300 Write block 2077 len 12288 data: 3 2 2
18800 020/300 Write block 3025 len 36864 data: 4 5 4 4 2 3 3 3 3
18801 021/300 Write block 1966 len 24576 data: 4 3 3 4 4 4
18802 022/300 Write block 1666 len 24576 data: 2 3 3 4 4 4
18803 023/300 Flush
18804 024/300 Read block 1965 len 4096
18805 025/300 Flush
18806 026/300 Read block 4191 len 28672
18807 027/300 Write block 363 len 20480 data: 2 2 2 3 3
18808 028/300 Flush
18809 029/300 Flush
18810 030/300 Read block 4800 len 4096
18811 031/300 Write block 810 len 12288 data: 3 3 3
18812 032/300 Flush
18813 033/300 Write block 4194 len 20480 data: 3 3 4 4 5
18814 034/300 Read block 1395 len 40960
18815 035/300 Write block 885 len 32768 data: 2 2 2 2 3 4 3 3
18816 036/300 Read block 3925 len 12288
18817 037/300 Read block 1997 len 12288
18818 038/300 Flush
18819 039/300 Write block 4688 len 40960 data: 5 4 4 6 6 5 7 8 8 7
18820 040/300 Flush
18821 041/300 Write block 4561 len 40960 data: 3 4 4 4 3 2 2 2 2 2
18822 042/300 Write block 4181 len 36864 data: 3 3 3 3 3 4 4 3 3
18823 043/300 Read block 2593 len 40960
18824 044/300 Read block 4652 len 4096
18825 045/300 Read block 591 len 8192
18826 046/300 Write block 1712 len 40960 data: 3 3 3 4 4 4 5 6 6 6
18827 047/300 Read block 873 len 8192
18828 048/300 Read block 1915 len 4096
18829 049/300 Write block 908 len 12288 data: 3 4 4
18830 050/300 Read block 4675 len 8192
18831 051/300 Read block 3210 len 32768
18832 052/300 Write block 432 len 32768 data: 4 4 5 4 2 2 2 2
18833 053/300 Read block 2290 len 16384
18834 054/300 Write block 3046 len 4096 data: 4
18835 055/300 Read block 3180 len 20480
18836 056/300 Read block 1985 len 32768
18837 057/300 Write block 4486 len 8192 data: 2 2
18838 058/300 Read block 3714 len 16384
18839 059/300 Flush
18840 060/300 Write block 678 len 36864 data: 5 4 2 3 3 4 4 4 6
18841 061/300 Read block 2036 len 16384
18842 062/300 Read block 3697 len 40960
18843 063/300 Write block 3380 len 20480 data: 3 3 3 4 5
18844 064/300 Write block 1766 len 12288 data: 4 4 3
18845 065/300 Write block 3094 len 20480 data: 3 3 4 5 5
18846 066/300 Write block 1552 len 28672 data: 4 5 3 2 2 2 2
18847 067/300 Read block 4261 len 28672
18848 068/300 Read block 447 len 24576
18849 069/300 Flush
18850 070/300 Read block 831 len 40960
18851 071/300 Write block 2156 len 8192 data: 2 3
18852 072/300 Read block 1288 len 36864
18853 073/300 Read block 928 len 28672
18854 074/300 Write block 3906 len 24576 data: 5 5 4 4 2 2
18855 075/300 Read block 3300 len 16384
18856 076/300 Write block 301 len 28672 data: 3 4 3 3 4 4 6
18857 077/300 Read block 4180 len 16384
18858 078/300 Read block 887 len 28672
18859 079/300 Read block 4430 len 28672
18860 080/300 Read block 4964 len 4096
18861 081/300 Read block 2157 len 8192
18862 082/300 Write block 4719 len 28672 data: 3 4 3 3 3 3 3
18863 083/300 Read block 2751 len 32768
18864 084/300 Write block 166 len 20480 data: 8 7 7 6 8
18865 085/300 Write block 3281 len 4096 data: 7
18866 086/300 Read block 2706 len 40960
18867 087/300 Write block 1359 len 20480 data: 3 5 6 6 5
18868 088/300 Read block 3192 len 28672
18869 089/300 Read block 749 len 8192
18870 090/300 Read block 3935 len 24576
18871 091/300 Read block 2408 len 40960
18872 092/300 Write block 4225 len 8192 data: 4 3
18873 093/300 Write block 2180 len 32768 data: 4 5 5 5 5 5 6 6
18874 094/300 Read block 496 len 16384
18875 095/300 Read block 3884 len 28672
18876 096/300 Read block 566 len 32768
18877 097/300 Read block 1376 len 28672
18878 098/300 Read block 3235 len 16384
18879 099/300 Write block 3650 len 4096 data: 5
18880 100/300 Write block 2702 len 40960 data: 5 4 2 2 2 3 3 3 3 4
18881 101/300 Write block 2154 len 4096 data: 2
18882 102/300 Read block 81 len 28672
18883 103/300 Read block 4726 len 20480
18884 104/300 Read block 3443 len 12288
18885 105/300 Flush
18886 106/300 Write block 1137 len 36864 data: 2 4 4 4 4 4 4 4 4
18887 107/300 Read block 4223 len 8192
18888 108/300 Write block 2111 len 36864 data: 6 5 5 6 6 5 4 5 7
18889 109/300 Read block 4501 len 8192
18890 110/300 Read block 1163 len 20480
18891 111/300 Write block 2654 len 24576 data: 4 5 5 6 6 5
18892 112/300 Flush
18893 113/300 Write block 1139 len 32768 data: 5 5 5 5 5 5 5 5
18894 114/300 Read block 3629 len 28672
18895 115/300 Write block 3363 len 20480 data: 4 4 4 4 3
18896 116/300 Flush
18897 117/300 Read block 2232 len 12288
18898 118/300 Read block 4826 len 28672
18899 119/300 Write block 152 len 40960 data: 4 4 3 3 4 4 3 5 5 5
18900 120/300 Flush
18901 121/300 Write block 1466 len 40960 data: 6 6 5 5 4 3 2 2 2 2
18902 122/300 Read block 1750 len 20480
18903 123/300 Write block 820 len 40960 data: 5 4 4 4 5 7 5 6 5 5
18904 124/300 Write block 3827 len 20480 data: 2 2 3 3 3
18905 125/300 Read block 737 len 20480
18906 126/300 Write block 2724 len 36864 data: 3 4 5 4 4 4 5 5 7
18907 127/300 Read block 4249 len 8192
18908 128/300 Write block 1657 len 28672 data: 4 4 3 3 3 3 3
18909 129/300 Write block 199 len 16384 data: 3 3 5 5
18910 130/300 Write block 3317 len 28672 data: 2 2 2 3 3 4 5
18911 131/300 Write block 1175 len 12288 data: 2 3 2
18912 132/300 Write block 4073 len 32768 data: 5 5 7 5 4 4 4 5
18913 133/300 Read block 384 len 20480
18914 134/300 Write block 839 len 8192 data: 3 3
18915 135/300 Write block 461 len 20480 data: 2 2 2 3 3
18916 136/300 Flush
18917 137/300 Write block 1423 len 4096 data: 4
18918 138/300 Read block 3017 len 4096
18919 139/300 Write block 2531 len 40960 data: 5 5 5 6 5 5 4 4 5 5
18920 140/300 Write block 1661 len 28672 data: 4 4 4 2 2 3 4
18921 141/300 Write block 3782 len 4096 data: 8
18922 142/300 Read block 4596 len 4096
18923 143/300 Write block 2819 len 20480 data: 5 5 5 5 5
18924 144/300 Write block 4456 len 12288 data: 6 6 4
18925 145/300 Read block 193 len 40960
18926 146/300 Write block 1900 len 28672 data: 3 3 4 5 5 5 5
18927 147/300 Read block 1284 len 12288
18928 148/300 Read block 282 len 36864
18929 149/300 Write block 3338 len 24576 data: 4 4 4 4 3 3
18930 150/300 Write block 3902 len 32768 data: 4 4 3 3 6 6 5 5
18931 151/300 Flush
18932 152/300 Write block 1256 len 8192 data: 2 2
18933 153/300 Read block 472 len 32768
18934 154/300 Read block 2488 len 8192
18935 155/300 Write block 3009 len 36864 data: 6 6 5 4 4 4 4 4 3
18936 156/300 Write block 3463 len 4096 data: 3
18937 157/300 Write block 115 len 40960 data: 3 3 3 3 3 4 4 4 4 4
18938 158/300 Read block 1992 len 20480
18939 159/300 Write block 422 len 4096 data: 5
18940 160/300 Write block 3392 len 16384 data: 2 2 2 3
18941 161/300 Write block 992 len 16384 data: 3 3 3 3
18942 162/300 Read block 3445 len 32768
18943 163/300 Read block 2473 len 8192
18944 164/300 Read block 3140 len 12288
18945 165/300 Read block 482 len 12288
18946 166/300 Read block 1400 len 28672
18947 167/300 Write block 4600 len 12288 data: 5 6 6
18948 168/300 Read block 838 len 24576
18949 169/300 Read block 4675 len 20480
18950 170/300 Flush
18951 171/300 Write block 632 len 28672 data: 5 4 5 5 5 4 5
18952 172/300 Read block 4344 len 28672
18953 173/300 Write block 2820 len 16384 data: 6 6 6 6
18954 174/300 Write block 227 len 28672 data: 3 3 3 4 4 4 4
18955 175/300 Flush
18956 176/300 Read block 3431 len 36864
18957 177/300 Read block 1366 len 20480
18958 178/300 Write block 4799 len 8192 data: 5 4
18959 179/300 Flush
18960 180/300 Read block 4523 len 24576
18961 181/300 Read block 2845 len 24576
18962 182/300 Read block 1609 len 16384
18963 183/300 Read block 4249 len 16384
18964 184/300 Write block 274 len 32768 data: 6 6 8 5 5 4 4 5
18965 185/300 Read block 211 len 32768
18966 186/300 Write block 2740 len 24576 data: 6 6 4 4 4 4
18967 187/300 Flush
18968 188/300 Flush
18969 189/300 Read block 3874 len 32768
18970 190/300 Read block 1860 len 4096
18971 191/300 Write block 3397 len 24576 data: 3 4 3 3 3 3
18972 192/300 Write block 2956 len 40960 data: 3 3 3 2 2 2 3 2 2 3
18973 193/300 Write block 1924 len 36864 data: 5 4 4 4 4 4 4 3 3
18974 194/300 Read block 2299 len 8192
18975 195/300 Read block 4095 len 4096
18976 196/300 Write block 1307 len 8192 data: 4 3
18977 197/300 Write block 1188 len 20480 data: 4 5 5 5 5
18978 198/300 Read block 4853 len 16384
18979 199/300 Write block 222 len 24576 data: 4 4 3 3 3 4
18980 200/300 Read block 4144 len 20480
18981 201/300 Read block 4615 len 8192
18982 202/300 Write block 1216 len 36864 data: 4 2 2 2 3 3 4 3 3
18983 203/300 Read block 2525 len 32768
18984 204/300 Read block 4018 len 20480
18985 205/300 Flush
18986 206/300 Read block 774 len 4096
18987 207/300 Read block 3852 len 28672
18988 208/300 Read block 305 len 12288
18989 209/300 Write block 3073 len 40960 data: 4 4 3 4 5 5 5 7 7 7
18990 210/300 Write block 2927 len 12288 data: 5 4 3
18991 211/300 Read block 2151 len 40960
18992 212/300 Write block 3645 len 16384 data: 4 4 4 4
18993 213/300 Write block 1529 len 20480 data: 3 3 5 5 4
18994 214/300 Read block 1575 len 8192
18995 215/300 Read block 4349 len 20480
18996 216/300 Read block 3562 len 16384
18997 217/300 Read block 2845 len 16384
18998 218/300 Read block 569 len 40960
18999 219/300 Write block 655 len 8192 data: 2 3
19000 220/300 Read block 3077 len 40960
19001 221/300 Write block 2847 len 36864 data: 3 2 2 2 2 2 2 3 2
19002 222/300 Write block 2560 len 32768 data: 4 5 4 3 3 3 2 2
19003 223/300 Read block 2545 len 36864
19004 224/300 Read block 3364 len 32768
19005 225/300 Read block 4259 len 40960
19006 226/300 Read block 101 len 4096
19007 227/300 Read block 3576 len 24576
19008 228/300 Read block 4006 len 20480
19009 229/300 Write block 1749 len 20480 data: 5 5 4 4 3
19010 230/300 Read block 2241 len 28672
19011 231/300 Write block 933 len 24576 data: 6 5 4 4 5 6
19012 232/300 Read block 2055 len 20480
19013 233/300 Write block 2329 len 20480 data: 3 3 4 4 4
19014 234/300 Read block 1589 len 4096
19015 235/300 Read block 3325 len 40960
19016 236/300 Write block 724 len 8192 data: 4 4
19017 237/300 Write block 3219 len 20480 data: 4 4 3 3 3
19018 238/300 Read block 2152 len 28672
19019 239/300 Read block 4395 len 40960
19020 240/300 Read block 295 len 20480
19021 241/300 Write block 2259 len 20480 data: 4 3 3 3 3
19022 242/300 Read block 1815 len 36864
19023 243/300 Read block 1399 len 24576
19024 244/300 Read block 2038 len 24576
19025 245/300 Read block 1384 len 16384
19026 246/300 Flush
19027 247/300 Read block 4301 len 28672
19028 248/300 Write block 3954 len 40960 data: 4 3 4 4 4 4 4 3 3 3
19029 249/300 Write block 1121 len 12288 data: 6 6 6
19030 250/300 Write block 1643 len 28672 data: 4 4 3 3 3 3 4
19031 251/300 Read block 3155 len 12288
19032 252/300 Write block 3738 len 28672 data: 2 2 3 3 3 4 6
19033 253/300 Flush
19034 254/300 Read block 3357 len 40960
19035 255/300 Write block 4116 len 40960 data: 3 3 3 3 4 4 6 6 6 7
19036 256/300 Write block 772 len 40960 data: 5 5 5 5 4 5 5 5 6 5
19037 257/300 Read block 4273 len 36864
19038 258/300 Write block 3733 len 20480 data: 3 3 2 2 2
19039 259/300 Write block 2997 len 16384 data: 3 3 3 3
19040 260/300 Write block 3983 len 8192 data: 4 5
19041 261/300 Flush
19042 262/300 Flush
19043 263/300 Read block 1694 len 40960
19044 264/300 Write block 957 len 40960 data: 6 6 5 6 5 5 4 4 3 3
19045 265/300 Write block 1731 len 32768 data: 3 3 3 3 3 3 2 2
19046 266/300 Write block 4577 len 40960 data: 4 4 4 3 3 4 3 4 4 5
19047 267/300 Write block 1078 len 36864 data: 3 3 4 4 4 3 3 3 3
19048 268/300 Write block 3912 len 20480 data: 2 2 3 3 4
19049 269/300 Read block 896 len 12288
19050 270/300 Read block 3886 len 12288
19051 271/300 Read block 4603 len 8192
19052 272/300 Write block 3437 len 16384 data: 5 4 4 4
19053 273/300 Read block 1624 len 12288
19054 274/300 Read block 50 len 36864
19055 275/300 Write block 1533 len 20480 data: 5 4 4 3 3
19056 276/300 Write block 2341 len 36864 data: 5 6 4 3 3 3 3 2 2
19057 277/300 Read block 4743 len 20480
19058 278/300 Write block 196 len 40960 data: 7 7 6 4 4 6 6 6 6 4
19059 279/300 Read block 1023 len 8192
19060 280/300 Read block 4157 len 8192
19061 281/300 Read block 3970 len 4096
19062 282/300 Write block 584 len 40960 data: 3 3 3 4 3 3 3 3 4 4
19063 283/300 Read block 2156 len 32768
19064 284/300 Flush
19065 285/300 Read block 239 len 8192
19066 286/300 Read block 3785 len 32768
19067 287/300 Flush
19068 288/300 Read block 3543 len 12288
19069 289/300 Write block 1625 len 28672 data: 6 6 5 4 5 4 3
19070 290/300 Read block 909 len 20480
19071 291/300 Write block 948 len 28672 data: 2 2 2 2 3 3 4
19072 292/300 Write block 4656 len 40960 data: 3 3 3 2 3 3 3 3 3 3
19073 293/300 Write block 2926 len 4096 data: 6
19074 294/300 Write block 1076 len 12288 data: 3 3 4
19075 295/300 Write block 4825 len 12288 data: 3 3 4
19076 296/300 Read block 576 len 32768
19077 297/300 Write block 4537 len 36864 data: 5 3 3 4 5 6 7 7 8
19078 298/300 Read block 3537 len 4096
19079 299/300 Write block 269 len 20480 data: 4 5 6 5 5
19080 300/300 Write block 4006 len 20480 data: 3 3 3 3 3
19081 301/300 Read block 4091 len 32768
19082 Main task has work to do, go find it
19083 got dsc Start(0)
19084 start 0
19085 Mar 07 18:16:48.601 INFO request completed, latency_us: 174, response_code: 204, uri: /start/cid/0, method: POST, req_id: e57ca270-8845-46ea-8b97-2eb844d1123b, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
19086 Replay: started 0, returned:Ok(())
19087 ----------------------------------------------------------------
19088 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19089 States: Offline Active Active
19090 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19091 5528 Acked Read 5 Sent Done Done false
19092 5529 Acked Read 7 Sent Done Done false
19093 5530 Acked Write 9 Sent Done Done false
19094 5531 Acked Write 8 Sent Done Done false
19095 5532 Acked Read 4 Sent Done Done false
19096 5533 Acked Write 6 Sent Done Done false
19097 5534 Acked Write 4 Sent Done Done false
19098 5535 Acked Write 10 Sent Done Done false
19099 5536 Acked Read 3 Sent Done Done false
19100 5537 Acked Read 3 Sent Done Done false
19101 5538 Acked Read 1 Sent Done Done false
19102 5539 Acked Read 3 Sent Done Done false
19103 5540 Acked Write 10 Sent Done Done false
19104 5541 Acked Flush 0 Sent Done Done false
19105 5542 Acked Write 5 Sent Done Done false
19106 5543 Acked Write 1 Sent Done Done false
19107 5544 Acked Read 1 Sent Done Done false
19108 5545 Acked Write 3 Sent Done Done false
19109 5546 Acked Write 3 Sent Done Done false
19110 5547 Acked Write 9 Sent Done Done false
19111 5548 Acked Write 6 Sent Done Done false
19112 5549 Acked Write 6 Sent Done Done false
19113 5550 Acked Flush 0 Sent Done Done false
19114 5551 Acked Read 1 Sent Done Done false
19115 5552 Acked Flush 0 Sent Done Done false
19116 5553 Acked Read 7 Sent Done Done false
19117 5554 Acked Write 5 Sent Done Done false
19118 5555 Acked Flush 0 Sent Done Done false
19119 5556 Acked Flush 0 Sent Done Done false
19120 5557 Acked Read 1 Sent Done Done false
19121 5558 Acked Write 3 Sent Done Done false
19122 5559 Acked Flush 0 Sent Done Done false
19123 5560 Acked Write 5 Sent Done Done false
19124 5561 Acked Read 10 Sent Done Done false
19125 5562 Acked Write 8 Sent Done Done false
19126 5563 Acked Read 3 Sent Done Done false
19127 5564 Acked Read 3 Sent Done Done false
19128 5565 Acked Flush 0 Sent Done Done false
19129 5566 Acked Write 10 Sent Done Done false
19130 5567 Acked Flush 0 Sent Done Done false
19131 5568 Acked Write 10 Sent Done Done false
19132 5569 Acked Write 9 Sent Done Done false
19133 5570 Acked Read 10 Sent Done Done false
19134 5571 Acked Read 1 Sent Done Done false
19135 5572 Acked Read 2 Sent Done Done false
19136 5573 Acked Write 10 Sent Done Done false
19137 5574 Acked Read 2 Sent Done Done false
19138 5575 Acked Read 1 Sent Done Done false
19139 5576 Acked Write 3 Sent Done Done false
19140 5577 Acked Read 2 Sent Done Done false
19141 5578 Acked Read 8 Sent Done Done false
19142 5579 Acked Write 8 Sent Done Done false
19143 5580 Acked Read 4 Sent Done Done false
19144 5581 Acked Write 1 Sent Done Done false
19145 5582 Acked Read 5 Sent Done Done false
19146 5583 Acked Read 8 Sent Done Done false
19147 5584 Acked Write 2 Sent Done Done false
19148 5585 Acked Read 4 Sent Done Done false
19149 5586 Acked Flush 0 Sent Done Done false
19150 5587 Acked Write 9 Sent Done Done false
19151 5588 Acked Read 4 Sent Done Done false
19152 5589 Acked Read 10 Sent Done Done false
19153 5590 Acked Write 5 Sent Done Done false
19154 5591 Acked Write 3 Sent Done Done false
19155 5592 Acked Write 5 Sent Done Done false
19156 5593 Acked Write 7 Sent Done Done false
19157 5594 Acked Read 7 Sent Done Done false
19158 5595 Acked Read 6 Sent Done Done false
19159 5596 Acked Flush 0 Sent Done Done false
19160 5597 Acked Read 10 Sent Done Done false
19161 5598 Acked Write 2 Sent Done Done false
19162 5599 Acked Read 9 Sent Done Done false
19163 5600 Acked Read 7 Sent Done Done false
19164 5601 Acked Write 6 Sent Done Done false
19165 5602 Acked Read 4 Sent Done Done false
19166 5603 Acked Write 7 Sent Done Done false
19167 5604 Acked Read 4 Sent Done Done false
19168 5605 Acked Read 7 Sent Done Done false
19169 5606 Acked Read 7 Sent Done Done false
19170 5607 Acked Read 1 Sent Done Done false
19171 5608 Acked Read 2 Sent Done Done false
19172 5609 Acked Write 7 Sent Done Done false
19173 5610 Acked Read 8 Sent Done Done false
19174 5611 Acked Write 5 Sent Done Done false
19175 5612 Acked Write 1 Sent Done Done false
19176 5613 Acked Read 10 Sent Done Done false
19177 5614 Acked Write 5 Sent Done Done false
19178 5615 Acked Read 7 Sent Done Done false
19179 5616 Acked Read 2 Sent Done Done false
19180 5617 Acked Read 6 Sent Done Done false
19181 5618 Acked Read 10 Sent Done Done false
19182 5619 Acked Write 2 Sent Done Done false
19183 5620 Acked Write 8 Sent Done Done false
19184 5621 Acked Read 4 Sent Done Done false
19185 5622 Acked Read 7 Sent Done Done false
19186 5623 Acked Read 8 Sent Done Done false
19187 5624 Acked Read 7 Sent Done Done false
19188 5625 Acked Read 4 Sent Done Done false
19189 5626 Acked Write 1 Sent Done Done false
19190 5627 Acked Write 10 Sent Done Done false
19191 5628 Acked Write 1 Sent Done Done false
19192 5629 Acked Read 7 Sent Done Done false
19193 5630 Acked Read 5 Sent Done Done false
19194 5631 Acked Read 3 Sent Done Done false
19195 5632 Acked Flush 0 Sent Done Done false
19196 5633 Acked Write 9 Sent Done Done false
19197 5634 Acked Read 2 Sent Done Done false
19198 5635 Acked Write 9 Sent Done Done false
19199 5636 Acked Read 2 Sent Done Done false
19200 5637 Acked Read 5 Sent Done Done false
19201 5638 Acked Write 6 Sent Done Done false
19202 5639 Acked Flush 0 Sent Done Done false
19203 5640 Acked Write 8 Sent Done Done false
19204 5641 Acked Read 7 Sent Done Done false
19205 5642 Acked Write 5 Sent Done Done false
19206 5643 Acked Flush 0 Sent Done Done false
19207 5644 Acked Read 3 Sent Done Done false
19208 5645 Acked Read 7 Sent Done Done false
19209 5646 Acked Write 10 Sent Done Done false
19210 5647 Acked Flush 0 Sent Done Done false
19211 5648 Acked Write 10 Sent Done Done false
19212 5649 Acked Read 5 Sent Done Done false
19213 5650 Acked Write 10 Sent Done Done false
19214 5651 Acked Write 5 Sent Done Done false
19215 5652 Acked Read 5 Sent Done Done false
19216 5653 Acked Write 9 Sent Done Done false
19217 5654 Acked Read 2 Sent Done Done false
19218 5655 Acked Write 7 Sent Done Done false
19219 5656 Acked Write 4 Sent Done Done false
19220 5657 Acked Write 7 Sent Done Done false
19221 5658 Acked Write 3 Sent Done Done false
19222 5659 Acked Write 8 Sent Done Done false
19223 5660 Acked Read 5 Sent Done Done false
19224 5661 Acked Write 2 Sent Done Done false
19225 5662 Acked Write 5 Sent Done Done false
19226 5663 Acked Flush 0 Sent Done Done false
19227 5664 Acked Write 1 Sent Done Done false
19228 5665 Acked Read 1 Sent Done Done false
19229 5666 Acked Write 10 Sent Done Done false
19230 5667 Acked Write 7 Sent Done Done false
19231 5668 Acked Write 1 Sent Done Done false
19232 5669 Acked Read 1 Sent Done Done false
19233 5670 Acked Write 5 Sent Done Done false
19234 5671 Acked Write 3 Sent Done Done false
19235 5672 Acked Read 10 Sent Done Done false
19236 5673 Acked Write 7 Sent Done Done false
19237 5674 Acked Read 3 Sent Done Done false
19238 5675 Acked Read 9 Sent Done Done false
19239 5676 Acked Write 6 Sent Done Done false
19240 5677 Acked Write 8 Sent Done Done false
19241 5678 Acked Flush 0 Sent Done Done false
19242 5679 Acked Write 2 Sent Done Done false
19243 5680 Acked Read 8 Sent Done Done false
19244 5681 Acked Read 2 Sent Done Done false
19245 5682 Acked Write 9 Sent Done Done false
19246 5683 Acked Write 1 Sent Done Done false
19247 5684 Acked Write 10 Sent Done Done false
19248 5685 Acked Read 5 Sent Done Done false
19249 5686 Acked Write 1 Sent Done Done false
19250 5687 Acked Write 4 Sent Done Done false
19251 5688 Acked Write 4 Sent Done Done false
19252 5689 Acked Read 8 Sent Done Done false
19253 5690 Acked Read 2 Sent Done Done false
19254 5691 Acked Read 3 Sent Done Done false
19255 5692 Acked Read 3 Sent Done Done false
19256 5693 Acked Read 7 Sent Done Done false
19257 5694 Acked Write 3 Sent Done Done false
19258 5695 Acked Read 6 Sent Done Done false
19259 5696 Acked Read 5 Sent Done Done false
19260 5697 Acked Flush 0 Sent Done Done false
19261 5698 Acked Write 7 Sent Done Done false
19262 5699 Acked Read 7 Sent Done Done false
19263 5700 Acked Write 4 Sent Done Done false
19264 5701 Acked Write 7 Sent Done Done false
19265 5702 Acked Flush 0 Sent Done Done false
19266 5703 Acked Read 9 Sent Done Done false
19267 5704 Acked Read 5 Sent Done Done false
19268 5705 Acked Write 2 Sent Done Done false
19269 5706 Acked Flush 0 Sent Done Done false
19270 5707 Acked Read 6 Sent Done Done false
19271 5708 Acked Read 6 Sent Done Done false
19272 5709 Acked Read 4 Sent Done Done false
19273 5710 Acked Read 4 Sent Done Done false
19274 5711 Acked Write 8 Sent Done Done false
19275 5712 Acked Read 8 Sent Done Done false
19276 5713 Acked Write 6 Sent Done Done false
19277 5714 Acked Flush 0 Sent Done Done false
19278 5715 Acked Flush 0 Sent Done Done false
19279 5716 Acked Read 8 Sent Done Done false
19280 5717 Acked Read 1 Sent Done Done false
19281 5718 Acked Write 6 Sent Done Done false
19282 5719 Acked Write 10 Sent Done Done false
19283 5720 Acked Write 9 Sent Done Done false
19284 5721 Acked Read 2 Sent Done Done false
19285 5722 Acked Read 1 Sent Done Done false
19286 5723 Acked Write 2 Sent Done Done false
19287 5724 Acked Write 5 Sent Done Done false
19288 5725 Acked Read 4 Sent Done Done false
19289 5726 Acked Write 6 Sent Done Done false
19290 5727 Acked Read 5 Sent Done Done false
19291 5728 Acked Read 2 Sent Done Done false
19292 5729 Acked Write 9 Sent Done Done false
19293 5730 Acked Read 8 Sent Done Done false
19294 5731 Acked Read 5 Sent Done Done false
19295 5732 Acked Flush 0 Sent Done Done false
19296 5733 Acked Read 1 Sent Done Done false
19297 5734 Acked Read 7 Sent Done Done false
19298 5735 Acked Read 3 Sent Done Done false
19299 5736 Acked Write 10 Sent Done Done false
19300 5737 Acked Write 3 Sent Done Done false
19301 5738 Acked Read 10 Sent Done Done false
19302 5739 Acked Write 4 Sent Done Done false
19303 5740 Acked Write 5 Sent Done Done false
19304 5741 Acked Read 2 Sent Done Done false
19305 5742 Acked Read 5 Sent Done Done false
19306 5743 Acked Read 4 Sent Done Done false
19307 5744 Acked Read 4 Sent Done Done false
19308 5745 Acked Read 10 Sent Done Done false
19309 5746 Acked Write 2 Sent Done Done false
19310 5747 Acked Read 10 Sent Done Done false
19311 5748 Acked Write 9 Sent Done Done false
19312 5749 Acked Write 8 Sent Done Done false
19313 5750 Acked Read 9 Sent Done Done false
19314 5751 Acked Read 8 Sent Done Done false
19315 5752 Acked Read 10 Sent Done Done false
19316 5753 Acked Read 1 Sent Done Done false
19317 5754 Acked Read 6 Sent Done Done false
19318 5755 Acked Read 5 Sent Done Done false
19319 5756 Acked Write 5 Sent Done Done false
19320 5757 Acked Read 7 Sent Done Done false
19321 5758 Acked Write 6 Sent Done Done false
19322 5759 Acked Read 5 Sent Done Done false
19323 5760 Acked Write 5 Sent Done Done false
19324 5761 Acked Read 1 Sent Done Done false
19325 5762 Acked Read 10 Sent Done Done false
19326 5763 Acked Write 2 Sent Done Done false
19327 5764 Acked Write 5 Sent Done Done false
19328 5765 Acked Read 7 Sent Done Done false
19329 5766 Acked Read 10 Sent Done Done false
19330 5767 Acked Read 5 Sent Done Done false
19331 5768 Acked Write 5 Sent Done Done false
19332 5769 Acked Read 9 Sent Done Done false
19333 5770 Acked Read 6 Sent Done Done false
19334 5771 Acked Read 6 Sent Done Done false
19335 5772 Acked Read 4 Sent Done Done false
19336 5773 Acked Flush 0 Sent Done Done false
19337 5774 Acked Read 7 Sent Done Done false
19338 5775 Acked Write 10 Sent Done Done false
19339 5776 Acked Write 3 Sent Done Done false
19340 5777 Acked Write 7 Sent Done Done false
19341 5778 Acked Read 3 Sent Done Done false
19342 5779 Acked Write 7 Sent Done Done false
19343 5780 Acked Flush 0 Sent Done Done false
19344 5781 Acked Read 10 Sent Done Done false
19345 5782 Acked Write 10 Sent Done Done false
19346 5783 Acked Write 10 Sent Done Done false
19347 5784 Acked Read 9 Sent Done Done false
19348 5785 Acked Write 5 Sent Done Done false
19349 5786 Acked Write 4 Sent Done Done false
19350 5787 Acked Write 2 Sent Done Done false
19351 5788 Acked Flush 0 Sent Done Done false
19352 5789 Acked Flush 0 Sent Done Done false
19353 5790 Acked Read 10 Sent Done Done false
19354 5791 Acked Write 10 Sent Done Done false
19355 5792 Acked Write 8 Sent Done Done false
19356 5793 Acked Write 10 Sent Done Done false
19357 5794 Acked Write 9 Sent Done Done false
19358 5795 Acked Write 5 Sent Done Done false
19359 5796 Acked Read 3 Sent Done Done false
19360 5797 Acked Read 3 Sent Done Done false
19361 5798 Acked Read 2 Sent Done Done false
19362 5799 Acked Write 4 Sent Done Done false
19363 5800 Acked Read 3 Sent Done Done false
19364 5801 Acked Read 9 Sent Done Done false
19365 5802 Acked Write 5 Sent Done Done false
19366 5803 Acked Write 9 Sent Done Done false
19367 5804 Acked Read 5 Sent Done Done false
19368 5805 Acked Write 10 Sent Done Done false
19369 5806 Acked Read 2 Sent Done Done false
19370 5807 Acked Read 2 Sent Done Done false
19371 5808 Acked Read 1 Sent Done Done false
19372 5809 Acked Write 10 Sent Done Done false
19373 5810 Acked Read 8 Sent Done Done false
19374 5811 Acked Flush 0 Sent Done Done false
19375 5812 Acked Read 2 Sent Done Done false
19376 5813 Acked Read 8 Sent Done Done false
19377 5814 Acked Flush 0 Sent Done Done false
19378 5815 Acked Read 3 Sent Done Done false
19379 5816 Acked Write 7 Sent Done Done false
19380 5817 Acked Read 5 Sent Done Done false
19381 5818 Acked Write 7 Sent Done Done false
19382 5819 Acked Write 10 Sent Done Done false
19383 5820 Acked Write 1 Sent Done Done false
19384 5821 Acked Write 3 Sent Done Done false
19385 5822 Acked Write 3 Sent Done Done false
19386 5823 Acked Read 8 Sent Done Done false
19387 5824 Acked Write 9 Sent Done Done false
19388 5825 Acked Read 1 Sent Done Done false
19389 5826 Acked Write 5 Sent Done Done false
19390 5827 Acked Write 5 Sent Done Done false
19391 5828 Acked Read 8 Sent Done Done false
19392 STATES DS:0 DS:1 DS:2 TOTAL
19393 Sent 301 0 0 301
19394 Done 0 301 301 602
19395 Skipped 0 0 0 0
19396 Error 0 0 0 0
19397 Last Flush: Some(JobId(5527)) Some(JobId(5814)) Some(JobId(5814))
19398 Downstairs last five completed: 5527 5526 5525 5524 5523
19399 Upstairs last five completed: 5828 5827 5826 5825 5824
19400 CLIENT: Up:0 ds:301 act:2
194012025-03-07T18:16:49.369ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
19402 [8810] got start action
19403 [8810] I am going to restart
19404 Starting downstairs at port 8810
19405 Make output file at "/tmp/dsc/downstairs-8810.txt"
19406 [8810][0] reports Starting
19407 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1224)
19408 [8810][0] reports Running
194092025-03-07T18:16:50.370ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
194102025-03-07T18:16:52.375ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
19411 ----------------------------------------------------------------
19412 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19413 States: Offline Active Active
19414 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19415 5528 Acked Read 5 Sent Done Done false
19416 5529 Acked Read 7 Sent Done Done false
19417 5530 Acked Write 9 Sent Done Done false
19418 5531 Acked Write 8 Sent Done Done false
19419 5532 Acked Read 4 Sent Done Done false
19420 5533 Acked Write 6 Sent Done Done false
19421 5534 Acked Write 4 Sent Done Done false
19422 5535 Acked Write 10 Sent Done Done false
19423 5536 Acked Read 3 Sent Done Done false
19424 5537 Acked Read 3 Sent Done Done false
19425 5538 Acked Read 1 Sent Done Done false
19426 5539 Acked Read 3 Sent Done Done false
19427 5540 Acked Write 10 Sent Done Done false
19428 5541 Acked Flush 0 Sent Done Done false
19429 5542 Acked Write 5 Sent Done Done false
19430 5543 Acked Write 1 Sent Done Done false
19431 5544 Acked Read 1 Sent Done Done false
19432 5545 Acked Write 3 Sent Done Done false
19433 5546 Acked Write 3 Sent Done Done false
19434 5547 Acked Write 9 Sent Done Done false
19435 5548 Acked Write 6 Sent Done Done false
19436 5549 Acked Write 6 Sent Done Done false
19437 5550 Acked Flush 0 Sent Done Done false
19438 5551 Acked Read 1 Sent Done Done false
19439 5552 Acked Flush 0 Sent Done Done false
19440 5553 Acked Read 7 Sent Done Done false
19441 5554 Acked Write 5 Sent Done Done false
19442 5555 Acked Flush 0 Sent Done Done false
19443 5556 Acked Flush 0 Sent Done Done false
19444 5557 Acked Read 1 Sent Done Done false
19445 5558 Acked Write 3 Sent Done Done false
19446 5559 Acked Flush 0 Sent Done Done false
19447 5560 Acked Write 5 Sent Done Done false
19448 5561 Acked Read 10 Sent Done Done false
19449 5562 Acked Write 8 Sent Done Done false
19450 5563 Acked Read 3 Sent Done Done false
19451 5564 Acked Read 3 Sent Done Done false
19452 5565 Acked Flush 0 Sent Done Done false
19453 5566 Acked Write 10 Sent Done Done false
19454 5567 Acked Flush 0 Sent Done Done false
19455 5568 Acked Write 10 Sent Done Done false
19456 5569 Acked Write 9 Sent Done Done false
19457 5570 Acked Read 10 Sent Done Done false
19458 5571 Acked Read 1 Sent Done Done false
19459 5572 Acked Read 2 Sent Done Done false
19460 5573 Acked Write 10 Sent Done Done false
19461 5574 Acked Read 2 Sent Done Done false
19462 5575 Acked Read 1 Sent Done Done false
19463 5576 Acked Write 3 Sent Done Done false
19464 5577 Acked Read 2 Sent Done Done false
19465 5578 Acked Read 8 Sent Done Done false
19466 5579 Acked Write 8 Sent Done Done false
19467 5580 Acked Read 4 Sent Done Done false
19468 5581 Acked Write 1 Sent Done Done false
19469 5582 Acked Read 5 Sent Done Done false
19470 5583 Acked Read 8 Sent Done Done false
19471 5584 Acked Write 2 Sent Done Done false
19472 5585 Acked Read 4 Sent Done Done false
19473 5586 Acked Flush 0 Sent Done Done false
19474 5587 Acked Write 9 Sent Done Done false
19475 5588 Acked Read 4 Sent Done Done false
19476 5589 Acked Read 10 Sent Done Done false
19477 5590 Acked Write 5 Sent Done Done false
19478 5591 Acked Write 3 Sent Done Done false
19479 5592 Acked Write 5 Sent Done Done false
19480 5593 Acked Write 7 Sent Done Done false
19481 5594 Acked Read 7 Sent Done Done false
19482 5595 Acked Read 6 Sent Done Done false
19483 5596 Acked Flush 0 Sent Done Done false
19484 5597 Acked Read 10 Sent Done Done false
19485 5598 Acked Write 2 Sent Done Done false
19486 5599 Acked Read 9 Sent Done Done false
19487 5600 Acked Read 7 Sent Done Done false
19488 5601 Acked Write 6 Sent Done Done false
19489 5602 Acked Read 4 Sent Done Done false
19490 5603 Acked Write 7 Sent Done Done false
19491 5604 Acked Read 4 Sent Done Done false
19492 5605 Acked Read 7 Sent Done Done false
19493 5606 Acked Read 7 Sent Done Done false
19494 5607 Acked Read 1 Sent Done Done false
19495 5608 Acked Read 2 Sent Done Done false
19496 5609 Acked Write 7 Sent Done Done false
19497 5610 Acked Read 8 Sent Done Done false
19498 5611 Acked Write 5 Sent Done Done false
19499 5612 Acked Write 1 Sent Done Done false
19500 5613 Acked Read 10 Sent Done Done false
19501 5614 Acked Write 5 Sent Done Done false
19502 5615 Acked Read 7 Sent Done Done false
19503 5616 Acked Read 2 Sent Done Done false
19504 5617 Acked Read 6 Sent Done Done false
19505 5618 Acked Read 10 Sent Done Done false
19506 5619 Acked Write 2 Sent Done Done false
19507 5620 Acked Write 8 Sent Done Done false
19508 5621 Acked Read 4 Sent Done Done false
19509 5622 Acked Read 7 Sent Done Done false
19510 5623 Acked Read 8 Sent Done Done false
19511 5624 Acked Read 7 Sent Done Done false
19512 5625 Acked Read 4 Sent Done Done false
19513 5626 Acked Write 1 Sent Done Done false
19514 5627 Acked Write 10 Sent Done Done false
19515 5628 Acked Write 1 Sent Done Done false
19516 5629 Acked Read 7 Sent Done Done false
19517 5630 Acked Read 5 Sent Done Done false
19518 5631 Acked Read 3 Sent Done Done false
19519 5632 Acked Flush 0 Sent Done Done false
19520 5633 Acked Write 9 Sent Done Done false
19521 5634 Acked Read 2 Sent Done Done false
19522 5635 Acked Write 9 Sent Done Done false
19523 5636 Acked Read 2 Sent Done Done false
19524 5637 Acked Read 5 Sent Done Done false
19525 5638 Acked Write 6 Sent Done Done false
19526 5639 Acked Flush 0 Sent Done Done false
19527 5640 Acked Write 8 Sent Done Done false
19528 5641 Acked Read 7 Sent Done Done false
19529 5642 Acked Write 5 Sent Done Done false
19530 5643 Acked Flush 0 Sent Done Done false
19531 5644 Acked Read 3 Sent Done Done false
19532 5645 Acked Read 7 Sent Done Done false
19533 5646 Acked Write 10 Sent Done Done false
19534 5647 Acked Flush 0 Sent Done Done false
19535 5648 Acked Write 10 Sent Done Done false
19536 5649 Acked Read 5 Sent Done Done false
19537 5650 Acked Write 10 Sent Done Done false
19538 5651 Acked Write 5 Sent Done Done false
19539 5652 Acked Read 5 Sent Done Done false
19540 5653 Acked Write 9 Sent Done Done false
19541 5654 Acked Read 2 Sent Done Done false
19542 5655 Acked Write 7 Sent Done Done false
19543 5656 Acked Write 4 Sent Done Done false
19544 5657 Acked Write 7 Sent Done Done false
19545 5658 Acked Write 3 Sent Done Done false
19546 5659 Acked Write 8 Sent Done Done false
19547 5660 Acked Read 5 Sent Done Done false
19548 5661 Acked Write 2 Sent Done Done false
19549 5662 Acked Write 5 Sent Done Done false
19550 5663 Acked Flush 0 Sent Done Done false
19551 5664 Acked Write 1 Sent Done Done false
19552 5665 Acked Read 1 Sent Done Done false
19553 5666 Acked Write 10 Sent Done Done false
19554 5667 Acked Write 7 Sent Done Done false
19555 5668 Acked Write 1 Sent Done Done false
19556 5669 Acked Read 1 Sent Done Done false
19557 5670 Acked Write 5 Sent Done Done false
19558 5671 Acked Write 3 Sent Done Done false
19559 5672 Acked Read 10 Sent Done Done false
19560 5673 Acked Write 7 Sent Done Done false
19561 5674 Acked Read 3 Sent Done Done false
19562 5675 Acked Read 9 Sent Done Done false
19563 5676 Acked Write 6 Sent Done Done false
19564 5677 Acked Write 8 Sent Done Done false
19565 5678 Acked Flush 0 Sent Done Done false
19566 5679 Acked Write 2 Sent Done Done false
19567 5680 Acked Read 8 Sent Done Done false
19568 5681 Acked Read 2 Sent Done Done false
19569 5682 Acked Write 9 Sent Done Done false
19570 5683 Acked Write 1 Sent Done Done false
19571 5684 Acked Write 10 Sent Done Done false
19572 5685 Acked Read 5 Sent Done Done false
19573 5686 Acked Write 1 Sent Done Done false
19574 5687 Acked Write 4 Sent Done Done false
19575 5688 Acked Write 4 Sent Done Done false
19576 5689 Acked Read 8 Sent Done Done false
19577 5690 Acked Read 2 Sent Done Done false
19578 5691 Acked Read 3 Sent Done Done false
19579 5692 Acked Read 3 Sent Done Done false
19580 5693 Acked Read 7 Sent Done Done false
19581 5694 Acked Write 3 Sent Done Done false
19582 5695 Acked Read 6 Sent Done Done false
19583 5696 Acked Read 5 Sent Done Done false
19584 5697 Acked Flush 0 Sent Done Done false
19585 5698 Acked Write 7 Sent Done Done false
19586 5699 Acked Read 7 Sent Done Done false
19587 5700 Acked Write 4 Sent Done Done false
19588 5701 Acked Write 7 Sent Done Done false
19589 5702 Acked Flush 0 Sent Done Done false
19590 5703 Acked Read 9 Sent Done Done false
19591 5704 Acked Read 5 Sent Done Done false
19592 5705 Acked Write 2 Sent Done Done false
19593 5706 Acked Flush 0 Sent Done Done false
19594 5707 Acked Read 6 Sent Done Done false
19595 5708 Acked Read 6 Sent Done Done false
19596 5709 Acked Read 4 Sent Done Done false
19597 5710 Acked Read 4 Sent Done Done false
19598 5711 Acked Write 8 Sent Done Done false
19599 5712 Acked Read 8 Sent Done Done false
19600 5713 Acked Write 6 Sent Done Done false
19601 5714 Acked Flush 0 Sent Done Done false
19602 5715 Acked Flush 0 Sent Done Done false
19603 5716 Acked Read 8 Sent Done Done false
19604 5717 Acked Read 1 Sent Done Done false
19605 5718 Acked Write 6 Sent Done Done false
19606 5719 Acked Write 10 Sent Done Done false
19607 5720 Acked Write 9 Sent Done Done false
19608 5721 Acked Read 2 Sent Done Done false
19609 5722 Acked Read 1 Sent Done Done false
19610 5723 Acked Write 2 Sent Done Done false
19611 5724 Acked Write 5 Sent Done Done false
19612 5725 Acked Read 4 Sent Done Done false
19613 5726 Acked Write 6 Sent Done Done false
19614 5727 Acked Read 5 Sent Done Done false
19615 5728 Acked Read 2 Sent Done Done false
19616 5729 Acked Write 9 Sent Done Done false
19617 5730 Acked Read 8 Sent Done Done false
19618 5731 Acked Read 5 Sent Done Done false
19619 5732 Acked Flush 0 Sent Done Done false
19620 5733 Acked Read 1 Sent Done Done false
19621 5734 Acked Read 7 Sent Done Done false
19622 5735 Acked Read 3 Sent Done Done false
19623 5736 Acked Write 10 Sent Done Done false
19624 5737 Acked Write 3 Sent Done Done false
19625 5738 Acked Read 10 Sent Done Done false
19626 5739 Acked Write 4 Sent Done Done false
19627 5740 Acked Write 5 Sent Done Done false
19628 5741 Acked Read 2 Sent Done Done false
19629 5742 Acked Read 5 Sent Done Done false
19630 5743 Acked Read 4 Sent Done Done false
19631 5744 Acked Read 4 Sent Done Done false
19632 5745 Acked Read 10 Sent Done Done false
19633 5746 Acked Write 2 Sent Done Done false
19634 5747 Acked Read 10 Sent Done Done false
19635 5748 Acked Write 9 Sent Done Done false
19636 5749 Acked Write 8 Sent Done Done false
19637 5750 Acked Read 9 Sent Done Done false
19638 5751 Acked Read 8 Sent Done Done false
19639 5752 Acked Read 10 Sent Done Done false
19640 5753 Acked Read 1 Sent Done Done false
19641 5754 Acked Read 6 Sent Done Done false
19642 5755 Acked Read 5 Sent Done Done false
19643 5756 Acked Write 5 Sent Done Done false
19644 5757 Acked Read 7 Sent Done Done false
19645 5758 Acked Write 6 Sent Done Done false
19646 5759 Acked Read 5 Sent Done Done false
19647 5760 Acked Write 5 Sent Done Done false
19648 5761 Acked Read 1 Sent Done Done false
19649 5762 Acked Read 10 Sent Done Done false
19650 5763 Acked Write 2 Sent Done Done false
19651 5764 Acked Write 5 Sent Done Done false
19652 5765 Acked Read 7 Sent Done Done false
19653 5766 Acked Read 10 Sent Done Done false
19654 5767 Acked Read 5 Sent Done Done false
19655 5768 Acked Write 5 Sent Done Done false
19656 5769 Acked Read 9 Sent Done Done false
19657 5770 Acked Read 6 Sent Done Done false
19658 5771 Acked Read 6 Sent Done Done false
19659 5772 Acked Read 4 Sent Done Done false
19660 5773 Acked Flush 0 Sent Done Done false
19661 5774 Acked Read 7 Sent Done Done false
19662 5775 Acked Write 10 Sent Done Done false
19663 5776 Acked Write 3 Sent Done Done false
19664 5777 Acked Write 7 Sent Done Done false
19665 5778 Acked Read 3 Sent Done Done false
19666 5779 Acked Write 7 Sent Done Done false
19667 5780 Acked Flush 0 Sent Done Done false
19668 5781 Acked Read 10 Sent Done Done false
19669 5782 Acked Write 10 Sent Done Done false
19670 5783 Acked Write 10 Sent Done Done false
19671 5784 Acked Read 9 Sent Done Done false
19672 5785 Acked Write 5 Sent Done Done false
19673 5786 Acked Write 4 Sent Done Done false
19674 5787 Acked Write 2 Sent Done Done false
19675 5788 Acked Flush 0 Sent Done Done false
19676 5789 Acked Flush 0 Sent Done Done false
19677 5790 Acked Read 10 Sent Done Done false
19678 5791 Acked Write 10 Sent Done Done false
19679 5792 Acked Write 8 Sent Done Done false
19680 5793 Acked Write 10 Sent Done Done false
19681 5794 Acked Write 9 Sent Done Done false
19682 5795 Acked Write 5 Sent Done Done false
19683 5796 Acked Read 3 Sent Done Done false
19684 5797 Acked Read 3 Sent Done Done false
19685 5798 Acked Read 2 Sent Done Done false
19686 5799 Acked Write 4 Sent Done Done false
19687 5800 Acked Read 3 Sent Done Done false
19688 5801 Acked Read 9 Sent Done Done false
19689 5802 Acked Write 5 Sent Done Done false
19690 5803 Acked Write 9 Sent Done Done false
19691 5804 Acked Read 5 Sent Done Done false
19692 5805 Acked Write 10 Sent Done Done false
19693 5806 Acked Read 2 Sent Done Done false
19694 5807 Acked Read 2 Sent Done Done false
19695 5808 Acked Read 1 Sent Done Done false
19696 5809 Acked Write 10 Sent Done Done false
19697 5810 Acked Read 8 Sent Done Done false
19698 5811 Acked Flush 0 Sent Done Done false
19699 5812 Acked Read 2 Sent Done Done false
19700 5813 Acked Read 8 Sent Done Done false
19701 5814 Acked Flush 0 Sent Done Done false
19702 5815 Acked Read 3 Sent Done Done false
19703 5816 Acked Write 7 Sent Done Done false
19704 5817 Acked Read 5 Sent Done Done false
19705 5818 Acked Write 7 Sent Done Done false
19706 5819 Acked Write 10 Sent Done Done false
19707 5820 Acked Write 1 Sent Done Done false
19708 5821 Acked Write 3 Sent Done Done false
19709 5822 Acked Write 3 Sent Done Done false
19710 5823 Acked Read 8 Sent Done Done false
19711 5824 Acked Write 9 Sent Done Done false
19712 5825 Acked Read 1 Sent Done Done false
19713 5826 Acked Write 5 Sent Done Done false
19714 5827 Acked Write 5 Sent Done Done false
19715 5828 Acked Read 8 Sent Done Done false
19716 STATES DS:0 DS:1 DS:2 TOTAL
19717 Sent 301 0 0 301
19718 Done 0 301 301 602
19719 Skipped 0 0 0 0
19720 Error 0 0 0 0
19721 Last Flush: Some(JobId(5527)) Some(JobId(5814)) Some(JobId(5814))
19722 Downstairs last five completed: 5527 5526 5525 5524 5523
19723 Upstairs last five completed: 5828 5827 5826 5825 5824
19724 CLIENT: Up:0 ds:301 act:2
197252025-03-07T18:16:53.377ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197262025-03-07T18:16:54.355ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197272025-03-07T18:16:54.355ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44097) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197282025-03-07T18:16:54.356ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197292025-03-07T18:16:54.356ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197302025-03-07T18:16:54.356ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197312025-03-07T18:16:54.356ZINFOcrucible: send last flush ID to this DS: Some(JobId(5527)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197322025-03-07T18:16:54.356ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197332025-03-07T18:16:54.356ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5527)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
197342025-03-07T18:16:54.356ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5527)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
19735 ----------------------------------------------------------------
19736 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19737 Downstairs last five completed: 5829 5828 5827 5826 5825
19738 Upstairs last five completed: 5829 5828 5827 5826 5825
19739 CLIENT: Up:0 ds:0 act:3
19740 Replay: All jobs finished
19741 Main task has work to do, go find it
19742 got dsc Stop(0)
19743 stop 0
19744 Mar 07 18:16:56.607 INFO request completed[8810] Got stop action so:false kr:false
19745 , latency_us: 353, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 08ea51b3-bb6e-4ac3-b571-85b0194ce994, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
197462025-03-07T18:16:56.607ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
19747 Mar 07 18:16:56.607{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:16:56.607840472Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
19748 INFO request completed, latency_us: 128, response_code:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:16:56.607862889Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
19749 200, uri: /state/cid/0, method: GET, req_id{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:16:56.607922792Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
19750 : 6ba3464a-0c43-4b35-a6a5-59c8e5958af8, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
19751 [8810] Exited with: ExitStatus(unix_wait_status(9))
19752 [8810][0] reports Exit
19753 Mar 07 18:17:00.610 INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/0, method: GET, req_id: c6d724c0-6f96-4c96-8a7a-e44d7bb77b3f, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
19754 001/300 Read block 3052 len 40960
19755 002/300 Write block 2944 len 20480 data: 6 6 5 5 5
19756 003/300 Write block 553 len 8192 data: 2 2
19757 004/300 Write block 98 len 32768 data: 3 3 3 3 3 3 3 3
19758 005/300 Read block 2227 len 36864
19759 006/300 Write block 2195 len 16384 data: 2 3 3 5
19760 007/300 Write block 4073 len 28672 data: 6 6 8 6 5 5 5
19761 008/300 Read block 157 len 12288
19762 009/300 Write block 4279 len 4096 data: 8
19763 010/300 Write block 3959 len 24576 data: 5 5 4 4 4 4
19764 011/300 Write block 4593 len 32768 data: 3 3 3 3 5 5 5 6
19765 012/300 Write block 1714 len 32768 data: 4 5 5 5 6 7 7 7
19766 013/300 Write block 2090 len 4096 data: 3
19767 014/300 Flush
19768 015/300 Flush
19769 016/300 Read block 4586 len 32768
19770 017/300 Write block 252 len 40960 data: 3 3 4 5 5 3 3 3 3 4
19771 018/300 Write block 4383 len 8192 data: 3 4
19772 019/300 Write block 3003 len 32768 data: 4 5 5 4 5 5 7 7
19773 020/300 Flush
19774 021/300 Write block 1682 len 20480 data: 2 2 2 3 3
19775 022/300 Read block 1439 len 4096
19776 023/300 Write block 303 len 16384 data: 4 4 5 5
19777 024/300 Read block 2074 len 24576
19778 025/300 Write block 1211 len 16384 data: 6 5 4 5
19779 026/300 Read block 3340 len 40960
19780 027/300 Read block 1658 len 24576
19781 028/300 Write block 3211 len 36864 data: 2 3 3 3 3 3 3 3 5
19782 029/300 Write block 4824 len 12288 data: 3 4 4
19783 030/300 Flush
19784 031/300 Read block 3611 len 24576
19785 032/300 Read block 2586 len 28672
19786 033/300 Read block 3713 len 20480
19787 034/300 Write block 2613 len 8192 data: 2 2
19788 035/300 Write block 4920 len 4096 data: 6
19789 036/300 Write block 3705 len 24576 data: 4 4 4 3 3 2
19790 037/300 Read block 2521 len 24576
19791 038/300 Write block 2835 len 28672 data: 2 2 3 3 4 5 5
19792 039/300 Read block 4752 len 4096
19793 040/300 Read block 97 len 32768
19794 041/300 Read block 907 len 12288
19795 042/300 Write block 1069 len 8192 data: 4 3
19796 043/300 Write block 4173 len 20480 data: 4 4 3 3 3
19797 044/300 Write block 3249 len 36864 data: 2 3 4 4 3 3 3 3 3
19798 045/300 Read block 4962 len 12288
19799 046/300 Read block 456 len 28672
19800 047/300 Read block 3417 len 16384
19801 048/300 Read block 4725 len 28672
19802 049/300 Read block 4122 len 28672
19803 050/300 Write block 1803 len 4096 data: 6
19804 051/300 Read block 2954 len 40960
19805 052/300 Read block 2077 len 16384
19806 053/300 Read block 3870 len 8192
19807 054/300 Read block 1532 len 20480
19808 055/300 Flush
19809 056/300 Write block 2913 len 40960 data: 4 6 5 5 5 6 4 4 7 7
19810 057/300 Read block 3837 len 32768
19811 058/300 Write block 4156 len 8192 data: 3 3
19812 059/300 Read block 2589 len 16384
19813 060/300 Write block 1573 len 8192 data: 5 5
19814 061/300 Write block 2617 len 8192 data: 3 4
19815 062/300 Read block 3352 len 4096
19816 063/300 Read block 4934 len 24576
19817 064/300 Read block 2457 len 4096
19818 065/300 Write block 2966 len 40960 data: 3 2 3 3 3 3 3 3 3 5
19819 066/300 Read block 3983 len 16384
19820 067/300 Read block 3599 len 40960
19821 068/300 Read block 2955 len 12288
19822 069/300 Read block 3689 len 24576
19823 070/300 Read block 3217 len 28672
19824 071/300 Read block 4903 len 40960
19825 072/300 Read block 3212 len 36864
19826 073/300 Write block 217 len 24576 data: 4 3 4 5 5 5
19827 074/300 Write block 2585 len 16384 data: 3 5 4 5
19828 075/300 Write block 3732 len 24576 data: 3 4 4 3 3 3
19829 076/300 Flush
19830 077/300 Flush
19831 078/300 Write block 1246 len 40960 data: 4 4 4 4 4 6 4 5 3 3
19832 079/300 Read block 254 len 12288
19833 080/300 Write block 2915 len 20480 data: 6 6 6 7 5
19834 081/300 Write block 1336 len 20480 data: 3 3 3 3 3
19835 082/300 Read block 1606 len 32768
19836 083/300 Read block 2764 len 4096
19837 084/300 Write block 4478 len 12288 data: 2 2 2
19838 085/300 Read block 948 len 32768
19839 086/300 Write block 3016 len 16384 data: 5 4 4 4
19840 087/300 Write block 3442 len 24576 data: 6 5 6 7 4 4
19841 088/300 Read block 4903 len 16384
19842 089/300 Read block 781 len 28672
19843 090/300 Write block 1736 len 28672 data: 4 3 3 3 3 3 5
19844 091/300 Read block 4440 len 36864
19845 092/300 Flush
19846 093/300 Write block 2310 len 20480 data: 3 3 2 2 4
19847 094/300 Write block 696 len 8192 data: 5 5
19848 095/300 Read block 3899 len 36864
19849 096/300 Read block 4271 len 12288
19850 097/300 Read block 2217 len 8192
19851 098/300 Write block 4124 len 8192 data: 7 8
19852 099/300 Read block 394 len 24576
19853 100/300 Write block 174 len 36864 data: 4 4 3 4 4 5 5 4 3
19854 101/300 Write block 1280 len 24576 data: 3 2 2 2 3 3
19855 102/300 Flush
19856 103/300 Write block 3084 len 4096 data: 9
19857 104/300 Write block 3692 len 20480 data: 2 2 3 4 4
19858 105/300 Read block 1550 len 4096
19859 106/300 Read block 2768 len 28672
19860 107/300 Read block 35 len 12288
19861 108/300 Write block 3182 len 8192 data: 2 2
19862 109/300 Read block 4317 len 28672
19863 110/300 Read block 2718 len 36864
19864 111/300 Write block 3062 len 32768 data: 5 8 7 7 7 6 6 6
19865 112/300 Write block 3972 len 16384 data: 3 3 2 2
19866 113/300 Write block 2122 len 20480 data: 5 5 4 4 4
19867 114/300 Read block 142 len 40960
19868 115/300 Write block 1205 len 40960 data: 3 5 5 5 6 6 7 6 5 6
19869 116/300 Read block 2009 len 20480
19870 117/300 Write block 573 len 8192 data: 3 3
19871 118/300 Read block 1779 len 24576
19872 119/300 Read block 4733 len 24576
19873 120/300 Flush
19874 121/300 Flush
19875 122/300 Read block 1699 len 4096
19876 123/300 Read block 4616 len 28672
19877 124/300 Read block 2664 len 32768
19878 125/300 Read block 2442 len 28672
19879 126/300 Read block 4275 len 36864
19880 127/300 Flush
19881 128/300 Read block 3576 len 24576
19882 129/300 Write block 3569 len 4096 data: 5
19883 130/300 Flush
19884 131/300 Write block 1240 len 20480 data: 2 2 2 2 4
19885 132/300 Write block 3931 len 28672 data: 5 6 4 5 5 4 4
19886 133/300 Read block 4674 len 36864
19887 134/300 Write block 1568 len 8192 data: 8 8
19888 135/300 Write block 1151 len 8192 data: 3 4
19889 136/300 Write block 3755 len 4096 data: 5
19890 137/300 Read block 4938 len 4096
19891 138/300 Write block 2221 len 24576 data: 4 4 4 4 4 5
19892 139/300 Read block 1724 len 40960
19893 140/300 Read block 847 len 8192
19894 141/300 Write block 3284 len 20480 data: 3 4 4 4 4
19895 142/300 Write block 1539 len 36864 data: 5 5 5 4 4 5 5 6 6
19896 143/300 Read block 4871 len 12288
19897 144/300 Write block 4309 len 36864 data: 3 3 4 4 5 5 5 5 5
19898 145/300 Write block 302 len 28672 data: 5 5 5 6 6 7 7
19899 146/300 Write block 4933 len 28672 data: 2 3 2 2 2 4 3
19900 147/300 Write block 2732 len 8192 data: 8 7
19901 148/300 Read block 1626 len 32768
19902 149/300 Write block 3827 len 16384 data: 3 3 4 4
19903 150/300 Read block 3489 len 16384
19904 151/300 Write block 1518 len 8192 data: 4 3
19905 152/300 Read block 143 len 4096
19906 153/300 Write block 199 len 4096 data: 5
19907 154/300 Flush
19908 155/300 Write block 351 len 8192 data: 7 8
19909 156/300 Write block 2233 len 32768 data: 2 2 2 2 3 3 4 4
19910 157/300 Read block 4192 len 12288
19911 158/300 Read block 1100 len 28672
19912 159/300 Write block 4172 len 28672 data: 6 5 5 4 4 4 3
19913 160/300 Flush
19914 161/300 Write block 1105 len 16384 data: 5 4 4 3
19915 162/300 Read block 2789 len 12288
19916 163/300 Read block 3736 len 16384
19917 164/300 Write block 806 len 16384 data: 4 4 4 4
19918 165/300 Write block 2096 len 16384 data: 2 2 2 2
19919 166/300 Read block 1685 len 24576
19920 167/300 Write block 3302 len 32768 data: 3 4 4 5 5 5 4 4
19921 168/300 Read block 1343 len 4096
19922 169/300 Read block 1654 len 8192
19923 170/300 Write block 921 len 24576 data: 2 3 3 3 3 3
19924 171/300 Read block 109 len 36864
19925 172/300 Read block 2632 len 40960
19926 173/300 Read block 770 len 4096
19927 174/300 Flush
19928 175/300 Write block 4120 len 16384 data: 5 5 7 7
19929 176/300 Write block 1339 len 4096 data: 4
19930 177/300 Write block 1119 len 16384 data: 5 5 7 7
19931 178/300 Read block 92 len 12288
19932 179/300 Write block 1530 len 4096 data: 4
19933 180/300 Write block 1287 len 32768 data: 4 3 3 4 5 6 5 6
19934 181/300 Read block 4288 len 36864
19935 182/300 Write block 4978 len 32768 data: 3 4 4 3 3 3 3 3
19936 183/300 Read block 771 len 4096
19937 184/300 Write block 2520 len 32768 data: 4 4 3 3 3 3 2 3
19938 185/300 Read block 4048 len 20480
19939 186/300 Write block 1460 len 12288 data: 5 7 8
19940 187/300 Read block 1328 len 24576
19941 188/300 Read block 4864 len 40960
19942 189/300 Read block 525 len 28672
19943 190/300 Read block 1434 len 40960
19944 191/300 Write block 3702 len 32768 data: 4 3 3 5 5 5 4 4
19945 192/300 Read block 4125 len 36864
19946 193/300 Flush
19947 194/300 Read block 3782 len 32768
19948 195/300 Flush
19949 196/300 Write block 3277 len 28672 data: 4 5 5 5 8 4 4
19950 197/300 Write block 3596 len 36864 data: 2 2 3 3 5 4 3 4 4
19951 198/300 Write block 261 len 20480 data: 5 4 5 5 6
19952 199/300 Write block 1359 len 24576 data: 4 6 7 7 6 5
19953 200/300 Read block 2900 len 16384
19954 201/300 Write block 1935 len 4096 data: 3
19955 202/300 Write block 2650 len 40960 data: 5 5 4 4 5 6 6 7 7 6
19956 203/300 Read block 4753 len 12288
19957 204/300 Write block 527 len 40960 data: 4 5 5 5 4 5 5 3 3 2
19958 205/300 Read block 1090 len 36864
19959 206/300 Write block 1917 len 28672 data: 5 4 3 4 4 6 6
19960 207/300 Write block 3282 len 32768 data: 5 5 4 5 5 5 5 4
19961 208/300 Write block 3240 len 40960 data: 3 4 4 4 4 4 4 4 3 3
19962 209/300 Read block 3545 len 16384
19963 210/300 Read block 3583 len 12288
19964 211/300 Write block 3793 len 32768 data: 4 4 5 5 6 5 3 4
19965 212/300 Flush
19966 213/300 Write block 2584 len 36864 data: 3 4 6 5 6 5 5 4 3
19967 214/300 Read block 4622 len 4096
19968 215/300 Read block 4163 len 24576
19969 216/300 Write block 375 len 4096 data: 3
19970 217/300 Read block 3580 len 16384
19971 218/300 Write block 1331 len 16384 data: 5 5 3 4
19972 219/300 Read block 140 len 12288
19973 220/300 Read block 1720 len 4096
19974 221/300 Read block 3869 len 32768
19975 222/300 Read block 1327 len 24576
19976 223/300 Read block 4653 len 16384
19977 224/300 Read block 2120 len 32768
19978 225/300 Read block 1 len 16384
19979 226/300 Read block 1978 len 36864
19980 227/300 Read block 359 len 28672
19981 228/300 Read block 2144 len 32768
19982 229/300 Read block 2426 len 12288
19983 230/300 Write block 2825 len 32768 data: 5 3 4 3 3 2 2 2
19984 231/300 Flush
19985 232/300 Write block 4911 len 24576 data: 5 5 5 6 6 7
19986 233/300 Write block 933 len 28672 data: 7 6 5 5 6 7 6
19987 234/300 Read block 3225 len 12288
19988 235/300 Write block 1044 len 28672 data: 2 2 3 3 3 3 3
19989 236/300 Flush
19990 237/300 Read block 4506 len 24576
19991 238/300 Write block 3196 len 28672 data: 4 4 4 5 5 6 6
19992 239/300 Read block 1531 len 16384
19993 240/300 Read block 1149 len 8192
19994 241/300 Read block 884 len 40960
19995 242/300 Write block 4126 len 28672 data: 6 7 8 7 6 4 2
19996 243/300 Read block 3980 len 28672
19997 244/300 Write block 4309 len 12288 data: 4 4 5
19998 245/300 Read block 3955 len 4096
19999 246/300 Read block 1223 len 24576
20000 247/300 Read block 979 len 4096
20001 248/300 Read block 794 len 32768
20002 249/300 Write block 1430 len 40960 data: 4 4 4 4 7 7 5 6 6 6
20003 250/300 Read block 2619 len 28672
20004 251/300 Write block 1085 len 20480 data: 4 4 3 3 3
20005 252/300 Read block 4809 len 36864
20006 253/300 Write block 3843 len 8192 data: 2 3
20007 254/300 Read block 3032 len 36864
20008 255/300 Write block 3035 len 8192 data: 3 3
20009 256/300 Read block 1665 len 12288
20010 257/300 Read block 1119 len 8192
20011 258/300 Read block 1128 len 32768
20012 259/300 Write block 2044 len 36864 data: 5 6 6 7 7 5 6 6 7
20013 260/300 Flush
20014 261/300 Write block 185 len 28672 data: 2 2 2 3 4 5 5
20015 262/300 Write block 1621 len 20480 data: 5 5 6 7 7
20016 263/300 Write block 4364 len 32768 data: 4 6 6 5 5 5 4 5
20017 264/300 Read block 2008 len 16384
20018 265/300 Write block 1674 len 12288 data: 4 3 3
20019 266/300 Write block 2112 len 28672 data: 6 6 7 7 6 5 6
20020 267/300 Read block 4975 len 36864
20021 268/300 Read block 4548 len 16384
20022 269/300 Flush
20023 270/300 Write block 3968 len 20480 data: 3 3 3 3 4
20024 271/300 Read block 3089 len 36864
20025 272/300 Read block 2035 len 40960
20026 273/300 Write block 545 len 8192 data: 3 3
20027 274/300 Write block 1196 len 8192 data: 4 3
20028 275/300 Read block 4669 len 8192
20029 276/300 Read block 3819 len 36864
20030 277/300 Read block 1998 len 28672
20031 278/300 Read block 624 len 20480
20032 279/300 Write block 3318 len 28672 data: 3 3 4 4 5 6 5
20033 280/300 Read block 3986 len 40960
20034 281/300 Write block 3162 len 12288 data: 2 2 2
20035 282/300 Read block 4005 len 12288
20036 283/300 Read block 2020 len 24576
20037 284/300 Write block 3984 len 12288 data: 6 3 3
20038 285/300 Write block 1798 len 32768 data: 3 3 3 5 6 7 6 6
20039 286/300 Flush
20040 287/300 Read block 1572 len 20480
20041 288/300 Read block 330 len 28672
20042 289/300 Write block 1196 len 8192 data: 5 4
20043 290/300 Write block 2735 len 24576 data: 6 6 7 7 8 7
20044 291/300 Read block 3953 len 40960
20045 292/300 Write block 2453 len 40960 data: 4 4 4 4 4 4 4 4 4 5
20046 293/300 Write block 1915 len 12288 data: 5 5 6
20047 294/300 Read block 4259 len 32768
20048 295/300 Write block 1917 len 40960 data: 7 5 4 5 5 7 7 6 5 5
20049 296/300 Read block 2054 len 8192
20050 297/300 Read block 202 len 28672
20051 298/300 Write block 1547 len 24576 data: 7 6 5 5 4 5
20052 299/300 Write block 243 len 16384 data: 4 5 5 4
20053 300/300 Flush
20054 301/300 Read block 4288 len 20480
20055 Main task has work to do, go find it
20056 got dsc Start(0)
20057 start 0
20058 Mar 07 18:17:00.856 INFO request completed, latency_us: 166, response_code: 204, uri: /start/cid/0, method: POST, req_id: 291accba-5ca3-4f97-8438-e1db07e1483e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
20059 Replay: started 0, returned:Ok(())
20060 ----------------------------------------------------------------
20061 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20062 States: Offline Active Active
20063 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20064 5830 Acked Read 10 Sent Done Done false
20065 5831 Acked Write 5 Sent Done Done false
20066 5832 Acked Write 2 Sent Done Done false
20067 5833 Acked Write 8 Sent Done Done false
20068 5834 Acked Read 9 Sent Done Done false
20069 5835 Acked Write 4 Sent Done Done false
20070 5836 Acked Write 7 Sent Done Done false
20071 5837 Acked Read 3 Sent Done Done false
20072 5838 Acked Write 1 Sent Done Done false
20073 5839 Acked Write 6 Sent Done Done false
20074 5840 Acked Write 8 Sent Done Done false
20075 5841 Acked Write 8 Sent Done Done false
20076 5842 Acked Write 1 Sent Done Done false
20077 5843 Acked Flush 0 Sent Done Done false
20078 5844 Acked Flush 0 Sent Done Done false
20079 5845 Acked Read 8 Sent Done Done false
20080 5846 Acked Write 10 Sent Done Done false
20081 5847 Acked Write 2 Sent Done Done false
20082 5848 Acked Write 8 Sent Done Done false
20083 5849 Acked Flush 0 Sent Done Done false
20084 5850 Acked Write 5 Sent Done Done false
20085 5851 Acked Read 1 Sent Done Done false
20086 5852 Acked Write 4 Sent Done Done false
20087 5853 Acked Read 6 Sent Done Done false
20088 5854 Acked Write 4 Sent Done Done false
20089 5855 Acked Read 10 Sent Done Done false
20090 5856 Acked Read 6 Sent Done Done false
20091 5857 Acked Write 9 Sent Done Done false
20092 5858 Acked Write 3 Sent Done Done false
20093 5859 Acked Flush 0 Sent Done Done false
20094 5860 Acked Read 6 Sent Done Done false
20095 5861 Acked Read 7 Sent Done Done false
20096 5862 Acked Read 5 Sent Done Done false
20097 5863 Acked Write 2 Sent Done Done false
20098 5864 Acked Write 1 Sent Done Done false
20099 5865 Acked Write 6 Sent Done Done false
20100 5866 Acked Read 6 Sent Done Done false
20101 5867 Acked Write 7 Sent Done Done false
20102 5868 Acked Read 1 Sent Done Done false
20103 5869 Acked Read 8 Sent Done Done false
20104 5870 Acked Read 3 Sent Done Done false
20105 5871 Acked Write 2 Sent Done Done false
20106 5872 Acked Write 5 Sent Done Done false
20107 5873 Acked Write 9 Sent Done Done false
20108 5874 Acked Read 3 Sent Done Done false
20109 5875 Acked Read 7 Sent Done Done false
20110 5876 Acked Read 4 Sent Done Done false
20111 5877 Acked Read 7 Sent Done Done false
20112 5878 Acked Read 7 Sent Done Done false
20113 5879 Acked Write 1 Sent Done Done false
20114 5880 Acked Read 10 Sent Done Done false
20115 5881 Acked Read 4 Sent Done Done false
20116 5882 Acked Read 2 Sent Done Done false
20117 5883 Acked Read 5 Sent Done Done false
20118 5884 Acked Flush 0 Sent Done Done false
20119 5885 Acked Write 10 Sent Done Done false
20120 5886 Acked Read 8 Sent Done Done false
20121 5887 Acked Write 2 Sent Done Done false
20122 5888 Acked Read 4 Sent Done Done false
20123 5889 Acked Write 2 Sent Done Done false
20124 5890 Acked Write 2 Sent Done Done false
20125 5891 Acked Read 1 Sent Done Done false
20126 5892 Acked Read 6 Sent Done Done false
20127 5893 Acked Read 1 Sent Done Done false
20128 5894 Acked Write 10 Sent Done Done false
20129 5895 Acked Read 4 Sent Done Done false
20130 5896 Acked Read 10 Sent Done Done false
20131 5897 Acked Read 3 Sent Done Done false
20132 5898 Acked Read 6 Sent Done Done false
20133 5899 Acked Read 7 Sent Done Done false
20134 5900 Acked Read 10 Sent Done Done false
20135 5901 Acked Read 9 Sent Done Done false
20136 5902 Acked Write 6 Sent Done Done false
20137 5903 Acked Write 4 Sent Done Done false
20138 5904 Acked Write 6 Sent Done Done false
20139 5905 Acked Flush 0 Sent Done Done false
20140 5906 Acked Flush 0 Sent Done Done false
20141 5907 Acked Write 10 Sent Done Done false
20142 5908 Acked Read 3 Sent Done Done false
20143 5909 Acked Write 5 Sent Done Done false
20144 5910 Acked Write 5 Sent Done Done false
20145 5911 Acked Read 8 Sent Done Done false
20146 5912 Acked Read 1 Sent Done Done false
20147 5913 Acked Write 3 Sent Done Done false
20148 5914 Acked Read 8 Sent Done Done false
20149 5915 Acked Write 4 Sent Done Done false
20150 5916 Acked Write 6 Sent Done Done false
20151 5917 Acked Read 4 Sent Done Done false
20152 5918 Acked Read 7 Sent Done Done false
20153 5919 Acked Write 7 Sent Done Done false
20154 5920 Acked Read 9 Sent Done Done false
20155 5921 Acked Flush 0 Sent Done Done false
20156 5922 Acked Write 5 Sent Done Done false
20157 5923 Acked Write 2 Sent Done Done false
20158 5924 Acked Read 9 Sent Done Done false
20159 5925 Acked Read 3 Sent Done Done false
20160 5926 Acked Read 2 Sent Done Done false
20161 5927 Acked Write 2 Sent Done Done false
20162 5928 Acked Read 6 Sent Done Done false
20163 5929 Acked Write 9 Sent Done Done false
20164 5930 Acked Write 6 Sent Done Done false
20165 5931 Acked Flush 0 Sent Done Done false
20166 5932 Acked Write 1 Sent Done Done false
20167 5933 Acked Write 5 Sent Done Done false
20168 5934 Acked Read 1 Sent Done Done false
20169 5935 Acked Read 7 Sent Done Done false
20170 5936 Acked Read 3 Sent Done Done false
20171 5937 Acked Write 2 Sent Done Done false
20172 5938 Acked Read 7 Sent Done Done false
20173 5939 Acked Read 9 Sent Done Done false
20174 5940 Acked Write 8 Sent Done Done false
20175 5941 Acked Write 4 Sent Done Done false
20176 5942 Acked Write 5 Sent Done Done false
20177 5943 Acked Read 10 Sent Done Done false
20178 5944 Acked Write 10 Sent Done Done false
20179 5945 Acked Read 5 Sent Done Done false
20180 5946 Acked Write 2 Sent Done Done false
20181 5947 Acked Read 6 Sent Done Done false
20182 5948 Acked Read 6 Sent Done Done false
20183 5949 Acked Flush 0 Sent Done Done false
20184 5950 Acked Flush 0 Sent Done Done false
20185 5951 Acked Read 1 Sent Done Done false
20186 5952 Acked Read 7 Sent Done Done false
20187 5953 Acked Read 8 Sent Done Done false
20188 5954 Acked Read 7 Sent Done Done false
20189 5955 Acked Read 9 Sent Done Done false
20190 5956 Acked Flush 0 Sent Done Done false
20191 5957 Acked Read 6 Sent Done Done false
20192 5958 Acked Write 1 Sent Done Done false
20193 5959 Acked Flush 0 Sent Done Done false
20194 5960 Acked Write 5 Sent Done Done false
20195 5961 Acked Write 7 Sent Done Done false
20196 5962 Acked Read 9 Sent Done Done false
20197 5963 Acked Write 2 Sent Done Done false
20198 5964 Acked Write 2 Sent Done Done false
20199 5965 Acked Write 1 Sent Done Done false
20200 5966 Acked Read 1 Sent Done Done false
20201 5967 Acked Write 6 Sent Done Done false
20202 5968 Acked Read 10 Sent Done Done false
20203 5969 Acked Read 2 Sent Done Done false
20204 5970 Acked Write 5 Sent Done Done false
20205 5971 Acked Write 9 Sent Done Done false
20206 5972 Acked Read 3 Sent Done Done false
20207 5973 Acked Write 9 Sent Done Done false
20208 5974 Acked Write 7 Sent Done Done false
20209 5975 Acked Write 7 Sent Done Done false
20210 5976 Acked Write 2 Sent Done Done false
20211 5977 Acked Read 8 Sent Done Done false
20212 5978 Acked Write 4 Sent Done Done false
20213 5979 Acked Read 4 Sent Done Done false
20214 5980 Acked Write 2 Sent Done Done false
20215 5981 Acked Read 1 Sent Done Done false
20216 5982 Acked Write 1 Sent Done Done false
20217 5983 Acked Flush 0 Sent Done Done false
20218 5984 Acked Write 2 Sent Done Done false
20219 5985 Acked Write 8 Sent Done Done false
20220 5986 Acked Read 3 Sent Done Done false
20221 5987 Acked Read 7 Sent Done Done false
20222 5988 Acked Write 7 Sent Done Done false
20223 5989 Acked Flush 0 Sent Done Done false
20224 5990 Acked Write 4 Sent Done Done false
20225 5991 Acked Read 3 Sent Done Done false
20226 5992 Acked Read 4 Sent Done Done false
20227 5993 Acked Write 4 Sent Done Done false
20228 5994 Acked Write 4 Sent Done Done false
20229 5995 Acked Read 6 Sent Done Done false
20230 5996 Acked Write 8 Sent Done Done false
20231 5997 Acked Read 1 Sent Done Done false
20232 5998 Acked Read 2 Sent Done Done false
20233 5999 Acked Write 6 Sent Done Done false
20234 6000 Acked Read 9 Sent Done Done false
20235 6001 Acked Read 10 Sent Done Done false
20236 6002 Acked Read 1 Sent Done Done false
20237 6003 Acked Flush 0 Sent Done Done false
20238 6004 Acked Write 4 Sent Done Done false
20239 6005 Acked Write 1 Sent Done Done false
20240 6006 Acked Write 4 Sent Done Done false
20241 6007 Acked Read 3 Sent Done Done false
20242 6008 Acked Write 1 Sent Done Done false
20243 6009 Acked Write 8 Sent Done Done false
20244 6010 Acked Read 9 Sent Done Done false
20245 6011 Acked Write 8 Sent Done Done false
20246 6012 Acked Read 1 Sent Done Done false
20247 6013 Acked Write 8 Sent Done Done false
20248 6014 Acked Read 5 Sent Done Done false
20249 6015 Acked Write 3 Sent Done Done false
20250 6016 Acked Read 6 Sent Done Done false
20251 6017 Acked Read 10 Sent Done Done false
20252 6018 Acked Read 7 Sent Done Done false
20253 6019 Acked Read 10 Sent Done Done false
20254 6020 Acked Write 8 Sent Done Done false
20255 6021 Acked Read 9 Sent Done Done false
20256 6022 Acked Flush 0 Sent Done Done false
20257 6023 Acked Read 8 Sent Done Done false
20258 6024 Acked Flush 0 Sent Done Done false
20259 6025 Acked Write 7 Sent Done Done false
20260 6026 Acked Write 9 Sent Done Done false
20261 6027 Acked Write 5 Sent Done Done false
20262 6028 Acked Write 6 Sent Done Done false
20263 6029 Acked Read 4 Sent Done Done false
20264 6030 Acked Write 1 Sent Done Done false
20265 6031 Acked Write 10 Sent Done Done false
20266 6032 Acked Read 3 Sent Done Done false
20267 6033 Acked Write 10 Sent Done Done false
20268 6034 Acked Read 9 Sent Done Done false
20269 6035 Acked Write 7 Sent Done Done false
20270 6036 Acked Write 8 Sent Done Done false
20271 6037 Acked Write 10 Sent Done Done false
20272 6038 Acked Read 4 Sent Done Done false
20273 6039 Acked Read 3 Sent Done Done false
20274 6040 Acked Write 8 Sent Done Done false
20275 6041 Acked Flush 0 Sent Done Done false
20276 6042 Acked Write 9 Sent Done Done false
20277 6043 Acked Read 1 Sent Done Done false
20278 6044 Acked Read 6 Sent Done Done false
20279 6045 Acked Write 1 Sent Done Done false
20280 6046 Acked Read 4 Sent Done Done false
20281 6047 Acked Write 4 Sent Done Done false
20282 6048 Acked Read 3 Sent Done Done false
20283 6049 Acked Read 1 Sent Done Done false
20284 6050 Acked Read 8 Sent Done Done false
20285 6051 Acked Read 6 Sent Done Done false
20286 6052 Acked Read 4 Sent Done Done false
20287 6053 Acked Read 8 Sent Done Done false
20288 6054 Acked Read 4 Sent Done Done false
20289 6055 Acked Read 9 Sent Done Done false
20290 6056 Acked Read 7 Sent Done Done false
20291 6057 Acked Read 8 Sent Done Done false
20292 6058 Acked Read 3 Sent Done Done false
20293 6059 Acked Write 8 Sent Done Done false
20294 6060 Acked Flush 0 Sent Done Done false
20295 6061 Acked Write 6 Sent Done Done false
20296 6062 Acked Write 7 Sent Done Done false
20297 6063 Acked Read 3 Sent Done Done false
20298 6064 Acked Write 7 Sent Done Done false
20299 6065 Acked Flush 0 Sent Done Done false
20300 6066 Acked Read 6 Sent Done Done false
20301 6067 Acked Write 7 Sent Done Done false
20302 6068 Acked Read 4 Sent Done Done false
20303 6069 Acked Read 2 Sent Done Done false
20304 6070 Acked Read 10 Sent Done Done false
20305 6071 Acked Write 7 Sent Done Done false
20306 6072 Acked Read 7 Sent Done Done false
20307 6073 Acked Write 3 Sent Done Done false
20308 6074 Acked Read 1 Sent Done Done false
20309 6075 Acked Read 6 Sent Done Done false
20310 6076 Acked Read 1 Sent Done Done false
20311 6077 Acked Read 8 Sent Done Done false
20312 6078 Acked Write 10 Sent Done Done false
20313 6079 Acked Read 7 Sent Done Done false
20314 6080 Acked Write 5 Sent Done Done false
20315 6081 Acked Read 9 Sent Done Done false
20316 6082 Acked Write 2 Sent Done Done false
20317 6083 Acked Read 9 Sent Done Done false
20318 6084 Acked Write 2 Sent Done Done false
20319 6085 Acked Read 3 Sent Done Done false
20320 6086 Acked Read 2 Sent Done Done false
20321 6087 Acked Read 8 Sent Done Done false
20322 6088 Acked Write 9 Sent Done Done false
20323 6089 Acked Flush 0 Sent Done Done false
20324 6090 Acked Write 7 Sent Done Done false
20325 6091 Acked Write 5 Sent Done Done false
20326 6092 Acked Write 8 Sent Done Done false
20327 6093 Acked Read 4 Sent Done Done false
20328 6094 Acked Write 3 Sent Done Done false
20329 6095 Acked Write 7 Sent Done Done false
20330 6096 Acked Read 9 Sent Done Done false
20331 6097 Acked Read 4 Sent Done Done false
20332 6098 Acked Flush 0 Sent Done Done false
20333 6099 Acked Write 5 Sent Done Done false
20334 6100 Acked Read 9 Sent Done Done false
20335 6101 Acked Read 10 Sent Done Done false
20336 6102 Acked Write 2 Sent Done Done false
20337 6103 Acked Write 2 Sent Done Done false
20338 6104 Acked Read 2 Sent Done Done false
20339 6105 Acked Read 9 Sent Done Done false
20340 6106 Acked Read 7 Sent Done Done false
20341 6107 Acked Read 5 Sent Done Done false
20342 6108 Acked Write 7 Sent Done Done false
20343 6109 Acked Read 10 Sent Done Done false
20344 6110 Acked Write 3 Sent Done Done false
20345 6111 Acked Read 3 Sent Done Done false
20346 6112 Acked Read 6 Sent Done Done false
20347 6113 Acked Write 3 Sent Done Done false
20348 6114 Acked Write 8 Sent Done Done false
20349 6115 Acked Flush 0 Sent Done Done false
20350 6116 Acked Read 5 Sent Done Done false
20351 6117 Acked Read 7 Sent Done Done false
20352 6118 Acked Write 2 Sent Done Done false
20353 6119 Acked Write 6 Sent Done Done false
20354 6120 Acked Read 10 Sent Done Done false
20355 6121 Acked Write 10 Sent Done Done false
20356 6122 Acked Write 3 Sent Done Done false
20357 6123 Acked Read 8 Sent Done Done false
20358 6124 Acked Write 10 Sent Done Done false
20359 6125 Acked Read 2 Sent Done Done false
20360 6126 Acked Read 7 Sent Done Done false
20361 6127 Acked Write 6 Sent Done Done false
20362 6128 Acked Write 4 Sent Done Done false
20363 6129 Acked Flush 0 Sent Done Done false
20364 6130 Acked Read 5 Sent Done Done false
20365 STATES DS:0 DS:1 DS:2 TOTAL
20366 Sent 301 0 0 301
20367 Done 0 301 301 602
20368 Skipped 0 0 0 0
20369 Error 0 0 0 0
20370 Last Flush: Some(JobId(5829)) Some(JobId(6129)) Some(JobId(6129))
20371 Downstairs last five completed: 5829 5828 5827 5826 5825
20372 Upstairs last five completed: 6130 6129 6128 6127 6126
20373 CLIENT: Up:0 ds:301 act:2
203742025-03-07T18:17:02.394ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
203752025-03-07T18:17:03.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
203762025-03-07T18:17:04.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
20377 [8810] got start action
20378 [8810] I am going to restart
20379 Starting downstairs at port 8810
20380 Make output file at "/tmp/dsc/downstairs-8810.txt"
20381 [8810][0] reports Starting
20382 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1225)
20383 [8810][0] reports Running
20384 ----------------------------------------------------------------
20385 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20386 States: Offline Active Active
20387 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20388 5830 Acked Read 10 Sent Done Done false
20389 5831 Acked Write 5 Sent Done Done false
20390 5832 Acked Write 2 Sent Done Done false
20391 5833 Acked Write 8 Sent Done Done false
20392 5834 Acked Read 9 Sent Done Done false
20393 5835 Acked Write 4 Sent Done Done false
20394 5836 Acked Write 7 Sent Done Done false
20395 5837 Acked Read 3 Sent Done Done false
20396 5838 Acked Write 1 Sent Done Done false
20397 5839 Acked Write 6 Sent Done Done false
20398 5840 Acked Write 8 Sent Done Done false
20399 5841 Acked Write 8 Sent Done Done false
20400 5842 Acked Write 1 Sent Done Done false
20401 5843 Acked Flush 0 Sent Done Done false
20402 5844 Acked Flush 0 Sent Done Done false
20403 5845 Acked Read 8 Sent Done Done false
20404 5846 Acked Write 10 Sent Done Done false
20405 5847 Acked Write 2 Sent Done Done false
20406 5848 Acked Write 8 Sent Done Done false
20407 5849 Acked Flush 0 Sent Done Done false
20408 5850 Acked Write 5 Sent Done Done false
20409 5851 Acked Read 1 Sent Done Done false
20410 5852 Acked Write 4 Sent Done Done false
20411 5853 Acked Read 6 Sent Done Done false
20412 5854 Acked Write 4 Sent Done Done false
20413 5855 Acked Read 10 Sent Done Done false
20414 5856 Acked Read 6 Sent Done Done false
20415 5857 Acked Write 9 Sent Done Done false
20416 5858 Acked Write 3 Sent Done Done false
20417 5859 Acked Flush 0 Sent Done Done false
20418 5860 Acked Read 6 Sent Done Done false
20419 5861 Acked Read 7 Sent Done Done false
20420 5862 Acked Read 5 Sent Done Done false
20421 5863 Acked Write 2 Sent Done Done false
20422 5864 Acked Write 1 Sent Done Done false
20423 5865 Acked Write 6 Sent Done Done false
20424 5866 Acked Read 6 Sent Done Done false
20425 5867 Acked Write 7 Sent Done Done false
20426 5868 Acked Read 1 Sent Done Done false
20427 5869 Acked Read 8 Sent Done Done false
20428 5870 Acked Read 3 Sent Done Done false
20429 5871 Acked Write 2 Sent Done Done false
20430 5872 Acked Write 5 Sent Done Done false
20431 5873 Acked Write 9 Sent Done Done false
20432 5874 Acked Read 3 Sent Done Done false
20433 5875 Acked Read 7 Sent Done Done false
20434 5876 Acked Read 4 Sent Done Done false
20435 5877 Acked Read 7 Sent Done Done false
20436 5878 Acked Read 7 Sent Done Done false
20437 5879 Acked Write 1 Sent Done Done false
20438 5880 Acked Read 10 Sent Done Done false
20439 5881 Acked Read 4 Sent Done Done false
20440 5882 Acked Read 2 Sent Done Done false
20441 5883 Acked Read 5 Sent Done Done false
20442 5884 Acked Flush 0 Sent Done Done false
20443 5885 Acked Write 10 Sent Done Done false
20444 5886 Acked Read 8 Sent Done Done false
20445 5887 Acked Write 2 Sent Done Done false
20446 5888 Acked Read 4 Sent Done Done false
20447 5889 Acked Write 2 Sent Done Done false
20448 5890 Acked Write 2 Sent Done Done false
20449 5891 Acked Read 1 Sent Done Done false
20450 5892 Acked Read 6 Sent Done Done false
20451 5893 Acked Read 1 Sent Done Done false
20452 5894 Acked Write 10 Sent Done Done false
20453 5895 Acked Read 4 Sent Done Done false
20454 5896 Acked Read 10 Sent Done Done false
20455 5897 Acked Read 3 Sent Done Done false
20456 5898 Acked Read 6 Sent Done Done false
20457 5899 Acked Read 7 Sent Done Done false
20458 5900 Acked Read 10 Sent Done Done false
20459 5901 Acked Read 9 Sent Done Done false
20460 5902 Acked Write 6 Sent Done Done false
20461 5903 Acked Write 4 Sent Done Done false
20462 5904 Acked Write 6 Sent Done Done false
20463 5905 Acked Flush 0 Sent Done Done false
20464 5906 Acked Flush 0 Sent Done Done false
20465 5907 Acked Write 10 Sent Done Done false
20466 5908 Acked Read 3 Sent Done Done false
20467 5909 Acked Write 5 Sent Done Done false
20468 5910 Acked Write 5 Sent Done Done false
20469 5911 Acked Read 8 Sent Done Done false
20470 5912 Acked Read 1 Sent Done Done false
20471 5913 Acked Write 3 Sent Done Done false
20472 5914 Acked Read 8 Sent Done Done false
20473 5915 Acked Write 4 Sent Done Done false
20474 5916 Acked Write 6 Sent Done Done false
20475 5917 Acked Read 4 Sent Done Done false
20476 5918 Acked Read 7 Sent Done Done false
20477 5919 Acked Write 7 Sent Done Done false
20478 5920 Acked Read 9 Sent Done Done false
20479 5921 Acked Flush 0 Sent Done Done false
20480 5922 Acked Write 5 Sent Done Done false
20481 5923 Acked Write 2 Sent Done Done false
20482 5924 Acked Read 9 Sent Done Done false
20483 5925 Acked Read 3 Sent Done Done false
20484 5926 Acked Read 2 Sent Done Done false
20485 5927 Acked Write 2 Sent Done Done false
20486 5928 Acked Read 6 Sent Done Done false
20487 5929 Acked Write 9 Sent Done Done false
20488 5930 Acked Write 6 Sent Done Done false
20489 5931 Acked Flush 0 Sent Done Done false
20490 5932 Acked Write 1 Sent Done Done false
20491 5933 Acked Write 5 Sent Done Done false
20492 5934 Acked Read 1 Sent Done Done false
20493 5935 Acked Read 7 Sent Done Done false
20494 5936 Acked Read 3 Sent Done Done false
20495 5937 Acked Write 2 Sent Done Done false
20496 5938 Acked Read 7 Sent Done Done false
20497 5939 Acked Read 9 Sent Done Done false
20498 5940 Acked Write 8 Sent Done Done false
20499 5941 Acked Write 4 Sent Done Done false
20500 5942 Acked Write 5 Sent Done Done false
20501 5943 Acked Read 10 Sent Done Done false
20502 5944 Acked Write 10 Sent Done Done false
20503 5945 Acked Read 5 Sent Done Done false
20504 5946 Acked Write 2 Sent Done Done false
20505 5947 Acked Read 6 Sent Done Done false
20506 5948 Acked Read 6 Sent Done Done false
20507 5949 Acked Flush 0 Sent Done Done false
20508 5950 Acked Flush 0 Sent Done Done false
20509 5951 Acked Read 1 Sent Done Done false
20510 5952 Acked Read 7 Sent Done Done false
20511 5953 Acked Read 8 Sent Done Done false
20512 5954 Acked Read 7 Sent Done Done false
20513 5955 Acked Read 9 Sent Done Done false
20514 5956 Acked Flush 0 Sent Done Done false
20515 5957 Acked Read 6 Sent Done Done false
20516 5958 Acked Write 1 Sent Done Done false
20517 5959 Acked Flush 0 Sent Done Done false
20518 5960 Acked Write 5 Sent Done Done false
20519 5961 Acked Write 7 Sent Done Done false
20520 5962 Acked Read 9 Sent Done Done false
20521 5963 Acked Write 2 Sent Done Done false
20522 5964 Acked Write 2 Sent Done Done false
20523 5965 Acked Write 1 Sent Done Done false
20524 5966 Acked Read 1 Sent Done Done false
20525 5967 Acked Write 6 Sent Done Done false
20526 5968 Acked Read 10 Sent Done Done false
20527 5969 Acked Read 2 Sent Done Done false
20528 5970 Acked Write 5 Sent Done Done false
20529 5971 Acked Write 9 Sent Done Done false
20530 5972 Acked Read 3 Sent Done Done false
20531 5973 Acked Write 9 Sent Done Done false
20532 5974 Acked Write 7 Sent Done Done false
20533 5975 Acked Write 7 Sent Done Done false
20534 5976 Acked Write 2 Sent Done Done false
20535 5977 Acked Read 8 Sent Done Done false
20536 5978 Acked Write 4 Sent Done Done false
20537 5979 Acked Read 4 Sent Done Done false
20538 5980 Acked Write 2 Sent Done Done false
20539 5981 Acked Read 1 Sent Done Done false
20540 5982 Acked Write 1 Sent Done Done false
20541 5983 Acked Flush 0 Sent Done Done false
20542 5984 Acked Write 2 Sent Done Done false
20543 5985 Acked Write 8 Sent Done Done false
20544 5986 Acked Read 3 Sent Done Done false
20545 5987 Acked Read 7 Sent Done Done false
20546 5988 Acked Write 7 Sent Done Done false
20547 5989 Acked Flush 0 Sent Done Done false
20548 5990 Acked Write 4 Sent Done Done false
20549 5991 Acked Read 3 Sent Done Done false
20550 5992 Acked Read 4 Sent Done Done false
20551 5993 Acked Write 4 Sent Done Done false
20552 5994 Acked Write 4 Sent Done Done false
20553 5995 Acked Read 6 Sent Done Done false
20554 5996 Acked Write 8 Sent Done Done false
20555 5997 Acked Read 1 Sent Done Done false
20556 5998 Acked Read 2 Sent Done Done false
20557 5999 Acked Write 6 Sent Done Done false
20558 6000 Acked Read 9 Sent Done Done false
20559 6001 Acked Read 10 Sent Done Done false
20560 6002 Acked Read 1 Sent Done Done false
20561 6003 Acked Flush 0 Sent Done Done false
20562 6004 Acked Write 4 Sent Done Done false
20563 6005 Acked Write 1 Sent Done Done false
20564 6006 Acked Write 4 Sent Done Done false
20565 6007 Acked Read 3 Sent Done Done false
20566 6008 Acked Write 1 Sent Done Done false
20567 6009 Acked Write 8 Sent Done Done false
20568 6010 Acked Read 9 Sent Done Done false
20569 6011 Acked Write 8 Sent Done Done false
20570 6012 Acked Read 1 Sent Done Done false
20571 6013 Acked Write 8 Sent Done Done false
20572 6014 Acked Read 5 Sent Done Done false
20573 6015 Acked Write 3 Sent Done Done false
20574 6016 Acked Read 6 Sent Done Done false
20575 6017 Acked Read 10 Sent Done Done false
20576 6018 Acked Read 7 Sent Done Done false
20577 6019 Acked Read 10 Sent Done Done false
20578 6020 Acked Write 8 Sent Done Done false
20579 6021 Acked Read 9 Sent Done Done false
20580 6022 Acked Flush 0 Sent Done Done false
20581 6023 Acked Read 8 Sent Done Done false
20582 6024 Acked Flush 0 Sent Done Done false
20583 6025 Acked Write 7 Sent Done Done false
20584 6026 Acked Write 9 Sent Done Done false
20585 6027 Acked Write 5 Sent Done Done false
20586 6028 Acked Write 6 Sent Done Done false
20587 6029 Acked Read 4 Sent Done Done false
20588 6030 Acked Write 1 Sent Done Done false
20589 6031 Acked Write 10 Sent Done Done false
20590 6032 Acked Read 3 Sent Done Done false
20591 6033 Acked Write 10 Sent Done Done false
20592 6034 Acked Read 9 Sent Done Done false
20593 6035 Acked Write 7 Sent Done Done false
20594 6036 Acked Write 8 Sent Done Done false
20595 6037 Acked Write 10 Sent Done Done false
20596 6038 Acked Read 4 Sent Done Done false
20597 6039 Acked Read 3 Sent Done Done false
20598 6040 Acked Write 8 Sent Done Done false
20599 6041 Acked Flush 0 Sent Done Done false
20600 6042 Acked Write 9 Sent Done Done false
20601 6043 Acked Read 1 Sent Done Done false
20602 6044 Acked Read 6 Sent Done Done false
20603 6045 Acked Write 1 Sent Done Done false
20604 6046 Acked Read 4 Sent Done Done false
20605 6047 Acked Write 4 Sent Done Done false
20606 6048 Acked Read 3 Sent Done Done false
20607 6049 Acked Read 1 Sent Done Done false
20608 6050 Acked Read 8 Sent Done Done false
20609 6051 Acked Read 6 Sent Done Done false
20610 6052 Acked Read 4 Sent Done Done false
20611 6053 Acked Read 8 Sent Done Done false
20612 6054 Acked Read 4 Sent Done Done false
20613 6055 Acked Read 9 Sent Done Done false
20614 6056 Acked Read 7 Sent Done Done false
20615 6057 Acked Read 8 Sent Done Done false
20616 6058 Acked Read 3 Sent Done Done false
20617 6059 Acked Write 8 Sent Done Done false
20618 6060 Acked Flush 0 Sent Done Done false
20619 6061 Acked Write 6 Sent Done Done false
20620 6062 Acked Write 7 Sent Done Done false
20621 6063 Acked Read 3 Sent Done Done false
20622 6064 Acked Write 7 Sent Done Done false
20623 6065 Acked Flush 0 Sent Done Done false
20624 6066 Acked Read 6 Sent Done Done false
20625 6067 Acked Write 7 Sent Done Done false
20626 6068 Acked Read 4 Sent Done Done false
20627 6069 Acked Read 2 Sent Done Done false
20628 6070 Acked Read 10 Sent Done Done false
20629 6071 Acked Write 7 Sent Done Done false
20630 6072 Acked Read 7 Sent Done Done false
20631 6073 Acked Write 3 Sent Done Done false
20632 6074 Acked Read 1 Sent Done Done false
20633 6075 Acked Read 6 Sent Done Done false
20634 6076 Acked Read 1 Sent Done Done false
20635 6077 Acked Read 8 Sent Done Done false
20636 6078 Acked Write 10 Sent Done Done false
20637 6079 Acked Read 7 Sent Done Done false
20638 6080 Acked Write 5 Sent Done Done false
20639 6081 Acked Read 9 Sent Done Done false
20640 6082 Acked Write 2 Sent Done Done false
20641 6083 Acked Read 9 Sent Done Done false
20642 6084 Acked Write 2 Sent Done Done false
20643 6085 Acked Read 3 Sent Done Done false
20644 6086 Acked Read 2 Sent Done Done false
20645 6087 Acked Read 8 Sent Done Done false
20646 6088 Acked Write 9 Sent Done Done false
20647 6089 Acked Flush 0 Sent Done Done false
20648 6090 Acked Write 7 Sent Done Done false
20649 6091 Acked Write 5 Sent Done Done false
20650 6092 Acked Write 8 Sent Done Done false
20651 6093 Acked Read 4 Sent Done Done false
20652 6094 Acked Write 3 Sent Done Done false
20653 6095 Acked Write 7 Sent Done Done false
20654 6096 Acked Read 9 Sent Done Done false
20655 6097 Acked Read 4 Sent Done Done false
20656 6098 Acked Flush 0 Sent Done Done false
20657 6099 Acked Write 5 Sent Done Done false
20658 6100 Acked Read 9 Sent Done Done false
20659 6101 Acked Read 10 Sent Done Done false
20660 6102 Acked Write 2 Sent Done Done false
20661 6103 Acked Write 2 Sent Done Done false
20662 6104 Acked Read 2 Sent Done Done false
20663 6105 Acked Read 9 Sent Done Done false
20664 6106 Acked Read 7 Sent Done Done false
20665 6107 Acked Read 5 Sent Done Done false
20666 6108 Acked Write 7 Sent Done Done false
20667 6109 Acked Read 10 Sent Done Done false
20668 6110 Acked Write 3 Sent Done Done false
20669 6111 Acked Read 3 Sent Done Done false
20670 6112 Acked Read 6 Sent Done Done false
20671 6113 Acked Write 3 Sent Done Done false
20672 6114 Acked Write 8 Sent Done Done false
20673 6115 Acked Flush 0 Sent Done Done false
20674 6116 Acked Read 5 Sent Done Done false
20675 6117 Acked Read 7 Sent Done Done false
20676 6118 Acked Write 2 Sent Done Done false
20677 6119 Acked Write 6 Sent Done Done false
20678 6120 Acked Read 10 Sent Done Done false
20679 6121 Acked Write 10 Sent Done Done false
20680 6122 Acked Write 3 Sent Done Done false
20681 6123 Acked Read 8 Sent Done Done false
20682 6124 Acked Write 10 Sent Done Done false
20683 6125 Acked Read 2 Sent Done Done false
20684 6126 Acked Read 7 Sent Done Done false
20685 6127 Acked Write 6 Sent Done Done false
20686 6128 Acked Write 4 Sent Done Done false
20687 6129 Acked Flush 0 Sent Done Done false
20688 6130 Acked Read 5 Sent Done Done false
20689 STATES DS:0 DS:1 DS:2 TOTAL
20690 Sent 301 0 0 301
20691 Done 0 301 301 602
20692 Skipped 0 0 0 0
20693 Error 0 0 0 0
20694 Last Flush: Some(JobId(5829)) Some(JobId(6129)) Some(JobId(6129))
20695 Downstairs last five completed: 5829 5828 5827 5826 5825
20696 Upstairs last five completed: 6130 6129 6128 6127 6126
20697 CLIENT: Up:0 ds:301 act:2
206982025-03-07T18:17:05.399ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
206992025-03-07T18:17:06.609ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207002025-03-07T18:17:06.609ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52738) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207012025-03-07T18:17:06.610ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207022025-03-07T18:17:06.611ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207032025-03-07T18:17:06.611ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207042025-03-07T18:17:06.611ZINFOcrucible: send last flush ID to this DS: Some(JobId(5829)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207052025-03-07T18:17:06.611ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207062025-03-07T18:17:06.611ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(5829)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207072025-03-07T18:17:06.611ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(5829)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
20708 ----------------------------------------------------------------
20709 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20710 Downstairs last five completed: 6131 6130 6129 6128 6127
20711 Upstairs last five completed: 6131 6130 6129 6128 6127
20712 CLIENT: Up:0 ds:0 act:3
20713 Replay: All jobs finished
20714 Main task has work to do, go find it
20715 got dsc Stop(2)
20716 stop 2
20717 Mar 07 18:17:08.861 INFO request completed, latency_us: 353, response_code: [8830] Got stop action so:false kr:false
20718 204, uri: /stop/cid/2, method: POST, req_id: b2769a2c-8a86-481f-b019-151c26359b36, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
20719 Mar 07 18:17:08.862 INFO request completed, latency_us: 97, response_code: 200, uri: /state/cid/2, method: GET, req_id:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:08.8623891Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
20720 44c2795f-bae9-472b-99fe-62c4c6a353a6, remote_addr: 127.0.0.1:64322, local_addr: 127.0.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:08.862485878Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
20721 0.1:9998
207222025-03-07T18:17:08.862ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
207232025-03-07T18:17:08.862ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
20724 [8830] Exited with: ExitStatus(unix_wait_status(9))
20725 [8830][2] reports Exit
20726 Mar 07 18:17:12.865 INFO request completed, latency_us: 287, response_code: 200, uri: /state/cid/2, method: GET, req_id: f52f3167-e1f6-40a8-90cc-fb629926b031, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
20727 001/300 Read block 4040 len 40960
20728 002/300 Read block 3850 len 36864
20729 003/300 Write block 3052 len 16384 data: 3 4 3 4
20730 004/300 Write block 1711 len 20480 data: 3 4 4 5 6
20731 005/300 Flush
20732 006/300 Read block 3948 len 4096
20733 007/300 Write block 314 len 8192 data: 3 3
20734 008/300 Flush
20735 009/300 Read block 2529 len 32768
20736 010/300 Read block 2219 len 24576
20737 011/300 Write block 2291 len 4096 data: 3
20738 012/300 Read block 471 len 12288
20739 013/300 Write block 4880 len 8192 data: 4 3
20740 014/300 Flush
20741 015/300 Write block 1599 len 24576 data: 3 5 6 6 6 5
20742 016/300 Read block 3676 len 20480
20743 017/300 Read block 581 len 36864
20744 018/300 Write block 3375 len 40960 data: 4 4 5 5 5 4 4 4 5 6
20745 019/300 Write block 4935 len 12288 data: 3 3 3
20746 020/300 Write block 397 len 16384 data: 4 4 5 5
20747 021/300 Write block 3344 len 4096 data: 3
20748 022/300 Read block 109 len 12288
20749 023/300 Write block 3673 len 20480 data: 3 3 3 4 3
20750 024/300 Write block 879 len 12288 data: 4 4 6
20751 025/300 Read block 4570 len 28672
20752 026/300 Write block 1777 len 4096 data: 4
20753 027/300 Write block 2894 len 36864 data: 5 4 4 4 4 5 6 6 6
20754 028/300 Read block 2901 len 32768
20755 029/300 Flush
20756 030/300 Read block 3861 len 36864
20757 031/300 Read block 4985 len 20480
20758 032/300 Write block 2429 len 4096 data: 4
20759 033/300 Write block 4163 len 20480 data: 2 2 2 3 3
20760 034/300 Read block 392 len 20480
20761 035/300 Write block 22 len 28672 data: 6 6 5 5 5 3 4
20762 036/300 Write block 2718 len 4096 data: 3
20763 037/300 Read block 2957 len 40960
20764 038/300 Read block 1691 len 32768
20765 039/300 Flush
20766 040/300 Read block 3899 len 12288
20767 041/300 Write block 428 len 40960 data: 5 6 6 4 5 5 6 5 3 3
20768 042/300 Write block 1245 len 24576 data: 4 5 5 5 5 5
20769 043/300 Read block 3557 len 24576
20770 044/300 Read block 3972 len 4096
20771 045/300 Read block 4150 len 4096
20772 046/300 Read block 3883 len 28672
20773 047/300 Flush
20774 048/300 Read block 2365 len 12288
20775 049/300 Write block 797 len 28672 data: 5 5 4 3 4 5 4
20776 050/300 Write block 3512 len 4096 data: 3
20777 051/300 Read block 465 len 36864
20778 052/300 Read block 3550 len 28672
20779 053/300 Read block 3284 len 28672
20780 054/300 Flush
20781 055/300 Read block 3431 len 4096
20782 056/300 Write block 3829 len 16384 data: 5 5 4 3
20783 057/300 Write block 4175 len 28672 data: 5 5 5 4 3 4 4
20784 058/300 Read block 1594 len 24576
20785 059/300 Write block 1007 len 12288 data: 5 7 7
20786 060/300 Read block 4637 len 36864
20787 061/300 Read block 3208 len 36864
20788 062/300 Write block 824 len 8192 data: 6 8
20789 063/300 Read block 1857 len 36864
20790 064/300 Flush
20791 065/300 Write block 600 len 28672 data: 3 5 6 6 6 6 6
20792 066/300 Write block 279 len 40960 data: 5 5 6 4 4 4 3 4 5 6
20793 067/300 Flush
20794 068/300 Read block 3187 len 24576
20795 069/300 Read block 3277 len 24576
20796 070/300 Write block 644 len 36864 data: 4 5 5 5 5 5 5 4 3
20797 071/300 Write block 3874 len 4096 data: 4
20798 072/300 Write block 2430 len 28672 data: 4 4 5 4 4 4 5
20799 073/300 Write block 1396 len 4096 data: 3
20800 074/300 Read block 2658 len 20480
20801 075/300 Read block 4427 len 28672
20802 076/300 Read block 548 len 24576
20803 077/300 Read block 1665 len 20480
20804 078/300 Read block 3995 len 12288
20805 079/300 Read block 1861 len 36864
20806 080/300 Read block 1641 len 20480
20807 081/300 Write block 2665 len 16384 data: 6 6 5 5
20808 082/300 Read block 560 len 36864
20809 083/300 Read block 3921 len 24576
20810 084/300 Read block 4332 len 32768
20811 085/300 Read block 490 len 12288
20812 086/300 Read block 2739 len 28672
20813 087/300 Read block 3096 len 40960
20814 088/300 Read block 4462 len 12288
20815 089/300 Read block 2507 len 28672
20816 090/300 Write block 1185 len 28672 data: 4 5 6 5 6 6 6
20817 091/300 Write block 676 len 24576 data: 4 4 6 5 3 4
20818 092/300 Read block 4656 len 40960
20819 093/300 Flush
20820 094/300 Read block 3564 len 12288
20821 095/300 Write block 1186 len 8192 data: 6 7
20822 096/300 Read block 2886 len 16384
20823 097/300 Write block 4503 len 12288 data: 4 5 6
20824 098/300 Read block 3733 len 36864
20825 099/300 Write block 232 len 20480 data: 5 5 3 3 3
20826 100/300 Read block 3922 len 24576
20827 101/300 Write block 4610 len 20480 data: 4 4 5 3 2
20828 102/300 Write block 3468 len 8192 data: 2 2
20829 103/300 Read block 4583 len 24576
20830 104/300 Write block 730 len 36864 data: 5 7 6 6 6 7 7 7 6
20831 105/300 Write block 4161 len 28672 data: 3 3 3 3 3 4 4
20832 106/300 Read block 1230 len 28672
20833 107/300 Read block 2193 len 8192
20834 108/300 Flush
20835 109/300 Read block 2390 len 8192
20836 110/300 Write block 2427 len 8192 data: 3 4
20837 111/300 Read block 4728 len 20480
20838 112/300 Read block 4548 len 36864
20839 113/300 Read block 1233 len 16384
20840 114/300 Read block 3458 len 32768
20841 115/300 Flush
20842 116/300 Read block 3128 len 24576
20843 117/300 Write block 2481 len 4096 data: 4
20844 118/300 Read block 3855 len 20480
20845 119/300 Write block 727 len 8192 data: 5 5
20846 120/300 Write block 222 len 20480 data: 6 5 4 4 4
20847 121/300 Write block 1921 len 32768 data: 6 8 8 7 6 6 5 5
20848 122/300 Write block 3537 len 20480 data: 3 2 2 2 2
20849 123/300 Read block 1694 len 4096
20850 124/300 Write block 4175 len 24576 data: 6 6 6 5 4 5
20851 125/300 Write block 3136 len 20480 data: 3 2 2 3 5
20852 126/300 Read block 1924 len 36864
20853 127/300 Write block 4108 len 24576 data: 5 4 5 4 5 6
20854 128/300 Write block 1872 len 32768 data: 4 4 3 3 3 4 3 3
20855 129/300 Write block 2191 len 40960 data: 4 4 4 3 3 4 4 6 5 4
20856 130/300 Write block 2595 len 20480 data: 2 2 2 4 4
20857 131/300 Read block 3864 len 40960
20858 132/300 Flush
20859 133/300 Read block 1485 len 24576
20860 134/300 Read block 2425 len 20480
20861 135/300 Read block 2941 len 32768
20862 136/300 Write block 3983 len 32768 data: 5 7 4 4 3 5 6 8
20863 137/300 Write block 3342 len 20480 data: 4 4 4 2 2
20864 138/300 Read block 1007 len 24576
20865 139/300 Read block 994 len 20480
20866 140/300 Read block 4817 len 28672
20867 141/300 Write block 2046 len 16384 data: 7 8 8 6
20868 142/300 Flush
20869 143/300 Write block 2191 len 36864 data: 5 5 5 4 4 5 5 7 6
20870 144/300 Write block 1521 len 36864 data: 3 4 5 6 5 3 3 3 4
20871 145/300 Read block 3058 len 4096
20872 146/300 Write block 2996 len 36864 data: 2 4 4 4 4 3 4 5 6
20873 147/300 Read block 435 len 28672
20874 148/300 Read block 2214 len 16384
20875 149/300 Read block 642 len 32768
20876 150/300 Read block 2920 len 8192
20877 151/300 Read block 2354 len 40960
20878 152/300 Write block 4202 len 20480 data: 7 8 7 7 7
20879 153/300 Write block 4931 len 36864 data: 2 2 3 4 4 4 4 5 4
20880 154/300 Read block 2525 len 20480
20881 155/300 Write block 3515 len 16384 data: 2 2 2 2
20882 156/300 Read block 4100 len 24576
20883 157/300 Write block 126 len 24576 data: 6 6 7 6 6 7
20884 158/300 Read block 2657 len 8192
20885 159/300 Read block 1282 len 28672
20886 160/300 Write block 1794 len 20480 data: 2 2 2 2 4
20887 161/300 Write block 145 len 16384 data: 4 5 5 6
20888 162/300 Write block 124 len 12288 data: 5 6 7
20889 163/300 Write block 2665 len 4096 data: 7
20890 164/300 Read block 4444 len 40960
20891 165/300 Write block 1351 len 4096 data: 2
20892 166/300 Write block 4793 len 28672 data: 4 5 4 4 5 5 6
20893 167/300 Flush
20894 168/300 Read block 3414 len 16384
20895 169/300 Read block 2749 len 36864
20896 170/300 Read block 1737 len 32768
20897 171/300 Write block 2742 len 4096 data: 5
20898 172/300 Read block 3398 len 4096
20899 173/300 Write block 4873 len 32768 data: 4 5 5 5 5 4 4 5
20900 174/300 Write block 1702 len 20480 data: 5 7 7 6 4
20901 175/300 Read block 3560 len 28672
20902 176/300 Write block 2866 len 12288 data: 3 3 3
20903 177/300 Read block 4386 len 16384
20904 178/300 Write block 218 len 24576 data: 4 5 6 6 7 6
20905 179/300 Read block 4740 len 8192
20906 180/300 Write block 2867 len 24576 data: 4 4 3 3 2 2
20907 181/300 Write block 3678 len 36864 data: 3 4 4 3 3 3 4 4 4
20908 182/300 Write block 2726 len 12288 data: 6 5 5
20909 183/300 Flush
20910 184/300 Write block 3545 len 8192 data: 2 2
20911 185/300 Write block 4486 len 12288 data: 3 3 2
20912 186/300 Write block 2333 len 28672 data: 5 4 5 5 4 4 5
20913 187/300 Write block 4969 len 12288 data: 3 3 3
20914 188/300 Flush
20915 189/300 Write block 2864 len 40960 data: 5 3 4 5 5 4 4 3 3 2
20916 190/300 Flush
20917 191/300 Flush
20918 192/300 Read block 1479 len 16384
20919 193/300 Write block 1836 len 36864 data: 3 5 5 6 5 5 4 4 4
20920 194/300 Write block 2488 len 32768 data: 4 4 4 4 3 4 6 6
20921 195/300 Write block 4980 len 4096 data: 5
20922 196/300 Write block 636 len 12288 data: 6 5 6
20923 197/300 Read block 229 len 36864
20924 198/300 Read block 2785 len 40960
20925 199/300 Write block 4384 len 16384 data: 5 4 4 4
20926 200/300 Read block 934 len 12288
20927 201/300 Read block 3128 len 8192
20928 202/300 Write block 3718 len 24576 data: 3 3 4 4 5 4
20929 203/300 Write block 3920 len 40960 data: 7 8 9 7 7 7 6 6 5 5
20930 204/300 Read block 1934 len 16384
20931 205/300 Read block 3535 len 24576
20932 206/300 Write block 2456 len 4096 data: 5
20933 207/300 Write block 788 len 12288 data: 6 6 6
20934 208/300 Read block 214 len 16384
20935 209/300 Read block 2209 len 12288
20936 210/300 Read block 4543 len 20480
20937 211/300 Write block 2468 len 20480 data: 5 3 3 3 3
20938 212/300 Write block 4404 len 40960 data: 2 2 3 3 3 3 4 5 5 5
20939 213/300 Write block 4732 len 40960 data: 3 3 3 3 3 4 4 4 3 3
20940 214/300 Write block 1242 len 4096 data: 3
20941 215/300 Write block 1647 len 24576 data: 4 4 5 5 4 4
20942 216/300 Write block 578 len 12288 data: 3 3 4
20943 217/300 Read block 2170 len 28672
20944 218/300 Write block 377 len 8192 data: 3 5
20945 219/300 Write block 352 len 28672 data: 9 7 6 5 5 3 2
20946 220/300 Read block 2415 len 36864
20947 221/300 Read block 401 len 24576
20948 222/300 Write block 244 len 28672 data: 6 6 5 4 4 4 3
20949 223/300 Write block 4547 len 16384 data: 8 8 8 6
20950 224/300 Write block 1981 len 20480 data: 4 3 4 4 3
20951 225/300 Read block 1034 len 28672
20952 226/300 Read block 4135 len 8192
20953 227/300 Write block 4068 len 40960 data: 4 4 4 5 5 7 7 9 7 6
20954 228/300 Write block 575 len 20480 data: 3 2 2 4 4
20955 229/300 Flush
20956 230/300 Read block 4090 len 12288
20957 231/300 Write block 1628 len 12288 data: 5 6 5
20958 232/300 Read block 2566 len 12288
20959 233/300 Read block 56 len 16384
20960 234/300 Flush
20961 235/300 Flush
20962 236/300 Read block 2425 len 20480
20963 237/300 Read block 3239 len 16384
20964 238/300 Flush
20965 239/300 Write block 701 len 12288 data: 5 4 4
20966 240/300 Read block 3001 len 28672
20967 241/300 Read block 2692 len 4096
20968 242/300 Write block 2667 len 40960 data: 6 6 4 3 3 4 3 3 3 3
20969 243/300 Write block 4866 len 8192 data: 4 3
20970 244/300 Write block 2713 len 20480 data: 4 4 2 3 3
20971 245/300 Write block 1143 len 4096 data: 6
20972 246/300 Read block 824 len 40960
20973 247/300 Read block 1901 len 32768
20974 248/300 Write block 2144 len 4096 data: 4
20975 249/300 Write block 4677 len 24576 data: 2 2 2 4 4 5
20976 250/300 Read block 687 len 40960
20977 251/300 Flush
20978 252/300 Read block 3917 len 24576
20979 253/300 Read block 1224 len 4096
20980 254/300 Write block 4156 len 20480 data: 4 4 3 3 3
20981 255/300 Write block 1888 len 36864 data: 4 6 6 5 5 5 4 3 3
20982 256/300 Read block 2031 len 8192
20983 257/300 Read block 1794 len 32768
20984 258/300 Flush
20985 259/300 Write block 1909 len 20480 data: 7 7 7 5 4
20986 260/300 Write block 1177 len 12288 data: 3 4 4
20987 261/300 Write block 1873 len 12288 data: 5 4 4
20988 262/300 Write block 1733 len 20480 data: 4 4 4 5 4
20989 263/300 Flush
20990 264/300 Write block 4463 len 28672 data: 5 6 6 5 5 5 4
20991 265/300 Read block 3306 len 8192
20992 266/300 Write block 217 len 40960 data: 5 5 6 7 7 8 7 5 5 5
20993 267/300 Read block 545 len 28672
20994 268/300 Read block 926 len 32768
20995 269/300 Write block 1918 len 20480 data: 6 5 6 7 9
20996 270/300 Write block 1488 len 20480 data: 5 4 4 5 4
20997 271/300 Write block 491 len 28672 data: 7 5 3 2 2 2 2
20998 272/300 Read block 2548 len 16384
20999 273/300 Read block 1238 len 4096
21000 274/300 Read block 317 len 24576
21001 275/300 Write block 4821 len 24576 data: 2 2 2 4 5 5
21002 276/300 Read block 4393 len 12288
21003 277/300 Write block 1467 len 28672 data: 7 6 6 5 4 3 3
21004 278/300 Read block 836 len 24576
21005 279/300 Flush
21006 280/300 Write block 2663 len 8192 data: 7 6
21007 281/300 Read block 2702 len 4096
21008 282/300 Write block 2382 len 28672 data: 4 3 3 3 3 3 4
21009 283/300 Read block 4678 len 28672
21010 284/300 Read block 3813 len 24576
21011 285/300 Write block 2279 len 40960 data: 4 4 5 5 5 5 5 5 5 5
21012 286/300 Write block 3505 len 20480 data: 3 3 3 4 4
21013 287/300 Read block 1265 len 36864
21014 288/300 Read block 264 len 12288
21015 289/300 Read block 3546 len 36864
21016 290/300 Write block 564 len 36864 data: 4 3 3 3 3 2 2 2 3
21017 291/300 Read block 4591 len 20480
21018 292/300 Read block 308 len 36864
21019 293/300 Write block 602 len 16384 data: 7 7 7 7
21020 294/300 Read block 2964 len 28672
21021 295/300 Read block 1613 len 20480
21022 296/300 Write block 1727 len 8192 data: 4 5
21023 297/300 Read block 2899 len 20480
21024 298/300 Write block 2376 len 4096 data: 4
21025 299/300 Write block 4606 len 20480 data: 4 4 4 4 5
21026 300/300 Write block 3935 len 12288 data: 6 5 5
21027 301/300 Read block 4923 len 40960
21028 Main task has work to do, go find it
21029 got dsc Start(2)
21030 start 2
21031 Mar 07 18:17:13.107 INFO request completed, latency_us: 182, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3ed89d5a-54c7-448d-a79c-02dd11a0365f, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
21032 Replay: started 2, returned:Ok(())
21033 ----------------------------------------------------------------
21034 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21035 States: Active Active Offline
21036 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21037 6132 Acked Read 10 Done Done Sent false
21038 6133 Acked Read 9 Done Done Sent false
21039 6134 Acked Write 4 Done Done Sent false
21040 6135 Acked Write 5 Done Done Sent false
21041 6136 Acked Flush 0 Done Done Sent false
21042 6137 Acked Read 1 Done Done Sent false
21043 6138 Acked Write 2 Done Done Sent false
21044 6139 Acked Flush 0 Done Done Sent false
21045 6140 Acked Read 8 Done Done Sent false
21046 6141 Acked Read 6 Done Done Sent false
21047 6142 Acked Write 1 Done Done Sent false
21048 6143 Acked Read 3 Done Done Sent false
21049 6144 Acked Write 2 Done Done Sent false
21050 6145 Acked Flush 0 Done Done Sent false
21051 6146 Acked Write 6 Done Done Sent false
21052 6147 Acked Read 5 Done Done Sent false
21053 6148 Acked Read 9 Done Done Sent false
21054 6149 Acked Write 10 Done Done Sent false
21055 6150 Acked Write 3 Done Done Sent false
21056 6151 Acked Write 4 Done Done Sent false
21057 6152 Acked Write 1 Done Done Sent false
21058 6153 Acked Read 3 Done Done Sent false
21059 6154 Acked Write 5 Done Done Sent false
21060 6155 Acked Write 3 Done Done Sent false
21061 6156 Acked Read 7 Done Done Sent false
21062 6157 Acked Write 1 Done Done Sent false
21063 6158 Acked Write 9 Done Done Sent false
21064 6159 Acked Read 8 Done Done Sent false
21065 6160 Acked Flush 0 Done Done Sent false
21066 6161 Acked Read 9 Done Done Sent false
21067 6162 Acked Read 5 Done Done Sent false
21068 6163 Acked Write 1 Done Done Sent false
21069 6164 Acked Write 5 Done Done Sent false
21070 6165 Acked Read 5 Done Done Sent false
21071 6166 Acked Write 7 Done Done Sent false
21072 6167 Acked Write 1 Done Done Sent false
21073 6168 Acked Read 10 Done Done Sent false
21074 6169 Acked Read 8 Done Done Sent false
21075 6170 Acked Flush 0 Done Done Sent false
21076 6171 Acked Read 3 Done Done Sent false
21077 6172 Acked Write 10 Done Done Sent false
21078 6173 Acked Write 6 Done Done Sent false
21079 6174 Acked Read 6 Done Done Sent false
21080 6175 Acked Read 1 Done Done Sent false
21081 6176 Acked Read 1 Done Done Sent false
21082 6177 Acked Read 7 Done Done Sent false
21083 6178 Acked Flush 0 Done Done Sent false
21084 6179 Acked Read 3 Done Done Sent false
21085 6180 Acked Write 7 Done Done Sent false
21086 6181 Acked Write 1 Done Done Sent false
21087 6182 Acked Read 9 Done Done Sent false
21088 6183 Acked Read 7 Done Done Sent false
21089 6184 Acked Read 7 Done Done Sent false
21090 6185 Acked Flush 0 Done Done Sent false
21091 6186 Acked Read 1 Done Done Sent false
21092 6187 Acked Write 4 Done Done Sent false
21093 6188 Acked Write 7 Done Done Sent false
21094 6189 Acked Read 6 Done Done Sent false
21095 6190 Acked Write 3 Done Done Sent false
21096 6191 Acked Read 9 Done Done Sent false
21097 6192 Acked Read 9 Done Done Sent false
21098 6193 Acked Write 2 Done Done Sent false
21099 6194 Acked Read 9 Done Done Sent false
21100 6195 Acked Flush 0 Done Done Sent false
21101 6196 Acked Write 7 Done Done Sent false
21102 6197 Acked Write 10 Done Done Sent false
21103 6198 Acked Flush 0 Done Done Sent false
21104 6199 Acked Read 6 Done Done Sent false
21105 6200 Acked Read 6 Done Done Sent false
21106 6201 Acked Write 9 Done Done Sent false
21107 6202 Acked Write 1 Done Done Sent false
21108 6203 Acked Write 7 Done Done Sent false
21109 6204 Acked Write 1 Done Done Sent false
21110 6205 Acked Read 5 Done Done Sent false
21111 6206 Acked Read 7 Done Done Sent false
21112 6207 Acked Read 6 Done Done Sent false
21113 6208 Acked Read 5 Done Done Sent false
21114 6209 Acked Read 3 Done Done Sent false
21115 6210 Acked Read 9 Done Done Sent false
21116 6211 Acked Read 5 Done Done Sent false
21117 6212 Acked Write 4 Done Done Sent false
21118 6213 Acked Read 9 Done Done Sent false
21119 6214 Acked Read 6 Done Done Sent false
21120 6215 Acked Read 8 Done Done Sent false
21121 6216 Acked Read 3 Done Done Sent false
21122 6217 Acked Read 7 Done Done Sent false
21123 6218 Acked Read 10 Done Done Sent false
21124 6219 Acked Read 3 Done Done Sent false
21125 6220 Acked Read 7 Done Done Sent false
21126 6221 Acked Write 7 Done Done Sent false
21127 6222 Acked Write 6 Done Done Sent false
21128 6223 Acked Read 10 Done Done Sent false
21129 6224 Acked Flush 0 Done Done Sent false
21130 6225 Acked Read 3 Done Done Sent false
21131 6226 Acked Write 2 Done Done Sent false
21132 6227 Acked Read 4 Done Done Sent false
21133 6228 Acked Write 3 Done Done Sent false
21134 6229 Acked Read 9 Done Done Sent false
21135 6230 Acked Write 5 Done Done Sent false
21136 6231 Acked Read 6 Done Done Sent false
21137 6232 Acked Write 5 Done Done Sent false
21138 6233 Acked Write 2 Done Done Sent false
21139 6234 Acked Read 6 Done Done Sent false
21140 6235 Acked Write 9 Done Done Sent false
21141 6236 Acked Write 7 Done Done Sent false
21142 6237 Acked Read 7 Done Done Sent false
21143 6238 Acked Read 2 Done Done Sent false
21144 6239 Acked Flush 0 Done Done Sent false
21145 6240 Acked Read 2 Done Done Sent false
21146 6241 Acked Write 2 Done Done Sent false
21147 6242 Acked Read 5 Done Done Sent false
21148 6243 Acked Read 9 Done Done Sent false
21149 6244 Acked Read 4 Done Done Sent false
21150 6245 Acked Read 8 Done Done Sent false
21151 6246 Acked Flush 0 Done Done Sent false
21152 6247 Acked Read 6 Done Done Sent false
21153 6248 Acked Write 1 Done Done Sent false
21154 6249 Acked Read 5 Done Done Sent false
21155 6250 Acked Write 2 Done Done Sent false
21156 6251 Acked Write 5 Done Done Sent false
21157 6252 Acked Write 8 Done Done Sent false
21158 6253 Acked Write 5 Done Done Sent false
21159 6254 Acked Read 1 Done Done Sent false
21160 6255 Acked Write 6 Done Done Sent false
21161 6256 Acked Write 5 Done Done Sent false
21162 6257 Acked Read 9 Done Done Sent false
21163 6258 Acked Write 6 Done Done Sent false
21164 6259 Acked Write 8 Done Done Sent false
21165 6260 Acked Write 10 Done Done Sent false
21166 6261 Acked Write 5 Done Done Sent false
21167 6262 Acked Read 10 Done Done Sent false
21168 6263 Acked Flush 0 Done Done Sent false
21169 6264 Acked Read 6 Done Done Sent false
21170 6265 Acked Read 5 Done Done Sent false
21171 6266 Acked Read 8 Done Done Sent false
21172 6267 Acked Write 8 Done Done Sent false
21173 6268 Acked Write 5 Done Done Sent false
21174 6269 Acked Read 6 Done Done Sent false
21175 6270 Acked Read 5 Done Done Sent false
21176 6271 Acked Read 7 Done Done Sent false
21177 6272 Acked Write 4 Done Done Sent false
21178 6273 Acked Flush 0 Done Done Sent false
21179 6274 Acked Write 9 Done Done Sent false
21180 6275 Acked Write 9 Done Done Sent false
21181 6276 Acked Read 1 Done Done Sent false
21182 6277 Acked Write 9 Done Done Sent false
21183 6278 Acked Read 7 Done Done Sent false
21184 6279 Acked Read 4 Done Done Sent false
21185 6280 Acked Read 8 Done Done Sent false
21186 6281 Acked Read 2 Done Done Sent false
21187 6282 Acked Read 10 Done Done Sent false
21188 6283 Acked Write 5 Done Done Sent false
21189 6284 Acked Write 9 Done Done Sent false
21190 6285 Acked Read 5 Done Done Sent false
21191 6286 Acked Write 4 Done Done Sent false
21192 6287 Acked Read 6 Done Done Sent false
21193 6288 Acked Write 6 Done Done Sent false
21194 6289 Acked Read 2 Done Done Sent false
21195 6290 Acked Read 7 Done Done Sent false
21196 6291 Acked Write 5 Done Done Sent false
21197 6292 Acked Write 4 Done Done Sent false
21198 6293 Acked Write 3 Done Done Sent false
21199 6294 Acked Write 1 Done Done Sent false
21200 6295 Acked Read 10 Done Done Sent false
21201 6296 Acked Write 1 Done Done Sent false
21202 6297 Acked Write 7 Done Done Sent false
21203 6298 Acked Flush 0 Done Done Sent false
21204 6299 Acked Read 4 Done Done Sent false
21205 6300 Acked Read 9 Done Done Sent false
21206 6301 Acked Read 8 Done Done Sent false
21207 6302 Acked Write 1 Done Done Sent false
21208 6303 Acked Read 1 Done Done Sent false
21209 6304 Acked Write 8 Done Done Sent false
21210 6305 Acked Write 5 Done Done Sent false
21211 6306 Acked Read 7 Done Done Sent false
21212 6307 Acked Write 3 Done Done Sent false
21213 6308 Acked Read 4 Done Done Sent false
21214 6309 Acked Write 6 Done Done Sent false
21215 6310 Acked Read 2 Done Done Sent false
21216 6311 Acked Write 6 Done Done Sent false
21217 6312 Acked Write 9 Done Done Sent false
21218 6313 Acked Write 3 Done Done Sent false
21219 6314 Acked Flush 0 Done Done Sent false
21220 6315 Acked Write 2 Done Done Sent false
21221 6316 Acked Write 3 Done Done Sent false
21222 6317 Acked Write 7 Done Done Sent false
21223 6318 Acked Write 3 Done Done Sent false
21224 6319 Acked Flush 0 Done Done Sent false
21225 6320 Acked Write 10 Done Done Sent false
21226 6321 Acked Flush 0 Done Done Sent false
21227 6322 Acked Flush 0 Done Done Sent false
21228 6323 Acked Read 4 Done Done Sent false
21229 6324 Acked Write 9 Done Done Sent false
21230 6325 Acked Write 8 Done Done Sent false
21231 6326 Acked Write 1 Done Done Sent false
21232 6327 Acked Write 3 Done Done Sent false
21233 6328 Acked Read 9 Done Done Sent false
21234 6329 Acked Read 10 Done Done Sent false
21235 6330 Acked Write 4 Done Done Sent false
21236 6331 Acked Read 3 Done Done Sent false
21237 6332 Acked Read 2 Done Done Sent false
21238 6333 Acked Write 6 Done Done Sent false
21239 6334 Acked Write 10 Done Done Sent false
21240 6335 Acked Read 4 Done Done Sent false
21241 6336 Acked Read 6 Done Done Sent false
21242 6337 Acked Write 1 Done Done Sent false
21243 6338 Acked Write 3 Done Done Sent false
21244 6339 Acked Read 4 Done Done Sent false
21245 6340 Acked Read 3 Done Done Sent false
21246 6341 Acked Read 5 Done Done Sent false
21247 6342 Acked Write 5 Done Done Sent false
21248 6343 Acked Write 10 Done Done Sent false
21249 6344 Acked Write 10 Done Done Sent false
21250 6345 Acked Write 1 Done Done Sent false
21251 6346 Acked Write 6 Done Done Sent false
21252 6347 Acked Write 3 Done Done Sent false
21253 6348 Acked Read 7 Done Done Sent false
21254 6349 Acked Write 2 Done Done Sent false
21255 6350 Acked Write 7 Done Done Sent false
21256 6351 Acked Read 9 Done Done Sent false
21257 6352 Acked Read 6 Done Done Sent false
21258 6353 Acked Write 7 Done Done Sent false
21259 6354 Acked Write 4 Done Done Sent false
21260 6355 Acked Write 5 Done Done Sent false
21261 6356 Acked Read 7 Done Done Sent false
21262 6357 Acked Read 2 Done Done Sent false
21263 6358 Acked Write 10 Done Done Sent false
21264 6359 Acked Write 5 Done Done Sent false
21265 6360 Acked Flush 0 Done Done Sent false
21266 6361 Acked Read 3 Done Done Sent false
21267 6362 Acked Write 3 Done Done Sent false
21268 6363 Acked Read 3 Done Done Sent false
21269 6364 Acked Read 4 Done Done Sent false
21270 6365 Acked Flush 0 Done Done Sent false
21271 6366 Acked Flush 0 Done Done Sent false
21272 6367 Acked Read 5 Done Done Sent false
21273 6368 Acked Read 4 Done Done Sent false
21274 6369 Acked Flush 0 Done Done Sent false
21275 6370 Acked Write 3 Done Done Sent false
21276 6371 Acked Read 7 Done Done Sent false
21277 6372 Acked Read 1 Done Done Sent false
21278 6373 Acked Write 10 Done Done Sent false
21279 6374 Acked Write 2 Done Done Sent false
21280 6375 Acked Write 5 Done Done Sent false
21281 6376 Acked Write 1 Done Done Sent false
21282 6377 Acked Read 10 Done Done Sent false
21283 6378 Acked Read 8 Done Done Sent false
21284 6379 Acked Write 1 Done Done Sent false
21285 6380 Acked Write 6 Done Done Sent false
21286 6381 Acked Read 10 Done Done Sent false
21287 6382 Acked Flush 0 Done Done Sent false
21288 6383 Acked Read 6 Done Done Sent false
21289 6384 Acked Read 1 Done Done Sent false
21290 6385 Acked Write 5 Done Done Sent false
21291 6386 Acked Write 9 Done Done Sent false
21292 6387 Acked Read 2 Done Done Sent false
21293 6388 Acked Read 8 Done Done Sent false
21294 6389 Acked Flush 0 Done Done Sent false
21295 6390 Acked Write 5 Done Done Sent false
21296 6391 Acked Write 3 Done Done Sent false
21297 6392 Acked Write 3 Done Done Sent false
21298 6393 Acked Write 5 Done Done Sent false
21299 6394 Acked Flush 0 Done Done Sent false
21300 6395 Acked Write 7 Done Done Sent false
21301 6396 Acked Read 2 Done Done Sent false
21302 6397 Acked Write 10 Done Done Sent false
21303 6398 Acked Read 7 Done Done Sent false
21304 6399 Acked Read 8 Done Done Sent false
21305 6400 Acked Write 5 Done Done Sent false
21306 6401 Acked Write 5 Done Done Sent false
21307 6402 Acked Write 7 Done Done Sent false
21308 6403 Acked Read 4 Done Done Sent false
21309 6404 Acked Read 1 Done Done Sent false
21310 6405 Acked Read 6 Done Done Sent false
21311 6406 Acked Write 6 Done Done Sent false
21312 6407 Acked Read 3 Done Done Sent false
21313 6408 Acked Write 7 Done Done Sent false
21314 6409 Acked Read 6 Done Done Sent false
21315 6410 Acked Flush 0 Done Done Sent false
21316 6411 Acked Write 2 Done Done Sent false
21317 6412 Acked Read 1 Done Done Sent false
21318 6413 Acked Write 7 Done Done Sent false
21319 6414 Acked Read 7 Done Done Sent false
21320 6415 Acked Read 6 Done Done Sent false
21321 6416 Acked Write 10 Done Done Sent false
21322 6417 Acked Write 5 Done Done Sent false
21323 6418 Acked Read 9 Done Done Sent false
21324 6419 Acked Read 3 Done Done Sent false
21325 6420 Acked Read 9 Done Done Sent false
21326 6421 Acked Write 9 Done Done Sent false
21327 6422 Acked Read 5 Done Done Sent false
21328 6423 Acked Read 9 Done Done Sent false
21329 6424 Acked Write 4 Done Done Sent false
21330 6425 Acked Read 7 Done Done Sent false
21331 6426 Acked Read 5 Done Done Sent false
21332 6427 Acked Write 2 Done Done Sent false
21333 6428 Acked Read 5 Done Done Sent false
21334 6429 Acked Write 1 Done Done Sent false
21335 6430 Acked Write 5 Done Done Sent false
21336 6431 Acked Write 3 Done Done Sent false
21337 6432 Acked Read 10 Done Done Sent false
21338 STATES DS:0 DS:1 DS:2 TOTAL
21339 Sent 0 0 301 301
21340 Done 301 301 0 602
21341 Skipped 0 0 0 0
21342 Error 0 0 0 0
21343 Last Flush: Some(JobId(6410)) Some(JobId(6410)) Some(JobId(6131))
21344 Downstairs last five completed: 6131 6130 6129 6128 6127
21345 Upstairs last five completed: 6432 6431 6430 6429 6428
21346 CLIENT: Up:0 ds:301 act:2
213472025-03-07T18:17:13.608ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
213482025-03-07T18:17:14.412ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
21349 [8830] got start action
21350 [8830] I am going to restart
21351 Starting downstairs at port 8830
21352 Make output file at "/tmp/dsc/downstairs-8830.txt"
21353 [8830][2] reports Starting
21354 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1226)
21355 [8830][2] reports Running
213562025-03-07T18:17:15.413ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
21357 ----------------------------------------------------------------
21358 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21359 States: Active Active Offline
21360 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21361 6132 Acked Read 10 Done Done Sent false
21362 6133 Acked Read 9 Done Done Sent false
21363 6134 Acked Write 4 Done Done Sent false
21364 6135 Acked Write 5 Done Done Sent false
21365 6136 Acked Flush 0 Done Done Sent false
21366 6137 Acked Read 1 Done Done Sent false
21367 6138 Acked Write 2 Done Done Sent false
21368 6139 Acked Flush 0 Done Done Sent false
21369 6140 Acked Read 8 Done Done Sent false
21370 6141 Acked Read 6 Done Done Sent false
21371 6142 Acked Write 1 Done Done Sent false
21372 6143 Acked Read 3 Done Done Sent false
21373 6144 Acked Write 2 Done Done Sent false
21374 6145 Acked Flush 0 Done Done Sent false
21375 6146 Acked Write 6 Done Done Sent false
21376 6147 Acked Read 5 Done Done Sent false
21377 6148 Acked Read 9 Done Done Sent false
21378 6149 Acked Write 10 Done Done Sent false
21379 6150 Acked Write 3 Done Done Sent false
21380 6151 Acked Write 4 Done Done Sent false
21381 6152 Acked Write 1 Done Done Sent false
21382 6153 Acked Read 3 Done Done Sent false
21383 6154 Acked Write 5 Done Done Sent false
21384 6155 Acked Write 3 Done Done Sent false
21385 6156 Acked Read 7 Done Done Sent false
21386 6157 Acked Write 1 Done Done Sent false
21387 6158 Acked Write 9 Done Done Sent false
21388 6159 Acked Read 8 Done Done Sent false
21389 6160 Acked Flush 0 Done Done Sent false
21390 6161 Acked Read 9 Done Done Sent false
21391 6162 Acked Read 5 Done Done Sent false
21392 6163 Acked Write 1 Done Done Sent false
21393 6164 Acked Write 5 Done Done Sent false
21394 6165 Acked Read 5 Done Done Sent false
21395 6166 Acked Write 7 Done Done Sent false
21396 6167 Acked Write 1 Done Done Sent false
21397 6168 Acked Read 10 Done Done Sent false
21398 6169 Acked Read 8 Done Done Sent false
21399 6170 Acked Flush 0 Done Done Sent false
21400 6171 Acked Read 3 Done Done Sent false
21401 6172 Acked Write 10 Done Done Sent false
21402 6173 Acked Write 6 Done Done Sent false
21403 6174 Acked Read 6 Done Done Sent false
21404 6175 Acked Read 1 Done Done Sent false
21405 6176 Acked Read 1 Done Done Sent false
21406 6177 Acked Read 7 Done Done Sent false
21407 6178 Acked Flush 0 Done Done Sent false
21408 6179 Acked Read 3 Done Done Sent false
21409 6180 Acked Write 7 Done Done Sent false
21410 6181 Acked Write 1 Done Done Sent false
21411 6182 Acked Read 9 Done Done Sent false
21412 6183 Acked Read 7 Done Done Sent false
21413 6184 Acked Read 7 Done Done Sent false
21414 6185 Acked Flush 0 Done Done Sent false
21415 6186 Acked Read 1 Done Done Sent false
21416 6187 Acked Write 4 Done Done Sent false
21417 6188 Acked Write 7 Done Done Sent false
21418 6189 Acked Read 6 Done Done Sent false
21419 6190 Acked Write 3 Done Done Sent false
21420 6191 Acked Read 9 Done Done Sent false
21421 6192 Acked Read 9 Done Done Sent false
21422 6193 Acked Write 2 Done Done Sent false
21423 6194 Acked Read 9 Done Done Sent false
21424 6195 Acked Flush 0 Done Done Sent false
21425 6196 Acked Write 7 Done Done Sent false
21426 6197 Acked Write 10 Done Done Sent false
21427 6198 Acked Flush 0 Done Done Sent false
21428 6199 Acked Read 6 Done Done Sent false
21429 6200 Acked Read 6 Done Done Sent false
21430 6201 Acked Write 9 Done Done Sent false
21431 6202 Acked Write 1 Done Done Sent false
21432 6203 Acked Write 7 Done Done Sent false
21433 6204 Acked Write 1 Done Done Sent false
21434 6205 Acked Read 5 Done Done Sent false
21435 6206 Acked Read 7 Done Done Sent false
21436 6207 Acked Read 6 Done Done Sent false
21437 6208 Acked Read 5 Done Done Sent false
21438 6209 Acked Read 3 Done Done Sent false
21439 6210 Acked Read 9 Done Done Sent false
21440 6211 Acked Read 5 Done Done Sent false
21441 6212 Acked Write 4 Done Done Sent false
21442 6213 Acked Read 9 Done Done Sent false
21443 6214 Acked Read 6 Done Done Sent false
21444 6215 Acked Read 8 Done Done Sent false
21445 6216 Acked Read 3 Done Done Sent false
21446 6217 Acked Read 7 Done Done Sent false
21447 6218 Acked Read 10 Done Done Sent false
21448 6219 Acked Read 3 Done Done Sent false
21449 6220 Acked Read 7 Done Done Sent false
21450 6221 Acked Write 7 Done Done Sent false
21451 6222 Acked Write 6 Done Done Sent false
21452 6223 Acked Read 10 Done Done Sent false
21453 6224 Acked Flush 0 Done Done Sent false
21454 6225 Acked Read 3 Done Done Sent false
21455 6226 Acked Write 2 Done Done Sent false
21456 6227 Acked Read 4 Done Done Sent false
21457 6228 Acked Write 3 Done Done Sent false
21458 6229 Acked Read 9 Done Done Sent false
21459 6230 Acked Write 5 Done Done Sent false
21460 6231 Acked Read 6 Done Done Sent false
21461 6232 Acked Write 5 Done Done Sent false
21462 6233 Acked Write 2 Done Done Sent false
21463 6234 Acked Read 6 Done Done Sent false
21464 6235 Acked Write 9 Done Done Sent false
21465 6236 Acked Write 7 Done Done Sent false
21466 6237 Acked Read 7 Done Done Sent false
21467 6238 Acked Read 2 Done Done Sent false
21468 6239 Acked Flush 0 Done Done Sent false
21469 6240 Acked Read 2 Done Done Sent false
21470 6241 Acked Write 2 Done Done Sent false
21471 6242 Acked Read 5 Done Done Sent false
21472 6243 Acked Read 9 Done Done Sent false
21473 6244 Acked Read 4 Done Done Sent false
21474 6245 Acked Read 8 Done Done Sent false
21475 6246 Acked Flush 0 Done Done Sent false
21476 6247 Acked Read 6 Done Done Sent false
21477 6248 Acked Write 1 Done Done Sent false
21478 6249 Acked Read 5 Done Done Sent false
21479 6250 Acked Write 2 Done Done Sent false
21480 6251 Acked Write 5 Done Done Sent false
21481 6252 Acked Write 8 Done Done Sent false
21482 6253 Acked Write 5 Done Done Sent false
21483 6254 Acked Read 1 Done Done Sent false
21484 6255 Acked Write 6 Done Done Sent false
21485 6256 Acked Write 5 Done Done Sent false
21486 6257 Acked Read 9 Done Done Sent false
21487 6258 Acked Write 6 Done Done Sent false
21488 6259 Acked Write 8 Done Done Sent false
21489 6260 Acked Write 10 Done Done Sent false
21490 6261 Acked Write 5 Done Done Sent false
21491 6262 Acked Read 10 Done Done Sent false
21492 6263 Acked Flush 0 Done Done Sent false
21493 6264 Acked Read 6 Done Done Sent false
21494 6265 Acked Read 5 Done Done Sent false
21495 6266 Acked Read 8 Done Done Sent false
21496 6267 Acked Write 8 Done Done Sent false
21497 6268 Acked Write 5 Done Done Sent false
21498 6269 Acked Read 6 Done Done Sent false
21499 6270 Acked Read 5 Done Done Sent false
21500 6271 Acked Read 7 Done Done Sent false
21501 6272 Acked Write 4 Done Done Sent false
21502 6273 Acked Flush 0 Done Done Sent false
21503 6274 Acked Write 9 Done Done Sent false
21504 6275 Acked Write 9 Done Done Sent false
21505 6276 Acked Read 1 Done Done Sent false
21506 6277 Acked Write 9 Done Done Sent false
21507 6278 Acked Read 7 Done Done Sent false
21508 6279 Acked Read 4 Done Done Sent false
21509 6280 Acked Read 8 Done Done Sent false
21510 6281 Acked Read 2 Done Done Sent false
21511 6282 Acked Read 10 Done Done Sent false
21512 6283 Acked Write 5 Done Done Sent false
21513 6284 Acked Write 9 Done Done Sent false
21514 6285 Acked Read 5 Done Done Sent false
21515 6286 Acked Write 4 Done Done Sent false
21516 6287 Acked Read 6 Done Done Sent false
21517 6288 Acked Write 6 Done Done Sent false
21518 6289 Acked Read 2 Done Done Sent false
21519 6290 Acked Read 7 Done Done Sent false
21520 6291 Acked Write 5 Done Done Sent false
21521 6292 Acked Write 4 Done Done Sent false
21522 6293 Acked Write 3 Done Done Sent false
21523 6294 Acked Write 1 Done Done Sent false
21524 6295 Acked Read 10 Done Done Sent false
21525 6296 Acked Write 1 Done Done Sent false
21526 6297 Acked Write 7 Done Done Sent false
21527 6298 Acked Flush 0 Done Done Sent false
21528 6299 Acked Read 4 Done Done Sent false
21529 6300 Acked Read 9 Done Done Sent false
21530 6301 Acked Read 8 Done Done Sent false
21531 6302 Acked Write 1 Done Done Sent false
21532 6303 Acked Read 1 Done Done Sent false
21533 6304 Acked Write 8 Done Done Sent false
21534 6305 Acked Write 5 Done Done Sent false
21535 6306 Acked Read 7 Done Done Sent false
21536 6307 Acked Write 3 Done Done Sent false
21537 6308 Acked Read 4 Done Done Sent false
21538 6309 Acked Write 6 Done Done Sent false
21539 6310 Acked Read 2 Done Done Sent false
21540 6311 Acked Write 6 Done Done Sent false
21541 6312 Acked Write 9 Done Done Sent false
21542 6313 Acked Write 3 Done Done Sent false
21543 6314 Acked Flush 0 Done Done Sent false
21544 6315 Acked Write 2 Done Done Sent false
21545 6316 Acked Write 3 Done Done Sent false
21546 6317 Acked Write 7 Done Done Sent false
21547 6318 Acked Write 3 Done Done Sent false
21548 6319 Acked Flush 0 Done Done Sent false
21549 6320 Acked Write 10 Done Done Sent false
21550 6321 Acked Flush 0 Done Done Sent false
21551 6322 Acked Flush 0 Done Done Sent false
21552 6323 Acked Read 4 Done Done Sent false
21553 6324 Acked Write 9 Done Done Sent false
21554 6325 Acked Write 8 Done Done Sent false
21555 6326 Acked Write 1 Done Done Sent false
21556 6327 Acked Write 3 Done Done Sent false
21557 6328 Acked Read 9 Done Done Sent false
21558 6329 Acked Read 10 Done Done Sent false
21559 6330 Acked Write 4 Done Done Sent false
21560 6331 Acked Read 3 Done Done Sent false
21561 6332 Acked Read 2 Done Done Sent false
21562 6333 Acked Write 6 Done Done Sent false
21563 6334 Acked Write 10 Done Done Sent false
21564 6335 Acked Read 4 Done Done Sent false
21565 6336 Acked Read 6 Done Done Sent false
21566 6337 Acked Write 1 Done Done Sent false
21567 6338 Acked Write 3 Done Done Sent false
21568 6339 Acked Read 4 Done Done Sent false
21569 6340 Acked Read 3 Done Done Sent false
21570 6341 Acked Read 5 Done Done Sent false
21571 6342 Acked Write 5 Done Done Sent false
21572 6343 Acked Write 10 Done Done Sent false
21573 6344 Acked Write 10 Done Done Sent false
21574 6345 Acked Write 1 Done Done Sent false
21575 6346 Acked Write 6 Done Done Sent false
21576 6347 Acked Write 3 Done Done Sent false
21577 6348 Acked Read 7 Done Done Sent false
21578 6349 Acked Write 2 Done Done Sent false
21579 6350 Acked Write 7 Done Done Sent false
21580 6351 Acked Read 9 Done Done Sent false
21581 6352 Acked Read 6 Done Done Sent false
21582 6353 Acked Write 7 Done Done Sent false
21583 6354 Acked Write 4 Done Done Sent false
21584 6355 Acked Write 5 Done Done Sent false
21585 6356 Acked Read 7 Done Done Sent false
21586 6357 Acked Read 2 Done Done Sent false
21587 6358 Acked Write 10 Done Done Sent false
21588 6359 Acked Write 5 Done Done Sent false
21589 6360 Acked Flush 0 Done Done Sent false
21590 6361 Acked Read 3 Done Done Sent false
21591 6362 Acked Write 3 Done Done Sent false
21592 6363 Acked Read 3 Done Done Sent false
21593 6364 Acked Read 4 Done Done Sent false
21594 6365 Acked Flush 0 Done Done Sent false
21595 6366 Acked Flush 0 Done Done Sent false
21596 6367 Acked Read 5 Done Done Sent false
21597 6368 Acked Read 4 Done Done Sent false
21598 6369 Acked Flush 0 Done Done Sent false
21599 6370 Acked Write 3 Done Done Sent false
21600 6371 Acked Read 7 Done Done Sent false
21601 6372 Acked Read 1 Done Done Sent false
21602 6373 Acked Write 10 Done Done Sent false
21603 6374 Acked Write 2 Done Done Sent false
21604 6375 Acked Write 5 Done Done Sent false
21605 6376 Acked Write 1 Done Done Sent false
21606 6377 Acked Read 10 Done Done Sent false
21607 6378 Acked Read 8 Done Done Sent false
21608 6379 Acked Write 1 Done Done Sent false
21609 6380 Acked Write 6 Done Done Sent false
21610 6381 Acked Read 10 Done Done Sent false
21611 6382 Acked Flush 0 Done Done Sent false
21612 6383 Acked Read 6 Done Done Sent false
21613 6384 Acked Read 1 Done Done Sent false
21614 6385 Acked Write 5 Done Done Sent false
21615 6386 Acked Write 9 Done Done Sent false
21616 6387 Acked Read 2 Done Done Sent false
21617 6388 Acked Read 8 Done Done Sent false
21618 6389 Acked Flush 0 Done Done Sent false
21619 6390 Acked Write 5 Done Done Sent false
21620 6391 Acked Write 3 Done Done Sent false
21621 6392 Acked Write 3 Done Done Sent false
21622 6393 Acked Write 5 Done Done Sent false
21623 6394 Acked Flush 0 Done Done Sent false
21624 6395 Acked Write 7 Done Done Sent false
21625 6396 Acked Read 2 Done Done Sent false
21626 6397 Acked Write 10 Done Done Sent false
21627 6398 Acked Read 7 Done Done Sent false
21628 6399 Acked Read 8 Done Done Sent false
21629 6400 Acked Write 5 Done Done Sent false
21630 6401 Acked Write 5 Done Done Sent false
21631 6402 Acked Write 7 Done Done Sent false
21632 6403 Acked Read 4 Done Done Sent false
21633 6404 Acked Read 1 Done Done Sent false
21634 6405 Acked Read 6 Done Done Sent false
21635 6406 Acked Write 6 Done Done Sent false
21636 6407 Acked Read 3 Done Done Sent false
21637 6408 Acked Write 7 Done Done Sent false
21638 6409 Acked Read 6 Done Done Sent false
21639 6410 Acked Flush 0 Done Done Sent false
21640 6411 Acked Write 2 Done Done Sent false
21641 6412 Acked Read 1 Done Done Sent false
21642 6413 Acked Write 7 Done Done Sent false
21643 6414 Acked Read 7 Done Done Sent false
21644 6415 Acked Read 6 Done Done Sent false
21645 6416 Acked Write 10 Done Done Sent false
21646 6417 Acked Write 5 Done Done Sent false
21647 6418 Acked Read 9 Done Done Sent false
21648 6419 Acked Read 3 Done Done Sent false
21649 6420 Acked Read 9 Done Done Sent false
21650 6421 Acked Write 9 Done Done Sent false
21651 6422 Acked Read 5 Done Done Sent false
21652 6423 Acked Read 9 Done Done Sent false
21653 6424 Acked Write 4 Done Done Sent false
21654 6425 Acked Read 7 Done Done Sent false
21655 6426 Acked Read 5 Done Done Sent false
21656 6427 Acked Write 2 Done Done Sent false
21657 6428 Acked Read 5 Done Done Sent false
21658 6429 Acked Write 1 Done Done Sent false
21659 6430 Acked Write 5 Done Done Sent false
21660 6431 Acked Write 3 Done Done Sent false
21661 6432 Acked Read 10 Done Done Sent false
21662 STATES DS:0 DS:1 DS:2 TOTAL
21663 Sent 0 0 301 301
21664 Done 301 301 0 602
21665 Skipped 0 0 0 0
21666 Error 0 0 0 0
21667 Last Flush: Some(JobId(6410)) Some(JobId(6410)) Some(JobId(6131))
21668 Downstairs last five completed: 6131 6130 6129 6128 6127
21669 Upstairs last five completed: 6432 6431 6430 6429 6428
21670 CLIENT: Up:0 ds:301 act:2
216712025-03-07T18:17:17.611ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216722025-03-07T18:17:18.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216732025-03-07T18:17:18.864ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216742025-03-07T18:17:18.864ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64104) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216752025-03-07T18:17:18.865ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216762025-03-07T18:17:18.865ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216772025-03-07T18:17:18.866ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216782025-03-07T18:17:18.866ZINFOcrucible: send last flush ID to this DS: Some(JobId(6131)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216792025-03-07T18:17:18.866ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216802025-03-07T18:17:18.866ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6131)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216812025-03-07T18:17:18.866ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6131)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
21682 ----------------------------------------------------------------
21683 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21684 Downstairs last five completed: 6433 6432 6431 6430 6429
21685 Upstairs last five completed: 6433 6432 6431 6430 6429
21686 CLIENT: Up:0 ds:0 act:3
21687 Replay: All jobs finished
21688 Main task has work to do, go find it
21689 got dsc Stop(0)
21690 stop 0
21691 Mar 07 18:17:21.112 INFO request completed, latency_us: 332, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 58661a79-5c5a-4782-973e-ec939b4e652e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998[8810] Got stop action so:false kr:false
21692 
216932025-03-07T18:17:21.112ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216942025-03-07T18:17:21.112ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216952025-03-07T18:17:21.112ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
216962025-03-07T18:17:21.112ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
21697 Mar 07 18:17:21.112 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4e483178-2913-47cc-951f-2b83001e1753, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
21698 [8810] Exited with: ExitStatus(unix_wait_status(9))
21699 [8810][0] reports Exit
21700 Mar 07 18:17:25.114 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/0, method: GET, req_id: 17503c9f-a839-4d72-ab78-0c17963e4815, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
21701 001/300 Write block 1666 len 36864 data: 4 5 4 5 5 5 4 4 5
21702 002/300 Read block 3050 len 32768
21703 003/300 Write block 1904 len 40960 data: 6 6 6 5 7 8 8 8 6 5
21704 004/300 Write block 268 len 40960 data: 5 5 6 7 6 6 7 7 9 6
21705 005/300 Write block 1208 len 36864 data: 6 7 7 8 7 6 7 4 5
21706 006/300 Write block 4953 len 36864 data: 3 4 4 4 5 7 6 6 6
21707 007/300 Read block 1108 len 24576
21708 008/300 Write block 4964 len 4096 data: 5
21709 009/300 Write block 1378 len 40960 data: 3 3 2 2 2 2 2 2 2 2
21710 010/300 Flush
21711 011/300 Write block 578 len 20480 data: 5 5 5 3 4
21712 012/300 Flush
21713 013/300 Read block 4756 len 24576
21714 014/300 Read block 1808 len 4096
21715 015/300 Write block 4035 len 40960 data: 4 6 6 7 5 3 2 5 5 6
21716 016/300 Read block 3993 len 12288
21717 017/300 Read block 3939 len 40960
21718 018/300 Flush
21719 019/300 Flush
21720 020/300 Write block 1525 len 24576 data: 6 4 4 4 5 5
21721 021/300 Write block 4194 len 8192 data: 4 4
21722 022/300 Write block 2327 len 40960 data: 2 3 4 4 5 5 6 5 6 6
21723 023/300 Read block 3517 len 36864
21724 024/300 Read block 4769 len 16384
21725 025/300 Read block 1084 len 36864
21726 026/300 Write block 4034 len 24576 data: 4 5 7 7 8 6
21727 027/300 Read block 3757 len 24576
21728 028/300 Write block 4718 len 12288 data: 2 4 5
21729 029/300 Read block 2728 len 24576
21730 030/300 Write block 1828 len 20480 data: 2 2 3 3 3
21731 031/300 Read block 767 len 36864
21732 032/300 Write block 3797 len 28672 data: 7 6 4 5 4 4 4
21733 033/300 Read block 4728 len 16384
21734 034/300 Flush
21735 035/300 Write block 4940 len 32768 data: 4 3 3 3 4 5 5 5
21736 036/300 Read block 1606 len 40960
21737 037/300 Write block 2233 len 28672 data: 3 3 3 3 4 4 5
21738 038/300 Read block 2958 len 4096
21739 039/300 Write block 3021 len 4096 data: 3
21740 040/300 Read block 1228 len 12288
21741 041/300 Read block 4982 len 32768
21742 042/300 Read block 4924 len 32768
21743 043/300 Read block 2267 len 40960
21744 044/300 Write block 2733 len 20480 data: 8 6 7 7 8
21745 045/300 Read block 400 len 4096
21746 046/300 Write block 3417 len 8192 data: 3 3
21747 047/300 Read block 4773 len 4096
21748 048/300 Flush
21749 049/300 Write block 1538 len 8192 data: 3 6
21750 050/300 Flush
21751 051/300 Read block 969 len 36864
21752 052/300 Read block 4681 len 24576
21753 053/300 Read block 1753 len 32768
21754 054/300 Read block 3355 len 20480
21755 055/300 Write block 993 len 12288 data: 4 4 4
21756 056/300 Read block 1342 len 12288
21757 057/300 Read block 2274 len 36864
21758 058/300 Write block 3792 len 8192 data: 2 5
21759 059/300 Read block 1675 len 8192
21760 060/300 Read block 2271 len 8192
21761 061/300 Read block 1642 len 36864
21762 062/300 Read block 4192 len 28672
21763 063/300 Read block 4499 len 16384
21764 064/300 Read block 4920 len 20480
21765 065/300 Read block 2851 len 32768
21766 066/300 Write block 2751 len 32768 data: 3 4 3 4 3 4 4 3
21767 067/300 Read block 3312 len 20480
21768 068/300 Write block 406 len 28672 data: 3 3 3 3 4 5 4
21769 069/300 Read block 4068 len 8192
21770 070/300 Read block 3340 len 36864
21771 071/300 Flush
21772 072/300 Read block 3559 len 24576
21773 073/300 Read block 3493 len 16384
21774 074/300 Read block 1676 len 40960
21775 075/300 Read block 56 len 24576
21776 076/300 Write block 262 len 12288 data: 5 6 6
21777 077/300 Read block 2562 len 20480
21778 078/300 Write block 977 len 20480 data: 2 2 3 3 3
21779 079/300 Read block 1017 len 24576
21780 080/300 Read block 4814 len 24576
21781 081/300 Flush
21782 082/300 Write block 2658 len 32768 data: 8 7 5 6 6 8 7 8
21783 083/300 Write block 3980 len 28672 data: 3 4 4 6 8 5 5
21784 084/300 Read block 3722 len 40960
21785 085/300 Read block 209 len 12288
21786 086/300 Read block 488 len 20480
21787 087/300 Read block 2378 len 4096
21788 088/300 Read block 4674 len 20480
21789 089/300 Write block 1453 len 28672 data: 5 5 4 3 4 4 5
21790 090/300 Write block 756 len 4096 data: 4
21791 091/300 Write block 235 len 12288 data: 4 4 3
21792 092/300 Write block 4942 len 16384 data: 4 4 5 6
21793 093/300 Read block 3941 len 36864
21794 094/300 Write block 4783 len 16384 data: 3 3 3 5
21795 095/300 Flush
21796 096/300 Read block 3742 len 24576
21797 097/300 Read block 367 len 32768
21798 098/300 Write block 585 len 8192 data: 4 4
21799 099/300 Write block 4292 len 12288 data: 5 4 3
21800 100/300 Read block 1344 len 4096
21801 101/300 Read block 4698 len 36864
21802 102/300 Write block 3510 len 8192 data: 4 3
21803 103/300 Read block 2655 len 24576
21804 104/300 Read block 2237 len 40960
21805 105/300 Write block 4609 len 8192 data: 5 6
21806 106/300 Write block 2643 len 20480 data: 4 4 4 4 3
21807 107/300 Write block 2665 len 40960 data: 9 7 7 7 5 4 4 5 4 4
21808 108/300 Read block 3918 len 32768
21809 109/300 Read block 413 len 4096
21810 110/300 Write block 3676 len 8192 data: 5 4
21811 111/300 Write block 2244 len 4096 data: 6
21812 112/300 Flush
21813 113/300 Read block 2592 len 40960
21814 114/300 Read block 2650 len 24576
21815 115/300 Read block 442 len 8192
21816 116/300 Write block 2289 len 24576 data: 3 3 4 3 2 2
21817 117/300 Read block 1980 len 12288
21818 118/300 Write block 400 len 20480 data: 6 5 5 5 5
21819 119/300 Write block 1562 len 36864 data: 2 2 5 6 7 8 9 9 7
21820 120/300 Read block 4946 len 16384
21821 121/300 Read block 3227 len 20480
21822 122/300 Write block 1436 len 4096 data: 6
21823 123/300 Read block 1572 len 4096
21824 124/300 Flush
21825 125/300 Read block 2265 len 16384
21826 126/300 Read block 3515 len 28672
21827 127/300 Read block 3439 len 8192
21828 128/300 Read block 2723 len 4096
21829 129/300 Write block 393 len 12288 data: 6 6 5
21830 130/300 Read block 1564 len 36864
21831 131/300 Read block 2537 len 8192
21832 132/300 Read block 2306 len 16384
21833 133/300 Write block 3283 len 36864 data: 6 5 6 6 6 6 5 3 4
21834 134/300 Read block 3760 len 12288
21835 135/300 Read block 4275 len 40960
21836 136/300 Read block 4757 len 36864
21837 137/300 Read block 511 len 8192
21838 138/300 Read block 143 len 32768
21839 139/300 Read block 825 len 40960
21840 140/300 Read block 2386 len 32768
21841 141/300 Read block 228 len 36864
21842 142/300 Read block 1995 len 28672
21843 143/300 Read block 4790 len 28672
21844 144/300 Write block 745 len 24576 data: 9 9 9 9 7 6
21845 145/300 Write block 4255 len 36864 data: 7 6 6 6 6 5 7 7 6
21846 146/300 Read block 3908 len 36864
21847 147/300 Read block 1450 len 16384
21848 148/300 Write block 2107 len 32768 data: 5 6 6 7 7 7 7 8
21849 149/300 Read block 4149 len 28672
21850 150/300 Read block 3011 len 20480
21851 151/300 Read block 689 len 20480
21852 152/300 Write block 3948 len 4096 data: 6
21853 153/300 Flush
21854 154/300 Read block 3599 len 40960
21855 155/300 Write block 1073 len 16384 data: 4 3 3 4
21856 156/300 Read block 3401 len 28672
21857 157/300 Read block 2926 len 20480
21858 158/300 Write block 3282 len 12288 data: 6 7 6
21859 159/300 Write block 3491 len 8192 data: 2 4
21860 160/300 Write block 1686 len 16384 data: 4 3 3 3
21861 161/300 Write block 655 len 28672 data: 3 4 3 2 2 2 3
21862 162/300 Write block 1210 len 36864 data: 8 9 8 7 8 5 6 3 3
21863 163/300 Read block 1811 len 12288
21864 164/300 Read block 1219 len 20480
21865 165/300 Write block 3177 len 12288 data: 3 3 3
21866 166/300 Write block 2010 len 40960 data: 6 5 4 5 5 5 6 5 7 6
21867 167/300 Write block 3551 len 8192 data: 3 4
21868 168/300 Read block 1989 len 16384
21869 169/300 Read block 3310 len 40960
21870 170/300 Read block 1936 len 4096
21871 171/300 Write block 1043 len 40960 data: 2 3 3 4 4 4 4 4 3 4
21872 172/300 Read block 3589 len 40960
21873 173/300 Write block 4086 len 20480 data: 4 4 3 3 3
21874 174/300 Write block 2460 len 20480 data: 5 5 6 7 7
21875 175/300 Write block 3318 len 40960 data: 4 4 5 5 6 7 6 5 6 6
21876 176/300 Write block 1053 len 24576 data: 5 4 4 3 2 3
21877 177/300 Read block 690 len 28672
21878 178/300 Write block 2886 len 36864 data: 6 6 5 6 6 5 5 5 6
21879 179/300 Write block 944 len 12288 data: 5 5 4
21880 180/300 Write block 2187 len 8192 data: 7 6
21881 181/300 Write block 3259 len 20480 data: 3 3 4 3 4
21882 182/300 Read block 2923 len 16384
21883 183/300 Write block 44 len 24576 data: 5 5 5 4 4 3
21884 184/300 Read block 2761 len 32768
21885 185/300 Read block 4965 len 20480
21886 186/300 Write block 1592 len 32768 data: 4 3 4 4 4 5 4 4
21887 187/300 Read block 4982 len 36864
21888 188/300 Write block 3959 len 28672 data: 6 6 5 5 5 5 2
21889 189/300 Flush
21890 190/300 Flush
21891 191/300 Read block 2220 len 32768
21892 192/300 Read block 2352 len 20480
21893 193/300 Read block 1666 len 4096
21894 194/300 Read block 129 len 36864
21895 195/300 Write block 796 len 12288 data: 4 6 6
21896 196/300 Write block 4853 len 8192 data: 4 4
21897 197/300 Flush
21898 198/300 Read block 1233 len 8192
21899 199/300 Read block 4129 len 24576
21900 200/300 Read block 4193 len 12288
21901 201/300 Read block 2033 len 28672
21902 202/300 Read block 1956 len 28672
21903 203/300 Read block 4392 len 32768
21904 204/300 Write block 1133 len 16384 data: 2 3 3 3
21905 205/300 Read block 292 len 24576
21906 206/300 Write block 567 len 4096 data: 4
21907 207/300 Write block 1125 len 32768 data: 4 4 4 4 4 3 2 2
21908 208/300 Read block 78 len 12288
21909 209/300 Read block 2852 len 32768
21910 210/300 Read block 339 len 40960
21911 211/300 Read block 2096 len 8192
21912 212/300 Read block 3514 len 8192
21913 213/300 Read block 657 len 28672
21914 214/300 Write block 3827 len 28672 data: 4 4 6 6 5 4 3
21915 215/300 Write block 1639 len 16384 data: 3 3 2 2
21916 216/300 Write block 4655 len 36864 data: 4 4 4 4 3 4 4 4 4
21917 217/300 Read block 3688 len 8192
21918 218/300 Read block 3696 len 12288
21919 219/300 Write block 1834 len 8192 data: 3 2
21920 220/300 Read block 1106 len 28672
21921 221/300 Write block 2022 len 28672 data: 4 4 5 5 6 5 5
21922 222/300 Write block 2446 len 12288 data: 3 3 2
21923 223/300 Write block 1693 len 28672 data: 6 6 6 5 5 5 7
21924 224/300 Write block 1345 len 20480 data: 3 3 2 2 2
21925 225/300 Read block 2218 len 16384
21926 226/300 Write block 4204 len 4096 data: 8
21927 227/300 Write block 2257 len 16384 data: 4 4 5 4
21928 228/300 Write block 1885 len 32768 data: 4 3 3 5 7 7 6 6
21929 229/300 Write block 2959 len 36864 data: 3 3 3 4 3 3 4 4 3
21930 230/300 Read block 2224 len 12288
21931 231/300 Flush
21932 232/300 Write block 1197 len 12288 data: 5 4 4
21933 233/300 Write block 3573 len 24576 data: 2 2 2 4 4 4
21934 234/300 Write block 563 len 4096 data: 4
21935 235/300 Flush
21936 236/300 Flush
21937 237/300 Read block 3500 len 12288
21938 238/300 Read block 967 len 28672
21939 239/300 Write block 717 len 12288 data: 4 3 2
21940 240/300 Read block 1126 len 8192
21941 241/300 Read block 617 len 32768
21942 242/300 Write block 3420 len 16384 data: 6 5 5 5
21943 243/300 Read block 2857 len 20480
21944 244/300 Read block 480 len 4096
21945 245/300 Read block 4651 len 24576
21946 246/300 Read block 220 len 8192
21947 247/300 Read block 2666 len 40960
21948 248/300 Read block 3970 len 24576
21949 249/300 Write block 2229 len 36864 data: 3 3 3 2 4 4 4 4 5
21950 250/300 Write block 1066 len 28672 data: 2 3 4 5 4 3 3
21951 251/300 Read block 1097 len 20480
21952 252/300 Write block 3359 len 20480 data: 3 3 3 4 5
21953 253/300 Flush
21954 254/300 Read block 971 len 28672
21955 255/300 Read block 2856 len 20480
21956 256/300 Write block 390 len 16384 data: 4 4 4 7
21957 257/300 Write block 455 len 24576 data: 3 3 3 2 4 3
21958 258/300 Read block 1438 len 8192
21959 259/300 Write block 3013 len 16384 data: 5 5 5 6
21960 260/300 Read block 593 len 8192
21961 261/300 Read block 884 len 32768
21962 262/300 Read block 266 len 24576
21963 263/300 Write block 4932 len 32768 data: 3 4 5 5 5 5 6 5
21964 264/300 Read block 1617 len 24576
21965 265/300 Write block 2547 len 20480 data: 5 4 4 3 3
21966 266/300 Flush
21967 267/300 Read block 2695 len 32768
21968 268/300 Read block 3513 len 36864
21969 269/300 Read block 1380 len 12288
21970 270/300 Read block 477 len 12288
21971 271/300 Read block 3020 len 12288
21972 272/300 Write block 811 len 8192 data: 4 4
21973 273/300 Read block 1446 len 36864
21974 274/300 Write block 1635 len 32768 data: 3 3 3 3 4 4 3 3
21975 275/300 Write block 1179 len 28672 data: 5 4 5 4 4 4 5
21976 276/300 Read block 310 len 16384
21977 277/300 Read block 336 len 12288
21978 278/300 Read block 2755 len 12288
21979 279/300 Read block 4388 len 4096
21980 280/300 Write block 289 len 4096 data: 6
21981 281/300 Read block 370 len 8192
21982 282/300 Write block 4183 len 8192 data: 4 4
21983 283/300 Read block 450 len 4096
21984 284/300 Write block 2838 len 8192 data: 4 5
21985 285/300 Write block 4072 len 4096 data: 6
21986 286/300 Write block 3966 len 24576 data: 2 3 4 4 4 4
21987 287/300 Read block 1210 len 32768
21988 288/300 Write block 2939 len 36864 data: 3 3 3 4 6 7 7 6 6
21989 289/300 Write block 1643 len 32768 data: 5 5 4 4 5 5 6 6
21990 290/300 Flush
21991 291/300 Read block 2901 len 16384
21992 292/300 Write block 3466 len 40960 data: 2 2 3 3 3 4 4 4 5 7
21993 293/300 Read block 3923 len 12288
21994 294/300 Write block 2838 len 24576 data: 5 6 6 6 4 4
21995 295/300 Read block 3292 len 40960
21996 296/300 Write block 3175 len 12288 data: 3 3 4
21997 297/300 Read block 4935 len 16384
21998 298/300 Read block 2744 len 40960
21999 299/300 Flush
22000 300/300 Read block 902 len 28672
22001 301/300 Read block 268 len 16384
22002 Main task has work to do, go find it
22003 got dsc Start(0)
22004 start 0
22005 Mar 07 18:17:25.368 INFO request completed, latency_us: 198, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5644b8bc-918c-4b9a-8dab-df64a4803b98, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
22006 Replay: started 0, returned:Ok(())
22007 ----------------------------------------------------------------
22008 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22009 States: Offline Active Active
22010 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22011 6434 Acked Write 9 Sent Done Done false
22012 6435 Acked Read 8 Sent Done Done false
22013 6436 Acked Write 10 Sent Done Done false
22014 6437 Acked Write 10 Sent Done Done false
22015 6438 Acked Write 9 Sent Done Done false
22016 6439 Acked Write 9 Sent Done Done false
22017 6440 Acked Read 6 Sent Done Done false
22018 6441 Acked Write 1 Sent Done Done false
22019 6442 Acked Write 10 Sent Done Done false
22020 6443 Acked Flush 0 Sent Done Done false
22021 6444 Acked Write 5 Sent Done Done false
22022 6445 Acked Flush 0 Sent Done Done false
22023 6446 Acked Read 6 Sent Done Done false
22024 6447 Acked Read 1 Sent Done Done false
22025 6448 Acked Write 10 Sent Done Done false
22026 6449 Acked Read 3 Sent Done Done false
22027 6450 Acked Read 10 Sent Done Done false
22028 6451 Acked Flush 0 Sent Done Done false
22029 6452 Acked Flush 0 Sent Done Done false
22030 6453 Acked Write 6 Sent Done Done false
22031 6454 Acked Write 2 Sent Done Done false
22032 6455 Acked Write 10 Sent Done Done false
22033 6456 Acked Read 9 Sent Done Done false
22034 6457 Acked Read 4 Sent Done Done false
22035 6458 Acked Read 9 Sent Done Done false
22036 6459 Acked Write 6 Sent Done Done false
22037 6460 Acked Read 6 Sent Done Done false
22038 6461 Acked Write 3 Sent Done Done false
22039 6462 Acked Read 6 Sent Done Done false
22040 6463 Acked Write 5 Sent Done Done false
22041 6464 Acked Read 9 Sent Done Done false
22042 6465 Acked Write 7 Sent Done Done false
22043 6466 Acked Read 4 Sent Done Done false
22044 6467 Acked Flush 0 Sent Done Done false
22045 6468 Acked Write 8 Sent Done Done false
22046 6469 Acked Read 10 Sent Done Done false
22047 6470 Acked Write 7 Sent Done Done false
22048 6471 Acked Read 1 Sent Done Done false
22049 6472 Acked Write 1 Sent Done Done false
22050 6473 Acked Read 3 Sent Done Done false
22051 6474 Acked Read 8 Sent Done Done false
22052 6475 Acked Read 8 Sent Done Done false
22053 6476 Acked Read 10 Sent Done Done false
22054 6477 Acked Write 5 Sent Done Done false
22055 6478 Acked Read 1 Sent Done Done false
22056 6479 Acked Write 2 Sent Done Done false
22057 6480 Acked Read 1 Sent Done Done false
22058 6481 Acked Flush 0 Sent Done Done false
22059 6482 Acked Write 2 Sent Done Done false
22060 6483 Acked Flush 0 Sent Done Done false
22061 6484 Acked Read 9 Sent Done Done false
22062 6485 Acked Read 6 Sent Done Done false
22063 6486 Acked Read 8 Sent Done Done false
22064 6487 Acked Read 5 Sent Done Done false
22065 6488 Acked Write 3 Sent Done Done false
22066 6489 Acked Read 3 Sent Done Done false
22067 6490 Acked Read 9 Sent Done Done false
22068 6491 Acked Write 2 Sent Done Done false
22069 6492 Acked Read 2 Sent Done Done false
22070 6493 Acked Read 2 Sent Done Done false
22071 6494 Acked Read 9 Sent Done Done false
22072 6495 Acked Read 7 Sent Done Done false
22073 6496 Acked Read 4 Sent Done Done false
22074 6497 Acked Read 5 Sent Done Done false
22075 6498 Acked Read 8 Sent Done Done false
22076 6499 Acked Write 8 Sent Done Done false
22077 6500 Acked Read 5 Sent Done Done false
22078 6501 Acked Write 7 Sent Done Done false
22079 6502 Acked Read 2 Sent Done Done false
22080 6503 Acked Read 9 Sent Done Done false
22081 6504 Acked Flush 0 Sent Done Done false
22082 6505 Acked Read 6 Sent Done Done false
22083 6506 Acked Read 4 Sent Done Done false
22084 6507 Acked Read 10 Sent Done Done false
22085 6508 Acked Read 6 Sent Done Done false
22086 6509 Acked Write 3 Sent Done Done false
22087 6510 Acked Read 5 Sent Done Done false
22088 6511 Acked Write 5 Sent Done Done false
22089 6512 Acked Read 6 Sent Done Done false
22090 6513 Acked Read 6 Sent Done Done false
22091 6514 Acked Flush 0 Sent Done Done false
22092 6515 Acked Write 8 Sent Done Done false
22093 6516 Acked Write 7 Sent Done Done false
22094 6517 Acked Read 10 Sent Done Done false
22095 6518 Acked Read 3 Sent Done Done false
22096 6519 Acked Read 5 Sent Done Done false
22097 6520 Acked Read 1 Sent Done Done false
22098 6521 Acked Read 5 Sent Done Done false
22099 6522 Acked Write 7 Sent Done Done false
22100 6523 Acked Write 1 Sent Done Done false
22101 6524 Acked Write 3 Sent Done Done false
22102 6525 Acked Write 4 Sent Done Done false
22103 6526 Acked Read 9 Sent Done Done false
22104 6527 Acked Write 4 Sent Done Done false
22105 6528 Acked Flush 0 Sent Done Done false
22106 6529 Acked Read 6 Sent Done Done false
22107 6530 Acked Read 8 Sent Done Done false
22108 6531 Acked Write 2 Sent Done Done false
22109 6532 Acked Write 3 Sent Done Done false
22110 6533 Acked Read 1 Sent Done Done false
22111 6534 Acked Read 9 Sent Done Done false
22112 6535 Acked Write 2 Sent Done Done false
22113 6536 Acked Read 6 Sent Done Done false
22114 6537 Acked Read 10 Sent Done Done false
22115 6538 Acked Write 2 Sent Done Done false
22116 6539 Acked Write 5 Sent Done Done false
22117 6540 Acked Write 10 Sent Done Done false
22118 6541 Acked Read 8 Sent Done Done false
22119 6542 Acked Read 1 Sent Done Done false
22120 6543 Acked Write 2 Sent Done Done false
22121 6544 Acked Write 1 Sent Done Done false
22122 6545 Acked Flush 0 Sent Done Done false
22123 6546 Acked Read 10 Sent Done Done false
22124 6547 Acked Read 6 Sent Done Done false
22125 6548 Acked Read 2 Sent Done Done false
22126 6549 Acked Write 6 Sent Done Done false
22127 6550 Acked Read 3 Sent Done Done false
22128 6551 Acked Write 5 Sent Done Done false
22129 6552 Acked Write 9 Sent Done Done false
22130 6553 Acked Read 4 Sent Done Done false
22131 6554 Acked Read 5 Sent Done Done false
22132 6555 Acked Write 1 Sent Done Done false
22133 6556 Acked Read 1 Sent Done Done false
22134 6557 Acked Flush 0 Sent Done Done false
22135 6558 Acked Read 4 Sent Done Done false
22136 6559 Acked Read 7 Sent Done Done false
22137 6560 Acked Read 2 Sent Done Done false
22138 6561 Acked Read 1 Sent Done Done false
22139 6562 Acked Write 3 Sent Done Done false
22140 6563 Acked Read 9 Sent Done Done false
22141 6564 Acked Read 2 Sent Done Done false
22142 6565 Acked Read 4 Sent Done Done false
22143 6566 Acked Write 9 Sent Done Done false
22144 6567 Acked Read 3 Sent Done Done false
22145 6568 Acked Read 10 Sent Done Done false
22146 6569 Acked Read 9 Sent Done Done false
22147 6570 Acked Read 2 Sent Done Done false
22148 6571 Acked Read 8 Sent Done Done false
22149 6572 Acked Read 10 Sent Done Done false
22150 6573 Acked Read 8 Sent Done Done false
22151 6574 Acked Read 9 Sent Done Done false
22152 6575 Acked Read 7 Sent Done Done false
22153 6576 Acked Read 7 Sent Done Done false
22154 6577 Acked Write 6 Sent Done Done false
22155 6578 Acked Write 9 Sent Done Done false
22156 6579 Acked Read 9 Sent Done Done false
22157 6580 Acked Read 4 Sent Done Done false
22158 6581 Acked Write 8 Sent Done Done false
22159 6582 Acked Read 7 Sent Done Done false
22160 6583 Acked Read 5 Sent Done Done false
22161 6584 Acked Read 5 Sent Done Done false
22162 6585 Acked Write 1 Sent Done Done false
22163 6586 Acked Flush 0 Sent Done Done false
22164 6587 Acked Read 10 Sent Done Done false
22165 6588 Acked Write 4 Sent Done Done false
22166 6589 Acked Read 7 Sent Done Done false
22167 6590 Acked Read 5 Sent Done Done false
22168 6591 Acked Write 3 Sent Done Done false
22169 6592 Acked Write 2 Sent Done Done false
22170 6593 Acked Write 4 Sent Done Done false
22171 6594 Acked Write 7 Sent Done Done false
22172 6595 Acked Write 9 Sent Done Done false
22173 6596 Acked Read 3 Sent Done Done false
22174 6597 Acked Read 5 Sent Done Done false
22175 6598 Acked Write 3 Sent Done Done false
22176 6599 Acked Write 10 Sent Done Done false
22177 6600 Acked Write 2 Sent Done Done false
22178 6601 Acked Read 4 Sent Done Done false
22179 6602 Acked Read 10 Sent Done Done false
22180 6603 Acked Read 1 Sent Done Done false
22181 6604 Acked Write 10 Sent Done Done false
22182 6605 Acked Read 10 Sent Done Done false
22183 6606 Acked Write 5 Sent Done Done false
22184 6607 Acked Write 5 Sent Done Done false
22185 6608 Acked Write 10 Sent Done Done false
22186 6609 Acked Write 6 Sent Done Done false
22187 6610 Acked Read 7 Sent Done Done false
22188 6611 Acked Write 9 Sent Done Done false
22189 6612 Acked Write 3 Sent Done Done false
22190 6613 Acked Write 2 Sent Done Done false
22191 6614 Acked Write 5 Sent Done Done false
22192 6615 Acked Read 4 Sent Done Done false
22193 6616 Acked Write 6 Sent Done Done false
22194 6617 Acked Read 8 Sent Done Done false
22195 6618 Acked Read 5 Sent Done Done false
22196 6619 Acked Write 8 Sent Done Done false
22197 6620 Acked Read 9 Sent Done Done false
22198 6621 Acked Write 7 Sent Done Done false
22199 6622 Acked Flush 0 Sent Done Done false
22200 6623 Acked Flush 0 Sent Done Done false
22201 6624 Acked Read 8 Sent Done Done false
22202 6625 Acked Read 5 Sent Done Done false
22203 6626 Acked Read 1 Sent Done Done false
22204 6627 Acked Read 9 Sent Done Done false
22205 6628 Acked Write 3 Sent Done Done false
22206 6629 Acked Write 2 Sent Done Done false
22207 6630 Acked Flush 0 Sent Done Done false
22208 6631 Acked Read 2 Sent Done Done false
22209 6632 Acked Read 6 Sent Done Done false
22210 6633 Acked Read 3 Sent Done Done false
22211 6634 Acked Read 7 Sent Done Done false
22212 6635 Acked Read 7 Sent Done Done false
22213 6636 Acked Read 8 Sent Done Done false
22214 6637 Acked Write 4 Sent Done Done false
22215 6638 Acked Read 6 Sent Done Done false
22216 6639 Acked Write 1 Sent Done Done false
22217 6640 Acked Write 8 Sent Done Done false
22218 6641 Acked Read 3 Sent Done Done false
22219 6642 Acked Read 8 Sent Done Done false
22220 6643 Acked Read 10 Sent Done Done false
22221 6644 Acked Read 2 Sent Done Done false
22222 6645 Acked Read 2 Sent Done Done false
22223 6646 Acked Read 7 Sent Done Done false
22224 6647 Acked Write 7 Sent Done Done false
22225 6648 Acked Write 4 Sent Done Done false
22226 6649 Acked Write 9 Sent Done Done false
22227 6650 Acked Read 2 Sent Done Done false
22228 6651 Acked Read 3 Sent Done Done false
22229 6652 Acked Write 2 Sent Done Done false
22230 6653 Acked Read 7 Sent Done Done false
22231 6654 Acked Write 7 Sent Done Done false
22232 6655 Acked Write 3 Sent Done Done false
22233 6656 Acked Write 7 Sent Done Done false
22234 6657 Acked Write 5 Sent Done Done false
22235 6658 Acked Read 4 Sent Done Done false
22236 6659 Acked Write 1 Sent Done Done false
22237 6660 Acked Write 4 Sent Done Done false
22238 6661 Acked Write 8 Sent Done Done false
22239 6662 Acked Write 9 Sent Done Done false
22240 6663 Acked Read 3 Sent Done Done false
22241 6664 Acked Flush 0 Sent Done Done false
22242 6665 Acked Write 3 Sent Done Done false
22243 6666 Acked Write 6 Sent Done Done false
22244 6667 Acked Write 1 Sent Done Done false
22245 6668 Acked Flush 0 Sent Done Done false
22246 6669 Acked Flush 0 Sent Done Done false
22247 6670 Acked Read 3 Sent Done Done false
22248 6671 Acked Read 7 Sent Done Done false
22249 6672 Acked Write 3 Sent Done Done false
22250 6673 Acked Read 2 Sent Done Done false
22251 6674 Acked Read 8 Sent Done Done false
22252 6675 Acked Write 4 Sent Done Done false
22253 6676 Acked Read 5 Sent Done Done false
22254 6677 Acked Read 1 Sent Done Done false
22255 6678 Acked Read 6 Sent Done Done false
22256 6679 Acked Read 2 Sent Done Done false
22257 6680 Acked Read 10 Sent Done Done false
22258 6681 Acked Read 6 Sent Done Done false
22259 6682 Acked Write 9 Sent Done Done false
22260 6683 Acked Write 7 Sent Done Done false
22261 6684 Acked Read 5 Sent Done Done false
22262 6685 Acked Write 5 Sent Done Done false
22263 6686 Acked Flush 0 Sent Done Done false
22264 6687 Acked Read 7 Sent Done Done false
22265 6688 Acked Read 5 Sent Done Done false
22266 6689 Acked Write 4 Sent Done Done false
22267 6690 Acked Write 6 Sent Done Done false
22268 6691 Acked Read 2 Sent Done Done false
22269 6692 Acked Write 4 Sent Done Done false
22270 6693 Acked Read 2 Sent Done Done false
22271 6694 Acked Read 8 Sent Done Done false
22272 6695 Acked Read 6 Sent Done Done false
22273 6696 Acked Write 8 Sent Done Done false
22274 6697 Acked Read 6 Sent Done Done false
22275 6698 Acked Write 5 Sent Done Done false
22276 6699 Acked Flush 0 Sent Done Done false
22277 6700 Acked Read 8 Sent Done Done false
22278 6701 Acked Read 9 Sent Done Done false
22279 6702 Acked Read 3 Sent Done Done false
22280 6703 Acked Read 3 Sent Done Done false
22281 6704 Acked Read 3 Sent Done Done false
22282 6705 Acked Write 2 Sent Done Done false
22283 6706 Acked Read 9 Sent Done Done false
22284 6707 Acked Write 8 Sent Done Done false
22285 6708 Acked Write 7 Sent Done Done false
22286 6709 Acked Read 4 Sent Done Done false
22287 6710 Acked Read 3 Sent Done Done false
22288 6711 Acked Read 3 Sent Done Done false
22289 6712 Acked Read 1 Sent Done Done false
22290 6713 Acked Write 1 Sent Done Done false
22291 6714 Acked Read 2 Sent Done Done false
22292 6715 Acked Write 2 Sent Done Done false
22293 6716 Acked Read 1 Sent Done Done false
22294 6717 Acked Write 2 Sent Done Done false
22295 6718 Acked Write 1 Sent Done Done false
22296 6719 Acked Write 6 Sent Done Done false
22297 6720 Acked Read 8 Sent Done Done false
22298 6721 Acked Write 9 Sent Done Done false
22299 6722 Acked Write 8 Sent Done Done false
22300 6723 Acked Flush 0 Sent Done Done false
22301 6724 Acked Read 4 Sent Done Done false
22302 6725 Acked Write 10 Sent Done Done false
22303 6726 Acked Read 3 Sent Done Done false
22304 6727 Acked Write 6 Sent Done Done false
22305 6728 Acked Read 10 Sent Done Done false
22306 6729 Acked Write 3 Sent Done Done false
22307 6730 Acked Read 4 Sent Done Done false
22308 6731 Acked Read 10 Sent Done Done false
22309 6732 Acked Flush 0 Sent Done Done false
22310 6733 Acked Read 7 Sent Done Done false
22311 6734 Acked Read 4 Sent Done Done false
22312 STATES DS:0 DS:1 DS:2 TOTAL
22313 Sent 301 0 0 301
22314 Done 0 301 301 602
22315 Skipped 0 0 0 0
22316 Error 0 0 0 0
22317 Last Flush: Some(JobId(6433)) Some(JobId(6732)) Some(JobId(6732))
22318 Downstairs last five completed: 6433 6432 6431 6430 6429
22319 Upstairs last five completed: 6734 6733 6732 6731 6730
22320 CLIENT: Up:0 ds:301 act:2
223212025-03-07T18:17:25.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
223222025-03-07T18:17:26.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
22323 [8810] got start action
22324 [8810] I am going to restart
22325 Starting downstairs at port 8810
22326 Make output file at "/tmp/dsc/downstairs-8810.txt"
22327 [8810][0] reports Starting
22328 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1227)
22329 [8810][0] reports Running
223302025-03-07T18:17:27.433ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
223312025-03-07T18:17:28.434ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
22332 ----------------------------------------------------------------
22333 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22334 States: Offline Active Active
22335 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22336 6434 Acked Write 9 Sent Done Done false
22337 6435 Acked Read 8 Sent Done Done false
22338 6436 Acked Write 10 Sent Done Done false
22339 6437 Acked Write 10 Sent Done Done false
22340 6438 Acked Write 9 Sent Done Done false
22341 6439 Acked Write 9 Sent Done Done false
22342 6440 Acked Read 6 Sent Done Done false
22343 6441 Acked Write 1 Sent Done Done false
22344 6442 Acked Write 10 Sent Done Done false
22345 6443 Acked Flush 0 Sent Done Done false
22346 6444 Acked Write 5 Sent Done Done false
22347 6445 Acked Flush 0 Sent Done Done false
22348 6446 Acked Read 6 Sent Done Done false
22349 6447 Acked Read 1 Sent Done Done false
22350 6448 Acked Write 10 Sent Done Done false
22351 6449 Acked Read 3 Sent Done Done false
22352 6450 Acked Read 10 Sent Done Done false
22353 6451 Acked Flush 0 Sent Done Done false
22354 6452 Acked Flush 0 Sent Done Done false
22355 6453 Acked Write 6 Sent Done Done false
22356 6454 Acked Write 2 Sent Done Done false
22357 6455 Acked Write 10 Sent Done Done false
22358 6456 Acked Read 9 Sent Done Done false
22359 6457 Acked Read 4 Sent Done Done false
22360 6458 Acked Read 9 Sent Done Done false
22361 6459 Acked Write 6 Sent Done Done false
22362 6460 Acked Read 6 Sent Done Done false
22363 6461 Acked Write 3 Sent Done Done false
22364 6462 Acked Read 6 Sent Done Done false
22365 6463 Acked Write 5 Sent Done Done false
22366 6464 Acked Read 9 Sent Done Done false
22367 6465 Acked Write 7 Sent Done Done false
22368 6466 Acked Read 4 Sent Done Done false
22369 6467 Acked Flush 0 Sent Done Done false
22370 6468 Acked Write 8 Sent Done Done false
22371 6469 Acked Read 10 Sent Done Done false
22372 6470 Acked Write 7 Sent Done Done false
22373 6471 Acked Read 1 Sent Done Done false
22374 6472 Acked Write 1 Sent Done Done false
22375 6473 Acked Read 3 Sent Done Done false
22376 6474 Acked Read 8 Sent Done Done false
22377 6475 Acked Read 8 Sent Done Done false
22378 6476 Acked Read 10 Sent Done Done false
22379 6477 Acked Write 5 Sent Done Done false
22380 6478 Acked Read 1 Sent Done Done false
22381 6479 Acked Write 2 Sent Done Done false
22382 6480 Acked Read 1 Sent Done Done false
22383 6481 Acked Flush 0 Sent Done Done false
22384 6482 Acked Write 2 Sent Done Done false
22385 6483 Acked Flush 0 Sent Done Done false
22386 6484 Acked Read 9 Sent Done Done false
22387 6485 Acked Read 6 Sent Done Done false
22388 6486 Acked Read 8 Sent Done Done false
22389 6487 Acked Read 5 Sent Done Done false
22390 6488 Acked Write 3 Sent Done Done false
22391 6489 Acked Read 3 Sent Done Done false
22392 6490 Acked Read 9 Sent Done Done false
22393 6491 Acked Write 2 Sent Done Done false
22394 6492 Acked Read 2 Sent Done Done false
22395 6493 Acked Read 2 Sent Done Done false
22396 6494 Acked Read 9 Sent Done Done false
22397 6495 Acked Read 7 Sent Done Done false
22398 6496 Acked Read 4 Sent Done Done false
22399 6497 Acked Read 5 Sent Done Done false
22400 6498 Acked Read 8 Sent Done Done false
22401 6499 Acked Write 8 Sent Done Done false
22402 6500 Acked Read 5 Sent Done Done false
22403 6501 Acked Write 7 Sent Done Done false
22404 6502 Acked Read 2 Sent Done Done false
22405 6503 Acked Read 9 Sent Done Done false
22406 6504 Acked Flush 0 Sent Done Done false
22407 6505 Acked Read 6 Sent Done Done false
22408 6506 Acked Read 4 Sent Done Done false
22409 6507 Acked Read 10 Sent Done Done false
22410 6508 Acked Read 6 Sent Done Done false
22411 6509 Acked Write 3 Sent Done Done false
22412 6510 Acked Read 5 Sent Done Done false
22413 6511 Acked Write 5 Sent Done Done false
22414 6512 Acked Read 6 Sent Done Done false
22415 6513 Acked Read 6 Sent Done Done false
22416 6514 Acked Flush 0 Sent Done Done false
22417 6515 Acked Write 8 Sent Done Done false
22418 6516 Acked Write 7 Sent Done Done false
22419 6517 Acked Read 10 Sent Done Done false
22420 6518 Acked Read 3 Sent Done Done false
22421 6519 Acked Read 5 Sent Done Done false
22422 6520 Acked Read 1 Sent Done Done false
22423 6521 Acked Read 5 Sent Done Done false
22424 6522 Acked Write 7 Sent Done Done false
22425 6523 Acked Write 1 Sent Done Done false
22426 6524 Acked Write 3 Sent Done Done false
22427 6525 Acked Write 4 Sent Done Done false
22428 6526 Acked Read 9 Sent Done Done false
22429 6527 Acked Write 4 Sent Done Done false
22430 6528 Acked Flush 0 Sent Done Done false
22431 6529 Acked Read 6 Sent Done Done false
22432 6530 Acked Read 8 Sent Done Done false
22433 6531 Acked Write 2 Sent Done Done false
22434 6532 Acked Write 3 Sent Done Done false
22435 6533 Acked Read 1 Sent Done Done false
22436 6534 Acked Read 9 Sent Done Done false
22437 6535 Acked Write 2 Sent Done Done false
22438 6536 Acked Read 6 Sent Done Done false
22439 6537 Acked Read 10 Sent Done Done false
22440 6538 Acked Write 2 Sent Done Done false
22441 6539 Acked Write 5 Sent Done Done false
22442 6540 Acked Write 10 Sent Done Done false
22443 6541 Acked Read 8 Sent Done Done false
22444 6542 Acked Read 1 Sent Done Done false
22445 6543 Acked Write 2 Sent Done Done false
22446 6544 Acked Write 1 Sent Done Done false
22447 6545 Acked Flush 0 Sent Done Done false
22448 6546 Acked Read 10 Sent Done Done false
22449 6547 Acked Read 6 Sent Done Done false
22450 6548 Acked Read 2 Sent Done Done false
22451 6549 Acked Write 6 Sent Done Done false
22452 6550 Acked Read 3 Sent Done Done false
22453 6551 Acked Write 5 Sent Done Done false
22454 6552 Acked Write 9 Sent Done Done false
22455 6553 Acked Read 4 Sent Done Done false
22456 6554 Acked Read 5 Sent Done Done false
22457 6555 Acked Write 1 Sent Done Done false
22458 6556 Acked Read 1 Sent Done Done false
22459 6557 Acked Flush 0 Sent Done Done false
22460 6558 Acked Read 4 Sent Done Done false
22461 6559 Acked Read 7 Sent Done Done false
22462 6560 Acked Read 2 Sent Done Done false
22463 6561 Acked Read 1 Sent Done Done false
22464 6562 Acked Write 3 Sent Done Done false
22465 6563 Acked Read 9 Sent Done Done false
22466 6564 Acked Read 2 Sent Done Done false
22467 6565 Acked Read 4 Sent Done Done false
22468 6566 Acked Write 9 Sent Done Done false
22469 6567 Acked Read 3 Sent Done Done false
22470 6568 Acked Read 10 Sent Done Done false
22471 6569 Acked Read 9 Sent Done Done false
22472 6570 Acked Read 2 Sent Done Done false
22473 6571 Acked Read 8 Sent Done Done false
22474 6572 Acked Read 10 Sent Done Done false
22475 6573 Acked Read 8 Sent Done Done false
22476 6574 Acked Read 9 Sent Done Done false
22477 6575 Acked Read 7 Sent Done Done false
22478 6576 Acked Read 7 Sent Done Done false
22479 6577 Acked Write 6 Sent Done Done false
22480 6578 Acked Write 9 Sent Done Done false
22481 6579 Acked Read 9 Sent Done Done false
22482 6580 Acked Read 4 Sent Done Done false
22483 6581 Acked Write 8 Sent Done Done false
22484 6582 Acked Read 7 Sent Done Done false
22485 6583 Acked Read 5 Sent Done Done false
22486 6584 Acked Read 5 Sent Done Done false
22487 6585 Acked Write 1 Sent Done Done false
22488 6586 Acked Flush 0 Sent Done Done false
22489 6587 Acked Read 10 Sent Done Done false
22490 6588 Acked Write 4 Sent Done Done false
22491 6589 Acked Read 7 Sent Done Done false
22492 6590 Acked Read 5 Sent Done Done false
22493 6591 Acked Write 3 Sent Done Done false
22494 6592 Acked Write 2 Sent Done Done false
22495 6593 Acked Write 4 Sent Done Done false
22496 6594 Acked Write 7 Sent Done Done false
22497 6595 Acked Write 9 Sent Done Done false
22498 6596 Acked Read 3 Sent Done Done false
22499 6597 Acked Read 5 Sent Done Done false
22500 6598 Acked Write 3 Sent Done Done false
22501 6599 Acked Write 10 Sent Done Done false
22502 6600 Acked Write 2 Sent Done Done false
22503 6601 Acked Read 4 Sent Done Done false
22504 6602 Acked Read 10 Sent Done Done false
22505 6603 Acked Read 1 Sent Done Done false
22506 6604 Acked Write 10 Sent Done Done false
22507 6605 Acked Read 10 Sent Done Done false
22508 6606 Acked Write 5 Sent Done Done false
22509 6607 Acked Write 5 Sent Done Done false
22510 6608 Acked Write 10 Sent Done Done false
22511 6609 Acked Write 6 Sent Done Done false
22512 6610 Acked Read 7 Sent Done Done false
22513 6611 Acked Write 9 Sent Done Done false
22514 6612 Acked Write 3 Sent Done Done false
22515 6613 Acked Write 2 Sent Done Done false
22516 6614 Acked Write 5 Sent Done Done false
22517 6615 Acked Read 4 Sent Done Done false
22518 6616 Acked Write 6 Sent Done Done false
22519 6617 Acked Read 8 Sent Done Done false
22520 6618 Acked Read 5 Sent Done Done false
22521 6619 Acked Write 8 Sent Done Done false
22522 6620 Acked Read 9 Sent Done Done false
22523 6621 Acked Write 7 Sent Done Done false
22524 6622 Acked Flush 0 Sent Done Done false
22525 6623 Acked Flush 0 Sent Done Done false
22526 6624 Acked Read 8 Sent Done Done false
22527 6625 Acked Read 5 Sent Done Done false
22528 6626 Acked Read 1 Sent Done Done false
22529 6627 Acked Read 9 Sent Done Done false
22530 6628 Acked Write 3 Sent Done Done false
22531 6629 Acked Write 2 Sent Done Done false
22532 6630 Acked Flush 0 Sent Done Done false
22533 6631 Acked Read 2 Sent Done Done false
22534 6632 Acked Read 6 Sent Done Done false
22535 6633 Acked Read 3 Sent Done Done false
22536 6634 Acked Read 7 Sent Done Done false
22537 6635 Acked Read 7 Sent Done Done false
22538 6636 Acked Read 8 Sent Done Done false
22539 6637 Acked Write 4 Sent Done Done false
22540 6638 Acked Read 6 Sent Done Done false
22541 6639 Acked Write 1 Sent Done Done false
22542 6640 Acked Write 8 Sent Done Done false
22543 6641 Acked Read 3 Sent Done Done false
22544 6642 Acked Read 8 Sent Done Done false
22545 6643 Acked Read 10 Sent Done Done false
22546 6644 Acked Read 2 Sent Done Done false
22547 6645 Acked Read 2 Sent Done Done false
22548 6646 Acked Read 7 Sent Done Done false
22549 6647 Acked Write 7 Sent Done Done false
22550 6648 Acked Write 4 Sent Done Done false
22551 6649 Acked Write 9 Sent Done Done false
22552 6650 Acked Read 2 Sent Done Done false
22553 6651 Acked Read 3 Sent Done Done false
22554 6652 Acked Write 2 Sent Done Done false
22555 6653 Acked Read 7 Sent Done Done false
22556 6654 Acked Write 7 Sent Done Done false
22557 6655 Acked Write 3 Sent Done Done false
22558 6656 Acked Write 7 Sent Done Done false
22559 6657 Acked Write 5 Sent Done Done false
22560 6658 Acked Read 4 Sent Done Done false
22561 6659 Acked Write 1 Sent Done Done false
22562 6660 Acked Write 4 Sent Done Done false
22563 6661 Acked Write 8 Sent Done Done false
22564 6662 Acked Write 9 Sent Done Done false
22565 6663 Acked Read 3 Sent Done Done false
22566 6664 Acked Flush 0 Sent Done Done false
22567 6665 Acked Write 3 Sent Done Done false
22568 6666 Acked Write 6 Sent Done Done false
22569 6667 Acked Write 1 Sent Done Done false
22570 6668 Acked Flush 0 Sent Done Done false
22571 6669 Acked Flush 0 Sent Done Done false
22572 6670 Acked Read 3 Sent Done Done false
22573 6671 Acked Read 7 Sent Done Done false
22574 6672 Acked Write 3 Sent Done Done false
22575 6673 Acked Read 2 Sent Done Done false
22576 6674 Acked Read 8 Sent Done Done false
22577 6675 Acked Write 4 Sent Done Done false
22578 6676 Acked Read 5 Sent Done Done false
22579 6677 Acked Read 1 Sent Done Done false
22580 6678 Acked Read 6 Sent Done Done false
22581 6679 Acked Read 2 Sent Done Done false
22582 6680 Acked Read 10 Sent Done Done false
22583 6681 Acked Read 6 Sent Done Done false
22584 6682 Acked Write 9 Sent Done Done false
22585 6683 Acked Write 7 Sent Done Done false
22586 6684 Acked Read 5 Sent Done Done false
22587 6685 Acked Write 5 Sent Done Done false
22588 6686 Acked Flush 0 Sent Done Done false
22589 6687 Acked Read 7 Sent Done Done false
22590 6688 Acked Read 5 Sent Done Done false
22591 6689 Acked Write 4 Sent Done Done false
22592 6690 Acked Write 6 Sent Done Done false
22593 6691 Acked Read 2 Sent Done Done false
22594 6692 Acked Write 4 Sent Done Done false
22595 6693 Acked Read 2 Sent Done Done false
22596 6694 Acked Read 8 Sent Done Done false
22597 6695 Acked Read 6 Sent Done Done false
22598 6696 Acked Write 8 Sent Done Done false
22599 6697 Acked Read 6 Sent Done Done false
22600 6698 Acked Write 5 Sent Done Done false
22601 6699 Acked Flush 0 Sent Done Done false
22602 6700 Acked Read 8 Sent Done Done false
22603 6701 Acked Read 9 Sent Done Done false
22604 6702 Acked Read 3 Sent Done Done false
22605 6703 Acked Read 3 Sent Done Done false
22606 6704 Acked Read 3 Sent Done Done false
22607 6705 Acked Write 2 Sent Done Done false
22608 6706 Acked Read 9 Sent Done Done false
22609 6707 Acked Write 8 Sent Done Done false
22610 6708 Acked Write 7 Sent Done Done false
22611 6709 Acked Read 4 Sent Done Done false
22612 6710 Acked Read 3 Sent Done Done false
22613 6711 Acked Read 3 Sent Done Done false
22614 6712 Acked Read 1 Sent Done Done false
22615 6713 Acked Write 1 Sent Done Done false
22616 6714 Acked Read 2 Sent Done Done false
22617 6715 Acked Write 2 Sent Done Done false
22618 6716 Acked Read 1 Sent Done Done false
22619 6717 Acked Write 2 Sent Done Done false
22620 6718 Acked Write 1 Sent Done Done false
22621 6719 Acked Write 6 Sent Done Done false
22622 6720 Acked Read 8 Sent Done Done false
22623 6721 Acked Write 9 Sent Done Done false
22624 6722 Acked Write 8 Sent Done Done false
22625 6723 Acked Flush 0 Sent Done Done false
22626 6724 Acked Read 4 Sent Done Done false
22627 6725 Acked Write 10 Sent Done Done false
22628 6726 Acked Read 3 Sent Done Done false
22629 6727 Acked Write 6 Sent Done Done false
22630 6728 Acked Read 10 Sent Done Done false
22631 6729 Acked Write 3 Sent Done Done false
22632 6730 Acked Read 4 Sent Done Done false
22633 6731 Acked Read 10 Sent Done Done false
22634 6732 Acked Flush 0 Sent Done Done false
22635 6733 Acked Read 7 Sent Done Done false
22636 6734 Acked Read 4 Sent Done Done false
22637 STATES DS:0 DS:1 DS:2 TOTAL
22638 Sent 301 0 0 301
22639 Done 0 301 301 602
22640 Skipped 0 0 0 0
22641 Error 0 0 0 0
22642 Last Flush: Some(JobId(6433)) Some(JobId(6732)) Some(JobId(6732))
22643 Downstairs last five completed: 6433 6432 6431 6430 6429
22644 Upstairs last five completed: 6734 6733 6732 6731 6730
22645 CLIENT: Up:0 ds:301 act:2
226462025-03-07T18:17:29.872ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226472025-03-07T18:17:30.438ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226482025-03-07T18:17:31.113ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226492025-03-07T18:17:31.114ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47412) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226502025-03-07T18:17:31.115ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226512025-03-07T18:17:31.115ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226522025-03-07T18:17:31.115ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226532025-03-07T18:17:31.115ZINFOcrucible: send last flush ID to this DS: Some(JobId(6433)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226542025-03-07T18:17:31.115ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226552025-03-07T18:17:31.115ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6433)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
226562025-03-07T18:17:31.116ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6433)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
22657 ----------------------------------------------------------------
22658 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22659 Downstairs last five completed: 6735 6734 6733 6732 6731
22660 Upstairs last five completed: 6735 6734 6733 6732 6731
22661 CLIENT: Up:0 ds:0 act:3
22662 Replay: All jobs finished
22663 Main task has work to do, go find it
22664 got dsc Stop(0)
22665 stop 0
22666 Mar 07 18:17:33.373 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/0, method: POST, req_id: [8810] Got stop action so:false kr:false
22667 9edf3980-0a45-4a19-9110-ed09b08b461d, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
226682025-03-07T18:17:33.374ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
22669 Mar 07 18:17:33.374 INFO {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:33.374627506Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
22670 request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method:{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:33.374688089Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
22671 GET, req_id: 559addc4-add0-441d-9489-15904712d54e, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:17:33.374716646Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"0","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
22672 127.0.0.1:64322, local_addr: 127.0.0.1:9998
22673 [8810] Exited with: ExitStatus(unix_wait_status(9))
22674 [8810][0] reports Exit
22675 Mar 07 18:17:37.377 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/0, method: GET, req_id: d2b16d20-fa19-4a49-8cf7-e06269ab7ed5, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
22676 001/300 Read block 3964 len 16384
22677 002/300 Read block 3734 len 28672
22678 003/300 Write block 2867 len 4096 data: 6
22679 004/300 Flush
22680 005/300 Write block 2246 len 20480 data: 2 2 2 3 3
22681 006/300 Write block 2859 len 4096 data: 2
22682 007/300 Write block 1626 len 12288 data: 7 6 6
22683 008/300 Read block 2419 len 24576
22684 009/300 Flush
22685 010/300 Write block 3598 len 8192 data: 4 4
22686 011/300 Read block 3183 len 36864
22687 012/300 Read block 4261 len 36864
22688 013/300 Read block 3011 len 32768
22689 014/300 Read block 2770 len 32768
22690 015/300 Read block 3666 len 12288
22691 016/300 Read block 4583 len 28672
22692 017/300 Write block 90 len 36864 data: 3 3 3 4 4 4 4 3 4
22693 018/300 Read block 696 len 24576
22694 019/300 Write block 676 len 20480 data: 5 5 7 6 4
22695 020/300 Write block 4158 len 40960 data: 4 4 4 4 4 4 4 4 5 5
22696 021/300 Write block 2666 len 28672 data: 8 8 8 6 5 5 6
22697 022/300 Read block 104 len 32768
22698 023/300 Read block 4934 len 20480
22699 024/300 Read block 1975 len 16384
22700 025/300 Read block 2056 len 16384
22701 026/300 Write block 164 len 36864 data: 6 7 9 8 8 7 9 5 5
22702 027/300 Read block 2680 len 4096
22703 028/300 Flush
22704 029/300 Read block 731 len 36864
22705 030/300 Read block 1652 len 36864
22706 031/300 Write block 819 len 36864 data: 4 6 5 5 5 7 9 6 7
22707 032/300 Write block 3299 len 32768 data: 3 3 3 4 5 5 6 6
22708 033/300 Write block 4770 len 28672 data: 5 5 5 4 5 4 4
22709 034/300 Write block 4218 len 16384 data: 4 4 4 5
22710 035/300 Read block 2147 len 32768
22711 036/300 Write block 3875 len 16384 data: 4 4 4 5
22712 037/300 Read block 2555 len 8192
22713 038/300 Write block 4147 len 40960 data: 2 2 2 2 2 2 2 2 3 5
22714 039/300 Write block 2390 len 12288 data: 6 7 6
22715 040/300 Write block 11 len 36864 data: 5 5 5 5 6 5 4 6 6
22716 041/300 Read block 4370 len 28672
22717 042/300 Read block 192 len 4096
22718 043/300 Read block 1089 len 16384
22719 044/300 Write block 2129 len 32768 data: 4 4 4 3 4 3 3 3
22720 045/300 Read block 754 len 16384
22721 046/300 Read block 3928 len 12288
22722 047/300 Write block 501 len 16384 data: 3 3 4 4
22723 048/300 Write block 865 len 36864 data: 2 3 4 4 4 5 5 2 2
22724 049/300 Read block 778 len 20480
22725 050/300 Write block 1714 len 24576 data: 6 7 6 6 7 8
22726 051/300 Read block 2867 len 36864
22727 052/300 Write block 4546 len 28672 data: 9 9 9 9 7 5 5
22728 053/300 Flush
22729 054/300 Write block 3942 len 24576 data: 5 5 5 5 5 4
22730 055/300 Write block 1467 len 36864 data: 8 7 7 6 5 4 4 3 3
22731 056/300 Write block 3687 len 40960 data: 4 3 3 2 2 3 3 4 5 5
22732 057/300 Read block 3708 len 16384
22733 058/300 Read block 254 len 4096
22734 059/300 Flush
22735 060/300 Read block 4587 len 32768
22736 061/300 Read block 4954 len 12288
22737 062/300 Write block 1875 len 28672 data: 5 4 5 4 4 3 3
22738 063/300 Write block 1955 len 12288 data: 2 2 3
22739 064/300 Read block 4078 len 36864
22740 065/300 Read block 185 len 16384
22741 066/300 Read block 3202 len 40960
22742 067/300 Flush
22743 068/300 Read block 4929 len 24576
22744 069/300 Write block 2171 len 4096 data: 2
22745 070/300 Read block 2949 len 20480
22746 071/300 Read block 746 len 40960
22747 072/300 Write block 2396 len 28672 data: 4 4 2 2 2 2 2
22748 073/300 Flush
22749 074/300 Write block 1634 len 36864 data: 3 4 4 4 4 5 5 4 4
22750 075/300 Read block 2679 len 36864
22751 076/300 Read block 4456 len 24576
22752 077/300 Write block 4250 len 16384 data: 6 6 7 7
22753 078/300 Write block 4157 len 24576 data: 5 5 5 5 5 5
22754 079/300 Read block 912 len 32768
22755 080/300 Read block 1505 len 36864
22756 081/300 Write block 2922 len 32768 data: 8 7 8 7 7 6 5 4
22757 082/300 Write block 2738 len 20480 data: 8 9 8 7 6
22758 083/300 Read block 585 len 8192
22759 084/300 Write block 4972 len 20480 data: 2 2 2 2 2
22760 085/300 Read block 2480 len 16384
22761 086/300 Write block 1351 len 4096 data: 3
22762 087/300 Read block 3553 len 8192
22763 088/300 Write block 4823 len 24576 data: 3 5 6 6 5 4
22764 089/300 Read block 4809 len 16384
22765 090/300 Read block 584 len 28672
22766 091/300 Write block 2397 len 4096 data: 5
22767 092/300 Read block 3437 len 28672
22768 093/300 Read block 2606 len 28672
22769 094/300 Write block 1736 len 24576 data: 6 5 4 4 4 4
22770 095/300 Write block 1268 len 8192 data: 7 6
22771 096/300 Flush
22772 097/300 Write block 1106 len 28672 data: 5 5 4 3 3 4 3
22773 098/300 Read block 4802 len 12288
22774 099/300 Read block 3909 len 28672
22775 100/300 Read block 2374 len 40960
22776 101/300 Read block 4781 len 4096
22777 102/300 Read block 4042 len 4096
22778 103/300 Read block 177 len 12288
22779 104/300 Flush
22780 105/300 Read block 619 len 32768
22781 106/300 Read block 1453 len 20480
22782 107/300 Write block 4984 len 24576 data: 4 4 3 2 2 2
22783 108/300 Write block 1215 len 4096 data: 6
22784 109/300 Read block 4839 len 32768
22785 110/300 Flush
22786 111/300 Read block 45 len 40960
22787 112/300 Write block 2968 len 36864 data: 4 4 4 4 4 4 4 6 3
22788 113/300 Write block 2789 len 24576 data: 3 3 3 3 3 3
22789 114/300 Read block 3154 len 40960
22790 115/300 Write block 4776 len 12288 data: 5 5 5
22791 116/300 Read block 2296 len 20480
22792 117/300 Write block 2455 len 28672 data: 5 6 5 5 5 6 6
22793 118/300 Write block 4786 len 20480 data: 6 4 4 4 4
22794 119/300 Write block 1127 len 4096 data: 5
22795 120/300 Read block 4891 len 36864
22796 121/300 Write block 3000 len 28672 data: 5 4 5 6 7 6 5
22797 122/300 Flush
22798 123/300 Read block 325 len 20480
22799 124/300 Read block 1642 len 32768
22800 125/300 Read block 3137 len 24576
22801 126/300 Write block 3140 len 16384 data: 6 4 4 5
22802 127/300 Read block 4516 len 24576
22803 128/300 Write block 3222 len 4096 data: 4
22804 129/300 Write block 3793 len 8192 data: 6 5
22805 130/300 Flush
22806 131/300 Read block 3658 len 36864
22807 132/300 Read block 1818 len 16384
22808 133/300 Write block 4821 len 12288 data: 3 3 4
22809 134/300 Read block 1068 len 28672
22810 135/300 Write block 4724 len 12288 data: 4 4 3
22811 136/300 Read block 4041 len 8192
22812 137/300 Read block 1832 len 40960
22813 138/300 Read block 3590 len 16384
22814 139/300 Read block 1171 len 12288
22815 140/300 Read block 603 len 8192
22816 141/300 Write block 2799 len 20480 data: 4 5 5 8 6
22817 142/300 Flush
22818 143/300 Read block 2533 len 36864
22819 144/300 Read block 2459 len 24576
22820 145/300 Flush
22821 146/300 Write block 4376 len 8192 data: 6 6
22822 147/300 Read block 2306 len 8192
22823 148/300 Read block 923 len 32768
22824 149/300 Write block 2383 len 40960 data: 4 4 4 4 4 5 6 7 8 7
22825 150/300 Write block 2333 len 36864 data: 7 6 7 7 5 5 6 5 6
22826 151/300 Write block 4053 len 40960 data: 5 6 7 8 7 5 4 4 4 4
22827 152/300 Read block 3516 len 24576
22828 153/300 Write block 1874 len 12288 data: 5 6 5
22829 154/300 Read block 3915 len 16384
22830 155/300 Read block 166 len 4096
22831 156/300 Read block 169 len 16384
22832 157/300 Read block 1756 len 28672
22833 158/300 Read block 1877 len 16384
22834 159/300 Write block 672 len 32768 data: 7 5 3 3 6 6 8 7
22835 160/300 Write block 1204 len 4096 data: 3
22836 161/300 Flush
22837 162/300 Read block 39 len 16384
22838 163/300 Write block 4385 len 8192 data: 5 5
22839 164/300 Read block 2412 len 20480
22840 165/300 Read block 3334 len 32768
22841 166/300 Write block 4109 len 28672 data: 5 6 5 6 7 5 4
22842 167/300 Read block 2156 len 32768
22843 168/300 Write block 1965 len 12288 data: 5 5 4
22844 169/300 Write block 1681 len 8192 data: 4 3
22845 170/300 Read block 444 len 20480
22846 171/300 Read block 364 len 40960
22847 172/300 Write block 4507 len 12288 data: 6 6 5
22848 173/300 Read block 3581 len 16384
22849 174/300 Read block 4980 len 12288
22850 175/300 Write block 2887 len 40960 data: 7 6 7 7 6 6 6 7 5 5
22851 176/300 Write block 3363 len 4096 data: 6
22852 177/300 Read block 2073 len 28672
22853 178/300 Write block 1741 len 36864 data: 5 6 5 5 7 5 5 5 6
22854 179/300 Write block 43 len 16384 data: 3 6 6 6
22855 180/300 Write block 2542 len 36864 data: 5 4 5 4 4 6 5 5 4
22856 181/300 Read block 185 len 20480
22857 182/300 Write block 3099 len 24576 data: 5 6 6 6 4 4
22858 183/300 Read block 2287 len 16384
22859 184/300 Read block 202 len 4096
22860 185/300 Read block 114 len 8192
22861 186/300 Write block 330 len 40960 data: 4 4 4 4 2 3 2 3 4 5
22862 187/300 Read block 3812 len 40960
22863 188/300 Read block 2323 len 36864
22864 189/300 Read block 4510 len 16384
22865 190/300 Flush
22866 191/300 Write block 4669 len 4096 data: 5
22867 192/300 Read block 2810 len 32768
22868 193/300 Read block 267 len 4096
22869 194/300 Read block 3781 len 24576
22870 195/300 Write block 2528 len 8192 data: 3 3
22871 196/300 Read block 38 len 28672
22872 197/300 Read block 4355 len 36864
22873 198/300 Write block 3607 len 36864 data: 3 3 3 2 2 2 3 3 4
22874 199/300 Read block 331 len 4096
22875 200/300 Write block 4680 len 12288 data: 5 5 6
22876 201/300 Flush
22877 202/300 Write block 2544 len 40960 data: 6 5 5 7 6 6 5 4 4 5
22878 203/300 Read block 1972 len 12288
22879 204/300 Write block 3386 len 28672 data: 6 6 4 4 3 2 3
22880 205/300 Read block 3917 len 36864
22881 206/300 Read block 4943 len 36864
22882 207/300 Read block 4303 len 12288
22883 208/300 Write block 1927 len 24576 data: 6 6 5 5 4 4
22884 209/300 Flush
22885 210/300 Write block 1473 len 24576 data: 5 4 4 2 2 2
22886 211/300 Read block 3902 len 8192
22887 212/300 Read block 3340 len 8192
22888 213/300 Read block 1008 len 4096
22889 214/300 Write block 1348 len 40960 data: 3 3 2 4 2 3 3 3 2 2
22890 215/300 Flush
22891 216/300 Read block 1556 len 36864
22892 217/300 Read block 1166 len 8192
22893 218/300 Read block 3900 len 24576
22894 219/300 Read block 3596 len 24576
22895 220/300 Write block 9 len 32768 data: 5 5 6 6 6 6 7 6
22896 221/300 Read block 2138 len 28672
22897 222/300 Read block 1088 len 36864
22898 223/300 Read block 1070 len 8192
22899 224/300 Read block 2854 len 28672
22900 225/300 Write block 554 len 4096 data: 3
22901 226/300 Flush
22902 227/300 Read block 4619 len 8192
22903 228/300 Read block 4077 len 40960
22904 229/300 Write block 2359 len 36864 data: 2 2 2 2 2 3 4 3 3
22905 230/300 Read block 4550 len 16384
22906 231/300 Read block 1418 len 12288
22907 232/300 Write block 4607 len 16384 data: 5 5 6 7
22908 233/300 Read block 2094 len 36864
22909 234/300 Read block 1760 len 20480
22910 235/300 Read block 3276 len 32768
22911 236/300 Write block 242 len 4096 data: 3
22912 237/300 Read block 3424 len 12288
22913 238/300 Read block 1124 len 24576
22914 239/300 Write block 4533 len 32768 data: 4 4 5 5 6 4 4 5
22915 240/300 Read block 1446 len 36864
22916 241/300 Read block 1970 len 36864
22917 242/300 Flush
22918 243/300 Write block 2166 len 28672 data: 3 3 3 2 2 3 2
22919 244/300 Flush
22920 245/300 Write block 3804 len 20480 data: 4 3 3 4 5
22921 246/300 Write block 2148 len 8192 data: 4 3
22922 247/300 Write block 3191 len 12288 data: 6 7 5
22923 248/300 Read block 4708 len 28672
22924 249/300 Write block 4450 len 4096 data: 5
22925 250/300 Write block 592 len 20480 data: 5 5 5 5 5
22926 251/300 Flush
22927 252/300 Write block 613 len 8192 data: 2 4
22928 253/300 Write block 1385 len 12288 data: 3 3 3
22929 254/300 Flush
22930 255/300 Read block 2619 len 20480
22931 256/300 Read block 803 len 16384
22932 257/300 Read block 1017 len 24576
22933 258/300 Write block 1917 len 20480 data: 8 7 6 7 8
22934 259/300 Read block 791 len 36864
22935 260/300 Write block 1249 len 8192 data: 6 6
22936 261/300 Read block 2216 len 16384
22937 262/300 Read block 2961 len 4096
22938 263/300 Write block 2985 len 40960 data: 3 3 2 2 2 2 2 2 2 2
22939 264/300 Write block 1126 len 20480 data: 5 6 5 5 4
22940 265/300 Flush
22941 266/300 Read block 3977 len 12288
22942 267/300 Read block 3402 len 32768
22943 268/300 Flush
22944 269/300 Read block 4178 len 32768
22945 270/300 Read block 1630 len 28672
22946 271/300 Read block 4699 len 36864
22947 272/300 Read block 1117 len 36864
22948 273/300 Read block 4025 len 12288
22949 274/300 Write block 2198 len 4096 data: 8
22950 275/300 Read block 4169 len 24576
22951 276/300 Write block 3454 len 4096 data: 5
22952 277/300 Write block 443 len 16384 data: 2 2 2 2
22953 278/300 Read block 919 len 40960
22954 279/300 Read block 3240 len 36864
22955 280/300 Write block 1350 len 32768 data: 3 5 3 4 4 4 3 3
22956 281/300 Read block 2449 len 28672
22957 282/300 Read block 2007 len 36864
22958 283/300 Read block 4445 len 24576
22959 284/300 Read block 3929 len 28672
22960 285/300 Write block 634 len 32768 data: 6 6 7 6 7 4 4 4
22961 286/300 Read block 2822 len 36864
22962 287/300 Write block 603 len 36864 data: 8 8 8 7 6 4 4 4 3
22963 288/300 Read block 2280 len 32768
22964 289/300 Write block 2458 len 8192 data: 6 6
22965 290/300 Read block 4894 len 32768
22966 291/300 Read block 1882 len 36864
22967 292/300 Read block 4629 len 20480
22968 293/300 Write block 1365 len 32768 data: 5 5 5 3 2 2 2 2
22969 294/300 Read block 873 len 32768
22970 295/300 Write block 1940 len 24576 data: 3 2 2 2 3 3
22971 296/300 Write block 298 len 28672 data: 3 3 3 4 6 6 6
22972 297/300 Flush
22973 298/300 Read block 474 len 36864
22974 299/300 Write block 389 len 28672 data: 4 5 5 5 8 7 6
22975 300/300 Flush
22976 301/300 Read block 639 len 4096
22977 Main task has work to do, go find it
22978 got dsc Start(0)
22979 start 0
22980 Mar 07 18:17:37.634 INFO request completed, latency_us: 164, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4d38c4e6-c51c-4d8e-a0c4-674e265fa2ea, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
22981 Replay: started 0, returned:Ok(())
22982 ----------------------------------------------------------------
22983 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22984 States: Offline Active Active
22985 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22986 6736 Acked Read 4 Sent Done Done false
22987 6737 Acked Read 7 Sent Done Done false
22988 6738 Acked Write 1 Sent Done Done false
22989 6739 Acked Flush 0 Sent Done Done false
22990 6740 Acked Write 5 Sent Done Done false
22991 6741 Acked Write 1 Sent Done Done false
22992 6742 Acked Write 3 Sent Done Done false
22993 6743 Acked Read 6 Sent Done Done false
22994 6744 Acked Flush 0 Sent Done Done false
22995 6745 Acked Write 2 Sent Done Done false
22996 6746 Acked Read 9 Sent Done Done false
22997 6747 Acked Read 9 Sent Done Done false
22998 6748 Acked Read 8 Sent Done Done false
22999 6749 Acked Read 8 Sent Done Done false
23000 6750 Acked Read 3 Sent Done Done false
23001 6751 Acked Read 7 Sent Done Done false
23002 6752 Acked Write 9 Sent Done Done false
23003 6753 Acked Read 6 Sent Done Done false
23004 6754 Acked Write 5 Sent Done Done false
23005 6755 Acked Write 10 Sent Done Done false
23006 6756 Acked Write 7 Sent Done Done false
23007 6757 Acked Read 8 Sent Done Done false
23008 6758 Acked Read 5 Sent Done Done false
23009 6759 Acked Read 4 Sent Done Done false
23010 6760 Acked Read 4 Sent Done Done false
23011 6761 Acked Write 9 Sent Done Done false
23012 6762 Acked Read 1 Sent Done Done false
23013 6763 Acked Flush 0 Sent Done Done false
23014 6764 Acked Read 9 Sent Done Done false
23015 6765 Acked Read 9 Sent Done Done false
23016 6766 Acked Write 9 Sent Done Done false
23017 6767 Acked Write 8 Sent Done Done false
23018 6768 Acked Write 7 Sent Done Done false
23019 6769 Acked Write 4 Sent Done Done false
23020 6770 Acked Read 8 Sent Done Done false
23021 6771 Acked Write 4 Sent Done Done false
23022 6772 Acked Read 2 Sent Done Done false
23023 6773 Acked Write 10 Sent Done Done false
23024 6774 Acked Write 3 Sent Done Done false
23025 6775 Acked Write 9 Sent Done Done false
23026 6776 Acked Read 7 Sent Done Done false
23027 6777 Acked Read 1 Sent Done Done false
23028 6778 Acked Read 4 Sent Done Done false
23029 6779 Acked Write 8 Sent Done Done false
23030 6780 Acked Read 4 Sent Done Done false
23031 6781 Acked Read 3 Sent Done Done false
23032 6782 Acked Write 4 Sent Done Done false
23033 6783 Acked Write 9 Sent Done Done false
23034 6784 Acked Read 5 Sent Done Done false
23035 6785 Acked Write 6 Sent Done Done false
23036 6786 Acked Read 9 Sent Done Done false
23037 6787 Acked Write 7 Sent Done Done false
23038 6788 Acked Flush 0 Sent Done Done false
23039 6789 Acked Write 6 Sent Done Done false
23040 6790 Acked Write 9 Sent Done Done false
23041 6791 Acked Write 10 Sent Done Done false
23042 6792 Acked Read 4 Sent Done Done false
23043 6793 Acked Read 1 Sent Done Done false
23044 6794 Acked Flush 0 Sent Done Done false
23045 6795 Acked Read 8 Sent Done Done false
23046 6796 Acked Read 3 Sent Done Done false
23047 6797 Acked Write 7 Sent Done Done false
23048 6798 Acked Write 3 Sent Done Done false
23049 6799 Acked Read 9 Sent Done Done false
23050 6800 Acked Read 4 Sent Done Done false
23051 6801 Acked Read 10 Sent Done Done false
23052 6802 Acked Flush 0 Sent Done Done false
23053 6803 Acked Read 6 Sent Done Done false
23054 6804 Acked Write 1 Sent Done Done false
23055 6805 Acked Read 5 Sent Done Done false
23056 6806 Acked Read 10 Sent Done Done false
23057 6807 Acked Write 7 Sent Done Done false
23058 6808 Acked Flush 0 Sent Done Done false
23059 6809 Acked Write 9 Sent Done Done false
23060 6810 Acked Read 9 Sent Done Done false
23061 6811 Acked Read 6 Sent Done Done false
23062 6812 Acked Write 4 Sent Done Done false
23063 6813 Acked Write 6 Sent Done Done false
23064 6814 Acked Read 8 Sent Done Done false
23065 6815 Acked Read 9 Sent Done Done false
23066 6816 Acked Write 8 Sent Done Done false
23067 6817 Acked Write 5 Sent Done Done false
23068 6818 Acked Read 2 Sent Done Done false
23069 6819 Acked Write 5 Sent Done Done false
23070 6820 Acked Read 4 Sent Done Done false
23071 6821 Acked Write 1 Sent Done Done false
23072 6822 Acked Read 2 Sent Done Done false
23073 6823 Acked Write 6 Sent Done Done false
23074 6824 Acked Read 4 Sent Done Done false
23075 6825 Acked Read 7 Sent Done Done false
23076 6826 Acked Write 1 Sent Done Done false
23077 6827 Acked Read 7 Sent Done Done false
23078 6828 Acked Read 7 Sent Done Done false
23079 6829 Acked Write 6 Sent Done Done false
23080 6830 Acked Write 2 Sent Done Done false
23081 6831 Acked Flush 0 Sent Done Done false
23082 6832 Acked Write 7 Sent Done Done false
23083 6833 Acked Read 3 Sent Done Done false
23084 6834 Acked Read 7 Sent Done Done false
23085 6835 Acked Read 10 Sent Done Done false
23086 6836 Acked Read 1 Sent Done Done false
23087 6837 Acked Read 1 Sent Done Done false
23088 6838 Acked Read 3 Sent Done Done false
23089 6839 Acked Flush 0 Sent Done Done false
23090 6840 Acked Read 8 Sent Done Done false
23091 6841 Acked Read 5 Sent Done Done false
23092 6842 Acked Write 6 Sent Done Done false
23093 6843 Acked Write 1 Sent Done Done false
23094 6844 Acked Read 8 Sent Done Done false
23095 6845 Acked Flush 0 Sent Done Done false
23096 6846 Acked Read 10 Sent Done Done false
23097 6847 Acked Write 9 Sent Done Done false
23098 6848 Acked Write 6 Sent Done Done false
23099 6849 Acked Read 10 Sent Done Done false
23100 6850 Acked Write 3 Sent Done Done false
23101 6851 Acked Read 5 Sent Done Done false
23102 6852 Acked Write 7 Sent Done Done false
23103 6853 Acked Write 5 Sent Done Done false
23104 6854 Acked Write 1 Sent Done Done false
23105 6855 Acked Read 9 Sent Done Done false
23106 6856 Acked Write 7 Sent Done Done false
23107 6857 Acked Flush 0 Sent Done Done false
23108 6858 Acked Read 5 Sent Done Done false
23109 6859 Acked Read 8 Sent Done Done false
23110 6860 Acked Read 6 Sent Done Done false
23111 6861 Acked Write 4 Sent Done Done false
23112 6862 Acked Read 6 Sent Done Done false
23113 6863 Acked Write 1 Sent Done Done false
23114 6864 Acked Write 2 Sent Done Done false
23115 6865 Acked Flush 0 Sent Done Done false
23116 6866 Acked Read 9 Sent Done Done false
23117 6867 Acked Read 4 Sent Done Done false
23118 6868 Acked Write 3 Sent Done Done false
23119 6869 Acked Read 7 Sent Done Done false
23120 6870 Acked Write 3 Sent Done Done false
23121 6871 Acked Read 2 Sent Done Done false
23122 6872 Acked Read 10 Sent Done Done false
23123 6873 Acked Read 4 Sent Done Done false
23124 6874 Acked Read 3 Sent Done Done false
23125 6875 Acked Read 2 Sent Done Done false
23126 6876 Acked Write 5 Sent Done Done false
23127 6877 Acked Flush 0 Sent Done Done false
23128 6878 Acked Read 9 Sent Done Done false
23129 6879 Acked Read 6 Sent Done Done false
23130 6880 Acked Flush 0 Sent Done Done false
23131 6881 Acked Write 2 Sent Done Done false
23132 6882 Acked Read 2 Sent Done Done false
23133 6883 Acked Read 8 Sent Done Done false
23134 6884 Acked Write 10 Sent Done Done false
23135 6885 Acked Write 9 Sent Done Done false
23136 6886 Acked Write 10 Sent Done Done false
23137 6887 Acked Read 6 Sent Done Done false
23138 6888 Acked Write 3 Sent Done Done false
23139 6889 Acked Read 4 Sent Done Done false
23140 6890 Acked Read 1 Sent Done Done false
23141 6891 Acked Read 4 Sent Done Done false
23142 6892 Acked Read 7 Sent Done Done false
23143 6893 Acked Read 4 Sent Done Done false
23144 6894 Acked Write 8 Sent Done Done false
23145 6895 Acked Write 1 Sent Done Done false
23146 6896 Acked Flush 0 Sent Done Done false
23147 6897 Acked Read 4 Sent Done Done false
23148 6898 Acked Write 2 Sent Done Done false
23149 6899 Acked Read 5 Sent Done Done false
23150 6900 Acked Read 8 Sent Done Done false
23151 6901 Acked Write 7 Sent Done Done false
23152 6902 Acked Read 8 Sent Done Done false
23153 6903 Acked Write 3 Sent Done Done false
23154 6904 Acked Write 2 Sent Done Done false
23155 6905 Acked Read 5 Sent Done Done false
23156 6906 Acked Read 10 Sent Done Done false
23157 6907 Acked Write 3 Sent Done Done false
23158 6908 Acked Read 4 Sent Done Done false
23159 6909 Acked Read 3 Sent Done Done false
23160 6910 Acked Write 10 Sent Done Done false
23161 6911 Acked Write 1 Sent Done Done false
23162 6912 Acked Read 7 Sent Done Done false
23163 6913 Acked Write 9 Sent Done Done false
23164 6914 Acked Write 4 Sent Done Done false
23165 6915 Acked Write 9 Sent Done Done false
23166 6916 Acked Read 5 Sent Done Done false
23167 6917 Acked Write 6 Sent Done Done false
23168 6918 Acked Read 4 Sent Done Done false
23169 6919 Acked Read 1 Sent Done Done false
23170 6920 Acked Read 2 Sent Done Done false
23171 6921 Acked Write 10 Sent Done Done false
23172 6922 Acked Read 10 Sent Done Done false
23173 6923 Acked Read 9 Sent Done Done false
23174 6924 Acked Read 4 Sent Done Done false
23175 6925 Acked Flush 0 Sent Done Done false
23176 6926 Acked Write 1 Sent Done Done false
23177 6927 Acked Read 8 Sent Done Done false
23178 6928 Acked Read 1 Sent Done Done false
23179 6929 Acked Read 6 Sent Done Done false
23180 6930 Acked Write 2 Sent Done Done false
23181 6931 Acked Read 7 Sent Done Done false
23182 6932 Acked Read 9 Sent Done Done false
23183 6933 Acked Write 9 Sent Done Done false
23184 6934 Acked Read 1 Sent Done Done false
23185 6935 Acked Write 3 Sent Done Done false
23186 6936 Acked Flush 0 Sent Done Done false
23187 6937 Acked Write 10 Sent Done Done false
23188 6938 Acked Read 3 Sent Done Done false
23189 6939 Acked Write 7 Sent Done Done false
23190 6940 Acked Read 9 Sent Done Done false
23191 6941 Acked Read 9 Sent Done Done false
23192 6942 Acked Read 3 Sent Done Done false
23193 6943 Acked Write 6 Sent Done Done false
23194 6944 Acked Flush 0 Sent Done Done false
23195 6945 Acked Write 6 Sent Done Done false
23196 6946 Acked Read 2 Sent Done Done false
23197 6947 Acked Read 2 Sent Done Done false
23198 6948 Acked Read 1 Sent Done Done false
23199 6949 Acked Write 10 Sent Done Done false
23200 6950 Acked Flush 0 Sent Done Done false
23201 6951 Acked Read 9 Sent Done Done false
23202 6952 Acked Read 2 Sent Done Done false
23203 6953 Acked Read 6 Sent Done Done false
23204 6954 Acked Read 6 Sent Done Done false
23205 6955 Acked Write 8 Sent Done Done false
23206 6956 Acked Read 7 Sent Done Done false
23207 6957 Acked Read 9 Sent Done Done false
23208 6958 Acked Read 2 Sent Done Done false
23209 6959 Acked Read 7 Sent Done Done false
23210 6960 Acked Write 1 Sent Done Done false
23211 6961 Acked Flush 0 Sent Done Done false
23212 6962 Acked Read 2 Sent Done Done false
23213 6963 Acked Read 10 Sent Done Done false
23214 6964 Acked Write 9 Sent Done Done false
23215 6965 Acked Read 4 Sent Done Done false
23216 6966 Acked Read 3 Sent Done Done false
23217 6967 Acked Write 4 Sent Done Done false
23218 6968 Acked Read 9 Sent Done Done false
23219 6969 Acked Read 5 Sent Done Done false
23220 6970 Acked Read 8 Sent Done Done false
23221 6971 Acked Write 1 Sent Done Done false
23222 6972 Acked Read 3 Sent Done Done false
23223 6973 Acked Read 6 Sent Done Done false
23224 6974 Acked Write 8 Sent Done Done false
23225 6975 Acked Read 9 Sent Done Done false
23226 6976 Acked Read 9 Sent Done Done false
23227 6977 Acked Flush 0 Sent Done Done false
23228 6978 Acked Write 7 Sent Done Done false
23229 6979 Acked Flush 0 Sent Done Done false
23230 6980 Acked Write 5 Sent Done Done false
23231 6981 Acked Write 2 Sent Done Done false
23232 6982 Acked Write 3 Sent Done Done false
23233 6983 Acked Read 7 Sent Done Done false
23234 6984 Acked Write 1 Sent Done Done false
23235 6985 Acked Write 5 Sent Done Done false
23236 6986 Acked Flush 0 Sent Done Done false
23237 6987 Acked Write 2 Sent Done Done false
23238 6988 Acked Write 3 Sent Done Done false
23239 6989 Acked Flush 0 Sent Done Done false
23240 6990 Acked Read 5 Sent Done Done false
23241 6991 Acked Read 4 Sent Done Done false
23242 6992 Acked Read 6 Sent Done Done false
23243 6993 Acked Write 5 Sent Done Done false
23244 6994 Acked Read 9 Sent Done Done false
23245 6995 Acked Write 2 Sent Done Done false
23246 6996 Acked Read 4 Sent Done Done false
23247 6997 Acked Read 1 Sent Done Done false
23248 6998 Acked Write 10 Sent Done Done false
23249 6999 Acked Write 5 Sent Done Done false
23250 7000 Acked Flush 0 Sent Done Done false
23251 7001 Acked Read 3 Sent Done Done false
23252 7002 Acked Read 8 Sent Done Done false
23253 7003 Acked Flush 0 Sent Done Done false
23254 7004 Acked Read 8 Sent Done Done false
23255 7005 Acked Read 7 Sent Done Done false
23256 7006 Acked Read 9 Sent Done Done false
23257 7007 Acked Read 9 Sent Done Done false
23258 7008 Acked Read 3 Sent Done Done false
23259 7009 Acked Write 1 Sent Done Done false
23260 7010 Acked Read 6 Sent Done Done false
23261 7011 Acked Write 1 Sent Done Done false
23262 7012 Acked Write 4 Sent Done Done false
23263 7013 Acked Read 10 Sent Done Done false
23264 7014 Acked Read 9 Sent Done Done false
23265 7015 Acked Write 8 Sent Done Done false
23266 7016 Acked Read 7 Sent Done Done false
23267 7017 Acked Read 9 Sent Done Done false
23268 7018 Acked Read 6 Sent Done Done false
23269 7019 Acked Read 7 Sent Done Done false
23270 7020 Acked Write 8 Sent Done Done false
23271 7021 Acked Read 9 Sent Done Done false
23272 7022 Acked Write 9 Sent Done Done false
23273 7023 Acked Read 8 Sent Done Done false
23274 7024 Acked Write 2 Sent Done Done false
23275 7025 Acked Read 8 Sent Done Done false
23276 7026 Acked Read 9 Sent Done Done false
23277 7027 Acked Read 5 Sent Done Done false
23278 7028 Acked Write 8 Sent Done Done false
23279 7029 Acked Read 8 Sent Done Done false
23280 7030 Acked Write 6 Sent Done Done false
23281 7031 Acked Write 7 Sent Done Done false
23282 7032 Acked Flush 0 Sent Done Done false
23283 7033 Acked Read 9 Sent Done Done false
23284 7034 Acked Write 7 Sent Done Done false
23285 7035 Acked Flush 0 Sent Done Done false
23286 7036 Acked Read 1 Sent Done Done false
23287 STATES DS:0 DS:1 DS:2 TOTAL
23288 Sent 301 0 0 301
23289 Done 0 301 301 602
23290 Skipped 0 0 0 0
23291 Error 0 0 0 0
23292 Last Flush: Some(JobId(6735)) Some(JobId(7035)) Some(JobId(7035))
23293 Downstairs last five completed: 6735 6734 6733 6732 6731
23294 Upstairs last five completed: 7036 7035 7034 7033 7032
23295 CLIENT: Up:0 ds:301 act:2
232962025-03-07T18:17:38.454ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
23297 [8810] got start action
23298 [8810] I am going to restart
23299 Starting downstairs at port 8810
23300 Make output file at "/tmp/dsc/downstairs-8810.txt"
23301 [8810][0] reports Starting
23302 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1228)
23303 [8810][0] reports Running
233042025-03-07T18:17:39.455ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
233052025-03-07T18:17:40.458ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
23306 ----------------------------------------------------------------
23307 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23308 States: Offline Active Active
23309 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23310 6736 Acked Read 4 Sent Done Done false
23311 6737 Acked Read 7 Sent Done Done false
23312 6738 Acked Write 1 Sent Done Done false
23313 6739 Acked Flush 0 Sent Done Done false
23314 6740 Acked Write 5 Sent Done Done false
23315 6741 Acked Write 1 Sent Done Done false
23316 6742 Acked Write 3 Sent Done Done false
23317 6743 Acked Read 6 Sent Done Done false
23318 6744 Acked Flush 0 Sent Done Done false
23319 6745 Acked Write 2 Sent Done Done false
23320 6746 Acked Read 9 Sent Done Done false
23321 6747 Acked Read 9 Sent Done Done false
23322 6748 Acked Read 8 Sent Done Done false
23323 6749 Acked Read 8 Sent Done Done false
23324 6750 Acked Read 3 Sent Done Done false
23325 6751 Acked Read 7 Sent Done Done false
23326 6752 Acked Write 9 Sent Done Done false
23327 6753 Acked Read 6 Sent Done Done false
23328 6754 Acked Write 5 Sent Done Done false
23329 6755 Acked Write 10 Sent Done Done false
23330 6756 Acked Write 7 Sent Done Done false
23331 6757 Acked Read 8 Sent Done Done false
23332 6758 Acked Read 5 Sent Done Done false
23333 6759 Acked Read 4 Sent Done Done false
23334 6760 Acked Read 4 Sent Done Done false
23335 6761 Acked Write 9 Sent Done Done false
23336 6762 Acked Read 1 Sent Done Done false
23337 6763 Acked Flush 0 Sent Done Done false
23338 6764 Acked Read 9 Sent Done Done false
23339 6765 Acked Read 9 Sent Done Done false
23340 6766 Acked Write 9 Sent Done Done false
23341 6767 Acked Write 8 Sent Done Done false
23342 6768 Acked Write 7 Sent Done Done false
23343 6769 Acked Write 4 Sent Done Done false
23344 6770 Acked Read 8 Sent Done Done false
23345 6771 Acked Write 4 Sent Done Done false
23346 6772 Acked Read 2 Sent Done Done false
23347 6773 Acked Write 10 Sent Done Done false
23348 6774 Acked Write 3 Sent Done Done false
23349 6775 Acked Write 9 Sent Done Done false
23350 6776 Acked Read 7 Sent Done Done false
23351 6777 Acked Read 1 Sent Done Done false
23352 6778 Acked Read 4 Sent Done Done false
23353 6779 Acked Write 8 Sent Done Done false
23354 6780 Acked Read 4 Sent Done Done false
23355 6781 Acked Read 3 Sent Done Done false
23356 6782 Acked Write 4 Sent Done Done false
23357 6783 Acked Write 9 Sent Done Done false
23358 6784 Acked Read 5 Sent Done Done false
23359 6785 Acked Write 6 Sent Done Done false
23360 6786 Acked Read 9 Sent Done Done false
23361 6787 Acked Write 7 Sent Done Done false
23362 6788 Acked Flush 0 Sent Done Done false
23363 6789 Acked Write 6 Sent Done Done false
23364 6790 Acked Write 9 Sent Done Done false
23365 6791 Acked Write 10 Sent Done Done false
23366 6792 Acked Read 4 Sent Done Done false
23367 6793 Acked Read 1 Sent Done Done false
23368 6794 Acked Flush 0 Sent Done Done false
23369 6795 Acked Read 8 Sent Done Done false
23370 6796 Acked Read 3 Sent Done Done false
23371 6797 Acked Write 7 Sent Done Done false
23372 6798 Acked Write 3 Sent Done Done false
23373 6799 Acked Read 9 Sent Done Done false
23374 6800 Acked Read 4 Sent Done Done false
23375 6801 Acked Read 10 Sent Done Done false
23376 6802 Acked Flush 0 Sent Done Done false
23377 6803 Acked Read 6 Sent Done Done false
23378 6804 Acked Write 1 Sent Done Done false
23379 6805 Acked Read 5 Sent Done Done false
23380 6806 Acked Read 10 Sent Done Done false
23381 6807 Acked Write 7 Sent Done Done false
23382 6808 Acked Flush 0 Sent Done Done false
23383 6809 Acked Write 9 Sent Done Done false
23384 6810 Acked Read 9 Sent Done Done false
23385 6811 Acked Read 6 Sent Done Done false
23386 6812 Acked Write 4 Sent Done Done false
23387 6813 Acked Write 6 Sent Done Done false
23388 6814 Acked Read 8 Sent Done Done false
23389 6815 Acked Read 9 Sent Done Done false
23390 6816 Acked Write 8 Sent Done Done false
23391 6817 Acked Write 5 Sent Done Done false
23392 6818 Acked Read 2 Sent Done Done false
23393 6819 Acked Write 5 Sent Done Done false
23394 6820 Acked Read 4 Sent Done Done false
23395 6821 Acked Write 1 Sent Done Done false
23396 6822 Acked Read 2 Sent Done Done false
23397 6823 Acked Write 6 Sent Done Done false
23398 6824 Acked Read 4 Sent Done Done false
23399 6825 Acked Read 7 Sent Done Done false
23400 6826 Acked Write 1 Sent Done Done false
23401 6827 Acked Read 7 Sent Done Done false
23402 6828 Acked Read 7 Sent Done Done false
23403 6829 Acked Write 6 Sent Done Done false
23404 6830 Acked Write 2 Sent Done Done false
23405 6831 Acked Flush 0 Sent Done Done false
23406 6832 Acked Write 7 Sent Done Done false
23407 6833 Acked Read 3 Sent Done Done false
23408 6834 Acked Read 7 Sent Done Done false
23409 6835 Acked Read 10 Sent Done Done false
23410 6836 Acked Read 1 Sent Done Done false
23411 6837 Acked Read 1 Sent Done Done false
23412 6838 Acked Read 3 Sent Done Done false
23413 6839 Acked Flush 0 Sent Done Done false
23414 6840 Acked Read 8 Sent Done Done false
23415 6841 Acked Read 5 Sent Done Done false
23416 6842 Acked Write 6 Sent Done Done false
23417 6843 Acked Write 1 Sent Done Done false
23418 6844 Acked Read 8 Sent Done Done false
23419 6845 Acked Flush 0 Sent Done Done false
23420 6846 Acked Read 10 Sent Done Done false
23421 6847 Acked Write 9 Sent Done Done false
23422 6848 Acked Write 6 Sent Done Done false
23423 6849 Acked Read 10 Sent Done Done false
23424 6850 Acked Write 3 Sent Done Done false
23425 6851 Acked Read 5 Sent Done Done false
23426 6852 Acked Write 7 Sent Done Done false
23427 6853 Acked Write 5 Sent Done Done false
23428 6854 Acked Write 1 Sent Done Done false
23429 6855 Acked Read 9 Sent Done Done false
23430 6856 Acked Write 7 Sent Done Done false
23431 6857 Acked Flush 0 Sent Done Done false
23432 6858 Acked Read 5 Sent Done Done false
23433 6859 Acked Read 8 Sent Done Done false
23434 6860 Acked Read 6 Sent Done Done false
23435 6861 Acked Write 4 Sent Done Done false
23436 6862 Acked Read 6 Sent Done Done false
23437 6863 Acked Write 1 Sent Done Done false
23438 6864 Acked Write 2 Sent Done Done false
23439 6865 Acked Flush 0 Sent Done Done false
23440 6866 Acked Read 9 Sent Done Done false
23441 6867 Acked Read 4 Sent Done Done false
23442 6868 Acked Write 3 Sent Done Done false
23443 6869 Acked Read 7 Sent Done Done false
23444 6870 Acked Write 3 Sent Done Done false
23445 6871 Acked Read 2 Sent Done Done false
23446 6872 Acked Read 10 Sent Done Done false
23447 6873 Acked Read 4 Sent Done Done false
23448 6874 Acked Read 3 Sent Done Done false
23449 6875 Acked Read 2 Sent Done Done false
23450 6876 Acked Write 5 Sent Done Done false
23451 6877 Acked Flush 0 Sent Done Done false
23452 6878 Acked Read 9 Sent Done Done false
23453 6879 Acked Read 6 Sent Done Done false
23454 6880 Acked Flush 0 Sent Done Done false
23455 6881 Acked Write 2 Sent Done Done false
23456 6882 Acked Read 2 Sent Done Done false
23457 6883 Acked Read 8 Sent Done Done false
23458 6884 Acked Write 10 Sent Done Done false
23459 6885 Acked Write 9 Sent Done Done false
23460 6886 Acked Write 10 Sent Done Done false
23461 6887 Acked Read 6 Sent Done Done false
23462 6888 Acked Write 3 Sent Done Done false
23463 6889 Acked Read 4 Sent Done Done false
23464 6890 Acked Read 1 Sent Done Done false
23465 6891 Acked Read 4 Sent Done Done false
23466 6892 Acked Read 7 Sent Done Done false
23467 6893 Acked Read 4 Sent Done Done false
23468 6894 Acked Write 8 Sent Done Done false
23469 6895 Acked Write 1 Sent Done Done false
23470 6896 Acked Flush 0 Sent Done Done false
23471 6897 Acked Read 4 Sent Done Done false
23472 6898 Acked Write 2 Sent Done Done false
23473 6899 Acked Read 5 Sent Done Done false
23474 6900 Acked Read 8 Sent Done Done false
23475 6901 Acked Write 7 Sent Done Done false
23476 6902 Acked Read 8 Sent Done Done false
23477 6903 Acked Write 3 Sent Done Done false
23478 6904 Acked Write 2 Sent Done Done false
23479 6905 Acked Read 5 Sent Done Done false
23480 6906 Acked Read 10 Sent Done Done false
23481 6907 Acked Write 3 Sent Done Done false
23482 6908 Acked Read 4 Sent Done Done false
23483 6909 Acked Read 3 Sent Done Done false
23484 6910 Acked Write 10 Sent Done Done false
23485 6911 Acked Write 1 Sent Done Done false
23486 6912 Acked Read 7 Sent Done Done false
23487 6913 Acked Write 9 Sent Done Done false
23488 6914 Acked Write 4 Sent Done Done false
23489 6915 Acked Write 9 Sent Done Done false
23490 6916 Acked Read 5 Sent Done Done false
23491 6917 Acked Write 6 Sent Done Done false
23492 6918 Acked Read 4 Sent Done Done false
23493 6919 Acked Read 1 Sent Done Done false
23494 6920 Acked Read 2 Sent Done Done false
23495 6921 Acked Write 10 Sent Done Done false
23496 6922 Acked Read 10 Sent Done Done false
23497 6923 Acked Read 9 Sent Done Done false
23498 6924 Acked Read 4 Sent Done Done false
23499 6925 Acked Flush 0 Sent Done Done false
23500 6926 Acked Write 1 Sent Done Done false
23501 6927 Acked Read 8 Sent Done Done false
23502 6928 Acked Read 1 Sent Done Done false
23503 6929 Acked Read 6 Sent Done Done false
23504 6930 Acked Write 2 Sent Done Done false
23505 6931 Acked Read 7 Sent Done Done false
23506 6932 Acked Read 9 Sent Done Done false
23507 6933 Acked Write 9 Sent Done Done false
23508 6934 Acked Read 1 Sent Done Done false
23509 6935 Acked Write 3 Sent Done Done false
23510 6936 Acked Flush 0 Sent Done Done false
23511 6937 Acked Write 10 Sent Done Done false
23512 6938 Acked Read 3 Sent Done Done false
23513 6939 Acked Write 7 Sent Done Done false
23514 6940 Acked Read 9 Sent Done Done false
23515 6941 Acked Read 9 Sent Done Done false
23516 6942 Acked Read 3 Sent Done Done false
23517 6943 Acked Write 6 Sent Done Done false
23518 6944 Acked Flush 0 Sent Done Done false
23519 6945 Acked Write 6 Sent Done Done false
23520 6946 Acked Read 2 Sent Done Done false
23521 6947 Acked Read 2 Sent Done Done false
23522 6948 Acked Read 1 Sent Done Done false
23523 6949 Acked Write 10 Sent Done Done false
23524 6950 Acked Flush 0 Sent Done Done false
23525 6951 Acked Read 9 Sent Done Done false
23526 6952 Acked Read 2 Sent Done Done false
23527 6953 Acked Read 6 Sent Done Done false
23528 6954 Acked Read 6 Sent Done Done false
23529 6955 Acked Write 8 Sent Done Done false
23530 6956 Acked Read 7 Sent Done Done false
23531 6957 Acked Read 9 Sent Done Done false
23532 6958 Acked Read 2 Sent Done Done false
23533 6959 Acked Read 7 Sent Done Done false
23534 6960 Acked Write 1 Sent Done Done false
23535 6961 Acked Flush 0 Sent Done Done false
23536 6962 Acked Read 2 Sent Done Done false
23537 6963 Acked Read 10 Sent Done Done false
23538 6964 Acked Write 9 Sent Done Done false
23539 6965 Acked Read 4 Sent Done Done false
23540 6966 Acked Read 3 Sent Done Done false
23541 6967 Acked Write 4 Sent Done Done false
23542 6968 Acked Read 9 Sent Done Done false
23543 6969 Acked Read 5 Sent Done Done false
23544 6970 Acked Read 8 Sent Done Done false
23545 6971 Acked Write 1 Sent Done Done false
23546 6972 Acked Read 3 Sent Done Done false
23547 6973 Acked Read 6 Sent Done Done false
23548 6974 Acked Write 8 Sent Done Done false
23549 6975 Acked Read 9 Sent Done Done false
23550 6976 Acked Read 9 Sent Done Done false
23551 6977 Acked Flush 0 Sent Done Done false
23552 6978 Acked Write 7 Sent Done Done false
23553 6979 Acked Flush 0 Sent Done Done false
23554 6980 Acked Write 5 Sent Done Done false
23555 6981 Acked Write 2 Sent Done Done false
23556 6982 Acked Write 3 Sent Done Done false
23557 6983 Acked Read 7 Sent Done Done false
23558 6984 Acked Write 1 Sent Done Done false
23559 6985 Acked Write 5 Sent Done Done false
23560 6986 Acked Flush 0 Sent Done Done false
23561 6987 Acked Write 2 Sent Done Done false
23562 6988 Acked Write 3 Sent Done Done false
23563 6989 Acked Flush 0 Sent Done Done false
23564 6990 Acked Read 5 Sent Done Done false
23565 6991 Acked Read 4 Sent Done Done false
23566 6992 Acked Read 6 Sent Done Done false
23567 6993 Acked Write 5 Sent Done Done false
23568 6994 Acked Read 9 Sent Done Done false
23569 6995 Acked Write 2 Sent Done Done false
23570 6996 Acked Read 4 Sent Done Done false
23571 6997 Acked Read 1 Sent Done Done false
23572 6998 Acked Write 10 Sent Done Done false
23573 6999 Acked Write 5 Sent Done Done false
23574 7000 Acked Flush 0 Sent Done Done false
23575 7001 Acked Read 3 Sent Done Done false
23576 7002 Acked Read 8 Sent Done Done false
23577 7003 Acked Flush 0 Sent Done Done false
23578 7004 Acked Read 8 Sent Done Done false
23579 7005 Acked Read 7 Sent Done Done false
23580 7006 Acked Read 9 Sent Done Done false
23581 7007 Acked Read 9 Sent Done Done false
23582 7008 Acked Read 3 Sent Done Done false
23583 7009 Acked Write 1 Sent Done Done false
23584 7010 Acked Read 6 Sent Done Done false
23585 7011 Acked Write 1 Sent Done Done false
23586 7012 Acked Write 4 Sent Done Done false
23587 7013 Acked Read 10 Sent Done Done false
23588 7014 Acked Read 9 Sent Done Done false
23589 7015 Acked Write 8 Sent Done Done false
23590 7016 Acked Read 7 Sent Done Done false
23591 7017 Acked Read 9 Sent Done Done false
23592 7018 Acked Read 6 Sent Done Done false
23593 7019 Acked Read 7 Sent Done Done false
23594 7020 Acked Write 8 Sent Done Done false
23595 7021 Acked Read 9 Sent Done Done false
23596 7022 Acked Write 9 Sent Done Done false
23597 7023 Acked Read 8 Sent Done Done false
23598 7024 Acked Write 2 Sent Done Done false
23599 7025 Acked Read 8 Sent Done Done false
23600 7026 Acked Read 9 Sent Done Done false
23601 7027 Acked Read 5 Sent Done Done false
23602 7028 Acked Write 8 Sent Done Done false
23603 7029 Acked Read 8 Sent Done Done false
23604 7030 Acked Write 6 Sent Done Done false
23605 7031 Acked Write 7 Sent Done Done false
23606 7032 Acked Flush 0 Sent Done Done false
23607 7033 Acked Read 9 Sent Done Done false
23608 7034 Acked Write 7 Sent Done Done false
23609 7035 Acked Flush 0 Sent Done Done false
23610 7036 Acked Read 1 Sent Done Done false
23611 STATES DS:0 DS:1 DS:2 TOTAL
23612 Sent 301 0 0 301
23613 Done 0 301 301 602
23614 Skipped 0 0 0 0
23615 Error 0 0 0 0
23616 Last Flush: Some(JobId(6735)) Some(JobId(7035)) Some(JobId(7035))
23617 Downstairs last five completed: 6735 6734 6733 6732 6731
23618 Upstairs last five completed: 7036 7035 7034 7033 7032
23619 CLIENT: Up:0 ds:301 act:2
236202025-03-07T18:17:42.461ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236212025-03-07T18:17:43.376ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236222025-03-07T18:17:43.376ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60136) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236232025-03-07T18:17:43.377ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236242025-03-07T18:17:43.378ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236252025-03-07T18:17:43.378ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236262025-03-07T18:17:43.378ZINFOcrucible: send last flush ID to this DS: Some(JobId(6735)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236272025-03-07T18:17:43.378ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236282025-03-07T18:17:43.378ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6735)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
236292025-03-07T18:17:43.378ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6735)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
23630 ----------------------------------------------------------------
23631 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23632 Downstairs last five completed: 7037 7036 7035 7034 7033
23633 Upstairs last five completed: 7037 7036 7035 7034 7033
23634 CLIENT: Up:0 ds:0 act:3
23635 Replay: All jobs finished
23636 Main task has work to do, go find it
23637 got dsc Stop(1)
23638 stop 1
23639 Mar 07 18:17:45.639 INFO request completed[8820] Got stop action so:false kr:false
23640 , latency_us: 334, response_code: 204, uri: /stop/cid/1, method: POST, req_id: cf8e75d7-685a-4084-9cdb-9d69d43471e9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
236412025-03-07T18:17:45.640ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
23642 Mar 07 18:17:45.640{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:45.640635305Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
23643 INFO request completed, latency_us: 147, response_code: {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:17:45.640674721Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
23644 200, uri: /state/cid/1, method: GET, req_id: 4c8d4d08-4e24-4910-91c8-88265bc00396, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
236452025-03-07T18:17:45.640ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
23646 [8820] Exited with: ExitStatus(unix_wait_status(9))
23647 [8820][1] reports Exit
23648 Mar 07 18:17:49.642 INFO request completed, latency_us: 290, response_code: 200, uri: /state/cid/1, method: GET, req_id: 61a4c6d7-d36e-497a-b520-3aca88821d01, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
23649 001/300 Flush
23650 002/300 Read block 1882 len 12288
23651 003/300 Write block 4882 len 16384 data: 2 2 2 3
23652 004/300 Read block 245 len 36864
23653 005/300 Read block 2830 len 40960
23654 006/300 Write block 4227 len 8192 data: 4 4
23655 007/300 Write block 15 len 8192 data: 8 7
23656 008/300 Write block 892 len 24576 data: 4 3 3 3 2 3
23657 009/300 Read block 416 len 24576
23658 010/300 Read block 4141 len 40960
23659 011/300 Read block 2909 len 4096
23660 012/300 Write block 4376 len 8192 data: 7 7
23661 013/300 Read block 4662 len 12288
23662 014/300 Write block 4331 len 12288 data: 7 7 7
23663 015/300 Write block 1616 len 16384 data: 7 8 7 8
23664 016/300 Write block 3078 len 4096 data: 6
23665 017/300 Read block 1064 len 12288
23666 018/300 Write block 668 len 16384 data: 5 5 5 5
23667 019/300 Flush
23668 020/300 Flush
23669 021/300 Read block 153 len 16384
23670 022/300 Read block 3636 len 16384
23671 023/300 Read block 3867 len 20480
23672 024/300 Write block 1128 len 16384 data: 6 6 5 3
23673 025/300 Write block 1580 len 8192 data: 4 4
23674 026/300 Write block 4790 len 28672 data: 5 5 5 5 6 5 5
23675 027/300 Read block 324 len 40960
23676 028/300 Write block 2312 len 32768 data: 3 3 5 3 2 3 2 3
23677 029/300 Write block 421 len 40960 data: 5 6 7 7 7 7 5 6 7 7
23678 030/300 Read block 3820 len 20480
23679 031/300 Write block 3301 len 16384 data: 4 5 6 6
23680 032/300 Read block 1706 len 16384
23681 033/300 Write block 1911 len 36864 data: 9 7 6 4 6 6 9 8 7
23682 034/300 Write block 2884 len 40960 data: 5 6 7 8 7 8 8 7 7 7
23683 035/300 Read block 2746 len 12288
23684 036/300 Read block 1677 len 12288
23685 037/300 Read block 1603 len 24576
23686 038/300 Write block 1326 len 16384 data: 4 4 5 4
23687 039/300 Read block 971 len 16384
23688 040/300 Read block 1011 len 20480
23689 041/300 Flush
23690 042/300 Read block 537 len 24576
23691 043/300 Write block 4488 len 28672 data: 3 2 2 2 2 3 4
23692 044/300 Read block 4063 len 16384
23693 045/300 Read block 3040 len 8192
23694 046/300 Read block 2401 len 24576
23695 047/300 Read block 1125 len 20480
23696 048/300 Flush
23697 049/300 Write block 3348 len 32768 data: 2 3 4 4 5 4 4 4
23698 050/300 Read block 3881 len 32768
23699 051/300 Read block 1144 len 40960
23700 052/300 Write block 2263 len 28672 data: 4 4 5 5 3 3 4
23701 053/300 Read block 3077 len 16384
23702 054/300 Read block 1847 len 28672
23703 055/300 Read block 3676 len 12288
23704 056/300 Write block 3178 len 36864 data: 4 4 4 2 3 3 2 3 4
23705 057/300 Write block 10 len 24576 data: 6 7 7 7 7 9
23706 058/300 Read block 4894 len 28672
23707 059/300 Write block 804 len 16384 data: 4 4 5 5
23708 060/300 Read block 447 len 40960
23709 061/300 Write block 4702 len 12288 data: 5 5 6
23710 062/300 Flush
23711 063/300 Write block 3838 len 16384 data: 2 2 2 2
23712 064/300 Read block 1802 len 8192
23713 065/300 Read block 2375 len 32768
23714 066/300 Read block 2268 len 28672
23715 067/300 Write block 285 len 36864 data: 4 5 6 7 7 7 6 6 5
23716 068/300 Write block 533 len 12288 data: 6 4 4
23717 069/300 Write block 1259 len 32768 data: 3 3 3 3 4 4 4 5
23718 070/300 Read block 4535 len 36864
23719 071/300 Read block 3820 len 40960
23720 072/300 Read block 1134 len 28672
23721 073/300 Write block 4623 len 16384 data: 4 7 5 4
23722 074/300 Write block 1988 len 12288 data: 4 4 3
23723 075/300 Write block 3783 len 4096 data: 6
23724 076/300 Write block 2120 len 20480 data: 7 6 6 6 5
23725 077/300 Write block 2653 len 40960 data: 5 6 7 7 8 9 8 6 7 7
23726 078/300 Read block 2104 len 12288
23727 079/300 Write block 100 len 20480 data: 4 4 4 4 4
23728 080/300 Read block 2254 len 4096
23729 081/300 Write block 4083 len 12288 data: 5 4 4
23730 082/300 Write block 3758 len 16384 data: 6 5 5 4
23731 083/300 Write block 753 len 16384 data: 4 4 4 5
23732 084/300 Write block 2022 len 8192 data: 5 5
23733 085/300 Flush
23734 086/300 Read block 3586 len 24576
23735 087/300 Write block 2531 len 16384 data: 6 6 6 7
23736 088/300 Read block 952 len 40960
23737 089/300 Write block 1507 len 40960 data: 3 3 3 3 3 5 5 5 6 6
23738 090/300 Read block 2311 len 4096
23739 091/300 Write block 3784 len 28672 data: 6 4 4 3 3 3 3
23740 092/300 Read block 4418 len 12288
23741 093/300 Write block 2072 len 32768 data: 3 3 3 3 3 4 3 3
23742 094/300 Write block 3169 len 28672 data: 4 4 4 3 3 2 4
23743 095/300 Write block 565 len 40960 data: 4 4 5 4 3 3 3 4 4 4
23744 096/300 Write block 2371 len 32768 data: 4 3 3 3 3 5 4 4
23745 097/300 Write block 2973 len 4096 data: 5
23746 098/300 Read block 4650 len 12288
23747 099/300 Write block 3546 len 28672 data: 3 2 2 2 3 4 5
23748 100/300 Read block 4308 len 32768
23749 101/300 Write block 2878 len 28672 data: 4 4 4 4 3 4 6
23750 102/300 Write block 2133 len 12288 data: 5 4 4
23751 103/300 Write block 2825 len 20480 data: 6 4 5 4 4
23752 104/300 Read block 2333 len 28672
23753 105/300 Read block 4408 len 36864
23754 106/300 Read block 729 len 12288
23755 107/300 Read block 2473 len 28672
23756 108/300 Read block 3640 len 8192
23757 109/300 Write block 2153 len 8192 data: 2 3
23758 110/300 Read block 2063 len 12288
23759 111/300 Flush
23760 112/300 Flush
23761 113/300 Read block 4392 len 32768
23762 114/300 Write block 4095 len 8192 data: 2 2
23763 115/300 Flush
23764 116/300 Write block 1586 len 8192 data: 4 5
23765 117/300 Read block 2490 len 8192
23766 118/300 Read block 3898 len 40960
23767 119/300 Flush
23768 120/300 Read block 508 len 28672
23769 121/300 Read block 1238 len 8192
23770 122/300 Read block 3106 len 16384
23771 123/300 Write block 4568 len 32768 data: 3 3 3 2 2 2 4 4
23772 124/300 Read block 809 len 20480
23773 125/300 Read block 2460 len 28672
23774 126/300 Flush
23775 127/300 Read block 3033 len 8192
23776 128/300 Write block 1326 len 4096 data: 5
23777 129/300 Read block 2510 len 28672
23778 130/300 Read block 4437 len 16384
23779 131/300 Read block 1120 len 28672
23780 132/300 Write block 531 len 16384 data: 5 6 7 5
23781 133/300 Read block 3477 len 4096
23782 134/300 Write block 3992 len 28672 data: 7 6 6 7 5 5 5
23783 135/300 Write block 189 len 24576 data: 5 6 6 5 7 6
23784 136/300 Read block 1921 len 36864
23785 137/300 Write block 4211 len 32768 data: 8 8 7 6 7 5 5 5
23786 138/300 Read block 1395 len 24576
23787 139/300 Write block 833 len 20480 data: 2 3 4 4 3
23788 140/300 Write block 3146 len 12288 data: 6 7 5
23789 141/300 Read block 2719 len 12288
23790 142/300 Write block 2445 len 12288 data: 3 4 4
23791 143/300 Read block 3938 len 36864
23792 144/300 Write block 2262 len 4096 data: 4
23793 145/300 Read block 703 len 24576
23794 146/300 Write block 2582 len 4096 data: 4
23795 147/300 Write block 4086 len 12288 data: 5 5 4
23796 148/300 Read block 7 len 8192
23797 149/300 Write block 3970 len 32768 data: 5 5 5 4 3 3 2 2
23798 150/300 Read block 2619 len 8192
23799 151/300 Read block 1700 len 28672
23800 152/300 Read block 526 len 8192
23801 153/300 Write block 3276 len 32768 data: 4 5 6 6 6 9 7 8
23802 154/300 Read block 61 len 12288
23803 155/300 Read block 1099 len 36864
23804 156/300 Flush
23805 157/300 Write block 2578 len 16384 data: 4 4 3 4
23806 158/300 Write block 3158 len 36864 data: 3 4 3 3 3 3 3 3 3
23807 159/300 Write block 1503 len 4096 data: 3
23808 160/300 Read block 1230 len 12288
23809 161/300 Write block 184 len 28672 data: 2 3 3 3 4 6 7
23810 162/300 Read block 3938 len 20480
23811 163/300 Write block 1179 len 28672 data: 6 5 6 5 5 5 6
23812 164/300 Read block 4308 len 36864
23813 165/300 Read block 4515 len 32768
23814 166/300 Read block 2231 len 4096
23815 167/300 Read block 473 len 24576
23816 168/300 Write block 3846 len 20480 data: 3 3 3 4 5
23817 169/300 Read block 1356 len 4096
23818 170/300 Write block 729 len 20480 data: 6 6 8 7 7
23819 171/300 Write block 4472 len 24576 data: 4 4 4 4 3 3
23820 172/300 Flush
23821 173/300 Write block 2720 len 12288 data: 4 4 4
23822 174/300 Read block 4098 len 20480
23823 175/300 Flush
23824 176/300 Write block 2791 len 36864 data: 4 4 4 4 2 2 3 3 5
23825 177/300 Read block 2441 len 16384
23826 178/300 Read block 299 len 12288
23827 179/300 Read block 2278 len 28672
23828 180/300 Read block 1286 len 12288
23829 181/300 Read block 2874 len 32768
23830 182/300 Write block 2146 len 16384 data: 4 4 5 4
23831 183/300 Write block 4078 len 32768 data: 6 6 6 5 5 6 5 5
23832 184/300 Write block 2902 len 32768 data: 7 6 6 5 4 3 3 4
23833 185/300 Write block 2467 len 24576 data: 6 6 4 4 4 4
23834 186/300 Write block 2662 len 4096 data: 8
23835 187/300 Write block 2677 len 16384 data: 3 3 4 4
23836 188/300 Read block 3454 len 4096
23837 189/300 Write block 86 len 36864 data: 3 3 3 3 4 4 4 5 5
23838 190/300 Read block 149 len 28672
23839 191/300 Flush
23840 192/300 Write block 1605 len 28672 data: 4 3 3 3 2 3 3
23841 193/300 Read block 4523 len 8192
23842 194/300 Read block 1188 len 32768
23843 195/300 Read block 1474 len 12288
23844 196/300 Read block 4215 len 28672
23845 197/300 Read block 195 len 12288
23846 198/300 Write block 1528 len 36864 data: 5 6 6 6 6 6 5 5 4
23847 199/300 Read block 866 len 36864
23848 200/300 Read block 4159 len 40960
23849 201/300 Read block 3126 len 28672
23850 202/300 Read block 1572 len 8192
23851 203/300 Flush
23852 204/300 Write block 1918 len 28672 data: 9 8 8 9 10 9 8
23853 205/300 Read block 1808 len 36864
23854 206/300 Read block 42 len 40960
23855 207/300 Write block 4825 len 16384 data: 7 7 6 5
23856 208/300 Write block 997 len 32768 data: 3 2 2 3 3 3 3 4
23857 209/300 Flush
23858 210/300 Read block 1654 len 28672
23859 211/300 Read block 2373 len 20480
23860 212/300 Read block 2012 len 32768
23861 213/300 Write block 2545 len 24576 data: 6 6 8 7 7 6
23862 214/300 Read block 4975 len 36864
23863 215/300 Read block 666 len 4096
23864 216/300 Write block 3082 len 8192 data: 8 8
23865 217/300 Write block 4013 len 4096 data: 3
23866 218/300 Read block 14 len 4096
23867 219/300 Read block 4416 len 24576
23868 220/300 Write block 2557 len 20480 data: 4 4 4 5 6
23869 221/300 Write block 3710 len 4096 data: 3
23870 222/300 Read block 2993 len 28672
23871 223/300 Write block 2805 len 40960 data: 4 4 5 5 5 4 4 4 3 2
23872 224/300 Read block 2003 len 32768
23873 225/300 Read block 1297 len 8192
23874 226/300 Read block 2483 len 16384
23875 227/300 Read block 3048 len 12288
23876 228/300 Write block 3827 len 20480 data: 5 5 7 7 6
23877 229/300 Write block 564 len 24576 data: 5 5 5 6 5 4
23878 230/300 Write block 2822 len 24576 data: 7 7 5 7 5 6
23879 231/300 Read block 1849 len 12288
23880 232/300 Read block 2268 len 12288
23881 233/300 Write block 4493 len 40960 data: 4 5 2 3 3 3 2 2 2 4
23882 234/300 Write block 2655 len 28672 data: 8 8 9 10 9 7 8
23883 235/300 Read block 4886 len 8192
23884 236/300 Write block 4917 len 16384 data: 7 8 7 7
23885 237/300 Read block 53 len 12288
23886 238/300 Write block 1459 len 12288 data: 6 6 8
23887 239/300 Write block 1509 len 20480 data: 4 4 4 6 6
23888 240/300 Write block 321 len 28672 data: 4 5 5 5 6 6 6
23889 241/300 Read block 3126 len 4096
23890 242/300 Read block 578 len 12288
23891 243/300 Write block 51 len 20480 data: 3 4 4 4 4
23892 244/300 Write block 2546 len 32768 data: 7 9 8 8 7 5 5 6
23893 245/300 Write block 4853 len 8192 data: 5 5
23894 246/300 Write block 3661 len 32768 data: 5 5 6 5 6 6 6 5
23895 247/300 Read block 2107 len 40960
23896 248/300 Read block 1887 len 36864
23897 249/300 Read block 3314 len 36864
23898 250/300 Read block 671 len 8192
23899 251/300 Flush
23900 252/300 Read block 2287 len 12288
23901 253/300 Write block 3447 len 24576 data: 5 5 6 6 6 5
23902 254/300 Read block 4453 len 32768
23903 255/300 Read block 3617 len 4096
23904 256/300 Write block 3107 len 40960 data: 4 4 4 3 4 4 6 6 5 5
23905 257/300 Read block 1119 len 20480
23906 258/300 Read block 4586 len 24576
23907 259/300 Write block 1727 len 16384 data: 5 6 4 3
23908 260/300 Write block 3707 len 28672 data: 6 5 5 4 2 2 2
23909 261/300 Write block 3650 len 40960 data: 6 5 4 4 4 5 4 4 4 5
23910 262/300 Write block 2956 len 4096 data: 4
23911 263/300 Write block 3360 len 16384 data: 4 4 5 7
23912 264/300 Write block 1698 len 28672 data: 6 8 5 5 6 8 8
23913 265/300 Read block 2945 len 4096
23914 266/300 Read block 1391 len 4096
23915 267/300 Write block 4712 len 40960 data: 3 3 2 2 2 2 3 5 6 4
23916 268/300 Read block 2643 len 28672
23917 269/300 Write block 1946 len 40960 data: 3 3 3 5 5 5 3 3 2 3
23918 270/300 Write block 1366 len 20480 data: 6 6 4 3 3
23919 271/300 Write block 2 len 20480 data: 4 4 4 5 5
23920 272/300 Read block 2565 len 20480
23921 273/300 Read block 3272 len 16384
23922 274/300 Read block 4466 len 32768
23923 275/300 Read block 3496 len 28672
23924 276/300 Write block 3946 len 16384 data: 6 5 7 7
23925 277/300 Write block 902 len 16384 data: 3 4 4 4
23926 278/300 Write block 2119 len 36864 data: 8 8 7 7 7 6 5 5 4
23927 279/300 Read block 4798 len 12288
23928 280/300 Read block 1627 len 20480
23929 281/300 Write block 3094 len 40960 data: 4 4 5 6 6 6 7 7 7 5
23930 282/300 Write block 1567 len 28672 data: 9 10 10 8 7 5 6
23931 283/300 Read block 4636 len 24576
23932 284/300 Read block 2276 len 20480
23933 285/300 Write block 3812 len 40960 data: 7 5 4 3 3 3 2 3 3 3
23934 286/300 Read block 4823 len 32768
23935 287/300 Read block 2318 len 36864
23936 288/300 Read block 3486 len 40960
23937 289/300 Read block 12 len 20480
23938 290/300 Read block 3327 len 4096
23939 291/300 Read block 4668 len 24576
23940 292/300 Write block 4233 len 32768 data: 2 2 2 3 3 3 3 4
23941 293/300 Read block 1006 len 16384
23942 294/300 Write block 433 len 4096 data: 6
23943 295/300 Read block 2122 len 24576
23944 296/300 Flush
23945 297/300 Write block 737 len 32768 data: 8 7 5 7 6 7 7 8
23946 298/300 Write block 1600 len 16384 data: 6 7 7 7
23947 299/300 Read block 1304 len 32768
23948 300/300 Read block 4774 len 4096
23949 301/300 Flush
23950 Main task has work to do, go find it
23951 got dsc Start(1)
23952 start 1
23953 Mar 07 18:17:49.887 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/1, method: POST, req_id: a742851e-695a-4782-ae8f-d969e17dc035, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
23954 Replay: started 1, returned:Ok(())
23955 ----------------------------------------------------------------
23956 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23957 States: Active Offline Active
23958 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23959 7038 Acked Flush 0 Done Sent Done false
23960 7039 Acked Read 3 Done Sent Done false
23961 7040 Acked Write 4 Done Sent Done false
23962 7041 Acked Read 9 Done Sent Done false
23963 7042 Acked Read 10 Done Sent Done false
23964 7043 Acked Write 2 Done Sent Done false
23965 7044 Acked Write 2 Done Sent Done false
23966 7045 Acked Write 6 Done Sent Done false
23967 7046 Acked Read 6 Done Sent Done false
23968 7047 Acked Read 10 Done Sent Done false
23969 7048 Acked Read 1 Done Sent Done false
23970 7049 Acked Write 2 Done Sent Done false
23971 7050 Acked Read 3 Done Sent Done false
23972 7051 Acked Write 3 Done Sent Done false
23973 7052 Acked Write 4 Done Sent Done false
23974 7053 Acked Write 1 Done Sent Done false
23975 7054 Acked Read 3 Done Sent Done false
23976 7055 Acked Write 4 Done Sent Done false
23977 7056 Acked Flush 0 Done Sent Done false
23978 7057 Acked Flush 0 Done Sent Done false
23979 7058 Acked Read 4 Done Sent Done false
23980 7059 Acked Read 4 Done Sent Done false
23981 7060 Acked Read 5 Done Sent Done false
23982 7061 Acked Write 4 Done Sent Done false
23983 7062 Acked Write 2 Done Sent Done false
23984 7063 Acked Write 7 Done Sent Done false
23985 7064 Acked Read 10 Done Sent Done false
23986 7065 Acked Write 8 Done Sent Done false
23987 7066 Acked Write 10 Done Sent Done false
23988 7067 Acked Read 5 Done Sent Done false
23989 7068 Acked Write 4 Done Sent Done false
23990 7069 Acked Read 4 Done Sent Done false
23991 7070 Acked Write 9 Done Sent Done false
23992 7071 Acked Write 10 Done Sent Done false
23993 7072 Acked Read 3 Done Sent Done false
23994 7073 Acked Read 3 Done Sent Done false
23995 7074 Acked Read 6 Done Sent Done false
23996 7075 Acked Write 4 Done Sent Done false
23997 7076 Acked Read 4 Done Sent Done false
23998 7077 Acked Read 5 Done Sent Done false
23999 7078 Acked Flush 0 Done Sent Done false
24000 7079 Acked Read 6 Done Sent Done false
24001 7080 Acked Write 7 Done Sent Done false
24002 7081 Acked Read 4 Done Sent Done false
24003 7082 Acked Read 2 Done Sent Done false
24004 7083 Acked Read 6 Done Sent Done false
24005 7084 Acked Read 5 Done Sent Done false
24006 7085 Acked Flush 0 Done Sent Done false
24007 7086 Acked Write 8 Done Sent Done false
24008 7087 Acked Read 8 Done Sent Done false
24009 7088 Acked Read 10 Done Sent Done false
24010 7089 Acked Write 7 Done Sent Done false
24011 7090 Acked Read 4 Done Sent Done false
24012 7091 Acked Read 7 Done Sent Done false
24013 7092 Acked Read 3 Done Sent Done false
24014 7093 Acked Write 9 Done Sent Done false
24015 7094 Acked Write 6 Done Sent Done false
24016 7095 Acked Read 7 Done Sent Done false
24017 7096 Acked Write 4 Done Sent Done false
24018 7097 Acked Read 10 Done Sent Done false
24019 7098 Acked Write 3 Done Sent Done false
24020 7099 Acked Flush 0 Done Sent Done false
24021 7100 Acked Write 4 Done Sent Done false
24022 7101 Acked Read 2 Done Sent Done false
24023 7102 Acked Read 8 Done Sent Done false
24024 7103 Acked Read 7 Done Sent Done false
24025 7104 Acked Write 9 Done Sent Done false
24026 7105 Acked Write 3 Done Sent Done false
24027 7106 Acked Write 8 Done Sent Done false
24028 7107 Acked Read 9 Done Sent Done false
24029 7108 Acked Read 10 Done Sent Done false
24030 7109 Acked Read 7 Done Sent Done false
24031 7110 Acked Write 4 Done Sent Done false
24032 7111 Acked Write 3 Done Sent Done false
24033 7112 Acked Write 1 Done Sent Done false
24034 7113 Acked Write 5 Done Sent Done false
24035 7114 Acked Write 10 Done Sent Done false
24036 7115 Acked Read 3 Done Sent Done false
24037 7116 Acked Write 5 Done Sent Done false
24038 7117 Acked Read 1 Done Sent Done false
24039 7118 Acked Write 3 Done Sent Done false
24040 7119 Acked Write 4 Done Sent Done false
24041 7120 Acked Write 4 Done Sent Done false
24042 7121 Acked Write 2 Done Sent Done false
24043 7122 Acked Flush 0 Done Sent Done false
24044 7123 Acked Read 6 Done Sent Done false
24045 7124 Acked Write 4 Done Sent Done false
24046 7125 Acked Read 10 Done Sent Done false
24047 7126 Acked Write 10 Done Sent Done false
24048 7127 Acked Read 1 Done Sent Done false
24049 7128 Acked Write 7 Done Sent Done false
24050 7129 Acked Read 3 Done Sent Done false
24051 7130 Acked Write 8 Done Sent Done false
24052 7131 Acked Write 7 Done Sent Done false
24053 7132 Acked Write 10 Done Sent Done false
24054 7133 Acked Write 8 Done Sent Done false
24055 7134 Acked Write 1 Done Sent Done false
24056 7135 Acked Read 3 Done Sent Done false
24057 7136 Acked Write 7 Done Sent Done false
24058 7137 Acked Read 8 Done Sent Done false
24059 7138 Acked Write 7 Done Sent Done false
24060 7139 Acked Write 3 Done Sent Done false
24061 7140 Acked Write 5 Done Sent Done false
24062 7141 Acked Read 7 Done Sent Done false
24063 7142 Acked Read 9 Done Sent Done false
24064 7143 Acked Read 3 Done Sent Done false
24065 7144 Acked Read 7 Done Sent Done false
24066 7145 Acked Read 2 Done Sent Done false
24067 7146 Acked Write 2 Done Sent Done false
24068 7147 Acked Read 3 Done Sent Done false
24069 7148 Acked Flush 0 Done Sent Done false
24070 7149 Acked Flush 0 Done Sent Done false
24071 7150 Acked Read 8 Done Sent Done false
24072 7151 Acked Write 2 Done Sent Done false
24073 7152 Acked Flush 0 Done Sent Done false
24074 7153 Acked Write 2 Done Sent Done false
24075 7154 Acked Read 2 Done Sent Done false
24076 7155 Acked Read 10 Done Sent Done false
24077 7156 Acked Flush 0 Done Sent Done false
24078 7157 Acked Read 7 Done Sent Done false
24079 7158 Acked Read 2 Done Sent Done false
24080 7159 Acked Read 4 Done Sent Done false
24081 7160 Acked Write 8 Done Sent Done false
24082 7161 Acked Read 5 Done Sent Done false
24083 7162 Acked Read 7 Done Sent Done false
24084 7163 Acked Flush 0 Done Sent Done false
24085 7164 Acked Read 2 Done Sent Done false
24086 7165 Acked Write 1 Done Sent Done false
24087 7166 Acked Read 7 Done Sent Done false
24088 7167 Acked Read 4 Done Sent Done false
24089 7168 Acked Read 7 Done Sent Done false
24090 7169 Acked Write 4 Done Sent Done false
24091 7170 Acked Read 1 Done Sent Done false
24092 7171 Acked Write 7 Done Sent Done false
24093 7172 Acked Write 6 Done Sent Done false
24094 7173 Acked Read 9 Done Sent Done false
24095 7174 Acked Write 8 Done Sent Done false
24096 7175 Acked Read 6 Done Sent Done false
24097 7176 Acked Write 5 Done Sent Done false
24098 7177 Acked Write 3 Done Sent Done false
24099 7178 Acked Read 3 Done Sent Done false
24100 7179 Acked Write 3 Done Sent Done false
24101 7180 Acked Read 9 Done Sent Done false
24102 7181 Acked Write 1 Done Sent Done false
24103 7182 Acked Read 6 Done Sent Done false
24104 7183 Acked Write 1 Done Sent Done false
24105 7184 Acked Write 3 Done Sent Done false
24106 7185 Acked Read 2 Done Sent Done false
24107 7186 Acked Write 8 Done Sent Done false
24108 7187 Acked Read 2 Done Sent Done false
24109 7188 Acked Read 7 Done Sent Done false
24110 7189 Acked Read 2 Done Sent Done false
24111 7190 Acked Write 8 Done Sent Done false
24112 7191 Acked Read 3 Done Sent Done false
24113 7192 Acked Read 9 Done Sent Done false
24114 7193 Acked Flush 0 Done Sent Done false
24115 7194 Acked Write 4 Done Sent Done false
24116 7195 Acked Write 9 Done Sent Done false
24117 7196 Acked Write 1 Done Sent Done false
24118 7197 Acked Read 3 Done Sent Done false
24119 7198 Acked Write 7 Done Sent Done false
24120 7199 Acked Read 5 Done Sent Done false
24121 7200 Acked Write 7 Done Sent Done false
24122 7201 Acked Read 9 Done Sent Done false
24123 7202 Acked Read 8 Done Sent Done false
24124 7203 Acked Read 1 Done Sent Done false
24125 7204 Acked Read 6 Done Sent Done false
24126 7205 Acked Write 5 Done Sent Done false
24127 7206 Acked Read 1 Done Sent Done false
24128 7207 Acked Write 5 Done Sent Done false
24129 7208 Acked Write 6 Done Sent Done false
24130 7209 Acked Flush 0 Done Sent Done false
24131 7210 Acked Write 3 Done Sent Done false
24132 7211 Acked Read 5 Done Sent Done false
24133 7212 Acked Flush 0 Done Sent Done false
24134 7213 Acked Write 9 Done Sent Done false
24135 7214 Acked Read 4 Done Sent Done false
24136 7215 Acked Read 3 Done Sent Done false
24137 7216 Acked Read 7 Done Sent Done false
24138 7217 Acked Read 3 Done Sent Done false
24139 7218 Acked Read 8 Done Sent Done false
24140 7219 Acked Write 4 Done Sent Done false
24141 7220 Acked Write 8 Done Sent Done false
24142 7221 Acked Write 8 Done Sent Done false
24143 7222 Acked Write 6 Done Sent Done false
24144 7223 Acked Write 1 Done Sent Done false
24145 7224 Acked Write 4 Done Sent Done false
24146 7225 Acked Read 1 Done Sent Done false
24147 7226 Acked Write 9 Done Sent Done false
24148 7227 Acked Read 7 Done Sent Done false
24149 7228 Acked Flush 0 Done Sent Done false
24150 7229 Acked Write 7 Done Sent Done false
24151 7230 Acked Read 2 Done Sent Done false
24152 7231 Acked Read 8 Done Sent Done false
24153 7232 Acked Read 3 Done Sent Done false
24154 7233 Acked Read 7 Done Sent Done false
24155 7234 Acked Read 3 Done Sent Done false
24156 7235 Acked Write 9 Done Sent Done false
24157 7236 Acked Read 9 Done Sent Done false
24158 7237 Acked Read 10 Done Sent Done false
24159 7238 Acked Read 7 Done Sent Done false
24160 7239 Acked Read 2 Done Sent Done false
24161 7240 Acked Flush 0 Done Sent Done false
24162 7241 Acked Write 7 Done Sent Done false
24163 7242 Acked Read 9 Done Sent Done false
24164 7243 Acked Read 10 Done Sent Done false
24165 7244 Acked Write 4 Done Sent Done false
24166 7245 Acked Write 8 Done Sent Done false
24167 7246 Acked Flush 0 Done Sent Done false
24168 7247 Acked Read 7 Done Sent Done false
24169 7248 Acked Read 5 Done Sent Done false
24170 7249 Acked Read 8 Done Sent Done false
24171 7250 Acked Write 6 Done Sent Done false
24172 7251 Acked Read 9 Done Sent Done false
24173 7252 Acked Read 1 Done Sent Done false
24174 7253 Acked Write 2 Done Sent Done false
24175 7254 Acked Write 1 Done Sent Done false
24176 7255 Acked Read 1 Done Sent Done false
24177 7256 Acked Read 6 Done Sent Done false
24178 7257 Acked Write 5 Done Sent Done false
24179 7258 Acked Write 1 Done Sent Done false
24180 7259 Acked Read 7 Done Sent Done false
24181 7260 Acked Write 10 Done Sent Done false
24182 7261 Acked Read 8 Done Sent Done false
24183 7262 Acked Read 2 Done Sent Done false
24184 7263 Acked Read 4 Done Sent Done false
24185 7264 Acked Read 3 Done Sent Done false
24186 7265 Acked Write 5 Done Sent Done false
24187 7266 Acked Write 6 Done Sent Done false
24188 7267 Acked Write 6 Done Sent Done false
24189 7268 Acked Read 3 Done Sent Done false
24190 7269 Acked Read 3 Done Sent Done false
24191 7270 Acked Write 10 Done Sent Done false
24192 7271 Acked Write 7 Done Sent Done false
24193 7272 Acked Read 2 Done Sent Done false
24194 7273 Acked Write 4 Done Sent Done false
24195 7274 Acked Read 3 Done Sent Done false
24196 7275 Acked Write 3 Done Sent Done false
24197 7276 Acked Write 5 Done Sent Done false
24198 7277 Acked Write 7 Done Sent Done false
24199 7278 Acked Read 1 Done Sent Done false
24200 7279 Acked Read 3 Done Sent Done false
24201 7280 Acked Write 5 Done Sent Done false
24202 7281 Acked Write 8 Done Sent Done false
24203 7282 Acked Write 2 Done Sent Done false
24204 7283 Acked Write 8 Done Sent Done false
24205 7284 Acked Read 10 Done Sent Done false
24206 7285 Acked Read 9 Done Sent Done false
24207 7286 Acked Read 9 Done Sent Done false
24208 7287 Acked Read 2 Done Sent Done false
24209 7288 Acked Flush 0 Done Sent Done false
24210 7289 Acked Read 3 Done Sent Done false
24211 7290 Acked Write 6 Done Sent Done false
24212 7291 Acked Read 8 Done Sent Done false
24213 7292 Acked Read 1 Done Sent Done false
24214 7293 Acked Write 10 Done Sent Done false
24215 7294 Acked Read 5 Done Sent Done false
24216 7295 Acked Read 6 Done Sent Done false
24217 7296 Acked Write 4 Done Sent Done false
24218 7297 Acked Write 7 Done Sent Done false
24219 7298 Acked Write 10 Done Sent Done false
24220 7299 Acked Write 1 Done Sent Done false
24221 7300 Acked Write 4 Done Sent Done false
24222 7301 Acked Write 7 Done Sent Done false
24223 7302 Acked Read 1 Done Sent Done false
24224 7303 Acked Read 1 Done Sent Done false
24225 7304 Acked Write 10 Done Sent Done false
24226 7305 Acked Read 7 Done Sent Done false
24227 7306 Acked Write 10 Done Sent Done false
24228 7307 Acked Write 5 Done Sent Done false
24229 7308 Acked Write 5 Done Sent Done false
24230 7309 Acked Read 5 Done Sent Done false
24231 7310 Acked Read 4 Done Sent Done false
24232 7311 Acked Read 8 Done Sent Done false
24233 7312 Acked Read 7 Done Sent Done false
24234 7313 Acked Write 4 Done Sent Done false
24235 7314 Acked Write 4 Done Sent Done false
24236 7315 Acked Write 9 Done Sent Done false
24237 7316 Acked Read 3 Done Sent Done false
24238 7317 Acked Read 5 Done Sent Done false
24239 7318 Acked Write 10 Done Sent Done false
24240 7319 Acked Write 7 Done Sent Done false
24241 7320 Acked Read 6 Done Sent Done false
24242 7321 Acked Read 5 Done Sent Done false
24243 7322 Acked Write 10 Done Sent Done false
24244 7323 Acked Read 8 Done Sent Done false
24245 7324 Acked Read 9 Done Sent Done false
24246 7325 Acked Read 10 Done Sent Done false
24247 7326 Acked Read 5 Done Sent Done false
24248 7327 Acked Read 1 Done Sent Done false
24249 7328 Acked Read 6 Done Sent Done false
24250 7329 Acked Write 8 Done Sent Done false
24251 7330 Acked Read 4 Done Sent Done false
24252 7331 Acked Write 1 Done Sent Done false
24253 7332 Acked Read 6 Done Sent Done false
24254 7333 Acked Flush 0 Done Sent Done false
24255 7334 Acked Write 8 Done Sent Done false
24256 7335 Acked Write 4 Done Sent Done false
24257 7336 Acked Read 8 Done Sent Done false
24258 7337 Acked Read 1 Done Sent Done false
24259 7338 Acked Flush 0 Done Sent Done false
24260 STATES DS:0 DS:1 DS:2 TOTAL
24261 Sent 0 301 0 301
24262 Done 301 0 301 602
24263 Skipped 0 0 0 0
24264 Error 0 0 0 0
24265 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
24266 Downstairs last five completed: 7037 7036 7035 7034 7033
24267 Upstairs last five completed: 7338 7337 7336 7335 7334
24268 CLIENT: Up:0 ds:301 act:2
242692025-03-07T18:17:50.473ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
242702025-03-07T18:17:51.475ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24271 [8820] got start action
24272 [8820] I am going to restart
24273 Starting downstairs at port 8820
24274 Make output file at "/tmp/dsc/downstairs-8820.txt"
24275 [8820][1] reports Starting
24276 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1229)
24277 [8820][1] reports Running
242782025-03-07T18:17:52.478ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24279 ----------------------------------------------------------------
24280 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24281 States: Active Offline Active
24282 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24283 7038 Acked Flush 0 Done Sent Done false
24284 7039 Acked Read 3 Done Sent Done false
24285 7040 Acked Write 4 Done Sent Done false
24286 7041 Acked Read 9 Done Sent Done false
24287 7042 Acked Read 10 Done Sent Done false
24288 7043 Acked Write 2 Done Sent Done false
24289 7044 Acked Write 2 Done Sent Done false
24290 7045 Acked Write 6 Done Sent Done false
24291 7046 Acked Read 6 Done Sent Done false
24292 7047 Acked Read 10 Done Sent Done false
24293 7048 Acked Read 1 Done Sent Done false
24294 7049 Acked Write 2 Done Sent Done false
24295 7050 Acked Read 3 Done Sent Done false
24296 7051 Acked Write 3 Done Sent Done false
24297 7052 Acked Write 4 Done Sent Done false
24298 7053 Acked Write 1 Done Sent Done false
24299 7054 Acked Read 3 Done Sent Done false
24300 7055 Acked Write 4 Done Sent Done false
24301 7056 Acked Flush 0 Done Sent Done false
24302 7057 Acked Flush 0 Done Sent Done false
24303 7058 Acked Read 4 Done Sent Done false
24304 7059 Acked Read 4 Done Sent Done false
24305 7060 Acked Read 5 Done Sent Done false
24306 7061 Acked Write 4 Done Sent Done false
24307 7062 Acked Write 2 Done Sent Done false
24308 7063 Acked Write 7 Done Sent Done false
24309 7064 Acked Read 10 Done Sent Done false
24310 7065 Acked Write 8 Done Sent Done false
24311 7066 Acked Write 10 Done Sent Done false
24312 7067 Acked Read 5 Done Sent Done false
24313 7068 Acked Write 4 Done Sent Done false
24314 7069 Acked Read 4 Done Sent Done false
24315 7070 Acked Write 9 Done Sent Done false
24316 7071 Acked Write 10 Done Sent Done false
24317 7072 Acked Read 3 Done Sent Done false
24318 7073 Acked Read 3 Done Sent Done false
24319 7074 Acked Read 6 Done Sent Done false
24320 7075 Acked Write 4 Done Sent Done false
24321 7076 Acked Read 4 Done Sent Done false
24322 7077 Acked Read 5 Done Sent Done false
24323 7078 Acked Flush 0 Done Sent Done false
24324 7079 Acked Read 6 Done Sent Done false
24325 7080 Acked Write 7 Done Sent Done false
24326 7081 Acked Read 4 Done Sent Done false
24327 7082 Acked Read 2 Done Sent Done false
24328 7083 Acked Read 6 Done Sent Done false
24329 7084 Acked Read 5 Done Sent Done false
24330 7085 Acked Flush 0 Done Sent Done false
24331 7086 Acked Write 8 Done Sent Done false
24332 7087 Acked Read 8 Done Sent Done false
24333 7088 Acked Read 10 Done Sent Done false
24334 7089 Acked Write 7 Done Sent Done false
24335 7090 Acked Read 4 Done Sent Done false
24336 7091 Acked Read 7 Done Sent Done false
24337 7092 Acked Read 3 Done Sent Done false
24338 7093 Acked Write 9 Done Sent Done false
24339 7094 Acked Write 6 Done Sent Done false
24340 7095 Acked Read 7 Done Sent Done false
24341 7096 Acked Write 4 Done Sent Done false
24342 7097 Acked Read 10 Done Sent Done false
24343 7098 Acked Write 3 Done Sent Done false
24344 7099 Acked Flush 0 Done Sent Done false
24345 7100 Acked Write 4 Done Sent Done false
24346 7101 Acked Read 2 Done Sent Done false
24347 7102 Acked Read 8 Done Sent Done false
24348 7103 Acked Read 7 Done Sent Done false
24349 7104 Acked Write 9 Done Sent Done false
24350 7105 Acked Write 3 Done Sent Done false
24351 7106 Acked Write 8 Done Sent Done false
24352 7107 Acked Read 9 Done Sent Done false
24353 7108 Acked Read 10 Done Sent Done false
24354 7109 Acked Read 7 Done Sent Done false
24355 7110 Acked Write 4 Done Sent Done false
24356 7111 Acked Write 3 Done Sent Done false
24357 7112 Acked Write 1 Done Sent Done false
24358 7113 Acked Write 5 Done Sent Done false
24359 7114 Acked Write 10 Done Sent Done false
24360 7115 Acked Read 3 Done Sent Done false
24361 7116 Acked Write 5 Done Sent Done false
24362 7117 Acked Read 1 Done Sent Done false
24363 7118 Acked Write 3 Done Sent Done false
24364 7119 Acked Write 4 Done Sent Done false
24365 7120 Acked Write 4 Done Sent Done false
24366 7121 Acked Write 2 Done Sent Done false
24367 7122 Acked Flush 0 Done Sent Done false
24368 7123 Acked Read 6 Done Sent Done false
24369 7124 Acked Write 4 Done Sent Done false
24370 7125 Acked Read 10 Done Sent Done false
24371 7126 Acked Write 10 Done Sent Done false
24372 7127 Acked Read 1 Done Sent Done false
24373 7128 Acked Write 7 Done Sent Done false
24374 7129 Acked Read 3 Done Sent Done false
24375 7130 Acked Write 8 Done Sent Done false
24376 7131 Acked Write 7 Done Sent Done false
24377 7132 Acked Write 10 Done Sent Done false
24378 7133 Acked Write 8 Done Sent Done false
24379 7134 Acked Write 1 Done Sent Done false
24380 7135 Acked Read 3 Done Sent Done false
24381 7136 Acked Write 7 Done Sent Done false
24382 7137 Acked Read 8 Done Sent Done false
24383 7138 Acked Write 7 Done Sent Done false
24384 7139 Acked Write 3 Done Sent Done false
24385 7140 Acked Write 5 Done Sent Done false
24386 7141 Acked Read 7 Done Sent Done false
24387 7142 Acked Read 9 Done Sent Done false
24388 7143 Acked Read 3 Done Sent Done false
24389 7144 Acked Read 7 Done Sent Done false
24390 7145 Acked Read 2 Done Sent Done false
24391 7146 Acked Write 2 Done Sent Done false
24392 7147 Acked Read 3 Done Sent Done false
24393 7148 Acked Flush 0 Done Sent Done false
24394 7149 Acked Flush 0 Done Sent Done false
24395 7150 Acked Read 8 Done Sent Done false
24396 7151 Acked Write 2 Done Sent Done false
24397 7152 Acked Flush 0 Done Sent Done false
24398 7153 Acked Write 2 Done Sent Done false
24399 7154 Acked Read 2 Done Sent Done false
24400 7155 Acked Read 10 Done Sent Done false
24401 7156 Acked Flush 0 Done Sent Done false
24402 7157 Acked Read 7 Done Sent Done false
24403 7158 Acked Read 2 Done Sent Done false
24404 7159 Acked Read 4 Done Sent Done false
24405 7160 Acked Write 8 Done Sent Done false
24406 7161 Acked Read 5 Done Sent Done false
24407 7162 Acked Read 7 Done Sent Done false
24408 7163 Acked Flush 0 Done Sent Done false
24409 7164 Acked Read 2 Done Sent Done false
24410 7165 Acked Write 1 Done Sent Done false
24411 7166 Acked Read 7 Done Sent Done false
24412 7167 Acked Read 4 Done Sent Done false
24413 7168 Acked Read 7 Done Sent Done false
24414 7169 Acked Write 4 Done Sent Done false
24415 7170 Acked Read 1 Done Sent Done false
24416 7171 Acked Write 7 Done Sent Done false
24417 7172 Acked Write 6 Done Sent Done false
24418 7173 Acked Read 9 Done Sent Done false
24419 7174 Acked Write 8 Done Sent Done false
24420 7175 Acked Read 6 Done Sent Done false
24421 7176 Acked Write 5 Done Sent Done false
24422 7177 Acked Write 3 Done Sent Done false
24423 7178 Acked Read 3 Done Sent Done false
24424 7179 Acked Write 3 Done Sent Done false
24425 7180 Acked Read 9 Done Sent Done false
24426 7181 Acked Write 1 Done Sent Done false
24427 7182 Acked Read 6 Done Sent Done false
24428 7183 Acked Write 1 Done Sent Done false
24429 7184 Acked Write 3 Done Sent Done false
24430 7185 Acked Read 2 Done Sent Done false
24431 7186 Acked Write 8 Done Sent Done false
24432 7187 Acked Read 2 Done Sent Done false
24433 7188 Acked Read 7 Done Sent Done false
24434 7189 Acked Read 2 Done Sent Done false
24435 7190 Acked Write 8 Done Sent Done false
24436 7191 Acked Read 3 Done Sent Done false
24437 7192 Acked Read 9 Done Sent Done false
24438 7193 Acked Flush 0 Done Sent Done false
24439 7194 Acked Write 4 Done Sent Done false
24440 7195 Acked Write 9 Done Sent Done false
24441 7196 Acked Write 1 Done Sent Done false
24442 7197 Acked Read 3 Done Sent Done false
24443 7198 Acked Write 7 Done Sent Done false
24444 7199 Acked Read 5 Done Sent Done false
24445 7200 Acked Write 7 Done Sent Done false
24446 7201 Acked Read 9 Done Sent Done false
24447 7202 Acked Read 8 Done Sent Done false
24448 7203 Acked Read 1 Done Sent Done false
24449 7204 Acked Read 6 Done Sent Done false
24450 7205 Acked Write 5 Done Sent Done false
24451 7206 Acked Read 1 Done Sent Done false
24452 7207 Acked Write 5 Done Sent Done false
24453 7208 Acked Write 6 Done Sent Done false
24454 7209 Acked Flush 0 Done Sent Done false
24455 7210 Acked Write 3 Done Sent Done false
24456 7211 Acked Read 5 Done Sent Done false
24457 7212 Acked Flush 0 Done Sent Done false
24458 7213 Acked Write 9 Done Sent Done false
24459 7214 Acked Read 4 Done Sent Done false
24460 7215 Acked Read 3 Done Sent Done false
24461 7216 Acked Read 7 Done Sent Done false
24462 7217 Acked Read 3 Done Sent Done false
24463 7218 Acked Read 8 Done Sent Done false
24464 7219 Acked Write 4 Done Sent Done false
24465 7220 Acked Write 8 Done Sent Done false
24466 7221 Acked Write 8 Done Sent Done false
24467 7222 Acked Write 6 Done Sent Done false
24468 7223 Acked Write 1 Done Sent Done false
24469 7224 Acked Write 4 Done Sent Done false
24470 7225 Acked Read 1 Done Sent Done false
24471 7226 Acked Write 9 Done Sent Done false
24472 7227 Acked Read 7 Done Sent Done false
24473 7228 Acked Flush 0 Done Sent Done false
24474 7229 Acked Write 7 Done Sent Done false
24475 7230 Acked Read 2 Done Sent Done false
24476 7231 Acked Read 8 Done Sent Done false
24477 7232 Acked Read 3 Done Sent Done false
24478 7233 Acked Read 7 Done Sent Done false
24479 7234 Acked Read 3 Done Sent Done false
24480 7235 Acked Write 9 Done Sent Done false
24481 7236 Acked Read 9 Done Sent Done false
24482 7237 Acked Read 10 Done Sent Done false
24483 7238 Acked Read 7 Done Sent Done false
24484 7239 Acked Read 2 Done Sent Done false
24485 7240 Acked Flush 0 Done Sent Done false
24486 7241 Acked Write 7 Done Sent Done false
24487 7242 Acked Read 9 Done Sent Done false
24488 7243 Acked Read 10 Done Sent Done false
24489 7244 Acked Write 4 Done Sent Done false
24490 7245 Acked Write 8 Done Sent Done false
24491 7246 Acked Flush 0 Done Sent Done false
24492 7247 Acked Read 7 Done Sent Done false
24493 7248 Acked Read 5 Done Sent Done false
24494 7249 Acked Read 8 Done Sent Done false
24495 7250 Acked Write 6 Done Sent Done false
24496 7251 Acked Read 9 Done Sent Done false
24497 7252 Acked Read 1 Done Sent Done false
24498 7253 Acked Write 2 Done Sent Done false
24499 7254 Acked Write 1 Done Sent Done false
24500 7255 Acked Read 1 Done Sent Done false
24501 7256 Acked Read 6 Done Sent Done false
24502 7257 Acked Write 5 Done Sent Done false
24503 7258 Acked Write 1 Done Sent Done false
24504 7259 Acked Read 7 Done Sent Done false
24505 7260 Acked Write 10 Done Sent Done false
24506 7261 Acked Read 8 Done Sent Done false
24507 7262 Acked Read 2 Done Sent Done false
24508 7263 Acked Read 4 Done Sent Done false
24509 7264 Acked Read 3 Done Sent Done false
24510 7265 Acked Write 5 Done Sent Done false
24511 7266 Acked Write 6 Done Sent Done false
24512 7267 Acked Write 6 Done Sent Done false
24513 7268 Acked Read 3 Done Sent Done false
24514 7269 Acked Read 3 Done Sent Done false
24515 7270 Acked Write 10 Done Sent Done false
24516 7271 Acked Write 7 Done Sent Done false
24517 7272 Acked Read 2 Done Sent Done false
24518 7273 Acked Write 4 Done Sent Done false
24519 7274 Acked Read 3 Done Sent Done false
24520 7275 Acked Write 3 Done Sent Done false
24521 7276 Acked Write 5 Done Sent Done false
24522 7277 Acked Write 7 Done Sent Done false
24523 7278 Acked Read 1 Done Sent Done false
24524 7279 Acked Read 3 Done Sent Done false
24525 7280 Acked Write 5 Done Sent Done false
24526 7281 Acked Write 8 Done Sent Done false
24527 7282 Acked Write 2 Done Sent Done false
24528 7283 Acked Write 8 Done Sent Done false
24529 7284 Acked Read 10 Done Sent Done false
24530 7285 Acked Read 9 Done Sent Done false
24531 7286 Acked Read 9 Done Sent Done false
24532 7287 Acked Read 2 Done Sent Done false
24533 7288 Acked Flush 0 Done Sent Done false
24534 7289 Acked Read 3 Done Sent Done false
24535 7290 Acked Write 6 Done Sent Done false
24536 7291 Acked Read 8 Done Sent Done false
24537 7292 Acked Read 1 Done Sent Done false
24538 7293 Acked Write 10 Done Sent Done false
24539 7294 Acked Read 5 Done Sent Done false
24540 7295 Acked Read 6 Done Sent Done false
24541 7296 Acked Write 4 Done Sent Done false
24542 7297 Acked Write 7 Done Sent Done false
24543 7298 Acked Write 10 Done Sent Done false
24544 7299 Acked Write 1 Done Sent Done false
24545 7300 Acked Write 4 Done Sent Done false
24546 7301 Acked Write 7 Done Sent Done false
24547 7302 Acked Read 1 Done Sent Done false
24548 7303 Acked Read 1 Done Sent Done false
24549 7304 Acked Write 10 Done Sent Done false
24550 7305 Acked Read 7 Done Sent Done false
24551 7306 Acked Write 10 Done Sent Done false
24552 7307 Acked Write 5 Done Sent Done false
24553 7308 Acked Write 5 Done Sent Done false
24554 7309 Acked Read 5 Done Sent Done false
24555 7310 Acked Read 4 Done Sent Done false
24556 7311 Acked Read 8 Done Sent Done false
24557 7312 Acked Read 7 Done Sent Done false
24558 7313 Acked Write 4 Done Sent Done false
24559 7314 Acked Write 4 Done Sent Done false
24560 7315 Acked Write 9 Done Sent Done false
24561 7316 Acked Read 3 Done Sent Done false
24562 7317 Acked Read 5 Done Sent Done false
24563 7318 Acked Write 10 Done Sent Done false
24564 7319 Acked Write 7 Done Sent Done false
24565 7320 Acked Read 6 Done Sent Done false
24566 7321 Acked Read 5 Done Sent Done false
24567 7322 Acked Write 10 Done Sent Done false
24568 7323 Acked Read 8 Done Sent Done false
24569 7324 Acked Read 9 Done Sent Done false
24570 7325 Acked Read 10 Done Sent Done false
24571 7326 Acked Read 5 Done Sent Done false
24572 7327 Acked Read 1 Done Sent Done false
24573 7328 Acked Read 6 Done Sent Done false
24574 7329 Acked Write 8 Done Sent Done false
24575 7330 Acked Read 4 Done Sent Done false
24576 7331 Acked Write 1 Done Sent Done false
24577 7332 Acked Read 6 Done Sent Done false
24578 7333 Acked Flush 0 Done Sent Done false
24579 7334 Acked Write 8 Done Sent Done false
24580 7335 Acked Write 4 Done Sent Done false
24581 7336 Acked Read 8 Done Sent Done false
24582 7337 Acked Read 1 Done Sent Done false
24583 7338 Acked Flush 0 Done Sent Done false
24584 STATES DS:0 DS:1 DS:2 TOTAL
24585 Sent 0 301 0 301
24586 Done 301 0 301 602
24587 Skipped 0 0 0 0
24588 Error 0 0 0 0
24589 Last Flush: Some(JobId(7338)) Some(JobId(7037)) Some(JobId(7338))
24590 Downstairs last five completed: 7037 7036 7035 7034 7033
24591 Upstairs last five completed: 7338 7337 7336 7335 7334
24592 CLIENT: Up:0 ds:301 act:2
245932025-03-07T18:17:54.481ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245942025-03-07T18:17:55.484ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245952025-03-07T18:17:55.642ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245962025-03-07T18:17:55.642ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46224) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245972025-03-07T18:17:55.643ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245982025-03-07T18:17:55.644ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
245992025-03-07T18:17:55.644ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246002025-03-07T18:17:55.644ZINFOcrucible: send last flush ID to this DS: Some(JobId(7037)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246012025-03-07T18:17:55.644ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246022025-03-07T18:17:55.644ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7037)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246032025-03-07T18:17:55.644ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7037)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24604 ----------------------------------------------------------------
24605 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24606 Downstairs last five completed: 7338 7337 7336 7335 7334
24607 Upstairs last five completed: 7338 7337 7336 7335 7334
24608 CLIENT: Up:0 ds:0 act:3
24609 Replay: All jobs finished
24610 Main task has work to do, go find it
24611 got dsc Stop(2)
24612 stop 2
24613 Mar 07 18:17:57.892 INFO request completed, latency_us[8830] Got stop action so:false kr:false
24614 : 316, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 9b32cbc8-0cb9-4a92-8fd2-0f94d7185b4c, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
246152025-03-07T18:17:57.893ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246162025-03-07T18:17:57.893ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246172025-03-07T18:17:57.893ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
246182025-03-07T18:17:57.893ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
24619 Mar 07 18:17:57.893 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/2, method: GET, req_id: abc30054-e255-4600-9ab3-daf1e47049a4, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
24620 [8830] Exited with: ExitStatus(unix_wait_status(9))
24621 [8830][2] reports Exit
24622 Mar 07 18:18:01.896 INFO request completed, latency_us: 309, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9621f8b9-1465-4555-add1-01c9a62bc667, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
24623 001/300 Read block 1477 len 8192
24624 002/300 Read block 3411 len 32768
24625 003/300 Read block 36 len 20480
24626 004/300 Read block 3996 len 16384
24627 005/300 Read block 3965 len 4096
24628 006/300 Write block 1271 len 36864 data: 5 5 3 3 2 3 3 3 3
24629 007/300 Read block 2335 len 12288
24630 008/300 Write block 2720 len 28672 data: 5 5 5 4 4 5 7
24631 009/300 Read block 2428 len 8192
24632 010/300 Read block 3987 len 24576
24633 011/300 Write block 3827 len 28672 data: 6 6 8 8 7 5 4
24634 012/300 Flush
24635 013/300 Flush
24636 014/300 Write block 3293 len 20480 data: 6 6 5 5 6
24637 015/300 Read block 1342 len 40960
24638 016/300 Read block 3254 len 8192
24639 017/300 Read block 1383 len 32768
24640 018/300 Read block 1603 len 16384
24641 019/300 Write block 1135 len 24576 data: 4 4 3 5 6 6
24642 020/300 Read block 210 len 36864
24643 021/300 Flush
24644 022/300 Write block 622 len 32768 data: 7 7 7 6 7 7 5 6
24645 023/300 Read block 86 len 28672
24646 024/300 Write block 937 len 40960 data: 7 8 7 4 4 5 5 6 6 5
24647 025/300 Read block 2836 len 28672
24648 026/300 Write block 1677 len 8192 data: 4 3
24649 027/300 Write block 974 len 24576 data: 3 2 2 3 3 4
24650 028/300 Read block 676 len 12288
24651 029/300 Read block 4200 len 16384
24652 030/300 Read block 4283 len 16384
24653 031/300 Flush
24654 032/300 Read block 2245 len 32768
24655 033/300 Write block 767 len 32768 data: 4 4 4 4 4 6 6 6
24656 034/300 Flush
24657 035/300 Write block 649 len 28672 data: 6 6 5 4 3 3 4
24658 036/300 Write block 3947 len 24576 data: 6 8 8 6 5 5
24659 037/300 Write block 4438 len 8192 data: 6 5
24660 038/300 Read block 2120 len 4096
24661 039/300 Read block 2769 len 20480
24662 040/300 Read block 3639 len 24576
24663 041/300 Read block 3926 len 36864
24664 042/300 Flush
24665 043/300 Read block 3042 len 8192
24666 044/300 Write block 4804 len 24576 data: 2 2 2 3 3 3
24667 045/300 Flush
24668 046/300 Read block 3301 len 28672
24669 047/300 Write block 2141 len 12288 data: 3 4 4
24670 048/300 Write block 2134 len 8192 data: 5 5
24671 049/300 Read block 1555 len 8192
24672 050/300 Flush
24673 051/300 Read block 3791 len 28672
24674 052/300 Read block 1534 len 24576
24675 053/300 Read block 2758 len 32768
24676 054/300 Read block 437 len 8192
24677 055/300 Flush
24678 056/300 Write block 2376 len 16384 data: 6 5 5 4
24679 057/300 Write block 1379 len 36864 data: 4 3 3 3 3 3 4 4 4
24680 058/300 Read block 2563 len 28672
24681 059/300 Flush
24682 060/300 Read block 1334 len 28672
24683 061/300 Write block 1939 len 28672 data: 4 4 3 3 3 4 4
24684 062/300 Write block 723 len 12288 data: 4 5 5
24685 063/300 Read block 3065 len 16384
24686 064/300 Write block 4532 len 40960 data: 3 5 5 6 6 7 5 5 6 6
24687 065/300 Write block 2012 len 24576 data: 5 6 6 6 7 6
24688 066/300 Write block 1946 len 12288 data: 4 4 4
24689 067/300 Read block 328 len 28672
24690 068/300 Write block 4042 len 24576 data: 6 6 7 6 7 6
24691 069/300 Write block 4791 len 36864 data: 6 6 6 7 6 6 6 6 7
24692 070/300 Write block 2585 len 16384 data: 5 7 6 7
24693 071/300 Flush
24694 072/300 Write block 1716 len 32768 data: 7 7 8 9 8 8 6 5
24695 073/300 Read block 3480 len 4096
24696 074/300 Write block 4670 len 8192 data: 5 7
24697 075/300 Write block 2467 len 24576 data: 7 7 5 5 5 5
24698 076/300 Read block 4593 len 12288
24699 077/300 Read block 1030 len 12288
24700 078/300 Flush
24701 079/300 Write block 2659 len 24576 data: 10 8 9 9 9 8
24702 080/300 Write block 1658 len 4096 data: 5
24703 081/300 Read block 3638 len 24576
24704 082/300 Write block 865 len 40960 data: 3 4 5 5 5 6 6 3 3 2
24705 083/300 Flush
24706 084/300 Read block 2202 len 20480
24707 085/300 Flush
24708 086/300 Write block 4361 len 16384 data: 4 4 5 5
24709 087/300 Read block 4398 len 8192
24710 088/300 Write block 3666 len 4096 data: 7
24711 089/300 Read block 4506 len 20480
24712 090/300 Read block 2852 len 8192
24713 091/300 Write block 1777 len 8192 data: 5 4
24714 092/300 Read block 2016 len 16384
24715 093/300 Read block 3843 len 36864
24716 094/300 Read block 1408 len 28672
24717 095/300 Write block 1128 len 4096 data: 7
24718 096/300 Write block 1147 len 24576 data: 5 3 2 3 4 5
24719 097/300 Read block 1806 len 24576
24720 098/300 Read block 2265 len 20480
24721 099/300 Write block 1444 len 36864 data: 2 2 2 2 2 2 5 5 5
24722 100/300 Flush
24723 101/300 Write block 572 len 12288 data: 5 5 5
24724 102/300 Read block 1639 len 40960
24725 103/300 Write block 2308 len 40960 data: 4 4 4 4 4 4 6 4 3 4
24726 104/300 Write block 2038 len 40960 data: 3 3 3 3 4 4 6 7 8 9
24727 105/300 Write block 33 len 12288 data: 4 4 4
24728 106/300 Write block 560 len 8192 data: 3 4
24729 107/300 Read block 273 len 20480
24730 108/300 Read block 2458 len 16384
24731 109/300 Read block 2118 len 28672
24732 110/300 Write block 1625 len 8192 data: 8 8
24733 111/300 Write block 325 len 4096 data: 7
24734 112/300 Write block 3822 len 32768 data: 2 2 3 3 3 7 7 9
24735 113/300 Read block 1979 len 16384
24736 114/300 Read block 2259 len 24576
24737 115/300 Write block 940 len 12288 data: 5 5 6
24738 116/300 Flush
24739 117/300 Read block 4550 len 40960
24740 118/300 Read block 1456 len 24576
24741 119/300 Read block 1276 len 32768
24742 120/300 Write block 650 len 4096 data: 7
24743 121/300 Write block 226 len 4096 data: 6
24744 122/300 Write block 494 len 4096 data: 3
24745 123/300 Write block 4815 len 40960 data: 3 3 3 3 2 2 4 4 5 6
24746 124/300 Read block 472 len 4096
24747 125/300 Read block 1331 len 12288
24748 126/300 Read block 424 len 8192
24749 127/300 Read block 1045 len 36864
24750 128/300 Flush
24751 129/300 Write block 1423 len 16384 data: 5 4 3 3
24752 130/300 Read block 2976 len 20480
24753 131/300 Read block 65 len 12288
24754 132/300 Read block 3590 len 28672
24755 133/300 Flush
24756 134/300 Read block 3852 len 8192
24757 135/300 Read block 358 len 28672
24758 136/300 Write block 4635 len 12288 data: 5 4 5
24759 137/300 Write block 3819 len 16384 data: 4 4 4 3
24760 138/300 Read block 4805 len 36864
24761 139/300 Read block 3870 len 24576
24762 140/300 Write block 4761 len 36864 data: 3 3 4 3 6 6 6 6 5
24763 141/300 Read block 2825 len 28672
24764 142/300 Write block 2560 len 4096 data: 6
24765 143/300 Read block 747 len 36864
24766 144/300 Write block 3817 len 8192 data: 4 3
24767 145/300 Write block 349 len 8192 data: 8 8
24768 146/300 Flush
24769 147/300 Read block 173 len 12288
24770 148/300 Read block 3515 len 16384
24771 149/300 Flush
24772 150/300 Write block 2718 len 32768 data: 4 4 6 6 6 5 5 6
24773 151/300 Write block 4865 len 20480 data: 4 5 4 3 2
24774 152/300 Flush
24775 153/300 Write block 1205 len 4096 data: 4
24776 154/300 Flush
24777 155/300 Read block 3681 len 24576
24778 156/300 Write block 2125 len 24576 data: 6 6 5 3 5 5
24779 157/300 Read block 221 len 28672
24780 158/300 Read block 1726 len 28672
24781 159/300 Write block 2002 len 24576 data: 3 4 5 6 7 7
24782 160/300 Write block 504 len 28672 data: 5 4 4 3 3 4 4
24783 161/300 Read block 2455 len 32768
24784 162/300 Write block 850 len 24576 data: 3 4 3 3 2 2
24785 163/300 Read block 3127 len 16384
24786 164/300 Read block 2891 len 24576
24787 165/300 Flush
24788 166/300 Read block 3013 len 8192
24789 167/300 Write block 3376 len 28672 data: 5 6 6 6 5 5 5
24790 168/300 Read block 4762 len 24576
24791 169/300 Flush
24792 170/300 Read block 4134 len 36864
24793 171/300 Read block 1149 len 8192
24794 172/300 Read block 158 len 12288
24795 173/300 Read block 1384 len 40960
24796 174/300 Read block 4862 len 16384
24797 175/300 Read block 4092 len 4096
24798 176/300 Write block 2161 len 32768 data: 3 3 4 4 4 4 4 4
24799 177/300 Write block 4159 len 8192 data: 6 6
24800 178/300 Read block 847 len 36864
24801 179/300 Read block 4228 len 24576
24802 180/300 Read block 403 len 8192
24803 181/300 Write block 4367 len 28672 data: 6 6 6 5 6 5 6
24804 182/300 Flush
24805 183/300 Read block 3930 len 32768
24806 184/300 Read block 1364 len 40960
24807 185/300 Write block 2690 len 24576 data: 3 3 4 4 4 4
24808 186/300 Read block 3362 len 4096
24809 187/300 Write block 843 len 36864 data: 2 2 2 2 3 3 3 4 5
24810 188/300 Write block 855 len 8192 data: 3 2
24811 189/300 Write block 1743 len 32768 data: 6 6 8 6 6 6 7 6
24812 190/300 Write block 1960 len 4096 data: 4
24813 191/300 Flush
24814 192/300 Read block 4234 len 28672
24815 193/300 Write block 13 len 8192 data: 8 8
24816 194/300 Write block 486 len 28672 data: 4 4 7 6 6 8 6
24817 195/300 Read block 4015 len 32768
24818 196/300 Read block 23 len 4096
24819 197/300 Read block 3888 len 40960
24820 198/300 Read block 3844 len 16384
24821 199/300 Write block 1825 len 32768 data: 4 3 3 3 3 4 4 4
24822 200/300 Flush
24823 201/300 Read block 431 len 12288
24824 202/300 Write block 3463 len 40960 data: 4 2 2 3 3 4 4 4 5 5
24825 203/300 Write block 3329 len 40960 data: 4 4 3 4 4 4 4 5 5 5
24826 204/300 Write block 3638 len 12288 data: 7 6 6
24827 205/300 Read block 3098 len 20480
24828 206/300 Write block 235 len 40960 data: 5 5 4 3 4 4 3 4 5 7
24829 207/300 Read block 1728 len 4096
24830 208/300 Read block 2877 len 12288
24831 209/300 Read block 2634 len 12288
24832 210/300 Read block 3733 len 8192
24833 211/300 Read block 445 len 4096
24834 212/300 Write block 2232 len 16384 data: 3 5 5 5
24835 213/300 Read block 1889 len 8192
24836 214/300 Read block 3544 len 20480
24837 215/300 Write block 580 len 8192 data: 6 4
24838 216/300 Flush
24839 217/300 Read block 4681 len 8192
24840 218/300 Read block 912 len 4096
24841 219/300 Write block 2002 len 4096 data: 4
24842 220/300 Write block 2392 len 4096 data: 8
24843 221/300 Read block 4358 len 4096
24844 222/300 Read block 3233 len 8192
24845 223/300 Read block 772 len 40960
24846 224/300 Read block 3176 len 28672
24847 225/300 Write block 4113 len 36864 data: 8 6 5 4 4 4 4 6 6
24848 226/300 Write block 1831 len 20480 data: 5 5 3 4 3
24849 227/300 Flush
24850 228/300 Read block 1142 len 32768
24851 229/300 Write block 1832 len 20480 data: 6 4 5 4 4
24852 230/300 Write block 111 len 28672 data: 3 4 4 3 4 4 4
24853 231/300 Write block 2002 len 12288 data: 5 5 6
24854 232/300 Read block 1338 len 4096
24855 233/300 Read block 3313 len 28672
24856 234/300 Write block 186 len 24576 data: 4 4 5 7 8 7
24857 235/300 Flush
24858 236/300 Read block 3753 len 4096
24859 237/300 Read block 412 len 32768
24860 238/300 Read block 2112 len 32768
24861 239/300 Read block 1184 len 20480
24862 240/300 Write block 119 len 24576 data: 4 5 5 5 5 6
24863 241/300 Write block 3513 len 4096 data: 3
24864 242/300 Read block 1208 len 32768
24865 243/300 Read block 3631 len 32768
24866 244/300 Read block 328 len 12288
24867 245/300 Flush
24868 246/300 Read block 639 len 8192
24869 247/300 Read block 578 len 4096
24870 248/300 Read block 2855 len 28672
24871 249/300 Read block 4908 len 32768
24872 250/300 Write block 4510 len 8192 data: 5 5
24873 251/300 Write block 4854 len 24576 data: 6 4 4 5 4 3
24874 252/300 Read block 1322 len 20480
24875 253/300 Write block 2736 len 8192 data: 8 9
24876 254/300 Flush
24877 255/300 Read block 4120 len 40960
24878 256/300 Write block 68 len 4096 data: 6
24879 257/300 Write block 664 len 4096 data: 6
24880 258/300 Flush
24881 259/300 Read block 2785 len 32768
24882 260/300 Read block 637 len 4096
24883 261/300 Write block 4157 len 24576 data: 6 6 7 7 6 6
24884 262/300 Read block 545 len 16384
24885 263/300 Read block 1709 len 32768
24886 264/300 Write block 2052 len 24576 data: 8 6 5 5 5 5
24887 265/300 Read block 2211 len 8192
24888 266/300 Write block 2335 len 20480 data: 8 8 6 6 7
24889 267/300 Write block 1509 len 32768 data: 5 5 5 7 7 6 7 7
24890 268/300 Read block 727 len 36864
24891 269/300 Read block 4720 len 40960
24892 270/300 Read block 4680 len 40960
24893 271/300 Read block 4740 len 32768
24894 272/300 Write block 2810 len 40960 data: 5 5 5 4 3 2 4 4 4 6
24895 273/300 Read block 1984 len 16384
24896 274/300 Read block 1119 len 24576
24897 275/300 Write block 28 len 36864 data: 5 3 4 3 3 5 5 5 3
24898 276/300 Write block 2573 len 36864 data: 2 3 3 3 3 5 5 4 5
24899 277/300 Read block 4294 len 4096
24900 278/300 Write block 506 len 40960 data: 5 4 4 5 5 4 3 3 4 4
24901 279/300 Write block 4116 len 36864 data: 5 5 5 5 7 7 8 8 8
24902 280/300 Write block 860 len 12288 data: 3 3 3
24903 281/300 Write block 2410 len 4096 data: 6
24904 282/300 Read block 147 len 8192
24905 283/300 Read block 1765 len 4096
24906 284/300 Read block 3620 len 20480
24907 285/300 Write block 3488 len 28672 data: 3 3 3 3 5 3 4
24908 286/300 Read block 4988 len 4096
24909 287/300 Write block 4281 len 32768 data: 6 5 6 6 5 4 4 4
24910 288/300 Read block 122 len 16384
24911 289/300 Write block 4315 len 12288 data: 6 6 6
24912 290/300 Flush
24913 291/300 Write block 1922 len 16384 data: 11 10 9 7
24914 292/300 Read block 4614 len 24576
24915 293/300 Read block 4299 len 40960
24916 294/300 Write block 1596 len 8192 data: 5 6
24917 295/300 Write block 4330 len 20480 data: 7 8 8 8 7
24918 296/300 Read block 2456 len 4096
24919 297/300 Read block 2284 len 24576
24920 298/300 Read block 427 len 36864
24921 299/300 Read block 4011 len 4096
24922 300/300 Read block 4519 len 20480
24923 301/300 Flush
24924 Main task has work to do, go find it
24925 got dsc Start(2)
24926 start 2
24927 Mar 07 18:18:02.137 INFO request completed, latency_us: 179, response_code: 204, uri: /start/cid/2, method: POST, req_id: b66105d7-d995-454c-a4ff-1edbcac0347b, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
24928 Replay: started 2, returned:Ok(())
24929 ----------------------------------------------------------------
24930 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24931 States: Active Active Offline
24932 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24933 7339 Acked Read 2 Done Done Sent false
24934 7340 Acked Read 8 Done Done Sent false
24935 7341 Acked Read 5 Done Done Sent false
24936 7342 Acked Read 4 Done Done Sent false
24937 7343 Acked Read 1 Done Done Sent false
24938 7344 Acked Write 9 Done Done Sent false
24939 7345 Acked Read 3 Done Done Sent false
24940 7346 Acked Write 7 Done Done Sent false
24941 7347 Acked Read 2 Done Done Sent false
24942 7348 Acked Read 6 Done Done Sent false
24943 7349 Acked Write 7 Done Done Sent false
24944 7350 Acked Flush 0 Done Done Sent false
24945 7351 Acked Flush 0 Done Done Sent false
24946 7352 Acked Write 5 Done Done Sent false
24947 7353 Acked Read 10 Done Done Sent false
24948 7354 Acked Read 2 Done Done Sent false
24949 7355 Acked Read 8 Done Done Sent false
24950 7356 Acked Read 4 Done Done Sent false
24951 7357 Acked Write 6 Done Done Sent false
24952 7358 Acked Read 9 Done Done Sent false
24953 7359 Acked Flush 0 Done Done Sent false
24954 7360 Acked Write 8 Done Done Sent false
24955 7361 Acked Read 7 Done Done Sent false
24956 7362 Acked Write 10 Done Done Sent false
24957 7363 Acked Read 7 Done Done Sent false
24958 7364 Acked Write 2 Done Done Sent false
24959 7365 Acked Write 6 Done Done Sent false
24960 7366 Acked Read 3 Done Done Sent false
24961 7367 Acked Read 4 Done Done Sent false
24962 7368 Acked Read 4 Done Done Sent false
24963 7369 Acked Flush 0 Done Done Sent false
24964 7370 Acked Read 8 Done Done Sent false
24965 7371 Acked Write 8 Done Done Sent false
24966 7372 Acked Flush 0 Done Done Sent false
24967 7373 Acked Write 7 Done Done Sent false
24968 7374 Acked Write 6 Done Done Sent false
24969 7375 Acked Write 2 Done Done Sent false
24970 7376 Acked Read 1 Done Done Sent false
24971 7377 Acked Read 5 Done Done Sent false
24972 7378 Acked Read 6 Done Done Sent false
24973 7379 Acked Read 9 Done Done Sent false
24974 7380 Acked Flush 0 Done Done Sent false
24975 7381 Acked Read 2 Done Done Sent false
24976 7382 Acked Write 6 Done Done Sent false
24977 7383 Acked Flush 0 Done Done Sent false
24978 7384 Acked Read 7 Done Done Sent false
24979 7385 Acked Write 3 Done Done Sent false
24980 7386 Acked Write 2 Done Done Sent false
24981 7387 Acked Read 2 Done Done Sent false
24982 7388 Acked Flush 0 Done Done Sent false
24983 7389 Acked Read 7 Done Done Sent false
24984 7390 Acked Read 6 Done Done Sent false
24985 7391 Acked Read 8 Done Done Sent false
24986 7392 Acked Read 2 Done Done Sent false
24987 7393 Acked Flush 0 Done Done Sent false
24988 7394 Acked Write 4 Done Done Sent false
24989 7395 Acked Write 9 Done Done Sent false
24990 7396 Acked Read 7 Done Done Sent false
24991 7397 Acked Flush 0 Done Done Sent false
24992 7398 Acked Read 7 Done Done Sent false
24993 7399 Acked Write 7 Done Done Sent false
24994 7400 Acked Write 3 Done Done Sent false
24995 7401 Acked Read 4 Done Done Sent false
24996 7402 Acked Write 10 Done Done Sent false
24997 7403 Acked Write 6 Done Done Sent false
24998 7404 Acked Write 3 Done Done Sent false
24999 7405 Acked Read 7 Done Done Sent false
25000 7406 Acked Write 6 Done Done Sent false
25001 7407 Acked Write 9 Done Done Sent false
25002 7408 Acked Write 4 Done Done Sent false
25003 7409 Acked Flush 0 Done Done Sent false
25004 7410 Acked Write 8 Done Done Sent false
25005 7411 Acked Read 1 Done Done Sent false
25006 7412 Acked Write 2 Done Done Sent false
25007 7413 Acked Write 6 Done Done Sent false
25008 7414 Acked Read 3 Done Done Sent false
25009 7415 Acked Read 3 Done Done Sent false
25010 7416 Acked Flush 0 Done Done Sent false
25011 7417 Acked Write 6 Done Done Sent false
25012 7418 Acked Write 1 Done Done Sent false
25013 7419 Acked Read 6 Done Done Sent false
25014 7420 Acked Write 10 Done Done Sent false
25015 7421 Acked Flush 0 Done Done Sent false
25016 7422 Acked Read 5 Done Done Sent false
25017 7423 Acked Flush 0 Done Done Sent false
25018 7424 Acked Write 4 Done Done Sent false
25019 7425 Acked Read 2 Done Done Sent false
25020 7426 Acked Write 1 Done Done Sent false
25021 7427 Acked Read 5 Done Done Sent false
25022 7428 Acked Read 2 Done Done Sent false
25023 7429 Acked Write 2 Done Done Sent false
25024 7430 Acked Read 4 Done Done Sent false
25025 7431 Acked Read 9 Done Done Sent false
25026 7432 Acked Read 7 Done Done Sent false
25027 7433 Acked Write 1 Done Done Sent false
25028 7434 Acked Write 6 Done Done Sent false
25029 7435 Acked Read 6 Done Done Sent false
25030 7436 Acked Read 5 Done Done Sent false
25031 7437 Acked Write 9 Done Done Sent false
25032 7438 Acked Flush 0 Done Done Sent false
25033 7439 Acked Write 3 Done Done Sent false
25034 7440 Acked Read 10 Done Done Sent false
25035 7441 Acked Write 10 Done Done Sent false
25036 7442 Acked Write 10 Done Done Sent false
25037 7443 Acked Write 3 Done Done Sent false
25038 7444 Acked Write 2 Done Done Sent false
25039 7445 Acked Read 5 Done Done Sent false
25040 7446 Acked Read 4 Done Done Sent false
25041 7447 Acked Read 7 Done Done Sent false
25042 7448 Acked Write 2 Done Done Sent false
25043 7449 Acked Write 1 Done Done Sent false
25044 7450 Acked Write 8 Done Done Sent false
25045 7451 Acked Read 4 Done Done Sent false
25046 7452 Acked Read 6 Done Done Sent false
25047 7453 Acked Write 3 Done Done Sent false
25048 7454 Acked Flush 0 Done Done Sent false
25049 7455 Acked Read 10 Done Done Sent false
25050 7456 Acked Read 6 Done Done Sent false
25051 7457 Acked Read 8 Done Done Sent false
25052 7458 Acked Write 1 Done Done Sent false
25053 7459 Acked Write 1 Done Done Sent false
25054 7460 Acked Write 1 Done Done Sent false
25055 7461 Acked Write 10 Done Done Sent false
25056 7462 Acked Read 1 Done Done Sent false
25057 7463 Acked Read 3 Done Done Sent false
25058 7464 Acked Read 2 Done Done Sent false
25059 7465 Acked Read 9 Done Done Sent false
25060 7466 Acked Flush 0 Done Done Sent false
25061 7467 Acked Write 4 Done Done Sent false
25062 7468 Acked Read 5 Done Done Sent false
25063 7469 Acked Read 3 Done Done Sent false
25064 7470 Acked Read 7 Done Done Sent false
25065 7471 Acked Flush 0 Done Done Sent false
25066 7472 Acked Read 2 Done Done Sent false
25067 7473 Acked Read 7 Done Done Sent false
25068 7474 Acked Write 3 Done Done Sent false
25069 7475 Acked Write 4 Done Done Sent false
25070 7476 Acked Read 9 Done Done Sent false
25071 7477 Acked Read 6 Done Done Sent false
25072 7478 Acked Write 9 Done Done Sent false
25073 7479 Acked Read 7 Done Done Sent false
25074 7480 Acked Write 1 Done Done Sent false
25075 7481 Acked Read 9 Done Done Sent false
25076 7482 Acked Write 2 Done Done Sent false
25077 7483 Acked Write 2 Done Done Sent false
25078 7484 Acked Flush 0 Done Done Sent false
25079 7485 Acked Read 3 Done Done Sent false
25080 7486 Acked Read 4 Done Done Sent false
25081 7487 Acked Flush 0 Done Done Sent false
25082 7488 Acked Write 8 Done Done Sent false
25083 7489 Acked Write 5 Done Done Sent false
25084 7490 Acked Flush 0 Done Done Sent false
25085 7491 Acked Write 1 Done Done Sent false
25086 7492 Acked Flush 0 Done Done Sent false
25087 7493 Acked Read 6 Done Done Sent false
25088 7494 Acked Write 6 Done Done Sent false
25089 7495 Acked Read 7 Done Done Sent false
25090 7496 Acked Read 7 Done Done Sent false
25091 7497 Acked Write 6 Done Done Sent false
25092 7498 Acked Write 7 Done Done Sent false
25093 7499 Acked Read 8 Done Done Sent false
25094 7500 Acked Write 6 Done Done Sent false
25095 7501 Acked Read 4 Done Done Sent false
25096 7502 Acked Read 6 Done Done Sent false
25097 7503 Acked Flush 0 Done Done Sent false
25098 7504 Acked Read 2 Done Done Sent false
25099 7505 Acked Write 7 Done Done Sent false
25100 7506 Acked Read 6 Done Done Sent false
25101 7507 Acked Flush 0 Done Done Sent false
25102 7508 Acked Read 9 Done Done Sent false
25103 7509 Acked Read 2 Done Done Sent false
25104 7510 Acked Read 3 Done Done Sent false
25105 7511 Acked Read 10 Done Done Sent false
25106 7512 Acked Read 4 Done Done Sent false
25107 7513 Acked Read 1 Done Done Sent false
25108 7514 Acked Write 8 Done Done Sent false
25109 7515 Acked Write 2 Done Done Sent false
25110 7516 Acked Read 9 Done Done Sent false
25111 7517 Acked Read 6 Done Done Sent false
25112 7518 Acked Read 2 Done Done Sent false
25113 7519 Acked Write 7 Done Done Sent false
25114 7520 Acked Flush 0 Done Done Sent false
25115 7521 Acked Read 8 Done Done Sent false
25116 7522 Acked Read 10 Done Done Sent false
25117 7523 Acked Write 6 Done Done Sent false
25118 7524 Acked Read 1 Done Done Sent false
25119 7525 Acked Write 9 Done Done Sent false
25120 7526 Acked Write 2 Done Done Sent false
25121 7527 Acked Write 8 Done Done Sent false
25122 7528 Acked Write 1 Done Done Sent false
25123 7529 Acked Flush 0 Done Done Sent false
25124 7530 Acked Read 7 Done Done Sent false
25125 7531 Acked Write 2 Done Done Sent false
25126 7532 Acked Write 7 Done Done Sent false
25127 7533 Acked Read 8 Done Done Sent false
25128 7534 Acked Read 1 Done Done Sent false
25129 7535 Acked Read 10 Done Done Sent false
25130 7536 Acked Read 4 Done Done Sent false
25131 7537 Acked Write 8 Done Done Sent false
25132 7538 Acked Flush 0 Done Done Sent false
25133 7539 Acked Read 3 Done Done Sent false
25134 7540 Acked Write 10 Done Done Sent false
25135 7541 Acked Write 10 Done Done Sent false
25136 7542 Acked Write 3 Done Done Sent false
25137 7543 Acked Read 5 Done Done Sent false
25138 7544 Acked Write 10 Done Done Sent false
25139 7545 Acked Read 1 Done Done Sent false
25140 7546 Acked Read 3 Done Done Sent false
25141 7547 Acked Read 3 Done Done Sent false
25142 7548 Acked Read 2 Done Done Sent false
25143 7549 Acked Read 1 Done Done Sent false
25144 7550 Acked Write 4 Done Done Sent false
25145 7551 Acked Read 2 Done Done Sent false
25146 7552 Acked Read 5 Done Done Sent false
25147 7553 Acked Write 2 Done Done Sent false
25148 7554 Acked Flush 0 Done Done Sent false
25149 7555 Acked Read 2 Done Done Sent false
25150 7556 Acked Read 1 Done Done Sent false
25151 7557 Acked Write 1 Done Done Sent false
25152 7558 Acked Write 1 Done Done Sent false
25153 7559 Acked Read 1 Done Done Sent false
25154 7560 Acked Read 2 Done Done Sent false
25155 7561 Acked Read 10 Done Done Sent false
25156 7562 Acked Read 7 Done Done Sent false
25157 7563 Acked Write 9 Done Done Sent false
25158 7564 Acked Write 5 Done Done Sent false
25159 7565 Acked Flush 0 Done Done Sent false
25160 7566 Acked Read 8 Done Done Sent false
25161 7567 Acked Write 5 Done Done Sent false
25162 7568 Acked Write 7 Done Done Sent false
25163 7569 Acked Write 3 Done Done Sent false
25164 7570 Acked Read 1 Done Done Sent false
25165 7571 Acked Read 7 Done Done Sent false
25166 7572 Acked Write 6 Done Done Sent false
25167 7573 Acked Flush 0 Done Done Sent false
25168 7574 Acked Read 1 Done Done Sent false
25169 7575 Acked Read 8 Done Done Sent false
25170 7576 Acked Read 8 Done Done Sent false
25171 7577 Acked Read 5 Done Done Sent false
25172 7578 Acked Write 6 Done Done Sent false
25173 7579 Acked Write 1 Done Done Sent false
25174 7580 Acked Read 8 Done Done Sent false
25175 7581 Acked Read 8 Done Done Sent false
25176 7582 Acked Read 3 Done Done Sent false
25177 7583 Acked Flush 0 Done Done Sent false
25178 7584 Acked Read 2 Done Done Sent false
25179 7585 Acked Read 1 Done Done Sent false
25180 7586 Acked Read 7 Done Done Sent false
25181 7587 Acked Read 8 Done Done Sent false
25182 7588 Acked Write 2 Done Done Sent false
25183 7589 Acked Write 6 Done Done Sent false
25184 7590 Acked Read 5 Done Done Sent false
25185 7591 Acked Write 2 Done Done Sent false
25186 7592 Acked Flush 0 Done Done Sent false
25187 7593 Acked Read 10 Done Done Sent false
25188 7594 Acked Write 1 Done Done Sent false
25189 7595 Acked Write 1 Done Done Sent false
25190 7596 Acked Flush 0 Done Done Sent false
25191 7597 Acked Read 8 Done Done Sent false
25192 7598 Acked Read 1 Done Done Sent false
25193 7599 Acked Write 6 Done Done Sent false
25194 7600 Acked Read 4 Done Done Sent false
25195 7601 Acked Read 8 Done Done Sent false
25196 7602 Acked Write 6 Done Done Sent false
25197 7603 Acked Read 2 Done Done Sent false
25198 7604 Acked Write 5 Done Done Sent false
25199 7605 Acked Write 8 Done Done Sent false
25200 7606 Acked Read 9 Done Done Sent false
25201 7607 Acked Read 10 Done Done Sent false
25202 7608 Acked Read 10 Done Done Sent false
25203 7609 Acked Read 8 Done Done Sent false
25204 7610 Acked Write 10 Done Done Sent false
25205 7611 Acked Read 4 Done Done Sent false
25206 7612 Acked Read 6 Done Done Sent false
25207 7613 Acked Write 9 Done Done Sent false
25208 7614 Acked Write 9 Done Done Sent false
25209 7615 Acked Read 1 Done Done Sent false
25210 7616 Acked Write 10 Done Done Sent false
25211 7617 Acked Write 9 Done Done Sent false
25212 7618 Acked Write 3 Done Done Sent false
25213 7619 Acked Write 1 Done Done Sent false
25214 7620 Acked Read 2 Done Done Sent false
25215 7621 Acked Read 1 Done Done Sent false
25216 7622 Acked Read 5 Done Done Sent false
25217 7623 Acked Write 7 Done Done Sent false
25218 7624 Acked Read 1 Done Done Sent false
25219 7625 Acked Write 8 Done Done Sent false
25220 7626 Acked Read 4 Done Done Sent false
25221 7627 Acked Write 3 Done Done Sent false
25222 7628 Acked Flush 0 Done Done Sent false
25223 7629 Acked Write 4 Done Done Sent false
25224 7630 Acked Read 6 Done Done Sent false
25225 7631 Acked Read 10 Done Done Sent false
25226 7632 Acked Write 2 Done Done Sent false
25227 7633 Acked Write 5 Done Done Sent false
25228 7634 Acked Read 1 Done Done Sent false
25229 7635 Acked Read 6 Done Done Sent false
25230 7636 Acked Read 9 Done Done Sent false
25231 7637 Acked Read 1 Done Done Sent false
25232 7638 Acked Read 5 Done Done Sent false
25233 7639 Acked Flush 0 Done Done Sent false
25234 STATES DS:0 DS:1 DS:2 TOTAL
25235 Sent 0 0 301 301
25236 Done 301 301 0 602
25237 Skipped 0 0 0 0
25238 Error 0 0 0 0
25239 Last Flush: Some(JobId(7639)) Some(JobId(7639)) Some(JobId(7338))
25240 Downstairs last five completed: 7338 7337 7336 7335 7334
25241 Upstairs last five completed: 7639 7638 7637 7636 7635
25242 CLIENT: Up:0 ds:301 act:2
252432025-03-07T18:18:02.639ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
252442025-03-07T18:18:03.881ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
252452025-03-07T18:18:04.497ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
252462025-03-07T18:18:05.500ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
25247 [8830] got start action
25248 [8830] I am going to restart
25249 Starting downstairs at port 8830
25250 Make output file at "/tmp/dsc/downstairs-8830.txt"
25251 [8830][2] reports Starting
25252 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
25253 [8830][2] reports Running
25254 ----------------------------------------------------------------
25255 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25256 States: Active Active Offline
25257 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25258 7339 Acked Read 2 Done Done Sent false
25259 7340 Acked Read 8 Done Done Sent false
25260 7341 Acked Read 5 Done Done Sent false
25261 7342 Acked Read 4 Done Done Sent false
25262 7343 Acked Read 1 Done Done Sent false
25263 7344 Acked Write 9 Done Done Sent false
25264 7345 Acked Read 3 Done Done Sent false
25265 7346 Acked Write 7 Done Done Sent false
25266 7347 Acked Read 2 Done Done Sent false
25267 7348 Acked Read 6 Done Done Sent false
25268 7349 Acked Write 7 Done Done Sent false
25269 7350 Acked Flush 0 Done Done Sent false
25270 7351 Acked Flush 0 Done Done Sent false
25271 7352 Acked Write 5 Done Done Sent false
25272 7353 Acked Read 10 Done Done Sent false
25273 7354 Acked Read 2 Done Done Sent false
25274 7355 Acked Read 8 Done Done Sent false
25275 7356 Acked Read 4 Done Done Sent false
25276 7357 Acked Write 6 Done Done Sent false
25277 7358 Acked Read 9 Done Done Sent false
25278 7359 Acked Flush 0 Done Done Sent false
25279 7360 Acked Write 8 Done Done Sent false
25280 7361 Acked Read 7 Done Done Sent false
25281 7362 Acked Write 10 Done Done Sent false
25282 7363 Acked Read 7 Done Done Sent false
25283 7364 Acked Write 2 Done Done Sent false
25284 7365 Acked Write 6 Done Done Sent false
25285 7366 Acked Read 3 Done Done Sent false
25286 7367 Acked Read 4 Done Done Sent false
25287 7368 Acked Read 4 Done Done Sent false
25288 7369 Acked Flush 0 Done Done Sent false
25289 7370 Acked Read 8 Done Done Sent false
25290 7371 Acked Write 8 Done Done Sent false
25291 7372 Acked Flush 0 Done Done Sent false
25292 7373 Acked Write 7 Done Done Sent false
25293 7374 Acked Write 6 Done Done Sent false
25294 7375 Acked Write 2 Done Done Sent false
25295 7376 Acked Read 1 Done Done Sent false
25296 7377 Acked Read 5 Done Done Sent false
25297 7378 Acked Read 6 Done Done Sent false
25298 7379 Acked Read 9 Done Done Sent false
25299 7380 Acked Flush 0 Done Done Sent false
25300 7381 Acked Read 2 Done Done Sent false
25301 7382 Acked Write 6 Done Done Sent false
25302 7383 Acked Flush 0 Done Done Sent false
25303 7384 Acked Read 7 Done Done Sent false
25304 7385 Acked Write 3 Done Done Sent false
25305 7386 Acked Write 2 Done Done Sent false
25306 7387 Acked Read 2 Done Done Sent false
25307 7388 Acked Flush 0 Done Done Sent false
25308 7389 Acked Read 7 Done Done Sent false
25309 7390 Acked Read 6 Done Done Sent false
25310 7391 Acked Read 8 Done Done Sent false
25311 7392 Acked Read 2 Done Done Sent false
25312 7393 Acked Flush 0 Done Done Sent false
25313 7394 Acked Write 4 Done Done Sent false
25314 7395 Acked Write 9 Done Done Sent false
25315 7396 Acked Read 7 Done Done Sent false
25316 7397 Acked Flush 0 Done Done Sent false
25317 7398 Acked Read 7 Done Done Sent false
25318 7399 Acked Write 7 Done Done Sent false
25319 7400 Acked Write 3 Done Done Sent false
25320 7401 Acked Read 4 Done Done Sent false
25321 7402 Acked Write 10 Done Done Sent false
25322 7403 Acked Write 6 Done Done Sent false
25323 7404 Acked Write 3 Done Done Sent false
25324 7405 Acked Read 7 Done Done Sent false
25325 7406 Acked Write 6 Done Done Sent false
25326 7407 Acked Write 9 Done Done Sent false
25327 7408 Acked Write 4 Done Done Sent false
25328 7409 Acked Flush 0 Done Done Sent false
25329 7410 Acked Write 8 Done Done Sent false
25330 7411 Acked Read 1 Done Done Sent false
25331 7412 Acked Write 2 Done Done Sent false
25332 7413 Acked Write 6 Done Done Sent false
25333 7414 Acked Read 3 Done Done Sent false
25334 7415 Acked Read 3 Done Done Sent false
25335 7416 Acked Flush 0 Done Done Sent false
25336 7417 Acked Write 6 Done Done Sent false
25337 7418 Acked Write 1 Done Done Sent false
25338 7419 Acked Read 6 Done Done Sent false
25339 7420 Acked Write 10 Done Done Sent false
25340 7421 Acked Flush 0 Done Done Sent false
25341 7422 Acked Read 5 Done Done Sent false
25342 7423 Acked Flush 0 Done Done Sent false
25343 7424 Acked Write 4 Done Done Sent false
25344 7425 Acked Read 2 Done Done Sent false
25345 7426 Acked Write 1 Done Done Sent false
25346 7427 Acked Read 5 Done Done Sent false
25347 7428 Acked Read 2 Done Done Sent false
25348 7429 Acked Write 2 Done Done Sent false
25349 7430 Acked Read 4 Done Done Sent false
25350 7431 Acked Read 9 Done Done Sent false
25351 7432 Acked Read 7 Done Done Sent false
25352 7433 Acked Write 1 Done Done Sent false
25353 7434 Acked Write 6 Done Done Sent false
25354 7435 Acked Read 6 Done Done Sent false
25355 7436 Acked Read 5 Done Done Sent false
25356 7437 Acked Write 9 Done Done Sent false
25357 7438 Acked Flush 0 Done Done Sent false
25358 7439 Acked Write 3 Done Done Sent false
25359 7440 Acked Read 10 Done Done Sent false
25360 7441 Acked Write 10 Done Done Sent false
25361 7442 Acked Write 10 Done Done Sent false
25362 7443 Acked Write 3 Done Done Sent false
25363 7444 Acked Write 2 Done Done Sent false
25364 7445 Acked Read 5 Done Done Sent false
25365 7446 Acked Read 4 Done Done Sent false
25366 7447 Acked Read 7 Done Done Sent false
25367 7448 Acked Write 2 Done Done Sent false
25368 7449 Acked Write 1 Done Done Sent false
25369 7450 Acked Write 8 Done Done Sent false
25370 7451 Acked Read 4 Done Done Sent false
25371 7452 Acked Read 6 Done Done Sent false
25372 7453 Acked Write 3 Done Done Sent false
25373 7454 Acked Flush 0 Done Done Sent false
25374 7455 Acked Read 10 Done Done Sent false
25375 7456 Acked Read 6 Done Done Sent false
25376 7457 Acked Read 8 Done Done Sent false
25377 7458 Acked Write 1 Done Done Sent false
25378 7459 Acked Write 1 Done Done Sent false
25379 7460 Acked Write 1 Done Done Sent false
25380 7461 Acked Write 10 Done Done Sent false
25381 7462 Acked Read 1 Done Done Sent false
25382 7463 Acked Read 3 Done Done Sent false
25383 7464 Acked Read 2 Done Done Sent false
25384 7465 Acked Read 9 Done Done Sent false
25385 7466 Acked Flush 0 Done Done Sent false
25386 7467 Acked Write 4 Done Done Sent false
25387 7468 Acked Read 5 Done Done Sent false
25388 7469 Acked Read 3 Done Done Sent false
25389 7470 Acked Read 7 Done Done Sent false
25390 7471 Acked Flush 0 Done Done Sent false
25391 7472 Acked Read 2 Done Done Sent false
25392 7473 Acked Read 7 Done Done Sent false
25393 7474 Acked Write 3 Done Done Sent false
25394 7475 Acked Write 4 Done Done Sent false
25395 7476 Acked Read 9 Done Done Sent false
25396 7477 Acked Read 6 Done Done Sent false
25397 7478 Acked Write 9 Done Done Sent false
25398 7479 Acked Read 7 Done Done Sent false
25399 7480 Acked Write 1 Done Done Sent false
25400 7481 Acked Read 9 Done Done Sent false
25401 7482 Acked Write 2 Done Done Sent false
25402 7483 Acked Write 2 Done Done Sent false
25403 7484 Acked Flush 0 Done Done Sent false
25404 7485 Acked Read 3 Done Done Sent false
25405 7486 Acked Read 4 Done Done Sent false
25406 7487 Acked Flush 0 Done Done Sent false
25407 7488 Acked Write 8 Done Done Sent false
25408 7489 Acked Write 5 Done Done Sent false
25409 7490 Acked Flush 0 Done Done Sent false
25410 7491 Acked Write 1 Done Done Sent false
25411 7492 Acked Flush 0 Done Done Sent false
25412 7493 Acked Read 6 Done Done Sent false
25413 7494 Acked Write 6 Done Done Sent false
25414 7495 Acked Read 7 Done Done Sent false
25415 7496 Acked Read 7 Done Done Sent false
25416 7497 Acked Write 6 Done Done Sent false
25417 7498 Acked Write 7 Done Done Sent false
25418 7499 Acked Read 8 Done Done Sent false
25419 7500 Acked Write 6 Done Done Sent false
25420 7501 Acked Read 4 Done Done Sent false
25421 7502 Acked Read 6 Done Done Sent false
25422 7503 Acked Flush 0 Done Done Sent false
25423 7504 Acked Read 2 Done Done Sent false
25424 7505 Acked Write 7 Done Done Sent false
25425 7506 Acked Read 6 Done Done Sent false
25426 7507 Acked Flush 0 Done Done Sent false
25427 7508 Acked Read 9 Done Done Sent false
25428 7509 Acked Read 2 Done Done Sent false
25429 7510 Acked Read 3 Done Done Sent false
25430 7511 Acked Read 10 Done Done Sent false
25431 7512 Acked Read 4 Done Done Sent false
25432 7513 Acked Read 1 Done Done Sent false
25433 7514 Acked Write 8 Done Done Sent false
25434 7515 Acked Write 2 Done Done Sent false
25435 7516 Acked Read 9 Done Done Sent false
25436 7517 Acked Read 6 Done Done Sent false
25437 7518 Acked Read 2 Done Done Sent false
25438 7519 Acked Write 7 Done Done Sent false
25439 7520 Acked Flush 0 Done Done Sent false
25440 7521 Acked Read 8 Done Done Sent false
25441 7522 Acked Read 10 Done Done Sent false
25442 7523 Acked Write 6 Done Done Sent false
25443 7524 Acked Read 1 Done Done Sent false
25444 7525 Acked Write 9 Done Done Sent false
25445 7526 Acked Write 2 Done Done Sent false
25446 7527 Acked Write 8 Done Done Sent false
25447 7528 Acked Write 1 Done Done Sent false
25448 7529 Acked Flush 0 Done Done Sent false
25449 7530 Acked Read 7 Done Done Sent false
25450 7531 Acked Write 2 Done Done Sent false
25451 7532 Acked Write 7 Done Done Sent false
25452 7533 Acked Read 8 Done Done Sent false
25453 7534 Acked Read 1 Done Done Sent false
25454 7535 Acked Read 10 Done Done Sent false
25455 7536 Acked Read 4 Done Done Sent false
25456 7537 Acked Write 8 Done Done Sent false
25457 7538 Acked Flush 0 Done Done Sent false
25458 7539 Acked Read 3 Done Done Sent false
25459 7540 Acked Write 10 Done Done Sent false
25460 7541 Acked Write 10 Done Done Sent false
25461 7542 Acked Write 3 Done Done Sent false
25462 7543 Acked Read 5 Done Done Sent false
25463 7544 Acked Write 10 Done Done Sent false
25464 7545 Acked Read 1 Done Done Sent false
25465 7546 Acked Read 3 Done Done Sent false
25466 7547 Acked Read 3 Done Done Sent false
25467 7548 Acked Read 2 Done Done Sent false
25468 7549 Acked Read 1 Done Done Sent false
25469 7550 Acked Write 4 Done Done Sent false
25470 7551 Acked Read 2 Done Done Sent false
25471 7552 Acked Read 5 Done Done Sent false
25472 7553 Acked Write 2 Done Done Sent false
25473 7554 Acked Flush 0 Done Done Sent false
25474 7555 Acked Read 2 Done Done Sent false
25475 7556 Acked Read 1 Done Done Sent false
25476 7557 Acked Write 1 Done Done Sent false
25477 7558 Acked Write 1 Done Done Sent false
25478 7559 Acked Read 1 Done Done Sent false
25479 7560 Acked Read 2 Done Done Sent false
25480 7561 Acked Read 10 Done Done Sent false
25481 7562 Acked Read 7 Done Done Sent false
25482 7563 Acked Write 9 Done Done Sent false
25483 7564 Acked Write 5 Done Done Sent false
25484 7565 Acked Flush 0 Done Done Sent false
25485 7566 Acked Read 8 Done Done Sent false
25486 7567 Acked Write 5 Done Done Sent false
25487 7568 Acked Write 7 Done Done Sent false
25488 7569 Acked Write 3 Done Done Sent false
25489 7570 Acked Read 1 Done Done Sent false
25490 7571 Acked Read 7 Done Done Sent false
25491 7572 Acked Write 6 Done Done Sent false
25492 7573 Acked Flush 0 Done Done Sent false
25493 7574 Acked Read 1 Done Done Sent false
25494 7575 Acked Read 8 Done Done Sent false
25495 7576 Acked Read 8 Done Done Sent false
25496 7577 Acked Read 5 Done Done Sent false
25497 7578 Acked Write 6 Done Done Sent false
25498 7579 Acked Write 1 Done Done Sent false
25499 7580 Acked Read 8 Done Done Sent false
25500 7581 Acked Read 8 Done Done Sent false
25501 7582 Acked Read 3 Done Done Sent false
25502 7583 Acked Flush 0 Done Done Sent false
25503 7584 Acked Read 2 Done Done Sent false
25504 7585 Acked Read 1 Done Done Sent false
25505 7586 Acked Read 7 Done Done Sent false
25506 7587 Acked Read 8 Done Done Sent false
25507 7588 Acked Write 2 Done Done Sent false
25508 7589 Acked Write 6 Done Done Sent false
25509 7590 Acked Read 5 Done Done Sent false
25510 7591 Acked Write 2 Done Done Sent false
25511 7592 Acked Flush 0 Done Done Sent false
25512 7593 Acked Read 10 Done Done Sent false
25513 7594 Acked Write 1 Done Done Sent false
25514 7595 Acked Write 1 Done Done Sent false
25515 7596 Acked Flush 0 Done Done Sent false
25516 7597 Acked Read 8 Done Done Sent false
25517 7598 Acked Read 1 Done Done Sent false
25518 7599 Acked Write 6 Done Done Sent false
25519 7600 Acked Read 4 Done Done Sent false
25520 7601 Acked Read 8 Done Done Sent false
25521 7602 Acked Write 6 Done Done Sent false
25522 7603 Acked Read 2 Done Done Sent false
25523 7604 Acked Write 5 Done Done Sent false
25524 7605 Acked Write 8 Done Done Sent false
25525 7606 Acked Read 9 Done Done Sent false
25526 7607 Acked Read 10 Done Done Sent false
25527 7608 Acked Read 10 Done Done Sent false
25528 7609 Acked Read 8 Done Done Sent false
25529 7610 Acked Write 10 Done Done Sent false
25530 7611 Acked Read 4 Done Done Sent false
25531 7612 Acked Read 6 Done Done Sent false
25532 7613 Acked Write 9 Done Done Sent false
25533 7614 Acked Write 9 Done Done Sent false
25534 7615 Acked Read 1 Done Done Sent false
25535 7616 Acked Write 10 Done Done Sent false
25536 7617 Acked Write 9 Done Done Sent false
25537 7618 Acked Write 3 Done Done Sent false
25538 7619 Acked Write 1 Done Done Sent false
25539 7620 Acked Read 2 Done Done Sent false
25540 7621 Acked Read 1 Done Done Sent false
25541 7622 Acked Read 5 Done Done Sent false
25542 7623 Acked Write 7 Done Done Sent false
25543 7624 Acked Read 1 Done Done Sent false
25544 7625 Acked Write 8 Done Done Sent false
25545 7626 Acked Read 4 Done Done Sent false
25546 7627 Acked Write 3 Done Done Sent false
25547 7628 Acked Flush 0 Done Done Sent false
25548 7629 Acked Write 4 Done Done Sent false
25549 7630 Acked Read 6 Done Done Sent false
25550 7631 Acked Read 10 Done Done Sent false
25551 7632 Acked Write 2 Done Done Sent false
25552 7633 Acked Write 5 Done Done Sent false
25553 7634 Acked Read 1 Done Done Sent false
25554 7635 Acked Read 6 Done Done Sent false
25555 7636 Acked Read 9 Done Done Sent false
25556 7637 Acked Read 1 Done Done Sent false
25557 7638 Acked Read 5 Done Done Sent false
25558 7639 Acked Flush 0 Done Done Sent false
25559 STATES DS:0 DS:1 DS:2 TOTAL
25560 Sent 0 0 301 301
25561 Done 301 301 0 602
25562 Skipped 0 0 0 0
25563 Error 0 0 0 0
25564 Last Flush: Some(JobId(7639)) Some(JobId(7639)) Some(JobId(7338))
25565 Downstairs last five completed: 7338 7337 7336 7335 7334
25566 Upstairs last five completed: 7639 7638 7637 7636 7635
25567 CLIENT: Up:0 ds:301 act:2
255682025-03-07T18:18:06.641ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255692025-03-07T18:18:07.503ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255702025-03-07T18:18:07.894ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255712025-03-07T18:18:07.894ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51039) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255722025-03-07T18:18:07.895ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255732025-03-07T18:18:07.895ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255742025-03-07T18:18:07.895ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255752025-03-07T18:18:07.895ZINFOcrucible: send last flush ID to this DS: Some(JobId(7338)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255762025-03-07T18:18:07.895ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255772025-03-07T18:18:07.895ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7338)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255782025-03-07T18:18:07.896ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7338)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
25579 ----------------------------------------------------------------
25580 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25581 Downstairs last five completed: 7639 7638 7637 7636 7635
25582 Upstairs last five completed: 7639 7638 7637 7636 7635
25583 CLIENT: Up:0 ds:0 act:3
25584 Replay: All jobs finished
25585 Main task has work to do, go find it
25586 got dsc Stop(0)
25587 stop 0
25588 Mar 07 18:18:10.142 INFO [8810] Got stop action so:false kr:false
25589 request completed, latency_us: 354, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 964351d5-7659-4dbc-9a5f-31d347e5c7e5, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
255902025-03-07T18:18:10.142ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255912025-03-07T18:18:10.142ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255922025-03-07T18:18:10.142ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
255932025-03-07T18:18:10.143ZINFOcrucible: client task is exiting = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
25594 Mar 07 18:18:10.143 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/0, method: GET, req_id: e1bd2ad2-1fa2-4668-b46e-28d4715f9544, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
25595 [8810] Exited with: ExitStatus(unix_wait_status(9))
25596 [8810][0] reports Exit
25597 Mar 07 18:18:14.144 INFO request completed, latency_us: 302, response_code: 200, uri: /state/cid/0, method: GET, req_id: 86dd2818-789d-46bb-ac1d-ee3c4637bb37, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
25598 001/300 Read block 1590 len 40960
25599 002/300 Write block 4952 len 8192 data: 3 4
25600 003/300 Write block 4946 len 4096 data: 6
25601 004/300 Read block 4361 len 32768
25602 005/300 Read block 4270 len 4096
25603 006/300 Flush
25604 007/300 Read block 1267 len 36864
25605 008/300 Write block 3380 len 32768 data: 6 6 6 6 7 6 7 7
25606 009/300 Write block 4852 len 24576 data: 5 6 7 5 5 6
25607 010/300 Read block 1373 len 32768
25608 011/300 Flush
25609 012/300 Read block 777 len 4096
25610 013/300 Write block 3603 len 24576 data: 5 5 4 4 4 4
25611 014/300 Write block 3117 len 32768 data: 5 4 3 3 2 2 3 5
25612 015/300 Flush
25613 016/300 Write block 937 len 20480 data: 8 9 8 6 6
25614 017/300 Read block 1738 len 4096
25615 018/300 Read block 3929 len 20480
25616 019/300 Read block 4606 len 36864
25617 020/300 Flush
25618 021/300 Read block 1163 len 16384
25619 022/300 Write block 4459 len 16384 data: 2 3 4 4
25620 023/300 Write block 3612 len 4096 data: 3
25621 024/300 Flush
25622 025/300 Read block 4977 len 12288
25623 026/300 Flush
25624 027/300 Write block 4243 len 12288 data: 4 5 5
25625 028/300 Write block 2576 len 16384 data: 4 4 6 6
25626 029/300 Read block 2131 len 16384
25627 030/300 Read block 1565 len 40960
25628 031/300 Flush
25629 032/300 Read block 1935 len 32768
25630 033/300 Read block 2079 len 40960
25631 034/300 Read block 2523 len 36864
25632 035/300 Write block 207 len 4096 data: 5
25633 036/300 Read block 751 len 32768
25634 037/300 Read block 4033 len 36864
25635 038/300 Write block 3073 len 20480 data: 5 5 4 5 6
25636 039/300 Read block 3616 len 8192
25637 040/300 Read block 4808 len 16384
25638 041/300 Flush
25639 042/300 Write block 2620 len 36864 data: 4 4 3 3 2 2 3 3 2
25640 043/300 Read block 4870 len 28672
25641 044/300 Write block 714 len 8192 data: 3 3
25642 045/300 Write block 2442 len 20480 data: 4 3 3 4 5
25643 046/300 Write block 265 len 12288 data: 7 6 6
25644 047/300 Read block 1560 len 28672
25645 048/300 Read block 4604 len 4096
25646 049/300 Write block 3166 len 28672 data: 4 5 3 5 5 5 4
25647 050/300 Read block 3825 len 16384
25648 051/300 Read block 1879 len 8192
25649 052/300 Write block 2760 len 36864 data: 2 3 4 4 4 4 4 5 5
25650 053/300 Read block 4161 len 36864
25651 054/300 Write block 1512 len 4096 data: 8
25652 055/300 Read block 254 len 16384
25653 056/300 Read block 1605 len 20480
25654 057/300 Read block 720 len 12288
25655 058/300 Read block 688 len 8192
25656 059/300 Read block 1201 len 16384
25657 060/300 Read block 3478 len 4096
25658 061/300 Read block 667 len 4096
25659 062/300 Write block 2360 len 20480 data: 3 3 3 3 4
25660 063/300 Write block 302 len 12288 data: 7 7 7
25661 064/300 Read block 2533 len 40960
25662 065/300 Write block 3922 len 40960 data: 10 8 8 8 7 7 6 6 5 6
25663 066/300 Read block 2653 len 32768
25664 067/300 Read block 1283 len 16384
25665 068/300 Flush
25666 069/300 Read block 1531 len 16384
25667 070/300 Flush
25668 071/300 Read block 1345 len 20480
25669 072/300 Write block 1083 len 12288 data: 4 4 5
25670 073/300 Write block 362 len 20480 data: 2 3 3 3 4
25671 074/300 Read block 2778 len 4096
25672 075/300 Read block 1918 len 28672
25673 076/300 Flush
25674 077/300 Flush
25675 078/300 Read block 83 len 40960
25676 079/300 Read block 3163 len 28672
25677 080/300 Read block 2360 len 32768
25678 081/300 Read block 3797 len 24576
25679 082/300 Write block 2657 len 20480 data: 10 11 11 9 10
25680 083/300 Write block 1363 len 8192 data: 7 6
25681 084/300 Read block 1511 len 8192
25682 085/300 Write block 1127 len 12288 data: 7 8 7
25683 086/300 Read block 1848 len 40960
25684 087/300 Write block 2262 len 4096 data: 5
25685 088/300 Flush
25686 089/300 Write block 445 len 28672 data: 3 3 2 2 3 3 3
25687 090/300 Read block 1160 len 24576
25688 091/300 Read block 1055 len 40960
25689 092/300 Read block 3626 len 40960
25690 093/300 Write block 4360 len 40960 data: 3 5 5 6 6 7 7 7 7 7
25691 094/300 Write block 4368 len 16384 data: 8 8 6 7
25692 095/300 Read block 2403 len 36864
25693 096/300 Write block 2269 len 8192 data: 5 5
25694 097/300 Write block 1097 len 28672 data: 5 5 5 4 3 3 5
25695 098/300 Read block 290 len 8192
25696 099/300 Write block 3796 len 36864 data: 6 8 7 5 6 5 5 5 5
25697 100/300 Write block 545 len 16384 data: 4 4 3 2
25698 101/300 Read block 16 len 12288
25699 102/300 Read block 100 len 24576
25700 103/300 Flush
25701 104/300 Flush
25702 105/300 Read block 3828 len 20480
25703 106/300 Write block 1479 len 24576 data: 2 3 4 4 4 4
25704 107/300 Write block 2827 len 16384 data: 7 5 5 3
25705 108/300 Flush
25706 109/300 Write block 621 len 36864 data: 6 8 8 8 7 8 8 6 7
25707 110/300 Write block 995 len 28672 data: 5 3 4 3 3 4 4
25708 111/300 Write block 264 len 28672 data: 7 8 7 7 6 6 7
25709 112/300 Write block 4444 len 8192 data: 3 2
25710 113/300 Write block 4556 len 36864 data: 3 3 3 3 2 4 5 5 5
25711 114/300 Read block 3080 len 32768
25712 115/300 Write block 1624 len 28672 data: 8 9 9 7 7 7 6
25713 116/300 Write block 1515 len 4096 data: 8
25714 117/300 Read block 776 len 16384
25715 118/300 Read block 4762 len 24576
25716 119/300 Flush
25717 120/300 Write block 2721 len 32768 data: 7 7 6 6 7 8 6 6
25718 121/300 Read block 1226 len 40960
25719 122/300 Read block 2277 len 36864
25720 123/300 Write block 316 len 20480 data: 3 3 3 4 4
25721 124/300 Write block 459 len 24576 data: 5 4 3 3 3 4
25722 125/300 Read block 1722 len 12288
25723 126/300 Write block 3691 len 12288 data: 3 4 4
25724 127/300 Write block 3788 len 12288 data: 4 4 4
25725 128/300 Write block 2217 len 16384 data: 3 3 3 4
25726 129/300 Write block 3926 len 20480 data: 8 8 7 7 6
25727 130/300 Write block 4770 len 24576 data: 6 6 6 5 6 5
25728 131/300 Write block 1003 len 12288 data: 4 5 4
25729 132/300 Write block 3252 len 40960 data: 5 4 4 4 4 4 3 4 4 5
25730 133/300 Flush
25731 134/300 Flush
25732 135/300 Write block 3498 len 16384 data: 4 4 4 4
25733 136/300 Write block 927 len 12288 data: 3 3 3
25734 137/300 Flush
25735 138/300 Read block 734 len 16384
25736 139/300 Write block 901 len 4096 data: 3
25737 140/300 Read block 329 len 32768
25738 141/300 Read block 627 len 36864
25739 142/300 Write block 4233 len 28672 data: 3 3 3 4 4 4 4
25740 143/300 Read block 2567 len 16384
25741 144/300 Flush
25742 145/300 Write block 3998 len 40960 data: 6 5 4 4 4 4 4 4 4 4
25743 146/300 Read block 4516 len 8192
25744 147/300 Write block 597 len 16384 data: 5 5 4 4
25745 148/300 Read block 616 len 12288
25746 149/300 Write block 4932 len 32768 data: 4 5 6 6 6 6 7 6
25747 150/300 Write block 1854 len 4096 data: 4
25748 151/300 Write block 4664 len 16384 data: 4 4 3 4
25749 152/300 Read block 4312 len 28672
25750 153/300 Read block 3996 len 36864
25751 154/300 Read block 2468 len 16384
25752 155/300 Read block 3342 len 20480
25753 156/300 Write block 4586 len 28672 data: 6 5 5 6 6 4 5
25754 157/300 Write block 144 len 40960 data: 4 5 6 6 7 6 5 4 5 5
25755 158/300 Write block 345 len 16384 data: 4 4 6 8
25756 159/300 Write block 2964 len 4096 data: 4
25757 160/300 Read block 3960 len 16384
25758 161/300 Read block 2460 len 24576
25759 162/300 Read block 1546 len 8192
25760 163/300 Read block 2256 len 20480
25761 164/300 Read block 2597 len 24576
25762 165/300 Flush
25763 166/300 Read block 4204 len 20480
25764 167/300 Read block 2660 len 24576
25765 168/300 Write block 2527 len 32768 data: 4 4 4 3 7 7 7 8
25766 169/300 Read block 430 len 4096
25767 170/300 Read block 2738 len 28672
25768 171/300 Write block 3541 len 32768 data: 3 2 2 2 3 4 3 3
25769 172/300 Read block 3860 len 12288
25770 173/300 Read block 1564 len 40960
25771 174/300 Write block 883 len 36864 data: 4 4 3 3 3 3 4 5 4
25772 175/300 Read block 2046 len 28672
25773 176/300 Write block 2467 len 12288 data: 8 8 6
25774 177/300 Write block 3840 len 24576 data: 3 3 2 3 4 3
25775 178/300 Read block 819 len 32768
25776 179/300 Flush
25777 180/300 Write block 1657 len 12288 data: 5 6 4
25778 181/300 Read block 2827 len 40960
25779 182/300 Write block 4800 len 40960 data: 5 4 3 2 3 3 3 4 4 4
25780 183/300 Read block 3466 len 36864
25781 184/300 Write block 1731 len 24576 data: 4 4 5 5 5 7
25782 185/300 Write block 205 len 20480 data: 5 5 6 6 5
25783 186/300 Write block 279 len 24576 data: 6 6 7 5 5 5
25784 187/300 Read block 2986 len 20480
25785 188/300 Read block 1056 len 20480
25786 189/300 Write block 3383 len 8192 data: 7 8
25787 190/300 Flush
25788 191/300 Write block 3188 len 8192 data: 5 5
25789 192/300 Read block 2531 len 8192
25790 193/300 Read block 1914 len 32768
25791 194/300 Read block 1805 len 8192
25792 195/300 Read block 4678 len 16384
25793 196/300 Write block 1635 len 4096 data: 5
25794 197/300 Write block 266 len 4096 data: 8
25795 198/300 Read block 4128 len 32768
25796 199/300 Write block 4459 len 20480 data: 3 4 5 5 6
25797 200/300 Flush
25798 201/300 Read block 3371 len 20480
25799 202/300 Read block 3810 len 28672
25800 203/300 Write block 1636 len 8192 data: 5 5
25801 204/300 Write block 4700 len 32768 data: 6 6 6 6 7 6 5 5
25802 205/300 Read block 971 len 28672
25803 206/300 Write block 2547 len 28672 data: 10 9 9 8 6 6 7
25804 207/300 Read block 2126 len 8192
25805 208/300 Write block 1389 len 32768 data: 2 3 3 3 3 3 3 4
25806 209/300 Read block 4038 len 16384
25807 210/300 Write block 2240 len 28672 data: 5 6 5 6 7 4 3
25808 211/300 Write block 3871 len 16384 data: 5 3 4 5
25809 212/300 Write block 595 len 20480 data: 6 6 6 6 5
25810 213/300 Read block 862 len 20480
25811 214/300 Read block 624 len 4096
25812 215/300 Read block 1302 len 24576
25813 216/300 Read block 1206 len 28672
25814 217/300 Write block 1987 len 12288 data: 3 5 5
25815 218/300 Read block 3355 len 20480
25816 219/300 Write block 2979 len 16384 data: 3 3 3 3
25817 220/300 Read block 3327 len 32768
25818 221/300 Write block 4020 len 40960 data: 4 4 3 3 3 3 2 2 2 3
25819 222/300 Read block 726 len 12288
25820 223/300 Write block 4883 len 32768 data: 3 3 4 3 4 4 3 3
25821 224/300 Write block 3586 len 32768 data: 3 4 3 3 3 3 3 3
25822 225/300 Read block 1700 len 32768
25823 226/300 Write block 3368 len 24576 data: 6 7 5 6 5 5
25824 227/300 Write block 1760 len 8192 data: 2 2
25825 228/300 Read block 4472 len 36864
25826 229/300 Write block 4871 len 8192 data: 4 4
25827 230/300 Read block 171 len 4096
25828 231/300 Flush
25829 232/300 Read block 4441 len 20480
25830 233/300 Read block 2322 len 4096
25831 234/300 Write block 4270 len 28672 data: 4 5 5 5 7 6 6
25832 235/300 Read block 3124 len 16384
25833 236/300 Read block 3861 len 4096
25834 237/300 Write block 3258 len 8192 data: 4 5
25835 238/300 Flush
25836 239/300 Read block 3813 len 8192
25837 240/300 Write block 4804 len 20480 data: 4 4 4 5 5
25838 241/300 Write block 1985 len 8192 data: 4 3
25839 242/300 Write block 3433 len 12288 data: 3 5 5
25840 243/300 Write block 2552 len 16384 data: 7 8 5 7
25841 244/300 Read block 3246 len 36864
25842 245/300 Write block 1804 len 8192 data: 7 7
25843 246/300 Read block 3135 len 28672
25844 247/300 Read block 3656 len 40960
25845 248/300 Write block 2672 len 36864 data: 7 5 5 4 4 4 4 5 5
25846 249/300 Read block 1976 len 36864
25847 250/300 Write block 164 len 12288 data: 7 8 10
25848 251/300 Write block 212 len 16384 data: 5 6 5 7
25849 252/300 Write block 3702 len 16384 data: 5 4 4 6
25850 253/300 Write block 3851 len 24576 data: 5 5 6 5 5 5
25851 254/300 Flush
25852 255/300 Read block 2560 len 36864
25853 256/300 Write block 3702 len 36864 data: 6 5 5 7 6 7 6 6 5
25854 257/300 Read block 448 len 12288
25855 258/300 Write block 582 len 20480 data: 5 3 4 5 5
25856 259/300 Write block 2858 len 16384 data: 2 3 2 3
25857 260/300 Read block 2058 len 16384
25858 261/300 Write block 4344 len 20480 data: 5 4 4 3 2
25859 262/300 Write block 1750 len 12288 data: 7 5 5
25860 263/300 Read block 3582 len 20480
25861 264/300 Read block 4272 len 4096
25862 265/300 Read block 336 len 28672
25863 266/300 Write block 4212 len 4096 data: 9
25864 267/300 Write block 2503 len 28672 data: 4 4 4 5 4 4 5
25865 268/300 Write block 4342 len 36864 data: 6 5 6 5 5 4 3 3 3
25866 269/300 Read block 2616 len 24576
25867 270/300 Read block 4720 len 40960
25868 271/300 Flush
25869 272/300 Read block 2967 len 12288
25870 273/300 Write block 1213 len 24576 data: 8 9 7 7 4 4
25871 274/300 Write block 4342 len 24576 data: 7 6 7 6 6 5
25872 275/300 Write block 4798 len 4096 data: 7
25873 276/300 Write block 1303 len 20480 data: 3 3 3 3 5
25874 277/300 Write block 3235 len 12288 data: 5 6 5
25875 278/300 Flush
25876 279/300 Write block 4702 len 40960 data: 7 7 8 7 6 6 4 3 3 3
25877 280/300 Write block 904 len 28672 data: 5 5 3 3 4 5 5
25878 281/300 Read block 4941 len 28672
25879 282/300 Read block 4609 len 20480
25880 283/300 Read block 904 len 8192
25881 284/300 Write block 4891 len 8192 data: 3 3
25882 285/300 Read block 2905 len 20480
25883 286/300 Write block 1135 len 28672 data: 5 5 4 6 7 7 6
25884 287/300 Read block 445 len 20480
25885 288/300 Read block 4056 len 4096
25886 289/300 Write block 90 len 24576 data: 5 5 5 6 6 5
25887 290/300 Read block 4667 len 12288
25888 291/300 Write block 1584 len 16384 data: 2 3 5 6
25889 292/300 Read block 4495 len 28672
25890 293/300 Read block 3185 len 4096
25891 294/300 Write block 4479 len 8192 data: 3 3
25892 295/300 Flush
25893 296/300 Write block 3928 len 4096 data: 8
25894 297/300 Write block 2229 len 28672 data: 4 4 4 4 6 6 6
25895 298/300 Read block 2920 len 4096
25896 299/300 Flush
25897 300/300 Write block 4580 len 24576 data: 4 4 5 4 5 5
25898 301/300 Read block 2528 len 36864
25899 Main task has work to do, go find it
25900 got dsc Start(0)
25901 start 0
25902 Mar 07 18:18:14.378 INFO request completed, latency_us: 168, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3d80489f-b800-47e7-815f-c06528b8010d, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
25903 Replay: started 0, returned:Ok(())
25904 ----------------------------------------------------------------
25905 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25906 States: Offline Active Active
25907 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25908 7640 Acked Read 10 Sent Done Done false
25909 7641 Acked Write 2 Sent Done Done false
25910 7642 Acked Write 1 Sent Done Done false
25911 7643 Acked Read 8 Sent Done Done false
25912 7644 Acked Read 1 Sent Done Done false
25913 7645 Acked Flush 0 Sent Done Done false
25914 7646 Acked Read 9 Sent Done Done false
25915 7647 Acked Write 8 Sent Done Done false
25916 7648 Acked Write 6 Sent Done Done false
25917 7649 Acked Read 8 Sent Done Done false
25918 7650 Acked Flush 0 Sent Done Done false
25919 7651 Acked Read 1 Sent Done Done false
25920 7652 Acked Write 6 Sent Done Done false
25921 7653 Acked Write 8 Sent Done Done false
25922 7654 Acked Flush 0 Sent Done Done false
25923 7655 Acked Write 5 Sent Done Done false
25924 7656 Acked Read 1 Sent Done Done false
25925 7657 Acked Read 5 Sent Done Done false
25926 7658 Acked Read 9 Sent Done Done false
25927 7659 Acked Flush 0 Sent Done Done false
25928 7660 Acked Read 4 Sent Done Done false
25929 7661 Acked Write 4 Sent Done Done false
25930 7662 Acked Write 1 Sent Done Done false
25931 7663 Acked Flush 0 Sent Done Done false
25932 7664 Acked Read 3 Sent Done Done false
25933 7665 Acked Flush 0 Sent Done Done false
25934 7666 Acked Write 3 Sent Done Done false
25935 7667 Acked Write 4 Sent Done Done false
25936 7668 Acked Read 4 Sent Done Done false
25937 7669 Acked Read 10 Sent Done Done false
25938 7670 Acked Flush 0 Sent Done Done false
25939 7671 Acked Read 8 Sent Done Done false
25940 7672 Acked Read 10 Sent Done Done false
25941 7673 Acked Read 9 Sent Done Done false
25942 7674 Acked Write 1 Sent Done Done false
25943 7675 Acked Read 8 Sent Done Done false
25944 7676 Acked Read 9 Sent Done Done false
25945 7677 Acked Write 5 Sent Done Done false
25946 7678 Acked Read 2 Sent Done Done false
25947 7679 Acked Read 4 Sent Done Done false
25948 7680 Acked Flush 0 Sent Done Done false
25949 7681 Acked Write 9 Sent Done Done false
25950 7682 Acked Read 7 Sent Done Done false
25951 7683 Acked Write 2 Sent Done Done false
25952 7684 Acked Write 5 Sent Done Done false
25953 7685 Acked Write 3 Sent Done Done false
25954 7686 Acked Read 7 Sent Done Done false
25955 7687 Acked Read 1 Sent Done Done false
25956 7688 Acked Write 7 Sent Done Done false
25957 7689 Acked Read 4 Sent Done Done false
25958 7690 Acked Read 2 Sent Done Done false
25959 7691 Acked Write 9 Sent Done Done false
25960 7692 Acked Read 9 Sent Done Done false
25961 7693 Acked Write 1 Sent Done Done false
25962 7694 Acked Read 4 Sent Done Done false
25963 7695 Acked Read 5 Sent Done Done false
25964 7696 Acked Read 3 Sent Done Done false
25965 7697 Acked Read 2 Sent Done Done false
25966 7698 Acked Read 4 Sent Done Done false
25967 7699 Acked Read 1 Sent Done Done false
25968 7700 Acked Read 1 Sent Done Done false
25969 7701 Acked Write 5 Sent Done Done false
25970 7702 Acked Write 3 Sent Done Done false
25971 7703 Acked Read 10 Sent Done Done false
25972 7704 Acked Write 10 Sent Done Done false
25973 7705 Acked Read 8 Sent Done Done false
25974 7706 Acked Read 4 Sent Done Done false
25975 7707 Acked Flush 0 Sent Done Done false
25976 7708 Acked Read 4 Sent Done Done false
25977 7709 Acked Flush 0 Sent Done Done false
25978 7710 Acked Read 5 Sent Done Done false
25979 7711 Acked Write 3 Sent Done Done false
25980 7712 Acked Write 5 Sent Done Done false
25981 7713 Acked Read 1 Sent Done Done false
25982 7714 Acked Read 7 Sent Done Done false
25983 7715 Acked Flush 0 Sent Done Done false
25984 7716 Acked Flush 0 Sent Done Done false
25985 7717 Acked Read 10 Sent Done Done false
25986 7718 Acked Read 7 Sent Done Done false
25987 7719 Acked Read 8 Sent Done Done false
25988 7720 Acked Read 6 Sent Done Done false
25989 7721 Acked Write 5 Sent Done Done false
25990 7722 Acked Write 2 Sent Done Done false
25991 7723 Acked Read 2 Sent Done Done false
25992 7724 Acked Write 3 Sent Done Done false
25993 7725 Acked Read 10 Sent Done Done false
25994 7726 Acked Write 1 Sent Done Done false
25995 7727 Acked Flush 0 Sent Done Done false
25996 7728 Acked Write 7 Sent Done Done false
25997 7729 Acked Read 6 Sent Done Done false
25998 7730 Acked Read 10 Sent Done Done false
25999 7731 Acked Read 10 Sent Done Done false
26000 7732 Acked Write 10 Sent Done Done false
26001 7733 Acked Write 4 Sent Done Done false
26002 7734 Acked Read 9 Sent Done Done false
26003 7735 Acked Write 2 Sent Done Done false
26004 7736 Acked Write 7 Sent Done Done false
26005 7737 Acked Read 2 Sent Done Done false
26006 7738 Acked Write 9 Sent Done Done false
26007 7739 Acked Write 4 Sent Done Done false
26008 7740 Acked Read 3 Sent Done Done false
26009 7741 Acked Read 6 Sent Done Done false
26010 7742 Acked Flush 0 Sent Done Done false
26011 7743 Acked Flush 0 Sent Done Done false
26012 7744 Acked Read 5 Sent Done Done false
26013 7745 Acked Write 6 Sent Done Done false
26014 7746 Acked Write 4 Sent Done Done false
26015 7747 Acked Flush 0 Sent Done Done false
26016 7748 Acked Write 9 Sent Done Done false
26017 7749 Acked Write 7 Sent Done Done false
26018 7750 Acked Write 7 Sent Done Done false
26019 7751 Acked Write 2 Sent Done Done false
26020 7752 Acked Write 9 Sent Done Done false
26021 7753 Acked Read 8 Sent Done Done false
26022 7754 Acked Write 7 Sent Done Done false
26023 7755 Acked Write 1 Sent Done Done false
26024 7756 Acked Read 4 Sent Done Done false
26025 7757 Acked Read 6 Sent Done Done false
26026 7758 Acked Flush 0 Sent Done Done false
26027 7759 Acked Write 8 Sent Done Done false
26028 7760 Acked Read 10 Sent Done Done false
26029 7761 Acked Read 9 Sent Done Done false
26030 7762 Acked Write 5 Sent Done Done false
26031 7763 Acked Write 6 Sent Done Done false
26032 7764 Acked Read 3 Sent Done Done false
26033 7765 Acked Write 3 Sent Done Done false
26034 7766 Acked Write 3 Sent Done Done false
26035 7767 Acked Write 4 Sent Done Done false
26036 7768 Acked Write 5 Sent Done Done false
26037 7769 Acked Write 6 Sent Done Done false
26038 7770 Acked Write 3 Sent Done Done false
26039 7771 Acked Write 10 Sent Done Done false
26040 7772 Acked Flush 0 Sent Done Done false
26041 7773 Acked Flush 0 Sent Done Done false
26042 7774 Acked Write 4 Sent Done Done false
26043 7775 Acked Write 3 Sent Done Done false
26044 7776 Acked Flush 0 Sent Done Done false
26045 7777 Acked Read 4 Sent Done Done false
26046 7778 Acked Write 1 Sent Done Done false
26047 7779 Acked Read 8 Sent Done Done false
26048 7780 Acked Read 9 Sent Done Done false
26049 7781 Acked Write 7 Sent Done Done false
26050 7782 Acked Read 4 Sent Done Done false
26051 7783 Acked Flush 0 Sent Done Done false
26052 7784 Acked Write 10 Sent Done Done false
26053 7785 Acked Read 2 Sent Done Done false
26054 7786 Acked Write 4 Sent Done Done false
26055 7787 Acked Read 3 Sent Done Done false
26056 7788 Acked Write 8 Sent Done Done false
26057 7789 Acked Write 1 Sent Done Done false
26058 7790 Acked Write 4 Sent Done Done false
26059 7791 Acked Read 7 Sent Done Done false
26060 7792 Acked Read 9 Sent Done Done false
26061 7793 Acked Read 4 Sent Done Done false
26062 7794 Acked Read 5 Sent Done Done false
26063 7795 Acked Write 7 Sent Done Done false
26064 7796 Acked Write 10 Sent Done Done false
26065 7797 Acked Write 4 Sent Done Done false
26066 7798 Acked Write 1 Sent Done Done false
26067 7799 Acked Read 4 Sent Done Done false
26068 7800 Acked Read 6 Sent Done Done false
26069 7801 Acked Read 2 Sent Done Done false
26070 7802 Acked Read 5 Sent Done Done false
26071 7803 Acked Read 6 Sent Done Done false
26072 7804 Acked Flush 0 Sent Done Done false
26073 7805 Acked Read 5 Sent Done Done false
26074 7806 Acked Read 6 Sent Done Done false
26075 7807 Acked Write 8 Sent Done Done false
26076 7808 Acked Read 1 Sent Done Done false
26077 7809 Acked Read 7 Sent Done Done false
26078 7810 Acked Write 8 Sent Done Done false
26079 7811 Acked Read 3 Sent Done Done false
26080 7812 Acked Read 10 Sent Done Done false
26081 7813 Acked Write 9 Sent Done Done false
26082 7814 Acked Read 7 Sent Done Done false
26083 7815 Acked Write 3 Sent Done Done false
26084 7816 Acked Write 6 Sent Done Done false
26085 7817 Acked Read 8 Sent Done Done false
26086 7818 Acked Flush 0 Sent Done Done false
26087 7819 Acked Write 3 Sent Done Done false
26088 7820 Acked Read 10 Sent Done Done false
26089 7821 Acked Write 10 Sent Done Done false
26090 7822 Acked Read 9 Sent Done Done false
26091 7823 Acked Write 6 Sent Done Done false
26092 7824 Acked Write 5 Sent Done Done false
26093 7825 Acked Write 6 Sent Done Done false
26094 7826 Acked Read 5 Sent Done Done false
26095 7827 Acked Read 5 Sent Done Done false
26096 7828 Acked Write 2 Sent Done Done false
26097 7829 Acked Flush 0 Sent Done Done false
26098 7830 Acked Write 2 Sent Done Done false
26099 7831 Acked Read 2 Sent Done Done false
26100 7832 Acked Read 8 Sent Done Done false
26101 7833 Acked Read 2 Sent Done Done false
26102 7834 Acked Read 4 Sent Done Done false
26103 7835 Acked Write 1 Sent Done Done false
26104 7836 Acked Write 1 Sent Done Done false
26105 7837 Acked Read 8 Sent Done Done false
26106 7838 Acked Write 5 Sent Done Done false
26107 7839 Acked Flush 0 Sent Done Done false
26108 7840 Acked Read 5 Sent Done Done false
26109 7841 Acked Read 7 Sent Done Done false
26110 7842 Acked Write 2 Sent Done Done false
26111 7843 Acked Write 8 Sent Done Done false
26112 7844 Acked Read 7 Sent Done Done false
26113 7845 Acked Write 7 Sent Done Done false
26114 7846 Acked Read 2 Sent Done Done false
26115 7847 Acked Write 8 Sent Done Done false
26116 7848 Acked Read 4 Sent Done Done false
26117 7849 Acked Write 7 Sent Done Done false
26118 7850 Acked Write 4 Sent Done Done false
26119 7851 Acked Write 5 Sent Done Done false
26120 7852 Acked Read 5 Sent Done Done false
26121 7853 Acked Read 1 Sent Done Done false
26122 7854 Acked Read 6 Sent Done Done false
26123 7855 Acked Read 7 Sent Done Done false
26124 7856 Acked Write 3 Sent Done Done false
26125 7857 Acked Read 5 Sent Done Done false
26126 7858 Acked Write 4 Sent Done Done false
26127 7859 Acked Read 8 Sent Done Done false
26128 7860 Acked Write 10 Sent Done Done false
26129 7861 Acked Read 3 Sent Done Done false
26130 7862 Acked Write 8 Sent Done Done false
26131 7863 Acked Write 8 Sent Done Done false
26132 7864 Acked Read 8 Sent Done Done false
26133 7865 Acked Write 6 Sent Done Done false
26134 7866 Acked Write 2 Sent Done Done false
26135 7867 Acked Read 9 Sent Done Done false
26136 7868 Acked Write 2 Sent Done Done false
26137 7869 Acked Read 1 Sent Done Done false
26138 7870 Acked Flush 0 Sent Done Done false
26139 7871 Acked Read 5 Sent Done Done false
26140 7872 Acked Read 1 Sent Done Done false
26141 7873 Acked Write 7 Sent Done Done false
26142 7874 Acked Read 4 Sent Done Done false
26143 7875 Acked Read 1 Sent Done Done false
26144 7876 Acked Write 2 Sent Done Done false
26145 7877 Acked Flush 0 Sent Done Done false
26146 7878 Acked Read 2 Sent Done Done false
26147 7879 Acked Write 5 Sent Done Done false
26148 7880 Acked Write 2 Sent Done Done false
26149 7881 Acked Write 3 Sent Done Done false
26150 7882 Acked Write 4 Sent Done Done false
26151 7883 Acked Read 9 Sent Done Done false
26152 7884 Acked Write 2 Sent Done Done false
26153 7885 Acked Read 7 Sent Done Done false
26154 7886 Acked Read 10 Sent Done Done false
26155 7887 Acked Write 9 Sent Done Done false
26156 7888 Acked Read 9 Sent Done Done false
26157 7889 Acked Write 3 Sent Done Done false
26158 7890 Acked Write 4 Sent Done Done false
26159 7891 Acked Write 4 Sent Done Done false
26160 7892 Acked Write 6 Sent Done Done false
26161 7893 Acked Flush 0 Sent Done Done false
26162 7894 Acked Read 9 Sent Done Done false
26163 7895 Acked Write 9 Sent Done Done false
26164 7896 Acked Read 3 Sent Done Done false
26165 7897 Acked Write 5 Sent Done Done false
26166 7898 Acked Write 4 Sent Done Done false
26167 7899 Acked Read 4 Sent Done Done false
26168 7900 Acked Write 5 Sent Done Done false
26169 7901 Acked Write 3 Sent Done Done false
26170 7902 Acked Read 5 Sent Done Done false
26171 7903 Acked Read 1 Sent Done Done false
26172 7904 Acked Read 7 Sent Done Done false
26173 7905 Acked Write 1 Sent Done Done false
26174 7906 Acked Write 7 Sent Done Done false
26175 7907 Acked Write 9 Sent Done Done false
26176 7908 Acked Read 6 Sent Done Done false
26177 7909 Acked Read 10 Sent Done Done false
26178 7910 Acked Flush 0 Sent Done Done false
26179 7911 Acked Read 3 Sent Done Done false
26180 7912 Acked Write 6 Sent Done Done false
26181 7913 Acked Write 6 Sent Done Done false
26182 7914 Acked Write 1 Sent Done Done false
26183 7915 Acked Write 5 Sent Done Done false
26184 7916 Acked Write 3 Sent Done Done false
26185 7917 Acked Flush 0 Sent Done Done false
26186 7918 Acked Write 10 Sent Done Done false
26187 7919 Acked Write 7 Sent Done Done false
26188 7920 Acked Read 7 Sent Done Done false
26189 7921 Acked Read 5 Sent Done Done false
26190 7922 Acked Read 2 Sent Done Done false
26191 7923 Acked Write 2 Sent Done Done false
26192 7924 Acked Read 5 Sent Done Done false
26193 7925 Acked Write 7 Sent Done Done false
26194 7926 Acked Read 5 Sent Done Done false
26195 7927 Acked Read 1 Sent Done Done false
26196 7928 Acked Write 6 Sent Done Done false
26197 7929 Acked Read 3 Sent Done Done false
26198 7930 Acked Write 4 Sent Done Done false
26199 7931 Acked Read 7 Sent Done Done false
26200 7932 Acked Read 1 Sent Done Done false
26201 7933 Acked Write 2 Sent Done Done false
26202 7934 Acked Flush 0 Sent Done Done false
26203 7935 Acked Write 1 Sent Done Done false
26204 7936 Acked Write 7 Sent Done Done false
26205 7937 Acked Read 1 Sent Done Done false
26206 7938 Acked Flush 0 Sent Done Done false
26207 7939 Acked Write 6 Sent Done Done false
26208 7940 Acked Read 9 Sent Done Done false
26209 STATES DS:0 DS:1 DS:2 TOTAL
26210 Sent 301 0 0 301
26211 Done 0 301 301 602
26212 Skipped 0 0 0 0
26213 Error 0 0 0 0
26214 Last Flush: Some(JobId(7639)) Some(JobId(7938)) Some(JobId(7938))
26215 Downstairs last five completed: 7639 7638 7637 7636 7635
26216 Upstairs last five completed: 7940 7939 7938 7937 7936
26217 CLIENT: Up:0 ds:301 act:2
262182025-03-07T18:18:14.881ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
262192025-03-07T18:18:15.516ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
26220 [8810] got start action
26221 [8810] I am going to restart
26222 Starting downstairs at port 8810
26223 Make output file at "/tmp/dsc/downstairs-8810.txt"
26224 [8810][0] reports Starting
26225 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1231)
26226 [8810][0] reports Running
262272025-03-07T18:18:16.518ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
262282025-03-07T18:18:17.520ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
26229 ----------------------------------------------------------------
26230 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26231 States: Offline Active Active
26232 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26233 7640 Acked Read 10 Sent Done Done false
26234 7641 Acked Write 2 Sent Done Done false
26235 7642 Acked Write 1 Sent Done Done false
26236 7643 Acked Read 8 Sent Done Done false
26237 7644 Acked Read 1 Sent Done Done false
26238 7645 Acked Flush 0 Sent Done Done false
26239 7646 Acked Read 9 Sent Done Done false
26240 7647 Acked Write 8 Sent Done Done false
26241 7648 Acked Write 6 Sent Done Done false
26242 7649 Acked Read 8 Sent Done Done false
26243 7650 Acked Flush 0 Sent Done Done false
26244 7651 Acked Read 1 Sent Done Done false
26245 7652 Acked Write 6 Sent Done Done false
26246 7653 Acked Write 8 Sent Done Done false
26247 7654 Acked Flush 0 Sent Done Done false
26248 7655 Acked Write 5 Sent Done Done false
26249 7656 Acked Read 1 Sent Done Done false
26250 7657 Acked Read 5 Sent Done Done false
26251 7658 Acked Read 9 Sent Done Done false
26252 7659 Acked Flush 0 Sent Done Done false
26253 7660 Acked Read 4 Sent Done Done false
26254 7661 Acked Write 4 Sent Done Done false
26255 7662 Acked Write 1 Sent Done Done false
26256 7663 Acked Flush 0 Sent Done Done false
26257 7664 Acked Read 3 Sent Done Done false
26258 7665 Acked Flush 0 Sent Done Done false
26259 7666 Acked Write 3 Sent Done Done false
26260 7667 Acked Write 4 Sent Done Done false
26261 7668 Acked Read 4 Sent Done Done false
26262 7669 Acked Read 10 Sent Done Done false
26263 7670 Acked Flush 0 Sent Done Done false
26264 7671 Acked Read 8 Sent Done Done false
26265 7672 Acked Read 10 Sent Done Done false
26266 7673 Acked Read 9 Sent Done Done false
26267 7674 Acked Write 1 Sent Done Done false
26268 7675 Acked Read 8 Sent Done Done false
26269 7676 Acked Read 9 Sent Done Done false
26270 7677 Acked Write 5 Sent Done Done false
26271 7678 Acked Read 2 Sent Done Done false
26272 7679 Acked Read 4 Sent Done Done false
26273 7680 Acked Flush 0 Sent Done Done false
26274 7681 Acked Write 9 Sent Done Done false
26275 7682 Acked Read 7 Sent Done Done false
26276 7683 Acked Write 2 Sent Done Done false
26277 7684 Acked Write 5 Sent Done Done false
26278 7685 Acked Write 3 Sent Done Done false
26279 7686 Acked Read 7 Sent Done Done false
26280 7687 Acked Read 1 Sent Done Done false
26281 7688 Acked Write 7 Sent Done Done false
26282 7689 Acked Read 4 Sent Done Done false
26283 7690 Acked Read 2 Sent Done Done false
26284 7691 Acked Write 9 Sent Done Done false
26285 7692 Acked Read 9 Sent Done Done false
26286 7693 Acked Write 1 Sent Done Done false
26287 7694 Acked Read 4 Sent Done Done false
26288 7695 Acked Read 5 Sent Done Done false
26289 7696 Acked Read 3 Sent Done Done false
26290 7697 Acked Read 2 Sent Done Done false
26291 7698 Acked Read 4 Sent Done Done false
26292 7699 Acked Read 1 Sent Done Done false
26293 7700 Acked Read 1 Sent Done Done false
26294 7701 Acked Write 5 Sent Done Done false
26295 7702 Acked Write 3 Sent Done Done false
26296 7703 Acked Read 10 Sent Done Done false
26297 7704 Acked Write 10 Sent Done Done false
26298 7705 Acked Read 8 Sent Done Done false
26299 7706 Acked Read 4 Sent Done Done false
26300 7707 Acked Flush 0 Sent Done Done false
26301 7708 Acked Read 4 Sent Done Done false
26302 7709 Acked Flush 0 Sent Done Done false
26303 7710 Acked Read 5 Sent Done Done false
26304 7711 Acked Write 3 Sent Done Done false
26305 7712 Acked Write 5 Sent Done Done false
26306 7713 Acked Read 1 Sent Done Done false
26307 7714 Acked Read 7 Sent Done Done false
26308 7715 Acked Flush 0 Sent Done Done false
26309 7716 Acked Flush 0 Sent Done Done false
26310 7717 Acked Read 10 Sent Done Done false
26311 7718 Acked Read 7 Sent Done Done false
26312 7719 Acked Read 8 Sent Done Done false
26313 7720 Acked Read 6 Sent Done Done false
26314 7721 Acked Write 5 Sent Done Done false
26315 7722 Acked Write 2 Sent Done Done false
26316 7723 Acked Read 2 Sent Done Done false
26317 7724 Acked Write 3 Sent Done Done false
26318 7725 Acked Read 10 Sent Done Done false
26319 7726 Acked Write 1 Sent Done Done false
26320 7727 Acked Flush 0 Sent Done Done false
26321 7728 Acked Write 7 Sent Done Done false
26322 7729 Acked Read 6 Sent Done Done false
26323 7730 Acked Read 10 Sent Done Done false
26324 7731 Acked Read 10 Sent Done Done false
26325 7732 Acked Write 10 Sent Done Done false
26326 7733 Acked Write 4 Sent Done Done false
26327 7734 Acked Read 9 Sent Done Done false
26328 7735 Acked Write 2 Sent Done Done false
26329 7736 Acked Write 7 Sent Done Done false
26330 7737 Acked Read 2 Sent Done Done false
26331 7738 Acked Write 9 Sent Done Done false
26332 7739 Acked Write 4 Sent Done Done false
26333 7740 Acked Read 3 Sent Done Done false
26334 7741 Acked Read 6 Sent Done Done false
26335 7742 Acked Flush 0 Sent Done Done false
26336 7743 Acked Flush 0 Sent Done Done false
26337 7744 Acked Read 5 Sent Done Done false
26338 7745 Acked Write 6 Sent Done Done false
26339 7746 Acked Write 4 Sent Done Done false
26340 7747 Acked Flush 0 Sent Done Done false
26341 7748 Acked Write 9 Sent Done Done false
26342 7749 Acked Write 7 Sent Done Done false
26343 7750 Acked Write 7 Sent Done Done false
26344 7751 Acked Write 2 Sent Done Done false
26345 7752 Acked Write 9 Sent Done Done false
26346 7753 Acked Read 8 Sent Done Done false
26347 7754 Acked Write 7 Sent Done Done false
26348 7755 Acked Write 1 Sent Done Done false
26349 7756 Acked Read 4 Sent Done Done false
26350 7757 Acked Read 6 Sent Done Done false
26351 7758 Acked Flush 0 Sent Done Done false
26352 7759 Acked Write 8 Sent Done Done false
26353 7760 Acked Read 10 Sent Done Done false
26354 7761 Acked Read 9 Sent Done Done false
26355 7762 Acked Write 5 Sent Done Done false
26356 7763 Acked Write 6 Sent Done Done false
26357 7764 Acked Read 3 Sent Done Done false
26358 7765 Acked Write 3 Sent Done Done false
26359 7766 Acked Write 3 Sent Done Done false
26360 7767 Acked Write 4 Sent Done Done false
26361 7768 Acked Write 5 Sent Done Done false
26362 7769 Acked Write 6 Sent Done Done false
26363 7770 Acked Write 3 Sent Done Done false
26364 7771 Acked Write 10 Sent Done Done false
26365 7772 Acked Flush 0 Sent Done Done false
26366 7773 Acked Flush 0 Sent Done Done false
26367 7774 Acked Write 4 Sent Done Done false
26368 7775 Acked Write 3 Sent Done Done false
26369 7776 Acked Flush 0 Sent Done Done false
26370 7777 Acked Read 4 Sent Done Done false
26371 7778 Acked Write 1 Sent Done Done false
26372 7779 Acked Read 8 Sent Done Done false
26373 7780 Acked Read 9 Sent Done Done false
26374 7781 Acked Write 7 Sent Done Done false
26375 7782 Acked Read 4 Sent Done Done false
26376 7783 Acked Flush 0 Sent Done Done false
26377 7784 Acked Write 10 Sent Done Done false
26378 7785 Acked Read 2 Sent Done Done false
26379 7786 Acked Write 4 Sent Done Done false
26380 7787 Acked Read 3 Sent Done Done false
26381 7788 Acked Write 8 Sent Done Done false
26382 7789 Acked Write 1 Sent Done Done false
26383 7790 Acked Write 4 Sent Done Done false
26384 7791 Acked Read 7 Sent Done Done false
26385 7792 Acked Read 9 Sent Done Done false
26386 7793 Acked Read 4 Sent Done Done false
26387 7794 Acked Read 5 Sent Done Done false
26388 7795 Acked Write 7 Sent Done Done false
26389 7796 Acked Write 10 Sent Done Done false
26390 7797 Acked Write 4 Sent Done Done false
26391 7798 Acked Write 1 Sent Done Done false
26392 7799 Acked Read 4 Sent Done Done false
26393 7800 Acked Read 6 Sent Done Done false
26394 7801 Acked Read 2 Sent Done Done false
26395 7802 Acked Read 5 Sent Done Done false
26396 7803 Acked Read 6 Sent Done Done false
26397 7804 Acked Flush 0 Sent Done Done false
26398 7805 Acked Read 5 Sent Done Done false
26399 7806 Acked Read 6 Sent Done Done false
26400 7807 Acked Write 8 Sent Done Done false
26401 7808 Acked Read 1 Sent Done Done false
26402 7809 Acked Read 7 Sent Done Done false
26403 7810 Acked Write 8 Sent Done Done false
26404 7811 Acked Read 3 Sent Done Done false
26405 7812 Acked Read 10 Sent Done Done false
26406 7813 Acked Write 9 Sent Done Done false
26407 7814 Acked Read 7 Sent Done Done false
26408 7815 Acked Write 3 Sent Done Done false
26409 7816 Acked Write 6 Sent Done Done false
26410 7817 Acked Read 8 Sent Done Done false
26411 7818 Acked Flush 0 Sent Done Done false
26412 7819 Acked Write 3 Sent Done Done false
26413 7820 Acked Read 10 Sent Done Done false
26414 7821 Acked Write 10 Sent Done Done false
26415 7822 Acked Read 9 Sent Done Done false
26416 7823 Acked Write 6 Sent Done Done false
26417 7824 Acked Write 5 Sent Done Done false
26418 7825 Acked Write 6 Sent Done Done false
26419 7826 Acked Read 5 Sent Done Done false
26420 7827 Acked Read 5 Sent Done Done false
26421 7828 Acked Write 2 Sent Done Done false
26422 7829 Acked Flush 0 Sent Done Done false
26423 7830 Acked Write 2 Sent Done Done false
26424 7831 Acked Read 2 Sent Done Done false
26425 7832 Acked Read 8 Sent Done Done false
26426 7833 Acked Read 2 Sent Done Done false
26427 7834 Acked Read 4 Sent Done Done false
26428 7835 Acked Write 1 Sent Done Done false
26429 7836 Acked Write 1 Sent Done Done false
26430 7837 Acked Read 8 Sent Done Done false
26431 7838 Acked Write 5 Sent Done Done false
26432 7839 Acked Flush 0 Sent Done Done false
26433 7840 Acked Read 5 Sent Done Done false
26434 7841 Acked Read 7 Sent Done Done false
26435 7842 Acked Write 2 Sent Done Done false
26436 7843 Acked Write 8 Sent Done Done false
26437 7844 Acked Read 7 Sent Done Done false
26438 7845 Acked Write 7 Sent Done Done false
26439 7846 Acked Read 2 Sent Done Done false
26440 7847 Acked Write 8 Sent Done Done false
26441 7848 Acked Read 4 Sent Done Done false
26442 7849 Acked Write 7 Sent Done Done false
26443 7850 Acked Write 4 Sent Done Done false
26444 7851 Acked Write 5 Sent Done Done false
26445 7852 Acked Read 5 Sent Done Done false
26446 7853 Acked Read 1 Sent Done Done false
26447 7854 Acked Read 6 Sent Done Done false
26448 7855 Acked Read 7 Sent Done Done false
26449 7856 Acked Write 3 Sent Done Done false
26450 7857 Acked Read 5 Sent Done Done false
26451 7858 Acked Write 4 Sent Done Done false
26452 7859 Acked Read 8 Sent Done Done false
26453 7860 Acked Write 10 Sent Done Done false
26454 7861 Acked Read 3 Sent Done Done false
26455 7862 Acked Write 8 Sent Done Done false
26456 7863 Acked Write 8 Sent Done Done false
26457 7864 Acked Read 8 Sent Done Done false
26458 7865 Acked Write 6 Sent Done Done false
26459 7866 Acked Write 2 Sent Done Done false
26460 7867 Acked Read 9 Sent Done Done false
26461 7868 Acked Write 2 Sent Done Done false
26462 7869 Acked Read 1 Sent Done Done false
26463 7870 Acked Flush 0 Sent Done Done false
26464 7871 Acked Read 5 Sent Done Done false
26465 7872 Acked Read 1 Sent Done Done false
26466 7873 Acked Write 7 Sent Done Done false
26467 7874 Acked Read 4 Sent Done Done false
26468 7875 Acked Read 1 Sent Done Done false
26469 7876 Acked Write 2 Sent Done Done false
26470 7877 Acked Flush 0 Sent Done Done false
26471 7878 Acked Read 2 Sent Done Done false
26472 7879 Acked Write 5 Sent Done Done false
26473 7880 Acked Write 2 Sent Done Done false
26474 7881 Acked Write 3 Sent Done Done false
26475 7882 Acked Write 4 Sent Done Done false
26476 7883 Acked Read 9 Sent Done Done false
26477 7884 Acked Write 2 Sent Done Done false
26478 7885 Acked Read 7 Sent Done Done false
26479 7886 Acked Read 10 Sent Done Done false
26480 7887 Acked Write 9 Sent Done Done false
26481 7888 Acked Read 9 Sent Done Done false
26482 7889 Acked Write 3 Sent Done Done false
26483 7890 Acked Write 4 Sent Done Done false
26484 7891 Acked Write 4 Sent Done Done false
26485 7892 Acked Write 6 Sent Done Done false
26486 7893 Acked Flush 0 Sent Done Done false
26487 7894 Acked Read 9 Sent Done Done false
26488 7895 Acked Write 9 Sent Done Done false
26489 7896 Acked Read 3 Sent Done Done false
26490 7897 Acked Write 5 Sent Done Done false
26491 7898 Acked Write 4 Sent Done Done false
26492 7899 Acked Read 4 Sent Done Done false
26493 7900 Acked Write 5 Sent Done Done false
26494 7901 Acked Write 3 Sent Done Done false
26495 7902 Acked Read 5 Sent Done Done false
26496 7903 Acked Read 1 Sent Done Done false
26497 7904 Acked Read 7 Sent Done Done false
26498 7905 Acked Write 1 Sent Done Done false
26499 7906 Acked Write 7 Sent Done Done false
26500 7907 Acked Write 9 Sent Done Done false
26501 7908 Acked Read 6 Sent Done Done false
26502 7909 Acked Read 10 Sent Done Done false
26503 7910 Acked Flush 0 Sent Done Done false
26504 7911 Acked Read 3 Sent Done Done false
26505 7912 Acked Write 6 Sent Done Done false
26506 7913 Acked Write 6 Sent Done Done false
26507 7914 Acked Write 1 Sent Done Done false
26508 7915 Acked Write 5 Sent Done Done false
26509 7916 Acked Write 3 Sent Done Done false
26510 7917 Acked Flush 0 Sent Done Done false
26511 7918 Acked Write 10 Sent Done Done false
26512 7919 Acked Write 7 Sent Done Done false
26513 7920 Acked Read 7 Sent Done Done false
26514 7921 Acked Read 5 Sent Done Done false
26515 7922 Acked Read 2 Sent Done Done false
26516 7923 Acked Write 2 Sent Done Done false
26517 7924 Acked Read 5 Sent Done Done false
26518 7925 Acked Write 7 Sent Done Done false
26519 7926 Acked Read 5 Sent Done Done false
26520 7927 Acked Read 1 Sent Done Done false
26521 7928 Acked Write 6 Sent Done Done false
26522 7929 Acked Read 3 Sent Done Done false
26523 7930 Acked Write 4 Sent Done Done false
26524 7931 Acked Read 7 Sent Done Done false
26525 7932 Acked Read 1 Sent Done Done false
26526 7933 Acked Write 2 Sent Done Done false
26527 7934 Acked Flush 0 Sent Done Done false
26528 7935 Acked Write 1 Sent Done Done false
26529 7936 Acked Write 7 Sent Done Done false
26530 7937 Acked Read 1 Sent Done Done false
26531 7938 Acked Flush 0 Sent Done Done false
26532 7939 Acked Write 6 Sent Done Done false
26533 7940 Acked Read 9 Sent Done Done false
26534 STATES DS:0 DS:1 DS:2 TOTAL
26535 Sent 301 0 0 301
26536 Done 0 301 301 602
26537 Skipped 0 0 0 0
26538 Error 0 0 0 0
26539 Last Flush: Some(JobId(7639)) Some(JobId(7938)) Some(JobId(7938))
26540 Downstairs last five completed: 7639 7638 7637 7636 7635
26541 Upstairs last five completed: 7940 7939 7938 7937 7936
26542 CLIENT: Up:0 ds:301 act:2
265432025-03-07T18:18:18.883ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265442025-03-07T18:18:19.524ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265452025-03-07T18:18:20.144ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265462025-03-07T18:18:20.144ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44187) = io task client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265472025-03-07T18:18:20.145ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265482025-03-07T18:18:20.146ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265492025-03-07T18:18:20.146ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265502025-03-07T18:18:20.146ZINFOcrucible: send last flush ID to this DS: Some(JobId(7639)) = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265512025-03-07T18:18:20.146ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265522025-03-07T18:18:20.146ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7639)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265532025-03-07T18:18:20.146ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7639)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
26554 ----------------------------------------------------------------
26555 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26556 Downstairs last five completed: 7941 7940 7939 7938 7937
26557 Upstairs last five completed: 7941 7940 7939 7938 7937
26558 CLIENT: Up:0 ds:0 act:3
26559 Replay: All jobs finished
26560 Main task has work to do, go find it
26561 got dsc Stop(2)
26562 stop 2
26563 Mar 07 18:18:22.384 INFO request completed, latency_us: 311, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 6ba77055-40b3-4e12-bac4-09a38fe42a94, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0[8830] Got stop action so:false kr:false
26564 .1:9998
265652025-03-07T18:18:22.385ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
26566 Mar 07 18:18:22.385 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 080c8a4b-ea71-40a0-946d-67ce20e12a0f, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
265672025-03-07T18:18:22.385ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265682025-03-07T18:18:22.385ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
265692025-03-07T18:18:22.385ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
26570 [8830] Exited with: ExitStatus(unix_wait_status(9))
26571 [8830][2] reports Exit
26572 Mar 07 18:18:26.388 INFO request completed, latency_us: 304, response_code: 200, uri: /state/cid/2, method: GET, req_id: 43dba39f-b0d9-4360-b70c-8b3b5db4a1ab, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
26573 001/300 Write block 4390 len 32768 data: 6 5 4 4 3 3 4 4
26574 002/300 Flush
26575 003/300 Write block 167 len 16384 data: 9 9 8 10
26576 004/300 Write block 3289 len 32768 data: 6 4 5 5 7 7 6 6
26577 005/300 Read block 2555 len 16384
26578 006/300 Read block 4997 len 4096
26579 007/300 Write block 4635 len 8192 data: 6 5
26580 008/300 Read block 3514 len 20480
26581 009/300 Write block 3005 len 40960 data: 7 6 6 6 8 8 6 5 6 6
26582 010/300 Read block 1279 len 32768
26583 011/300 Read block 2952 len 20480
26584 012/300 Write block 2919 len 20480 data: 6 5 8 9 8
26585 013/300 Write block 2195 len 28672 data: 5 6 6 9 7 5 4
26586 014/300 Read block 3439 len 20480
26587 015/300 Read block 4767 len 12288
26588 016/300 Read block 1323 len 20480
26589 017/300 Read block 396 len 32768
26590 018/300 Read block 763 len 40960
26591 019/300 Read block 743 len 28672
26592 020/300 Write block 1883 len 40960 data: 3 4 5 4 4 6 8 8 7 7
26593 021/300 Write block 28 len 32768 data: 6 4 5 4 4 6 6 6
26594 022/300 Read block 552 len 16384
26595 023/300 Read block 3985 len 28672
26596 024/300 Read block 2882 len 24576
26597 025/300 Read block 1068 len 40960
26598 026/300 Read block 2619 len 8192
26599 027/300 Flush
26600 028/300 Read block 569 len 4096
26601 029/300 Write block 386 len 4096 data: 4
26602 030/300 Write block 3517 len 4096 data: 3
26603 031/300 Read block 82 len 4096
26604 032/300 Write block 1723 len 16384 data: 6 6 5 5
26605 033/300 Flush
26606 034/300 Write block 1201 len 20480 data: 4 3 3 4 5
26607 035/300 Write block 4133 len 16384 data: 2 2 3 5
26608 036/300 Read block 1034 len 4096
26609 037/300 Write block 3626 len 20480 data: 3 3 3 3 3
26610 038/300 Read block 4820 len 32768
26611 039/300 Write block 1008 len 28672 data: 8 8 7 7 6 6 6
26612 040/300 Read block 4631 len 8192
26613 041/300 Write block 3529 len 24576 data: 5 4 4 4 4 2
26614 042/300 Flush
26615 043/300 Read block 2485 len 24576
26616 044/300 Write block 1815 len 12288 data: 5 5 5
26617 045/300 Read block 3552 len 32768
26618 046/300 Write block 761 len 20480 data: 8 7 5 5 4
26619 047/300 Read block 2474 len 20480
26620 048/300 Write block 1540 len 4096 data: 6
26621 049/300 Flush
26622 050/300 Write block 2530 len 4096 data: 4
26623 051/300 Read block 1548 len 32768
26624 052/300 Write block 4649 len 40960 data: 2 2 2 2 4 4 5 5 5 5
26625 053/300 Write block 1139 len 20480 data: 8 8 7 6 7
26626 054/300 Write block 1831 len 28672 data: 6 7 5 6 5 5 6
26627 055/300 Flush
26628 056/300 Write block 3019 len 8192 data: 5 4
26629 057/300 Read block 2481 len 28672
26630 058/300 Write block 2819 len 32768 data: 7 7 7 8 8 6 8 6
26631 059/300 Write block 1814 len 4096 data: 3
26632 060/300 Read block 4748 len 40960
26633 061/300 Write block 3465 len 40960 data: 3 4 4 5 5 5 6 6 5 6
26634 062/300 Write block 2112 len 4096 data: 8
26635 063/300 Read block 4109 len 16384
26636 064/300 Read block 3495 len 32768
26637 065/300 Write block 3448 len 32768 data: 6 7 7 7 6 5 6 5
26638 066/300 Read block 1682 len 24576
26639 067/300 Read block 93 len 20480
26640 068/300 Read block 14 len 24576
26641 069/300 Write block 1460 len 28672 data: 7 9 9 9 8 6 7
26642 070/300 Read block 4480 len 16384
26643 071/300 Write block 3236 len 36864 data: 7 6 3 2 4 5 5 5 5
26644 072/300 Write block 4963 len 28672 data: 5 6 6 4 3 3 4
26645 073/300 Read block 1878 len 8192
26646 074/300 Read block 672 len 24576
26647 075/300 Read block 1413 len 24576
26648 076/300 Write block 121 len 28672 data: 6 6 6 7 7 8 7
26649 077/300 Read block 258 len 32768
26650 078/300 Write block 598 len 24576 data: 7 6 5 6 8 9
26651 079/300 Flush
26652 080/300 Write block 3782 len 16384 data: 9 7 7 5
26653 081/300 Read block 4724 len 20480
26654 082/300 Read block 1932 len 40960
26655 083/300 Flush
26656 084/300 Write block 3006 len 16384 data: 7 7 7 9
26657 085/300 Read block 2562 len 4096
26658 086/300 Read block 2328 len 40960
26659 087/300 Read block 1222 len 28672
26660 088/300 Write block 964 len 12288 data: 5 4 4
26661 089/300 Read block 99 len 28672
26662 090/300 Read block 378 len 8192
26663 091/300 Write block 2689 len 28672 data: 3 4 4 5 5 5 5
26664 092/300 Read block 2034 len 4096
26665 093/300 Read block 1086 len 40960
26666 094/300 Flush
26667 095/300 Read block 1885 len 28672
26668 096/300 Read block 1606 len 28672
26669 097/300 Read block 1785 len 16384
26670 098/300 Write block 204 len 28672 data: 7 6 6 7 7 6 5
26671 099/300 Read block 291 len 16384
26672 100/300 Write block 974 len 20480 data: 4 3 3 4 4
26673 101/300 Read block 2442 len 32768
26674 102/300 Write block 3267 len 8192 data: 6 5
26675 103/300 Write block 2716 len 40960 data: 4 4 5 5 7 8 8 7 7 8
26676 104/300 Write block 2860 len 16384 data: 3 4 3 4
26677 105/300 Read block 3399 len 32768
26678 106/300 Read block 3973 len 20480
26679 107/300 Read block 482 len 12288
26680 108/300 Read block 4138 len 16384
26681 109/300 Flush
26682 110/300 Write block 3424 len 8192 data: 5 6
26683 111/300 Read block 2767 len 20480
26684 112/300 Read block 1317 len 4096
26685 113/300 Write block 881 len 12288 data: 7 5 5
26686 114/300 Flush
26687 115/300 Read block 4423 len 32768
26688 116/300 Read block 3163 len 32768
26689 117/300 Read block 2363 len 28672
26690 118/300 Write block 3658 len 32768 data: 5 6 3 6 6 7 6 7
26691 119/300 Write block 301 len 28672 data: 5 8 8 8 7 7 8
26692 120/300 Flush
26693 121/300 Flush
26694 122/300 Read block 2127 len 36864
26695 123/300 Read block 2403 len 24576
26696 124/300 Write block 1563 len 32768 data: 3 6 7 8 10 11 11 9
26697 125/300 Flush
26698 126/300 Read block 3834 len 8192
26699 127/300 Write block 2849 len 8192 data: 3 3
26700 128/300 Write block 1296 len 28672 data: 5 5 5 6 5 4 3
26701 129/300 Read block 330 len 24576
26702 130/300 Flush
26703 131/300 Read block 3143 len 32768
26704 132/300 Write block 3072 len 8192 data: 4 6
26705 133/300 Write block 971 len 8192 data: 4 4
26706 134/300 Write block 3392 len 40960 data: 4 3 3 4 3 4 5 4 4 4
26707 135/300 Write block 2950 len 24576 data: 6 5 4 3 3 3
26708 136/300 Read block 988 len 8192
26709 137/300 Write block 3853 len 16384 data: 7 6 6 6
26710 138/300 Write block 46 len 36864 data: 7 5 5 4 3 4 5 5 5
26711 139/300 Read block 402 len 28672
26712 140/300 Read block 4257 len 32768
26713 141/300 Read block 459 len 4096
26714 142/300 Read block 4122 len 8192
26715 143/300 Read block 927 len 32768
26716 144/300 Read block 1868 len 40960
26717 145/300 Read block 1400 len 8192
26718 146/300 Write block 574 len 24576 data: 6 4 3 3 6 6
26719 147/300 Read block 3579 len 40960
26720 148/300 Flush
26721 149/300 Read block 3555 len 8192
26722 150/300 Write block 1736 len 32768 data: 8 6 5 5 5 6 7 7
26723 151/300 Write block 991 len 12288 data: 3 4 5
26724 152/300 Flush
26725 153/300 Read block 2859 len 40960
26726 154/300 Write block 3728 len 4096 data: 3
26727 155/300 Read block 4697 len 28672
26728 156/300 Read block 1424 len 12288
26729 157/300 Flush
26730 158/300 Write block 3611 len 40960 data: 3 4 4 4 5 4 4 4 3 3
26731 159/300 Read block 200 len 32768
26732 160/300 Write block 3511 len 20480 data: 4 4 4 3 3
26733 161/300 Read block 2667 len 28672
26734 162/300 Read block 4449 len 12288
26735 163/300 Read block 947 len 28672
26736 164/300 Read block 218 len 24576
26737 165/300 Read block 56 len 20480
26738 166/300 Write block 1306 len 36864 data: 4 6 4 3 5 5 5 3 4
26739 167/300 Write block 4280 len 20480 data: 6 7 6 7 7
26740 168/300 Write block 1410 len 4096 data: 4
26741 169/300 Read block 1406 len 28672
26742 170/300 Read block 2072 len 8192
26743 171/300 Write block 1660 len 8192 data: 4 5
26744 172/300 Write block 3577 len 36864 data: 5 5 4 4 4 5 4 3 3
26745 173/300 Flush
26746 174/300 Read block 3567 len 32768
26747 175/300 Read block 965 len 4096
26748 176/300 Write block 1384 len 32768 data: 4 5 5 5 2 3 4 4
26749 177/300 Read block 3204 len 4096
26750 178/300 Write block 3812 len 28672 data: 8 6 5 4 4 5 4
26751 179/300 Read block 278 len 20480
26752 180/300 Write block 4329 len 12288 data: 6 8 9
26753 181/300 Write block 4850 len 40960 data: 3 5 6 7 8 6 6 7 5 4
26754 182/300 Read block 344 len 20480
26755 183/300 Read block 198 len 36864
26756 184/300 Read block 3568 len 8192
26757 185/300 Write block 299 len 36864 data: 4 4 6 9 9 9 8 8 9
26758 186/300 Read block 3060 len 16384
26759 187/300 Write block 1572 len 12288 data: 6 7 6
26760 188/300 Read block 2271 len 32768
26761 189/300 Write block 3757 len 28672 data: 6 7 6 6 5 4 5
26762 190/300 Read block 2664 len 16384
26763 191/300 Read block 4202 len 16384
26764 192/300 Read block 2764 len 28672
26765 193/300 Write block 1134 len 36864 data: 4 6 6 5 7 9 9 8 7
26766 194/300 Flush
26767 195/300 Read block 1253 len 32768
26768 196/300 Write block 698 len 40960 data: 5 5 5 6 5 5 4 4 3 2
26769 197/300 Write block 3041 len 16384 data: 4 4 3 4
26770 198/300 Write block 2450 len 12288 data: 4 3 3
26771 199/300 Write block 3189 len 8192 data: 6 4
26772 200/300 Write block 1398 len 32768 data: 4 4 4 3 3 4 4 6
26773 201/300 Write block 906 len 12288 data: 4 4 5
26774 202/300 Write block 953 len 28672 data: 4 5 4 6 7 7 6
26775 203/300 Write block 4737 len 16384 data: 5 5 5 4
26776 204/300 Write block 4477 len 24576 data: 4 3 4 4 3 3
26777 205/300 Flush
26778 206/300 Write block 2631 len 4096 data: 3
26779 207/300 Read block 237 len 36864
26780 208/300 Read block 4553 len 40960
26781 209/300 Read block 442 len 32768
26782 210/300 Flush
26783 211/300 Write block 1126 len 4096 data: 6
26784 212/300 Read block 1848 len 36864
26785 213/300 Write block 1514 len 4096 data: 7
26786 214/300 Write block 2300 len 32768 data: 4 4 3 4 4 4 4 4
26787 215/300 Read block 2103 len 40960
26788 216/300 Read block 902 len 12288
26789 217/300 Read block 156 len 40960
26790 218/300 Write block 1383 len 12288 data: 4 5 6
26791 219/300 Read block 1263 len 40960
26792 220/300 Write block 2809 len 32768 data: 6 6 6 6 5 4 3 5
26793 221/300 Read block 94 len 20480
26794 222/300 Read block 4451 len 20480
26795 223/300 Write block 2519 len 16384 data: 4 5 5 4
26796 224/300 Read block 1642 len 8192
26797 225/300 Read block 4543 len 40960
26798 226/300 Write block 2108 len 20480 data: 7 7 8 8 9
26799 227/300 Read block 2886 len 40960
26800 228/300 Read block 2991 len 12288
26801 229/300 Write block 4726 len 8192 data: 4 3
26802 230/300 Read block 165 len 4096
26803 231/300 Write block 111 len 36864 data: 4 5 5 4 5 5 5 4 5
26804 232/300 Write block 4922 len 12288 data: 6 6 6
26805 233/300 Read block 2438 len 20480
26806 234/300 Read block 2866 len 36864
26807 235/300 Flush
26808 236/300 Flush
26809 237/300 Read block 4556 len 28672
26810 238/300 Write block 1459 len 16384 data: 7 8 10 10
26811 239/300 Write block 2223 len 20480 data: 5 5 5 6 5
26812 240/300 Write block 4499 len 20480 data: 3 3 3 5 5
26813 241/300 Read block 4823 len 24576
26814 242/300 Write block 2512 len 20480 data: 7 7 6 6 7
26815 243/300 Write block 1353 len 16384 data: 5 5 5 4
26816 244/300 Read block 3559 len 8192
26817 245/300 Write block 1575 len 32768 data: 4 6 4 4 4 5 5 3
26818 246/300 Read block 4828 len 8192
26819 247/300 Write block 1924 len 36864 data: 10 8 7 7 7 6 6 5 5
26820 248/300 Write block 1797 len 24576 data: 3 5 4 4 6 7
26821 249/300 Read block 4900 len 16384
26822 250/300 Write block 2965 len 40960 data: 5 5 4 5 5 5 5 5 6 5
26823 251/300 Read block 1157 len 28672
26824 252/300 Flush
26825 253/300 Write block 3565 len 20480 data: 3 5 6 5 6
26826 254/300 Write block 2580 len 40960 data: 5 6 5 5 4 6 8 7 8 6
26827 255/300 Write block 3107 len 36864 data: 5 5 5 4 5 5 7 7 6
26828 256/300 Write block 3 len 40960 data: 5 5 6 6 6 6 6 7 8 8
26829 257/300 Write block 3500 len 28672 data: 5 5 3 3 2 4 4
26830 258/300 Write block 1358 len 16384 data: 2 5 7 8
26831 259/300 Write block 77 len 4096 data: 4
26832 260/300 Write block 4627 len 24576 data: 4 4 5 3 3 3
26833 261/300 Read block 2502 len 4096
26834 262/300 Read block 3431 len 8192
26835 263/300 Read block 598 len 32768
26836 264/300 Write block 4074 len 16384 data: 8 10 8 7
26837 265/300 Write block 2382 len 20480 data: 5 5 5 5 5
26838 266/300 Read block 4648 len 24576
26839 267/300 Write block 1823 len 40960 data: 4 4 5 4 4 4 4 5 7 8
26840 268/300 Read block 1057 len 20480
26841 269/300 Write block 4250 len 16384 data: 7 7 8 8
26842 270/300 Read block 1940 len 4096
26843 271/300 Read block 1967 len 36864
26844 272/300 Read block 53 len 36864
26845 273/300 Write block 3427 len 4096 data: 5
26846 274/300 Read block 675 len 28672
26847 275/300 Read block 2073 len 12288
26848 276/300 Read block 3732 len 32768
26849 277/300 Read block 3985 len 24576
26850 278/300 Read block 2891 len 4096
26851 279/300 Read block 4651 len 24576
26852 280/300 Read block 1454 len 12288
26853 281/300 Read block 2814 len 28672
26854 282/300 Read block 123 len 36864
26855 283/300 Write block 4283 len 20480 data: 8 8 6 5 5
26856 284/300 Write block 3682 len 24576 data: 4 4 5 5 5 5
26857 285/300 Read block 1621 len 32768
26858 286/300 Read block 4689 len 32768
26859 287/300 Read block 2587 len 16384
26860 288/300 Read block 614 len 24576
26861 289/300 Flush
26862 290/300 Read block 1823 len 8192
26863 291/300 Read block 1126 len 4096
26864 292/300 Read block 1049 len 4096
26865 293/300 Read block 1774 len 36864
26866 294/300 Read block 3936 len 16384
26867 295/300 Write block 4461 len 16384 data: 6 6 7 7
26868 296/300 Flush
26869 297/300 Write block 910 len 24576 data: 6 3 3 3 3 4
26870 298/300 Write block 3038 len 8192 data: 2 2
26871 299/300 Read block 2668 len 28672
26872 300/300 Read block 1849 len 8192
26873 301/300 Write block 4548 len 20480 data: 10 10 8 6 6
26874 Main task has work to do, go find it
26875 got dsc Start(2)
26876 start 2
26877 Mar 07 18:18:26.635 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2dcd613c-69c0-4d0c-9867-ccac46cd65fb, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
26878 Replay: started 2, returned:Ok(())
26879 ----------------------------------------------------------------
26880 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26881 States: Active Active Offline
26882 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26883 7942 Acked Write 8 Done Done Sent false
26884 7943 Acked Flush 0 Done Done Sent false
26885 7944 Acked Write 4 Done Done Sent false
26886 7945 Acked Write 8 Done Done Sent false
26887 7946 Acked Read 4 Done Done Sent false
26888 7947 Acked Read 1 Done Done Sent false
26889 7948 Acked Write 2 Done Done Sent false
26890 7949 Acked Read 5 Done Done Sent false
26891 7950 Acked Write 10 Done Done Sent false
26892 7951 Acked Read 8 Done Done Sent false
26893 7952 Acked Read 5 Done Done Sent false
26894 7953 Acked Write 5 Done Done Sent false
26895 7954 Acked Write 7 Done Done Sent false
26896 7955 Acked Read 5 Done Done Sent false
26897 7956 Acked Read 3 Done Done Sent false
26898 7957 Acked Read 5 Done Done Sent false
26899 7958 Acked Read 8 Done Done Sent false
26900 7959 Acked Read 10 Done Done Sent false
26901 7960 Acked Read 7 Done Done Sent false
26902 7961 Acked Write 10 Done Done Sent false
26903 7962 Acked Write 8 Done Done Sent false
26904 7963 Acked Read 4 Done Done Sent false
26905 7964 Acked Read 7 Done Done Sent false
26906 7965 Acked Read 6 Done Done Sent false
26907 7966 Acked Read 10 Done Done Sent false
26908 7967 Acked Read 2 Done Done Sent false
26909 7968 Acked Flush 0 Done Done Sent false
26910 7969 Acked Read 1 Done Done Sent false
26911 7970 Acked Write 1 Done Done Sent false
26912 7971 Acked Write 1 Done Done Sent false
26913 7972 Acked Read 1 Done Done Sent false
26914 7973 Acked Write 4 Done Done Sent false
26915 7974 Acked Flush 0 Done Done Sent false
26916 7975 Acked Write 5 Done Done Sent false
26917 7976 Acked Write 4 Done Done Sent false
26918 7977 Acked Read 1 Done Done Sent false
26919 7978 Acked Write 5 Done Done Sent false
26920 7979 Acked Read 8 Done Done Sent false
26921 7980 Acked Write 7 Done Done Sent false
26922 7981 Acked Read 2 Done Done Sent false
26923 7982 Acked Write 6 Done Done Sent false
26924 7983 Acked Flush 0 Done Done Sent false
26925 7984 Acked Read 6 Done Done Sent false
26926 7985 Acked Write 3 Done Done Sent false
26927 7986 Acked Read 8 Done Done Sent false
26928 7987 Acked Write 5 Done Done Sent false
26929 7988 Acked Read 5 Done Done Sent false
26930 7989 Acked Write 1 Done Done Sent false
26931 7990 Acked Flush 0 Done Done Sent false
26932 7991 Acked Write 1 Done Done Sent false
26933 7992 Acked Read 8 Done Done Sent false
26934 7993 Acked Write 10 Done Done Sent false
26935 7994 Acked Write 5 Done Done Sent false
26936 7995 Acked Write 7 Done Done Sent false
26937 7996 Acked Flush 0 Done Done Sent false
26938 7997 Acked Write 2 Done Done Sent false
26939 7998 Acked Read 7 Done Done Sent false
26940 7999 Acked Write 8 Done Done Sent false
26941 8000 Acked Write 1 Done Done Sent false
26942 8001 Acked Read 10 Done Done Sent false
26943 8002 Acked Write 10 Done Done Sent false
26944 8003 Acked Write 1 Done Done Sent false
26945 8004 Acked Read 4 Done Done Sent false
26946 8005 Acked Read 8 Done Done Sent false
26947 8006 Acked Write 8 Done Done Sent false
26948 8007 Acked Read 6 Done Done Sent false
26949 8008 Acked Read 5 Done Done Sent false
26950 8009 Acked Read 6 Done Done Sent false
26951 8010 Acked Write 7 Done Done Sent false
26952 8011 Acked Read 4 Done Done Sent false
26953 8012 Acked Write 9 Done Done Sent false
26954 8013 Acked Write 7 Done Done Sent false
26955 8014 Acked Read 2 Done Done Sent false
26956 8015 Acked Read 6 Done Done Sent false
26957 8016 Acked Read 6 Done Done Sent false
26958 8017 Acked Write 7 Done Done Sent false
26959 8018 Acked Read 8 Done Done Sent false
26960 8019 Acked Write 6 Done Done Sent false
26961 8020 Acked Flush 0 Done Done Sent false
26962 8021 Acked Write 4 Done Done Sent false
26963 8022 Acked Read 5 Done Done Sent false
26964 8023 Acked Read 10 Done Done Sent false
26965 8024 Acked Flush 0 Done Done Sent false
26966 8025 Acked Write 4 Done Done Sent false
26967 8026 Acked Read 1 Done Done Sent false
26968 8027 Acked Read 10 Done Done Sent false
26969 8028 Acked Read 7 Done Done Sent false
26970 8029 Acked Write 3 Done Done Sent false
26971 8030 Acked Read 7 Done Done Sent false
26972 8031 Acked Read 2 Done Done Sent false
26973 8032 Acked Write 7 Done Done Sent false
26974 8033 Acked Read 1 Done Done Sent false
26975 8034 Acked Read 10 Done Done Sent false
26976 8035 Acked Flush 0 Done Done Sent false
26977 8036 Acked Read 7 Done Done Sent false
26978 8037 Acked Read 7 Done Done Sent false
26979 8038 Acked Read 4 Done Done Sent false
26980 8039 Acked Write 7 Done Done Sent false
26981 8040 Acked Read 4 Done Done Sent false
26982 8041 Acked Write 5 Done Done Sent false
26983 8042 Acked Read 8 Done Done Sent false
26984 8043 Acked Write 2 Done Done Sent false
26985 8044 Acked Write 10 Done Done Sent false
26986 8045 Acked Write 4 Done Done Sent false
26987 8046 Acked Read 8 Done Done Sent false
26988 8047 Acked Read 5 Done Done Sent false
26989 8048 Acked Read 3 Done Done Sent false
26990 8049 Acked Read 4 Done Done Sent false
26991 8050 Acked Flush 0 Done Done Sent false
26992 8051 Acked Write 2 Done Done Sent false
26993 8052 Acked Read 5 Done Done Sent false
26994 8053 Acked Read 1 Done Done Sent false
26995 8054 Acked Write 3 Done Done Sent false
26996 8055 Acked Flush 0 Done Done Sent false
26997 8056 Acked Read 8 Done Done Sent false
26998 8057 Acked Read 8 Done Done Sent false
26999 8058 Acked Read 7 Done Done Sent false
27000 8059 Acked Write 8 Done Done Sent false
27001 8060 Acked Write 7 Done Done Sent false
27002 8061 Acked Flush 0 Done Done Sent false
27003 8062 Acked Flush 0 Done Done Sent false
27004 8063 Acked Read 9 Done Done Sent false
27005 8064 Acked Read 6 Done Done Sent false
27006 8065 Acked Write 8 Done Done Sent false
27007 8066 Acked Flush 0 Done Done Sent false
27008 8067 Acked Read 2 Done Done Sent false
27009 8068 Acked Write 2 Done Done Sent false
27010 8069 Acked Write 7 Done Done Sent false
27011 8070 Acked Read 6 Done Done Sent false
27012 8071 Acked Flush 0 Done Done Sent false
27013 8072 Acked Read 8 Done Done Sent false
27014 8073 Acked Write 2 Done Done Sent false
27015 8074 Acked Write 2 Done Done Sent false
27016 8075 Acked Write 10 Done Done Sent false
27017 8076 Acked Write 6 Done Done Sent false
27018 8077 Acked Read 2 Done Done Sent false
27019 8078 Acked Write 4 Done Done Sent false
27020 8079 Acked Write 9 Done Done Sent false
27021 8080 Acked Read 7 Done Done Sent false
27022 8081 Acked Read 8 Done Done Sent false
27023 8082 Acked Read 1 Done Done Sent false
27024 8083 Acked Read 2 Done Done Sent false
27025 8084 Acked Read 8 Done Done Sent false
27026 8085 Acked Read 10 Done Done Sent false
27027 8086 Acked Read 2 Done Done Sent false
27028 8087 Acked Write 6 Done Done Sent false
27029 8088 Acked Read 10 Done Done Sent false
27030 8089 Acked Flush 0 Done Done Sent false
27031 8090 Acked Read 2 Done Done Sent false
27032 8091 Acked Write 8 Done Done Sent false
27033 8092 Acked Write 3 Done Done Sent false
27034 8093 Acked Flush 0 Done Done Sent false
27035 8094 Acked Read 10 Done Done Sent false
27036 8095 Acked Write 1 Done Done Sent false
27037 8096 Acked Read 7 Done Done Sent false
27038 8097 Acked Read 3 Done Done Sent false
27039 8098 Acked Flush 0 Done Done Sent false
27040 8099 Acked Write 10 Done Done Sent false
27041 8100 Acked Read 8 Done Done Sent false
27042 8101 Acked Write 5 Done Done Sent false
27043 8102 Acked Read 7 Done Done Sent false
27044 8103 Acked Read 3 Done Done Sent false
27045 8104 Acked Read 7 Done Done Sent false
27046 8105 Acked Read 6 Done Done Sent false
27047 8106 Acked Read 5 Done Done Sent false
27048 8107 Acked Write 9 Done Done Sent false
27049 8108 Acked Write 5 Done Done Sent false
27050 8109 Acked Write 1 Done Done Sent false
27051 8110 Acked Read 7 Done Done Sent false
27052 8111 Acked Read 2 Done Done Sent false
27053 8112 Acked Write 2 Done Done Sent false
27054 8113 Acked Write 9 Done Done Sent false
27055 8114 Acked Flush 0 Done Done Sent false
27056 8115 Acked Read 8 Done Done Sent false
27057 8116 Acked Read 1 Done Done Sent false
27058 8117 Acked Write 8 Done Done Sent false
27059 8118 Acked Read 1 Done Done Sent false
27060 8119 Acked Write 7 Done Done Sent false
27061 8120 Acked Read 5 Done Done Sent false
27062 8121 Acked Write 3 Done Done Sent false
27063 8122 Acked Write 10 Done Done Sent false
27064 8123 Acked Read 5 Done Done Sent false
27065 8124 Acked Read 9 Done Done Sent false
27066 8125 Acked Read 2 Done Done Sent false
27067 8126 Acked Write 9 Done Done Sent false
27068 8127 Acked Read 4 Done Done Sent false
27069 8128 Acked Write 3 Done Done Sent false
27070 8129 Acked Read 8 Done Done Sent false
27071 8130 Acked Write 7 Done Done Sent false
27072 8131 Acked Read 4 Done Done Sent false
27073 8132 Acked Read 4 Done Done Sent false
27074 8133 Acked Read 7 Done Done Sent false
27075 8134 Acked Write 9 Done Done Sent false
27076 8135 Acked Flush 0 Done Done Sent false
27077 8136 Acked Read 8 Done Done Sent false
27078 8137 Acked Write 10 Done Done Sent false
27079 8138 Acked Write 4 Done Done Sent false
27080 8139 Acked Write 3 Done Done Sent false
27081 8140 Acked Write 2 Done Done Sent false
27082 8141 Acked Write 8 Done Done Sent false
27083 8142 Acked Write 3 Done Done Sent false
27084 8143 Acked Write 7 Done Done Sent false
27085 8144 Acked Write 4 Done Done Sent false
27086 8145 Acked Write 6 Done Done Sent false
27087 8146 Acked Flush 0 Done Done Sent false
27088 8147 Acked Write 1 Done Done Sent false
27089 8148 Acked Read 9 Done Done Sent false
27090 8149 Acked Read 10 Done Done Sent false
27091 8150 Acked Read 8 Done Done Sent false
27092 8151 Acked Flush 0 Done Done Sent false
27093 8152 Acked Write 1 Done Done Sent false
27094 8153 Acked Read 9 Done Done Sent false
27095 8154 Acked Write 1 Done Done Sent false
27096 8155 Acked Write 8 Done Done Sent false
27097 8156 Acked Read 10 Done Done Sent false
27098 8157 Acked Read 3 Done Done Sent false
27099 8158 Acked Read 10 Done Done Sent false
27100 8159 Acked Write 3 Done Done Sent false
27101 8160 Acked Read 10 Done Done Sent false
27102 8161 Acked Write 8 Done Done Sent false
27103 8162 Acked Read 5 Done Done Sent false
27104 8163 Acked Read 5 Done Done Sent false
27105 8164 Acked Write 4 Done Done Sent false
27106 8165 Acked Read 2 Done Done Sent false
27107 8166 Acked Read 10 Done Done Sent false
27108 8167 Acked Write 5 Done Done Sent false
27109 8168 Acked Read 10 Done Done Sent false
27110 8169 Acked Read 3 Done Done Sent false
27111 8170 Acked Write 2 Done Done Sent false
27112 8171 Acked Read 1 Done Done Sent false
27113 8172 Acked Write 9 Done Done Sent false
27114 8173 Acked Write 3 Done Done Sent false
27115 8174 Acked Read 5 Done Done Sent false
27116 8175 Acked Read 9 Done Done Sent false
27117 8176 Acked Flush 0 Done Done Sent false
27118 8177 Acked Flush 0 Done Done Sent false
27119 8178 Acked Read 7 Done Done Sent false
27120 8179 Acked Write 4 Done Done Sent false
27121 8180 Acked Write 5 Done Done Sent false
27122 8181 Acked Write 5 Done Done Sent false
27123 8182 Acked Read 6 Done Done Sent false
27124 8183 Acked Write 5 Done Done Sent false
27125 8184 Acked Write 4 Done Done Sent false
27126 8185 Acked Read 2 Done Done Sent false
27127 8186 Acked Write 8 Done Done Sent false
27128 8187 Acked Read 2 Done Done Sent false
27129 8188 Acked Write 9 Done Done Sent false
27130 8189 Acked Write 6 Done Done Sent false
27131 8190 Acked Read 4 Done Done Sent false
27132 8191 Acked Write 10 Done Done Sent false
27133 8192 Acked Read 7 Done Done Sent false
27134 8193 Acked Flush 0 Done Done Sent false
27135 8194 Acked Write 5 Done Done Sent false
27136 8195 Acked Write 10 Done Done Sent false
27137 8196 Acked Write 9 Done Done Sent false
27138 8197 Acked Write 10 Done Done Sent false
27139 8198 Acked Write 7 Done Done Sent false
27140 8199 Acked Write 4 Done Done Sent false
27141 8200 Acked Write 1 Done Done Sent false
27142 8201 Acked Write 6 Done Done Sent false
27143 8202 Acked Read 1 Done Done Sent false
27144 8203 Acked Read 2 Done Done Sent false
27145 8204 Acked Read 8 Done Done Sent false
27146 8205 Acked Write 4 Done Done Sent false
27147 8206 Acked Write 5 Done Done Sent false
27148 8207 Acked Read 6 Done Done Sent false
27149 8208 Acked Write 10 Done Done Sent false
27150 8209 Acked Read 5 Done Done Sent false
27151 8210 Acked Write 4 Done Done Sent false
27152 8211 Acked Read 1 Done Done Sent false
27153 8212 Acked Read 9 Done Done Sent false
27154 8213 Acked Read 9 Done Done Sent false
27155 8214 Acked Write 1 Done Done Sent false
27156 8215 Acked Read 7 Done Done Sent false
27157 8216 Acked Read 3 Done Done Sent false
27158 8217 Acked Read 8 Done Done Sent false
27159 8218 Acked Read 6 Done Done Sent false
27160 8219 Acked Read 1 Done Done Sent false
27161 8220 Acked Read 6 Done Done Sent false
27162 8221 Acked Read 3 Done Done Sent false
27163 8222 Acked Read 7 Done Done Sent false
27164 8223 Acked Read 9 Done Done Sent false
27165 8224 Acked Write 5 Done Done Sent false
27166 8225 Acked Write 6 Done Done Sent false
27167 8226 Acked Read 8 Done Done Sent false
27168 8227 Acked Read 8 Done Done Sent false
27169 8228 Acked Read 4 Done Done Sent false
27170 8229 Acked Read 6 Done Done Sent false
27171 8230 Acked Flush 0 Done Done Sent false
27172 8231 Acked Read 2 Done Done Sent false
27173 8232 Acked Read 1 Done Done Sent false
27174 8233 Acked Read 1 Done Done Sent false
27175 8234 Acked Read 9 Done Done Sent false
27176 8235 Acked Read 4 Done Done Sent false
27177 8236 Acked Write 4 Done Done Sent false
27178 8237 Acked Flush 0 Done Done Sent false
27179 8238 Acked Write 6 Done Done Sent false
27180 8239 Acked Write 2 Done Done Sent false
27181 8240 Acked Read 7 Done Done Sent false
27182 8241 Acked Read 2 Done Done Sent false
27183 8242 Acked Write 5 Done Done Sent false
27184 STATES DS:0 DS:1 DS:2 TOTAL
27185 Sent 0 0 301 301
27186 Done 301 301 0 602
27187 Skipped 0 0 0 0
27188 Error 0 0 0 0
27189 Last Flush: Some(JobId(8237)) Some(JobId(8237)) Some(JobId(7941))
27190 Downstairs last five completed: 7941 7940 7939 7938 7937
27191 Upstairs last five completed: 8242 8241 8240 8239 8238
27192 CLIENT: Up:0 ds:301 act:2
271932025-03-07T18:18:27.538ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
271942025-03-07T18:18:28.540ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
271952025-03-07T18:18:29.541ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27196 ----------------------------------------------------------------
27197 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27198 States: Active Active Offline
27199 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27200 7942 Acked Write 8 Done Done Sent false
27201 7943 Acked Flush 0 Done Done Sent false
27202 7944 Acked Write 4 Done Done Sent false
27203 7945 Acked Write 8 Done Done Sent false
27204 7946 Acked Read 4 Done Done Sent false
27205 7947 Acked Read 1 Done Done Sent false
27206 7948 Acked Write 2 Done Done Sent false
27207 7949 Acked Read 5 Done Done Sent false
27208 7950 Acked Write 10 Done Done Sent false
27209 7951 Acked Read 8 Done Done Sent false
27210 7952 Acked Read 5 Done Done Sent false
27211 7953 Acked Write 5 Done Done Sent false
27212 7954 Acked Write 7 Done Done Sent false
27213 7955 Acked Read 5 Done Done Sent false
27214 7956 Acked Read 3 Done Done Sent false
27215 7957 Acked Read 5 Done Done Sent false
27216 7958 Acked Read 8 Done Done Sent false
27217 7959 Acked Read 10 Done Done Sent false
27218 7960 Acked Read 7 Done Done Sent false
27219 7961 Acked Write 10 Done Done Sent false
27220 7962 Acked Write 8 Done Done Sent false
27221 7963 Acked Read 4 Done Done Sent false
27222 7964 Acked Read 7 Done Done Sent false
27223 7965 Acked Read 6 Done Done Sent false
27224 7966 Acked Read 10 Done Done Sent false
27225 7967 Acked Read 2 Done Done Sent false
27226 7968 Acked Flush 0 Done Done Sent false
27227 7969 Acked Read 1 Done Done Sent false
27228 7970 Acked Write 1 Done Done Sent false
27229 7971 Acked Write 1 Done Done Sent false
27230 7972 Acked Read 1 Done Done Sent false
27231 7973 Acked Write 4 Done Done Sent false
27232 7974 Acked Flush 0 Done Done Sent false
27233 7975 Acked Write 5 Done Done Sent false
27234 7976 Acked Write 4 Done Done Sent false
27235 7977 Acked Read 1 Done Done Sent false
27236 7978 Acked Write 5 Done Done Sent false
27237 7979 Acked Read 8 Done Done Sent false
27238 7980 Acked Write 7 Done Done Sent false
27239 7981 Acked Read 2 Done Done Sent false
27240 7982 Acked Write 6 Done Done Sent false
27241 7983 Acked Flush 0 Done Done Sent false
27242 7984 Acked Read 6 Done Done Sent false
27243 7985 Acked Write 3 Done Done Sent false
27244 7986 Acked Read 8 Done Done Sent false
27245 7987 Acked Write 5 Done Done Sent false
27246 7988 Acked Read 5 Done Done Sent false
27247 7989 Acked Write 1 Done Done Sent false
27248 7990 Acked Flush 0 Done Done Sent false
27249 7991 Acked Write 1 Done Done Sent false
27250 7992 Acked Read 8 Done Done Sent false
27251 7993 Acked Write 10 Done Done Sent false
27252 7994 Acked Write 5 Done Done Sent false
27253 7995 Acked Write 7 Done Done Sent false
27254 7996 Acked Flush 0 Done Done Sent false
27255 7997 Acked Write 2 Done Done Sent false
27256 7998 Acked Read 7 Done Done Sent false
27257 7999 Acked Write 8 Done Done Sent false
27258 8000 Acked Write 1 Done Done Sent false
27259 8001 Acked Read 10 Done Done Sent false
27260 8002 Acked Write 10 Done Done Sent false
27261 8003 Acked Write 1 Done Done Sent false
27262 8004 Acked Read 4 Done Done Sent false
27263 8005 Acked Read 8 Done Done Sent false
27264 8006 Acked Write 8 Done Done Sent false
27265 8007 Acked Read 6 Done Done Sent false
27266 8008 Acked Read 5 Done Done Sent false
27267 8009 Acked Read 6 Done Done Sent false
27268 8010 Acked Write 7 Done Done Sent false
27269 8011 Acked Read 4 Done Done Sent false
27270 8012 Acked Write 9 Done Done Sent false
27271 8013 Acked Write 7 Done Done Sent false
27272 8014 Acked Read 2 Done Done Sent false
27273 8015 Acked Read 6 Done Done Sent false
27274 8016 Acked Read 6 Done Done Sent false
27275 8017 Acked Write 7 Done Done Sent false
27276 8018 Acked Read 8 Done Done Sent false
27277 8019 Acked Write 6 Done Done Sent false
27278 8020 Acked Flush 0 Done Done Sent false
27279 8021 Acked Write 4 Done Done Sent false
27280 8022 Acked Read 5 Done Done Sent false
27281 8023 Acked Read 10 Done Done Sent false
27282 8024 Acked Flush 0 Done Done Sent false
27283 8025 Acked Write 4 Done Done Sent false
27284 8026 Acked Read 1 Done Done Sent false
27285 8027 Acked Read 10 Done Done Sent false
27286 8028 Acked Read 7 Done Done Sent false
27287 8029 Acked Write 3 Done Done Sent false
27288 8030 Acked Read 7 Done Done Sent false
27289 8031 Acked Read 2 Done Done Sent false
27290 8032 Acked Write 7 Done Done Sent false
27291 8033 Acked Read 1 Done Done Sent false
27292 8034 Acked Read 10 Done Done Sent false
27293 8035 Acked Flush 0 Done Done Sent false
27294 8036 Acked Read 7 Done Done Sent false
27295 8037 Acked Read 7 Done Done Sent false
27296 8038 Acked Read 4 Done Done Sent false
27297 8039 Acked Write 7 Done Done Sent false
27298 8040 Acked Read 4 Done Done Sent false
27299 8041 Acked Write 5 Done Done Sent false
27300 8042 Acked Read 8 Done Done Sent false
27301 8043 Acked Write 2 Done Done Sent false
27302 8044 Acked Write 10 Done Done Sent false
27303 8045 Acked Write 4 Done Done Sent false
27304 8046 Acked Read 8 Done Done Sent false
27305 8047 Acked Read 5 Done Done Sent false
27306 8048 Acked Read 3 Done Done Sent false
27307 8049 Acked Read 4 Done Done Sent false
27308 8050 Acked Flush 0 Done Done Sent false
27309 8051 Acked Write 2 Done Done Sent false
27310 8052 Acked Read 5 Done Done Sent false
27311 8053 Acked Read 1 Done Done Sent false
27312 8054 Acked Write 3 Done Done Sent false
27313 8055 Acked Flush 0 Done Done Sent false
27314 8056 Acked Read 8 Done Done Sent false
27315 8057 Acked Read 8 Done Done Sent false
27316 8058 Acked Read 7 Done Done Sent false
27317 8059 Acked Write 8 Done Done Sent false
27318 8060 Acked Write 7 Done Done Sent false
27319 8061 Acked Flush 0 Done Done Sent false
27320 8062 Acked Flush 0 Done Done Sent false
27321 8063 Acked Read 9 Done Done Sent false
27322 8064 Acked Read 6 Done Done Sent false
27323 8065 Acked Write 8 Done Done Sent false
27324 8066 Acked Flush 0 Done Done Sent false
27325 8067 Acked Read 2 Done Done Sent false
27326 8068 Acked Write 2 Done Done Sent false
27327 8069 Acked Write 7 Done Done Sent false
27328 8070 Acked Read 6 Done Done Sent false
27329 8071 Acked Flush 0 Done Done Sent false
27330 8072 Acked Read 8 Done Done Sent false
27331 8073 Acked Write 2 Done Done Sent false
27332 8074 Acked Write 2 Done Done Sent false
27333 8075 Acked Write 10 Done Done Sent false
27334 8076 Acked Write 6 Done Done Sent false
27335 8077 Acked Read 2 Done Done Sent false
27336 8078 Acked Write 4 Done Done Sent false
27337 8079 Acked Write 9 Done Done Sent false
27338 8080 Acked Read 7 Done Done Sent false
27339 8081 Acked Read 8 Done Done Sent false
27340 8082 Acked Read 1 Done Done Sent false
27341 8083 Acked Read 2 Done Done Sent false
27342 8084 Acked Read 8 Done Done Sent false
27343 8085 Acked Read 10 Done Done Sent false
27344 8086 Acked Read 2 Done Done Sent false
27345 8087 Acked Write 6 Done Done Sent false
27346 8088 Acked Read 10 Done Done Sent false
27347 8089 Acked Flush 0 Done Done Sent false
27348 8090 Acked Read 2 Done Done Sent false
27349 8091 Acked Write 8 Done Done Sent false
27350 8092 Acked Write 3 Done Done Sent false
27351 8093 Acked Flush 0 Done Done Sent false
27352 8094 Acked Read 10 Done Done Sent false
27353 8095 Acked Write 1 Done Done Sent false
27354 8096 Acked Read 7 Done Done Sent false
27355 8097 Acked Read 3 Done Done Sent false
27356 8098 Acked Flush 0 Done Done Sent false
27357 8099 Acked Write 10 Done Done Sent false
27358 8100 Acked Read 8 Done Done Sent false
27359 8101 Acked Write 5 Done Done Sent false
27360 8102 Acked Read 7 Done Done Sent false
27361 8103 Acked Read 3 Done Done Sent false
27362 8104 Acked Read 7 Done Done Sent false
27363 8105 Acked Read 6 Done Done Sent false
27364 8106 Acked Read 5 Done Done Sent false
27365 8107 Acked Write 9 Done Done Sent false
27366 8108 Acked Write 5 Done Done Sent false
27367 8109 Acked Write 1 Done Done Sent false
27368 8110 Acked Read 7 Done Done Sent false
27369 8111 Acked Read 2 Done Done Sent false
27370 8112 Acked Write 2 Done Done Sent false
27371 8113 Acked Write 9 Done Done Sent false
27372 8114 Acked Flush 0 Done Done Sent false
27373 8115 Acked Read 8 Done Done Sent false
27374 8116 Acked Read 1 Done Done Sent false
27375 8117 Acked Write 8 Done Done Sent false
27376 8118 Acked Read 1 Done Done Sent false
27377 8119 Acked Write 7 Done Done Sent false
27378 8120 Acked Read 5 Done Done Sent false
27379 8121 Acked Write 3 Done Done Sent false
27380 8122 Acked Write 10 Done Done Sent false
27381 8123 Acked Read 5 Done Done Sent false
27382 8124 Acked Read 9 Done Done Sent false
27383 8125 Acked Read 2 Done Done Sent false
27384 8126 Acked Write 9 Done Done Sent false
27385 8127 Acked Read 4 Done Done Sent false
27386 8128 Acked Write 3 Done Done Sent false
27387 8129 Acked Read 8 Done Done Sent false
27388 8130 Acked Write 7 Done Done Sent false
27389 8131 Acked Read 4 Done Done Sent false
27390 8132 Acked Read 4 Done Done Sent false
27391 8133 Acked Read 7 Done Done Sent false
27392 8134 Acked Write 9 Done Done Sent false
27393 8135 Acked Flush 0 Done Done Sent false
27394 8136 Acked Read 8 Done Done Sent false
27395 8137 Acked Write 10 Done Done Sent false
27396 8138 Acked Write 4 Done Done Sent false
27397 8139 Acked Write 3 Done Done Sent false
27398 8140 Acked Write 2 Done Done Sent false
27399 8141 Acked Write 8 Done Done Sent false
27400 8142 Acked Write 3 Done Done Sent false
27401 8143 Acked Write 7 Done Done Sent false
27402 8144 Acked Write 4 Done Done Sent false
27403 8145 Acked Write 6 Done Done Sent false
27404 8146 Acked Flush 0 Done Done Sent false
27405 8147 Acked Write 1 Done Done Sent false
27406 8148 Acked Read 9 Done Done Sent false
27407 8149 Acked Read 10 Done Done Sent false
27408 8150 Acked Read 8 Done Done Sent false
27409 8151 Acked Flush 0 Done Done Sent false
27410 8152 Acked Write 1 Done Done Sent false
27411 8153 Acked Read 9 Done Done Sent false
27412 8154 Acked Write 1 Done Done Sent false
27413 8155 Acked Write 8 Done Done Sent false
27414 8156 Acked Read 10 Done Done Sent false
27415 8157 Acked Read 3 Done Done Sent false
27416 8158 Acked Read 10 Done Done Sent false
27417 8159 Acked Write 3 Done Done Sent false
27418 8160 Acked Read 10 Done Done Sent false
27419 8161 Acked Write 8 Done Done Sent false
27420 8162 Acked Read 5 Done Done Sent false
27421 8163 Acked Read 5 Done Done Sent false
27422 8164 Acked Write 4 Done Done Sent false
27423 8165 Acked Read 2 Done Done Sent false
27424 8166 Acked Read 10 Done Done Sent false
27425 8167 Acked Write 5 Done Done Sent false
27426 8168 Acked Read 10 Done Done Sent false
27427 8169 Acked Read 3 Done Done Sent false
27428 8170 Acked Write 2 Done Done Sent false
27429 8171 Acked Read 1 Done Done Sent false
27430 8172 Acked Write 9 Done Done Sent false
27431 8173 Acked Write 3 Done Done Sent false
27432 8174 Acked Read 5 Done Done Sent false
27433 8175 Acked Read 9 Done Done Sent false
27434 8176 Acked Flush 0 Done Done Sent false
27435 8177 Acked Flush 0 Done Done Sent false
27436 8178 Acked Read 7 Done Done Sent false
27437 8179 Acked Write 4 Done Done Sent false
27438 8180 Acked Write 5 Done Done Sent false
27439 8181 Acked Write 5 Done Done Sent false
27440 8182 Acked Read 6 Done Done Sent false
27441 8183 Acked Write 5 Done Done Sent false
27442 8184 Acked Write 4 Done Done Sent false
27443 8185 Acked Read 2 Done Done Sent false
27444 8186 Acked Write 8 Done Done Sent false
27445 8187 Acked Read 2 Done Done Sent false
27446 8188 Acked Write 9 Done Done Sent false
27447 8189 Acked Write 6 Done Done Sent false
27448 8190 Acked Read 4 Done Done Sent false
27449 8191 Acked Write 10 Done Done Sent false
27450 8192 Acked Read 7 Done Done Sent false
27451 8193 Acked Flush 0 Done Done Sent false
27452 8194 Acked Write 5 Done Done Sent false
27453 8195 Acked Write 10 Done Done Sent false
27454 8196 Acked Write 9 Done Done Sent false
27455 8197 Acked Write 10 Done Done Sent false
27456 8198 Acked Write 7 Done Done Sent false
27457 8199 Acked Write 4 Done Done Sent false
27458 8200 Acked Write 1 Done Done Sent false
27459 8201 Acked Write 6 Done Done Sent false
27460 8202 Acked Read 1 Done Done Sent false
27461 8203 Acked Read 2 Done Done Sent false
27462 8204 Acked Read 8 Done Done Sent false
27463 8205 Acked Write 4 Done Done Sent false
27464 8206 Acked Write 5 Done Done Sent false
27465 8207 Acked Read 6 Done Done Sent false
27466 8208 Acked Write 10 Done Done Sent false
27467 8209 Acked Read 5 Done Done Sent false
27468 8210 Acked Write 4 Done Done Sent false
27469 8211 Acked Read 1 Done Done Sent false
27470 8212 Acked Read 9 Done Done Sent false
27471 8213 Acked Read 9 Done Done Sent false
27472 8214 Acked Write 1 Done Done Sent false
27473 8215 Acked Read 7 Done Done Sent false
27474 8216 Acked Read 3 Done Done Sent false
27475 8217 Acked Read 8 Done Done Sent false
27476 8218 Acked Read 6 Done Done Sent false
27477 8219 Acked Read 1 Done Done Sent false
27478 8220 Acked Read 6 Done Done Sent false
27479 8221 Acked Read 3 Done Done Sent false
27480 8222 Acked Read 7 Done Done Sent false
27481 8223 Acked Read 9 Done Done Sent false
27482 8224 Acked Write 5 Done Done Sent false
27483 8225 Acked Write 6 Done Done Sent false
27484 8226 Acked Read 8 Done Done Sent false
27485 8227 Acked Read 8 Done Done Sent false
27486 8228 Acked Read 4 Done Done Sent false
27487 8229 Acked Read 6 Done Done Sent false
27488 8230 Acked Flush 0 Done Done Sent false
27489 8231 Acked Read 2 Done Done Sent false
27490 8232 Acked Read 1 Done Done Sent false
27491 8233 Acked Read 1 Done Done Sent false
27492 8234 Acked Read 9 Done Done Sent false
27493 8235 Acked Read 4 Done Done Sent false
27494 8236 Acked Write 4 Done Done Sent false
27495 8237 Acked Flush 0 Done Done Sent false
27496 8238 Acked Write 6 Done Done Sent false
27497 8239 Acked Write 2 Done Done Sent false
27498 8240 Acked Read 7 Done Done Sent false
27499 8241 Acked Read 2 Done Done Sent false
27500 8242 Acked Write 5 Done Done Sent false
27501 STATES DS:0 DS:1 DS:2 TOTAL
27502 Sent 0 0 301 301
27503 Done 301 301 0 602
27504 Skipped 0 0 0 0
27505 Error 0 0 0 0
27506 Last Flush: Some(JobId(8237)) Some(JobId(8237)) Some(JobId(7941))
27507 Downstairs last five completed: 7941 7940 7939 7938 7937
27508 Upstairs last five completed: 8242 8241 8240 8239 8238
27509 CLIENT: Up:0 ds:301 act:2
275102025-03-07T18:18:31.545ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275112025-03-07T18:18:32.386ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275122025-03-07T18:18:32.386ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275132025-03-07T18:18:32.386ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275142025-03-07T18:18:32.386ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275152025-03-07T18:18:32.386ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27516 [8830] got start action
27517 [8830] I am going to restart
27518 Starting downstairs at port 8830
27519 Make output file at "/tmp/dsc/downstairs-8830.txt"
27520 [8830][2] reports Starting
27521 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1232)
27522 [8830][2] reports Running
275232025-03-07T18:18:32.888ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275242025-03-07T18:18:33.549ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
275252025-03-07T18:18:34.551ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27526 ----------------------------------------------------------------
27527 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27528 States: Active Active Offline
27529 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27530 7942 Acked Write 8 Done Done Sent false
27531 7943 Acked Flush 0 Done Done Sent false
27532 7944 Acked Write 4 Done Done Sent false
27533 7945 Acked Write 8 Done Done Sent false
27534 7946 Acked Read 4 Done Done Sent false
27535 7947 Acked Read 1 Done Done Sent false
27536 7948 Acked Write 2 Done Done Sent false
27537 7949 Acked Read 5 Done Done Sent false
27538 7950 Acked Write 10 Done Done Sent false
27539 7951 Acked Read 8 Done Done Sent false
27540 7952 Acked Read 5 Done Done Sent false
27541 7953 Acked Write 5 Done Done Sent false
27542 7954 Acked Write 7 Done Done Sent false
27543 7955 Acked Read 5 Done Done Sent false
27544 7956 Acked Read 3 Done Done Sent false
27545 7957 Acked Read 5 Done Done Sent false
27546 7958 Acked Read 8 Done Done Sent false
27547 7959 Acked Read 10 Done Done Sent false
27548 7960 Acked Read 7 Done Done Sent false
27549 7961 Acked Write 10 Done Done Sent false
27550 7962 Acked Write 8 Done Done Sent false
27551 7963 Acked Read 4 Done Done Sent false
27552 7964 Acked Read 7 Done Done Sent false
27553 7965 Acked Read 6 Done Done Sent false
27554 7966 Acked Read 10 Done Done Sent false
27555 7967 Acked Read 2 Done Done Sent false
27556 7968 Acked Flush 0 Done Done Sent false
27557 7969 Acked Read 1 Done Done Sent false
27558 7970 Acked Write 1 Done Done Sent false
27559 7971 Acked Write 1 Done Done Sent false
27560 7972 Acked Read 1 Done Done Sent false
27561 7973 Acked Write 4 Done Done Sent false
27562 7974 Acked Flush 0 Done Done Sent false
27563 7975 Acked Write 5 Done Done Sent false
27564 7976 Acked Write 4 Done Done Sent false
27565 7977 Acked Read 1 Done Done Sent false
27566 7978 Acked Write 5 Done Done Sent false
27567 7979 Acked Read 8 Done Done Sent false
27568 7980 Acked Write 7 Done Done Sent false
27569 7981 Acked Read 2 Done Done Sent false
27570 7982 Acked Write 6 Done Done Sent false
27571 7983 Acked Flush 0 Done Done Sent false
27572 7984 Acked Read 6 Done Done Sent false
27573 7985 Acked Write 3 Done Done Sent false
27574 7986 Acked Read 8 Done Done Sent false
27575 7987 Acked Write 5 Done Done Sent false
27576 7988 Acked Read 5 Done Done Sent false
27577 7989 Acked Write 1 Done Done Sent false
27578 7990 Acked Flush 0 Done Done Sent false
27579 7991 Acked Write 1 Done Done Sent false
27580 7992 Acked Read 8 Done Done Sent false
27581 7993 Acked Write 10 Done Done Sent false
27582 7994 Acked Write 5 Done Done Sent false
27583 7995 Acked Write 7 Done Done Sent false
27584 7996 Acked Flush 0 Done Done Sent false
27585 7997 Acked Write 2 Done Done Sent false
27586 7998 Acked Read 7 Done Done Sent false
27587 7999 Acked Write 8 Done Done Sent false
27588 8000 Acked Write 1 Done Done Sent false
27589 8001 Acked Read 10 Done Done Sent false
27590 8002 Acked Write 10 Done Done Sent false
27591 8003 Acked Write 1 Done Done Sent false
27592 8004 Acked Read 4 Done Done Sent false
27593 8005 Acked Read 8 Done Done Sent false
27594 8006 Acked Write 8 Done Done Sent false
27595 8007 Acked Read 6 Done Done Sent false
27596 8008 Acked Read 5 Done Done Sent false
27597 8009 Acked Read 6 Done Done Sent false
27598 8010 Acked Write 7 Done Done Sent false
27599 8011 Acked Read 4 Done Done Sent false
27600 8012 Acked Write 9 Done Done Sent false
27601 8013 Acked Write 7 Done Done Sent false
27602 8014 Acked Read 2 Done Done Sent false
27603 8015 Acked Read 6 Done Done Sent false
27604 8016 Acked Read 6 Done Done Sent false
27605 8017 Acked Write 7 Done Done Sent false
27606 8018 Acked Read 8 Done Done Sent false
27607 8019 Acked Write 6 Done Done Sent false
27608 8020 Acked Flush 0 Done Done Sent false
27609 8021 Acked Write 4 Done Done Sent false
27610 8022 Acked Read 5 Done Done Sent false
27611 8023 Acked Read 10 Done Done Sent false
27612 8024 Acked Flush 0 Done Done Sent false
27613 8025 Acked Write 4 Done Done Sent false
27614 8026 Acked Read 1 Done Done Sent false
27615 8027 Acked Read 10 Done Done Sent false
27616 8028 Acked Read 7 Done Done Sent false
27617 8029 Acked Write 3 Done Done Sent false
27618 8030 Acked Read 7 Done Done Sent false
27619 8031 Acked Read 2 Done Done Sent false
27620 8032 Acked Write 7 Done Done Sent false
27621 8033 Acked Read 1 Done Done Sent false
27622 8034 Acked Read 10 Done Done Sent false
27623 8035 Acked Flush 0 Done Done Sent false
27624 8036 Acked Read 7 Done Done Sent false
27625 8037 Acked Read 7 Done Done Sent false
27626 8038 Acked Read 4 Done Done Sent false
27627 8039 Acked Write 7 Done Done Sent false
27628 8040 Acked Read 4 Done Done Sent false
27629 8041 Acked Write 5 Done Done Sent false
27630 8042 Acked Read 8 Done Done Sent false
27631 8043 Acked Write 2 Done Done Sent false
27632 8044 Acked Write 10 Done Done Sent false
27633 8045 Acked Write 4 Done Done Sent false
27634 8046 Acked Read 8 Done Done Sent false
27635 8047 Acked Read 5 Done Done Sent false
27636 8048 Acked Read 3 Done Done Sent false
27637 8049 Acked Read 4 Done Done Sent false
27638 8050 Acked Flush 0 Done Done Sent false
27639 8051 Acked Write 2 Done Done Sent false
27640 8052 Acked Read 5 Done Done Sent false
27641 8053 Acked Read 1 Done Done Sent false
27642 8054 Acked Write 3 Done Done Sent false
27643 8055 Acked Flush 0 Done Done Sent false
27644 8056 Acked Read 8 Done Done Sent false
27645 8057 Acked Read 8 Done Done Sent false
27646 8058 Acked Read 7 Done Done Sent false
27647 8059 Acked Write 8 Done Done Sent false
27648 8060 Acked Write 7 Done Done Sent false
27649 8061 Acked Flush 0 Done Done Sent false
27650 8062 Acked Flush 0 Done Done Sent false
27651 8063 Acked Read 9 Done Done Sent false
27652 8064 Acked Read 6 Done Done Sent false
27653 8065 Acked Write 8 Done Done Sent false
27654 8066 Acked Flush 0 Done Done Sent false
27655 8067 Acked Read 2 Done Done Sent false
27656 8068 Acked Write 2 Done Done Sent false
27657 8069 Acked Write 7 Done Done Sent false
27658 8070 Acked Read 6 Done Done Sent false
27659 8071 Acked Flush 0 Done Done Sent false
27660 8072 Acked Read 8 Done Done Sent false
27661 8073 Acked Write 2 Done Done Sent false
27662 8074 Acked Write 2 Done Done Sent false
27663 8075 Acked Write 10 Done Done Sent false
27664 8076 Acked Write 6 Done Done Sent false
27665 8077 Acked Read 2 Done Done Sent false
27666 8078 Acked Write 4 Done Done Sent false
27667 8079 Acked Write 9 Done Done Sent false
27668 8080 Acked Read 7 Done Done Sent false
27669 8081 Acked Read 8 Done Done Sent false
27670 8082 Acked Read 1 Done Done Sent false
27671 8083 Acked Read 2 Done Done Sent false
27672 8084 Acked Read 8 Done Done Sent false
27673 8085 Acked Read 10 Done Done Sent false
27674 8086 Acked Read 2 Done Done Sent false
27675 8087 Acked Write 6 Done Done Sent false
27676 8088 Acked Read 10 Done Done Sent false
27677 8089 Acked Flush 0 Done Done Sent false
27678 8090 Acked Read 2 Done Done Sent false
27679 8091 Acked Write 8 Done Done Sent false
27680 8092 Acked Write 3 Done Done Sent false
27681 8093 Acked Flush 0 Done Done Sent false
27682 8094 Acked Read 10 Done Done Sent false
27683 8095 Acked Write 1 Done Done Sent false
27684 8096 Acked Read 7 Done Done Sent false
27685 8097 Acked Read 3 Done Done Sent false
27686 8098 Acked Flush 0 Done Done Sent false
27687 8099 Acked Write 10 Done Done Sent false
27688 8100 Acked Read 8 Done Done Sent false
27689 8101 Acked Write 5 Done Done Sent false
27690 8102 Acked Read 7 Done Done Sent false
27691 8103 Acked Read 3 Done Done Sent false
27692 8104 Acked Read 7 Done Done Sent false
27693 8105 Acked Read 6 Done Done Sent false
27694 8106 Acked Read 5 Done Done Sent false
27695 8107 Acked Write 9 Done Done Sent false
27696 8108 Acked Write 5 Done Done Sent false
27697 8109 Acked Write 1 Done Done Sent false
27698 8110 Acked Read 7 Done Done Sent false
27699 8111 Acked Read 2 Done Done Sent false
27700 8112 Acked Write 2 Done Done Sent false
27701 8113 Acked Write 9 Done Done Sent false
27702 8114 Acked Flush 0 Done Done Sent false
27703 8115 Acked Read 8 Done Done Sent false
27704 8116 Acked Read 1 Done Done Sent false
27705 8117 Acked Write 8 Done Done Sent false
27706 8118 Acked Read 1 Done Done Sent false
27707 8119 Acked Write 7 Done Done Sent false
27708 8120 Acked Read 5 Done Done Sent false
27709 8121 Acked Write 3 Done Done Sent false
27710 8122 Acked Write 10 Done Done Sent false
27711 8123 Acked Read 5 Done Done Sent false
27712 8124 Acked Read 9 Done Done Sent false
27713 8125 Acked Read 2 Done Done Sent false
27714 8126 Acked Write 9 Done Done Sent false
27715 8127 Acked Read 4 Done Done Sent false
27716 8128 Acked Write 3 Done Done Sent false
27717 8129 Acked Read 8 Done Done Sent false
27718 8130 Acked Write 7 Done Done Sent false
27719 8131 Acked Read 4 Done Done Sent false
27720 8132 Acked Read 4 Done Done Sent false
27721 8133 Acked Read 7 Done Done Sent false
27722 8134 Acked Write 9 Done Done Sent false
27723 8135 Acked Flush 0 Done Done Sent false
27724 8136 Acked Read 8 Done Done Sent false
27725 8137 Acked Write 10 Done Done Sent false
27726 8138 Acked Write 4 Done Done Sent false
27727 8139 Acked Write 3 Done Done Sent false
27728 8140 Acked Write 2 Done Done Sent false
27729 8141 Acked Write 8 Done Done Sent false
27730 8142 Acked Write 3 Done Done Sent false
27731 8143 Acked Write 7 Done Done Sent false
27732 8144 Acked Write 4 Done Done Sent false
27733 8145 Acked Write 6 Done Done Sent false
27734 8146 Acked Flush 0 Done Done Sent false
27735 8147 Acked Write 1 Done Done Sent false
27736 8148 Acked Read 9 Done Done Sent false
27737 8149 Acked Read 10 Done Done Sent false
27738 8150 Acked Read 8 Done Done Sent false
27739 8151 Acked Flush 0 Done Done Sent false
27740 8152 Acked Write 1 Done Done Sent false
27741 8153 Acked Read 9 Done Done Sent false
27742 8154 Acked Write 1 Done Done Sent false
27743 8155 Acked Write 8 Done Done Sent false
27744 8156 Acked Read 10 Done Done Sent false
27745 8157 Acked Read 3 Done Done Sent false
27746 8158 Acked Read 10 Done Done Sent false
27747 8159 Acked Write 3 Done Done Sent false
27748 8160 Acked Read 10 Done Done Sent false
27749 8161 Acked Write 8 Done Done Sent false
27750 8162 Acked Read 5 Done Done Sent false
27751 8163 Acked Read 5 Done Done Sent false
27752 8164 Acked Write 4 Done Done Sent false
27753 8165 Acked Read 2 Done Done Sent false
27754 8166 Acked Read 10 Done Done Sent false
27755 8167 Acked Write 5 Done Done Sent false
27756 8168 Acked Read 10 Done Done Sent false
27757 8169 Acked Read 3 Done Done Sent false
27758 8170 Acked Write 2 Done Done Sent false
27759 8171 Acked Read 1 Done Done Sent false
27760 8172 Acked Write 9 Done Done Sent false
27761 8173 Acked Write 3 Done Done Sent false
27762 8174 Acked Read 5 Done Done Sent false
27763 8175 Acked Read 9 Done Done Sent false
27764 8176 Acked Flush 0 Done Done Sent false
27765 8177 Acked Flush 0 Done Done Sent false
27766 8178 Acked Read 7 Done Done Sent false
27767 8179 Acked Write 4 Done Done Sent false
27768 8180 Acked Write 5 Done Done Sent false
27769 8181 Acked Write 5 Done Done Sent false
27770 8182 Acked Read 6 Done Done Sent false
27771 8183 Acked Write 5 Done Done Sent false
27772 8184 Acked Write 4 Done Done Sent false
27773 8185 Acked Read 2 Done Done Sent false
27774 8186 Acked Write 8 Done Done Sent false
27775 8187 Acked Read 2 Done Done Sent false
27776 8188 Acked Write 9 Done Done Sent false
27777 8189 Acked Write 6 Done Done Sent false
27778 8190 Acked Read 4 Done Done Sent false
27779 8191 Acked Write 10 Done Done Sent false
27780 8192 Acked Read 7 Done Done Sent false
27781 8193 Acked Flush 0 Done Done Sent false
27782 8194 Acked Write 5 Done Done Sent false
27783 8195 Acked Write 10 Done Done Sent false
27784 8196 Acked Write 9 Done Done Sent false
27785 8197 Acked Write 10 Done Done Sent false
27786 8198 Acked Write 7 Done Done Sent false
27787 8199 Acked Write 4 Done Done Sent false
27788 8200 Acked Write 1 Done Done Sent false
27789 8201 Acked Write 6 Done Done Sent false
27790 8202 Acked Read 1 Done Done Sent false
27791 8203 Acked Read 2 Done Done Sent false
27792 8204 Acked Read 8 Done Done Sent false
27793 8205 Acked Write 4 Done Done Sent false
27794 8206 Acked Write 5 Done Done Sent false
27795 8207 Acked Read 6 Done Done Sent false
27796 8208 Acked Write 10 Done Done Sent false
27797 8209 Acked Read 5 Done Done Sent false
27798 8210 Acked Write 4 Done Done Sent false
27799 8211 Acked Read 1 Done Done Sent false
27800 8212 Acked Read 9 Done Done Sent false
27801 8213 Acked Read 9 Done Done Sent false
27802 8214 Acked Write 1 Done Done Sent false
27803 8215 Acked Read 7 Done Done Sent false
27804 8216 Acked Read 3 Done Done Sent false
27805 8217 Acked Read 8 Done Done Sent false
27806 8218 Acked Read 6 Done Done Sent false
27807 8219 Acked Read 1 Done Done Sent false
27808 8220 Acked Read 6 Done Done Sent false
27809 8221 Acked Read 3 Done Done Sent false
27810 8222 Acked Read 7 Done Done Sent false
27811 8223 Acked Read 9 Done Done Sent false
27812 8224 Acked Write 5 Done Done Sent false
27813 8225 Acked Write 6 Done Done Sent false
27814 8226 Acked Read 8 Done Done Sent false
27815 8227 Acked Read 8 Done Done Sent false
27816 8228 Acked Read 4 Done Done Sent false
27817 8229 Acked Read 6 Done Done Sent false
27818 8230 Acked Flush 0 Done Done Sent false
27819 8231 Acked Read 2 Done Done Sent false
27820 8232 Acked Read 1 Done Done Sent false
27821 8233 Acked Read 1 Done Done Sent false
27822 8234 Acked Read 9 Done Done Sent false
27823 8235 Acked Read 4 Done Done Sent false
27824 8236 Acked Write 4 Done Done Sent false
27825 8237 Acked Flush 0 Done Done Sent false
27826 8238 Acked Write 6 Done Done Sent false
27827 8239 Acked Write 2 Done Done Sent false
27828 8240 Acked Read 7 Done Done Sent false
27829 8241 Acked Read 2 Done Done Sent false
27830 8242 Acked Write 5 Done Done Sent false
27831 STATES DS:0 DS:1 DS:2 TOTAL
27832 Sent 0 0 301 301
27833 Done 301 301 0 602
27834 Skipped 0 0 0 0
27835 Error 0 0 0 0
27836 Last Flush: Some(JobId(8237)) Some(JobId(8237)) Some(JobId(7941))
27837 Downstairs last five completed: 7941 7940 7939 7938 7937
27838 Upstairs last five completed: 8242 8241 8240 8239 8238
27839 CLIENT: Up:0 ds:301 act:2
278402025-03-07T18:18:35.650ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
278412025-03-07T18:18:36.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
278422025-03-07T18:18:37.556ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
278432025-03-07T18:18:38.556ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
27844 ----------------------------------------------------------------
27845 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27846 States: Active Active Offline
27847 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27848 7942 Acked Write 8 Done Done Sent false
27849 7943 Acked Flush 0 Done Done Sent false
27850 7944 Acked Write 4 Done Done Sent false
27851 7945 Acked Write 8 Done Done Sent false
27852 7946 Acked Read 4 Done Done Sent false
27853 7947 Acked Read 1 Done Done Sent false
27854 7948 Acked Write 2 Done Done Sent false
27855 7949 Acked Read 5 Done Done Sent false
27856 7950 Acked Write 10 Done Done Sent false
27857 7951 Acked Read 8 Done Done Sent false
27858 7952 Acked Read 5 Done Done Sent false
27859 7953 Acked Write 5 Done Done Sent false
27860 7954 Acked Write 7 Done Done Sent false
27861 7955 Acked Read 5 Done Done Sent false
27862 7956 Acked Read 3 Done Done Sent false
27863 7957 Acked Read 5 Done Done Sent false
27864 7958 Acked Read 8 Done Done Sent false
27865 7959 Acked Read 10 Done Done Sent false
27866 7960 Acked Read 7 Done Done Sent false
27867 7961 Acked Write 10 Done Done Sent false
27868 7962 Acked Write 8 Done Done Sent false
27869 7963 Acked Read 4 Done Done Sent false
27870 7964 Acked Read 7 Done Done Sent false
27871 7965 Acked Read 6 Done Done Sent false
27872 7966 Acked Read 10 Done Done Sent false
27873 7967 Acked Read 2 Done Done Sent false
27874 7968 Acked Flush 0 Done Done Sent false
27875 7969 Acked Read 1 Done Done Sent false
27876 7970 Acked Write 1 Done Done Sent false
27877 7971 Acked Write 1 Done Done Sent false
27878 7972 Acked Read 1 Done Done Sent false
27879 7973 Acked Write 4 Done Done Sent false
27880 7974 Acked Flush 0 Done Done Sent false
27881 7975 Acked Write 5 Done Done Sent false
27882 7976 Acked Write 4 Done Done Sent false
27883 7977 Acked Read 1 Done Done Sent false
27884 7978 Acked Write 5 Done Done Sent false
27885 7979 Acked Read 8 Done Done Sent false
27886 7980 Acked Write 7 Done Done Sent false
27887 7981 Acked Read 2 Done Done Sent false
27888 7982 Acked Write 6 Done Done Sent false
27889 7983 Acked Flush 0 Done Done Sent false
27890 7984 Acked Read 6 Done Done Sent false
27891 7985 Acked Write 3 Done Done Sent false
27892 7986 Acked Read 8 Done Done Sent false
27893 7987 Acked Write 5 Done Done Sent false
27894 7988 Acked Read 5 Done Done Sent false
27895 7989 Acked Write 1 Done Done Sent false
27896 7990 Acked Flush 0 Done Done Sent false
27897 7991 Acked Write 1 Done Done Sent false
27898 7992 Acked Read 8 Done Done Sent false
27899 7993 Acked Write 10 Done Done Sent false
27900 7994 Acked Write 5 Done Done Sent false
27901 7995 Acked Write 7 Done Done Sent false
27902 7996 Acked Flush 0 Done Done Sent false
27903 7997 Acked Write 2 Done Done Sent false
27904 7998 Acked Read 7 Done Done Sent false
27905 7999 Acked Write 8 Done Done Sent false
27906 8000 Acked Write 1 Done Done Sent false
27907 8001 Acked Read 10 Done Done Sent false
27908 8002 Acked Write 10 Done Done Sent false
27909 8003 Acked Write 1 Done Done Sent false
27910 8004 Acked Read 4 Done Done Sent false
27911 8005 Acked Read 8 Done Done Sent false
27912 8006 Acked Write 8 Done Done Sent false
27913 8007 Acked Read 6 Done Done Sent false
27914 8008 Acked Read 5 Done Done Sent false
27915 8009 Acked Read 6 Done Done Sent false
27916 8010 Acked Write 7 Done Done Sent false
27917 8011 Acked Read 4 Done Done Sent false
27918 8012 Acked Write 9 Done Done Sent false
27919 8013 Acked Write 7 Done Done Sent false
27920 8014 Acked Read 2 Done Done Sent false
27921 8015 Acked Read 6 Done Done Sent false
27922 8016 Acked Read 6 Done Done Sent false
27923 8017 Acked Write 7 Done Done Sent false
27924 8018 Acked Read 8 Done Done Sent false
27925 8019 Acked Write 6 Done Done Sent false
27926 8020 Acked Flush 0 Done Done Sent false
27927 8021 Acked Write 4 Done Done Sent false
27928 8022 Acked Read 5 Done Done Sent false
27929 8023 Acked Read 10 Done Done Sent false
27930 8024 Acked Flush 0 Done Done Sent false
27931 8025 Acked Write 4 Done Done Sent false
27932 8026 Acked Read 1 Done Done Sent false
27933 8027 Acked Read 10 Done Done Sent false
27934 8028 Acked Read 7 Done Done Sent false
27935 8029 Acked Write 3 Done Done Sent false
27936 8030 Acked Read 7 Done Done Sent false
27937 8031 Acked Read 2 Done Done Sent false
27938 8032 Acked Write 7 Done Done Sent false
27939 8033 Acked Read 1 Done Done Sent false
27940 8034 Acked Read 10 Done Done Sent false
27941 8035 Acked Flush 0 Done Done Sent false
27942 8036 Acked Read 7 Done Done Sent false
27943 8037 Acked Read 7 Done Done Sent false
27944 8038 Acked Read 4 Done Done Sent false
27945 8039 Acked Write 7 Done Done Sent false
27946 8040 Acked Read 4 Done Done Sent false
27947 8041 Acked Write 5 Done Done Sent false
27948 8042 Acked Read 8 Done Done Sent false
27949 8043 Acked Write 2 Done Done Sent false
27950 8044 Acked Write 10 Done Done Sent false
27951 8045 Acked Write 4 Done Done Sent false
27952 8046 Acked Read 8 Done Done Sent false
27953 8047 Acked Read 5 Done Done Sent false
27954 8048 Acked Read 3 Done Done Sent false
27955 8049 Acked Read 4 Done Done Sent false
27956 8050 Acked Flush 0 Done Done Sent false
27957 8051 Acked Write 2 Done Done Sent false
27958 8052 Acked Read 5 Done Done Sent false
27959 8053 Acked Read 1 Done Done Sent false
27960 8054 Acked Write 3 Done Done Sent false
27961 8055 Acked Flush 0 Done Done Sent false
27962 8056 Acked Read 8 Done Done Sent false
27963 8057 Acked Read 8 Done Done Sent false
27964 8058 Acked Read 7 Done Done Sent false
27965 8059 Acked Write 8 Done Done Sent false
27966 8060 Acked Write 7 Done Done Sent false
27967 8061 Acked Flush 0 Done Done Sent false
27968 8062 Acked Flush 0 Done Done Sent false
27969 8063 Acked Read 9 Done Done Sent false
27970 8064 Acked Read 6 Done Done Sent false
27971 8065 Acked Write 8 Done Done Sent false
27972 8066 Acked Flush 0 Done Done Sent false
27973 8067 Acked Read 2 Done Done Sent false
27974 8068 Acked Write 2 Done Done Sent false
27975 8069 Acked Write 7 Done Done Sent false
27976 8070 Acked Read 6 Done Done Sent false
27977 8071 Acked Flush 0 Done Done Sent false
27978 8072 Acked Read 8 Done Done Sent false
27979 8073 Acked Write 2 Done Done Sent false
27980 8074 Acked Write 2 Done Done Sent false
27981 8075 Acked Write 10 Done Done Sent false
27982 8076 Acked Write 6 Done Done Sent false
27983 8077 Acked Read 2 Done Done Sent false
27984 8078 Acked Write 4 Done Done Sent false
27985 8079 Acked Write 9 Done Done Sent false
27986 8080 Acked Read 7 Done Done Sent false
27987 8081 Acked Read 8 Done Done Sent false
27988 8082 Acked Read 1 Done Done Sent false
27989 8083 Acked Read 2 Done Done Sent false
27990 8084 Acked Read 8 Done Done Sent false
27991 8085 Acked Read 10 Done Done Sent false
27992 8086 Acked Read 2 Done Done Sent false
27993 8087 Acked Write 6 Done Done Sent false
27994 8088 Acked Read 10 Done Done Sent false
27995 8089 Acked Flush 0 Done Done Sent false
27996 8090 Acked Read 2 Done Done Sent false
27997 8091 Acked Write 8 Done Done Sent false
27998 8092 Acked Write 3 Done Done Sent false
27999 8093 Acked Flush 0 Done Done Sent false
28000 8094 Acked Read 10 Done Done Sent false
28001 8095 Acked Write 1 Done Done Sent false
28002 8096 Acked Read 7 Done Done Sent false
28003 8097 Acked Read 3 Done Done Sent false
28004 8098 Acked Flush 0 Done Done Sent false
28005 8099 Acked Write 10 Done Done Sent false
28006 8100 Acked Read 8 Done Done Sent false
28007 8101 Acked Write 5 Done Done Sent false
28008 8102 Acked Read 7 Done Done Sent false
28009 8103 Acked Read 3 Done Done Sent false
28010 8104 Acked Read 7 Done Done Sent false
28011 8105 Acked Read 6 Done Done Sent false
28012 8106 Acked Read 5 Done Done Sent false
28013 8107 Acked Write 9 Done Done Sent false
28014 8108 Acked Write 5 Done Done Sent false
28015 8109 Acked Write 1 Done Done Sent false
28016 8110 Acked Read 7 Done Done Sent false
28017 8111 Acked Read 2 Done Done Sent false
28018 8112 Acked Write 2 Done Done Sent false
28019 8113 Acked Write 9 Done Done Sent false
28020 8114 Acked Flush 0 Done Done Sent false
28021 8115 Acked Read 8 Done Done Sent false
28022 8116 Acked Read 1 Done Done Sent false
28023 8117 Acked Write 8 Done Done Sent false
28024 8118 Acked Read 1 Done Done Sent false
28025 8119 Acked Write 7 Done Done Sent false
28026 8120 Acked Read 5 Done Done Sent false
28027 8121 Acked Write 3 Done Done Sent false
28028 8122 Acked Write 10 Done Done Sent false
28029 8123 Acked Read 5 Done Done Sent false
28030 8124 Acked Read 9 Done Done Sent false
28031 8125 Acked Read 2 Done Done Sent false
28032 8126 Acked Write 9 Done Done Sent false
28033 8127 Acked Read 4 Done Done Sent false
28034 8128 Acked Write 3 Done Done Sent false
28035 8129 Acked Read 8 Done Done Sent false
28036 8130 Acked Write 7 Done Done Sent false
28037 8131 Acked Read 4 Done Done Sent false
28038 8132 Acked Read 4 Done Done Sent false
28039 8133 Acked Read 7 Done Done Sent false
28040 8134 Acked Write 9 Done Done Sent false
28041 8135 Acked Flush 0 Done Done Sent false
28042 8136 Acked Read 8 Done Done Sent false
28043 8137 Acked Write 10 Done Done Sent false
28044 8138 Acked Write 4 Done Done Sent false
28045 8139 Acked Write 3 Done Done Sent false
28046 8140 Acked Write 2 Done Done Sent false
28047 8141 Acked Write 8 Done Done Sent false
28048 8142 Acked Write 3 Done Done Sent false
28049 8143 Acked Write 7 Done Done Sent false
28050 8144 Acked Write 4 Done Done Sent false
28051 8145 Acked Write 6 Done Done Sent false
28052 8146 Acked Flush 0 Done Done Sent false
28053 8147 Acked Write 1 Done Done Sent false
28054 8148 Acked Read 9 Done Done Sent false
28055 8149 Acked Read 10 Done Done Sent false
28056 8150 Acked Read 8 Done Done Sent false
28057 8151 Acked Flush 0 Done Done Sent false
28058 8152 Acked Write 1 Done Done Sent false
28059 8153 Acked Read 9 Done Done Sent false
28060 8154 Acked Write 1 Done Done Sent false
28061 8155 Acked Write 8 Done Done Sent false
28062 8156 Acked Read 10 Done Done Sent false
28063 8157 Acked Read 3 Done Done Sent false
28064 8158 Acked Read 10 Done Done Sent false
28065 8159 Acked Write 3 Done Done Sent false
28066 8160 Acked Read 10 Done Done Sent false
28067 8161 Acked Write 8 Done Done Sent false
28068 8162 Acked Read 5 Done Done Sent false
28069 8163 Acked Read 5 Done Done Sent false
28070 8164 Acked Write 4 Done Done Sent false
28071 8165 Acked Read 2 Done Done Sent false
28072 8166 Acked Read 10 Done Done Sent false
28073 8167 Acked Write 5 Done Done Sent false
28074 8168 Acked Read 10 Done Done Sent false
28075 8169 Acked Read 3 Done Done Sent false
28076 8170 Acked Write 2 Done Done Sent false
28077 8171 Acked Read 1 Done Done Sent false
28078 8172 Acked Write 9 Done Done Sent false
28079 8173 Acked Write 3 Done Done Sent false
28080 8174 Acked Read 5 Done Done Sent false
28081 8175 Acked Read 9 Done Done Sent false
28082 8176 Acked Flush 0 Done Done Sent false
28083 8177 Acked Flush 0 Done Done Sent false
28084 8178 Acked Read 7 Done Done Sent false
28085 8179 Acked Write 4 Done Done Sent false
28086 8180 Acked Write 5 Done Done Sent false
28087 8181 Acked Write 5 Done Done Sent false
28088 8182 Acked Read 6 Done Done Sent false
28089 8183 Acked Write 5 Done Done Sent false
28090 8184 Acked Write 4 Done Done Sent false
28091 8185 Acked Read 2 Done Done Sent false
28092 8186 Acked Write 8 Done Done Sent false
28093 8187 Acked Read 2 Done Done Sent false
28094 8188 Acked Write 9 Done Done Sent false
28095 8189 Acked Write 6 Done Done Sent false
28096 8190 Acked Read 4 Done Done Sent false
28097 8191 Acked Write 10 Done Done Sent false
28098 8192 Acked Read 7 Done Done Sent false
28099 8193 Acked Flush 0 Done Done Sent false
28100 8194 Acked Write 5 Done Done Sent false
28101 8195 Acked Write 10 Done Done Sent false
28102 8196 Acked Write 9 Done Done Sent false
28103 8197 Acked Write 10 Done Done Sent false
28104 8198 Acked Write 7 Done Done Sent false
28105 8199 Acked Write 4 Done Done Sent false
28106 8200 Acked Write 1 Done Done Sent false
28107 8201 Acked Write 6 Done Done Sent false
28108 8202 Acked Read 1 Done Done Sent false
28109 8203 Acked Read 2 Done Done Sent false
28110 8204 Acked Read 8 Done Done Sent false
28111 8205 Acked Write 4 Done Done Sent false
28112 8206 Acked Write 5 Done Done Sent false
28113 8207 Acked Read 6 Done Done Sent false
28114 8208 Acked Write 10 Done Done Sent false
28115 8209 Acked Read 5 Done Done Sent false
28116 8210 Acked Write 4 Done Done Sent false
28117 8211 Acked Read 1 Done Done Sent false
28118 8212 Acked Read 9 Done Done Sent false
28119 8213 Acked Read 9 Done Done Sent false
28120 8214 Acked Write 1 Done Done Sent false
28121 8215 Acked Read 7 Done Done Sent false
28122 8216 Acked Read 3 Done Done Sent false
28123 8217 Acked Read 8 Done Done Sent false
28124 8218 Acked Read 6 Done Done Sent false
28125 8219 Acked Read 1 Done Done Sent false
28126 8220 Acked Read 6 Done Done Sent false
28127 8221 Acked Read 3 Done Done Sent false
28128 8222 Acked Read 7 Done Done Sent false
28129 8223 Acked Read 9 Done Done Sent false
28130 8224 Acked Write 5 Done Done Sent false
28131 8225 Acked Write 6 Done Done Sent false
28132 8226 Acked Read 8 Done Done Sent false
28133 8227 Acked Read 8 Done Done Sent false
28134 8228 Acked Read 4 Done Done Sent false
28135 8229 Acked Read 6 Done Done Sent false
28136 8230 Acked Flush 0 Done Done Sent false
28137 8231 Acked Read 2 Done Done Sent false
28138 8232 Acked Read 1 Done Done Sent false
28139 8233 Acked Read 1 Done Done Sent false
28140 8234 Acked Read 9 Done Done Sent false
28141 8235 Acked Read 4 Done Done Sent false
28142 8236 Acked Write 4 Done Done Sent false
28143 8237 Acked Flush 0 Done Done Sent false
28144 8238 Acked Write 6 Done Done Sent false
28145 8239 Acked Write 2 Done Done Sent false
28146 8240 Acked Read 7 Done Done Sent false
28147 8241 Acked Read 2 Done Done Sent false
28148 8242 Acked Write 5 Done Done Sent false
28149 STATES DS:0 DS:1 DS:2 TOTAL
28150 Sent 0 0 301 301
28151 Done 301 301 0 602
28152 Skipped 0 0 0 0
28153 Error 0 0 0 0
28154 Last Flush: Some(JobId(8237)) Some(JobId(8237)) Some(JobId(7941))
28155 Downstairs last five completed: 7941 7940 7939 7938 7937
28156 Upstairs last five completed: 8242 8241 8240 8239 8238
28157 CLIENT: Up:0 ds:301 act:2
281582025-03-07T18:18:39.558ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281592025-03-07T18:18:40.651ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281602025-03-07T18:18:41.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281612025-03-07T18:18:42.388ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281622025-03-07T18:18:42.388ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36532) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281632025-03-07T18:18:42.389ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281642025-03-07T18:18:42.390ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281652025-03-07T18:18:42.390ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281662025-03-07T18:18:42.390ZINFOcrucible: send last flush ID to this DS: Some(JobId(7941)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281672025-03-07T18:18:42.390ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281682025-03-07T18:18:42.390ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7941)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
281692025-03-07T18:18:42.390ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7941)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
28170 ----------------------------------------------------------------
28171 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
28172 States: Active Active Active
28173 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28174 8238 Acked Write 6 Done Done Done true
28175 8239 Acked Write 2 Done Done Done true
28176 8240 Acked Read 7 Done Done Done true
28177 8241 Acked Read 2 Done Done Done true
28178 8242 Acked Write 5 Done Done Done true
28179 STATES DS:0 DS:1 DS:2 TOTAL
28180 Sent 0 0 0 0
28181 Done 5 5 5 15
28182 Skipped 0 0 0 0
28183 Error 0 0 0 0
28184 Last Flush: Some(JobId(8237)) Some(JobId(8237)) Some(JobId(8237))
28185 Downstairs last five completed: 8237 8236 8235 8234 8233
28186 Upstairs last five completed: 8242 8241 8240 8239 8238
28187 CLIENT: Up:0 ds:5 act:3
28188 ----------------------------------------------------------------
28189 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28190 Downstairs last five completed: 8243 8242 8241 8240 8239
28191 Upstairs last five completed: 8243 8242 8241 8240 8239
28192 CLIENT: Up:0 ds:0 act:3
28193 Replay: All jobs finished
28194 Main task has work to do, go find it
28195 got dsc Stop(1)
28196 stop 1
28197 Mar 07 18:18:46.648 INFO request completed[8820] Got stop action so:false kr:false
28198 , latency_us: 314, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 71cdab64-a9a9-4e84-84cd-92f227762d13, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
28199 Mar 07 18:18:46.648 INFO request completed, latency_us: 146, response_code: 200, uri: /state/cid/1, method: GET, req_id: c33df265-5353-435c-8d89-ece7f58854ed, remote_addr: 127.0.0.1{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:18:46.648867476Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
28200 :64322, local_addr: 127.0.0.1{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:18:46.648931038Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
28201 :9998
282022025-03-07T18:18:46.648ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
282032025-03-07T18:18:46.649ZINFOcrucible: client task is exiting = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
28204 [8820] Exited with: ExitStatus(unix_wait_status(9))
28205 [8820][1] reports Exit
28206 Mar 07 18:18:50.651 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/1, method: GET, req_id: 60cc28ab-ef04-4c49-954a-97d7086f2332, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
28207 001/300 Read block 871 len 36864
28208 002/300 Write block 2855 len 32768 data: 3 2 2 3 4 4 5 4
28209 003/300 Flush
28210 004/300 Read block 1809 len 4096
28211 005/300 Read block 2356 len 4096
28212 006/300 Read block 2393 len 40960
28213 007/300 Write block 1101 len 24576 data: 4 4 6 6 6 6
28214 008/300 Read block 1745 len 40960
28215 009/300 Read block 1552 len 28672
28216 010/300 Read block 1383 len 4096
28217 011/300 Read block 2548 len 40960
28218 012/300 Read block 2387 len 24576
28219 013/300 Write block 746 len 8192 data: 10 10
28220 014/300 Flush
28221 015/300 Write block 3520 len 20480 data: 3 4 4 5 5
28222 016/300 Read block 4647 len 28672
28223 017/300 Read block 932 len 16384
28224 018/300 Read block 4559 len 24576
28225 019/300 Read block 1235 len 16384
28226 020/300 Write block 1389 len 12288 data: 4 5 5
28227 021/300 Read block 3774 len 32768
28228 022/300 Read block 2788 len 20480
28229 023/300 Read block 1369 len 4096
28230 024/300 Read block 1430 len 12288
28231 025/300 Write block 2328 len 20480 data: 4 5 5 6 6
28232 026/300 Read block 1794 len 40960
28233 027/300 Write block 1604 len 8192 data: 6 5
28234 028/300 Write block 1930 len 20480 data: 7 6 6 2 3
28235 029/300 Flush
28236 030/300 Write block 3054 len 36864 data: 4 5 3 5 5 5 5 6 6
28237 031/300 Write block 69 len 40960 data: 6 7 7 8 8 8 8 7 5 4
28238 032/300 Read block 2863 len 4096
28239 033/300 Write block 81 len 16384 data: 4 4 5 5
28240 034/300 Write block 3809 len 36864 data: 5 5 6 9 7 6 5 5 6
28241 035/300 Write block 4101 len 28672 data: 3 3 4 4 3 5 5
28242 036/300 Read block 4629 len 36864
28243 037/300 Read block 1268 len 16384
28244 038/300 Read block 4573 len 16384
28245 039/300 Read block 3333 len 4096
28246 040/300 Write block 3442 len 40960 data: 7 6 7 8 5 6 7 8 8 8
28247 041/300 Flush
28248 042/300 Write block 970 len 12288 data: 4 5 5
28249 043/300 Write block 4393 len 40960 data: 5 4 4 5 5 3 3 3 3 4
28250 044/300 Flush
28251 045/300 Write block 1629 len 4096 data: 8
28252 046/300 Read block 1246 len 4096
28253 047/300 Read block 1264 len 12288
28254 048/300 Write block 408 len 20480 data: 4 4 5 6 5
28255 049/300 Read block 4362 len 12288
28256 050/300 Write block 3091 len 16384 data: 4 4 3 5
28257 051/300 Read block 958 len 4096
28258 052/300 Read block 1235 len 20480
28259 053/300 Read block 1566 len 28672
28260 054/300 Read block 883 len 4096
28261 055/300 Read block 1397 len 32768
28262 056/300 Write block 2344 len 24576 data: 4 4 4 4 3 3
28263 057/300 Write block 4071 len 4096 data: 6
28264 058/300 Read block 678 len 8192
28265 059/300 Read block 521 len 20480
28266 060/300 Read block 603 len 40960
28267 061/300 Read block 4600 len 20480
28268 062/300 Write block 4327 len 20480 data: 5 7 7 9 10
28269 063/300 Write block 4265 len 20480 data: 4 2 2 2 2
28270 064/300 Read block 3161 len 36864
28271 065/300 Write block 1043 len 32768 data: 3 4 4 5 5 5 5 5
28272 066/300 Write block 1860 len 20480 data: 4 4 3 3 2
28273 067/300 Write block 2549 len 32768 data: 10 9 7 8 9 6 8 7
28274 068/300 Read block 3103 len 8192
28275 069/300 Read block 4577 len 28672
28276 070/300 Flush
28277 071/300 Write block 1142 len 36864 data: 8 8 6 6 6 6 4 3 4
28278 072/300 Read block 1877 len 28672
28279 073/300 Write block 4558 len 16384 data: 4 4 3 5
28280 074/300 Write block 839 len 4096 data: 4
28281 075/300 Read block 4743 len 32768
28282 076/300 Write block 1919 len 16384 data: 9 9 10 12
28283 077/300 Write block 1239 len 28672 data: 2 3 3 4 3 5 5
28284 078/300 Flush
28285 079/300 Read block 2927 len 40960
28286 080/300 Write block 3330 len 36864 data: 5 4 5 5 5 5 6 6 6
28287 081/300 Write block 383 len 16384 data: 6 5 4 5
28288 082/300 Write block 2427 len 28672 data: 4 5 5 5 5 6 5
28289 083/300 Write block 2039 len 8192 data: 4 4
28290 084/300 Read block 3700 len 20480
28291 085/300 Read block 599 len 40960
28292 086/300 Read block 4214 len 20480
28293 087/300 Flush
28294 088/300 Flush
28295 089/300 Read block 1732 len 32768
28296 090/300 Read block 2793 len 8192
28297 091/300 Read block 465 len 24576
28298 092/300 Write block 4175 len 32768 data: 7 7 7 6 5 6 5 4
28299 093/300 Read block 1539 len 12288
28300 094/300 Read block 1586 len 28672
28301 095/300 Write block 2841 len 8192 data: 7 5
28302 096/300 Write block 4726 len 32768 data: 5 4 3 2 2 2 4 4
28303 097/300 Read block 3104 len 40960
28304 098/300 Read block 4986 len 24576
28305 099/300 Read block 2252 len 8192
28306 100/300 Write block 475 len 12288 data: 6 5 5
28307 101/300 Read block 2187 len 16384
28308 102/300 Read block 552 len 28672
28309 103/300 Write block 3568 len 28672 data: 6 7 4 2 2 3 3
28310 104/300 Flush
28311 105/300 Flush
28312 106/300 Read block 2217 len 4096
28313 107/300 Write block 3782 len 24576 data: 10 8 8 6 5 4
28314 108/300 Read block 4909 len 8192
28315 109/300 Write block 4513 len 20480 data: 4 4 5 5 4
28316 110/300 Write block 1229 len 28672 data: 3 3 3 3 3 5 6
28317 111/300 Write block 1849 len 20480 data: 6 6 6 6 4
28318 112/300 Flush
28319 113/300 Write block 195 len 36864 data: 7 8 8 7 6 5 7 7 7
28320 114/300 Read block 4111 len 4096
28321 115/300 Write block 4387 len 16384 data: 5 5 7 7
28322 116/300 Read block 1327 len 12288
28323 117/300 Read block 2696 len 4096
28324 118/300 Read block 3427 len 8192
28325 119/300 Read block 2330 len 20480
28326 120/300 Write block 2717 len 40960 data: 5 6 6 8 9 9 8 8 9 9
28327 121/300 Read block 1612 len 20480
28328 122/300 Read block 352 len 32768
28329 123/300 Write block 3792 len 24576 data: 3 7 6 6 7 9
28330 124/300 Write block 399 len 12288 data: 6 7 6
28331 125/300 Write block 3693 len 4096 data: 5
28332 126/300 Write block 2596 len 20480 data: 3 3 5 5 4
28333 127/300 Read block 4352 len 20480
28334 128/300 Write block 92 len 24576 data: 6 7 7 6 5 4
28335 129/300 Read block 1867 len 28672
28336 130/300 Write block 2863 len 32768 data: 5 6 4 5 7 6 5 5
28337 131/300 Write block 1847 len 40960 data: 6 6 7 7 7 7 5 5 4 4
28338 132/300 Read block 4918 len 12288
28339 133/300 Write block 678 len 8192 data: 9 8
28340 134/300 Read block 271 len 8192
28341 135/300 Flush
28342 136/300 Read block 4555 len 16384
28343 137/300 Write block 3828 len 12288 data: 8 10 9
28344 138/300 Read block 2723 len 20480
28345 139/300 Flush
28346 140/300 Flush
28347 141/300 Read block 878 len 20480
28348 142/300 Read block 285 len 24576
28349 143/300 Write block 1644 len 8192 data: 6 5
28350 144/300 Read block 762 len 12288
28351 145/300 Write block 3166 len 32768 data: 5 6 4 6 6 6 5 4
28352 146/300 Read block 1340 len 16384
28353 147/300 Read block 1449 len 8192
28354 148/300 Write block 3258 len 16384 data: 5 6 5 6
28355 149/300 Write block 4454 len 28672 data: 6 6 7 7 5 4 5
28356 150/300 Flush
28357 151/300 Write block 160 len 32768 data: 6 6 6 6 8 9 11 10
28358 152/300 Read block 2219 len 8192
28359 153/300 Read block 918 len 32768
28360 154/300 Flush
28361 155/300 Write block 630 len 4096 data: 6
28362 156/300 Flush
28363 157/300 Flush
28364 158/300 Read block 4452 len 24576
28365 159/300 Read block 4669 len 28672
28366 160/300 Read block 4162 len 16384
28367 161/300 Read block 2118 len 36864
28368 162/300 Write block 4768 len 20480 data: 7 6 7 7 7
28369 163/300 Write block 285 len 16384 data: 5 6 7 8
28370 164/300 Read block 4192 len 4096
28371 165/300 Write block 3908 len 40960 data: 6 6 3 3 3 3 4 4 5 4
28372 166/300 Write block 3434 len 24576 data: 6 6 5 6 5 5
28373 167/300 Read block 765 len 4096
28374 168/300 Read block 281 len 40960
28375 169/300 Read block 3064 len 28672
28376 170/300 Flush
28377 171/300 Read block 4180 len 24576
28378 172/300 Read block 4658 len 16384
28379 173/300 Read block 4307 len 20480
28380 174/300 Read block 3157 len 16384
28381 175/300 Read block 3863 len 28672
28382 176/300 Write block 4224 len 32768 data: 4 5 4 5 5 5 4 3
28383 177/300 Read block 1777 len 24576
28384 178/300 Read block 301 len 4096
28385 179/300 Read block 4961 len 4096
28386 180/300 Flush
28387 181/300 Write block 1970 len 40960 data: 5 5 4 5 4 4 3 3 2 3
28388 182/300 Read block 2491 len 28672
28389 183/300 Read block 3906 len 20480
28390 184/300 Read block 35 len 8192
28391 185/300 Write block 3390 len 28672 data: 4 3 5 4 4 5 4
28392 186/300 Write block 1029 len 8192 data: 5 4
28393 187/300 Write block 1076 len 20480 data: 5 4 5 4 5
28394 188/300 Read block 1605 len 4096
28395 189/300 Write block 3157 len 40960 data: 3 4 5 4 4 4 4 4 4 6
28396 190/300 Read block 51 len 20480
28397 191/300 Read block 3791 len 4096
28398 192/300 Write block 4033 len 40960 data: 4 5 6 8 8 9 7 4 3 7
28399 193/300 Write block 3429 len 12288 data: 3 3 3
28400 194/300 Write block 2606 len 4096 data: 4
28401 195/300 Write block 3699 len 4096 data: 5
28402 196/300 Write block 4237 len 40960 data: 5 5 5 5 5 5 5 6 6 3
28403 197/300 Read block 4011 len 24576
28404 198/300 Read block 2315 len 16384
28405 199/300 Read block 4631 len 4096
28406 200/300 Write block 2682 len 32768 data: 3 6 5 4 4 3 3 4
28407 201/300 Read block 994 len 4096
28408 202/300 Flush
28409 203/300 Write block 1633 len 20480 data: 2 4 6 6 6
28410 204/300 Read block 4095 len 40960
28411 205/300 Read block 3295 len 36864
28412 206/300 Write block 3804 len 8192 data: 6 4
28413 207/300 Flush
28414 208/300 Read block 2070 len 24576
28415 209/300 Write block 4233 len 12288 data: 4 4 4
28416 210/300 Write block 544 len 32768 data: 3 5 5 4 3 2 2 2
28417 211/300 Write block 4799 len 16384 data: 8 6 5 4
28418 212/300 Read block 4161 len 20480
28419 213/300 Write block 1890 len 24576 data: 9 8 8 6 5 4
28420 214/300 Write block 2569 len 36864 data: 3 3 2 2 3 4 4 5 5
28421 215/300 Write block 3969 len 24576 data: 5 6 6 6 5 4
28422 216/300 Read block 133 len 28672
28423 217/300 Read block 502 len 40960
28424 218/300 Flush
28425 219/300 Read block 1632 len 12288
28426 220/300 Write block 4863 len 16384 data: 4 4 5 6
28427 221/300 Write block 327 len 36864 data: 7 5 4 5 5 5 5 3 4
28428 222/300 Read block 483 len 20480
28429 223/300 Read block 1408 len 28672
28430 224/300 Read block 1219 len 4096
28431 225/300 Read block 1293 len 40960
28432 226/300 Write block 1749 len 4096 data: 8
28433 227/300 Write block 738 len 32768 data: 8 6 8 7 8 8 9 10
28434 228/300 Read block 2208 len 20480
28435 229/300 Write block 3000 len 20480 data: 6 5 6 7 8
28436 230/300 Read block 4091 len 24576
28437 231/300 Read block 1939 len 24576
28438 232/300 Flush
28439 233/300 Read block 502 len 24576
28440 234/300 Read block 539 len 24576
28441 235/300 Read block 3948 len 40960
28442 236/300 Read block 4326 len 4096
28443 237/300 Write block 1358 len 36864 data: 3 6 8 9 8 8 7 6 7
28444 238/300 Write block 2844 len 40960 data: 3 3 3 4 3 4 4 3 3 3
28445 239/300 Flush
28446 240/300 Read block 3086 len 28672
28447 241/300 Read block 3043 len 8192
28448 242/300 Read block 509 len 16384
28449 243/300 Write block 1777 len 32768 data: 6 5 4 4 3 3 3 3
28450 244/300 Flush
28451 245/300 Flush
28452 246/300 Read block 2636 len 36864
28453 247/300 Read block 2855 len 28672
28454 248/300 Write block 4008 len 20480 data: 4 4 4 3 3
28455 249/300 Read block 646 len 28672
28456 250/300 Read block 4007 len 28672
28457 251/300 Read block 4068 len 8192
28458 252/300 Write block 2024 len 12288 data: 6 6 7
28459 253/300 Read block 4615 len 8192
28460 254/300 Read block 1426 len 4096
28461 255/300 Read block 3951 len 28672
28462 256/300 Write block 2443 len 20480 data: 4 4 5 6 5
28463 257/300 Write block 1079 len 20480 data: 5 6 5 5 5
28464 258/300 Write block 4979 len 36864 data: 5 6 4 4 4 5 5 4 3
28465 259/300 Read block 1425 len 28672
28466 260/300 Write block 3464 len 24576 data: 3 4 5 5 6 6
28467 261/300 Read block 2408 len 16384
28468 262/300 Read block 2061 len 8192
28469 263/300 Flush
28470 264/300 Read block 3615 len 40960
28471 265/300 Write block 1274 len 32768 data: 4 3 4 4 4 4 4 3
28472 266/300 Write block 2036 len 32768 data: 4 4 4 5 5 4 5 5
28473 267/300 Read block 2727 len 16384
28474 268/300 Read block 3484 len 32768
28475 269/300 Write block 4321 len 20480 data: 4 4 3 3 4
28476 270/300 Read block 2834 len 36864
28477 271/300 Read block 4190 len 20480
28478 272/300 Write block 2652 len 16384 data: 5 6 7 9
28479 273/300 Write block 4881 len 4096 data: 4
28480 274/300 Write block 3150 len 40960 data: 5 6 6 5 5 4 3 4 5 6
28481 275/300 Read block 2812 len 36864
28482 276/300 Read block 2761 len 36864
28483 277/300 Read block 1671 len 4096
28484 278/300 Write block 3913 len 28672 data: 4 5 5 6 5 6 7
28485 279/300 Read block 3383 len 16384
28486 280/300 Read block 4741 len 36864
28487 281/300 Flush
28488 282/300 Write block 529 len 28672 data: 6 6 6 7 8 6 5
28489 283/300 Read block 4108 len 8192
28490 284/300 Read block 3107 len 16384
28491 285/300 Write block 3490 len 36864 data: 4 4 6 4 5 4 3 3 5
28492 286/300 Flush
28493 287/300 Read block 1186 len 4096
28494 288/300 Write block 4015 len 28672 data: 3 4 5 5 4 5 5
28495 289/300 Read block 3466 len 32768
28496 290/300 Read block 181 len 12288
28497 291/300 Write block 3374 len 16384 data: 4 5 6 7
28498 292/300 Write block 446 len 28672 data: 4 3 3 4 4 4 3
28499 293/300 Write block 2394 len 40960 data: 6 6 5 6 3 3 3 3 3 2
28500 294/300 Write block 284 len 4096 data: 6
28501 295/300 Write block 3489 len 36864 data: 4 5 5 7 5 6 5 4 4
28502 296/300 Read block 3531 len 20480
28503 297/300 Flush
28504 298/300 Read block 3874 len 40960
28505 299/300 Read block 3236 len 16384
28506 300/300 Write block 2665 len 36864 data: 10 9 9 9 7 6 6 8 6
28507 301/300 Read block 4249 len 32768
28508 Main task has work to do, go find it
28509 got dsc Start(1)
28510 start 1
28511 Mar 07 18:18:50.899 INFO request completed, latency_us: 167, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9ffe1747-fc59-4572-a710-b1fd7e14968c, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
28512 Replay: started 1, returned:Ok(())
28513 ----------------------------------------------------------------
28514 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28515 States: Active Offline Active
28516 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28517 8244 Acked Read 9 Done Sent Done false
28518 8245 Acked Write 8 Done Sent Done false
28519 8246 Acked Flush 0 Done Sent Done false
28520 8247 Acked Read 1 Done Sent Done false
28521 8248 Acked Read 1 Done Sent Done false
28522 8249 Acked Read 10 Done Sent Done false
28523 8250 Acked Write 6 Done Sent Done false
28524 8251 Acked Read 10 Done Sent Done false
28525 8252 Acked Read 7 Done Sent Done false
28526 8253 Acked Read 1 Done Sent Done false
28527 8254 Acked Read 10 Done Sent Done false
28528 8255 Acked Read 6 Done Sent Done false
28529 8256 Acked Write 2 Done Sent Done false
28530 8257 Acked Flush 0 Done Sent Done false
28531 8258 Acked Write 5 Done Sent Done false
28532 8259 Acked Read 7 Done Sent Done false
28533 8260 Acked Read 4 Done Sent Done false
28534 8261 Acked Read 6 Done Sent Done false
28535 8262 Acked Read 4 Done Sent Done false
28536 8263 Acked Write 3 Done Sent Done false
28537 8264 Acked Read 8 Done Sent Done false
28538 8265 Acked Read 5 Done Sent Done false
28539 8266 Acked Read 1 Done Sent Done false
28540 8267 Acked Read 3 Done Sent Done false
28541 8268 Acked Write 5 Done Sent Done false
28542 8269 Acked Read 10 Done Sent Done false
28543 8270 Acked Write 2 Done Sent Done false
28544 8271 Acked Write 5 Done Sent Done false
28545 8272 Acked Flush 0 Done Sent Done false
28546 8273 Acked Write 9 Done Sent Done false
28547 8274 Acked Write 10 Done Sent Done false
28548 8275 Acked Read 1 Done Sent Done false
28549 8276 Acked Write 4 Done Sent Done false
28550 8277 Acked Write 9 Done Sent Done false
28551 8278 Acked Write 7 Done Sent Done false
28552 8279 Acked Read 9 Done Sent Done false
28553 8280 Acked Read 4 Done Sent Done false
28554 8281 Acked Read 4 Done Sent Done false
28555 8282 Acked Read 1 Done Sent Done false
28556 8283 Acked Write 10 Done Sent Done false
28557 8284 Acked Flush 0 Done Sent Done false
28558 8285 Acked Write 3 Done Sent Done false
28559 8286 Acked Write 10 Done Sent Done false
28560 8287 Acked Flush 0 Done Sent Done false
28561 8288 Acked Write 1 Done Sent Done false
28562 8289 Acked Read 1 Done Sent Done false
28563 8290 Acked Read 3 Done Sent Done false
28564 8291 Acked Write 5 Done Sent Done false
28565 8292 Acked Read 3 Done Sent Done false
28566 8293 Acked Write 4 Done Sent Done false
28567 8294 Acked Read 1 Done Sent Done false
28568 8295 Acked Read 5 Done Sent Done false
28569 8296 Acked Read 7 Done Sent Done false
28570 8297 Acked Read 1 Done Sent Done false
28571 8298 Acked Read 8 Done Sent Done false
28572 8299 Acked Write 6 Done Sent Done false
28573 8300 Acked Write 1 Done Sent Done false
28574 8301 Acked Read 2 Done Sent Done false
28575 8302 Acked Read 5 Done Sent Done false
28576 8303 Acked Read 10 Done Sent Done false
28577 8304 Acked Read 5 Done Sent Done false
28578 8305 Acked Write 5 Done Sent Done false
28579 8306 Acked Write 5 Done Sent Done false
28580 8307 Acked Read 9 Done Sent Done false
28581 8308 Acked Write 8 Done Sent Done false
28582 8309 Acked Write 5 Done Sent Done false
28583 8310 Acked Write 8 Done Sent Done false
28584 8311 Acked Read 2 Done Sent Done false
28585 8312 Acked Read 7 Done Sent Done false
28586 8313 Acked Flush 0 Done Sent Done false
28587 8314 Acked Write 9 Done Sent Done false
28588 8315 Acked Read 7 Done Sent Done false
28589 8316 Acked Write 4 Done Sent Done false
28590 8317 Acked Write 1 Done Sent Done false
28591 8318 Acked Read 8 Done Sent Done false
28592 8319 Acked Write 4 Done Sent Done false
28593 8320 Acked Write 7 Done Sent Done false
28594 8321 Acked Flush 0 Done Sent Done false
28595 8322 Acked Read 10 Done Sent Done false
28596 8323 Acked Write 9 Done Sent Done false
28597 8324 Acked Write 4 Done Sent Done false
28598 8325 Acked Write 7 Done Sent Done false
28599 8326 Acked Write 2 Done Sent Done false
28600 8327 Acked Read 5 Done Sent Done false
28601 8328 Acked Read 10 Done Sent Done false
28602 8329 Acked Read 5 Done Sent Done false
28603 8330 Acked Flush 0 Done Sent Done false
28604 8331 Acked Flush 0 Done Sent Done false
28605 8332 Acked Read 8 Done Sent Done false
28606 8333 Acked Read 2 Done Sent Done false
28607 8334 Acked Read 6 Done Sent Done false
28608 8335 Acked Write 8 Done Sent Done false
28609 8336 Acked Read 3 Done Sent Done false
28610 8337 Acked Read 7 Done Sent Done false
28611 8338 Acked Write 2 Done Sent Done false
28612 8339 Acked Write 8 Done Sent Done false
28613 8340 Acked Read 10 Done Sent Done false
28614 8341 Acked Read 6 Done Sent Done false
28615 8342 Acked Read 2 Done Sent Done false
28616 8343 Acked Write 3 Done Sent Done false
28617 8344 Acked Read 4 Done Sent Done false
28618 8345 Acked Read 7 Done Sent Done false
28619 8346 Acked Write 7 Done Sent Done false
28620 8347 Acked Flush 0 Done Sent Done false
28621 8348 Acked Flush 0 Done Sent Done false
28622 8349 Acked Read 1 Done Sent Done false
28623 8350 Acked Write 6 Done Sent Done false
28624 8351 Acked Read 2 Done Sent Done false
28625 8352 Acked Write 5 Done Sent Done false
28626 8353 Acked Write 7 Done Sent Done false
28627 8354 Acked Write 5 Done Sent Done false
28628 8355 Acked Flush 0 Done Sent Done false
28629 8356 Acked Write 9 Done Sent Done false
28630 8357 Acked Read 1 Done Sent Done false
28631 8358 Acked Write 4 Done Sent Done false
28632 8359 Acked Read 3 Done Sent Done false
28633 8360 Acked Read 1 Done Sent Done false
28634 8361 Acked Read 2 Done Sent Done false
28635 8362 Acked Read 5 Done Sent Done false
28636 8363 Acked Write 10 Done Sent Done false
28637 8364 Acked Read 5 Done Sent Done false
28638 8365 Acked Read 8 Done Sent Done false
28639 8366 Acked Write 6 Done Sent Done false
28640 8367 Acked Write 3 Done Sent Done false
28641 8368 Acked Write 1 Done Sent Done false
28642 8369 Acked Write 5 Done Sent Done false
28643 8370 Acked Read 5 Done Sent Done false
28644 8371 Acked Write 6 Done Sent Done false
28645 8372 Acked Read 7 Done Sent Done false
28646 8373 Acked Write 8 Done Sent Done false
28647 8374 Acked Write 10 Done Sent Done false
28648 8375 Acked Read 3 Done Sent Done false
28649 8376 Acked Write 2 Done Sent Done false
28650 8377 Acked Read 2 Done Sent Done false
28651 8378 Acked Flush 0 Done Sent Done false
28652 8379 Acked Read 4 Done Sent Done false
28653 8380 Acked Write 3 Done Sent Done false
28654 8381 Acked Read 5 Done Sent Done false
28655 8382 Acked Flush 0 Done Sent Done false
28656 8383 Acked Flush 0 Done Sent Done false
28657 8384 Acked Read 5 Done Sent Done false
28658 8385 Acked Read 6 Done Sent Done false
28659 8386 Acked Write 2 Done Sent Done false
28660 8387 Acked Read 3 Done Sent Done false
28661 8388 Acked Write 8 Done Sent Done false
28662 8389 Acked Read 4 Done Sent Done false
28663 8390 Acked Read 2 Done Sent Done false
28664 8391 Acked Write 4 Done Sent Done false
28665 8392 Acked Write 7 Done Sent Done false
28666 8393 Acked Flush 0 Done Sent Done false
28667 8394 Acked Write 8 Done Sent Done false
28668 8395 Acked Read 2 Done Sent Done false
28669 8396 Acked Read 8 Done Sent Done false
28670 8397 Acked Flush 0 Done Sent Done false
28671 8398 Acked Write 1 Done Sent Done false
28672 8399 Acked Flush 0 Done Sent Done false
28673 8400 Acked Flush 0 Done Sent Done false
28674 8401 Acked Read 6 Done Sent Done false
28675 8402 Acked Read 7 Done Sent Done false
28676 8403 Acked Read 4 Done Sent Done false
28677 8404 Acked Read 9 Done Sent Done false
28678 8405 Acked Write 5 Done Sent Done false
28679 8406 Acked Write 4 Done Sent Done false
28680 8407 Acked Read 1 Done Sent Done false
28681 8408 Acked Write 10 Done Sent Done false
28682 8409 Acked Write 6 Done Sent Done false
28683 8410 Acked Read 1 Done Sent Done false
28684 8411 Acked Read 10 Done Sent Done false
28685 8412 Acked Read 7 Done Sent Done false
28686 8413 Acked Flush 0 Done Sent Done false
28687 8414 Acked Read 6 Done Sent Done false
28688 8415 Acked Read 4 Done Sent Done false
28689 8416 Acked Read 5 Done Sent Done false
28690 8417 Acked Read 4 Done Sent Done false
28691 8418 Acked Read 7 Done Sent Done false
28692 8419 Acked Write 8 Done Sent Done false
28693 8420 Acked Read 6 Done Sent Done false
28694 8421 Acked Read 1 Done Sent Done false
28695 8422 Acked Read 1 Done Sent Done false
28696 8423 Acked Flush 0 Done Sent Done false
28697 8424 Acked Write 10 Done Sent Done false
28698 8425 Acked Read 7 Done Sent Done false
28699 8426 Acked Read 5 Done Sent Done false
28700 8427 Acked Read 2 Done Sent Done false
28701 8428 Acked Write 7 Done Sent Done false
28702 8429 Acked Write 2 Done Sent Done false
28703 8430 Acked Write 5 Done Sent Done false
28704 8431 Acked Read 1 Done Sent Done false
28705 8432 Acked Write 10 Done Sent Done false
28706 8433 Acked Read 5 Done Sent Done false
28707 8434 Acked Read 1 Done Sent Done false
28708 8435 Acked Write 10 Done Sent Done false
28709 8436 Acked Write 3 Done Sent Done false
28710 8437 Acked Write 1 Done Sent Done false
28711 8438 Acked Write 1 Done Sent Done false
28712 8439 Acked Write 10 Done Sent Done false
28713 8440 Acked Read 6 Done Sent Done false
28714 8441 Acked Read 4 Done Sent Done false
28715 8442 Acked Read 1 Done Sent Done false
28716 8443 Acked Write 8 Done Sent Done false
28717 8444 Acked Read 1 Done Sent Done false
28718 8445 Acked Flush 0 Done Sent Done false
28719 8446 Acked Write 5 Done Sent Done false
28720 8447 Acked Read 10 Done Sent Done false
28721 8448 Acked Read 9 Done Sent Done false
28722 8449 Acked Write 2 Done Sent Done false
28723 8450 Acked Flush 0 Done Sent Done false
28724 8451 Acked Read 6 Done Sent Done false
28725 8452 Acked Write 3 Done Sent Done false
28726 8453 Acked Write 8 Done Sent Done false
28727 8454 Acked Write 4 Done Sent Done false
28728 8455 Acked Read 5 Done Sent Done false
28729 8456 Acked Write 6 Done Sent Done false
28730 8457 Acked Write 9 Done Sent Done false
28731 8458 Acked Write 6 Done Sent Done false
28732 8459 Acked Read 7 Done Sent Done false
28733 8460 Acked Read 10 Done Sent Done false
28734 8461 Acked Flush 0 Done Sent Done false
28735 8462 Acked Read 3 Done Sent Done false
28736 8463 Acked Write 4 Done Sent Done false
28737 8464 Acked Write 9 Done Sent Done false
28738 8465 Acked Read 5 Done Sent Done false
28739 8466 Acked Read 7 Done Sent Done false
28740 8467 Acked Read 1 Done Sent Done false
28741 8468 Acked Read 10 Done Sent Done false
28742 8469 Acked Write 1 Done Sent Done false
28743 8470 Acked Write 8 Done Sent Done false
28744 8471 Acked Read 5 Done Sent Done false
28745 8472 Acked Write 5 Done Sent Done false
28746 8473 Acked Read 6 Done Sent Done false
28747 8474 Acked Read 6 Done Sent Done false
28748 8475 Acked Flush 0 Done Sent Done false
28749 8476 Acked Read 6 Done Sent Done false
28750 8477 Acked Read 6 Done Sent Done false
28751 8478 Acked Read 10 Done Sent Done false
28752 8479 Acked Read 1 Done Sent Done false
28753 8480 Acked Write 9 Done Sent Done false
28754 8481 Acked Write 10 Done Sent Done false
28755 8482 Acked Flush 0 Done Sent Done false
28756 8483 Acked Read 7 Done Sent Done false
28757 8484 Acked Read 2 Done Sent Done false
28758 8485 Acked Read 4 Done Sent Done false
28759 8486 Acked Write 8 Done Sent Done false
28760 8487 Acked Flush 0 Done Sent Done false
28761 8488 Acked Flush 0 Done Sent Done false
28762 8489 Acked Read 9 Done Sent Done false
28763 8490 Acked Read 7 Done Sent Done false
28764 8491 Acked Write 5 Done Sent Done false
28765 8492 Acked Read 7 Done Sent Done false
28766 8493 Acked Read 7 Done Sent Done false
28767 8494 Acked Read 2 Done Sent Done false
28768 8495 Acked Write 3 Done Sent Done false
28769 8496 Acked Read 2 Done Sent Done false
28770 8497 Acked Read 1 Done Sent Done false
28771 8498 Acked Read 7 Done Sent Done false
28772 8499 Acked Write 5 Done Sent Done false
28773 8500 Acked Write 5 Done Sent Done false
28774 8501 Acked Write 9 Done Sent Done false
28775 8502 Acked Read 7 Done Sent Done false
28776 8503 Acked Write 6 Done Sent Done false
28777 8504 Acked Read 4 Done Sent Done false
28778 8505 Acked Read 2 Done Sent Done false
28779 8506 Acked Flush 0 Done Sent Done false
28780 8507 Acked Read 10 Done Sent Done false
28781 8508 Acked Write 8 Done Sent Done false
28782 8509 Acked Write 8 Done Sent Done false
28783 8510 Acked Read 4 Done Sent Done false
28784 8511 Acked Read 8 Done Sent Done false
28785 8512 Acked Write 5 Done Sent Done false
28786 8513 Acked Read 9 Done Sent Done false
28787 8514 Acked Read 5 Done Sent Done false
28788 8515 Acked Write 4 Done Sent Done false
28789 8516 Acked Write 1 Done Sent Done false
28790 8517 Acked Write 10 Done Sent Done false
28791 8518 Acked Read 9 Done Sent Done false
28792 8519 Acked Read 9 Done Sent Done false
28793 8520 Acked Read 1 Done Sent Done false
28794 8521 Acked Write 7 Done Sent Done false
28795 8522 Acked Read 4 Done Sent Done false
28796 8523 Acked Read 9 Done Sent Done false
28797 8524 Acked Flush 0 Done Sent Done false
28798 8525 Acked Write 7 Done Sent Done false
28799 8526 Acked Read 2 Done Sent Done false
28800 8527 Acked Read 4 Done Sent Done false
28801 8528 Acked Write 9 Done Sent Done false
28802 8529 Acked Flush 0 Done Sent Done false
28803 8530 Acked Read 1 Done Sent Done false
28804 8531 Acked Write 7 Done Sent Done false
28805 8532 Acked Read 8 Done Sent Done false
28806 8533 Acked Read 3 Done Sent Done false
28807 8534 Acked Write 4 Done Sent Done false
28808 8535 Acked Write 7 Done Sent Done false
28809 8536 Acked Write 10 Done Sent Done false
28810 8537 Acked Write 1 Done Sent Done false
28811 8538 Acked Write 9 Done Sent Done false
28812 8539 Acked Read 5 Done Sent Done false
28813 8540 Acked Flush 0 Done Sent Done false
28814 8541 Acked Read 10 Done Sent Done false
28815 8542 Acked Read 4 Done Sent Done false
28816 8543 Acked Write 9 Done Sent Done false
28817 8544 Acked Read 8 Done Sent Done false
28818 STATES DS:0 DS:1 DS:2 TOTAL
28819 Sent 0 301 0 301
28820 Done 301 0 301 602
28821 Skipped 0 0 0 0
28822 Error 0 0 0 0
28823 Last Flush: Some(JobId(8540)) Some(JobId(8243)) Some(JobId(8540))
28824 Downstairs last five completed: 8243 8242 8241 8240 8239
28825 Upstairs last five completed: 8544 8543 8542 8541 8540
28826 CLIENT: Up:0 ds:301 act:2
288272025-03-07T18:18:51.576ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
28828 [8820] got start action
28829 [8820] I am going to restart
28830 Starting downstairs at port 8820
28831 Make output file at "/tmp/dsc/downstairs-8820.txt"
28832 [8820][1] reports Starting
28833 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
28834 [8820][1] reports Running
288352025-03-07T18:18:52.891ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
288362025-03-07T18:18:53.580ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
288372025-03-07T18:18:54.582ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
28838 ----------------------------------------------------------------
28839 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28840 States: Active Offline Active
28841 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28842 8244 Acked Read 9 Done Sent Done false
28843 8245 Acked Write 8 Done Sent Done false
28844 8246 Acked Flush 0 Done Sent Done false
28845 8247 Acked Read 1 Done Sent Done false
28846 8248 Acked Read 1 Done Sent Done false
28847 8249 Acked Read 10 Done Sent Done false
28848 8250 Acked Write 6 Done Sent Done false
28849 8251 Acked Read 10 Done Sent Done false
28850 8252 Acked Read 7 Done Sent Done false
28851 8253 Acked Read 1 Done Sent Done false
28852 8254 Acked Read 10 Done Sent Done false
28853 8255 Acked Read 6 Done Sent Done false
28854 8256 Acked Write 2 Done Sent Done false
28855 8257 Acked Flush 0 Done Sent Done false
28856 8258 Acked Write 5 Done Sent Done false
28857 8259 Acked Read 7 Done Sent Done false
28858 8260 Acked Read 4 Done Sent Done false
28859 8261 Acked Read 6 Done Sent Done false
28860 8262 Acked Read 4 Done Sent Done false
28861 8263 Acked Write 3 Done Sent Done false
28862 8264 Acked Read 8 Done Sent Done false
28863 8265 Acked Read 5 Done Sent Done false
28864 8266 Acked Read 1 Done Sent Done false
28865 8267 Acked Read 3 Done Sent Done false
28866 8268 Acked Write 5 Done Sent Done false
28867 8269 Acked Read 10 Done Sent Done false
28868 8270 Acked Write 2 Done Sent Done false
28869 8271 Acked Write 5 Done Sent Done false
28870 8272 Acked Flush 0 Done Sent Done false
28871 8273 Acked Write 9 Done Sent Done false
28872 8274 Acked Write 10 Done Sent Done false
28873 8275 Acked Read 1 Done Sent Done false
28874 8276 Acked Write 4 Done Sent Done false
28875 8277 Acked Write 9 Done Sent Done false
28876 8278 Acked Write 7 Done Sent Done false
28877 8279 Acked Read 9 Done Sent Done false
28878 8280 Acked Read 4 Done Sent Done false
28879 8281 Acked Read 4 Done Sent Done false
28880 8282 Acked Read 1 Done Sent Done false
28881 8283 Acked Write 10 Done Sent Done false
28882 8284 Acked Flush 0 Done Sent Done false
28883 8285 Acked Write 3 Done Sent Done false
28884 8286 Acked Write 10 Done Sent Done false
28885 8287 Acked Flush 0 Done Sent Done false
28886 8288 Acked Write 1 Done Sent Done false
28887 8289 Acked Read 1 Done Sent Done false
28888 8290 Acked Read 3 Done Sent Done false
28889 8291 Acked Write 5 Done Sent Done false
28890 8292 Acked Read 3 Done Sent Done false
28891 8293 Acked Write 4 Done Sent Done false
28892 8294 Acked Read 1 Done Sent Done false
28893 8295 Acked Read 5 Done Sent Done false
28894 8296 Acked Read 7 Done Sent Done false
28895 8297 Acked Read 1 Done Sent Done false
28896 8298 Acked Read 8 Done Sent Done false
28897 8299 Acked Write 6 Done Sent Done false
28898 8300 Acked Write 1 Done Sent Done false
28899 8301 Acked Read 2 Done Sent Done false
28900 8302 Acked Read 5 Done Sent Done false
28901 8303 Acked Read 10 Done Sent Done false
28902 8304 Acked Read 5 Done Sent Done false
28903 8305 Acked Write 5 Done Sent Done false
28904 8306 Acked Write 5 Done Sent Done false
28905 8307 Acked Read 9 Done Sent Done false
28906 8308 Acked Write 8 Done Sent Done false
28907 8309 Acked Write 5 Done Sent Done false
28908 8310 Acked Write 8 Done Sent Done false
28909 8311 Acked Read 2 Done Sent Done false
28910 8312 Acked Read 7 Done Sent Done false
28911 8313 Acked Flush 0 Done Sent Done false
28912 8314 Acked Write 9 Done Sent Done false
28913 8315 Acked Read 7 Done Sent Done false
28914 8316 Acked Write 4 Done Sent Done false
28915 8317 Acked Write 1 Done Sent Done false
28916 8318 Acked Read 8 Done Sent Done false
28917 8319 Acked Write 4 Done Sent Done false
28918 8320 Acked Write 7 Done Sent Done false
28919 8321 Acked Flush 0 Done Sent Done false
28920 8322 Acked Read 10 Done Sent Done false
28921 8323 Acked Write 9 Done Sent Done false
28922 8324 Acked Write 4 Done Sent Done false
28923 8325 Acked Write 7 Done Sent Done false
28924 8326 Acked Write 2 Done Sent Done false
28925 8327 Acked Read 5 Done Sent Done false
28926 8328 Acked Read 10 Done Sent Done false
28927 8329 Acked Read 5 Done Sent Done false
28928 8330 Acked Flush 0 Done Sent Done false
28929 8331 Acked Flush 0 Done Sent Done false
28930 8332 Acked Read 8 Done Sent Done false
28931 8333 Acked Read 2 Done Sent Done false
28932 8334 Acked Read 6 Done Sent Done false
28933 8335 Acked Write 8 Done Sent Done false
28934 8336 Acked Read 3 Done Sent Done false
28935 8337 Acked Read 7 Done Sent Done false
28936 8338 Acked Write 2 Done Sent Done false
28937 8339 Acked Write 8 Done Sent Done false
28938 8340 Acked Read 10 Done Sent Done false
28939 8341 Acked Read 6 Done Sent Done false
28940 8342 Acked Read 2 Done Sent Done false
28941 8343 Acked Write 3 Done Sent Done false
28942 8344 Acked Read 4 Done Sent Done false
28943 8345 Acked Read 7 Done Sent Done false
28944 8346 Acked Write 7 Done Sent Done false
28945 8347 Acked Flush 0 Done Sent Done false
28946 8348 Acked Flush 0 Done Sent Done false
28947 8349 Acked Read 1 Done Sent Done false
28948 8350 Acked Write 6 Done Sent Done false
28949 8351 Acked Read 2 Done Sent Done false
28950 8352 Acked Write 5 Done Sent Done false
28951 8353 Acked Write 7 Done Sent Done false
28952 8354 Acked Write 5 Done Sent Done false
28953 8355 Acked Flush 0 Done Sent Done false
28954 8356 Acked Write 9 Done Sent Done false
28955 8357 Acked Read 1 Done Sent Done false
28956 8358 Acked Write 4 Done Sent Done false
28957 8359 Acked Read 3 Done Sent Done false
28958 8360 Acked Read 1 Done Sent Done false
28959 8361 Acked Read 2 Done Sent Done false
28960 8362 Acked Read 5 Done Sent Done false
28961 8363 Acked Write 10 Done Sent Done false
28962 8364 Acked Read 5 Done Sent Done false
28963 8365 Acked Read 8 Done Sent Done false
28964 8366 Acked Write 6 Done Sent Done false
28965 8367 Acked Write 3 Done Sent Done false
28966 8368 Acked Write 1 Done Sent Done false
28967 8369 Acked Write 5 Done Sent Done false
28968 8370 Acked Read 5 Done Sent Done false
28969 8371 Acked Write 6 Done Sent Done false
28970 8372 Acked Read 7 Done Sent Done false
28971 8373 Acked Write 8 Done Sent Done false
28972 8374 Acked Write 10 Done Sent Done false
28973 8375 Acked Read 3 Done Sent Done false
28974 8376 Acked Write 2 Done Sent Done false
28975 8377 Acked Read 2 Done Sent Done false
28976 8378 Acked Flush 0 Done Sent Done false
28977 8379 Acked Read 4 Done Sent Done false
28978 8380 Acked Write 3 Done Sent Done false
28979 8381 Acked Read 5 Done Sent Done false
28980 8382 Acked Flush 0 Done Sent Done false
28981 8383 Acked Flush 0 Done Sent Done false
28982 8384 Acked Read 5 Done Sent Done false
28983 8385 Acked Read 6 Done Sent Done false
28984 8386 Acked Write 2 Done Sent Done false
28985 8387 Acked Read 3 Done Sent Done false
28986 8388 Acked Write 8 Done Sent Done false
28987 8389 Acked Read 4 Done Sent Done false
28988 8390 Acked Read 2 Done Sent Done false
28989 8391 Acked Write 4 Done Sent Done false
28990 8392 Acked Write 7 Done Sent Done false
28991 8393 Acked Flush 0 Done Sent Done false
28992 8394 Acked Write 8 Done Sent Done false
28993 8395 Acked Read 2 Done Sent Done false
28994 8396 Acked Read 8 Done Sent Done false
28995 8397 Acked Flush 0 Done Sent Done false
28996 8398 Acked Write 1 Done Sent Done false
28997 8399 Acked Flush 0 Done Sent Done false
28998 8400 Acked Flush 0 Done Sent Done false
28999 8401 Acked Read 6 Done Sent Done false
29000 8402 Acked Read 7 Done Sent Done false
29001 8403 Acked Read 4 Done Sent Done false
29002 8404 Acked Read 9 Done Sent Done false
29003 8405 Acked Write 5 Done Sent Done false
29004 8406 Acked Write 4 Done Sent Done false
29005 8407 Acked Read 1 Done Sent Done false
29006 8408 Acked Write 10 Done Sent Done false
29007 8409 Acked Write 6 Done Sent Done false
29008 8410 Acked Read 1 Done Sent Done false
29009 8411 Acked Read 10 Done Sent Done false
29010 8412 Acked Read 7 Done Sent Done false
29011 8413 Acked Flush 0 Done Sent Done false
29012 8414 Acked Read 6 Done Sent Done false
29013 8415 Acked Read 4 Done Sent Done false
29014 8416 Acked Read 5 Done Sent Done false
29015 8417 Acked Read 4 Done Sent Done false
29016 8418 Acked Read 7 Done Sent Done false
29017 8419 Acked Write 8 Done Sent Done false
29018 8420 Acked Read 6 Done Sent Done false
29019 8421 Acked Read 1 Done Sent Done false
29020 8422 Acked Read 1 Done Sent Done false
29021 8423 Acked Flush 0 Done Sent Done false
29022 8424 Acked Write 10 Done Sent Done false
29023 8425 Acked Read 7 Done Sent Done false
29024 8426 Acked Read 5 Done Sent Done false
29025 8427 Acked Read 2 Done Sent Done false
29026 8428 Acked Write 7 Done Sent Done false
29027 8429 Acked Write 2 Done Sent Done false
29028 8430 Acked Write 5 Done Sent Done false
29029 8431 Acked Read 1 Done Sent Done false
29030 8432 Acked Write 10 Done Sent Done false
29031 8433 Acked Read 5 Done Sent Done false
29032 8434 Acked Read 1 Done Sent Done false
29033 8435 Acked Write 10 Done Sent Done false
29034 8436 Acked Write 3 Done Sent Done false
29035 8437 Acked Write 1 Done Sent Done false
29036 8438 Acked Write 1 Done Sent Done false
29037 8439 Acked Write 10 Done Sent Done false
29038 8440 Acked Read 6 Done Sent Done false
29039 8441 Acked Read 4 Done Sent Done false
29040 8442 Acked Read 1 Done Sent Done false
29041 8443 Acked Write 8 Done Sent Done false
29042 8444 Acked Read 1 Done Sent Done false
29043 8445 Acked Flush 0 Done Sent Done false
29044 8446 Acked Write 5 Done Sent Done false
29045 8447 Acked Read 10 Done Sent Done false
29046 8448 Acked Read 9 Done Sent Done false
29047 8449 Acked Write 2 Done Sent Done false
29048 8450 Acked Flush 0 Done Sent Done false
29049 8451 Acked Read 6 Done Sent Done false
29050 8452 Acked Write 3 Done Sent Done false
29051 8453 Acked Write 8 Done Sent Done false
29052 8454 Acked Write 4 Done Sent Done false
29053 8455 Acked Read 5 Done Sent Done false
29054 8456 Acked Write 6 Done Sent Done false
29055 8457 Acked Write 9 Done Sent Done false
29056 8458 Acked Write 6 Done Sent Done false
29057 8459 Acked Read 7 Done Sent Done false
29058 8460 Acked Read 10 Done Sent Done false
29059 8461 Acked Flush 0 Done Sent Done false
29060 8462 Acked Read 3 Done Sent Done false
29061 8463 Acked Write 4 Done Sent Done false
29062 8464 Acked Write 9 Done Sent Done false
29063 8465 Acked Read 5 Done Sent Done false
29064 8466 Acked Read 7 Done Sent Done false
29065 8467 Acked Read 1 Done Sent Done false
29066 8468 Acked Read 10 Done Sent Done false
29067 8469 Acked Write 1 Done Sent Done false
29068 8470 Acked Write 8 Done Sent Done false
29069 8471 Acked Read 5 Done Sent Done false
29070 8472 Acked Write 5 Done Sent Done false
29071 8473 Acked Read 6 Done Sent Done false
29072 8474 Acked Read 6 Done Sent Done false
29073 8475 Acked Flush 0 Done Sent Done false
29074 8476 Acked Read 6 Done Sent Done false
29075 8477 Acked Read 6 Done Sent Done false
29076 8478 Acked Read 10 Done Sent Done false
29077 8479 Acked Read 1 Done Sent Done false
29078 8480 Acked Write 9 Done Sent Done false
29079 8481 Acked Write 10 Done Sent Done false
29080 8482 Acked Flush 0 Done Sent Done false
29081 8483 Acked Read 7 Done Sent Done false
29082 8484 Acked Read 2 Done Sent Done false
29083 8485 Acked Read 4 Done Sent Done false
29084 8486 Acked Write 8 Done Sent Done false
29085 8487 Acked Flush 0 Done Sent Done false
29086 8488 Acked Flush 0 Done Sent Done false
29087 8489 Acked Read 9 Done Sent Done false
29088 8490 Acked Read 7 Done Sent Done false
29089 8491 Acked Write 5 Done Sent Done false
29090 8492 Acked Read 7 Done Sent Done false
29091 8493 Acked Read 7 Done Sent Done false
29092 8494 Acked Read 2 Done Sent Done false
29093 8495 Acked Write 3 Done Sent Done false
29094 8496 Acked Read 2 Done Sent Done false
29095 8497 Acked Read 1 Done Sent Done false
29096 8498 Acked Read 7 Done Sent Done false
29097 8499 Acked Write 5 Done Sent Done false
29098 8500 Acked Write 5 Done Sent Done false
29099 8501 Acked Write 9 Done Sent Done false
29100 8502 Acked Read 7 Done Sent Done false
29101 8503 Acked Write 6 Done Sent Done false
29102 8504 Acked Read 4 Done Sent Done false
29103 8505 Acked Read 2 Done Sent Done false
29104 8506 Acked Flush 0 Done Sent Done false
29105 8507 Acked Read 10 Done Sent Done false
29106 8508 Acked Write 8 Done Sent Done false
29107 8509 Acked Write 8 Done Sent Done false
29108 8510 Acked Read 4 Done Sent Done false
29109 8511 Acked Read 8 Done Sent Done false
29110 8512 Acked Write 5 Done Sent Done false
29111 8513 Acked Read 9 Done Sent Done false
29112 8514 Acked Read 5 Done Sent Done false
29113 8515 Acked Write 4 Done Sent Done false
29114 8516 Acked Write 1 Done Sent Done false
29115 8517 Acked Write 10 Done Sent Done false
29116 8518 Acked Read 9 Done Sent Done false
29117 8519 Acked Read 9 Done Sent Done false
29118 8520 Acked Read 1 Done Sent Done false
29119 8521 Acked Write 7 Done Sent Done false
29120 8522 Acked Read 4 Done Sent Done false
29121 8523 Acked Read 9 Done Sent Done false
29122 8524 Acked Flush 0 Done Sent Done false
29123 8525 Acked Write 7 Done Sent Done false
29124 8526 Acked Read 2 Done Sent Done false
29125 8527 Acked Read 4 Done Sent Done false
29126 8528 Acked Write 9 Done Sent Done false
29127 8529 Acked Flush 0 Done Sent Done false
29128 8530 Acked Read 1 Done Sent Done false
29129 8531 Acked Write 7 Done Sent Done false
29130 8532 Acked Read 8 Done Sent Done false
29131 8533 Acked Read 3 Done Sent Done false
29132 8534 Acked Write 4 Done Sent Done false
29133 8535 Acked Write 7 Done Sent Done false
29134 8536 Acked Write 10 Done Sent Done false
29135 8537 Acked Write 1 Done Sent Done false
29136 8538 Acked Write 9 Done Sent Done false
29137 8539 Acked Read 5 Done Sent Done false
29138 8540 Acked Flush 0 Done Sent Done false
29139 8541 Acked Read 10 Done Sent Done false
29140 8542 Acked Read 4 Done Sent Done false
29141 8543 Acked Write 9 Done Sent Done false
29142 8544 Acked Read 8 Done Sent Done false
29143 STATES DS:0 DS:1 DS:2 TOTAL
29144 Sent 0 301 0 301
29145 Done 301 0 301 602
29146 Skipped 0 0 0 0
29147 Error 0 0 0 0
29148 Last Flush: Some(JobId(8540)) Some(JobId(8243)) Some(JobId(8540))
29149 Downstairs last five completed: 8243 8242 8241 8240 8239
29150 Upstairs last five completed: 8544 8543 8542 8541 8540
29151 CLIENT: Up:0 ds:301 act:2
291522025-03-07T18:18:55.655ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291532025-03-07T18:18:56.585ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291542025-03-07T18:18:56.649ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291552025-03-07T18:18:56.649ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63539) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291562025-03-07T18:18:56.651ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291572025-03-07T18:18:56.651ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291582025-03-07T18:18:56.651ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291592025-03-07T18:18:56.651ZINFOcrucible: send last flush ID to this DS: Some(JobId(8243)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291602025-03-07T18:18:56.651ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291612025-03-07T18:18:56.651ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8243)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291622025-03-07T18:18:56.651ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8243)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
29163 ----------------------------------------------------------------
29164 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29165 Downstairs last five completed: 8545 8544 8543 8542 8541
29166 Upstairs last five completed: 8545 8544 8543 8542 8541
29167 CLIENT: Up:0 ds:0 act:3
29168 Replay: All jobs finished
29169 Main task has work to do, go find it
29170 got dsc Stop(2)
29171 stop 2
29172 Mar 07 18:18:58.905 INFO request completed, latency_us[8830] Got stop action so:false kr:false
29173 : 296, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e9c466bf-c620-440b-b6cc-8a07b0779807, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
291742025-03-07T18:18:58.906ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291752025-03-07T18:18:58.906ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291762025-03-07T18:18:58.906ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
291772025-03-07T18:18:58.906ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
29178 Mar 07 18:18:58.906 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: 38264df8-503d-41e4-8d9c-f7d80f7f338a, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
29179 [8830] Exited with: ExitStatus(unix_wait_status(9))
29180 [8830][2] reports Exit
29181 Mar 07 18:19:02.909 INFO request completed, latency_us: 301, response_code: 200, uri: /state/cid/2, method: GET, req_id: d9cf5149-dda7-4ebc-a9a7-1a4802b6f980, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
29182 001/300 Write block 4112 len 36864 data: 7 9 7 6 6 6 6 6 8
29183 002/300 Read block 495 len 36864
29184 003/300 Write block 5 len 8192 data: 7 7
29185 004/300 Read block 714 len 28672
29186 005/300 Read block 3662 len 32768
29187 006/300 Read block 3714 len 4096
29188 007/300 Read block 3450 len 32768
29189 008/300 Flush
29190 009/300 Flush
29191 010/300 Write block 1708 len 36864 data: 4 4 4 4 5 5 7 8 8
29192 011/300 Write block 3913 len 40960 data: 5 6 6 7 6 7 8 8 9 11
29193 012/300 Read block 1714 len 32768
29194 013/300 Read block 3399 len 36864
29195 014/300 Write block 3463 len 16384 data: 5 4 5 6
29196 015/300 Write block 4583 len 20480 data: 5 6 6 7 6
29197 016/300 Flush
29198 017/300 Write block 3642 len 4096 data: 5
29199 018/300 Flush
29200 019/300 Write block 3492 len 8192 data: 8 6
29201 020/300 Flush
29202 021/300 Write block 3066 len 8192 data: 8 7
29203 022/300 Write block 3751 len 24576 data: 5 5 4 5 6 5
29204 023/300 Write block 1572 len 4096 data: 7
29205 024/300 Write block 2504 len 12288 data: 5 5 6
29206 025/300 Write block 2043 len 24576 data: 6 7 8 9 10 9
29207 026/300 Write block 2855 len 24576 data: 4 3 3 4 5 5
29208 027/300 Read block 2790 len 40960
29209 028/300 Write block 444 len 36864 data: 3 4 5 4 4 5 5 5 4
29210 029/300 Write block 1853 len 8192 data: 6 6
29211 030/300 Read block 3125 len 40960
29212 031/300 Write block 3860 len 4096 data: 4
29213 032/300 Read block 4001 len 8192
29214 033/300 Write block 956 len 28672 data: 7 8 8 7 7 6 6
29215 034/300 Read block 780 len 16384
29216 035/300 Write block 2264 len 24576 data: 5 6 6 4 4 6
29217 036/300 Read block 1893 len 40960
29218 037/300 Write block 3083 len 24576 data: 9 10 8 8 7 5
29219 038/300 Flush
29220 039/300 Flush
29221 040/300 Write block 4957 len 40960 data: 6 8 7 7 7 5 6 7 7 5
29222 041/300 Read block 995 len 12288
29223 042/300 Read block 2298 len 8192
29224 043/300 Read block 3226 len 8192
29225 044/300 Read block 1395 len 12288
29226 045/300 Read block 2917 len 8192
29227 046/300 Write block 2041 len 20480 data: 5 6 7 8 9
29228 047/300 Write block 2927 len 36864 data: 7 6 5 4 4 4 4 3 3
29229 048/300 Write block 2091 len 20480 data: 3 2 2 2 2
29230 049/300 Read block 4422 len 16384
29231 050/300 Write block 2895 len 28672 data: 6 6 5 5 6 7 7
29232 051/300 Read block 103 len 28672
29233 052/300 Read block 145 len 36864
29234 053/300 Read block 4258 len 16384
29235 054/300 Read block 4853 len 4096
29236 055/300 Write block 3377 len 28672 data: 8 7 7 7 7 7 8
29237 056/300 Read block 2989 len 4096
29238 057/300 Write block 3820 len 8192 data: 5 5
29239 058/300 Write block 4438 len 32768 data: 7 6 3 2 3 3 4 3
29240 059/300 Write block 836 len 4096 data: 5
29241 060/300 Read block 3711 len 4096
29242 061/300 Write block 1771 len 36864 data: 4 5 5 5 5 6 7 6 5
29243 062/300 Flush
29244 063/300 Read block 4027 len 12288
29245 064/300 Read block 4604 len 32768
29246 065/300 Read block 2137 len 4096
29247 066/300 Read block 3172 len 40960
29248 067/300 Flush
29249 068/300 Write block 2117 len 8192 data: 6 7
29250 069/300 Read block 386 len 24576
29251 070/300 Write block 3371 len 32768 data: 7 6 6 5 6 7 9 8
29252 071/300 Read block 86 len 20480
29253 072/300 Flush
29254 073/300 Read block 3248 len 16384
29255 074/300 Write block 4240 len 4096 data: 6
29256 075/300 Flush
29257 076/300 Write block 1124 len 28672 data: 5 5 7 8 9 8 6
29258 077/300 Write block 3428 len 36864 data: 5 4 4 4 3 4 7 7 6
29259 078/300 Flush
29260 079/300 Write block 1705 len 12288 data: 7 5 4
29261 080/300 Write block 259 len 36864 data: 4 4 6 6 7 8 9 9 8
29262 081/300 Read block 4959 len 8192
29263 082/300 Write block 1135 len 12288 data: 7 7 6
29264 083/300 Read block 3459 len 40960
29265 084/300 Write block 4601 len 40960 data: 7 7 6 5 4 5 6 6 7 8
29266 085/300 Read block 1095 len 24576
29267 086/300 Read block 674 len 8192
29268 087/300 Read block 1055 len 32768
29269 088/300 Write block 1264 len 8192 data: 5 5
29270 089/300 Write block 4542 len 24576 data: 7 8 8 9 10 10
29271 090/300 Read block 4445 len 24576
29272 091/300 Flush
29273 092/300 Write block 2804 len 32768 data: 5 5 5 6 6 7 7 7
29274 093/300 Flush
29275 094/300 Read block 790 len 40960
29276 095/300 Write block 2415 len 4096 data: 4
29277 096/300 Read block 1641 len 24576
29278 097/300 Read block 3585 len 16384
29279 098/300 Read block 2954 len 32768
29280 099/300 Write block 3686 len 36864 data: 6 6 4 4 3 4 5 6 5
29281 100/300 Write block 2468 len 8192 data: 9 7
29282 101/300 Flush
29283 102/300 Write block 3904 len 4096 data: 4
29284 103/300 Flush
29285 104/300 Read block 2926 len 24576
29286 105/300 Write block 1446 len 20480 data: 3 3 3 3 6
29287 106/300 Write block 4923 len 32768 data: 7 7 5 5 5 4 3 2
29288 107/300 Write block 3566 len 24576 data: 6 7 7 8 5 3
29289 108/300 Read block 4533 len 36864
29290 109/300 Read block 4798 len 4096
29291 110/300 Read block 530 len 28672
29292 111/300 Read block 3066 len 20480
29293 112/300 Read block 2722 len 32768
29294 113/300 Read block 4187 len 8192
29295 114/300 Write block 4560 len 32768 data: 4 6 6 6 6 4 3 3
29296 115/300 Read block 1144 len 20480
29297 116/300 Write block 4136 len 12288 data: 6 4 4
29298 117/300 Read block 3795 len 20480
29299 118/300 Write block 1512 len 20480 data: 9 8 8 9 8
29300 119/300 Write block 2246 len 20480 data: 4 3 3 4 4
29301 120/300 Read block 4928 len 36864
29302 121/300 Write block 2271 len 4096 data: 5
29303 122/300 Write block 4879 len 24576 data: 5 6 5 3 4 4
29304 123/300 Flush
29305 124/300 Read block 4150 len 8192
29306 125/300 Write block 2507 len 20480 data: 5 5 6 5 5
29307 126/300 Read block 3826 len 8192
29308 127/300 Read block 2623 len 40960
29309 128/300 Read block 2950 len 28672
29310 129/300 Write block 1798 len 16384 data: 6 5 5 7
29311 130/300 Read block 692 len 28672
29312 131/300 Read block 4854 len 32768
29313 132/300 Write block 4748 len 12288 data: 6 4 4
29314 133/300 Write block 3030 len 28672 data: 4 4 4 4 3 4 4
29315 134/300 Read block 2139 len 12288
29316 135/300 Read block 19 len 36864
29317 136/300 Write block 3853 len 8192 data: 8 7
29318 137/300 Read block 1414 len 20480
29319 138/300 Read block 4406 len 16384
29320 139/300 Read block 874 len 12288
29321 140/300 Read block 3167 len 36864
29322 141/300 Write block 70 len 28672 data: 8 8 9 9 9 9 8
29323 142/300 Write block 2880 len 8192 data: 5 5
29324 143/300 Write block 1914 len 32768 data: 5 7 7 10 10 10 10 11
29325 144/300 Read block 3250 len 28672
29326 145/300 Write block 2945 len 36864 data: 8 7 7 6 5 7 6 5 4
29327 146/300 Read block 4060 len 16384
29328 147/300 Read block 288 len 36864
29329 148/300 Write block 1708 len 40960 data: 5 5 5 5 6 6 8 9 9 8
29330 149/300 Write block 4714 len 20480 data: 3 3 3 3 4
29331 150/300 Write block 846 len 20480 data: 3 4 4 4 5
29332 151/300 Flush
29333 152/300 Read block 2241 len 4096
29334 153/300 Read block 4454 len 4096
29335 154/300 Read block 3532 len 32768
29336 155/300 Flush
29337 156/300 Write block 2486 len 40960 data: 3 5 5 5 5 5 4 5 7 7
29338 157/300 Write block 3158 len 36864 data: 6 7 5 5 5 5 5 5 7
29339 158/300 Write block 4676 len 16384 data: 5 3 3 3
29340 159/300 Write block 2724 len 12288 data: 9 10 10
29341 160/300 Read block 3605 len 24576
29342 161/300 Write block 4461 len 12288 data: 7 7 8
29343 162/300 Write block 3052 len 40960 data: 4 5 5 6 4 6 6 6 6 7
29344 163/300 Flush
29345 164/300 Read block 2264 len 28672
29346 165/300 Flush
29347 166/300 Write block 1341 len 12288 data: 3 3 3
29348 167/300 Read block 2635 len 28672
29349 168/300 Write block 3267 len 16384 data: 7 6 6 4
29350 169/300 Read block 3779 len 24576
29351 170/300 Write block 3371 len 32768 data: 8 7 7 6 7 8 10 9
29352 171/300 Read block 3915 len 40960
29353 172/300 Write block 2532 len 28672 data: 8 8 9 6 6 5 5
29354 173/300 Write block 1534 len 8192 data: 6 6
29355 174/300 Read block 1388 len 32768
29356 175/300 Read block 1308 len 16384
29357 176/300 Read block 1696 len 36864
29358 177/300 Write block 1132 len 24576 data: 3 3 5 8 8 7
29359 178/300 Read block 2785 len 24576
29360 179/300 Flush
29361 180/300 Flush
29362 181/300 Flush
29363 182/300 Read block 2543 len 36864
29364 183/300 Read block 4673 len 20480
29365 184/300 Write block 4532 len 36864 data: 4 6 6 7 7 8 6 6 7
29366 185/300 Write block 799 len 8192 data: 5 4
29367 186/300 Flush
29368 187/300 Write block 454 len 16384 data: 3 4 4 4
29369 188/300 Read block 743 len 28672
29370 189/300 Write block 649 len 28672 data: 7 8 6 5 4 4 5
29371 190/300 Read block 2100 len 16384
29372 191/300 Flush
29373 192/300 Write block 4009 len 12288 data: 5 5 4
29374 193/300 Flush
29375 194/300 Read block 1559 len 16384
29376 195/300 Write block 3620 len 12288 data: 4 2 3
29377 196/300 Read block 4635 len 28672
29378 197/300 Read block 4508 len 36864
29379 198/300 Write block 2255 len 4096 data: 2
29380 199/300 Write block 2520 len 16384 data: 6 6 5 4
29381 200/300 Write block 4481 len 36864 data: 4 4 3 2 2 4 4 4 3
29382 201/300 Write block 4578 len 20480 data: 5 5 5 5 6
29383 202/300 Read block 3018 len 28672
29384 203/300 Read block 2447 len 24576
29385 204/300 Write block 663 len 4096 data: 6
29386 205/300 Read block 995 len 32768
29387 206/300 Read block 2334 len 16384
29388 207/300 Read block 3083 len 24576
29389 208/300 Write block 356 len 8192 data: 6 4
29390 209/300 Write block 4945 len 24576 data: 7 7 6 4 3 4
29391 210/300 Read block 177 len 40960
29392 211/300 Write block 1117 len 8192 data: 4 4
29393 212/300 Read block 3257 len 20480
29394 213/300 Write block 4976 len 40960 data: 3 3 4 6 7 5 5 5 6 6
29395 214/300 Read block 3906 len 8192
29396 215/300 Read block 422 len 40960
29397 216/300 Read block 4578 len 40960
29398 217/300 Write block 2443 len 32768 data: 5 5 6 7 6 3 3 5
29399 218/300 Read block 2404 len 40960
29400 219/300 Read block 3652 len 8192
29401 220/300 Write block 3970 len 32768 data: 7 7 7 6 5 4 3 3
29402 221/300 Read block 3639 len 4096
29403 222/300 Write block 4348 len 20480 data: 4 4 4 4 4
29404 223/300 Write block 3371 len 12288 data: 9 8 8
29405 224/300 Read block 580 len 28672
29406 225/300 Write block 1693 len 28672 data: 7 7 7 6 6 7 9
29407 226/300 Read block 4783 len 28672
29408 227/300 Write block 1075 len 8192 data: 4 6
29409 228/300 Write block 2248 len 28672 data: 4 5 5 3 3 3 3
29410 229/300 Read block 610 len 28672
29411 230/300 Write block 1360 len 12288 data: 9 10 9
29412 231/300 Flush
29413 232/300 Write block 4863 len 16384 data: 5 5 6 7
29414 233/300 Flush
29415 234/300 Write block 2513 len 8192 data: 8 7
29416 235/300 Write block 360 len 24576 data: 2 2 3 4 4 4
29417 236/300 Write block 3346 len 32768 data: 3 2 3 4 5 5 6 5
29418 237/300 Write block 8 len 16384 data: 7 7 8 9
29419 238/300 Read block 2524 len 12288
29420 239/300 Read block 4576 len 40960
29421 240/300 Write block 4641 len 32768 data: 5 3 4 6 6 5 5 5
29422 241/300 Read block 2446 len 24576
29423 242/300 Read block 4539 len 40960
29424 243/300 Write block 4672 len 8192 data: 7 6
29425 244/300 Flush
29426 245/300 Read block 623 len 36864
29427 246/300 Write block 4104 len 24576 data: 5 4 6 6 6 6
29428 247/300 Write block 2222 len 20480 data: 5 6 6 6 7
29429 248/300 Read block 2076 len 24576
29430 249/300 Write block 4352 len 32768 data: 5 5 5 4 5 3 3 3
29431 250/300 Write block 2426 len 16384 data: 3 5 6 6
29432 251/300 Read block 369 len 32768
29433 252/300 Flush
29434 253/300 Flush
29435 254/300 Flush
29436 255/300 Read block 562 len 12288
29437 256/300 Flush
29438 257/300 Write block 2698 len 24576 data: 2 4 5 5 6 5
29439 258/300 Flush
29440 259/300 Read block 3042 len 12288
29441 260/300 Write block 1201 len 24576 data: 5 4 4 5 6 6
29442 261/300 Write block 2275 len 8192 data: 3 4
29443 262/300 Read block 4912 len 28672
29444 263/300 Write block 3015 len 28672 data: 6 7 5 5 6 5 4
29445 264/300 Write block 560 len 24576 data: 4 5 4 5 6 6
29446 265/300 Read block 4458 len 36864
29447 266/300 Write block 3733 len 8192 data: 5 5
29448 267/300 Write block 2027 len 16384 data: 6 6 6 4
29449 268/300 Read block 3817 len 24576
29450 269/300 Read block 1584 len 20480
29451 270/300 Read block 269 len 16384
29452 271/300 Write block 4952 len 40960 data: 4 5 5 5 5 7 9 8 8 8
29453 272/300 Read block 189 len 40960
29454 273/300 Read block 1481 len 32768
29455 274/300 Write block 1532 len 12288 data: 7 7 7
29456 275/300 Flush
29457 276/300 Flush
29458 277/300 Flush
29459 278/300 Write block 1476 len 36864 data: 3 3 3 3 4 5 5 5 5
29460 279/300 Read block 4910 len 8192
29461 280/300 Read block 3995 len 24576
29462 281/300 Flush
29463 282/300 Write block 4795 len 4096 data: 7
29464 283/300 Flush
29465 284/300 Read block 4537 len 28672
29466 285/300 Read block 4883 len 28672
29467 286/300 Read block 1818 len 40960
29468 287/300 Write block 4174 len 36864 data: 6 8 8 8 7 6 7 6 5
29469 288/300 Read block 2987 len 32768
29470 289/300 Read block 3301 len 12288
29471 290/300 Read block 2171 len 36864
29472 291/300 Write block 4877 len 4096 data: 6
29473 292/300 Flush
29474 293/300 Read block 49 len 16384
29475 294/300 Write block 2526 len 28672 data: 3 5 5 5 5 8 9
29476 295/300 Write block 2052 len 36864 data: 9 7 6 6 6 6 3 4 2
29477 296/300 Read block 1915 len 4096
29478 297/300 Write block 4541 len 16384 data: 7 8 9 9
29479 298/300 Read block 2548 len 8192
29480 299/300 Read block 4481 len 24576
29481 300/300 Write block 3564 len 4096 data: 3
29482 301/300 Write block 4019 len 16384 data: 5 6 6 4
29483 Main task has work to do, go find it
29484 got dsc Start(2)
29485 start 2
29486 Mar 07 18:19:03.149 INFO request completed, latency_us: 178, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6bee260b-0ab1-467b-be42-9c1342313b6e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
29487 Replay: started 2, returned:Ok(())
29488 ----------------------------------------------------------------
29489 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29490 States: Active Active Offline
29491 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29492 8546 Acked Write 9 Done Done Sent false
29493 8547 Acked Read 9 Done Done Sent false
29494 8548 Acked Write 2 Done Done Sent false
29495 8549 Acked Read 7 Done Done Sent false
29496 8550 Acked Read 8 Done Done Sent false
29497 8551 Acked Read 1 Done Done Sent false
29498 8552 Acked Read 8 Done Done Sent false
29499 8553 Acked Flush 0 Done Done Sent false
29500 8554 Acked Flush 0 Done Done Sent false
29501 8555 Acked Write 9 Done Done Sent false
29502 8556 Acked Write 10 Done Done Sent false
29503 8557 Acked Read 8 Done Done Sent false
29504 8558 Acked Read 9 Done Done Sent false
29505 8559 Acked Write 4 Done Done Sent false
29506 8560 Acked Write 5 Done Done Sent false
29507 8561 Acked Flush 0 Done Done Sent false
29508 8562 Acked Write 1 Done Done Sent false
29509 8563 Acked Flush 0 Done Done Sent false
29510 8564 Acked Write 2 Done Done Sent false
29511 8565 Acked Flush 0 Done Done Sent false
29512 8566 Acked Write 2 Done Done Sent false
29513 8567 Acked Write 6 Done Done Sent false
29514 8568 Acked Write 1 Done Done Sent false
29515 8569 Acked Write 3 Done Done Sent false
29516 8570 Acked Write 6 Done Done Sent false
29517 8571 Acked Write 6 Done Done Sent false
29518 8572 Acked Read 10 Done Done Sent false
29519 8573 Acked Write 9 Done Done Sent false
29520 8574 Acked Write 2 Done Done Sent false
29521 8575 Acked Read 10 Done Done Sent false
29522 8576 Acked Write 1 Done Done Sent false
29523 8577 Acked Read 2 Done Done Sent false
29524 8578 Acked Write 7 Done Done Sent false
29525 8579 Acked Read 4 Done Done Sent false
29526 8580 Acked Write 6 Done Done Sent false
29527 8581 Acked Read 10 Done Done Sent false
29528 8582 Acked Write 6 Done Done Sent false
29529 8583 Acked Flush 0 Done Done Sent false
29530 8584 Acked Flush 0 Done Done Sent false
29531 8585 Acked Write 10 Done Done Sent false
29532 8586 Acked Read 3 Done Done Sent false
29533 8587 Acked Read 2 Done Done Sent false
29534 8588 Acked Read 2 Done Done Sent false
29535 8589 Acked Read 3 Done Done Sent false
29536 8590 Acked Read 2 Done Done Sent false
29537 8591 Acked Write 5 Done Done Sent false
29538 8592 Acked Write 9 Done Done Sent false
29539 8593 Acked Write 5 Done Done Sent false
29540 8594 Acked Read 4 Done Done Sent false
29541 8595 Acked Write 7 Done Done Sent false
29542 8596 Acked Read 7 Done Done Sent false
29543 8597 Acked Read 9 Done Done Sent false
29544 8598 Acked Read 4 Done Done Sent false
29545 8599 Acked Read 1 Done Done Sent false
29546 8600 Acked Write 7 Done Done Sent false
29547 8601 Acked Read 1 Done Done Sent false
29548 8602 Acked Write 2 Done Done Sent false
29549 8603 Acked Write 8 Done Done Sent false
29550 8604 Acked Write 1 Done Done Sent false
29551 8605 Acked Read 1 Done Done Sent false
29552 8606 Acked Write 9 Done Done Sent false
29553 8607 Acked Flush 0 Done Done Sent false
29554 8608 Acked Read 3 Done Done Sent false
29555 8609 Acked Read 8 Done Done Sent false
29556 8610 Acked Read 1 Done Done Sent false
29557 8611 Acked Read 10 Done Done Sent false
29558 8612 Acked Flush 0 Done Done Sent false
29559 8613 Acked Write 2 Done Done Sent false
29560 8614 Acked Read 6 Done Done Sent false
29561 8615 Acked Write 8 Done Done Sent false
29562 8616 Acked Read 5 Done Done Sent false
29563 8617 Acked Flush 0 Done Done Sent false
29564 8618 Acked Read 4 Done Done Sent false
29565 8619 Acked Write 1 Done Done Sent false
29566 8620 Acked Flush 0 Done Done Sent false
29567 8621 Acked Write 7 Done Done Sent false
29568 8622 Acked Write 9 Done Done Sent false
29569 8623 Acked Flush 0 Done Done Sent false
29570 8624 Acked Write 3 Done Done Sent false
29571 8625 Acked Write 9 Done Done Sent false
29572 8626 Acked Read 2 Done Done Sent false
29573 8627 Acked Write 3 Done Done Sent false
29574 8628 Acked Read 10 Done Done Sent false
29575 8629 Acked Write 10 Done Done Sent false
29576 8630 Acked Read 6 Done Done Sent false
29577 8631 Acked Read 2 Done Done Sent false
29578 8632 Acked Read 8 Done Done Sent false
29579 8633 Acked Write 2 Done Done Sent false
29580 8634 Acked Write 6 Done Done Sent false
29581 8635 Acked Read 6 Done Done Sent false
29582 8636 Acked Flush 0 Done Done Sent false
29583 8637 Acked Write 8 Done Done Sent false
29584 8638 Acked Flush 0 Done Done Sent false
29585 8639 Acked Read 10 Done Done Sent false
29586 8640 Acked Write 1 Done Done Sent false
29587 8641 Acked Read 6 Done Done Sent false
29588 8642 Acked Read 4 Done Done Sent false
29589 8643 Acked Read 8 Done Done Sent false
29590 8644 Acked Write 9 Done Done Sent false
29591 8645 Acked Write 2 Done Done Sent false
29592 8646 Acked Flush 0 Done Done Sent false
29593 8647 Acked Write 1 Done Done Sent false
29594 8648 Acked Flush 0 Done Done Sent false
29595 8649 Acked Read 6 Done Done Sent false
29596 8650 Acked Write 5 Done Done Sent false
29597 8651 Acked Write 8 Done Done Sent false
29598 8652 Acked Write 6 Done Done Sent false
29599 8653 Acked Read 9 Done Done Sent false
29600 8654 Acked Read 1 Done Done Sent false
29601 8655 Acked Read 7 Done Done Sent false
29602 8656 Acked Read 5 Done Done Sent false
29603 8657 Acked Read 8 Done Done Sent false
29604 8658 Acked Read 2 Done Done Sent false
29605 8659 Acked Write 8 Done Done Sent false
29606 8660 Acked Read 5 Done Done Sent false
29607 8661 Acked Write 3 Done Done Sent false
29608 8662 Acked Read 5 Done Done Sent false
29609 8663 Acked Write 5 Done Done Sent false
29610 8664 Acked Write 5 Done Done Sent false
29611 8665 Acked Read 9 Done Done Sent false
29612 8666 Acked Write 1 Done Done Sent false
29613 8667 Acked Write 6 Done Done Sent false
29614 8668 Acked Flush 0 Done Done Sent false
29615 8669 Acked Read 2 Done Done Sent false
29616 8670 Acked Write 5 Done Done Sent false
29617 8671 Acked Read 2 Done Done Sent false
29618 8672 Acked Read 10 Done Done Sent false
29619 8673 Acked Read 7 Done Done Sent false
29620 8674 Acked Write 4 Done Done Sent false
29621 8675 Acked Read 7 Done Done Sent false
29622 8676 Acked Read 8 Done Done Sent false
29623 8677 Acked Write 3 Done Done Sent false
29624 8678 Acked Write 7 Done Done Sent false
29625 8679 Acked Read 3 Done Done Sent false
29626 8680 Acked Read 9 Done Done Sent false
29627 8681 Acked Write 2 Done Done Sent false
29628 8682 Acked Read 5 Done Done Sent false
29629 8683 Acked Read 4 Done Done Sent false
29630 8684 Acked Read 3 Done Done Sent false
29631 8685 Acked Read 9 Done Done Sent false
29632 8686 Acked Write 7 Done Done Sent false
29633 8687 Acked Write 2 Done Done Sent false
29634 8688 Acked Write 8 Done Done Sent false
29635 8689 Acked Read 7 Done Done Sent false
29636 8690 Acked Write 9 Done Done Sent false
29637 8691 Acked Read 4 Done Done Sent false
29638 8692 Acked Read 9 Done Done Sent false
29639 8693 Acked Write 10 Done Done Sent false
29640 8694 Acked Write 5 Done Done Sent false
29641 8695 Acked Write 5 Done Done Sent false
29642 8696 Acked Flush 0 Done Done Sent false
29643 8697 Acked Read 1 Done Done Sent false
29644 8698 Acked Read 1 Done Done Sent false
29645 8699 Acked Read 8 Done Done Sent false
29646 8700 Acked Flush 0 Done Done Sent false
29647 8701 Acked Write 10 Done Done Sent false
29648 8702 Acked Write 9 Done Done Sent false
29649 8703 Acked Write 4 Done Done Sent false
29650 8704 Acked Write 3 Done Done Sent false
29651 8705 Acked Read 6 Done Done Sent false
29652 8706 Acked Write 3 Done Done Sent false
29653 8707 Acked Write 10 Done Done Sent false
29654 8708 Acked Flush 0 Done Done Sent false
29655 8709 Acked Read 7 Done Done Sent false
29656 8710 Acked Flush 0 Done Done Sent false
29657 8711 Acked Write 3 Done Done Sent false
29658 8712 Acked Read 7 Done Done Sent false
29659 8713 Acked Write 4 Done Done Sent false
29660 8714 Acked Read 6 Done Done Sent false
29661 8715 Acked Write 8 Done Done Sent false
29662 8716 Acked Read 10 Done Done Sent false
29663 8717 Acked Write 7 Done Done Sent false
29664 8718 Acked Write 2 Done Done Sent false
29665 8719 Acked Read 8 Done Done Sent false
29666 8720 Acked Read 4 Done Done Sent false
29667 8721 Acked Read 9 Done Done Sent false
29668 8722 Acked Write 6 Done Done Sent false
29669 8723 Acked Read 6 Done Done Sent false
29670 8724 Acked Flush 0 Done Done Sent false
29671 8725 Acked Flush 0 Done Done Sent false
29672 8726 Acked Flush 0 Done Done Sent false
29673 8727 Acked Read 9 Done Done Sent false
29674 8728 Acked Read 5 Done Done Sent false
29675 8729 Acked Write 9 Done Done Sent false
29676 8730 Acked Write 2 Done Done Sent false
29677 8731 Acked Flush 0 Done Done Sent false
29678 8732 Acked Write 4 Done Done Sent false
29679 8733 Acked Read 7 Done Done Sent false
29680 8734 Acked Write 7 Done Done Sent false
29681 8735 Acked Read 4 Done Done Sent false
29682 8736 Acked Flush 0 Done Done Sent false
29683 8737 Acked Write 3 Done Done Sent false
29684 8738 Acked Flush 0 Done Done Sent false
29685 8739 Acked Read 4 Done Done Sent false
29686 8740 Acked Write 3 Done Done Sent false
29687 8741 Acked Read 7 Done Done Sent false
29688 8742 Acked Read 9 Done Done Sent false
29689 8743 Acked Write 1 Done Done Sent false
29690 8744 Acked Write 4 Done Done Sent false
29691 8745 Acked Write 9 Done Done Sent false
29692 8746 Acked Write 5 Done Done Sent false
29693 8747 Acked Read 7 Done Done Sent false
29694 8748 Acked Read 6 Done Done Sent false
29695 8749 Acked Write 1 Done Done Sent false
29696 8750 Acked Read 8 Done Done Sent false
29697 8751 Acked Read 4 Done Done Sent false
29698 8752 Acked Read 6 Done Done Sent false
29699 8753 Acked Write 2 Done Done Sent false
29700 8754 Acked Write 6 Done Done Sent false
29701 8755 Acked Read 10 Done Done Sent false
29702 8756 Acked Write 2 Done Done Sent false
29703 8757 Acked Read 5 Done Done Sent false
29704 8758 Acked Write 10 Done Done Sent false
29705 8759 Acked Read 2 Done Done Sent false
29706 8760 Acked Read 10 Done Done Sent false
29707 8761 Acked Read 10 Done Done Sent false
29708 8762 Acked Write 8 Done Done Sent false
29709 8763 Acked Read 10 Done Done Sent false
29710 8764 Acked Read 2 Done Done Sent false
29711 8765 Acked Write 8 Done Done Sent false
29712 8766 Acked Read 1 Done Done Sent false
29713 8767 Acked Write 5 Done Done Sent false
29714 8768 Acked Write 3 Done Done Sent false
29715 8769 Acked Read 7 Done Done Sent false
29716 8770 Acked Write 7 Done Done Sent false
29717 8771 Acked Read 7 Done Done Sent false
29718 8772 Acked Write 2 Done Done Sent false
29719 8773 Acked Write 7 Done Done Sent false
29720 8774 Acked Read 7 Done Done Sent false
29721 8775 Acked Write 3 Done Done Sent false
29722 8776 Acked Flush 0 Done Done Sent false
29723 8777 Acked Write 4 Done Done Sent false
29724 8778 Acked Flush 0 Done Done Sent false
29725 8779 Acked Write 2 Done Done Sent false
29726 8780 Acked Write 6 Done Done Sent false
29727 8781 Acked Write 8 Done Done Sent false
29728 8782 Acked Write 4 Done Done Sent false
29729 8783 Acked Read 3 Done Done Sent false
29730 8784 Acked Read 10 Done Done Sent false
29731 8785 Acked Write 8 Done Done Sent false
29732 8786 Acked Read 6 Done Done Sent false
29733 8787 Acked Read 10 Done Done Sent false
29734 8788 Acked Write 2 Done Done Sent false
29735 8789 Acked Flush 0 Done Done Sent false
29736 8790 Acked Read 9 Done Done Sent false
29737 8791 Acked Write 6 Done Done Sent false
29738 8792 Acked Write 5 Done Done Sent false
29739 8793 Acked Read 6 Done Done Sent false
29740 8794 Acked Write 8 Done Done Sent false
29741 8795 Acked Write 4 Done Done Sent false
29742 8796 Acked Read 8 Done Done Sent false
29743 8797 Acked Flush 0 Done Done Sent false
29744 8798 Acked Flush 0 Done Done Sent false
29745 8799 Acked Flush 0 Done Done Sent false
29746 8800 Acked Read 3 Done Done Sent false
29747 8801 Acked Flush 0 Done Done Sent false
29748 8802 Acked Write 6 Done Done Sent false
29749 8803 Acked Flush 0 Done Done Sent false
29750 8804 Acked Read 3 Done Done Sent false
29751 8805 Acked Write 6 Done Done Sent false
29752 8806 Acked Write 2 Done Done Sent false
29753 8807 Acked Read 7 Done Done Sent false
29754 8808 Acked Write 7 Done Done Sent false
29755 8809 Acked Write 6 Done Done Sent false
29756 8810 Acked Read 9 Done Done Sent false
29757 8811 Acked Write 2 Done Done Sent false
29758 8812 Acked Write 4 Done Done Sent false
29759 8813 Acked Read 6 Done Done Sent false
29760 8814 Acked Read 5 Done Done Sent false
29761 8815 Acked Read 4 Done Done Sent false
29762 8816 Acked Write 10 Done Done Sent false
29763 8817 Acked Read 10 Done Done Sent false
29764 8818 Acked Read 8 Done Done Sent false
29765 8819 Acked Write 3 Done Done Sent false
29766 8820 Acked Flush 0 Done Done Sent false
29767 8821 Acked Flush 0 Done Done Sent false
29768 8822 Acked Flush 0 Done Done Sent false
29769 8823 Acked Write 9 Done Done Sent false
29770 8824 Acked Read 2 Done Done Sent false
29771 8825 Acked Read 6 Done Done Sent false
29772 8826 Acked Flush 0 Done Done Sent false
29773 8827 Acked Write 1 Done Done Sent false
29774 8828 Acked Flush 0 Done Done Sent false
29775 8829 Acked Read 7 Done Done Sent false
29776 8830 Acked Read 7 Done Done Sent false
29777 8831 Acked Read 10 Done Done Sent false
29778 8832 Acked Write 9 Done Done Sent false
29779 8833 Acked Read 8 Done Done Sent false
29780 8834 Acked Read 3 Done Done Sent false
29781 8835 Acked Read 9 Done Done Sent false
29782 8836 Acked Write 1 Done Done Sent false
29783 8837 Acked Flush 0 Done Done Sent false
29784 8838 Acked Read 4 Done Done Sent false
29785 8839 Acked Write 7 Done Done Sent false
29786 8840 Acked Write 9 Done Done Sent false
29787 8841 Acked Read 1 Done Done Sent false
29788 8842 Acked Write 4 Done Done Sent false
29789 8843 Acked Read 2 Done Done Sent false
29790 8844 Acked Read 6 Done Done Sent false
29791 8845 Acked Write 1 Done Done Sent false
29792 8846 Acked Write 4 Done Done Sent false
29793 STATES DS:0 DS:1 DS:2 TOTAL
29794 Sent 0 0 301 301
29795 Done 301 301 0 602
29796 Skipped 0 0 0 0
29797 Error 0 0 0 0
29798 Last Flush: Some(JobId(8837)) Some(JobId(8837)) Some(JobId(8545))
29799 Downstairs last five completed: 8545 8544 8543 8542 8541
29800 Upstairs last five completed: 8846 8845 8844 8843 8842
29801 CLIENT: Up:0 ds:301 act:2
298022025-03-07T18:19:03.651ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
298032025-03-07T18:19:04.597ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
298042025-03-07T18:19:05.655ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
298052025-03-07T18:19:06.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
29806 [8830] got start action
29807 [8830] I am going to restart
29808 Starting downstairs at port 8830
29809 Make output file at "/tmp/dsc/downstairs-8830.txt"
29810 [8830][2] reports Starting
29811 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1234)
29812 [8830][2] reports Running
29813 ----------------------------------------------------------------
29814 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29815 States: Active Active Offline
29816 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29817 8546 Acked Write 9 Done Done Sent false
29818 8547 Acked Read 9 Done Done Sent false
29819 8548 Acked Write 2 Done Done Sent false
29820 8549 Acked Read 7 Done Done Sent false
29821 8550 Acked Read 8 Done Done Sent false
29822 8551 Acked Read 1 Done Done Sent false
29823 8552 Acked Read 8 Done Done Sent false
29824 8553 Acked Flush 0 Done Done Sent false
29825 8554 Acked Flush 0 Done Done Sent false
29826 8555 Acked Write 9 Done Done Sent false
29827 8556 Acked Write 10 Done Done Sent false
29828 8557 Acked Read 8 Done Done Sent false
29829 8558 Acked Read 9 Done Done Sent false
29830 8559 Acked Write 4 Done Done Sent false
29831 8560 Acked Write 5 Done Done Sent false
29832 8561 Acked Flush 0 Done Done Sent false
29833 8562 Acked Write 1 Done Done Sent false
29834 8563 Acked Flush 0 Done Done Sent false
29835 8564 Acked Write 2 Done Done Sent false
29836 8565 Acked Flush 0 Done Done Sent false
29837 8566 Acked Write 2 Done Done Sent false
29838 8567 Acked Write 6 Done Done Sent false
29839 8568 Acked Write 1 Done Done Sent false
29840 8569 Acked Write 3 Done Done Sent false
29841 8570 Acked Write 6 Done Done Sent false
29842 8571 Acked Write 6 Done Done Sent false
29843 8572 Acked Read 10 Done Done Sent false
29844 8573 Acked Write 9 Done Done Sent false
29845 8574 Acked Write 2 Done Done Sent false
29846 8575 Acked Read 10 Done Done Sent false
29847 8576 Acked Write 1 Done Done Sent false
29848 8577 Acked Read 2 Done Done Sent false
29849 8578 Acked Write 7 Done Done Sent false
29850 8579 Acked Read 4 Done Done Sent false
29851 8580 Acked Write 6 Done Done Sent false
29852 8581 Acked Read 10 Done Done Sent false
29853 8582 Acked Write 6 Done Done Sent false
29854 8583 Acked Flush 0 Done Done Sent false
29855 8584 Acked Flush 0 Done Done Sent false
29856 8585 Acked Write 10 Done Done Sent false
29857 8586 Acked Read 3 Done Done Sent false
29858 8587 Acked Read 2 Done Done Sent false
29859 8588 Acked Read 2 Done Done Sent false
29860 8589 Acked Read 3 Done Done Sent false
29861 8590 Acked Read 2 Done Done Sent false
29862 8591 Acked Write 5 Done Done Sent false
29863 8592 Acked Write 9 Done Done Sent false
29864 8593 Acked Write 5 Done Done Sent false
29865 8594 Acked Read 4 Done Done Sent false
29866 8595 Acked Write 7 Done Done Sent false
29867 8596 Acked Read 7 Done Done Sent false
29868 8597 Acked Read 9 Done Done Sent false
29869 8598 Acked Read 4 Done Done Sent false
29870 8599 Acked Read 1 Done Done Sent false
29871 8600 Acked Write 7 Done Done Sent false
29872 8601 Acked Read 1 Done Done Sent false
29873 8602 Acked Write 2 Done Done Sent false
29874 8603 Acked Write 8 Done Done Sent false
29875 8604 Acked Write 1 Done Done Sent false
29876 8605 Acked Read 1 Done Done Sent false
29877 8606 Acked Write 9 Done Done Sent false
29878 8607 Acked Flush 0 Done Done Sent false
29879 8608 Acked Read 3 Done Done Sent false
29880 8609 Acked Read 8 Done Done Sent false
29881 8610 Acked Read 1 Done Done Sent false
29882 8611 Acked Read 10 Done Done Sent false
29883 8612 Acked Flush 0 Done Done Sent false
29884 8613 Acked Write 2 Done Done Sent false
29885 8614 Acked Read 6 Done Done Sent false
29886 8615 Acked Write 8 Done Done Sent false
29887 8616 Acked Read 5 Done Done Sent false
29888 8617 Acked Flush 0 Done Done Sent false
29889 8618 Acked Read 4 Done Done Sent false
29890 8619 Acked Write 1 Done Done Sent false
29891 8620 Acked Flush 0 Done Done Sent false
29892 8621 Acked Write 7 Done Done Sent false
29893 8622 Acked Write 9 Done Done Sent false
29894 8623 Acked Flush 0 Done Done Sent false
29895 8624 Acked Write 3 Done Done Sent false
29896 8625 Acked Write 9 Done Done Sent false
29897 8626 Acked Read 2 Done Done Sent false
29898 8627 Acked Write 3 Done Done Sent false
29899 8628 Acked Read 10 Done Done Sent false
29900 8629 Acked Write 10 Done Done Sent false
29901 8630 Acked Read 6 Done Done Sent false
29902 8631 Acked Read 2 Done Done Sent false
29903 8632 Acked Read 8 Done Done Sent false
29904 8633 Acked Write 2 Done Done Sent false
29905 8634 Acked Write 6 Done Done Sent false
29906 8635 Acked Read 6 Done Done Sent false
29907 8636 Acked Flush 0 Done Done Sent false
29908 8637 Acked Write 8 Done Done Sent false
29909 8638 Acked Flush 0 Done Done Sent false
29910 8639 Acked Read 10 Done Done Sent false
29911 8640 Acked Write 1 Done Done Sent false
29912 8641 Acked Read 6 Done Done Sent false
29913 8642 Acked Read 4 Done Done Sent false
29914 8643 Acked Read 8 Done Done Sent false
29915 8644 Acked Write 9 Done Done Sent false
29916 8645 Acked Write 2 Done Done Sent false
29917 8646 Acked Flush 0 Done Done Sent false
29918 8647 Acked Write 1 Done Done Sent false
29919 8648 Acked Flush 0 Done Done Sent false
29920 8649 Acked Read 6 Done Done Sent false
29921 8650 Acked Write 5 Done Done Sent false
29922 8651 Acked Write 8 Done Done Sent false
29923 8652 Acked Write 6 Done Done Sent false
29924 8653 Acked Read 9 Done Done Sent false
29925 8654 Acked Read 1 Done Done Sent false
29926 8655 Acked Read 7 Done Done Sent false
29927 8656 Acked Read 5 Done Done Sent false
29928 8657 Acked Read 8 Done Done Sent false
29929 8658 Acked Read 2 Done Done Sent false
29930 8659 Acked Write 8 Done Done Sent false
29931 8660 Acked Read 5 Done Done Sent false
29932 8661 Acked Write 3 Done Done Sent false
29933 8662 Acked Read 5 Done Done Sent false
29934 8663 Acked Write 5 Done Done Sent false
29935 8664 Acked Write 5 Done Done Sent false
29936 8665 Acked Read 9 Done Done Sent false
29937 8666 Acked Write 1 Done Done Sent false
29938 8667 Acked Write 6 Done Done Sent false
29939 8668 Acked Flush 0 Done Done Sent false
29940 8669 Acked Read 2 Done Done Sent false
29941 8670 Acked Write 5 Done Done Sent false
29942 8671 Acked Read 2 Done Done Sent false
29943 8672 Acked Read 10 Done Done Sent false
29944 8673 Acked Read 7 Done Done Sent false
29945 8674 Acked Write 4 Done Done Sent false
29946 8675 Acked Read 7 Done Done Sent false
29947 8676 Acked Read 8 Done Done Sent false
29948 8677 Acked Write 3 Done Done Sent false
29949 8678 Acked Write 7 Done Done Sent false
29950 8679 Acked Read 3 Done Done Sent false
29951 8680 Acked Read 9 Done Done Sent false
29952 8681 Acked Write 2 Done Done Sent false
29953 8682 Acked Read 5 Done Done Sent false
29954 8683 Acked Read 4 Done Done Sent false
29955 8684 Acked Read 3 Done Done Sent false
29956 8685 Acked Read 9 Done Done Sent false
29957 8686 Acked Write 7 Done Done Sent false
29958 8687 Acked Write 2 Done Done Sent false
29959 8688 Acked Write 8 Done Done Sent false
29960 8689 Acked Read 7 Done Done Sent false
29961 8690 Acked Write 9 Done Done Sent false
29962 8691 Acked Read 4 Done Done Sent false
29963 8692 Acked Read 9 Done Done Sent false
29964 8693 Acked Write 10 Done Done Sent false
29965 8694 Acked Write 5 Done Done Sent false
29966 8695 Acked Write 5 Done Done Sent false
29967 8696 Acked Flush 0 Done Done Sent false
29968 8697 Acked Read 1 Done Done Sent false
29969 8698 Acked Read 1 Done Done Sent false
29970 8699 Acked Read 8 Done Done Sent false
29971 8700 Acked Flush 0 Done Done Sent false
29972 8701 Acked Write 10 Done Done Sent false
29973 8702 Acked Write 9 Done Done Sent false
29974 8703 Acked Write 4 Done Done Sent false
29975 8704 Acked Write 3 Done Done Sent false
29976 8705 Acked Read 6 Done Done Sent false
29977 8706 Acked Write 3 Done Done Sent false
29978 8707 Acked Write 10 Done Done Sent false
29979 8708 Acked Flush 0 Done Done Sent false
29980 8709 Acked Read 7 Done Done Sent false
29981 8710 Acked Flush 0 Done Done Sent false
29982 8711 Acked Write 3 Done Done Sent false
29983 8712 Acked Read 7 Done Done Sent false
29984 8713 Acked Write 4 Done Done Sent false
29985 8714 Acked Read 6 Done Done Sent false
29986 8715 Acked Write 8 Done Done Sent false
29987 8716 Acked Read 10 Done Done Sent false
29988 8717 Acked Write 7 Done Done Sent false
29989 8718 Acked Write 2 Done Done Sent false
29990 8719 Acked Read 8 Done Done Sent false
29991 8720 Acked Read 4 Done Done Sent false
29992 8721 Acked Read 9 Done Done Sent false
29993 8722 Acked Write 6 Done Done Sent false
29994 8723 Acked Read 6 Done Done Sent false
29995 8724 Acked Flush 0 Done Done Sent false
29996 8725 Acked Flush 0 Done Done Sent false
29997 8726 Acked Flush 0 Done Done Sent false
29998 8727 Acked Read 9 Done Done Sent false
29999 8728 Acked Read 5 Done Done Sent false
30000 8729 Acked Write 9 Done Done Sent false
30001 8730 Acked Write 2 Done Done Sent false
30002 8731 Acked Flush 0 Done Done Sent false
30003 8732 Acked Write 4 Done Done Sent false
30004 8733 Acked Read 7 Done Done Sent false
30005 8734 Acked Write 7 Done Done Sent false
30006 8735 Acked Read 4 Done Done Sent false
30007 8736 Acked Flush 0 Done Done Sent false
30008 8737 Acked Write 3 Done Done Sent false
30009 8738 Acked Flush 0 Done Done Sent false
30010 8739 Acked Read 4 Done Done Sent false
30011 8740 Acked Write 3 Done Done Sent false
30012 8741 Acked Read 7 Done Done Sent false
30013 8742 Acked Read 9 Done Done Sent false
30014 8743 Acked Write 1 Done Done Sent false
30015 8744 Acked Write 4 Done Done Sent false
30016 8745 Acked Write 9 Done Done Sent false
30017 8746 Acked Write 5 Done Done Sent false
30018 8747 Acked Read 7 Done Done Sent false
30019 8748 Acked Read 6 Done Done Sent false
30020 8749 Acked Write 1 Done Done Sent false
30021 8750 Acked Read 8 Done Done Sent false
30022 8751 Acked Read 4 Done Done Sent false
30023 8752 Acked Read 6 Done Done Sent false
30024 8753 Acked Write 2 Done Done Sent false
30025 8754 Acked Write 6 Done Done Sent false
30026 8755 Acked Read 10 Done Done Sent false
30027 8756 Acked Write 2 Done Done Sent false
30028 8757 Acked Read 5 Done Done Sent false
30029 8758 Acked Write 10 Done Done Sent false
30030 8759 Acked Read 2 Done Done Sent false
30031 8760 Acked Read 10 Done Done Sent false
30032 8761 Acked Read 10 Done Done Sent false
30033 8762 Acked Write 8 Done Done Sent false
30034 8763 Acked Read 10 Done Done Sent false
30035 8764 Acked Read 2 Done Done Sent false
30036 8765 Acked Write 8 Done Done Sent false
30037 8766 Acked Read 1 Done Done Sent false
30038 8767 Acked Write 5 Done Done Sent false
30039 8768 Acked Write 3 Done Done Sent false
30040 8769 Acked Read 7 Done Done Sent false
30041 8770 Acked Write 7 Done Done Sent false
30042 8771 Acked Read 7 Done Done Sent false
30043 8772 Acked Write 2 Done Done Sent false
30044 8773 Acked Write 7 Done Done Sent false
30045 8774 Acked Read 7 Done Done Sent false
30046 8775 Acked Write 3 Done Done Sent false
30047 8776 Acked Flush 0 Done Done Sent false
30048 8777 Acked Write 4 Done Done Sent false
30049 8778 Acked Flush 0 Done Done Sent false
30050 8779 Acked Write 2 Done Done Sent false
30051 8780 Acked Write 6 Done Done Sent false
30052 8781 Acked Write 8 Done Done Sent false
30053 8782 Acked Write 4 Done Done Sent false
30054 8783 Acked Read 3 Done Done Sent false
30055 8784 Acked Read 10 Done Done Sent false
30056 8785 Acked Write 8 Done Done Sent false
30057 8786 Acked Read 6 Done Done Sent false
30058 8787 Acked Read 10 Done Done Sent false
30059 8788 Acked Write 2 Done Done Sent false
30060 8789 Acked Flush 0 Done Done Sent false
30061 8790 Acked Read 9 Done Done Sent false
30062 8791 Acked Write 6 Done Done Sent false
30063 8792 Acked Write 5 Done Done Sent false
30064 8793 Acked Read 6 Done Done Sent false
30065 8794 Acked Write 8 Done Done Sent false
30066 8795 Acked Write 4 Done Done Sent false
30067 8796 Acked Read 8 Done Done Sent false
30068 8797 Acked Flush 0 Done Done Sent false
30069 8798 Acked Flush 0 Done Done Sent false
30070 8799 Acked Flush 0 Done Done Sent false
30071 8800 Acked Read 3 Done Done Sent false
30072 8801 Acked Flush 0 Done Done Sent false
30073 8802 Acked Write 6 Done Done Sent false
30074 8803 Acked Flush 0 Done Done Sent false
30075 8804 Acked Read 3 Done Done Sent false
30076 8805 Acked Write 6 Done Done Sent false
30077 8806 Acked Write 2 Done Done Sent false
30078 8807 Acked Read 7 Done Done Sent false
30079 8808 Acked Write 7 Done Done Sent false
30080 8809 Acked Write 6 Done Done Sent false
30081 8810 Acked Read 9 Done Done Sent false
30082 8811 Acked Write 2 Done Done Sent false
30083 8812 Acked Write 4 Done Done Sent false
30084 8813 Acked Read 6 Done Done Sent false
30085 8814 Acked Read 5 Done Done Sent false
30086 8815 Acked Read 4 Done Done Sent false
30087 8816 Acked Write 10 Done Done Sent false
30088 8817 Acked Read 10 Done Done Sent false
30089 8818 Acked Read 8 Done Done Sent false
30090 8819 Acked Write 3 Done Done Sent false
30091 8820 Acked Flush 0 Done Done Sent false
30092 8821 Acked Flush 0 Done Done Sent false
30093 8822 Acked Flush 0 Done Done Sent false
30094 8823 Acked Write 9 Done Done Sent false
30095 8824 Acked Read 2 Done Done Sent false
30096 8825 Acked Read 6 Done Done Sent false
30097 8826 Acked Flush 0 Done Done Sent false
30098 8827 Acked Write 1 Done Done Sent false
30099 8828 Acked Flush 0 Done Done Sent false
30100 8829 Acked Read 7 Done Done Sent false
30101 8830 Acked Read 7 Done Done Sent false
30102 8831 Acked Read 10 Done Done Sent false
30103 8832 Acked Write 9 Done Done Sent false
30104 8833 Acked Read 8 Done Done Sent false
30105 8834 Acked Read 3 Done Done Sent false
30106 8835 Acked Read 9 Done Done Sent false
30107 8836 Acked Write 1 Done Done Sent false
30108 8837 Acked Flush 0 Done Done Sent false
30109 8838 Acked Read 4 Done Done Sent false
30110 8839 Acked Write 7 Done Done Sent false
30111 8840 Acked Write 9 Done Done Sent false
30112 8841 Acked Read 1 Done Done Sent false
30113 8842 Acked Write 4 Done Done Sent false
30114 8843 Acked Read 2 Done Done Sent false
30115 8844 Acked Read 6 Done Done Sent false
30116 8845 Acked Write 1 Done Done Sent false
30117 8846 Acked Write 4 Done Done Sent false
30118 STATES DS:0 DS:1 DS:2 TOTAL
30119 Sent 0 0 301 301
30120 Done 301 301 0 602
30121 Skipped 0 0 0 0
30122 Error 0 0 0 0
30123 Last Flush: Some(JobId(8837)) Some(JobId(8837)) Some(JobId(8545))
30124 Downstairs last five completed: 8545 8544 8543 8542 8541
30125 Upstairs last five completed: 8846 8845 8844 8843 8842
30126 CLIENT: Up:0 ds:301 act:2
301272025-03-07T18:19:07.653ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301282025-03-07T18:19:08.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301292025-03-07T18:19:08.906ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301302025-03-07T18:19:08.907ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42754) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301312025-03-07T18:19:08.908ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301322025-03-07T18:19:08.909ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301332025-03-07T18:19:08.909ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301342025-03-07T18:19:08.909ZINFOcrucible: send last flush ID to this DS: Some(JobId(8545)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301352025-03-07T18:19:08.909ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301362025-03-07T18:19:08.909ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8545)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301372025-03-07T18:19:08.909ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8545)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
30138 ----------------------------------------------------------------
30139 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30140 Downstairs last five completed: 8847 8846 8845 8844 8843
30141 Upstairs last five completed: 8847 8846 8845 8844 8843
30142 CLIENT: Up:0 ds:0 act:3
30143 Replay: All jobs finished
30144 Main task has work to do, go find it
30145 got dsc Stop(2)
30146 stop 2
30147 Mar 07 18:19:11.154 INFO request completed[8830] Got stop action so:false kr:false
30148 , latency_us: 309, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8f11186c-d534-4a30-916c-24738a122404, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
301492025-03-07T18:19:11.154ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
301502025-03-07T18:19:11.154ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
30151 Mar 07 18:19:11.154 INFO request completed, latency_us:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:19:11.154974563Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
30152 136, response_code: 200, uri: /state/cid/2, method: GET, req_id: b013019e-b24d-46a9-acd8-49d4413c744e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:19:11.155034016Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"2","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
30153 :9998
30154 [8830] Exited with: ExitStatus(unix_wait_status(9))
30155 [8830][2] reports Exit
30156 Mar 07 18:19:15.157 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: 345d2d5c-87c5-4ad3-a980-7fbd0a0cdf04, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
30157 001/300 Read block 2455 len 40960
30158 002/300 Read block 3514 len 28672
30159 003/300 Write block 4594 len 16384 data: 4 4 4 6
30160 004/300 Read block 3274 len 28672
30161 005/300 Read block 1876 len 32768
30162 006/300 Flush
30163 007/300 Read block 2238 len 16384
30164 008/300 Flush
30165 009/300 Read block 3549 len 28672
30166 010/300 Flush
30167 011/300 Read block 1956 len 40960
30168 012/300 Read block 84 len 24576
30169 013/300 Read block 1929 len 36864
30170 014/300 Read block 3594 len 28672
30171 015/300 Read block 661 len 32768
30172 016/300 Read block 2103 len 8192
30173 017/300 Read block 2443 len 4096
30174 018/300 Flush
30175 019/300 Write block 3129 len 36864 data: 5 5 6 7 6 5 4 4 3
30176 020/300 Read block 132 len 40960
30177 021/300 Write block 3960 len 16384 data: 7 6 6 6
30178 022/300 Read block 4847 len 36864
30179 023/300 Write block 3501 len 32768 data: 6 4 4 3 5 5 4 5
30180 024/300 Write block 4347 len 28672 data: 6 5 5 5 5 6 6
30181 025/300 Write block 3885 len 4096 data: 7
30182 026/300 Write block 4791 len 24576 data: 7 7 7 8 8 7
30183 027/300 Write block 2711 len 36864 data: 5 4 5 5 3 5 6 7 7
30184 028/300 Read block 4724 len 40960
30185 029/300 Write block 1803 len 16384 data: 8 8 8 4
30186 030/300 Write block 4698 len 40960 data: 7 7 7 7 8 8 9 8 7 7
30187 031/300 Write block 1686 len 12288 data: 5 4 4
30188 032/300 Write block 668 len 36864 data: 6 6 6 6 8 6 4 4 7
30189 033/300 Read block 1310 len 40960
30190 034/300 Read block 4805 len 20480
30191 035/300 Write block 2028 len 36864 data: 7 7 5 3 5 5 4 4 5
30192 036/300 Write block 124 len 32768 data: 8 8 9 8 8 7 7 8
30193 037/300 Read block 290 len 12288
30194 038/300 Write block 4886 len 40960 data: 4 5 5 4 4 4 4 3 4 5
30195 039/300 Write block 471 len 4096 data: 3
30196 040/300 Write block 1077 len 24576 data: 5 6 6 7 6 6
30197 041/300 Read block 87 len 28672
30198 042/300 Write block 3313 len 24576 data: 4 3 4 3 3 5
30199 043/300 Write block 3902 len 16384 data: 5 5 5 4
30200 044/300 Flush
30201 045/300 Read block 468 len 20480
30202 046/300 Read block 4252 len 4096
30203 047/300 Write block 1859 len 12288 data: 3 5 5
30204 048/300 Read block 4405 len 4096
30205 049/300 Write block 2529 len 36864 data: 6 6 9 10 9 10 7 7 6
30206 050/300 Write block 379 len 36864 data: 5 4 3 5 7 6 5 6 5
30207 051/300 Write block 1080 len 40960 data: 8 7 7 6 5 6 5 4 4 4
30208 052/300 Read block 2655 len 32768
30209 053/300 Write block 2913 len 8192 data: 5 7
30210 054/300 Write block 4466 len 28672 data: 6 6 6 5 5 4 5
30211 055/300 Read block 2987 len 12288
30212 056/300 Read block 3646 len 20480
30213 057/300 Read block 4468 len 32768
30214 058/300 Read block 2941 len 8192
30215 059/300 Write block 3059 len 32768 data: 7 7 8 7 9 8 8 9
30216 060/300 Write block 2354 len 16384 data: 3 3 3 3
30217 061/300 Read block 2858 len 32768
30218 062/300 Write block 2153 len 4096 data: 3
30219 063/300 Read block 1336 len 4096
30220 064/300 Read block 4821 len 36864
30221 065/300 Read block 2008 len 36864
30222 066/300 Read block 4011 len 40960
30223 067/300 Read block 1152 len 40960
30224 068/300 Read block 2624 len 24576
30225 069/300 Read block 1843 len 24576
30226 070/300 Read block 1744 len 24576
30227 071/300 Read block 4652 len 36864
30228 072/300 Write block 496 len 28672 data: 3 3 2 2 3 4 4
30229 073/300 Write block 4692 len 36864 data: 7 6 8 9 9 8 8 8 8
30230 074/300 Read block 2258 len 32768
30231 075/300 Read block 83 len 40960
30232 076/300 Read block 1970 len 12288
30233 077/300 Read block 3331 len 20480
30234 078/300 Read block 655 len 12288
30235 079/300 Flush
30236 080/300 Flush
30237 081/300 Read block 938 len 4096
30238 082/300 Read block 37 len 40960
30239 083/300 Write block 782 len 20480 data: 4 4 5 5 7
30240 084/300 Read block 3430 len 4096
30241 085/300 Flush
30242 086/300 Flush
30243 087/300 Read block 1617 len 24576
30244 088/300 Flush
30245 089/300 Write block 199 len 16384 data: 7 6 8 8
30246 090/300 Write block 398 len 36864 data: 5 7 8 7 6 6 6 4 4
30247 091/300 Write block 157 len 16384 data: 5 4 6 7
30248 092/300 Read block 869 len 28672
30249 093/300 Read block 3496 len 8192
30250 094/300 Read block 628 len 16384
30251 095/300 Write block 1272 len 12288 data: 6 4 5
30252 096/300 Read block 1853 len 24576
30253 097/300 Read block 4017 len 24576
30254 098/300 Read block 4909 len 24576
30255 099/300 Read block 75 len 32768
30256 100/300 Write block 1401 len 24576 data: 4 4 5 5 7 5
30257 101/300 Write block 3975 len 32768 data: 5 4 4 2 2 4 5 5
30258 102/300 Write block 992 len 28672 data: 5 6 5 6 4 5 4
30259 103/300 Read block 3718 len 4096
30260 104/300 Read block 1324 len 40960
30261 105/300 Read block 2801 len 16384
30262 106/300 Read block 4360 len 24576
30263 107/300 Write block 1400 len 20480 data: 5 5 5 6 6
30264 108/300 Write block 106 len 12288 data: 2 2 3
30265 109/300 Write block 2533 len 28672 data: 10 11 8 8 7 6 6
30266 110/300 Write block 1217 len 4096 data: 5
30267 111/300 Flush
30268 112/300 Read block 2654 len 4096
30269 113/300 Write block 4328 len 28672 data: 8 8 10 11 9 9 8
30270 114/300 Write block 2021 len 16384 data: 4 6 6 7
30271 115/300 Read block 3651 len 16384
30272 116/300 Read block 351 len 40960
30273 117/300 Write block 3040 len 20480 data: 3 5 5 4 5
30274 118/300 Flush
30275 119/300 Write block 2029 len 28672 data: 8 6 4 6 6 5 5
30276 120/300 Read block 2586 len 12288
30277 121/300 Read block 3363 len 8192
30278 122/300 Read block 1371 len 20480
30279 123/300 Read block 4038 len 8192
30280 124/300 Write block 694 len 16384 data: 5 4 6 6
30281 125/300 Read block 1308 len 28672
30282 126/300 Write block 1567 len 8192 data: 11 12
30283 127/300 Write block 3339 len 28672 data: 5 5 5 5 5 5 3
30284 128/300 Read block 1586 len 12288
30285 129/300 Write block 3040 len 32768 data: 4 6 6 5 6 4 5 4
30286 130/300 Read block 3551 len 40960
30287 131/300 Read block 4048 len 24576
30288 132/300 Write block 678 len 28672 data: 10 9 5 5 4 5 5
30289 133/300 Write block 4978 len 20480 data: 5 7 8 6 6
30290 134/300 Read block 210 len 20480
30291 135/300 Write block 825 len 8192 data: 10 7
30292 136/300 Read block 3638 len 28672
30293 137/300 Read block 1841 len 4096
30294 138/300 Write block 4687 len 12288 data: 4 6 5
30295 139/300 Write block 3200 len 32768 data: 6 7 7 7 6 6 6 5
30296 140/300 Read block 1865 len 40960
30297 141/300 Write block 477 len 4096 data: 6
30298 142/300 Read block 808 len 24576
30299 143/300 Read block 3531 len 12288
30300 144/300 Write block 3529 len 24576 data: 6 5 5 5 5 3
30301 145/300 Write block 4917 len 36864 data: 8 9 8 8 6 7 8 8 6
30302 146/300 Read block 2825 len 12288
30303 147/300 Read block 3394 len 24576
30304 148/300 Write block 3785 len 8192 data: 7 6
30305 149/300 Write block 2874 len 4096 data: 2
30306 150/300 Read block 3380 len 8192
30307 151/300 Write block 2016 len 32768 data: 8 7 8 7 4 5 7 7
30308 152/300 Write block 1712 len 20480 data: 7 7 9 10 10
30309 153/300 Write block 660 len 20480 data: 3 4 3 7 7
30310 154/300 Read block 1476 len 32768
30311 155/300 Write block 879 len 32768 data: 5 5 8 6 6 5 4 4
30312 156/300 Write block 2708 len 36864 data: 4 4 4 6 5 6 6 4 6
30313 157/300 Write block 912 len 32768 data: 4 4 4 5 4 4 4 3
30314 158/300 Read block 71 len 12288
30315 159/300 Read block 3142 len 36864
30316 160/300 Read block 113 len 20480
30317 161/300 Write block 3060 len 16384 data: 8 9 8 10
30318 162/300 Write block 2141 len 20480 data: 4 5 5 5 4
30319 163/300 Write block 275 len 4096 data: 8
30320 164/300 Read block 4310 len 8192
30321 165/300 Read block 2391 len 12288
30322 166/300 Read block 159 len 8192
30323 167/300 Read block 3377 len 40960
30324 168/300 Write block 354 len 12288 data: 7 6 7
30325 169/300 Write block 2969 len 8192 data: 6 6
30326 170/300 Write block 3302 len 32768 data: 6 7 7 7 7 6 5 5
30327 171/300 Read block 2861 len 16384
30328 172/300 Read block 4779 len 36864
30329 173/300 Write block 3612 len 8192 data: 5 5
30330 174/300 Flush
30331 175/300 Write block 3066 len 4096 data: 10
30332 176/300 Write block 4632 len 36864 data: 4 3 5 7 6 6 5 5 5
30333 177/300 Read block 689 len 20480
30334 178/300 Write block 1560 len 4096 data: 3
30335 179/300 Read block 1211 len 36864
30336 180/300 Read block 2344 len 28672
30337 181/300 Flush
30338 182/300 Read block 982 len 20480
30339 183/300 Read block 1986 len 4096
30340 184/300 Write block 1715 len 4096 data: 11
30341 185/300 Read block 1518 len 24576
30342 186/300 Read block 4625 len 24576
30343 187/300 Write block 2087 len 24576 data: 3 3 3 4 4 3
30344 188/300 Write block 1040 len 4096 data: 2
30345 189/300 Flush
30346 190/300 Write block 2722 len 16384 data: 10 9 10 11
30347 191/300 Read block 2229 len 20480
30348 192/300 Read block 3576 len 20480
30349 193/300 Read block 562 len 16384
30350 194/300 Read block 2741 len 40960
30351 195/300 Write block 1371 len 16384 data: 3 3 2 2
30352 196/300 Read block 4645 len 40960
30353 197/300 Read block 4328 len 24576
30354 198/300 Flush
30355 199/300 Flush
30356 200/300 Write block 1890 len 20480 data: 10 9 9 7 6
30357 201/300 Write block 2328 len 36864 data: 5 6 6 7 7 8 7 9 9
30358 202/300 Read block 626 len 12288
30359 203/300 Write block 767 len 24576 data: 5 5 5 5 5 7
30360 204/300 Write block 1588 len 40960 data: 4 4 5 4 5 4 5 5 6 7
30361 205/300 Write block 2212 len 36864 data: 3 3 4 3 3 4 4 4 5
30362 206/300 Read block 4269 len 36864
30363 207/300 Write block 4858 len 12288 data: 6 5 3
30364 208/300 Read block 3953 len 32768
30365 209/300 Write block 1631 len 40960 data: 4 2 3 5 7 7 7 5 6 6
30366 210/300 Flush
30367 211/300 Write block 3116 len 36864 data: 6 6 5 4 4 3 3 4 6
30368 212/300 Flush
30369 213/300 Flush
30370 214/300 Read block 494 len 20480
30371 215/300 Read block 4568 len 32768
30372 216/300 Flush
30373 217/300 Write block 464 len 20480 data: 5 4 2 2 2
30374 218/300 Flush
30375 219/300 Flush
30376 220/300 Read block 1538 len 28672
30377 221/300 Write block 1599 len 4096 data: 5
30378 222/300 Read block 4667 len 28672
30379 223/300 Flush
30380 224/300 Flush
30381 225/300 Write block 512 len 24576 data: 4 4 5 5 4 3
30382 226/300 Write block 3956 len 40960 data: 5 5 5 7 8 7 7 7 6 3
30383 227/300 Write block 4334 len 16384 data: 9 6 5 5
30384 228/300 Write block 3354 len 16384 data: 5 5 4 5
30385 229/300 Read block 4177 len 28672
30386 230/300 Read block 1499 len 36864
30387 231/300 Flush
30388 232/300 Read block 2041 len 40960
30389 233/300 Read block 3877 len 40960
30390 234/300 Read block 3410 len 20480
30391 235/300 Write block 3306 len 24576 data: 8 7 6 6 4 4
30392 236/300 Write block 3548 len 12288 data: 4 3 4
30393 237/300 Read block 824 len 16384
30394 238/300 Write block 3521 len 32768 data: 5 5 6 6 5 5 5 5
30395 239/300 Write block 3696 len 28672 data: 6 3 3 6 5 4 7
30396 240/300 Write block 914 len 32768 data: 5 6 5 5 5 4 3 3
30397 241/300 Read block 477 len 20480
30398 242/300 Write block 4735 len 8192 data: 4 4
30399 243/300 Read block 4893 len 16384
30400 244/300 Read block 1681 len 8192
30401 245/300 Read block 3881 len 12288
30402 246/300 Write block 2353 len 20480 data: 3 4 4 4 4
30403 247/300 Write block 4095 len 36864 data: 3 3 3 3 3 3 4 4 5
30404 248/300 Write block 2157 len 36864 data: 4 3 3 3 4 4 5 5 5
30405 249/300 Read block 914 len 40960
30406 250/300 Read block 893 len 32768
30407 251/300 Write block 378 len 4096 data: 6
30408 252/300 Write block 1506 len 4096 data: 2
30409 253/300 Read block 297 len 20480
30410 254/300 Read block 2913 len 8192
30411 255/300 Write block 277 len 40960 data: 7 6 7 7 8 6 6 7 6 7
30412 256/300 Write block 1487 len 12288 data: 6 6 5
30413 257/300 Read block 909 len 36864
30414 258/300 Read block 3718 len 12288
30415 259/300 Read block 3560 len 40960
30416 260/300 Read block 1315 len 32768
30417 261/300 Write block 2428 len 28672 data: 7 7 6 6 7 6 5
30418 262/300 Read block 4556 len 8192
30419 263/300 Write block 629 len 24576 data: 8 7 5 6 5 7
30420 264/300 Read block 4497 len 8192
30421 265/300 Read block 1724 len 24576
30422 266/300 Read block 1364 len 24576
30423 267/300 Read block 2118 len 32768
30424 268/300 Write block 4409 len 4096 data: 4
30425 269/300 Flush
30426 270/300 Read block 284 len 32768
30427 271/300 Flush
30428 272/300 Write block 243 len 16384 data: 6 8 7 6
30429 273/300 Read block 1103 len 8192
30430 274/300 Write block 4831 len 32768 data: 5 6 6 6 6 6 7 7
30431 275/300 Write block 1225 len 16384 data: 3 3 4 2
30432 276/300 Write block 3097 len 28672 data: 7 7 7 8 8 8 6
30433 277/300 Read block 1940 len 24576
30434 278/300 Read block 1114 len 32768
30435 279/300 Flush
30436 280/300 Read block 1170 len 12288
30437 281/300 Write block 2698 len 12288 data: 3 5 6
30438 282/300 Write block 3048 len 36864 data: 4 4 4 3 5 6 6 7 5
30439 283/300 Read block 2280 len 4096
30440 284/300 Read block 4058 len 8192
30441 285/300 Flush
30442 286/300 Read block 2931 len 12288
30443 287/300 Read block 2344 len 12288
30444 288/300 Write block 4199 len 20480 data: 6 6 6 8 9
30445 289/300 Write block 2062 len 8192 data: 5 7
30446 290/300 Read block 4067 len 20480
30447 291/300 Write block 1158 len 32768 data: 6 5 6 6 6 6 8 5
30448 292/300 Read block 2210 len 8192
30449 293/300 Write block 3445 len 12288 data: 9 6 7
30450 294/300 Flush
30451 295/300 Read block 3182 len 12288
30452 296/300 Write block 339 len 36864 data: 6 5 5 4 4 3 5 5 7
30453 297/300 Read block 1189 len 24576
30454 298/300 Write block 4608 len 32768 data: 7 8 9 5 6 4 3 2
30455 299/300 Read block 3041 len 20480
30456 300/300 Read block 2746 len 36864
30457 301/300 Write block 2802 len 40960 data: 9 7 6 6 6 7 7 8 8 8
30458 Main task has work to do, go find it
30459 got dsc Start(2)
30460 start 2
30461 Mar 07 18:19:15.402 INFO request completed, latency_us: 194, response_code: 204, uri: /start/cid/2, method: POST, req_id: 25cbfe3a-7bc4-45e0-8308-64e39b17bacc, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
30462 Replay: started 2, returned:Ok(())
30463 ----------------------------------------------------------------
30464 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30465 States: Active Active Offline
30466 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30467 8848 Acked Read 10 Done Done Sent false
30468 8849 Acked Read 7 Done Done Sent false
30469 8850 Acked Write 4 Done Done Sent false
30470 8851 Acked Read 7 Done Done Sent false
30471 8852 Acked Read 8 Done Done Sent false
30472 8853 Acked Flush 0 Done Done Sent false
30473 8854 Acked Read 4 Done Done Sent false
30474 8855 Acked Flush 0 Done Done Sent false
30475 8856 Acked Read 7 Done Done Sent false
30476 8857 Acked Flush 0 Done Done Sent false
30477 8858 Acked Read 10 Done Done Sent false
30478 8859 Acked Read 6 Done Done Sent false
30479 8860 Acked Read 9 Done Done Sent false
30480 8861 Acked Read 7 Done Done Sent false
30481 8862 Acked Read 8 Done Done Sent false
30482 8863 Acked Read 2 Done Done Sent false
30483 8864 Acked Read 1 Done Done Sent false
30484 8865 Acked Flush 0 Done Done Sent false
30485 8866 Acked Write 9 Done Done Sent false
30486 8867 Acked Read 10 Done Done Sent false
30487 8868 Acked Write 4 Done Done Sent false
30488 8869 Acked Read 9 Done Done Sent false
30489 8870 Acked Write 8 Done Done Sent false
30490 8871 Acked Write 7 Done Done Sent false
30491 8872 Acked Write 1 Done Done Sent false
30492 8873 Acked Write 6 Done Done Sent false
30493 8874 Acked Write 9 Done Done Sent false
30494 8875 Acked Read 10 Done Done Sent false
30495 8876 Acked Write 4 Done Done Sent false
30496 8877 Acked Write 10 Done Done Sent false
30497 8878 Acked Write 3 Done Done Sent false
30498 8879 Acked Write 9 Done Done Sent false
30499 8880 Acked Read 10 Done Done Sent false
30500 8881 Acked Read 5 Done Done Sent false
30501 8882 Acked Write 9 Done Done Sent false
30502 8883 Acked Write 8 Done Done Sent false
30503 8884 Acked Read 3 Done Done Sent false
30504 8885 Acked Write 10 Done Done Sent false
30505 8886 Acked Write 1 Done Done Sent false
30506 8887 Acked Write 6 Done Done Sent false
30507 8888 Acked Read 7 Done Done Sent false
30508 8889 Acked Write 6 Done Done Sent false
30509 8890 Acked Write 4 Done Done Sent false
30510 8891 Acked Flush 0 Done Done Sent false
30511 8892 Acked Read 5 Done Done Sent false
30512 8893 Acked Read 1 Done Done Sent false
30513 8894 Acked Write 3 Done Done Sent false
30514 8895 Acked Read 1 Done Done Sent false
30515 8896 Acked Write 9 Done Done Sent false
30516 8897 Acked Write 9 Done Done Sent false
30517 8898 Acked Write 10 Done Done Sent false
30518 8899 Acked Read 8 Done Done Sent false
30519 8900 Acked Write 2 Done Done Sent false
30520 8901 Acked Write 7 Done Done Sent false
30521 8902 Acked Read 3 Done Done Sent false
30522 8903 Acked Read 5 Done Done Sent false
30523 8904 Acked Read 8 Done Done Sent false
30524 8905 Acked Read 2 Done Done Sent false
30525 8906 Acked Write 8 Done Done Sent false
30526 8907 Acked Write 4 Done Done Sent false
30527 8908 Acked Read 8 Done Done Sent false
30528 8909 Acked Write 1 Done Done Sent false
30529 8910 Acked Read 1 Done Done Sent false
30530 8911 Acked Read 9 Done Done Sent false
30531 8912 Acked Read 9 Done Done Sent false
30532 8913 Acked Read 10 Done Done Sent false
30533 8914 Acked Read 10 Done Done Sent false
30534 8915 Acked Read 6 Done Done Sent false
30535 8916 Acked Read 6 Done Done Sent false
30536 8917 Acked Read 6 Done Done Sent false
30537 8918 Acked Read 9 Done Done Sent false
30538 8919 Acked Write 7 Done Done Sent false
30539 8920 Acked Write 9 Done Done Sent false
30540 8921 Acked Read 8 Done Done Sent false
30541 8922 Acked Read 10 Done Done Sent false
30542 8923 Acked Read 3 Done Done Sent false
30543 8924 Acked Read 5 Done Done Sent false
30544 8925 Acked Read 3 Done Done Sent false
30545 8926 Acked Flush 0 Done Done Sent false
30546 8927 Acked Flush 0 Done Done Sent false
30547 8928 Acked Read 1 Done Done Sent false
30548 8929 Acked Read 10 Done Done Sent false
30549 8930 Acked Write 5 Done Done Sent false
30550 8931 Acked Read 1 Done Done Sent false
30551 8932 Acked Flush 0 Done Done Sent false
30552 8933 Acked Flush 0 Done Done Sent false
30553 8934 Acked Read 6 Done Done Sent false
30554 8935 Acked Flush 0 Done Done Sent false
30555 8936 Acked Write 4 Done Done Sent false
30556 8937 Acked Write 9 Done Done Sent false
30557 8938 Acked Write 4 Done Done Sent false
30558 8939 Acked Read 7 Done Done Sent false
30559 8940 Acked Read 2 Done Done Sent false
30560 8941 Acked Read 4 Done Done Sent false
30561 8942 Acked Write 3 Done Done Sent false
30562 8943 Acked Read 6 Done Done Sent false
30563 8944 Acked Read 6 Done Done Sent false
30564 8945 Acked Read 6 Done Done Sent false
30565 8946 Acked Read 8 Done Done Sent false
30566 8947 Acked Write 6 Done Done Sent false
30567 8948 Acked Write 8 Done Done Sent false
30568 8949 Acked Write 7 Done Done Sent false
30569 8950 Acked Read 1 Done Done Sent false
30570 8951 Acked Read 10 Done Done Sent false
30571 8952 Acked Read 4 Done Done Sent false
30572 8953 Acked Read 6 Done Done Sent false
30573 8954 Acked Write 5 Done Done Sent false
30574 8955 Acked Write 3 Done Done Sent false
30575 8956 Acked Write 7 Done Done Sent false
30576 8957 Acked Write 1 Done Done Sent false
30577 8958 Acked Flush 0 Done Done Sent false
30578 8959 Acked Read 1 Done Done Sent false
30579 8960 Acked Write 7 Done Done Sent false
30580 8961 Acked Write 4 Done Done Sent false
30581 8962 Acked Read 4 Done Done Sent false
30582 8963 Acked Read 10 Done Done Sent false
30583 8964 Acked Write 5 Done Done Sent false
30584 8965 Acked Flush 0 Done Done Sent false
30585 8966 Acked Write 7 Done Done Sent false
30586 8967 Acked Read 3 Done Done Sent false
30587 8968 Acked Read 2 Done Done Sent false
30588 8969 Acked Read 5 Done Done Sent false
30589 8970 Acked Read 2 Done Done Sent false
30590 8971 Acked Write 4 Done Done Sent false
30591 8972 Acked Read 7 Done Done Sent false
30592 8973 Acked Write 2 Done Done Sent false
30593 8974 Acked Write 7 Done Done Sent false
30594 8975 Acked Read 3 Done Done Sent false
30595 8976 Acked Write 8 Done Done Sent false
30596 8977 Acked Read 10 Done Done Sent false
30597 8978 Acked Read 6 Done Done Sent false
30598 8979 Acked Write 7 Done Done Sent false
30599 8980 Acked Write 5 Done Done Sent false
30600 8981 Acked Read 5 Done Done Sent false
30601 8982 Acked Write 2 Done Done Sent false
30602 8983 Acked Read 7 Done Done Sent false
30603 8984 Acked Read 1 Done Done Sent false
30604 8985 Acked Write 3 Done Done Sent false
30605 8986 Acked Write 8 Done Done Sent false
30606 8987 Acked Read 10 Done Done Sent false
30607 8988 Acked Write 1 Done Done Sent false
30608 8989 Acked Read 6 Done Done Sent false
30609 8990 Acked Read 3 Done Done Sent false
30610 8991 Acked Write 6 Done Done Sent false
30611 8992 Acked Write 9 Done Done Sent false
30612 8993 Acked Read 3 Done Done Sent false
30613 8994 Acked Read 6 Done Done Sent false
30614 8995 Acked Write 2 Done Done Sent false
30615 8996 Acked Write 1 Done Done Sent false
30616 8997 Acked Read 2 Done Done Sent false
30617 8998 Acked Write 8 Done Done Sent false
30618 8999 Acked Write 5 Done Done Sent false
30619 9000 Acked Write 5 Done Done Sent false
30620 9001 Acked Read 8 Done Done Sent false
30621 9002 Acked Write 8 Done Done Sent false
30622 9003 Acked Write 9 Done Done Sent false
30623 9004 Acked Write 8 Done Done Sent false
30624 9005 Acked Read 3 Done Done Sent false
30625 9006 Acked Read 9 Done Done Sent false
30626 9007 Acked Read 5 Done Done Sent false
30627 9008 Acked Write 4 Done Done Sent false
30628 9009 Acked Write 5 Done Done Sent false
30629 9010 Acked Write 1 Done Done Sent false
30630 9011 Acked Read 2 Done Done Sent false
30631 9012 Acked Read 3 Done Done Sent false
30632 9013 Acked Read 2 Done Done Sent false
30633 9014 Acked Read 10 Done Done Sent false
30634 9015 Acked Write 3 Done Done Sent false
30635 9016 Acked Write 2 Done Done Sent false
30636 9017 Acked Write 8 Done Done Sent false
30637 9018 Acked Read 4 Done Done Sent false
30638 9019 Acked Read 9 Done Done Sent false
30639 9020 Acked Write 2 Done Done Sent false
30640 9021 Acked Flush 0 Done Done Sent false
30641 9022 Acked Write 1 Done Done Sent false
30642 9023 Acked Write 9 Done Done Sent false
30643 9024 Acked Read 5 Done Done Sent false
30644 9025 Acked Write 1 Done Done Sent false
30645 9026 Acked Read 9 Done Done Sent false
30646 9027 Acked Read 7 Done Done Sent false
30647 9028 Acked Flush 0 Done Done Sent false
30648 9029 Acked Read 5 Done Done Sent false
30649 9030 Acked Read 1 Done Done Sent false
30650 9031 Acked Write 1 Done Done Sent false
30651 9032 Acked Read 6 Done Done Sent false
30652 9033 Acked Read 6 Done Done Sent false
30653 9034 Acked Write 6 Done Done Sent false
30654 9035 Acked Write 1 Done Done Sent false
30655 9036 Acked Flush 0 Done Done Sent false
30656 9037 Acked Write 4 Done Done Sent false
30657 9038 Acked Read 5 Done Done Sent false
30658 9039 Acked Read 5 Done Done Sent false
30659 9040 Acked Read 4 Done Done Sent false
30660 9041 Acked Read 10 Done Done Sent false
30661 9042 Acked Write 4 Done Done Sent false
30662 9043 Acked Read 10 Done Done Sent false
30663 9044 Acked Read 6 Done Done Sent false
30664 9045 Acked Flush 0 Done Done Sent false
30665 9046 Acked Flush 0 Done Done Sent false
30666 9047 Acked Write 5 Done Done Sent false
30667 9048 Acked Write 9 Done Done Sent false
30668 9049 Acked Read 3 Done Done Sent false
30669 9050 Acked Write 6 Done Done Sent false
30670 9051 Acked Write 10 Done Done Sent false
30671 9052 Acked Write 9 Done Done Sent false
30672 9053 Acked Read 9 Done Done Sent false
30673 9054 Acked Write 3 Done Done Sent false
30674 9055 Acked Read 8 Done Done Sent false
30675 9056 Acked Write 10 Done Done Sent false
30676 9057 Acked Flush 0 Done Done Sent false
30677 9058 Acked Write 9 Done Done Sent false
30678 9059 Acked Flush 0 Done Done Sent false
30679 9060 Acked Flush 0 Done Done Sent false
30680 9061 Acked Read 5 Done Done Sent false
30681 9062 Acked Read 8 Done Done Sent false
30682 9063 Acked Flush 0 Done Done Sent false
30683 9064 Acked Write 5 Done Done Sent false
30684 9065 Acked Flush 0 Done Done Sent false
30685 9066 Acked Flush 0 Done Done Sent false
30686 9067 Acked Read 7 Done Done Sent false
30687 9068 Acked Write 1 Done Done Sent false
30688 9069 Acked Read 7 Done Done Sent false
30689 9070 Acked Flush 0 Done Done Sent false
30690 9071 Acked Flush 0 Done Done Sent false
30691 9072 Acked Write 6 Done Done Sent false
30692 9073 Acked Write 10 Done Done Sent false
30693 9074 Acked Write 4 Done Done Sent false
30694 9075 Acked Write 4 Done Done Sent false
30695 9076 Acked Read 7 Done Done Sent false
30696 9077 Acked Read 9 Done Done Sent false
30697 9078 Acked Flush 0 Done Done Sent false
30698 9079 Acked Read 10 Done Done Sent false
30699 9080 Acked Read 10 Done Done Sent false
30700 9081 Acked Read 5 Done Done Sent false
30701 9082 Acked Write 6 Done Done Sent false
30702 9083 Acked Write 3 Done Done Sent false
30703 9084 Acked Read 4 Done Done Sent false
30704 9085 Acked Write 8 Done Done Sent false
30705 9086 Acked Write 7 Done Done Sent false
30706 9087 Acked Write 8 Done Done Sent false
30707 9088 Acked Read 5 Done Done Sent false
30708 9089 Acked Write 2 Done Done Sent false
30709 9090 Acked Read 4 Done Done Sent false
30710 9091 Acked Read 2 Done Done Sent false
30711 9092 Acked Read 3 Done Done Sent false
30712 9093 Acked Write 5 Done Done Sent false
30713 9094 Acked Write 9 Done Done Sent false
30714 9095 Acked Write 9 Done Done Sent false
30715 9096 Acked Read 10 Done Done Sent false
30716 9097 Acked Read 8 Done Done Sent false
30717 9098 Acked Write 1 Done Done Sent false
30718 9099 Acked Write 1 Done Done Sent false
30719 9100 Acked Read 5 Done Done Sent false
30720 9101 Acked Read 2 Done Done Sent false
30721 9102 Acked Write 10 Done Done Sent false
30722 9103 Acked Write 3 Done Done Sent false
30723 9104 Acked Read 9 Done Done Sent false
30724 9105 Acked Read 3 Done Done Sent false
30725 9106 Acked Read 10 Done Done Sent false
30726 9107 Acked Read 8 Done Done Sent false
30727 9108 Acked Write 7 Done Done Sent false
30728 9109 Acked Read 2 Done Done Sent false
30729 9110 Acked Write 6 Done Done Sent false
30730 9111 Acked Read 2 Done Done Sent false
30731 9112 Acked Read 6 Done Done Sent false
30732 9113 Acked Read 6 Done Done Sent false
30733 9114 Acked Read 8 Done Done Sent false
30734 9115 Acked Write 1 Done Done Sent false
30735 9116 Acked Flush 0 Done Done Sent false
30736 9117 Acked Read 8 Done Done Sent false
30737 9118 Acked Flush 0 Done Done Sent false
30738 9119 Acked Write 4 Done Done Sent false
30739 9120 Acked Read 2 Done Done Sent false
30740 9121 Acked Write 8 Done Done Sent false
30741 9122 Acked Write 4 Done Done Sent false
30742 9123 Acked Write 7 Done Done Sent false
30743 9124 Acked Read 6 Done Done Sent false
30744 9125 Acked Read 8 Done Done Sent false
30745 9126 Acked Flush 0 Done Done Sent false
30746 9127 Acked Read 3 Done Done Sent false
30747 9128 Acked Write 3 Done Done Sent false
30748 9129 Acked Write 9 Done Done Sent false
30749 9130 Acked Read 1 Done Done Sent false
30750 9131 Acked Read 2 Done Done Sent false
30751 9132 Acked Flush 0 Done Done Sent false
30752 9133 Acked Read 3 Done Done Sent false
30753 9134 Acked Read 3 Done Done Sent false
30754 9135 Acked Write 5 Done Done Sent false
30755 9136 Acked Write 2 Done Done Sent false
30756 9137 Acked Read 5 Done Done Sent false
30757 9138 Acked Write 8 Done Done Sent false
30758 9139 Acked Read 2 Done Done Sent false
30759 9140 Acked Write 3 Done Done Sent false
30760 9141 Acked Flush 0 Done Done Sent false
30761 9142 Acked Read 3 Done Done Sent false
30762 9143 Acked Write 9 Done Done Sent false
30763 9144 Acked Read 6 Done Done Sent false
30764 9145 Acked Write 8 Done Done Sent false
30765 9146 Acked Read 5 Done Done Sent false
30766 9147 Acked Read 9 Done Done Sent false
30767 9148 Acked Write 10 Done Done Sent false
30768 STATES DS:0 DS:1 DS:2 TOTAL
30769 Sent 0 0 301 301
30770 Done 301 301 0 602
30771 Skipped 0 0 0 0
30772 Error 0 0 0 0
30773 Last Flush: Some(JobId(9141)) Some(JobId(9141)) Some(JobId(8847))
30774 Downstairs last five completed: 8847 8846 8845 8844 8843
30775 Upstairs last five completed: 9148 9147 9146 9145 9144
30776 CLIENT: Up:0 ds:301 act:2
307772025-03-07T18:19:15.905ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
307782025-03-07T18:19:16.616ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
30779 [8830] got start action
30780 [8830] I am going to restart
30781 Starting downstairs at port 8830
30782 Make output file at "/tmp/dsc/downstairs-8830.txt"
30783 [8830][2] reports Starting
30784 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1235)
30785 [8830][2] reports Running
307862025-03-07T18:19:17.618ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
307872025-03-07T18:19:18.619ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
30788 ----------------------------------------------------------------
30789 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30790 States: Active Active Offline
30791 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30792 8848 Acked Read 10 Done Done Sent false
30793 8849 Acked Read 7 Done Done Sent false
30794 8850 Acked Write 4 Done Done Sent false
30795 8851 Acked Read 7 Done Done Sent false
30796 8852 Acked Read 8 Done Done Sent false
30797 8853 Acked Flush 0 Done Done Sent false
30798 8854 Acked Read 4 Done Done Sent false
30799 8855 Acked Flush 0 Done Done Sent false
30800 8856 Acked Read 7 Done Done Sent false
30801 8857 Acked Flush 0 Done Done Sent false
30802 8858 Acked Read 10 Done Done Sent false
30803 8859 Acked Read 6 Done Done Sent false
30804 8860 Acked Read 9 Done Done Sent false
30805 8861 Acked Read 7 Done Done Sent false
30806 8862 Acked Read 8 Done Done Sent false
30807 8863 Acked Read 2 Done Done Sent false
30808 8864 Acked Read 1 Done Done Sent false
30809 8865 Acked Flush 0 Done Done Sent false
30810 8866 Acked Write 9 Done Done Sent false
30811 8867 Acked Read 10 Done Done Sent false
30812 8868 Acked Write 4 Done Done Sent false
30813 8869 Acked Read 9 Done Done Sent false
30814 8870 Acked Write 8 Done Done Sent false
30815 8871 Acked Write 7 Done Done Sent false
30816 8872 Acked Write 1 Done Done Sent false
30817 8873 Acked Write 6 Done Done Sent false
30818 8874 Acked Write 9 Done Done Sent false
30819 8875 Acked Read 10 Done Done Sent false
30820 8876 Acked Write 4 Done Done Sent false
30821 8877 Acked Write 10 Done Done Sent false
30822 8878 Acked Write 3 Done Done Sent false
30823 8879 Acked Write 9 Done Done Sent false
30824 8880 Acked Read 10 Done Done Sent false
30825 8881 Acked Read 5 Done Done Sent false
30826 8882 Acked Write 9 Done Done Sent false
30827 8883 Acked Write 8 Done Done Sent false
30828 8884 Acked Read 3 Done Done Sent false
30829 8885 Acked Write 10 Done Done Sent false
30830 8886 Acked Write 1 Done Done Sent false
30831 8887 Acked Write 6 Done Done Sent false
30832 8888 Acked Read 7 Done Done Sent false
30833 8889 Acked Write 6 Done Done Sent false
30834 8890 Acked Write 4 Done Done Sent false
30835 8891 Acked Flush 0 Done Done Sent false
30836 8892 Acked Read 5 Done Done Sent false
30837 8893 Acked Read 1 Done Done Sent false
30838 8894 Acked Write 3 Done Done Sent false
30839 8895 Acked Read 1 Done Done Sent false
30840 8896 Acked Write 9 Done Done Sent false
30841 8897 Acked Write 9 Done Done Sent false
30842 8898 Acked Write 10 Done Done Sent false
30843 8899 Acked Read 8 Done Done Sent false
30844 8900 Acked Write 2 Done Done Sent false
30845 8901 Acked Write 7 Done Done Sent false
30846 8902 Acked Read 3 Done Done Sent false
30847 8903 Acked Read 5 Done Done Sent false
30848 8904 Acked Read 8 Done Done Sent false
30849 8905 Acked Read 2 Done Done Sent false
30850 8906 Acked Write 8 Done Done Sent false
30851 8907 Acked Write 4 Done Done Sent false
30852 8908 Acked Read 8 Done Done Sent false
30853 8909 Acked Write 1 Done Done Sent false
30854 8910 Acked Read 1 Done Done Sent false
30855 8911 Acked Read 9 Done Done Sent false
30856 8912 Acked Read 9 Done Done Sent false
30857 8913 Acked Read 10 Done Done Sent false
30858 8914 Acked Read 10 Done Done Sent false
30859 8915 Acked Read 6 Done Done Sent false
30860 8916 Acked Read 6 Done Done Sent false
30861 8917 Acked Read 6 Done Done Sent false
30862 8918 Acked Read 9 Done Done Sent false
30863 8919 Acked Write 7 Done Done Sent false
30864 8920 Acked Write 9 Done Done Sent false
30865 8921 Acked Read 8 Done Done Sent false
30866 8922 Acked Read 10 Done Done Sent false
30867 8923 Acked Read 3 Done Done Sent false
30868 8924 Acked Read 5 Done Done Sent false
30869 8925 Acked Read 3 Done Done Sent false
30870 8926 Acked Flush 0 Done Done Sent false
30871 8927 Acked Flush 0 Done Done Sent false
30872 8928 Acked Read 1 Done Done Sent false
30873 8929 Acked Read 10 Done Done Sent false
30874 8930 Acked Write 5 Done Done Sent false
30875 8931 Acked Read 1 Done Done Sent false
30876 8932 Acked Flush 0 Done Done Sent false
30877 8933 Acked Flush 0 Done Done Sent false
30878 8934 Acked Read 6 Done Done Sent false
30879 8935 Acked Flush 0 Done Done Sent false
30880 8936 Acked Write 4 Done Done Sent false
30881 8937 Acked Write 9 Done Done Sent false
30882 8938 Acked Write 4 Done Done Sent false
30883 8939 Acked Read 7 Done Done Sent false
30884 8940 Acked Read 2 Done Done Sent false
30885 8941 Acked Read 4 Done Done Sent false
30886 8942 Acked Write 3 Done Done Sent false
30887 8943 Acked Read 6 Done Done Sent false
30888 8944 Acked Read 6 Done Done Sent false
30889 8945 Acked Read 6 Done Done Sent false
30890 8946 Acked Read 8 Done Done Sent false
30891 8947 Acked Write 6 Done Done Sent false
30892 8948 Acked Write 8 Done Done Sent false
30893 8949 Acked Write 7 Done Done Sent false
30894 8950 Acked Read 1 Done Done Sent false
30895 8951 Acked Read 10 Done Done Sent false
30896 8952 Acked Read 4 Done Done Sent false
30897 8953 Acked Read 6 Done Done Sent false
30898 8954 Acked Write 5 Done Done Sent false
30899 8955 Acked Write 3 Done Done Sent false
30900 8956 Acked Write 7 Done Done Sent false
30901 8957 Acked Write 1 Done Done Sent false
30902 8958 Acked Flush 0 Done Done Sent false
30903 8959 Acked Read 1 Done Done Sent false
30904 8960 Acked Write 7 Done Done Sent false
30905 8961 Acked Write 4 Done Done Sent false
30906 8962 Acked Read 4 Done Done Sent false
30907 8963 Acked Read 10 Done Done Sent false
30908 8964 Acked Write 5 Done Done Sent false
30909 8965 Acked Flush 0 Done Done Sent false
30910 8966 Acked Write 7 Done Done Sent false
30911 8967 Acked Read 3 Done Done Sent false
30912 8968 Acked Read 2 Done Done Sent false
30913 8969 Acked Read 5 Done Done Sent false
30914 8970 Acked Read 2 Done Done Sent false
30915 8971 Acked Write 4 Done Done Sent false
30916 8972 Acked Read 7 Done Done Sent false
30917 8973 Acked Write 2 Done Done Sent false
30918 8974 Acked Write 7 Done Done Sent false
30919 8975 Acked Read 3 Done Done Sent false
30920 8976 Acked Write 8 Done Done Sent false
30921 8977 Acked Read 10 Done Done Sent false
30922 8978 Acked Read 6 Done Done Sent false
30923 8979 Acked Write 7 Done Done Sent false
30924 8980 Acked Write 5 Done Done Sent false
30925 8981 Acked Read 5 Done Done Sent false
30926 8982 Acked Write 2 Done Done Sent false
30927 8983 Acked Read 7 Done Done Sent false
30928 8984 Acked Read 1 Done Done Sent false
30929 8985 Acked Write 3 Done Done Sent false
30930 8986 Acked Write 8 Done Done Sent false
30931 8987 Acked Read 10 Done Done Sent false
30932 8988 Acked Write 1 Done Done Sent false
30933 8989 Acked Read 6 Done Done Sent false
30934 8990 Acked Read 3 Done Done Sent false
30935 8991 Acked Write 6 Done Done Sent false
30936 8992 Acked Write 9 Done Done Sent false
30937 8993 Acked Read 3 Done Done Sent false
30938 8994 Acked Read 6 Done Done Sent false
30939 8995 Acked Write 2 Done Done Sent false
30940 8996 Acked Write 1 Done Done Sent false
30941 8997 Acked Read 2 Done Done Sent false
30942 8998 Acked Write 8 Done Done Sent false
30943 8999 Acked Write 5 Done Done Sent false
30944 9000 Acked Write 5 Done Done Sent false
30945 9001 Acked Read 8 Done Done Sent false
30946 9002 Acked Write 8 Done Done Sent false
30947 9003 Acked Write 9 Done Done Sent false
30948 9004 Acked Write 8 Done Done Sent false
30949 9005 Acked Read 3 Done Done Sent false
30950 9006 Acked Read 9 Done Done Sent false
30951 9007 Acked Read 5 Done Done Sent false
30952 9008 Acked Write 4 Done Done Sent false
30953 9009 Acked Write 5 Done Done Sent false
30954 9010 Acked Write 1 Done Done Sent false
30955 9011 Acked Read 2 Done Done Sent false
30956 9012 Acked Read 3 Done Done Sent false
30957 9013 Acked Read 2 Done Done Sent false
30958 9014 Acked Read 10 Done Done Sent false
30959 9015 Acked Write 3 Done Done Sent false
30960 9016 Acked Write 2 Done Done Sent false
30961 9017 Acked Write 8 Done Done Sent false
30962 9018 Acked Read 4 Done Done Sent false
30963 9019 Acked Read 9 Done Done Sent false
30964 9020 Acked Write 2 Done Done Sent false
30965 9021 Acked Flush 0 Done Done Sent false
30966 9022 Acked Write 1 Done Done Sent false
30967 9023 Acked Write 9 Done Done Sent false
30968 9024 Acked Read 5 Done Done Sent false
30969 9025 Acked Write 1 Done Done Sent false
30970 9026 Acked Read 9 Done Done Sent false
30971 9027 Acked Read 7 Done Done Sent false
30972 9028 Acked Flush 0 Done Done Sent false
30973 9029 Acked Read 5 Done Done Sent false
30974 9030 Acked Read 1 Done Done Sent false
30975 9031 Acked Write 1 Done Done Sent false
30976 9032 Acked Read 6 Done Done Sent false
30977 9033 Acked Read 6 Done Done Sent false
30978 9034 Acked Write 6 Done Done Sent false
30979 9035 Acked Write 1 Done Done Sent false
30980 9036 Acked Flush 0 Done Done Sent false
30981 9037 Acked Write 4 Done Done Sent false
30982 9038 Acked Read 5 Done Done Sent false
30983 9039 Acked Read 5 Done Done Sent false
30984 9040 Acked Read 4 Done Done Sent false
30985 9041 Acked Read 10 Done Done Sent false
30986 9042 Acked Write 4 Done Done Sent false
30987 9043 Acked Read 10 Done Done Sent false
30988 9044 Acked Read 6 Done Done Sent false
30989 9045 Acked Flush 0 Done Done Sent false
30990 9046 Acked Flush 0 Done Done Sent false
30991 9047 Acked Write 5 Done Done Sent false
30992 9048 Acked Write 9 Done Done Sent false
30993 9049 Acked Read 3 Done Done Sent false
30994 9050 Acked Write 6 Done Done Sent false
30995 9051 Acked Write 10 Done Done Sent false
30996 9052 Acked Write 9 Done Done Sent false
30997 9053 Acked Read 9 Done Done Sent false
30998 9054 Acked Write 3 Done Done Sent false
30999 9055 Acked Read 8 Done Done Sent false
31000 9056 Acked Write 10 Done Done Sent false
31001 9057 Acked Flush 0 Done Done Sent false
31002 9058 Acked Write 9 Done Done Sent false
31003 9059 Acked Flush 0 Done Done Sent false
31004 9060 Acked Flush 0 Done Done Sent false
31005 9061 Acked Read 5 Done Done Sent false
31006 9062 Acked Read 8 Done Done Sent false
31007 9063 Acked Flush 0 Done Done Sent false
31008 9064 Acked Write 5 Done Done Sent false
31009 9065 Acked Flush 0 Done Done Sent false
31010 9066 Acked Flush 0 Done Done Sent false
31011 9067 Acked Read 7 Done Done Sent false
31012 9068 Acked Write 1 Done Done Sent false
31013 9069 Acked Read 7 Done Done Sent false
31014 9070 Acked Flush 0 Done Done Sent false
31015 9071 Acked Flush 0 Done Done Sent false
31016 9072 Acked Write 6 Done Done Sent false
31017 9073 Acked Write 10 Done Done Sent false
31018 9074 Acked Write 4 Done Done Sent false
31019 9075 Acked Write 4 Done Done Sent false
31020 9076 Acked Read 7 Done Done Sent false
31021 9077 Acked Read 9 Done Done Sent false
31022 9078 Acked Flush 0 Done Done Sent false
31023 9079 Acked Read 10 Done Done Sent false
31024 9080 Acked Read 10 Done Done Sent false
31025 9081 Acked Read 5 Done Done Sent false
31026 9082 Acked Write 6 Done Done Sent false
31027 9083 Acked Write 3 Done Done Sent false
31028 9084 Acked Read 4 Done Done Sent false
31029 9085 Acked Write 8 Done Done Sent false
31030 9086 Acked Write 7 Done Done Sent false
31031 9087 Acked Write 8 Done Done Sent false
31032 9088 Acked Read 5 Done Done Sent false
31033 9089 Acked Write 2 Done Done Sent false
31034 9090 Acked Read 4 Done Done Sent false
31035 9091 Acked Read 2 Done Done Sent false
31036 9092 Acked Read 3 Done Done Sent false
31037 9093 Acked Write 5 Done Done Sent false
31038 9094 Acked Write 9 Done Done Sent false
31039 9095 Acked Write 9 Done Done Sent false
31040 9096 Acked Read 10 Done Done Sent false
31041 9097 Acked Read 8 Done Done Sent false
31042 9098 Acked Write 1 Done Done Sent false
31043 9099 Acked Write 1 Done Done Sent false
31044 9100 Acked Read 5 Done Done Sent false
31045 9101 Acked Read 2 Done Done Sent false
31046 9102 Acked Write 10 Done Done Sent false
31047 9103 Acked Write 3 Done Done Sent false
31048 9104 Acked Read 9 Done Done Sent false
31049 9105 Acked Read 3 Done Done Sent false
31050 9106 Acked Read 10 Done Done Sent false
31051 9107 Acked Read 8 Done Done Sent false
31052 9108 Acked Write 7 Done Done Sent false
31053 9109 Acked Read 2 Done Done Sent false
31054 9110 Acked Write 6 Done Done Sent false
31055 9111 Acked Read 2 Done Done Sent false
31056 9112 Acked Read 6 Done Done Sent false
31057 9113 Acked Read 6 Done Done Sent false
31058 9114 Acked Read 8 Done Done Sent false
31059 9115 Acked Write 1 Done Done Sent false
31060 9116 Acked Flush 0 Done Done Sent false
31061 9117 Acked Read 8 Done Done Sent false
31062 9118 Acked Flush 0 Done Done Sent false
31063 9119 Acked Write 4 Done Done Sent false
31064 9120 Acked Read 2 Done Done Sent false
31065 9121 Acked Write 8 Done Done Sent false
31066 9122 Acked Write 4 Done Done Sent false
31067 9123 Acked Write 7 Done Done Sent false
31068 9124 Acked Read 6 Done Done Sent false
31069 9125 Acked Read 8 Done Done Sent false
31070 9126 Acked Flush 0 Done Done Sent false
31071 9127 Acked Read 3 Done Done Sent false
31072 9128 Acked Write 3 Done Done Sent false
31073 9129 Acked Write 9 Done Done Sent false
31074 9130 Acked Read 1 Done Done Sent false
31075 9131 Acked Read 2 Done Done Sent false
31076 9132 Acked Flush 0 Done Done Sent false
31077 9133 Acked Read 3 Done Done Sent false
31078 9134 Acked Read 3 Done Done Sent false
31079 9135 Acked Write 5 Done Done Sent false
31080 9136 Acked Write 2 Done Done Sent false
31081 9137 Acked Read 5 Done Done Sent false
31082 9138 Acked Write 8 Done Done Sent false
31083 9139 Acked Read 2 Done Done Sent false
31084 9140 Acked Write 3 Done Done Sent false
31085 9141 Acked Flush 0 Done Done Sent false
31086 9142 Acked Read 3 Done Done Sent false
31087 9143 Acked Write 9 Done Done Sent false
31088 9144 Acked Read 6 Done Done Sent false
31089 9145 Acked Write 8 Done Done Sent false
31090 9146 Acked Read 5 Done Done Sent false
31091 9147 Acked Read 9 Done Done Sent false
31092 9148 Acked Write 10 Done Done Sent false
31093 STATES DS:0 DS:1 DS:2 TOTAL
31094 Sent 0 0 301 301
31095 Done 301 301 0 602
31096 Skipped 0 0 0 0
31097 Error 0 0 0 0
31098 Last Flush: Some(JobId(9141)) Some(JobId(9141)) Some(JobId(8847))
31099 Downstairs last five completed: 8847 8846 8845 8844 8843
31100 Upstairs last five completed: 9148 9147 9146 9145 9144
31101 CLIENT: Up:0 ds:301 act:2
311022025-03-07T18:19:19.907ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311032025-03-07T18:19:20.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311042025-03-07T18:19:21.155ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311052025-03-07T18:19:21.155ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61445) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311062025-03-07T18:19:21.157ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311072025-03-07T18:19:21.157ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311082025-03-07T18:19:21.157ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311092025-03-07T18:19:21.157ZINFOcrucible: send last flush ID to this DS: Some(JobId(8847)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311102025-03-07T18:19:21.157ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311112025-03-07T18:19:21.157ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(8847)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311122025-03-07T18:19:21.158ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(8847)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31113 ----------------------------------------------------------------
31114 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31115 Downstairs last five completed: 9149 9148 9147 9146 9145
31116 Upstairs last five completed: 9149 9148 9147 9146 9145
31117 CLIENT: Up:0 ds:0 act:3
31118 Replay: All jobs finished
31119 Main task has work to do, go find it
31120 got dsc Stop(2)
31121 stop 2
31122 Mar 07 18:19:23.408 INFO request completed, latency_us: 399, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0b126fa9-83a0-493a-946d-a813fdd095a9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
31123 [8830] Got stop action so:false kr:false
31124 Mar 07 18:19:23.408 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/2, method: GET, req_id: 17670dcf-49f3-4a5c-ae89-3521ba2fb064, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
311252025-03-07T18:19:23.409ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311262025-03-07T18:19:23.409ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311272025-03-07T18:19:23.409ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
311282025-03-07T18:19:23.409ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31129 [8830] Exited with: ExitStatus(unix_wait_status(9))
31130 [8830][2] reports Exit
31131 Mar 07 18:19:27.412 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/2, method: GET, req_id: e2e586cd-9eae-4d63-a97d-fa133a51d8d8, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
31132 001/300 Write block 2290 len 20480 data: 4 5 4 3 3
31133 002/300 Write block 1221 len 8192 data: 4 5
31134 003/300 Write block 3203 len 24576 data: 8 7 7 7 6 5
31135 004/300 Read block 3017 len 4096
31136 005/300 Read block 2634 len 16384
31137 006/300 Write block 3991 len 40960 data: 7 8 7 7 8 6 6 7 6 5
31138 007/300 Write block 3264 len 40960 data: 4 5 7 8 7 7 5 3 3 3
31139 008/300 Read block 1785 len 8192
31140 009/300 Read block 1915 len 28672
31141 010/300 Read block 4896 len 12288
31142 011/300 Write block 119 len 8192 data: 6 6
31143 012/300 Read block 1238 len 16384
31144 013/300 Write block 2569 len 40960 data: 4 4 3 3 4 5 5 6 6 7
31145 014/300 Write block 2199 len 16384 data: 8 6 5 5
31146 015/300 Read block 2491 len 20480
31147 016/300 Read block 1761 len 12288
31148 017/300 Read block 4909 len 12288
31149 018/300 Flush
31150 019/300 Write block 1535 len 12288 data: 7 5 4
31151 020/300 Read block 2716 len 20480
31152 021/300 Read block 3189 len 36864
31153 022/300 Read block 1542 len 20480
31154 023/300 Read block 3812 len 20480
31155 024/300 Read block 3518 len 4096
31156 025/300 Read block 1237 len 32768
31157 026/300 Write block 604 len 4096 data: 9
31158 027/300 Write block 3420 len 32768 data: 7 6 6 6 6 7 6 6
31159 028/300 Flush
31160 029/300 Write block 14 len 16384 data: 9 10 8 5
31161 030/300 Write block 679 len 16384 data: 10 6 6 5
31162 031/300 Read block 1151 len 16384
31163 032/300 Write block 2924 len 12288 data: 9 8 8
31164 033/300 Read block 2882 len 20480
31165 034/300 Read block 3227 len 16384
31166 035/300 Read block 1248 len 36864
31167 036/300 Write block 1636 len 24576 data: 8 8 6 7 7 5
31168 037/300 Write block 3088 len 28672 data: 6 4 4 5 5 4 6
31169 038/300 Read block 856 len 28672
31170 039/300 Read block 2677 len 28672
31171 040/300 Read block 1452 len 36864
31172 041/300 Read block 1033 len 28672
31173 042/300 Write block 4312 len 16384 data: 5 6 6 7
31174 043/300 Write block 4542 len 20480 data: 9 10 10 10 11
31175 044/300 Write block 4189 len 32768 data: 4 3 3 3 2 5 5 5
31176 045/300 Read block 4915 len 32768
31177 046/300 Read block 61 len 36864
31178 047/300 Read block 2208 len 12288
31179 048/300 Read block 2817 len 20480
31180 049/300 Read block 3758 len 36864
31181 050/300 Write block 2971 len 12288 data: 6 6 7
31182 051/300 Write block 4706 len 16384 data: 8 8 5 4
31183 052/300 Write block 4247 len 36864 data: 4 6 6 8 8 9 9 7 8
31184 053/300 Read block 2845 len 4096
31185 054/300 Flush
31186 055/300 Write block 136 len 12288 data: 5 5 4
31187 056/300 Read block 2733 len 20480
31188 057/300 Write block 2005 len 4096 data: 7
31189 058/300 Read block 3199 len 12288
31190 059/300 Write block 613 len 8192 data: 3 5
31191 060/300 Read block 3812 len 12288
31192 061/300 Read block 2437 len 8192
31193 062/300 Read block 688 len 12288
31194 063/300 Read block 27 len 20480
31195 064/300 Read block 1938 len 32768
31196 065/300 Read block 552 len 16384
31197 066/300 Read block 484 len 24576
31198 067/300 Write block 2231 len 12288 data: 5 5 7
31199 068/300 Write block 673 len 24576 data: 7 5 5 8 7 11
31200 069/300 Write block 4213 len 24576 data: 8 7 8 6 6 6
31201 070/300 Write block 4536 len 24576 data: 8 9 7 7 8 8
31202 071/300 Read block 4436 len 24576
31203 072/300 Read block 1478 len 28672
31204 073/300 Read block 857 len 32768
31205 074/300 Read block 1548 len 24576
31206 075/300 Write block 2638 len 12288 data: 4 3 4
31207 076/300 Read block 2229 len 4096
31208 077/300 Read block 1386 len 8192
31209 078/300 Write block 994 len 16384 data: 6 7 5 6
31210 079/300 Write block 977 len 12288 data: 5 5 5
31211 080/300 Flush
31212 081/300 Read block 1331 len 4096
31213 082/300 Read block 3493 len 28672
31214 083/300 Flush
31215 084/300 Read block 1592 len 16384
31216 085/300 Read block 4903 len 4096
31217 086/300 Read block 2835 len 8192
31218 087/300 Read block 2965 len 40960
31219 088/300 Read block 4380 len 16384
31220 089/300 Read block 936 len 4096
31221 090/300 Read block 1766 len 28672
31222 091/300 Read block 4707 len 4096
31223 092/300 Flush
31224 093/300 Read block 2395 len 28672
31225 094/300 Read block 2750 len 4096
31226 095/300 Read block 3632 len 16384
31227 096/300 Read block 1446 len 24576
31228 097/300 Read block 1811 len 40960
31229 098/300 Flush
31230 099/300 Write block 320 len 24576 data: 5 5 6 6 6 8
31231 100/300 Read block 486 len 20480
31232 101/300 Read block 3322 len 12288
31233 102/300 Flush
31234 103/300 Read block 4182 len 28672
31235 104/300 Write block 2659 len 32768 data: 12 10 11 10 10 9 11 10
31236 105/300 Read block 2826 len 40960
31237 106/300 Write block 800 len 28672 data: 5 5 6 5 5 5 6
31238 107/300 Write block 3241 len 36864 data: 6 6 6 6 5 5 5 4 4
31239 108/300 Read block 2607 len 4096
31240 109/300 Read block 487 len 28672
31241 110/300 Read block 4510 len 4096
31242 111/300 Write block 3631 len 20480 data: 3 3 4 5 5
31243 112/300 Read block 2253 len 4096
31244 113/300 Read block 1640 len 4096
31245 114/300 Read block 4716 len 20480
31246 115/300 Write block 324 len 8192 data: 7 9
31247 116/300 Write block 4508 len 12288 data: 7 6 6
31248 117/300 Write block 3316 len 4096 data: 4
31249 118/300 Write block 1432 len 20480 data: 5 5 8 8 7
31250 119/300 Read block 166 len 40960
31251 120/300 Read block 4159 len 12288
31252 121/300 Read block 4543 len 28672
31253 122/300 Write block 341 len 24576 data: 6 5 5 4 6 6
31254 123/300 Write block 1782 len 8192 data: 4 4
31255 124/300 Read block 1863 len 8192
31256 125/300 Read block 2386 len 8192
31257 126/300 Read block 3179 len 16384
31258 127/300 Write block 4247 len 24576 data: 5 7 7 9 9 10
31259 128/300 Flush
31260 129/300 Read block 3427 len 16384
31261 130/300 Read block 1540 len 8192
31262 131/300 Write block 2739 len 16384 data: 10 9 8 7
31263 132/300 Write block 2016 len 4096 data: 9
31264 133/300 Read block 4430 len 32768
31265 134/300 Write block 3822 len 12288 data: 4 3 4
31266 135/300 Write block 4863 len 32768 data: 6 6 7 8 5 4 3 3
31267 136/300 Read block 4108 len 24576
31268 137/300 Write block 3339 len 40960 data: 6 6 6 6 6 6 4 4 3 4
31269 138/300 Read block 902 len 20480
31270 139/300 Read block 4901 len 8192
31271 140/300 Flush
31272 141/300 Read block 3685 len 4096
31273 142/300 Read block 1355 len 32768
31274 143/300 Write block 946 len 28672 data: 6 2 3 3 3 3 4
31275 144/300 Write block 4210 len 16384 data: 7 9 10 9
31276 145/300 Flush
31277 146/300 Write block 314 len 12288 data: 4 4 4
31278 147/300 Read block 4143 len 8192
31279 148/300 Flush
31280 149/300 Read block 242 len 24576
31281 150/300 Read block 1101 len 4096
31282 151/300 Write block 2265 len 16384 data: 7 7 5 5
31283 152/300 Write block 562 len 16384 data: 5 6 7 7
31284 153/300 Flush
31285 154/300 Write block 838 len 36864 data: 3 5 4 3 3 3 3 3 4
31286 155/300 Read block 1465 len 12288
31287 156/300 Write block 3356 len 32768 data: 5 6 4 4 5 5 6 8
31288 157/300 Read block 3843 len 20480
31289 158/300 Write block 4120 len 8192 data: 9 8
31290 159/300 Read block 3207 len 20480
31291 160/300 Read block 3662 len 16384
31292 161/300 Read block 3902 len 24576
31293 162/300 Read block 1796 len 28672
31294 163/300 Write block 2459 len 16384 data: 7 7 7 7
31295 164/300 Read block 326 len 28672
31296 165/300 Write block 2298 len 16384 data: 4 4 5 5
31297 166/300 Read block 1819 len 28672
31298 167/300 Write block 2483 len 36864 data: 5 4 4 4 6 6 6 6 6
31299 168/300 Read block 3538 len 36864
31300 169/300 Read block 3800 len 20480
31301 170/300 Read block 4660 len 24576
31302 171/300 Read block 2290 len 24576
31303 172/300 Read block 2535 len 8192
31304 173/300 Read block 3963 len 36864
31305 174/300 Read block 3282 len 16384
31306 175/300 Read block 746 len 40960
31307 176/300 Read block 972 len 24576
31308 177/300 Read block 2558 len 8192
31309 178/300 Read block 2039 len 32768
31310 179/300 Flush
31311 180/300 Read block 2329 len 24576
31312 181/300 Read block 2106 len 40960
31313 182/300 Read block 3590 len 28672
31314 183/300 Read block 673 len 28672
31315 184/300 Read block 3799 len 32768
31316 185/300 Write block 3873 len 20480 data: 5 6 5 5 5
31317 186/300 Write block 3208 len 4096 data: 6
31318 187/300 Write block 3894 len 24576 data: 4 4 5 5 4 4
31319 188/300 Flush
31320 189/300 Flush
31321 190/300 Write block 210 len 16384 data: 6 3 6 7
31322 191/300 Read block 4028 len 36864
31323 192/300 Read block 4246 len 12288
31324 193/300 Write block 3690 len 4096 data: 4
31325 194/300 Write block 1511 len 4096 data: 6
31326 195/300 Write block 1013 len 36864 data: 7 7 6 5 6 6 5 5 6
31327 196/300 Read block 2839 len 36864
31328 197/300 Write block 4769 len 28672 data: 7 8 8 8 6 7 6
31329 198/300 Read block 3434 len 12288
31330 199/300 Read block 2261 len 36864
31331 200/300 Read block 4506 len 40960
31332 201/300 Read block 4337 len 40960
31333 202/300 Write block 3792 len 16384 data: 4 8 7 7
31334 203/300 Read block 3566 len 8192
31335 204/300 Read block 1389 len 16384
31336 205/300 Write block 2714 len 4096 data: 7
31337 206/300 Read block 3808 len 4096
31338 207/300 Write block 3348 len 24576 data: 5 5 6 6 7 6
31339 208/300 Read block 2046 len 36864
31340 209/300 Read block 4972 len 32768
31341 210/300 Read block 3317 len 8192
31342 211/300 Read block 2978 len 36864
31343 212/300 Read block 2312 len 32768
31344 213/300 Flush
31345 214/300 Write block 2390 len 4096 data: 8
31346 215/300 Write block 3115 len 12288 data: 7 7 7
31347 216/300 Write block 3860 len 12288 data: 5 3 3
31348 217/300 Write block 4330 len 32768 data: 11 12 10 10 10 7 6 6
31349 218/300 Read block 3781 len 12288
31350 219/300 Read block 1300 len 24576
31351 220/300 Flush
31352 221/300 Read block 2071 len 8192
31353 222/300 Write block 1830 len 36864 data: 6 8 9 6 7 6 6 7 6
31354 223/300 Write block 130 len 20480 data: 8 9 7 6 7
31355 224/300 Read block 2898 len 16384
31356 225/300 Read block 3221 len 8192
31357 226/300 Read block 572 len 8192
31358 227/300 Read block 3496 len 4096
31359 228/300 Read block 3683 len 4096
31360 229/300 Read block 1659 len 32768
31361 230/300 Write block 412 len 4096 data: 6
31362 231/300 Read block 1014 len 28672
31363 232/300 Read block 3336 len 16384
31364 233/300 Read block 2861 len 12288
31365 234/300 Read block 67 len 20480
31366 235/300 Read block 3849 len 8192
31367 236/300 Read block 4979 len 12288
31368 237/300 Read block 327 len 20480
31369 238/300 Write block 4773 len 20480 data: 7 8 7 6 6
31370 239/300 Write block 4231 len 20480 data: 4 2 5 5 5
31371 240/300 Read block 2286 len 32768
31372 241/300 Write block 3452 len 12288 data: 7 6 7
31373 242/300 Read block 1095 len 20480
31374 243/300 Flush
31375 244/300 Read block 1314 len 12288
31376 245/300 Write block 2242 len 8192 data: 6 7
31377 246/300 Read block 4545 len 40960
31378 247/300 Read block 4109 len 12288
31379 248/300 Read block 3268 len 12288
31380 249/300 Read block 4173 len 36864
31381 250/300 Read block 486 len 8192
31382 251/300 Read block 2630 len 24576
31383 252/300 Read block 3123 len 12288
31384 253/300 Flush
31385 254/300 Read block 2961 len 4096
31386 255/300 Read block 1057 len 8192
31387 256/300 Write block 1831 len 32768 data: 9 10 7 8 7 7 8 7
31388 257/300 Read block 1289 len 16384
31389 258/300 Flush
31390 259/300 Read block 1161 len 8192
31391 260/300 Read block 4384 len 8192
31392 261/300 Write block 2248 len 4096 data: 5
31393 262/300 Read block 2028 len 12288
31394 263/300 Read block 2120 len 4096
31395 264/300 Read block 4522 len 24576
31396 265/300 Write block 788 len 16384 data: 7 7 7 6
31397 266/300 Write block 4807 len 24576 data: 6 6 5 2 3 3
31398 267/300 Read block 47 len 36864
31399 268/300 Write block 201 len 8192 data: 9 9
31400 269/300 Write block 3111 len 4096 data: 6
31401 270/300 Write block 3928 len 4096 data: 9
31402 271/300 Read block 3011 len 24576
31403 272/300 Flush
31404 273/300 Write block 2161 len 8192 data: 5 5
31405 274/300 Flush
31406 275/300 Read block 4784 len 28672
31407 276/300 Read block 664 len 4096
31408 277/300 Read block 2814 len 36864
31409 278/300 Read block 2975 len 28672
31410 279/300 Write block 1443 len 4096 data: 3
31411 280/300 Write block 1286 len 40960 data: 4 5 4 4 5 6 7 6 7 6
31412 281/300 Read block 1854 len 8192
31413 282/300 Write block 485 len 12288 data: 4 5 5
31414 283/300 Write block 1075 len 4096 data: 5
31415 284/300 Write block 1415 len 12288 data: 3 3 2
31416 285/300 Write block 892 len 32768 data: 5 4 4 4 3 4 2 2
31417 286/300 Read block 1978 len 20480
31418 287/300 Flush
31419 288/300 Write block 1100 len 20480 data: 5 5 5 7 7
31420 289/300 Write block 2167 len 12288 data: 5 5 3
31421 290/300 Write block 4956 len 32768 data: 6 8 10 9 9 9 6 7
31422 291/300 Read block 212 len 4096
31423 292/300 Flush
31424 293/300 Write block 2335 len 28672 data: 10 10 7 7 8 6 7
31425 294/300 Read block 4022 len 24576
31426 295/300 Write block 3460 len 16384 data: 4 3 3 6
31427 296/300 Read block 1955 len 36864
31428 297/300 Read block 307 len 12288
31429 298/300 Read block 753 len 40960
31430 299/300 Read block 2866 len 16384
31431 300/300 Write block 4335 len 40960 data: 8 7 7 4 4 5 5 8 7 8
31432 301/300 Write block 1567 len 28672 data: 12 13 12 10 8 8 8
31433 Main task has work to do, go find it
31434 got dsc Start(2)
31435 start 2
31436 Mar 07 18:19:27.654 INFO request completed, latency_us: 193, response_code: 204, uri: /start/cid/2, method: POST, req_id: fa3d8bdf-66f3-49bc-b589-12208f6f91db, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
31437 Replay: started 2, returned:Ok(())
31438 ----------------------------------------------------------------
31439 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31440 States: Active Active Offline
31441 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31442 9150 Acked Write 5 Done Done Sent false
31443 9151 Acked Write 2 Done Done Sent false
31444 9152 Acked Write 6 Done Done Sent false
31445 9153 Acked Read 1 Done Done Sent false
31446 9154 Acked Read 4 Done Done Sent false
31447 9155 Acked Write 10 Done Done Sent false
31448 9156 Acked Write 10 Done Done Sent false
31449 9157 Acked Read 2 Done Done Sent false
31450 9158 Acked Read 7 Done Done Sent false
31451 9159 Acked Read 3 Done Done Sent false
31452 9160 Acked Write 2 Done Done Sent false
31453 9161 Acked Read 4 Done Done Sent false
31454 9162 Acked Write 10 Done Done Sent false
31455 9163 Acked Write 4 Done Done Sent false
31456 9164 Acked Read 5 Done Done Sent false
31457 9165 Acked Read 3 Done Done Sent false
31458 9166 Acked Read 3 Done Done Sent false
31459 9167 Acked Flush 0 Done Done Sent false
31460 9168 Acked Write 3 Done Done Sent false
31461 9169 Acked Read 5 Done Done Sent false
31462 9170 Acked Read 9 Done Done Sent false
31463 9171 Acked Read 5 Done Done Sent false
31464 9172 Acked Read 5 Done Done Sent false
31465 9173 Acked Read 1 Done Done Sent false
31466 9174 Acked Read 8 Done Done Sent false
31467 9175 Acked Write 1 Done Done Sent false
31468 9176 Acked Write 8 Done Done Sent false
31469 9177 Acked Flush 0 Done Done Sent false
31470 9178 Acked Write 4 Done Done Sent false
31471 9179 Acked Write 4 Done Done Sent false
31472 9180 Acked Read 4 Done Done Sent false
31473 9181 Acked Write 3 Done Done Sent false
31474 9182 Acked Read 5 Done Done Sent false
31475 9183 Acked Read 4 Done Done Sent false
31476 9184 Acked Read 9 Done Done Sent false
31477 9185 Acked Write 6 Done Done Sent false
31478 9186 Acked Write 7 Done Done Sent false
31479 9187 Acked Read 7 Done Done Sent false
31480 9188 Acked Read 7 Done Done Sent false
31481 9189 Acked Read 9 Done Done Sent false
31482 9190 Acked Read 7 Done Done Sent false
31483 9191 Acked Write 4 Done Done Sent false
31484 9192 Acked Write 5 Done Done Sent false
31485 9193 Acked Write 8 Done Done Sent false
31486 9194 Acked Read 8 Done Done Sent false
31487 9195 Acked Read 9 Done Done Sent false
31488 9196 Acked Read 3 Done Done Sent false
31489 9197 Acked Read 5 Done Done Sent false
31490 9198 Acked Read 9 Done Done Sent false
31491 9199 Acked Write 3 Done Done Sent false
31492 9200 Acked Write 4 Done Done Sent false
31493 9201 Acked Write 9 Done Done Sent false
31494 9202 Acked Read 1 Done Done Sent false
31495 9203 Acked Flush 0 Done Done Sent false
31496 9204 Acked Write 3 Done Done Sent false
31497 9205 Acked Read 5 Done Done Sent false
31498 9206 Acked Write 1 Done Done Sent false
31499 9207 Acked Read 3 Done Done Sent false
31500 9208 Acked Write 2 Done Done Sent false
31501 9209 Acked Read 3 Done Done Sent false
31502 9210 Acked Read 2 Done Done Sent false
31503 9211 Acked Read 3 Done Done Sent false
31504 9212 Acked Read 5 Done Done Sent false
31505 9213 Acked Read 8 Done Done Sent false
31506 9214 Acked Read 4 Done Done Sent false
31507 9215 Acked Read 6 Done Done Sent false
31508 9216 Acked Write 3 Done Done Sent false
31509 9217 Acked Write 6 Done Done Sent false
31510 9218 Acked Write 6 Done Done Sent false
31511 9219 Acked Write 6 Done Done Sent false
31512 9220 Acked Read 6 Done Done Sent false
31513 9221 Acked Read 7 Done Done Sent false
31514 9222 Acked Read 8 Done Done Sent false
31515 9223 Acked Read 6 Done Done Sent false
31516 9224 Acked Write 3 Done Done Sent false
31517 9225 Acked Read 1 Done Done Sent false
31518 9226 Acked Read 2 Done Done Sent false
31519 9227 Acked Write 4 Done Done Sent false
31520 9228 Acked Write 3 Done Done Sent false
31521 9229 Acked Flush 0 Done Done Sent false
31522 9230 Acked Read 1 Done Done Sent false
31523 9231 Acked Read 7 Done Done Sent false
31524 9232 Acked Flush 0 Done Done Sent false
31525 9233 Acked Read 4 Done Done Sent false
31526 9234 Acked Read 1 Done Done Sent false
31527 9235 Acked Read 2 Done Done Sent false
31528 9236 Acked Read 10 Done Done Sent false
31529 9237 Acked Read 4 Done Done Sent false
31530 9238 Acked Read 1 Done Done Sent false
31531 9239 Acked Read 7 Done Done Sent false
31532 9240 Acked Read 1 Done Done Sent false
31533 9241 Acked Flush 0 Done Done Sent false
31534 9242 Acked Read 7 Done Done Sent false
31535 9243 Acked Read 1 Done Done Sent false
31536 9244 Acked Read 4 Done Done Sent false
31537 9245 Acked Read 6 Done Done Sent false
31538 9246 Acked Read 10 Done Done Sent false
31539 9247 Acked Flush 0 Done Done Sent false
31540 9248 Acked Write 6 Done Done Sent false
31541 9249 Acked Read 5 Done Done Sent false
31542 9250 Acked Read 3 Done Done Sent false
31543 9251 Acked Flush 0 Done Done Sent false
31544 9252 Acked Read 7 Done Done Sent false
31545 9253 Acked Write 8 Done Done Sent false
31546 9254 Acked Read 10 Done Done Sent false
31547 9255 Acked Write 7 Done Done Sent false
31548 9256 Acked Write 9 Done Done Sent false
31549 9257 Acked Read 1 Done Done Sent false
31550 9258 Acked Read 7 Done Done Sent false
31551 9259 Acked Read 1 Done Done Sent false
31552 9260 Acked Write 5 Done Done Sent false
31553 9261 Acked Read 1 Done Done Sent false
31554 9262 Acked Read 1 Done Done Sent false
31555 9263 Acked Read 5 Done Done Sent false
31556 9264 Acked Write 2 Done Done Sent false
31557 9265 Acked Write 3 Done Done Sent false
31558 9266 Acked Write 1 Done Done Sent false
31559 9267 Acked Write 5 Done Done Sent false
31560 9268 Acked Read 10 Done Done Sent false
31561 9269 Acked Read 3 Done Done Sent false
31562 9270 Acked Read 7 Done Done Sent false
31563 9271 Acked Write 6 Done Done Sent false
31564 9272 Acked Write 2 Done Done Sent false
31565 9273 Acked Read 2 Done Done Sent false
31566 9274 Acked Read 2 Done Done Sent false
31567 9275 Acked Read 4 Done Done Sent false
31568 9276 Acked Write 6 Done Done Sent false
31569 9277 Acked Flush 0 Done Done Sent false
31570 9278 Acked Read 4 Done Done Sent false
31571 9279 Acked Read 2 Done Done Sent false
31572 9280 Acked Write 4 Done Done Sent false
31573 9281 Acked Write 1 Done Done Sent false
31574 9282 Acked Read 8 Done Done Sent false
31575 9283 Acked Write 3 Done Done Sent false
31576 9284 Acked Write 8 Done Done Sent false
31577 9285 Acked Read 6 Done Done Sent false
31578 9286 Acked Write 10 Done Done Sent false
31579 9287 Acked Read 5 Done Done Sent false
31580 9288 Acked Read 2 Done Done Sent false
31581 9289 Acked Flush 0 Done Done Sent false
31582 9290 Acked Read 1 Done Done Sent false
31583 9291 Acked Read 8 Done Done Sent false
31584 9292 Acked Write 7 Done Done Sent false
31585 9293 Acked Write 4 Done Done Sent false
31586 9294 Acked Flush 0 Done Done Sent false
31587 9295 Acked Write 3 Done Done Sent false
31588 9296 Acked Read 2 Done Done Sent false
31589 9297 Acked Flush 0 Done Done Sent false
31590 9298 Acked Read 6 Done Done Sent false
31591 9299 Acked Read 1 Done Done Sent false
31592 9300 Acked Write 4 Done Done Sent false
31593 9301 Acked Write 4 Done Done Sent false
31594 9302 Acked Flush 0 Done Done Sent false
31595 9303 Acked Write 9 Done Done Sent false
31596 9304 Acked Read 3 Done Done Sent false
31597 9305 Acked Write 8 Done Done Sent false
31598 9306 Acked Read 5 Done Done Sent false
31599 9307 Acked Write 2 Done Done Sent false
31600 9308 Acked Read 5 Done Done Sent false
31601 9309 Acked Read 4 Done Done Sent false
31602 9310 Acked Read 6 Done Done Sent false
31603 9311 Acked Read 7 Done Done Sent false
31604 9312 Acked Write 4 Done Done Sent false
31605 9313 Acked Read 7 Done Done Sent false
31606 9314 Acked Write 4 Done Done Sent false
31607 9315 Acked Read 7 Done Done Sent false
31608 9316 Acked Write 9 Done Done Sent false
31609 9317 Acked Read 9 Done Done Sent false
31610 9318 Acked Read 5 Done Done Sent false
31611 9319 Acked Read 6 Done Done Sent false
31612 9320 Acked Read 6 Done Done Sent false
31613 9321 Acked Read 2 Done Done Sent false
31614 9322 Acked Read 9 Done Done Sent false
31615 9323 Acked Read 4 Done Done Sent false
31616 9324 Acked Read 10 Done Done Sent false
31617 9325 Acked Read 6 Done Done Sent false
31618 9326 Acked Read 2 Done Done Sent false
31619 9327 Acked Read 8 Done Done Sent false
31620 9328 Acked Flush 0 Done Done Sent false
31621 9329 Acked Read 6 Done Done Sent false
31622 9330 Acked Read 10 Done Done Sent false
31623 9331 Acked Read 7 Done Done Sent false
31624 9332 Acked Read 7 Done Done Sent false
31625 9333 Acked Read 8 Done Done Sent false
31626 9334 Acked Write 5 Done Done Sent false
31627 9335 Acked Write 1 Done Done Sent false
31628 9336 Acked Write 6 Done Done Sent false
31629 9337 Acked Flush 0 Done Done Sent false
31630 9338 Acked Flush 0 Done Done Sent false
31631 9339 Acked Write 4 Done Done Sent false
31632 9340 Acked Read 9 Done Done Sent false
31633 9341 Acked Read 3 Done Done Sent false
31634 9342 Acked Write 1 Done Done Sent false
31635 9343 Acked Write 1 Done Done Sent false
31636 9344 Acked Write 9 Done Done Sent false
31637 9345 Acked Read 9 Done Done Sent false
31638 9346 Acked Write 7 Done Done Sent false
31639 9347 Acked Read 3 Done Done Sent false
31640 9348 Acked Read 9 Done Done Sent false
31641 9349 Acked Read 10 Done Done Sent false
31642 9350 Acked Read 10 Done Done Sent false
31643 9351 Acked Write 4 Done Done Sent false
31644 9352 Acked Read 2 Done Done Sent false
31645 9353 Acked Read 4 Done Done Sent false
31646 9354 Acked Write 1 Done Done Sent false
31647 9355 Acked Read 1 Done Done Sent false
31648 9356 Acked Write 6 Done Done Sent false
31649 9357 Acked Read 9 Done Done Sent false
31650 9358 Acked Read 8 Done Done Sent false
31651 9359 Acked Read 2 Done Done Sent false
31652 9360 Acked Read 9 Done Done Sent false
31653 9361 Acked Read 8 Done Done Sent false
31654 9362 Acked Flush 0 Done Done Sent false
31655 9363 Acked Write 1 Done Done Sent false
31656 9364 Acked Write 3 Done Done Sent false
31657 9365 Acked Write 3 Done Done Sent false
31658 9366 Acked Write 8 Done Done Sent false
31659 9367 Acked Read 3 Done Done Sent false
31660 9368 Acked Read 6 Done Done Sent false
31661 9369 Acked Flush 0 Done Done Sent false
31662 9370 Acked Read 2 Done Done Sent false
31663 9371 Acked Write 9 Done Done Sent false
31664 9372 Acked Write 5 Done Done Sent false
31665 9373 Acked Read 4 Done Done Sent false
31666 9374 Acked Read 2 Done Done Sent false
31667 9375 Acked Read 2 Done Done Sent false
31668 9376 Acked Read 1 Done Done Sent false
31669 9377 Acked Read 1 Done Done Sent false
31670 9378 Acked Read 8 Done Done Sent false
31671 9379 Acked Write 1 Done Done Sent false
31672 9380 Acked Read 7 Done Done Sent false
31673 9381 Acked Read 4 Done Done Sent false
31674 9382 Acked Read 3 Done Done Sent false
31675 9383 Acked Read 5 Done Done Sent false
31676 9384 Acked Read 2 Done Done Sent false
31677 9385 Acked Read 3 Done Done Sent false
31678 9386 Acked Read 5 Done Done Sent false
31679 9387 Acked Write 5 Done Done Sent false
31680 9388 Acked Write 5 Done Done Sent false
31681 9389 Acked Read 8 Done Done Sent false
31682 9390 Acked Write 3 Done Done Sent false
31683 9391 Acked Read 5 Done Done Sent false
31684 9392 Acked Flush 0 Done Done Sent false
31685 9393 Acked Read 3 Done Done Sent false
31686 9394 Acked Write 2 Done Done Sent false
31687 9395 Acked Read 10 Done Done Sent false
31688 9396 Acked Read 3 Done Done Sent false
31689 9397 Acked Read 3 Done Done Sent false
31690 9398 Acked Read 9 Done Done Sent false
31691 9399 Acked Read 2 Done Done Sent false
31692 9400 Acked Read 6 Done Done Sent false
31693 9401 Acked Read 3 Done Done Sent false
31694 9402 Acked Flush 0 Done Done Sent false
31695 9403 Acked Read 1 Done Done Sent false
31696 9404 Acked Read 2 Done Done Sent false
31697 9405 Acked Write 8 Done Done Sent false
31698 9406 Acked Read 4 Done Done Sent false
31699 9407 Acked Flush 0 Done Done Sent false
31700 9408 Acked Read 2 Done Done Sent false
31701 9409 Acked Read 2 Done Done Sent false
31702 9410 Acked Write 1 Done Done Sent false
31703 9411 Acked Read 3 Done Done Sent false
31704 9412 Acked Read 1 Done Done Sent false
31705 9413 Acked Read 6 Done Done Sent false
31706 9414 Acked Write 4 Done Done Sent false
31707 9415 Acked Write 6 Done Done Sent false
31708 9416 Acked Read 9 Done Done Sent false
31709 9417 Acked Write 2 Done Done Sent false
31710 9418 Acked Write 1 Done Done Sent false
31711 9419 Acked Write 1 Done Done Sent false
31712 9420 Acked Read 6 Done Done Sent false
31713 9421 Acked Flush 0 Done Done Sent false
31714 9422 Acked Write 2 Done Done Sent false
31715 9423 Acked Flush 0 Done Done Sent false
31716 9424 Acked Read 7 Done Done Sent false
31717 9425 Acked Read 1 Done Done Sent false
31718 9426 Acked Read 9 Done Done Sent false
31719 9427 Acked Read 7 Done Done Sent false
31720 9428 Acked Write 1 Done Done Sent false
31721 9429 Acked Write 10 Done Done Sent false
31722 9430 Acked Read 2 Done Done Sent false
31723 9431 Acked Write 3 Done Done Sent false
31724 9432 Acked Write 1 Done Done Sent false
31725 9433 Acked Write 3 Done Done Sent false
31726 9434 Acked Write 8 Done Done Sent false
31727 9435 Acked Read 5 Done Done Sent false
31728 9436 Acked Flush 0 Done Done Sent false
31729 9437 Acked Write 5 Done Done Sent false
31730 9438 Acked Write 3 Done Done Sent false
31731 9439 Acked Write 8 Done Done Sent false
31732 9440 Acked Read 1 Done Done Sent false
31733 9441 Acked Flush 0 Done Done Sent false
31734 9442 Acked Write 7 Done Done Sent false
31735 9443 Acked Read 6 Done Done Sent false
31736 9444 Acked Write 4 Done Done Sent false
31737 9445 Acked Read 9 Done Done Sent false
31738 9446 Acked Read 3 Done Done Sent false
31739 9447 Acked Read 10 Done Done Sent false
31740 9448 Acked Read 4 Done Done Sent false
31741 9449 Acked Write 10 Done Done Sent false
31742 9450 Acked Write 7 Done Done Sent false
31743 STATES DS:0 DS:1 DS:2 TOTAL
31744 Sent 0 0 301 301
31745 Done 301 301 0 602
31746 Skipped 0 0 0 0
31747 Error 0 0 0 0
31748 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
31749 Downstairs last five completed: 9149 9148 9147 9146 9145
31750 Upstairs last five completed: 9450 9449 9448 9447 9446
31751 CLIENT: Up:0 ds:301 act:2
317522025-03-07T18:19:28.636ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
317532025-03-07T18:19:29.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
317542025-03-07T18:19:30.661ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
317552025-03-07T18:19:31.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
31756 ----------------------------------------------------------------
31757 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31758 States: Active Active Offline
31759 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31760 9150 Acked Write 5 Done Done Sent false
31761 9151 Acked Write 2 Done Done Sent false
31762 9152 Acked Write 6 Done Done Sent false
31763 9153 Acked Read 1 Done Done Sent false
31764 9154 Acked Read 4 Done Done Sent false
31765 9155 Acked Write 10 Done Done Sent false
31766 9156 Acked Write 10 Done Done Sent false
31767 9157 Acked Read 2 Done Done Sent false
31768 9158 Acked Read 7 Done Done Sent false
31769 9159 Acked Read 3 Done Done Sent false
31770 9160 Acked Write 2 Done Done Sent false
31771 9161 Acked Read 4 Done Done Sent false
31772 9162 Acked Write 10 Done Done Sent false
31773 9163 Acked Write 4 Done Done Sent false
31774 9164 Acked Read 5 Done Done Sent false
31775 9165 Acked Read 3 Done Done Sent false
31776 9166 Acked Read 3 Done Done Sent false
31777 9167 Acked Flush 0 Done Done Sent false
31778 9168 Acked Write 3 Done Done Sent false
31779 9169 Acked Read 5 Done Done Sent false
31780 9170 Acked Read 9 Done Done Sent false
31781 9171 Acked Read 5 Done Done Sent false
31782 9172 Acked Read 5 Done Done Sent false
31783 9173 Acked Read 1 Done Done Sent false
31784 9174 Acked Read 8 Done Done Sent false
31785 9175 Acked Write 1 Done Done Sent false
31786 9176 Acked Write 8 Done Done Sent false
31787 9177 Acked Flush 0 Done Done Sent false
31788 9178 Acked Write 4 Done Done Sent false
31789 9179 Acked Write 4 Done Done Sent false
31790 9180 Acked Read 4 Done Done Sent false
31791 9181 Acked Write 3 Done Done Sent false
31792 9182 Acked Read 5 Done Done Sent false
31793 9183 Acked Read 4 Done Done Sent false
31794 9184 Acked Read 9 Done Done Sent false
31795 9185 Acked Write 6 Done Done Sent false
31796 9186 Acked Write 7 Done Done Sent false
31797 9187 Acked Read 7 Done Done Sent false
31798 9188 Acked Read 7 Done Done Sent false
31799 9189 Acked Read 9 Done Done Sent false
31800 9190 Acked Read 7 Done Done Sent false
31801 9191 Acked Write 4 Done Done Sent false
31802 9192 Acked Write 5 Done Done Sent false
31803 9193 Acked Write 8 Done Done Sent false
31804 9194 Acked Read 8 Done Done Sent false
31805 9195 Acked Read 9 Done Done Sent false
31806 9196 Acked Read 3 Done Done Sent false
31807 9197 Acked Read 5 Done Done Sent false
31808 9198 Acked Read 9 Done Done Sent false
31809 9199 Acked Write 3 Done Done Sent false
31810 9200 Acked Write 4 Done Done Sent false
31811 9201 Acked Write 9 Done Done Sent false
31812 9202 Acked Read 1 Done Done Sent false
31813 9203 Acked Flush 0 Done Done Sent false
31814 9204 Acked Write 3 Done Done Sent false
31815 9205 Acked Read 5 Done Done Sent false
31816 9206 Acked Write 1 Done Done Sent false
31817 9207 Acked Read 3 Done Done Sent false
31818 9208 Acked Write 2 Done Done Sent false
31819 9209 Acked Read 3 Done Done Sent false
31820 9210 Acked Read 2 Done Done Sent false
31821 9211 Acked Read 3 Done Done Sent false
31822 9212 Acked Read 5 Done Done Sent false
31823 9213 Acked Read 8 Done Done Sent false
31824 9214 Acked Read 4 Done Done Sent false
31825 9215 Acked Read 6 Done Done Sent false
31826 9216 Acked Write 3 Done Done Sent false
31827 9217 Acked Write 6 Done Done Sent false
31828 9218 Acked Write 6 Done Done Sent false
31829 9219 Acked Write 6 Done Done Sent false
31830 9220 Acked Read 6 Done Done Sent false
31831 9221 Acked Read 7 Done Done Sent false
31832 9222 Acked Read 8 Done Done Sent false
31833 9223 Acked Read 6 Done Done Sent false
31834 9224 Acked Write 3 Done Done Sent false
31835 9225 Acked Read 1 Done Done Sent false
31836 9226 Acked Read 2 Done Done Sent false
31837 9227 Acked Write 4 Done Done Sent false
31838 9228 Acked Write 3 Done Done Sent false
31839 9229 Acked Flush 0 Done Done Sent false
31840 9230 Acked Read 1 Done Done Sent false
31841 9231 Acked Read 7 Done Done Sent false
31842 9232 Acked Flush 0 Done Done Sent false
31843 9233 Acked Read 4 Done Done Sent false
31844 9234 Acked Read 1 Done Done Sent false
31845 9235 Acked Read 2 Done Done Sent false
31846 9236 Acked Read 10 Done Done Sent false
31847 9237 Acked Read 4 Done Done Sent false
31848 9238 Acked Read 1 Done Done Sent false
31849 9239 Acked Read 7 Done Done Sent false
31850 9240 Acked Read 1 Done Done Sent false
31851 9241 Acked Flush 0 Done Done Sent false
31852 9242 Acked Read 7 Done Done Sent false
31853 9243 Acked Read 1 Done Done Sent false
31854 9244 Acked Read 4 Done Done Sent false
31855 9245 Acked Read 6 Done Done Sent false
31856 9246 Acked Read 10 Done Done Sent false
31857 9247 Acked Flush 0 Done Done Sent false
31858 9248 Acked Write 6 Done Done Sent false
31859 9249 Acked Read 5 Done Done Sent false
31860 9250 Acked Read 3 Done Done Sent false
31861 9251 Acked Flush 0 Done Done Sent false
31862 9252 Acked Read 7 Done Done Sent false
31863 9253 Acked Write 8 Done Done Sent false
31864 9254 Acked Read 10 Done Done Sent false
31865 9255 Acked Write 7 Done Done Sent false
31866 9256 Acked Write 9 Done Done Sent false
31867 9257 Acked Read 1 Done Done Sent false
31868 9258 Acked Read 7 Done Done Sent false
31869 9259 Acked Read 1 Done Done Sent false
31870 9260 Acked Write 5 Done Done Sent false
31871 9261 Acked Read 1 Done Done Sent false
31872 9262 Acked Read 1 Done Done Sent false
31873 9263 Acked Read 5 Done Done Sent false
31874 9264 Acked Write 2 Done Done Sent false
31875 9265 Acked Write 3 Done Done Sent false
31876 9266 Acked Write 1 Done Done Sent false
31877 9267 Acked Write 5 Done Done Sent false
31878 9268 Acked Read 10 Done Done Sent false
31879 9269 Acked Read 3 Done Done Sent false
31880 9270 Acked Read 7 Done Done Sent false
31881 9271 Acked Write 6 Done Done Sent false
31882 9272 Acked Write 2 Done Done Sent false
31883 9273 Acked Read 2 Done Done Sent false
31884 9274 Acked Read 2 Done Done Sent false
31885 9275 Acked Read 4 Done Done Sent false
31886 9276 Acked Write 6 Done Done Sent false
31887 9277 Acked Flush 0 Done Done Sent false
31888 9278 Acked Read 4 Done Done Sent false
31889 9279 Acked Read 2 Done Done Sent false
31890 9280 Acked Write 4 Done Done Sent false
31891 9281 Acked Write 1 Done Done Sent false
31892 9282 Acked Read 8 Done Done Sent false
31893 9283 Acked Write 3 Done Done Sent false
31894 9284 Acked Write 8 Done Done Sent false
31895 9285 Acked Read 6 Done Done Sent false
31896 9286 Acked Write 10 Done Done Sent false
31897 9287 Acked Read 5 Done Done Sent false
31898 9288 Acked Read 2 Done Done Sent false
31899 9289 Acked Flush 0 Done Done Sent false
31900 9290 Acked Read 1 Done Done Sent false
31901 9291 Acked Read 8 Done Done Sent false
31902 9292 Acked Write 7 Done Done Sent false
31903 9293 Acked Write 4 Done Done Sent false
31904 9294 Acked Flush 0 Done Done Sent false
31905 9295 Acked Write 3 Done Done Sent false
31906 9296 Acked Read 2 Done Done Sent false
31907 9297 Acked Flush 0 Done Done Sent false
31908 9298 Acked Read 6 Done Done Sent false
31909 9299 Acked Read 1 Done Done Sent false
31910 9300 Acked Write 4 Done Done Sent false
31911 9301 Acked Write 4 Done Done Sent false
31912 9302 Acked Flush 0 Done Done Sent false
31913 9303 Acked Write 9 Done Done Sent false
31914 9304 Acked Read 3 Done Done Sent false
31915 9305 Acked Write 8 Done Done Sent false
31916 9306 Acked Read 5 Done Done Sent false
31917 9307 Acked Write 2 Done Done Sent false
31918 9308 Acked Read 5 Done Done Sent false
31919 9309 Acked Read 4 Done Done Sent false
31920 9310 Acked Read 6 Done Done Sent false
31921 9311 Acked Read 7 Done Done Sent false
31922 9312 Acked Write 4 Done Done Sent false
31923 9313 Acked Read 7 Done Done Sent false
31924 9314 Acked Write 4 Done Done Sent false
31925 9315 Acked Read 7 Done Done Sent false
31926 9316 Acked Write 9 Done Done Sent false
31927 9317 Acked Read 9 Done Done Sent false
31928 9318 Acked Read 5 Done Done Sent false
31929 9319 Acked Read 6 Done Done Sent false
31930 9320 Acked Read 6 Done Done Sent false
31931 9321 Acked Read 2 Done Done Sent false
31932 9322 Acked Read 9 Done Done Sent false
31933 9323 Acked Read 4 Done Done Sent false
31934 9324 Acked Read 10 Done Done Sent false
31935 9325 Acked Read 6 Done Done Sent false
31936 9326 Acked Read 2 Done Done Sent false
31937 9327 Acked Read 8 Done Done Sent false
31938 9328 Acked Flush 0 Done Done Sent false
31939 9329 Acked Read 6 Done Done Sent false
31940 9330 Acked Read 10 Done Done Sent false
31941 9331 Acked Read 7 Done Done Sent false
31942 9332 Acked Read 7 Done Done Sent false
31943 9333 Acked Read 8 Done Done Sent false
31944 9334 Acked Write 5 Done Done Sent false
31945 9335 Acked Write 1 Done Done Sent false
31946 9336 Acked Write 6 Done Done Sent false
31947 9337 Acked Flush 0 Done Done Sent false
31948 9338 Acked Flush 0 Done Done Sent false
31949 9339 Acked Write 4 Done Done Sent false
31950 9340 Acked Read 9 Done Done Sent false
31951 9341 Acked Read 3 Done Done Sent false
31952 9342 Acked Write 1 Done Done Sent false
31953 9343 Acked Write 1 Done Done Sent false
31954 9344 Acked Write 9 Done Done Sent false
31955 9345 Acked Read 9 Done Done Sent false
31956 9346 Acked Write 7 Done Done Sent false
31957 9347 Acked Read 3 Done Done Sent false
31958 9348 Acked Read 9 Done Done Sent false
31959 9349 Acked Read 10 Done Done Sent false
31960 9350 Acked Read 10 Done Done Sent false
31961 9351 Acked Write 4 Done Done Sent false
31962 9352 Acked Read 2 Done Done Sent false
31963 9353 Acked Read 4 Done Done Sent false
31964 9354 Acked Write 1 Done Done Sent false
31965 9355 Acked Read 1 Done Done Sent false
31966 9356 Acked Write 6 Done Done Sent false
31967 9357 Acked Read 9 Done Done Sent false
31968 9358 Acked Read 8 Done Done Sent false
31969 9359 Acked Read 2 Done Done Sent false
31970 9360 Acked Read 9 Done Done Sent false
31971 9361 Acked Read 8 Done Done Sent false
31972 9362 Acked Flush 0 Done Done Sent false
31973 9363 Acked Write 1 Done Done Sent false
31974 9364 Acked Write 3 Done Done Sent false
31975 9365 Acked Write 3 Done Done Sent false
31976 9366 Acked Write 8 Done Done Sent false
31977 9367 Acked Read 3 Done Done Sent false
31978 9368 Acked Read 6 Done Done Sent false
31979 9369 Acked Flush 0 Done Done Sent false
31980 9370 Acked Read 2 Done Done Sent false
31981 9371 Acked Write 9 Done Done Sent false
31982 9372 Acked Write 5 Done Done Sent false
31983 9373 Acked Read 4 Done Done Sent false
31984 9374 Acked Read 2 Done Done Sent false
31985 9375 Acked Read 2 Done Done Sent false
31986 9376 Acked Read 1 Done Done Sent false
31987 9377 Acked Read 1 Done Done Sent false
31988 9378 Acked Read 8 Done Done Sent false
31989 9379 Acked Write 1 Done Done Sent false
31990 9380 Acked Read 7 Done Done Sent false
31991 9381 Acked Read 4 Done Done Sent false
31992 9382 Acked Read 3 Done Done Sent false
31993 9383 Acked Read 5 Done Done Sent false
31994 9384 Acked Read 2 Done Done Sent false
31995 9385 Acked Read 3 Done Done Sent false
31996 9386 Acked Read 5 Done Done Sent false
31997 9387 Acked Write 5 Done Done Sent false
31998 9388 Acked Write 5 Done Done Sent false
31999 9389 Acked Read 8 Done Done Sent false
32000 9390 Acked Write 3 Done Done Sent false
32001 9391 Acked Read 5 Done Done Sent false
32002 9392 Acked Flush 0 Done Done Sent false
32003 9393 Acked Read 3 Done Done Sent false
32004 9394 Acked Write 2 Done Done Sent false
32005 9395 Acked Read 10 Done Done Sent false
32006 9396 Acked Read 3 Done Done Sent false
32007 9397 Acked Read 3 Done Done Sent false
32008 9398 Acked Read 9 Done Done Sent false
32009 9399 Acked Read 2 Done Done Sent false
32010 9400 Acked Read 6 Done Done Sent false
32011 9401 Acked Read 3 Done Done Sent false
32012 9402 Acked Flush 0 Done Done Sent false
32013 9403 Acked Read 1 Done Done Sent false
32014 9404 Acked Read 2 Done Done Sent false
32015 9405 Acked Write 8 Done Done Sent false
32016 9406 Acked Read 4 Done Done Sent false
32017 9407 Acked Flush 0 Done Done Sent false
32018 9408 Acked Read 2 Done Done Sent false
32019 9409 Acked Read 2 Done Done Sent false
32020 9410 Acked Write 1 Done Done Sent false
32021 9411 Acked Read 3 Done Done Sent false
32022 9412 Acked Read 1 Done Done Sent false
32023 9413 Acked Read 6 Done Done Sent false
32024 9414 Acked Write 4 Done Done Sent false
32025 9415 Acked Write 6 Done Done Sent false
32026 9416 Acked Read 9 Done Done Sent false
32027 9417 Acked Write 2 Done Done Sent false
32028 9418 Acked Write 1 Done Done Sent false
32029 9419 Acked Write 1 Done Done Sent false
32030 9420 Acked Read 6 Done Done Sent false
32031 9421 Acked Flush 0 Done Done Sent false
32032 9422 Acked Write 2 Done Done Sent false
32033 9423 Acked Flush 0 Done Done Sent false
32034 9424 Acked Read 7 Done Done Sent false
32035 9425 Acked Read 1 Done Done Sent false
32036 9426 Acked Read 9 Done Done Sent false
32037 9427 Acked Read 7 Done Done Sent false
32038 9428 Acked Write 1 Done Done Sent false
32039 9429 Acked Write 10 Done Done Sent false
32040 9430 Acked Read 2 Done Done Sent false
32041 9431 Acked Write 3 Done Done Sent false
32042 9432 Acked Write 1 Done Done Sent false
32043 9433 Acked Write 3 Done Done Sent false
32044 9434 Acked Write 8 Done Done Sent false
32045 9435 Acked Read 5 Done Done Sent false
32046 9436 Acked Flush 0 Done Done Sent false
32047 9437 Acked Write 5 Done Done Sent false
32048 9438 Acked Write 3 Done Done Sent false
32049 9439 Acked Write 8 Done Done Sent false
32050 9440 Acked Read 1 Done Done Sent false
32051 9441 Acked Flush 0 Done Done Sent false
32052 9442 Acked Write 7 Done Done Sent false
32053 9443 Acked Read 6 Done Done Sent false
32054 9444 Acked Write 4 Done Done Sent false
32055 9445 Acked Read 9 Done Done Sent false
32056 9446 Acked Read 3 Done Done Sent false
32057 9447 Acked Read 10 Done Done Sent false
32058 9448 Acked Read 4 Done Done Sent false
32059 9449 Acked Write 10 Done Done Sent false
32060 9450 Acked Write 7 Done Done Sent false
32061 STATES DS:0 DS:1 DS:2 TOTAL
32062 Sent 0 0 301 301
32063 Done 301 301 0 602
32064 Skipped 0 0 0 0
32065 Error 0 0 0 0
32066 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
32067 Downstairs last five completed: 9149 9148 9147 9146 9145
32068 Upstairs last five completed: 9450 9449 9448 9447 9446
32069 CLIENT: Up:0 ds:301 act:2
320702025-03-07T18:19:32.642ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320712025-03-07T18:19:33.410ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320722025-03-07T18:19:33.410ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320732025-03-07T18:19:33.410ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320742025-03-07T18:19:33.410ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320752025-03-07T18:19:33.410ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320762025-03-07T18:19:33.911ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
320772025-03-07T18:19:34.645ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32078 ----------------------------------------------------------------
32079 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32080 States: Active Active Offline
32081 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32082 9150 Acked Write 5 Done Done Sent false
32083 9151 Acked Write 2 Done Done Sent false
32084 9152 Acked Write 6 Done Done Sent false
32085 9153 Acked Read 1 Done Done Sent false
32086 9154 Acked Read 4 Done Done Sent false
32087 9155 Acked Write 10 Done Done Sent false
32088 9156 Acked Write 10 Done Done Sent false
32089 9157 Acked Read 2 Done Done Sent false
32090 9158 Acked Read 7 Done Done Sent false
32091 9159 Acked Read 3 Done Done Sent false
32092 9160 Acked Write 2 Done Done Sent false
32093 9161 Acked Read 4 Done Done Sent false
32094 9162 Acked Write 10 Done Done Sent false
32095 9163 Acked Write 4 Done Done Sent false
32096 9164 Acked Read 5 Done Done Sent false
32097 9165 Acked Read 3 Done Done Sent false
32098 9166 Acked Read 3 Done Done Sent false
32099 9167 Acked Flush 0 Done Done Sent false
32100 9168 Acked Write 3 Done Done Sent false
32101 9169 Acked Read 5 Done Done Sent false
32102 9170 Acked Read 9 Done Done Sent false
32103 9171 Acked Read 5 Done Done Sent false
32104 9172 Acked Read 5 Done Done Sent false
32105 9173 Acked Read 1 Done Done Sent false
32106 9174 Acked Read 8 Done Done Sent false
32107 9175 Acked Write 1 Done Done Sent false
32108 9176 Acked Write 8 Done Done Sent false
32109 9177 Acked Flush 0 Done Done Sent false
32110 9178 Acked Write 4 Done Done Sent false
32111 9179 Acked Write 4 Done Done Sent false
32112 9180 Acked Read 4 Done Done Sent false
32113 9181 Acked Write 3 Done Done Sent false
32114 9182 Acked Read 5 Done Done Sent false
32115 9183 Acked Read 4 Done Done Sent false
32116 9184 Acked Read 9 Done Done Sent false
32117 9185 Acked Write 6 Done Done Sent false
32118 9186 Acked Write 7 Done Done Sent false
32119 9187 Acked Read 7 Done Done Sent false
32120 9188 Acked Read 7 Done Done Sent false
32121 9189 Acked Read 9 Done Done Sent false
32122 9190 Acked Read 7 Done Done Sent false
32123 9191 Acked Write 4 Done Done Sent false
32124 9192 Acked Write 5 Done Done Sent false
32125 9193 Acked Write 8 Done Done Sent false
32126 9194 Acked Read 8 Done Done Sent false
32127 9195 Acked Read 9 Done Done Sent false
32128 9196 Acked Read 3 Done Done Sent false
32129 9197 Acked Read 5 Done Done Sent false
32130 9198 Acked Read 9 Done Done Sent false
32131 9199 Acked Write 3 Done Done Sent false
32132 9200 Acked Write 4 Done Done Sent false
32133 9201 Acked Write 9 Done Done Sent false
32134 9202 Acked Read 1 Done Done Sent false
32135 9203 Acked Flush 0 Done Done Sent false
32136 9204 Acked Write 3 Done Done Sent false
32137 9205 Acked Read 5 Done Done Sent false
32138 9206 Acked Write 1 Done Done Sent false
32139 9207 Acked Read 3 Done Done Sent false
32140 9208 Acked Write 2 Done Done Sent false
32141 9209 Acked Read 3 Done Done Sent false
32142 9210 Acked Read 2 Done Done Sent false
32143 9211 Acked Read 3 Done Done Sent false
32144 9212 Acked Read 5 Done Done Sent false
32145 9213 Acked Read 8 Done Done Sent false
32146 9214 Acked Read 4 Done Done Sent false
32147 9215 Acked Read 6 Done Done Sent false
32148 9216 Acked Write 3 Done Done Sent false
32149 9217 Acked Write 6 Done Done Sent false
32150 9218 Acked Write 6 Done Done Sent false
32151 9219 Acked Write 6 Done Done Sent false
32152 9220 Acked Read 6 Done Done Sent false
32153 9221 Acked Read 7 Done Done Sent false
32154 9222 Acked Read 8 Done Done Sent false
32155 9223 Acked Read 6 Done Done Sent false
32156 9224 Acked Write 3 Done Done Sent false
32157 9225 Acked Read 1 Done Done Sent false
32158 9226 Acked Read 2 Done Done Sent false
32159 9227 Acked Write 4 Done Done Sent false
32160 9228 Acked Write 3 Done Done Sent false
32161 9229 Acked Flush 0 Done Done Sent false
32162 9230 Acked Read 1 Done Done Sent false
32163 9231 Acked Read 7 Done Done Sent false
32164 9232 Acked Flush 0 Done Done Sent false
32165 9233 Acked Read 4 Done Done Sent false
32166 9234 Acked Read 1 Done Done Sent false
32167 9235 Acked Read 2 Done Done Sent false
32168 9236 Acked Read 10 Done Done Sent false
32169 9237 Acked Read 4 Done Done Sent false
32170 9238 Acked Read 1 Done Done Sent false
32171 9239 Acked Read 7 Done Done Sent false
32172 9240 Acked Read 1 Done Done Sent false
32173 9241 Acked Flush 0 Done Done Sent false
32174 9242 Acked Read 7 Done Done Sent false
32175 9243 Acked Read 1 Done Done Sent false
32176 9244 Acked Read 4 Done Done Sent false
32177 9245 Acked Read 6 Done Done Sent false
32178 9246 Acked Read 10 Done Done Sent false
32179 9247 Acked Flush 0 Done Done Sent false
32180 9248 Acked Write 6 Done Done Sent false
32181 9249 Acked Read 5 Done Done Sent false
32182 9250 Acked Read 3 Done Done Sent false
32183 9251 Acked Flush 0 Done Done Sent false
32184 9252 Acked Read 7 Done Done Sent false
32185 9253 Acked Write 8 Done Done Sent false
32186 9254 Acked Read 10 Done Done Sent false
32187 9255 Acked Write 7 Done Done Sent false
32188 9256 Acked Write 9 Done Done Sent false
32189 9257 Acked Read 1 Done Done Sent false
32190 9258 Acked Read 7 Done Done Sent false
32191 9259 Acked Read 1 Done Done Sent false
32192 9260 Acked Write 5 Done Done Sent false
32193 9261 Acked Read 1 Done Done Sent false
32194 9262 Acked Read 1 Done Done Sent false
32195 9263 Acked Read 5 Done Done Sent false
32196 9264 Acked Write 2 Done Done Sent false
32197 9265 Acked Write 3 Done Done Sent false
32198 9266 Acked Write 1 Done Done Sent false
32199 9267 Acked Write 5 Done Done Sent false
32200 9268 Acked Read 10 Done Done Sent false
32201 9269 Acked Read 3 Done Done Sent false
32202 9270 Acked Read 7 Done Done Sent false
32203 9271 Acked Write 6 Done Done Sent false
32204 9272 Acked Write 2 Done Done Sent false
32205 9273 Acked Read 2 Done Done Sent false
32206 9274 Acked Read 2 Done Done Sent false
32207 9275 Acked Read 4 Done Done Sent false
32208 9276 Acked Write 6 Done Done Sent false
32209 9277 Acked Flush 0 Done Done Sent false
32210 9278 Acked Read 4 Done Done Sent false
32211 9279 Acked Read 2 Done Done Sent false
32212 9280 Acked Write 4 Done Done Sent false
32213 9281 Acked Write 1 Done Done Sent false
32214 9282 Acked Read 8 Done Done Sent false
32215 9283 Acked Write 3 Done Done Sent false
32216 9284 Acked Write 8 Done Done Sent false
32217 9285 Acked Read 6 Done Done Sent false
32218 9286 Acked Write 10 Done Done Sent false
32219 9287 Acked Read 5 Done Done Sent false
32220 9288 Acked Read 2 Done Done Sent false
32221 9289 Acked Flush 0 Done Done Sent false
32222 9290 Acked Read 1 Done Done Sent false
32223 9291 Acked Read 8 Done Done Sent false
32224 9292 Acked Write 7 Done Done Sent false
32225 9293 Acked Write 4 Done Done Sent false
32226 9294 Acked Flush 0 Done Done Sent false
32227 9295 Acked Write 3 Done Done Sent false
32228 9296 Acked Read 2 Done Done Sent false
32229 9297 Acked Flush 0 Done Done Sent false
32230 9298 Acked Read 6 Done Done Sent false
32231 9299 Acked Read 1 Done Done Sent false
32232 9300 Acked Write 4 Done Done Sent false
32233 9301 Acked Write 4 Done Done Sent false
32234 9302 Acked Flush 0 Done Done Sent false
32235 9303 Acked Write 9 Done Done Sent false
32236 9304 Acked Read 3 Done Done Sent false
32237 9305 Acked Write 8 Done Done Sent false
32238 9306 Acked Read 5 Done Done Sent false
32239 9307 Acked Write 2 Done Done Sent false
32240 9308 Acked Read 5 Done Done Sent false
32241 9309 Acked Read 4 Done Done Sent false
32242 9310 Acked Read 6 Done Done Sent false
32243 9311 Acked Read 7 Done Done Sent false
32244 9312 Acked Write 4 Done Done Sent false
32245 9313 Acked Read 7 Done Done Sent false
32246 9314 Acked Write 4 Done Done Sent false
32247 9315 Acked Read 7 Done Done Sent false
32248 9316 Acked Write 9 Done Done Sent false
32249 9317 Acked Read 9 Done Done Sent false
32250 9318 Acked Read 5 Done Done Sent false
32251 9319 Acked Read 6 Done Done Sent false
32252 9320 Acked Read 6 Done Done Sent false
32253 9321 Acked Read 2 Done Done Sent false
32254 9322 Acked Read 9 Done Done Sent false
32255 9323 Acked Read 4 Done Done Sent false
32256 9324 Acked Read 10 Done Done Sent false
32257 9325 Acked Read 6 Done Done Sent false
32258 9326 Acked Read 2 Done Done Sent false
32259 9327 Acked Read 8 Done Done Sent false
32260 9328 Acked Flush 0 Done Done Sent false
32261 9329 Acked Read 6 Done Done Sent false
32262 9330 Acked Read 10 Done Done Sent false
32263 9331 Acked Read 7 Done Done Sent false
32264 9332 Acked Read 7 Done Done Sent false
32265 9333 Acked Read 8 Done Done Sent false
32266 9334 Acked Write 5 Done Done Sent false
32267 9335 Acked Write 1 Done Done Sent false
32268 9336 Acked Write 6 Done Done Sent false
32269 9337 Acked Flush 0 Done Done Sent false
32270 9338 Acked Flush 0 Done Done Sent false
32271 9339 Acked Write 4 Done Done Sent false
32272 9340 Acked Read 9 Done Done Sent false
32273 9341 Acked Read 3 Done Done Sent false
32274 9342 Acked Write 1 Done Done Sent false
32275 9343 Acked Write 1 Done Done Sent false
32276 9344 Acked Write 9 Done Done Sent false
32277 9345 Acked Read 9 Done Done Sent false
32278 9346 Acked Write 7 Done Done Sent false
32279 9347 Acked Read 3 Done Done Sent false
32280 9348 Acked Read 9 Done Done Sent false
32281 9349 Acked Read 10 Done Done Sent false
32282 9350 Acked Read 10 Done Done Sent false
32283 9351 Acked Write 4 Done Done Sent false
32284 9352 Acked Read 2 Done Done Sent false
32285 9353 Acked Read 4 Done Done Sent false
32286 9354 Acked Write 1 Done Done Sent false
32287 9355 Acked Read 1 Done Done Sent false
32288 9356 Acked Write 6 Done Done Sent false
32289 9357 Acked Read 9 Done Done Sent false
32290 9358 Acked Read 8 Done Done Sent false
32291 9359 Acked Read 2 Done Done Sent false
32292 9360 Acked Read 9 Done Done Sent false
32293 9361 Acked Read 8 Done Done Sent false
32294 9362 Acked Flush 0 Done Done Sent false
32295 9363 Acked Write 1 Done Done Sent false
32296 9364 Acked Write 3 Done Done Sent false
32297 9365 Acked Write 3 Done Done Sent false
32298 9366 Acked Write 8 Done Done Sent false
32299 9367 Acked Read 3 Done Done Sent false
32300 9368 Acked Read 6 Done Done Sent false
32301 9369 Acked Flush 0 Done Done Sent false
32302 9370 Acked Read 2 Done Done Sent false
32303 9371 Acked Write 9 Done Done Sent false
32304 9372 Acked Write 5 Done Done Sent false
32305 9373 Acked Read 4 Done Done Sent false
32306 9374 Acked Read 2 Done Done Sent false
32307 9375 Acked Read 2 Done Done Sent false
32308 9376 Acked Read 1 Done Done Sent false
32309 9377 Acked Read 1 Done Done Sent false
32310 9378 Acked Read 8 Done Done Sent false
32311 9379 Acked Write 1 Done Done Sent false
32312 9380 Acked Read 7 Done Done Sent false
32313 9381 Acked Read 4 Done Done Sent false
32314 9382 Acked Read 3 Done Done Sent false
32315 9383 Acked Read 5 Done Done Sent false
32316 9384 Acked Read 2 Done Done Sent false
32317 9385 Acked Read 3 Done Done Sent false
32318 9386 Acked Read 5 Done Done Sent false
32319 9387 Acked Write 5 Done Done Sent false
32320 9388 Acked Write 5 Done Done Sent false
32321 9389 Acked Read 8 Done Done Sent false
32322 9390 Acked Write 3 Done Done Sent false
32323 9391 Acked Read 5 Done Done Sent false
32324 9392 Acked Flush 0 Done Done Sent false
32325 9393 Acked Read 3 Done Done Sent false
32326 9394 Acked Write 2 Done Done Sent false
32327 9395 Acked Read 10 Done Done Sent false
32328 9396 Acked Read 3 Done Done Sent false
32329 9397 Acked Read 3 Done Done Sent false
32330 9398 Acked Read 9 Done Done Sent false
32331 9399 Acked Read 2 Done Done Sent false
32332 9400 Acked Read 6 Done Done Sent false
32333 9401 Acked Read 3 Done Done Sent false
32334 9402 Acked Flush 0 Done Done Sent false
32335 9403 Acked Read 1 Done Done Sent false
32336 9404 Acked Read 2 Done Done Sent false
32337 9405 Acked Write 8 Done Done Sent false
32338 9406 Acked Read 4 Done Done Sent false
32339 9407 Acked Flush 0 Done Done Sent false
32340 9408 Acked Read 2 Done Done Sent false
32341 9409 Acked Read 2 Done Done Sent false
32342 9410 Acked Write 1 Done Done Sent false
32343 9411 Acked Read 3 Done Done Sent false
32344 9412 Acked Read 1 Done Done Sent false
32345 9413 Acked Read 6 Done Done Sent false
32346 9414 Acked Write 4 Done Done Sent false
32347 9415 Acked Write 6 Done Done Sent false
32348 9416 Acked Read 9 Done Done Sent false
32349 9417 Acked Write 2 Done Done Sent false
32350 9418 Acked Write 1 Done Done Sent false
32351 9419 Acked Write 1 Done Done Sent false
32352 9420 Acked Read 6 Done Done Sent false
32353 9421 Acked Flush 0 Done Done Sent false
32354 9422 Acked Write 2 Done Done Sent false
32355 9423 Acked Flush 0 Done Done Sent false
32356 9424 Acked Read 7 Done Done Sent false
32357 9425 Acked Read 1 Done Done Sent false
32358 9426 Acked Read 9 Done Done Sent false
32359 9427 Acked Read 7 Done Done Sent false
32360 9428 Acked Write 1 Done Done Sent false
32361 9429 Acked Write 10 Done Done Sent false
32362 9430 Acked Read 2 Done Done Sent false
32363 9431 Acked Write 3 Done Done Sent false
32364 9432 Acked Write 1 Done Done Sent false
32365 9433 Acked Write 3 Done Done Sent false
32366 9434 Acked Write 8 Done Done Sent false
32367 9435 Acked Read 5 Done Done Sent false
32368 9436 Acked Flush 0 Done Done Sent false
32369 9437 Acked Write 5 Done Done Sent false
32370 9438 Acked Write 3 Done Done Sent false
32371 9439 Acked Write 8 Done Done Sent false
32372 9440 Acked Read 1 Done Done Sent false
32373 9441 Acked Flush 0 Done Done Sent false
32374 9442 Acked Write 7 Done Done Sent false
32375 9443 Acked Read 6 Done Done Sent false
32376 9444 Acked Write 4 Done Done Sent false
32377 9445 Acked Read 9 Done Done Sent false
32378 9446 Acked Read 3 Done Done Sent false
32379 9447 Acked Read 10 Done Done Sent false
32380 9448 Acked Read 4 Done Done Sent false
32381 9449 Acked Write 10 Done Done Sent false
32382 9450 Acked Write 7 Done Done Sent false
32383 STATES DS:0 DS:1 DS:2 TOTAL
32384 Sent 0 0 301 301
32385 Done 301 301 0 602
32386 Skipped 0 0 0 0
32387 Error 0 0 0 0
32388 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
32389 Downstairs last five completed: 9149 9148 9147 9146 9145
32390 Upstairs last five completed: 9450 9449 9448 9447 9446
32391 CLIENT: Up:0 ds:301 act:2
323922025-03-07T18:19:36.648ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
323932025-03-07T18:19:37.651ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
323942025-03-07T18:19:38.652ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
323952025-03-07T18:19:39.653ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32396 ----------------------------------------------------------------
32397 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32398 States: Active Active Offline
32399 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32400 9150 Acked Write 5 Done Done Sent false
32401 9151 Acked Write 2 Done Done Sent false
32402 9152 Acked Write 6 Done Done Sent false
32403 9153 Acked Read 1 Done Done Sent false
32404 9154 Acked Read 4 Done Done Sent false
32405 9155 Acked Write 10 Done Done Sent false
32406 9156 Acked Write 10 Done Done Sent false
32407 9157 Acked Read 2 Done Done Sent false
32408 9158 Acked Read 7 Done Done Sent false
32409 9159 Acked Read 3 Done Done Sent false
32410 9160 Acked Write 2 Done Done Sent false
32411 9161 Acked Read 4 Done Done Sent false
32412 9162 Acked Write 10 Done Done Sent false
32413 9163 Acked Write 4 Done Done Sent false
32414 9164 Acked Read 5 Done Done Sent false
32415 9165 Acked Read 3 Done Done Sent false
32416 9166 Acked Read 3 Done Done Sent false
32417 9167 Acked Flush 0 Done Done Sent false
32418 9168 Acked Write 3 Done Done Sent false
32419 9169 Acked Read 5 Done Done Sent false
32420 9170 Acked Read 9 Done Done Sent false
32421 9171 Acked Read 5 Done Done Sent false
32422 9172 Acked Read 5 Done Done Sent false
32423 9173 Acked Read 1 Done Done Sent false
32424 9174 Acked Read 8 Done Done Sent false
32425 9175 Acked Write 1 Done Done Sent false
32426 9176 Acked Write 8 Done Done Sent false
32427 9177 Acked Flush 0 Done Done Sent false
32428 9178 Acked Write 4 Done Done Sent false
32429 9179 Acked Write 4 Done Done Sent false
32430 9180 Acked Read 4 Done Done Sent false
32431 9181 Acked Write 3 Done Done Sent false
32432 9182 Acked Read 5 Done Done Sent false
32433 9183 Acked Read 4 Done Done Sent false
32434 9184 Acked Read 9 Done Done Sent false
32435 9185 Acked Write 6 Done Done Sent false
32436 9186 Acked Write 7 Done Done Sent false
32437 9187 Acked Read 7 Done Done Sent false
32438 9188 Acked Read 7 Done Done Sent false
32439 9189 Acked Read 9 Done Done Sent false
32440 9190 Acked Read 7 Done Done Sent false
32441 9191 Acked Write 4 Done Done Sent false
32442 9192 Acked Write 5 Done Done Sent false
32443 9193 Acked Write 8 Done Done Sent false
32444 9194 Acked Read 8 Done Done Sent false
32445 9195 Acked Read 9 Done Done Sent false
32446 9196 Acked Read 3 Done Done Sent false
32447 9197 Acked Read 5 Done Done Sent false
32448 9198 Acked Read 9 Done Done Sent false
32449 9199 Acked Write 3 Done Done Sent false
32450 9200 Acked Write 4 Done Done Sent false
32451 9201 Acked Write 9 Done Done Sent false
32452 9202 Acked Read 1 Done Done Sent false
32453 9203 Acked Flush 0 Done Done Sent false
32454 9204 Acked Write 3 Done Done Sent false
32455 9205 Acked Read 5 Done Done Sent false
32456 9206 Acked Write 1 Done Done Sent false
32457 9207 Acked Read 3 Done Done Sent false
32458 9208 Acked Write 2 Done Done Sent false
32459 9209 Acked Read 3 Done Done Sent false
32460 9210 Acked Read 2 Done Done Sent false
32461 9211 Acked Read 3 Done Done Sent false
32462 9212 Acked Read 5 Done Done Sent false
32463 9213 Acked Read 8 Done Done Sent false
32464 9214 Acked Read 4 Done Done Sent false
32465 9215 Acked Read 6 Done Done Sent false
32466 9216 Acked Write 3 Done Done Sent false
32467 9217 Acked Write 6 Done Done Sent false
32468 9218 Acked Write 6 Done Done Sent false
32469 9219 Acked Write 6 Done Done Sent false
32470 9220 Acked Read 6 Done Done Sent false
32471 9221 Acked Read 7 Done Done Sent false
32472 9222 Acked Read 8 Done Done Sent false
32473 9223 Acked Read 6 Done Done Sent false
32474 9224 Acked Write 3 Done Done Sent false
32475 9225 Acked Read 1 Done Done Sent false
32476 9226 Acked Read 2 Done Done Sent false
32477 9227 Acked Write 4 Done Done Sent false
32478 9228 Acked Write 3 Done Done Sent false
32479 9229 Acked Flush 0 Done Done Sent false
32480 9230 Acked Read 1 Done Done Sent false
32481 9231 Acked Read 7 Done Done Sent false
32482 9232 Acked Flush 0 Done Done Sent false
32483 9233 Acked Read 4 Done Done Sent false
32484 9234 Acked Read 1 Done Done Sent false
32485 9235 Acked Read 2 Done Done Sent false
32486 9236 Acked Read 10 Done Done Sent false
32487 9237 Acked Read 4 Done Done Sent false
32488 9238 Acked Read 1 Done Done Sent false
32489 9239 Acked Read 7 Done Done Sent false
32490 9240 Acked Read 1 Done Done Sent false
32491 9241 Acked Flush 0 Done Done Sent false
32492 9242 Acked Read 7 Done Done Sent false
32493 9243 Acked Read 1 Done Done Sent false
32494 9244 Acked Read 4 Done Done Sent false
32495 9245 Acked Read 6 Done Done Sent false
32496 9246 Acked Read 10 Done Done Sent false
32497 9247 Acked Flush 0 Done Done Sent false
32498 9248 Acked Write 6 Done Done Sent false
32499 9249 Acked Read 5 Done Done Sent false
32500 9250 Acked Read 3 Done Done Sent false
32501 9251 Acked Flush 0 Done Done Sent false
32502 9252 Acked Read 7 Done Done Sent false
32503 9253 Acked Write 8 Done Done Sent false
32504 9254 Acked Read 10 Done Done Sent false
32505 9255 Acked Write 7 Done Done Sent false
32506 9256 Acked Write 9 Done Done Sent false
32507 9257 Acked Read 1 Done Done Sent false
32508 9258 Acked Read 7 Done Done Sent false
32509 9259 Acked Read 1 Done Done Sent false
32510 9260 Acked Write 5 Done Done Sent false
32511 9261 Acked Read 1 Done Done Sent false
32512 9262 Acked Read 1 Done Done Sent false
32513 9263 Acked Read 5 Done Done Sent false
32514 9264 Acked Write 2 Done Done Sent false
32515 9265 Acked Write 3 Done Done Sent false
32516 9266 Acked Write 1 Done Done Sent false
32517 9267 Acked Write 5 Done Done Sent false
32518 9268 Acked Read 10 Done Done Sent false
32519 9269 Acked Read 3 Done Done Sent false
32520 9270 Acked Read 7 Done Done Sent false
32521 9271 Acked Write 6 Done Done Sent false
32522 9272 Acked Write 2 Done Done Sent false
32523 9273 Acked Read 2 Done Done Sent false
32524 9274 Acked Read 2 Done Done Sent false
32525 9275 Acked Read 4 Done Done Sent false
32526 9276 Acked Write 6 Done Done Sent false
32527 9277 Acked Flush 0 Done Done Sent false
32528 9278 Acked Read 4 Done Done Sent false
32529 9279 Acked Read 2 Done Done Sent false
32530 9280 Acked Write 4 Done Done Sent false
32531 9281 Acked Write 1 Done Done Sent false
32532 9282 Acked Read 8 Done Done Sent false
32533 9283 Acked Write 3 Done Done Sent false
32534 9284 Acked Write 8 Done Done Sent false
32535 9285 Acked Read 6 Done Done Sent false
32536 9286 Acked Write 10 Done Done Sent false
32537 9287 Acked Read 5 Done Done Sent false
32538 9288 Acked Read 2 Done Done Sent false
32539 9289 Acked Flush 0 Done Done Sent false
32540 9290 Acked Read 1 Done Done Sent false
32541 9291 Acked Read 8 Done Done Sent false
32542 9292 Acked Write 7 Done Done Sent false
32543 9293 Acked Write 4 Done Done Sent false
32544 9294 Acked Flush 0 Done Done Sent false
32545 9295 Acked Write 3 Done Done Sent false
32546 9296 Acked Read 2 Done Done Sent false
32547 9297 Acked Flush 0 Done Done Sent false
32548 9298 Acked Read 6 Done Done Sent false
32549 9299 Acked Read 1 Done Done Sent false
32550 9300 Acked Write 4 Done Done Sent false
32551 9301 Acked Write 4 Done Done Sent false
32552 9302 Acked Flush 0 Done Done Sent false
32553 9303 Acked Write 9 Done Done Sent false
32554 9304 Acked Read 3 Done Done Sent false
32555 9305 Acked Write 8 Done Done Sent false
32556 9306 Acked Read 5 Done Done Sent false
32557 9307 Acked Write 2 Done Done Sent false
32558 9308 Acked Read 5 Done Done Sent false
32559 9309 Acked Read 4 Done Done Sent false
32560 9310 Acked Read 6 Done Done Sent false
32561 9311 Acked Read 7 Done Done Sent false
32562 9312 Acked Write 4 Done Done Sent false
32563 9313 Acked Read 7 Done Done Sent false
32564 9314 Acked Write 4 Done Done Sent false
32565 9315 Acked Read 7 Done Done Sent false
32566 9316 Acked Write 9 Done Done Sent false
32567 9317 Acked Read 9 Done Done Sent false
32568 9318 Acked Read 5 Done Done Sent false
32569 9319 Acked Read 6 Done Done Sent false
32570 9320 Acked Read 6 Done Done Sent false
32571 9321 Acked Read 2 Done Done Sent false
32572 9322 Acked Read 9 Done Done Sent false
32573 9323 Acked Read 4 Done Done Sent false
32574 9324 Acked Read 10 Done Done Sent false
32575 9325 Acked Read 6 Done Done Sent false
32576 9326 Acked Read 2 Done Done Sent false
32577 9327 Acked Read 8 Done Done Sent false
32578 9328 Acked Flush 0 Done Done Sent false
32579 9329 Acked Read 6 Done Done Sent false
32580 9330 Acked Read 10 Done Done Sent false
32581 9331 Acked Read 7 Done Done Sent false
32582 9332 Acked Read 7 Done Done Sent false
32583 9333 Acked Read 8 Done Done Sent false
32584 9334 Acked Write 5 Done Done Sent false
32585 9335 Acked Write 1 Done Done Sent false
32586 9336 Acked Write 6 Done Done Sent false
32587 9337 Acked Flush 0 Done Done Sent false
32588 9338 Acked Flush 0 Done Done Sent false
32589 9339 Acked Write 4 Done Done Sent false
32590 9340 Acked Read 9 Done Done Sent false
32591 9341 Acked Read 3 Done Done Sent false
32592 9342 Acked Write 1 Done Done Sent false
32593 9343 Acked Write 1 Done Done Sent false
32594 9344 Acked Write 9 Done Done Sent false
32595 9345 Acked Read 9 Done Done Sent false
32596 9346 Acked Write 7 Done Done Sent false
32597 9347 Acked Read 3 Done Done Sent false
32598 9348 Acked Read 9 Done Done Sent false
32599 9349 Acked Read 10 Done Done Sent false
32600 9350 Acked Read 10 Done Done Sent false
32601 9351 Acked Write 4 Done Done Sent false
32602 9352 Acked Read 2 Done Done Sent false
32603 9353 Acked Read 4 Done Done Sent false
32604 9354 Acked Write 1 Done Done Sent false
32605 9355 Acked Read 1 Done Done Sent false
32606 9356 Acked Write 6 Done Done Sent false
32607 9357 Acked Read 9 Done Done Sent false
32608 9358 Acked Read 8 Done Done Sent false
32609 9359 Acked Read 2 Done Done Sent false
32610 9360 Acked Read 9 Done Done Sent false
32611 9361 Acked Read 8 Done Done Sent false
32612 9362 Acked Flush 0 Done Done Sent false
32613 9363 Acked Write 1 Done Done Sent false
32614 9364 Acked Write 3 Done Done Sent false
32615 9365 Acked Write 3 Done Done Sent false
32616 9366 Acked Write 8 Done Done Sent false
32617 9367 Acked Read 3 Done Done Sent false
32618 9368 Acked Read 6 Done Done Sent false
32619 9369 Acked Flush 0 Done Done Sent false
32620 9370 Acked Read 2 Done Done Sent false
32621 9371 Acked Write 9 Done Done Sent false
32622 9372 Acked Write 5 Done Done Sent false
32623 9373 Acked Read 4 Done Done Sent false
32624 9374 Acked Read 2 Done Done Sent false
32625 9375 Acked Read 2 Done Done Sent false
32626 9376 Acked Read 1 Done Done Sent false
32627 9377 Acked Read 1 Done Done Sent false
32628 9378 Acked Read 8 Done Done Sent false
32629 9379 Acked Write 1 Done Done Sent false
32630 9380 Acked Read 7 Done Done Sent false
32631 9381 Acked Read 4 Done Done Sent false
32632 9382 Acked Read 3 Done Done Sent false
32633 9383 Acked Read 5 Done Done Sent false
32634 9384 Acked Read 2 Done Done Sent false
32635 9385 Acked Read 3 Done Done Sent false
32636 9386 Acked Read 5 Done Done Sent false
32637 9387 Acked Write 5 Done Done Sent false
32638 9388 Acked Write 5 Done Done Sent false
32639 9389 Acked Read 8 Done Done Sent false
32640 9390 Acked Write 3 Done Done Sent false
32641 9391 Acked Read 5 Done Done Sent false
32642 9392 Acked Flush 0 Done Done Sent false
32643 9393 Acked Read 3 Done Done Sent false
32644 9394 Acked Write 2 Done Done Sent false
32645 9395 Acked Read 10 Done Done Sent false
32646 9396 Acked Read 3 Done Done Sent false
32647 9397 Acked Read 3 Done Done Sent false
32648 9398 Acked Read 9 Done Done Sent false
32649 9399 Acked Read 2 Done Done Sent false
32650 9400 Acked Read 6 Done Done Sent false
32651 9401 Acked Read 3 Done Done Sent false
32652 9402 Acked Flush 0 Done Done Sent false
32653 9403 Acked Read 1 Done Done Sent false
32654 9404 Acked Read 2 Done Done Sent false
32655 9405 Acked Write 8 Done Done Sent false
32656 9406 Acked Read 4 Done Done Sent false
32657 9407 Acked Flush 0 Done Done Sent false
32658 9408 Acked Read 2 Done Done Sent false
32659 9409 Acked Read 2 Done Done Sent false
32660 9410 Acked Write 1 Done Done Sent false
32661 9411 Acked Read 3 Done Done Sent false
32662 9412 Acked Read 1 Done Done Sent false
32663 9413 Acked Read 6 Done Done Sent false
32664 9414 Acked Write 4 Done Done Sent false
32665 9415 Acked Write 6 Done Done Sent false
32666 9416 Acked Read 9 Done Done Sent false
32667 9417 Acked Write 2 Done Done Sent false
32668 9418 Acked Write 1 Done Done Sent false
32669 9419 Acked Write 1 Done Done Sent false
32670 9420 Acked Read 6 Done Done Sent false
32671 9421 Acked Flush 0 Done Done Sent false
32672 9422 Acked Write 2 Done Done Sent false
32673 9423 Acked Flush 0 Done Done Sent false
32674 9424 Acked Read 7 Done Done Sent false
32675 9425 Acked Read 1 Done Done Sent false
32676 9426 Acked Read 9 Done Done Sent false
32677 9427 Acked Read 7 Done Done Sent false
32678 9428 Acked Write 1 Done Done Sent false
32679 9429 Acked Write 10 Done Done Sent false
32680 9430 Acked Read 2 Done Done Sent false
32681 9431 Acked Write 3 Done Done Sent false
32682 9432 Acked Write 1 Done Done Sent false
32683 9433 Acked Write 3 Done Done Sent false
32684 9434 Acked Write 8 Done Done Sent false
32685 9435 Acked Read 5 Done Done Sent false
32686 9436 Acked Flush 0 Done Done Sent false
32687 9437 Acked Write 5 Done Done Sent false
32688 9438 Acked Write 3 Done Done Sent false
32689 9439 Acked Write 8 Done Done Sent false
32690 9440 Acked Read 1 Done Done Sent false
32691 9441 Acked Flush 0 Done Done Sent false
32692 9442 Acked Write 7 Done Done Sent false
32693 9443 Acked Read 6 Done Done Sent false
32694 9444 Acked Write 4 Done Done Sent false
32695 9445 Acked Read 9 Done Done Sent false
32696 9446 Acked Read 3 Done Done Sent false
32697 9447 Acked Read 10 Done Done Sent false
32698 9448 Acked Read 4 Done Done Sent false
32699 9449 Acked Write 10 Done Done Sent false
32700 9450 Acked Write 7 Done Done Sent false
32701 STATES DS:0 DS:1 DS:2 TOTAL
32702 Sent 0 0 301 301
32703 Done 301 301 0 602
32704 Skipped 0 0 0 0
32705 Error 0 0 0 0
32706 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9149))
32707 Downstairs last five completed: 9149 9148 9147 9146 9145
32708 Upstairs last five completed: 9450 9449 9448 9447 9446
32709 CLIENT: Up:0 ds:301 act:2
327102025-03-07T18:19:40.663ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32711 [8830] got start action
32712 [8830] I am going to restart
32713 Starting downstairs at port 8830
32714 Make output file at "/tmp/dsc/downstairs-8830.txt"
32715 [8830][2] reports Starting
32716 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1236)
32717 [8830][2] reports Running
327182025-03-07T18:19:41.657ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327192025-03-07T18:19:42.658ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327202025-03-07T18:19:43.411ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327212025-03-07T18:19:43.411ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51677) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327222025-03-07T18:19:43.412ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327232025-03-07T18:19:43.413ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327242025-03-07T18:19:43.413ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327252025-03-07T18:19:43.413ZINFOcrucible: send last flush ID to this DS: Some(JobId(9149)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327262025-03-07T18:19:43.413ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327272025-03-07T18:19:43.413ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9149)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327282025-03-07T18:19:43.413ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9149)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32729 ----------------------------------------------------------------
32730 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
32731 States: Active Active Active
32732 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32733 9442 Acked Write 7 Done Done Done true
32734 9443 Acked Read 6 Done Done Done true
32735 9444 Acked Write 4 Done Done Done true
32736 9445 Acked Read 9 Done Done Done true
32737 9446 Acked Read 3 Done Done Done true
32738 9447 Acked Read 10 Done Done Done true
32739 9448 Acked Read 4 Done Done Done true
32740 9449 Acked Write 10 Done Done Done true
32741 9450 Acked Write 7 Done Done Done true
32742 STATES DS:0 DS:1 DS:2 TOTAL
32743 Sent 0 0 0 0
32744 Done 9 9 9 27
32745 Skipped 0 0 0 0
32746 Error 0 0 0 0
32747 Last Flush: Some(JobId(9441)) Some(JobId(9441)) Some(JobId(9441))
32748 Downstairs last five completed: 9441 9440 9439 9438 9437
32749 Upstairs last five completed: 9450 9449 9448 9447 9446
32750 CLIENT: Up:0 ds:9 act:3
32751 ----------------------------------------------------------------
32752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32753 Downstairs last five completed: 9451 9450 9449 9448 9447
32754 Upstairs last five completed: 9451 9450 9449 9448 9447
32755 CLIENT: Up:0 ds:0 act:3
32756 Replay: All jobs finished
32757 Main task has work to do, go find it
32758 got dsc Stop(2)
32759 stop 2
32760 [8830] Got stop action so:false kr:false
32761 Mar 07 18:19:47.667 INFO request completed, latency_us: 370, response_code: 204, uri: /stop/cid/2, method: POST, req_id: b44fa6ee-b985-4a87-a584-2df853f36acb, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
327622025-03-07T18:19:47.668ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
327632025-03-07T18:19:47.668ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32764 Mar 07 18:19:47.668 INFO {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-07T18:19:47.66826791Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
32765 request completed, latency_us: 114, response_code: 200, uri: /state/cid/2, method: GET, req_id: e8c76318-b3d0-418b-9594-b6b9aa648fc6, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
327662025-03-07T18:19:47.668ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
32767 [8830] Exited with: ExitStatus(unix_wait_status(9))
32768 [8830][2] reports Exit
32769 Mar 07 18:19:51.670 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: c47fc60b-a470-4f7e-9140-1ed312148fcc, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
32770 001/300 Write block 523 len 20480 data: 4 4 4 5 5
32771 002/300 Write block 888 len 12288 data: 4 5 6
32772 003/300 Read block 3461 len 16384
32773 004/300 Write block 4017 len 4096 data: 6
32774 005/300 Write block 1199 len 28672 data: 5 4 6 5 5 6 7
32775 006/300 Read block 1377 len 40960
32776 007/300 Read block 482 len 24576
32777 008/300 Read block 10 len 12288
32778 009/300 Read block 6 len 24576
32779 010/300 Write block 872 len 8192 data: 4 4
32780 011/300 Read block 1935 len 20480
32781 012/300 Write block 2805 len 16384 data: 7 7 8 8
32782 013/300 Write block 1778 len 40960 data: 7 6 5 4 5 5 4 3 4 3
32783 014/300 Read block 507 len 32768
32784 015/300 Read block 1397 len 20480
32785 016/300 Read block 4654 len 36864
32786 017/300 Read block 3465 len 36864
32787 018/300 Write block 2484 len 28672 data: 5 5 5 7 7 7 7
32788 019/300 Read block 4773 len 4096
32789 020/300 Read block 3475 len 36864
32790 021/300 Read block 966 len 36864
32791 022/300 Read block 64 len 24576
32792 023/300 Read block 1170 len 20480
32793 024/300 Write block 1276 len 40960 data: 5 5 5 5 5 4 3 3 4 4
32794 025/300 Write block 927 len 12288 data: 4 4 4
32795 026/300 Write block 4371 len 16384 data: 8 6 7 5
32796 027/300 Read block 2509 len 40960
32797 028/300 Read block 4613 len 32768
32798 029/300 Read block 4100 len 40960
32799 030/300 Read block 47 len 28672
32800 031/300 Write block 521 len 12288 data: 3 3 5
32801 032/300 Read block 3042 len 36864
32802 033/300 Read block 682 len 36864
32803 034/300 Read block 3970 len 24576
32804 035/300 Flush
32805 036/300 Read block 4387 len 20480
32806 037/300 Write block 2870 len 16384 data: 6 4 4 3
32807 038/300 Write block 3634 len 32768 data: 6 6 6 6 8 7 7 6
32808 039/300 Read block 2544 len 8192
32809 040/300 Flush
32810 041/300 Flush
32811 042/300 Flush
32812 043/300 Read block 4277 len 32768
32813 044/300 Flush
32814 045/300 Write block 3361 len 32768 data: 6 7 9 5 5 5 4 7
32815 046/300 Read block 1922 len 32768
32816 047/300 Read block 2177 len 28672
32817 048/300 Write block 479 len 24576 data: 3 3 3 3 4 4
32818 049/300 Write block 2877 len 20480 data: 4 5 5 6 6
32819 050/300 Read block 1367 len 32768
32820 051/300 Write block 3601 len 28672 data: 5 4 6 6 5 5 5
32821 052/300 Read block 4538 len 36864
32822 053/300 Read block 1878 len 40960
32823 054/300 Write block 113 len 8192 data: 6 5
32824 055/300 Read block 237 len 32768
32825 056/300 Read block 2958 len 20480
32826 057/300 Write block 4921 len 4096 data: 7
32827 058/300 Read block 3753 len 28672
32828 059/300 Write block 3865 len 40960 data: 4 3 3 4 5 5 6 4 6 7
32829 060/300 Read block 3767 len 40960
32830 061/300 Write block 733 len 4096 data: 8
32831 062/300 Read block 4069 len 32768
32832 063/300 Write block 1654 len 16384 data: 5 4 5 6
32833 064/300 Write block 2279 len 36864 data: 5 5 6 6 6 6 6 6 6
32834 065/300 Write block 2580 len 24576 data: 6 7 6 6 5 7
32835 066/300 Read block 4172 len 4096
32836 067/300 Read block 4776 len 16384
32837 068/300 Read block 132 len 16384
32838 069/300 Read block 3997 len 4096
32839 070/300 Read block 1890 len 20480
32840 071/300 Write block 3005 len 40960 data: 8 8 8 8 10 9 7 6 7 7
32841 072/300 Write block 2027 len 8192 data: 7 8
32842 073/300 Write block 4958 len 8192 data: 11 10
32843 074/300 Write block 463 len 20480 data: 4 6 5 3 3
32844 075/300 Read block 1675 len 32768
32845 076/300 Read block 395 len 24576
32846 077/300 Write block 4348 len 24576 data: 6 6 6 6 7 7
32847 078/300 Write block 1117 len 32768 data: 5 5 6 6 8 8 7 6
32848 079/300 Read block 988 len 20480
32849 080/300 Write block 3890 len 4096 data: 3
32850 081/300 Write block 2839 len 16384 data: 7 7 8 6
32851 082/300 Write block 785 len 20480 data: 6 8 7 8 8
32852 083/300 Read block 4314 len 36864
32853 084/300 Write block 575 len 8192 data: 5 4
32854 085/300 Read block 608 len 16384
32855 086/300 Write block 3356 len 32768 data: 6 7 5 5 6 7 8 10
32856 087/300 Write block 4212 len 32768 data: 11 10 8 9 7 7 7 5
32857 088/300 Write block 392 len 32768 data: 6 9 8 7 4 5 6 8
32858 089/300 Write block 2533 len 16384 data: 11 12 9 9
32859 090/300 Read block 2266 len 24576
32860 091/300 Write block 4418 len 40960 data: 4 4 2 3 3 3 4 5 5 6
32861 092/300 Write block 1682 len 36864 data: 4 3 3 4 6 5 5 4 3
32862 093/300 Read block 957 len 20480
32863 094/300 Write block 4019 len 24576 data: 6 7 7 5 4 4
32864 095/300 Read block 854 len 4096
32865 096/300 Read block 2861 len 24576
32866 097/300 Flush
32867 098/300 Write block 3634 len 12288 data: 7 7 7
32868 099/300 Read block 3462 len 12288
32869 100/300 Read block 712 len 16384
32870 101/300 Write block 1687 len 28672 data: 6 6 5 4 3 4 8
32871 102/300 Read block 626 len 12288
32872 103/300 Read block 298 len 28672
32873 104/300 Write block 1775 len 40960 data: 6 7 8 8 7 6 5 6 6 5
32874 105/300 Read block 445 len 24576
32875 106/300 Flush
32876 107/300 Read block 628 len 40960
32877 108/300 Write block 2892 len 12288 data: 8 8 8
32878 109/300 Read block 2749 len 40960
32879 110/300 Flush
32880 111/300 Read block 3318 len 16384
32881 112/300 Write block 2851 len 24576 data: 4 4 4 4 5 4
32882 113/300 Read block 2254 len 16384
32883 114/300 Write block 2883 len 24576 data: 5 7 7 8 9 8
32884 115/300 Write block 3785 len 16384 data: 8 7 5 5
32885 116/300 Write block 3290 len 36864 data: 5 6 6 8 8 7 7 7 5
32886 117/300 Read block 2219 len 8192
32887 118/300 Read block 3500 len 28672
32888 119/300 Write block 4725 len 12288 data: 5 6 5
32889 120/300 Write block 2256 len 12288 data: 2 5 5
32890 121/300 Write block 2535 len 24576 data: 10 10 8 7 7 6
32891 122/300 Read block 2760 len 36864
32892 123/300 Write block 3789 len 12288 data: 5 5 2
32893 124/300 Write block 3985 len 32768 data: 6 6 4 6 7 9 8 9
32894 125/300 Read block 576 len 20480
32895 126/300 Flush
32896 127/300 Read block 1326 len 36864
32897 128/300 Write block 3976 len 40960 data: 5 5 3 3 5 6 6 7 9 7
32898 129/300 Read block 1008 len 8192
32899 130/300 Write block 1436 len 20480 data: 8 7 7 7 6
32900 131/300 Read block 1626 len 28672
32901 132/300 Read block 4757 len 12288
32902 133/300 Read block 2492 len 40960
32903 134/300 Read block 580 len 12288
32904 135/300 Read block 3552 len 8192
32905 136/300 Write block 2524 len 40960 data: 4 4 4 6 6 7 7 10 11 12
32906 137/300 Read block 2500 len 20480
32907 138/300 Write block 46 len 16384 data: 8 6 6 5
32908 139/300 Write block 4787 len 36864 data: 5 5 5 6 8 8 8 9 9
32909 140/300 Write block 1568 len 8192 data: 14 13
32910 141/300 Read block 1450 len 4096
32911 142/300 Read block 1354 len 32768
32912 143/300 Write block 2961 len 36864 data: 4 5 4 5 6 6 5 6 7
32913 144/300 Read block 90 len 16384
32914 145/300 Write block 4237 len 4096 data: 6
32915 146/300 Flush
32916 147/300 Read block 3825 len 28672
32917 148/300 Read block 2100 len 8192
32918 149/300 Write block 2874 len 32768 data: 3 3 3 5 6 6 7 7
32919 150/300 Read block 1048 len 40960
32920 151/300 Read block 714 len 12288
32921 152/300 Write block 742 len 20480 data: 9 9 10 11 11
32922 153/300 Write block 4872 len 24576 data: 5 5 6 6 6 7
32923 154/300 Write block 765 len 4096 data: 5
32924 155/300 Read block 354 len 20480
32925 156/300 Flush
32926 157/300 Read block 4603 len 4096
32927 158/300 Write block 475 len 4096 data: 7
32928 159/300 Read block 4907 len 36864
32929 160/300 Write block 1850 len 12288 data: 8 8 8
32930 161/300 Read block 2033 len 32768
32931 162/300 Write block 2221 len 12288 data: 5 6 7
32932 163/300 Read block 2424 len 12288
32933 164/300 Flush
32934 165/300 Read block 686 len 32768
32935 166/300 Read block 2951 len 28672
32936 167/300 Read block 2932 len 20480
32937 168/300 Read block 1138 len 28672
32938 169/300 Read block 1243 len 40960
32939 170/300 Write block 3852 len 36864 data: 6 9 8 7 7 6 5 4 6
32940 171/300 Write block 4433 len 24576 data: 8 8 6 6 6 8
32941 172/300 Flush
32942 173/300 Flush
32943 174/300 Read block 4798 len 4096
32944 175/300 Write block 1140 len 4096 data: 10
32945 176/300 Flush
32946 177/300 Write block 3990 len 12288 data: 10 9 10
32947 178/300 Write block 2791 len 36864 data: 5 5 5 5 3 3 4 4 6
32948 179/300 Write block 960 len 16384 data: 8 7 7 5
32949 180/300 Write block 1836 len 32768 data: 8 9 8 7 6 6 5 5
32950 181/300 Flush
32951 182/300 Write block 2337 len 16384 data: 8 8 9 7
32952 183/300 Write block 2083 len 40960 data: 2 2 2 2 4 4 4 5 5 4
32953 184/300 Write block 4885 len 16384 data: 5 5 6 6
32954 185/300 Read block 2859 len 24576
32955 186/300 Read block 1507 len 4096
32956 187/300 Read block 3259 len 12288
32957 188/300 Write block 4753 len 8192 data: 3 3
32958 189/300 Write block 2964 len 40960 data: 6 7 7 6 7 8 7 7 7 8
32959 190/300 Read block 3740 len 20480
32960 191/300 Write block 1766 len 40960 data: 5 5 4 4 4 5 6 6 6 7
32961 192/300 Write block 3456 len 4096 data: 5
32962 193/300 Read block 4762 len 4096
32963 194/300 Read block 830 len 8192
32964 195/300 Write block 1958 len 40960 data: 5 5 5 3 3 4 5 6 6 5
32965 196/300 Read block 1949 len 40960
32966 197/300 Flush
32967 198/300 Read block 1601 len 28672
32968 199/300 Read block 4560 len 12288
32969 200/300 Write block 2465 len 36864 data: 7 7 9 10 8 6 6 6 3
32970 201/300 Read block 977 len 32768
32971 202/300 Read block 506 len 36864
32972 203/300 Write block 4114 len 40960 data: 8 7 7 7 7 7 10 9 9 9
32973 204/300 Flush
32974 205/300 Read block 3151 len 40960
32975 206/300 Read block 2837 len 8192
32976 207/300 Flush
32977 208/300 Read block 1336 len 20480
32978 209/300 Read block 3930 len 28672
32979 210/300 Write block 3292 len 4096 data: 7
32980 211/300 Read block 160 len 40960
32981 212/300 Write block 178 len 8192 data: 5 6
32982 213/300 Read block 1110 len 36864
32983 214/300 Read block 2848 len 24576
32984 215/300 Read block 4285 len 20480
32985 216/300 Write block 830 len 28672 data: 4 4 3 3 4 5 6
32986 217/300 Write block 4755 len 24576 data: 4 4 3 4 3 3
32987 218/300 Read block 2366 len 8192
32988 219/300 Read block 281 len 16384
32989 220/300 Read block 1682 len 20480
32990 221/300 Write block 2970 len 24576 data: 8 8 8 9 6 7
32991 222/300 Write block 186 len 36864 data: 5 5 6 8 9 8 6 8 7
32992 223/300 Read block 64 len 24576
32993 224/300 Write block 2034 len 36864 data: 6 6 6 5 5 6 6 6 7
32994 225/300 Write block 3841 len 32768 data: 4 3 4 5 4 4 4 4
32995 226/300 Read block 2085 len 4096
32996 227/300 Write block 2792 len 16384 data: 6 6 6 4
32997 228/300 Write block 3860 len 32768 data: 7 4 4 3 3 5 4 4
32998 229/300 Read block 4899 len 32768
32999 230/300 Read block 3727 len 12288
33000 231/300 Write block 1014 len 12288 data: 8 7 6
33001 232/300 Read block 2119 len 4096
33002 233/300 Write block 4346 len 32768 data: 7 7 7 7 7 7 8 8
33003 234/300 Read block 2844 len 20480
33004 235/300 Flush
33005 236/300 Write block 3805 len 16384 data: 5 4 5 6
33006 237/300 Read block 430 len 28672
33007 238/300 Write block 4862 len 28672 data: 4 7 7 8 9 6 5
33008 239/300 Read block 1011 len 16384
33009 240/300 Write block 4720 len 28672 data: 7 5 4 4 5 6 7
33010 241/300 Write block 4982 len 24576 data: 7 6 7 7 5 4
33011 242/300 Flush
33012 243/300 Write block 1308 len 8192 data: 5 4
33013 244/300 Read block 4438 len 20480
33014 245/300 Flush
33015 246/300 Write block 4246 len 4096 data: 4
33016 247/300 Read block 685 len 36864
33017 248/300 Read block 1620 len 40960
33018 249/300 Read block 4547 len 36864
33019 250/300 Read block 4317 len 16384
33020 251/300 Write block 4283 len 20480 data: 9 9 7 6 6
33021 252/300 Write block 4365 len 4096 data: 8
33022 253/300 Read block 3442 len 16384
33023 254/300 Read block 2255 len 36864
33024 255/300 Flush
33025 256/300 Read block 2752 len 20480
33026 257/300 Read block 3476 len 28672
33027 258/300 Read block 3765 len 32768
33028 259/300 Write block 3811 len 4096 data: 7
33029 260/300 Write block 1251 len 36864 data: 7 5 6 4 4 3 3 3 4
33030 261/300 Read block 3770 len 16384
33031 262/300 Write block 321 len 8192 data: 6 7
33032 263/300 Write block 2953 len 20480 data: 5 4 4 5 4
33033 264/300 Read block 4008 len 12288
33034 265/300 Write block 1830 len 20480 data: 7 10 11 8 9
33035 266/300 Write block 2244 len 16384 data: 8 5 5 4
33036 267/300 Read block 4916 len 4096
33037 268/300 Write block 993 len 16384 data: 7 7 8 6
33038 269/300 Read block 560 len 16384
33039 270/300 Read block 2467 len 32768
33040 271/300 Write block 805 len 12288 data: 6 7 6
33041 272/300 Read block 943 len 20480
33042 273/300 Write block 444 len 8192 data: 4 5
33043 274/300 Write block 3932 len 20480 data: 7 5 6 7 6
33044 275/300 Read block 2383 len 36864
33045 276/300 Read block 4828 len 16384
33046 277/300 Flush
33047 278/300 Read block 849 len 8192
33048 279/300 Write block 2316 len 4096 data: 4
33049 280/300 Write block 1211 len 28672 data: 10 9 9 10 8 8 6
33050 281/300 Write block 4044 len 28672 data: 8 7 8 7 7 7 6
33051 282/300 Write block 3402 len 4096 data: 4
33052 283/300 Read block 4630 len 32768
33053 284/300 Write block 4965 len 4096 data: 8
33054 285/300 Write block 4542 len 16384 data: 10 11 11 11
33055 286/300 Write block 2229 len 40960 data: 5 5 6 6 8 7 7 5 6 5
33056 287/300 Write block 1184 len 24576 data: 6 7 7 8 6 7
33057 288/300 Read block 2176 len 16384
33058 289/300 Flush
33059 290/300 Read block 2426 len 16384
33060 291/300 Read block 1652 len 28672
33061 292/300 Read block 4985 len 28672
33062 293/300 Read block 2824 len 8192
33063 294/300 Read block 2001 len 20480
33064 295/300 Write block 1446 len 20480 data: 4 4 4 4 7
33065 296/300 Read block 3124 len 8192
33066 297/300 Flush
33067 298/300 Write block 270 len 20480 data: 8 8 7 7 8
33068 299/300 Read block 876 len 40960
33069 300/300 Read block 4918 len 36864
33070 301/300 Write block 3827 len 12288 data: 8 9 11
33071 Main task has work to do, go find it
33072 got dsc Start(2)
33073 start 2
33074 Mar 07 18:19:51.918 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/2, method: POST, req_id: 95ff86cb-b7a2-49fe-b6c9-91894f1ca5c9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
33075 Replay: started 2, returned:Ok(())
33076 ----------------------------------------------------------------
33077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33078 States: Active Active Offline
33079 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33080 9452 Acked Write 5 Done Done Sent false
33081 9453 Acked Write 3 Done Done Sent false
33082 9454 Acked Read 4 Done Done Sent false
33083 9455 Acked Write 1 Done Done Sent false
33084 9456 Acked Write 7 Done Done Sent false
33085 9457 Acked Read 10 Done Done Sent false
33086 9458 Acked Read 6 Done Done Sent false
33087 9459 Acked Read 3 Done Done Sent false
33088 9460 Acked Read 6 Done Done Sent false
33089 9461 Acked Write 2 Done Done Sent false
33090 9462 Acked Read 5 Done Done Sent false
33091 9463 Acked Write 4 Done Done Sent false
33092 9464 Acked Write 10 Done Done Sent false
33093 9465 Acked Read 8 Done Done Sent false
33094 9466 Acked Read 5 Done Done Sent false
33095 9467 Acked Read 9 Done Done Sent false
33096 9468 Acked Read 9 Done Done Sent false
33097 9469 Acked Write 7 Done Done Sent false
33098 9470 Acked Read 1 Done Done Sent false
33099 9471 Acked Read 9 Done Done Sent false
33100 9472 Acked Read 9 Done Done Sent false
33101 9473 Acked Read 6 Done Done Sent false
33102 9474 Acked Read 5 Done Done Sent false
33103 9475 Acked Write 10 Done Done Sent false
33104 9476 Acked Write 3 Done Done Sent false
33105 9477 Acked Write 4 Done Done Sent false
33106 9478 Acked Read 10 Done Done Sent false
33107 9479 Acked Read 8 Done Done Sent false
33108 9480 Acked Read 10 Done Done Sent false
33109 9481 Acked Read 7 Done Done Sent false
33110 9482 Acked Write 3 Done Done Sent false
33111 9483 Acked Read 9 Done Done Sent false
33112 9484 Acked Read 9 Done Done Sent false
33113 9485 Acked Read 6 Done Done Sent false
33114 9486 Acked Flush 0 Done Done Sent false
33115 9487 Acked Read 5 Done Done Sent false
33116 9488 Acked Write 4 Done Done Sent false
33117 9489 Acked Write 8 Done Done Sent false
33118 9490 Acked Read 2 Done Done Sent false
33119 9491 Acked Flush 0 Done Done Sent false
33120 9492 Acked Flush 0 Done Done Sent false
33121 9493 Acked Flush 0 Done Done Sent false
33122 9494 Acked Read 8 Done Done Sent false
33123 9495 Acked Flush 0 Done Done Sent false
33124 9496 Acked Write 8 Done Done Sent false
33125 9497 Acked Read 8 Done Done Sent false
33126 9498 Acked Read 7 Done Done Sent false
33127 9499 Acked Write 6 Done Done Sent false
33128 9500 Acked Write 5 Done Done Sent false
33129 9501 Acked Read 8 Done Done Sent false
33130 9502 Acked Write 7 Done Done Sent false
33131 9503 Acked Read 9 Done Done Sent false
33132 9504 Acked Read 10 Done Done Sent false
33133 9505 Acked Write 2 Done Done Sent false
33134 9506 Acked Read 8 Done Done Sent false
33135 9507 Acked Read 5 Done Done Sent false
33136 9508 Acked Write 1 Done Done Sent false
33137 9509 Acked Read 7 Done Done Sent false
33138 9510 Acked Write 10 Done Done Sent false
33139 9511 Acked Read 10 Done Done Sent false
33140 9512 Acked Write 1 Done Done Sent false
33141 9513 Acked Read 8 Done Done Sent false
33142 9514 Acked Write 4 Done Done Sent false
33143 9515 Acked Write 9 Done Done Sent false
33144 9516 Acked Write 6 Done Done Sent false
33145 9517 Acked Read 1 Done Done Sent false
33146 9518 Acked Read 4 Done Done Sent false
33147 9519 Acked Read 4 Done Done Sent false
33148 9520 Acked Read 1 Done Done Sent false
33149 9521 Acked Read 5 Done Done Sent false
33150 9522 Acked Write 10 Done Done Sent false
33151 9523 Acked Write 2 Done Done Sent false
33152 9524 Acked Write 2 Done Done Sent false
33153 9525 Acked Write 5 Done Done Sent false
33154 9526 Acked Read 8 Done Done Sent false
33155 9527 Acked Read 6 Done Done Sent false
33156 9528 Acked Write 6 Done Done Sent false
33157 9529 Acked Write 8 Done Done Sent false
33158 9530 Acked Read 5 Done Done Sent false
33159 9531 Acked Write 1 Done Done Sent false
33160 9532 Acked Write 4 Done Done Sent false
33161 9533 Acked Write 5 Done Done Sent false
33162 9534 Acked Read 9 Done Done Sent false
33163 9535 Acked Write 2 Done Done Sent false
33164 9536 Acked Read 4 Done Done Sent false
33165 9537 Acked Write 8 Done Done Sent false
33166 9538 Acked Write 8 Done Done Sent false
33167 9539 Acked Write 8 Done Done Sent false
33168 9540 Acked Write 4 Done Done Sent false
33169 9541 Acked Read 6 Done Done Sent false
33170 9542 Acked Write 10 Done Done Sent false
33171 9543 Acked Write 9 Done Done Sent false
33172 9544 Acked Read 5 Done Done Sent false
33173 9545 Acked Write 6 Done Done Sent false
33174 9546 Acked Read 1 Done Done Sent false
33175 9547 Acked Read 6 Done Done Sent false
33176 9548 Acked Flush 0 Done Done Sent false
33177 9549 Acked Write 3 Done Done Sent false
33178 9550 Acked Read 3 Done Done Sent false
33179 9551 Acked Read 4 Done Done Sent false
33180 9552 Acked Write 7 Done Done Sent false
33181 9553 Acked Read 3 Done Done Sent false
33182 9554 Acked Read 7 Done Done Sent false
33183 9555 Acked Write 10 Done Done Sent false
33184 9556 Acked Read 6 Done Done Sent false
33185 9557 Acked Flush 0 Done Done Sent false
33186 9558 Acked Read 10 Done Done Sent false
33187 9559 Acked Write 3 Done Done Sent false
33188 9560 Acked Read 10 Done Done Sent false
33189 9561 Acked Flush 0 Done Done Sent false
33190 9562 Acked Read 4 Done Done Sent false
33191 9563 Acked Write 6 Done Done Sent false
33192 9564 Acked Read 4 Done Done Sent false
33193 9565 Acked Write 6 Done Done Sent false
33194 9566 Acked Write 4 Done Done Sent false
33195 9567 Acked Write 9 Done Done Sent false
33196 9568 Acked Read 2 Done Done Sent false
33197 9569 Acked Read 7 Done Done Sent false
33198 9570 Acked Write 3 Done Done Sent false
33199 9571 Acked Write 3 Done Done Sent false
33200 9572 Acked Write 6 Done Done Sent false
33201 9573 Acked Read 9 Done Done Sent false
33202 9574 Acked Write 3 Done Done Sent false
33203 9575 Acked Write 8 Done Done Sent false
33204 9576 Acked Read 5 Done Done Sent false
33205 9577 Acked Flush 0 Done Done Sent false
33206 9578 Acked Read 9 Done Done Sent false
33207 9579 Acked Write 10 Done Done Sent false
33208 9580 Acked Read 2 Done Done Sent false
33209 9581 Acked Write 5 Done Done Sent false
33210 9582 Acked Read 7 Done Done Sent false
33211 9583 Acked Read 3 Done Done Sent false
33212 9584 Acked Read 10 Done Done Sent false
33213 9585 Acked Read 3 Done Done Sent false
33214 9586 Acked Read 2 Done Done Sent false
33215 9587 Acked Write 10 Done Done Sent false
33216 9588 Acked Read 5 Done Done Sent false
33217 9589 Acked Write 4 Done Done Sent false
33218 9590 Acked Write 9 Done Done Sent false
33219 9591 Acked Write 2 Done Done Sent false
33220 9592 Acked Read 1 Done Done Sent false
33221 9593 Acked Read 8 Done Done Sent false
33222 9594 Acked Write 9 Done Done Sent false
33223 9595 Acked Read 4 Done Done Sent false
33224 9596 Acked Write 1 Done Done Sent false
33225 9597 Acked Flush 0 Done Done Sent false
33226 9598 Acked Read 7 Done Done Sent false
33227 9599 Acked Read 2 Done Done Sent false
33228 9600 Acked Write 8 Done Done Sent false
33229 9601 Acked Read 10 Done Done Sent false
33230 9602 Acked Read 3 Done Done Sent false
33231 9603 Acked Write 5 Done Done Sent false
33232 9604 Acked Write 6 Done Done Sent false
33233 9605 Acked Write 1 Done Done Sent false
33234 9606 Acked Read 5 Done Done Sent false
33235 9607 Acked Flush 0 Done Done Sent false
33236 9608 Acked Read 1 Done Done Sent false
33237 9609 Acked Write 1 Done Done Sent false
33238 9610 Acked Read 9 Done Done Sent false
33239 9611 Acked Write 3 Done Done Sent false
33240 9612 Acked Read 8 Done Done Sent false
33241 9613 Acked Write 3 Done Done Sent false
33242 9614 Acked Read 3 Done Done Sent false
33243 9615 Acked Flush 0 Done Done Sent false
33244 9616 Acked Read 8 Done Done Sent false
33245 9617 Acked Read 7 Done Done Sent false
33246 9618 Acked Read 5 Done Done Sent false
33247 9619 Acked Read 7 Done Done Sent false
33248 9620 Acked Read 10 Done Done Sent false
33249 9621 Acked Write 9 Done Done Sent false
33250 9622 Acked Write 6 Done Done Sent false
33251 9623 Acked Flush 0 Done Done Sent false
33252 9624 Acked Flush 0 Done Done Sent false
33253 9625 Acked Read 1 Done Done Sent false
33254 9626 Acked Write 1 Done Done Sent false
33255 9627 Acked Flush 0 Done Done Sent false
33256 9628 Acked Write 3 Done Done Sent false
33257 9629 Acked Write 9 Done Done Sent false
33258 9630 Acked Write 4 Done Done Sent false
33259 9631 Acked Write 8 Done Done Sent false
33260 9632 Acked Flush 0 Done Done Sent false
33261 9633 Acked Write 4 Done Done Sent false
33262 9634 Acked Write 10 Done Done Sent false
33263 9635 Acked Write 4 Done Done Sent false
33264 9636 Acked Read 6 Done Done Sent false
33265 9637 Acked Read 1 Done Done Sent false
33266 9638 Acked Read 3 Done Done Sent false
33267 9639 Acked Write 2 Done Done Sent false
33268 9640 Acked Write 10 Done Done Sent false
33269 9641 Acked Read 5 Done Done Sent false
33270 9642 Acked Write 10 Done Done Sent false
33271 9643 Acked Write 1 Done Done Sent false
33272 9644 Acked Read 1 Done Done Sent false
33273 9645 Acked Read 2 Done Done Sent false
33274 9646 Acked Write 10 Done Done Sent false
33275 9647 Acked Read 10 Done Done Sent false
33276 9648 Acked Flush 0 Done Done Sent false
33277 9649 Acked Read 7 Done Done Sent false
33278 9650 Acked Read 3 Done Done Sent false
33279 9651 Acked Write 9 Done Done Sent false
33280 9652 Acked Read 8 Done Done Sent false
33281 9653 Acked Read 9 Done Done Sent false
33282 9654 Acked Write 10 Done Done Sent false
33283 9655 Acked Flush 0 Done Done Sent false
33284 9656 Acked Read 10 Done Done Sent false
33285 9657 Acked Read 2 Done Done Sent false
33286 9658 Acked Flush 0 Done Done Sent false
33287 9659 Acked Read 5 Done Done Sent false
33288 9660 Acked Read 7 Done Done Sent false
33289 9661 Acked Write 1 Done Done Sent false
33290 9662 Acked Read 10 Done Done Sent false
33291 9663 Acked Write 2 Done Done Sent false
33292 9664 Acked Read 9 Done Done Sent false
33293 9665 Acked Read 6 Done Done Sent false
33294 9666 Acked Read 5 Done Done Sent false
33295 9667 Acked Write 7 Done Done Sent false
33296 9668 Acked Write 6 Done Done Sent false
33297 9669 Acked Read 2 Done Done Sent false
33298 9670 Acked Read 4 Done Done Sent false
33299 9671 Acked Read 5 Done Done Sent false
33300 9672 Acked Write 6 Done Done Sent false
33301 9673 Acked Write 9 Done Done Sent false
33302 9674 Acked Read 6 Done Done Sent false
33303 9675 Acked Write 9 Done Done Sent false
33304 9676 Acked Write 8 Done Done Sent false
33305 9677 Acked Read 1 Done Done Sent false
33306 9678 Acked Write 4 Done Done Sent false
33307 9679 Acked Write 8 Done Done Sent false
33308 9680 Acked Read 8 Done Done Sent false
33309 9681 Acked Read 3 Done Done Sent false
33310 9682 Acked Write 3 Done Done Sent false
33311 9683 Acked Read 1 Done Done Sent false
33312 9684 Acked Write 8 Done Done Sent false
33313 9685 Acked Read 5 Done Done Sent false
33314 9686 Acked Flush 0 Done Done Sent false
33315 9687 Acked Write 4 Done Done Sent false
33316 9688 Acked Read 7 Done Done Sent false
33317 9689 Acked Write 7 Done Done Sent false
33318 9690 Acked Read 4 Done Done Sent false
33319 9691 Acked Write 7 Done Done Sent false
33320 9692 Acked Write 6 Done Done Sent false
33321 9693 Acked Flush 0 Done Done Sent false
33322 9694 Acked Write 2 Done Done Sent false
33323 9695 Acked Read 5 Done Done Sent false
33324 9696 Acked Flush 0 Done Done Sent false
33325 9697 Acked Write 1 Done Done Sent false
33326 9698 Acked Read 9 Done Done Sent false
33327 9699 Acked Read 10 Done Done Sent false
33328 9700 Acked Read 9 Done Done Sent false
33329 9701 Acked Read 4 Done Done Sent false
33330 9702 Acked Write 5 Done Done Sent false
33331 9703 Acked Write 1 Done Done Sent false
33332 9704 Acked Read 4 Done Done Sent false
33333 9705 Acked Read 9 Done Done Sent false
33334 9706 Acked Flush 0 Done Done Sent false
33335 9707 Acked Read 5 Done Done Sent false
33336 9708 Acked Read 7 Done Done Sent false
33337 9709 Acked Read 8 Done Done Sent false
33338 9710 Acked Write 1 Done Done Sent false
33339 9711 Acked Write 9 Done Done Sent false
33340 9712 Acked Read 4 Done Done Sent false
33341 9713 Acked Write 2 Done Done Sent false
33342 9714 Acked Write 5 Done Done Sent false
33343 9715 Acked Read 3 Done Done Sent false
33344 9716 Acked Write 5 Done Done Sent false
33345 9717 Acked Write 4 Done Done Sent false
33346 9718 Acked Read 1 Done Done Sent false
33347 9719 Acked Write 4 Done Done Sent false
33348 9720 Acked Read 4 Done Done Sent false
33349 9721 Acked Read 8 Done Done Sent false
33350 9722 Acked Write 3 Done Done Sent false
33351 9723 Acked Read 5 Done Done Sent false
33352 9724 Acked Write 2 Done Done Sent false
33353 9725 Acked Write 5 Done Done Sent false
33354 9726 Acked Read 9 Done Done Sent false
33355 9727 Acked Read 4 Done Done Sent false
33356 9728 Acked Flush 0 Done Done Sent false
33357 9729 Acked Read 2 Done Done Sent false
33358 9730 Acked Write 1 Done Done Sent false
33359 9731 Acked Write 7 Done Done Sent false
33360 9732 Acked Write 7 Done Done Sent false
33361 9733 Acked Write 1 Done Done Sent false
33362 9734 Acked Read 8 Done Done Sent false
33363 9735 Acked Write 1 Done Done Sent false
33364 9736 Acked Write 4 Done Done Sent false
33365 9737 Acked Write 10 Done Done Sent false
33366 9738 Acked Write 6 Done Done Sent false
33367 9739 Acked Read 4 Done Done Sent false
33368 9740 Acked Flush 0 Done Done Sent false
33369 9741 Acked Read 4 Done Done Sent false
33370 9742 Acked Read 7 Done Done Sent false
33371 9743 Acked Read 7 Done Done Sent false
33372 9744 Acked Read 2 Done Done Sent false
33373 9745 Acked Read 5 Done Done Sent false
33374 9746 Acked Write 5 Done Done Sent false
33375 9747 Acked Read 2 Done Done Sent false
33376 9748 Acked Flush 0 Done Done Sent false
33377 9749 Acked Write 5 Done Done Sent false
33378 9750 Acked Read 10 Done Done Sent false
33379 9751 Acked Read 9 Done Done Sent false
33380 9752 Acked Write 3 Done Done Sent false
33381 STATES DS:0 DS:1 DS:2 TOTAL
33382 Sent 0 0 301 301
33383 Done 301 301 0 602
33384 Skipped 0 0 0 0
33385 Error 0 0 0 0
33386 Last Flush: Some(JobId(9748)) Some(JobId(9748)) Some(JobId(9451))
33387 Downstairs last five completed: 9451 9450 9449 9448 9447
33388 Upstairs last five completed: 9752 9751 9750 9749 9748
33389 CLIENT: Up:0 ds:301 act:2
333902025-03-07T18:19:52.677ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
333912025-03-07T18:19:53.678ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
333922025-03-07T18:19:54.679ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
33393 [8830] got start action
33394 [8830] I am going to restart
33395 Starting downstairs at port 8830
33396 Make output file at "/tmp/dsc/downstairs-8830.txt"
33397 [8830][2] reports Starting
333982025-03-07T18:19:55.681ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
33399 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1237)
33400 [8830][2] reports Running
33401 ----------------------------------------------------------------
33402 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33403 States: Active Active Offline
33404 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33405 9452 Acked Write 5 Done Done Sent false
33406 9453 Acked Write 3 Done Done Sent false
33407 9454 Acked Read 4 Done Done Sent false
33408 9455 Acked Write 1 Done Done Sent false
33409 9456 Acked Write 7 Done Done Sent false
33410 9457 Acked Read 10 Done Done Sent false
33411 9458 Acked Read 6 Done Done Sent false
33412 9459 Acked Read 3 Done Done Sent false
33413 9460 Acked Read 6 Done Done Sent false
33414 9461 Acked Write 2 Done Done Sent false
33415 9462 Acked Read 5 Done Done Sent false
33416 9463 Acked Write 4 Done Done Sent false
33417 9464 Acked Write 10 Done Done Sent false
33418 9465 Acked Read 8 Done Done Sent false
33419 9466 Acked Read 5 Done Done Sent false
33420 9467 Acked Read 9 Done Done Sent false
33421 9468 Acked Read 9 Done Done Sent false
33422 9469 Acked Write 7 Done Done Sent false
33423 9470 Acked Read 1 Done Done Sent false
33424 9471 Acked Read 9 Done Done Sent false
33425 9472 Acked Read 9 Done Done Sent false
33426 9473 Acked Read 6 Done Done Sent false
33427 9474 Acked Read 5 Done Done Sent false
33428 9475 Acked Write 10 Done Done Sent false
33429 9476 Acked Write 3 Done Done Sent false
33430 9477 Acked Write 4 Done Done Sent false
33431 9478 Acked Read 10 Done Done Sent false
33432 9479 Acked Read 8 Done Done Sent false
33433 9480 Acked Read 10 Done Done Sent false
33434 9481 Acked Read 7 Done Done Sent false
33435 9482 Acked Write 3 Done Done Sent false
33436 9483 Acked Read 9 Done Done Sent false
33437 9484 Acked Read 9 Done Done Sent false
33438 9485 Acked Read 6 Done Done Sent false
33439 9486 Acked Flush 0 Done Done Sent false
33440 9487 Acked Read 5 Done Done Sent false
33441 9488 Acked Write 4 Done Done Sent false
33442 9489 Acked Write 8 Done Done Sent false
33443 9490 Acked Read 2 Done Done Sent false
33444 9491 Acked Flush 0 Done Done Sent false
33445 9492 Acked Flush 0 Done Done Sent false
33446 9493 Acked Flush 0 Done Done Sent false
33447 9494 Acked Read 8 Done Done Sent false
33448 9495 Acked Flush 0 Done Done Sent false
33449 9496 Acked Write 8 Done Done Sent false
33450 9497 Acked Read 8 Done Done Sent false
33451 9498 Acked Read 7 Done Done Sent false
33452 9499 Acked Write 6 Done Done Sent false
33453 9500 Acked Write 5 Done Done Sent false
33454 9501 Acked Read 8 Done Done Sent false
33455 9502 Acked Write 7 Done Done Sent false
33456 9503 Acked Read 9 Done Done Sent false
33457 9504 Acked Read 10 Done Done Sent false
33458 9505 Acked Write 2 Done Done Sent false
33459 9506 Acked Read 8 Done Done Sent false
33460 9507 Acked Read 5 Done Done Sent false
33461 9508 Acked Write 1 Done Done Sent false
33462 9509 Acked Read 7 Done Done Sent false
33463 9510 Acked Write 10 Done Done Sent false
33464 9511 Acked Read 10 Done Done Sent false
33465 9512 Acked Write 1 Done Done Sent false
33466 9513 Acked Read 8 Done Done Sent false
33467 9514 Acked Write 4 Done Done Sent false
33468 9515 Acked Write 9 Done Done Sent false
33469 9516 Acked Write 6 Done Done Sent false
33470 9517 Acked Read 1 Done Done Sent false
33471 9518 Acked Read 4 Done Done Sent false
33472 9519 Acked Read 4 Done Done Sent false
33473 9520 Acked Read 1 Done Done Sent false
33474 9521 Acked Read 5 Done Done Sent false
33475 9522 Acked Write 10 Done Done Sent false
33476 9523 Acked Write 2 Done Done Sent false
33477 9524 Acked Write 2 Done Done Sent false
33478 9525 Acked Write 5 Done Done Sent false
33479 9526 Acked Read 8 Done Done Sent false
33480 9527 Acked Read 6 Done Done Sent false
33481 9528 Acked Write 6 Done Done Sent false
33482 9529 Acked Write 8 Done Done Sent false
33483 9530 Acked Read 5 Done Done Sent false
33484 9531 Acked Write 1 Done Done Sent false
33485 9532 Acked Write 4 Done Done Sent false
33486 9533 Acked Write 5 Done Done Sent false
33487 9534 Acked Read 9 Done Done Sent false
33488 9535 Acked Write 2 Done Done Sent false
33489 9536 Acked Read 4 Done Done Sent false
33490 9537 Acked Write 8 Done Done Sent false
33491 9538 Acked Write 8 Done Done Sent false
33492 9539 Acked Write 8 Done Done Sent false
33493 9540 Acked Write 4 Done Done Sent false
33494 9541 Acked Read 6 Done Done Sent false
33495 9542 Acked Write 10 Done Done Sent false
33496 9543 Acked Write 9 Done Done Sent false
33497 9544 Acked Read 5 Done Done Sent false
33498 9545 Acked Write 6 Done Done Sent false
33499 9546 Acked Read 1 Done Done Sent false
33500 9547 Acked Read 6 Done Done Sent false
33501 9548 Acked Flush 0 Done Done Sent false
33502 9549 Acked Write 3 Done Done Sent false
33503 9550 Acked Read 3 Done Done Sent false
33504 9551 Acked Read 4 Done Done Sent false
33505 9552 Acked Write 7 Done Done Sent false
33506 9553 Acked Read 3 Done Done Sent false
33507 9554 Acked Read 7 Done Done Sent false
33508 9555 Acked Write 10 Done Done Sent false
33509 9556 Acked Read 6 Done Done Sent false
33510 9557 Acked Flush 0 Done Done Sent false
33511 9558 Acked Read 10 Done Done Sent false
33512 9559 Acked Write 3 Done Done Sent false
33513 9560 Acked Read 10 Done Done Sent false
33514 9561 Acked Flush 0 Done Done Sent false
33515 9562 Acked Read 4 Done Done Sent false
33516 9563 Acked Write 6 Done Done Sent false
33517 9564 Acked Read 4 Done Done Sent false
33518 9565 Acked Write 6 Done Done Sent false
33519 9566 Acked Write 4 Done Done Sent false
33520 9567 Acked Write 9 Done Done Sent false
33521 9568 Acked Read 2 Done Done Sent false
33522 9569 Acked Read 7 Done Done Sent false
33523 9570 Acked Write 3 Done Done Sent false
33524 9571 Acked Write 3 Done Done Sent false
33525 9572 Acked Write 6 Done Done Sent false
33526 9573 Acked Read 9 Done Done Sent false
33527 9574 Acked Write 3 Done Done Sent false
33528 9575 Acked Write 8 Done Done Sent false
33529 9576 Acked Read 5 Done Done Sent false
33530 9577 Acked Flush 0 Done Done Sent false
33531 9578 Acked Read 9 Done Done Sent false
33532 9579 Acked Write 10 Done Done Sent false
33533 9580 Acked Read 2 Done Done Sent false
33534 9581 Acked Write 5 Done Done Sent false
33535 9582 Acked Read 7 Done Done Sent false
33536 9583 Acked Read 3 Done Done Sent false
33537 9584 Acked Read 10 Done Done Sent false
33538 9585 Acked Read 3 Done Done Sent false
33539 9586 Acked Read 2 Done Done Sent false
33540 9587 Acked Write 10 Done Done Sent false
33541 9588 Acked Read 5 Done Done Sent false
33542 9589 Acked Write 4 Done Done Sent false
33543 9590 Acked Write 9 Done Done Sent false
33544 9591 Acked Write 2 Done Done Sent false
33545 9592 Acked Read 1 Done Done Sent false
33546 9593 Acked Read 8 Done Done Sent false
33547 9594 Acked Write 9 Done Done Sent false
33548 9595 Acked Read 4 Done Done Sent false
33549 9596 Acked Write 1 Done Done Sent false
33550 9597 Acked Flush 0 Done Done Sent false
33551 9598 Acked Read 7 Done Done Sent false
33552 9599 Acked Read 2 Done Done Sent false
33553 9600 Acked Write 8 Done Done Sent false
33554 9601 Acked Read 10 Done Done Sent false
33555 9602 Acked Read 3 Done Done Sent false
33556 9603 Acked Write 5 Done Done Sent false
33557 9604 Acked Write 6 Done Done Sent false
33558 9605 Acked Write 1 Done Done Sent false
33559 9606 Acked Read 5 Done Done Sent false
33560 9607 Acked Flush 0 Done Done Sent false
33561 9608 Acked Read 1 Done Done Sent false
33562 9609 Acked Write 1 Done Done Sent false
33563 9610 Acked Read 9 Done Done Sent false
33564 9611 Acked Write 3 Done Done Sent false
33565 9612 Acked Read 8 Done Done Sent false
33566 9613 Acked Write 3 Done Done Sent false
33567 9614 Acked Read 3 Done Done Sent false
33568 9615 Acked Flush 0 Done Done Sent false
33569 9616 Acked Read 8 Done Done Sent false
33570 9617 Acked Read 7 Done Done Sent false
33571 9618 Acked Read 5 Done Done Sent false
33572 9619 Acked Read 7 Done Done Sent false
33573 9620 Acked Read 10 Done Done Sent false
33574 9621 Acked Write 9 Done Done Sent false
33575 9622 Acked Write 6 Done Done Sent false
33576 9623 Acked Flush 0 Done Done Sent false
33577 9624 Acked Flush 0 Done Done Sent false
33578 9625 Acked Read 1 Done Done Sent false
33579 9626 Acked Write 1 Done Done Sent false
33580 9627 Acked Flush 0 Done Done Sent false
33581 9628 Acked Write 3 Done Done Sent false
33582 9629 Acked Write 9 Done Done Sent false
33583 9630 Acked Write 4 Done Done Sent false
33584 9631 Acked Write 8 Done Done Sent false
33585 9632 Acked Flush 0 Done Done Sent false
33586 9633 Acked Write 4 Done Done Sent false
33587 9634 Acked Write 10 Done Done Sent false
33588 9635 Acked Write 4 Done Done Sent false
33589 9636 Acked Read 6 Done Done Sent false
33590 9637 Acked Read 1 Done Done Sent false
33591 9638 Acked Read 3 Done Done Sent false
33592 9639 Acked Write 2 Done Done Sent false
33593 9640 Acked Write 10 Done Done Sent false
33594 9641 Acked Read 5 Done Done Sent false
33595 9642 Acked Write 10 Done Done Sent false
33596 9643 Acked Write 1 Done Done Sent false
33597 9644 Acked Read 1 Done Done Sent false
33598 9645 Acked Read 2 Done Done Sent false
33599 9646 Acked Write 10 Done Done Sent false
33600 9647 Acked Read 10 Done Done Sent false
33601 9648 Acked Flush 0 Done Done Sent false
33602 9649 Acked Read 7 Done Done Sent false
33603 9650 Acked Read 3 Done Done Sent false
33604 9651 Acked Write 9 Done Done Sent false
33605 9652 Acked Read 8 Done Done Sent false
33606 9653 Acked Read 9 Done Done Sent false
33607 9654 Acked Write 10 Done Done Sent false
33608 9655 Acked Flush 0 Done Done Sent false
33609 9656 Acked Read 10 Done Done Sent false
33610 9657 Acked Read 2 Done Done Sent false
33611 9658 Acked Flush 0 Done Done Sent false
33612 9659 Acked Read 5 Done Done Sent false
33613 9660 Acked Read 7 Done Done Sent false
33614 9661 Acked Write 1 Done Done Sent false
33615 9662 Acked Read 10 Done Done Sent false
33616 9663 Acked Write 2 Done Done Sent false
33617 9664 Acked Read 9 Done Done Sent false
33618 9665 Acked Read 6 Done Done Sent false
33619 9666 Acked Read 5 Done Done Sent false
33620 9667 Acked Write 7 Done Done Sent false
33621 9668 Acked Write 6 Done Done Sent false
33622 9669 Acked Read 2 Done Done Sent false
33623 9670 Acked Read 4 Done Done Sent false
33624 9671 Acked Read 5 Done Done Sent false
33625 9672 Acked Write 6 Done Done Sent false
33626 9673 Acked Write 9 Done Done Sent false
33627 9674 Acked Read 6 Done Done Sent false
33628 9675 Acked Write 9 Done Done Sent false
33629 9676 Acked Write 8 Done Done Sent false
33630 9677 Acked Read 1 Done Done Sent false
33631 9678 Acked Write 4 Done Done Sent false
33632 9679 Acked Write 8 Done Done Sent false
33633 9680 Acked Read 8 Done Done Sent false
33634 9681 Acked Read 3 Done Done Sent false
33635 9682 Acked Write 3 Done Done Sent false
33636 9683 Acked Read 1 Done Done Sent false
33637 9684 Acked Write 8 Done Done Sent false
33638 9685 Acked Read 5 Done Done Sent false
33639 9686 Acked Flush 0 Done Done Sent false
33640 9687 Acked Write 4 Done Done Sent false
33641 9688 Acked Read 7 Done Done Sent false
33642 9689 Acked Write 7 Done Done Sent false
33643 9690 Acked Read 4 Done Done Sent false
33644 9691 Acked Write 7 Done Done Sent false
33645 9692 Acked Write 6 Done Done Sent false
33646 9693 Acked Flush 0 Done Done Sent false
33647 9694 Acked Write 2 Done Done Sent false
33648 9695 Acked Read 5 Done Done Sent false
33649 9696 Acked Flush 0 Done Done Sent false
33650 9697 Acked Write 1 Done Done Sent false
33651 9698 Acked Read 9 Done Done Sent false
33652 9699 Acked Read 10 Done Done Sent false
33653 9700 Acked Read 9 Done Done Sent false
33654 9701 Acked Read 4 Done Done Sent false
33655 9702 Acked Write 5 Done Done Sent false
33656 9703 Acked Write 1 Done Done Sent false
33657 9704 Acked Read 4 Done Done Sent false
33658 9705 Acked Read 9 Done Done Sent false
33659 9706 Acked Flush 0 Done Done Sent false
33660 9707 Acked Read 5 Done Done Sent false
33661 9708 Acked Read 7 Done Done Sent false
33662 9709 Acked Read 8 Done Done Sent false
33663 9710 Acked Write 1 Done Done Sent false
33664 9711 Acked Write 9 Done Done Sent false
33665 9712 Acked Read 4 Done Done Sent false
33666 9713 Acked Write 2 Done Done Sent false
33667 9714 Acked Write 5 Done Done Sent false
33668 9715 Acked Read 3 Done Done Sent false
33669 9716 Acked Write 5 Done Done Sent false
33670 9717 Acked Write 4 Done Done Sent false
33671 9718 Acked Read 1 Done Done Sent false
33672 9719 Acked Write 4 Done Done Sent false
33673 9720 Acked Read 4 Done Done Sent false
33674 9721 Acked Read 8 Done Done Sent false
33675 9722 Acked Write 3 Done Done Sent false
33676 9723 Acked Read 5 Done Done Sent false
33677 9724 Acked Write 2 Done Done Sent false
33678 9725 Acked Write 5 Done Done Sent false
33679 9726 Acked Read 9 Done Done Sent false
33680 9727 Acked Read 4 Done Done Sent false
33681 9728 Acked Flush 0 Done Done Sent false
33682 9729 Acked Read 2 Done Done Sent false
33683 9730 Acked Write 1 Done Done Sent false
33684 9731 Acked Write 7 Done Done Sent false
33685 9732 Acked Write 7 Done Done Sent false
33686 9733 Acked Write 1 Done Done Sent false
33687 9734 Acked Read 8 Done Done Sent false
33688 9735 Acked Write 1 Done Done Sent false
33689 9736 Acked Write 4 Done Done Sent false
33690 9737 Acked Write 10 Done Done Sent false
33691 9738 Acked Write 6 Done Done Sent false
33692 9739 Acked Read 4 Done Done Sent false
33693 9740 Acked Flush 0 Done Done Sent false
33694 9741 Acked Read 4 Done Done Sent false
33695 9742 Acked Read 7 Done Done Sent false
33696 9743 Acked Read 7 Done Done Sent false
33697 9744 Acked Read 2 Done Done Sent false
33698 9745 Acked Read 5 Done Done Sent false
33699 9746 Acked Write 5 Done Done Sent false
33700 9747 Acked Read 2 Done Done Sent false
33701 9748 Acked Flush 0 Done Done Sent false
33702 9749 Acked Write 5 Done Done Sent false
33703 9750 Acked Read 10 Done Done Sent false
33704 9751 Acked Read 9 Done Done Sent false
33705 9752 Acked Write 3 Done Done Sent false
33706 STATES DS:0 DS:1 DS:2 TOTAL
33707 Sent 0 0 301 301
33708 Done 301 301 0 602
33709 Skipped 0 0 0 0
33710 Error 0 0 0 0
33711 Last Flush: Some(JobId(9748)) Some(JobId(9748)) Some(JobId(9451))
33712 Downstairs last five completed: 9451 9450 9449 9448 9447
33713 Upstairs last five completed: 9752 9751 9750 9749 9748
33714 CLIENT: Up:0 ds:301 act:2
337152025-03-07T18:19:57.163ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337162025-03-07T18:19:57.669ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337172025-03-07T18:19:57.669ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53260) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337182025-03-07T18:19:57.671ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337192025-03-07T18:19:57.671ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337202025-03-07T18:19:57.671ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337212025-03-07T18:19:57.671ZINFOcrucible: send last flush ID to this DS: Some(JobId(9451)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337222025-03-07T18:19:57.671ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337232025-03-07T18:19:57.671ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9451)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337242025-03-07T18:19:57.671ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9451)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
33725 ----------------------------------------------------------------
33726 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33727 Downstairs last five completed: 9753 9752 9751 9750 9749
33728 Upstairs last five completed: 9753 9752 9751 9750 9749
33729 CLIENT: Up:0 ds:0 act:3
33730 Replay: All jobs finished
33731 Main task has work to do, go find it
33732 got dsc Stop(2)
33733 stop 2
33734 Mar 07 18:19:59.924 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d2da73fe-0712-4a64-8a97-9846f10e79f4, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
33735 [8830] Got stop action so:false kr:false
33736 Mar 07 18:19:59.925 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/2, method: GET, req_id: fb3e3af1-a27c-417a-a77a-9034f2a3eac9, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
337372025-03-07T18:19:59.925ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337382025-03-07T18:19:59.925ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337392025-03-07T18:19:59.925ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
337402025-03-07T18:19:59.925ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
33741 [8830] Exited with: ExitStatus(unix_wait_status(9))
33742 [8830][2] reports Exit
33743 Mar 07 18:20:03.928 INFO request completed, latency_us: 285, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2a60a21f-2d1a-46b5-b9b0-ab40ebeedb79, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
33744 001/300 Write block 227 len 32768 data: 5 4 4 5 5 6 6 4
33745 002/300 Read block 739 len 24576
33746 003/300 Write block 3613 len 8192 data: 6 5
33747 004/300 Flush
33748 005/300 Read block 3291 len 24576
33749 006/300 Read block 4756 len 40960
33750 007/300 Write block 576 len 12288 data: 5 4 7
33751 008/300 Read block 3641 len 20480
33752 009/300 Read block 1351 len 20480
33753 010/300 Read block 965 len 40960
33754 011/300 Read block 4231 len 4096
33755 012/300 Read block 630 len 40960
33756 013/300 Read block 64 len 8192
33757 014/300 Flush
33758 015/300 Read block 1560 len 8192
33759 016/300 Flush
33760 017/300 Write block 3433 len 4096 data: 5
33761 018/300 Read block 4213 len 36864
33762 019/300 Write block 3054 len 36864 data: 7 8 6 7 7 8 9 10 9
33763 020/300 Write block 4616 len 32768 data: 2 3 4 4 3 4 4 5
33764 021/300 Write block 4973 len 20480 data: 3 3 3 4 4
33765 022/300 Read block 2459 len 40960
33766 023/300 Write block 4064 len 36864 data: 4 3 3 4 5 5 5 7 7
33767 024/300 Read block 35 len 28672
33768 025/300 Write block 609 len 8192 data: 5 5
33769 026/300 Write block 2713 len 16384 data: 7 8 5 7
33770 027/300 Read block 4039 len 20480
33771 028/300 Write block 4684 len 40960 data: 4 4 4 5 7 6 5 7 8 7
33772 029/300 Write block 4500 len 16384 data: 4 4 6 6
33773 030/300 Write block 1440 len 40960 data: 7 5 3 4 3 3 5 5 5 5
33774 031/300 Read block 1680 len 20480
33775 032/300 Flush
33776 033/300 Read block 836 len 12288
33777 034/300 Read block 4169 len 32768
33778 035/300 Read block 235 len 28672
33779 036/300 Read block 283 len 24576
33780 037/300 Write block 4873 len 4096 data: 6
33781 038/300 Write block 4372 len 16384 data: 7 8 6 5
33782 039/300 Write block 1746 len 36864 data: 7 7 7 9 8 6 6 4 3
33783 040/300 Read block 1219 len 36864
33784 041/300 Write block 1273 len 8192 data: 5 6
33785 042/300 Write block 3766 len 28672 data: 3 3 3 3 4 4 5
33786 043/300 Write block 739 len 40960 data: 7 9 8 10 10 11 12 12 11 10
33787 044/300 Read block 2274 len 4096
33788 045/300 Read block 1094 len 16384
33789 046/300 Flush
33790 047/300 Write block 2284 len 4096 data: 7
33791 048/300 Read block 2693 len 36864
33792 049/300 Read block 2253 len 12288
33793 050/300 Write block 3888 len 12288 data: 3 3 4
33794 051/300 Write block 3945 len 24576 data: 6 7 7 9 9 7
33795 052/300 Write block 154 len 32768 data: 4 4 5 6 5 7 8 7
33796 053/300 Read block 1738 len 12288
33797 054/300 Write block 929 len 24576 data: 5 2 3 3 8 7
33798 055/300 Write block 755 len 24576 data: 5 6 6 8 8 8
33799 056/300 Read block 816 len 28672
33800 057/300 Read block 2246 len 32768
33801 058/300 Write block 3484 len 36864 data: 3 4 3 3 4 5 6 6 9
33802 059/300 Read block 3020 len 24576
33803 060/300 Write block 1296 len 28672 data: 6 6 6 7 6 5 4
33804 061/300 Write block 1139 len 36864 data: 10 11 9 9 9 7 7 7 7
33805 062/300 Read block 2036 len 32768
33806 063/300 Flush
33807 064/300 Write block 867 len 24576 data: 6 6 6 7 7 5
33808 065/300 Write block 2353 len 16384 data: 4 5 5 5
33809 066/300 Write block 191 len 4096 data: 9
33810 067/300 Read block 2146 len 40960
33811 068/300 Write block 473 len 32768 data: 4 5 8 6 7 4 4 4
33812 069/300 Read block 2557 len 4096
33813 070/300 Read block 4669 len 28672
33814 071/300 Read block 836 len 28672
33815 072/300 Read block 2329 len 40960
33816 073/300 Write block 1079 len 36864 data: 7 9 8 8 7 6 7 6 5
33817 074/300 Write block 456 len 36864 data: 5 5 3 6 5 4 4 5 7
33818 075/300 Read block 814 len 16384
33819 076/300 Read block 3808 len 20480
33820 077/300 Write block 3172 len 36864 data: 6 5 3 5 4 5 5 5 5
33821 078/300 Write block 4640 len 4096 data: 6
33822 079/300 Read block 26 len 4096
33823 080/300 Write block 1088 len 20480 data: 5 5 3 3 5
33824 081/300 Read block 635 len 4096
33825 082/300 Write block 3056 len 40960 data: 7 8 8 9 10 11 10 11 9 9
33826 083/300 Read block 1381 len 24576
33827 084/300 Read block 1629 len 24576
33828 085/300 Flush
33829 086/300 Read block 264 len 28672
33830 087/300 Flush
33831 088/300 Read block 3604 len 36864
33832 089/300 Read block 266 len 20480
33833 090/300 Write block 388 len 20480 data: 4 5 6 6 7
33834 091/300 Read block 1910 len 12288
33835 092/300 Read block 2423 len 8192
33836 093/300 Flush
33837 094/300 Write block 103 len 12288 data: 5 5 4
33838 095/300 Read block 4732 len 40960
33839 096/300 Read block 1507 len 20480
33840 097/300 Read block 1617 len 28672
33841 098/300 Read block 3913 len 28672
33842 099/300 Read block 3468 len 36864
33843 100/300 Write block 3787 len 4096 data: 6
33844 101/300 Write block 3908 len 40960 data: 7 7 4 4 4 6 7 7 8 7
33845 102/300 Flush
33846 103/300 Read block 3743 len 20480
33847 104/300 Write block 2189 len 24576 data: 5 4 6 6 6 5
33848 105/300 Flush
33849 106/300 Write block 3791 len 12288 data: 3 5 9
33850 107/300 Read block 4776 len 12288
33851 108/300 Read block 4805 len 32768
33852 109/300 Read block 4955 len 16384
33853 110/300 Read block 2650 len 36864
33854 111/300 Read block 768 len 40960
33855 112/300 Write block 3453 len 20480 data: 7 8 6 6 6
33856 113/300 Write block 742 len 32768 data: 11 11 12 13 13 12 11 8
33857 114/300 Read block 533 len 16384
33858 115/300 Flush
33859 116/300 Read block 1803 len 32768
33860 117/300 Read block 609 len 16384
33861 118/300 Read block 2012 len 8192
33862 119/300 Read block 597 len 32768
33863 120/300 Write block 4391 len 4096 data: 6
33864 121/300 Read block 4409 len 16384
33865 122/300 Read block 837 len 24576
33866 123/300 Write block 1105 len 8192 data: 7 7
33867 124/300 Write block 2369 len 24576 data: 3 4 5 4 4 4
33868 125/300 Write block 3950 len 24576 data: 8 6 6 5 5 4
33869 126/300 Write block 2106 len 40960 data: 6 6 8 8 9 9 10 8 9 8
33870 127/300 Write block 290 len 40960 data: 8 7 7 6 4 4 3 3 4 5
33871 128/300 Write block 1696 len 28672 data: 7 7 8 10 6 6 7
33872 129/300 Write block 2352 len 32768 data: 2 5 6 6 6 5 3 3
33873 130/300 Read block 4042 len 8192
33874 131/300 Write block 4621 len 24576 data: 5 5 6 8 6 5
33875 132/300 Write block 4923 len 40960 data: 9 9 7 6 6 5 4 3 3 5
33876 133/300 Read block 2012 len 32768
33877 134/300 Write block 1841 len 24576 data: 7 6 6 5 4 6
33878 135/300 Read block 3877 len 28672
33879 136/300 Write block 2561 len 40960 data: 7 5 4 4 4 3 3 3 5 5
33880 137/300 Read block 4723 len 24576
33881 138/300 Flush
33882 139/300 Write block 3139 len 8192 data: 4 7
33883 140/300 Read block 3049 len 16384
33884 141/300 Read block 1389 len 4096
33885 142/300 Read block 1763 len 4096
33886 143/300 Read block 984 len 28672
33887 144/300 Read block 3965 len 40960
33888 145/300 Read block 4692 len 8192
33889 146/300 Read block 4379 len 24576
33890 147/300 Read block 3372 len 4096
33891 148/300 Write block 3119 len 36864 data: 5 5 4 4 5 7 7 7 7
33892 149/300 Read block 1007 len 36864
33893 150/300 Read block 2154 len 24576
33894 151/300 Read block 2611 len 4096
33895 152/300 Write block 4634 len 20480 data: 6 8 7 7 6
33896 153/300 Read block 1485 len 16384
33897 154/300 Read block 96 len 20480
33898 155/300 Read block 3995 len 40960
33899 156/300 Read block 4986 len 8192
33900 157/300 Write block 2509 len 8192 data: 7 6
33901 158/300 Read block 1250 len 28672
33902 159/300 Read block 179 len 8192
33903 160/300 Read block 3494 len 40960
33904 161/300 Write block 3309 len 24576 data: 7 5 5 4 5 4
33905 162/300 Read block 3804 len 36864
33906 163/300 Write block 2826 len 20480 data: 7 8 6 6 4
33907 164/300 Read block 3937 len 8192
33908 165/300 Write block 1016 len 40960 data: 7 7 7 6 6 7 5 6 6 6
33909 166/300 Write block 2360 len 32768 data: 4 4 4 4 5 5 4 4
33910 167/300 Read block 3487 len 4096
33911 168/300 Write block 1649 len 12288 data: 7 7 5
33912 169/300 Read block 4760 len 36864
33913 170/300 Write block 3345 len 36864 data: 5 5 4 6 6 7 7 8 7
33914 171/300 Read block 880 len 20480
33915 172/300 Write block 3233 len 12288 data: 6 5 6
33916 173/300 Read block 1199 len 20480
33917 174/300 Write block 1169 len 20480 data: 4 3 2 2 2
33918 175/300 Read block 4700 len 16384
33919 176/300 Flush
33920 177/300 Write block 2762 len 28672 data: 5 5 5 5 5 6 6
33921 178/300 Read block 1250 len 8192
33922 179/300 Read block 46 len 8192
33923 180/300 Flush
33924 181/300 Write block 2563 len 24576 data: 5 5 5 4 4 4
33925 182/300 Write block 453 len 20480 data: 3 4 5 6 6
33926 183/300 Write block 1027 len 40960 data: 5 5 6 5 4 4 3 3 3 3
33927 184/300 Write block 2763 len 28672 data: 6 6 6 6 7 7 3
33928 185/300 Flush
33929 186/300 Read block 3922 len 20480
33930 187/300 Write block 2988 len 16384 data: 3 3 3 3
33931 188/300 Write block 3789 len 32768 data: 6 6 4 6 10 8 8 8
33932 189/300 Read block 3045 len 28672
33933 190/300 Flush
33934 191/300 Read block 4746 len 24576
33935 192/300 Read block 1072 len 40960
33936 193/300 Write block 3780 len 24576 data: 4 6 11 9 9 9
33937 194/300 Read block 104 len 4096
33938 195/300 Flush
33939 196/300 Write block 2051 len 16384 data: 7 10 8 7
33940 197/300 Write block 3960 len 24576 data: 9 8 8 8 7 4
33941 198/300 Read block 4650 len 28672
33942 199/300 Flush
33943 200/300 Read block 3340 len 36864
33944 201/300 Read block 1776 len 20480
33945 202/300 Read block 3805 len 4096
33946 203/300 Write block 3725 len 8192 data: 6 4
33947 204/300 Read block 2241 len 20480
33948 205/300 Write block 842 len 40960 data: 4 4 4 4 5 5 5 5 6 6
33949 206/300 Write block 1722 len 32768 data: 7 7 7 6 6 6 7 5
33950 207/300 Write block 4235 len 24576 data: 6 5 7 6 6 7
33951 208/300 Write block 1151 len 16384 data: 5 6 5 5
33952 209/300 Read block 2807 len 32768
33953 210/300 Write block 3536 len 36864 data: 3 4 3 3 3 4 3 3 3
33954 211/300 Read block 2225 len 32768
33955 212/300 Write block 2247 len 40960 data: 5 6 6 6 4 4 4 4 3 3
33956 213/300 Read block 2417 len 16384
33957 214/300 Read block 2430 len 36864
33958 215/300 Write block 706 len 24576 data: 4 3 2 2 2 2
33959 216/300 Read block 4160 len 32768
33960 217/300 Read block 2970 len 16384
33961 218/300 Read block 3517 len 32768
33962 219/300 Write block 4681 len 24576 data: 6 7 5 5 5 5
33963 220/300 Read block 2082 len 16384
33964 221/300 Write block 3998 len 12288 data: 8 7 6
33965 222/300 Read block 3981 len 32768
33966 223/300 Read block 4740 len 24576
33967 224/300 Write block 3218 len 28672 data: 4 6 5 4 5 4 3
33968 225/300 Write block 1444 len 20480 data: 4 4 6 6 6
33969 226/300 Write block 1198 len 20480 data: 5 6 5 7 6
33970 227/300 Write block 3741 len 28672 data: 4 4 5 7 5 5 5
33971 228/300 Read block 263 len 24576
33972 229/300 Flush
33973 230/300 Write block 822 len 32768 data: 6 6 8 11 8 8 6 6
33974 231/300 Write block 1402 len 40960 data: 6 7 7 8 6 5 5 5 5 5
33975 232/300 Read block 3864 len 12288
33976 233/300 Read block 3422 len 28672
33977 234/300 Write block 2300 len 8192 data: 6 6
33978 235/300 Write block 3912 len 24576 data: 5 7 8 8 9 8
33979 236/300 Write block 4577 len 28672 data: 5 6 6 6 6 7 6
33980 237/300 Read block 9 len 12288
33981 238/300 Write block 1335 len 8192 data: 4 4
33982 239/300 Write block 1257 len 28672 data: 4 4 5 4 4 4 5
33983 240/300 Read block 4711 len 16384
33984 241/300 Read block 2129 len 24576
33985 242/300 Write block 1261 len 12288 data: 5 5 6
33986 243/300 Read block 748 len 8192
33987 244/300 Read block 4478 len 12288
33988 245/300 Read block 2983 len 24576
33989 246/300 Read block 889 len 36864
33990 247/300 Read block 1226 len 40960
33991 248/300 Write block 415 len 4096 data: 4
33992 249/300 Read block 897 len 16384
33993 250/300 Write block 4798 len 4096 data: 8
33994 251/300 Write block 1977 len 8192 data: 4 3
33995 252/300 Read block 829 len 36864
33996 253/300 Write block 1123 len 32768 data: 8 7 6 8 9 10 9 7
33997 254/300 Write block 1293 len 20480 data: 7 8 7 7 7
33998 255/300 Read block 2745 len 8192
33999 256/300 Write block 1260 len 40960 data: 5 6 6 7 6 6 6 6 8 7
34000 257/300 Write block 2858 len 28672 data: 5 6 6 6 5 6 7
34001 258/300 Write block 2324 len 40960 data: 2 2 2 3 6 7 7 8 8 9
34002 259/300 Read block 2986 len 36864
34003 260/300 Read block 2969 len 12288
34004 261/300 Read block 291 len 28672
34005 262/300 Read block 2308 len 12288
34006 263/300 Read block 4156 len 4096
34007 264/300 Write block 4970 len 40960 data: 4 4 3 4 4 4 5 5 6 8
34008 265/300 Write block 1486 len 24576 data: 5 7 7 6 5 6
34009 266/300 Read block 3116 len 28672
34010 267/300 Flush
34011 268/300 Read block 2715 len 40960
34012 269/300 Write block 4412 len 28672 data: 6 6 6 6 5 4 5
34013 270/300 Write block 1795 len 20480 data: 3 3 4 7 6
34014 271/300 Read block 4823 len 32768
34015 272/300 Write block 3033 len 8192 data: 5 4
34016 273/300 Read block 2435 len 28672
34017 274/300 Write block 2065 len 28672 data: 6 6 5 3 3 3 3
34018 275/300 Read block 3122 len 4096
34019 276/300 Write block 4705 len 16384 data: 9 9 9 6
34020 277/300 Write block 1861 len 32768 data: 6 4 4 3 2 3 4 3
34021 278/300 Write block 4987 len 16384 data: 5 3 3 2
34022 279/300 Read block 3216 len 40960
34023 280/300 Read block 395 len 32768
34024 281/300 Flush
34025 282/300 Read block 1259 len 8192
34026 283/300 Read block 2560 len 24576
34027 284/300 Write block 856 len 4096 data: 3
34028 285/300 Read block 4199 len 8192
34029 286/300 Read block 2201 len 28672
34030 287/300 Flush
34031 288/300 Read block 4558 len 32768
34032 289/300 Read block 84 len 4096
34033 290/300 Flush
34034 291/300 Write block 302 len 40960 data: 10 10 10 9 9 10 8 7 7 6
34035 292/300 Flush
34036 293/300 Read block 4920 len 20480
34037 294/300 Write block 1388 len 36864 data: 3 5 6 6 4 4 4 4 5
34038 295/300 Read block 4200 len 12288
34039 296/300 Write block 3051 len 28672 data: 4 6 7 8 9 8 9
34040 297/300 Write block 2930 len 4096 data: 5
34041 298/300 Read block 3856 len 32768
34042 299/300 Write block 713 len 40960 data: 3 4 4 4 5 4 3 2 2 2
34043 300/300 Read block 3973 len 40960
34044 301/300 Read block 462 len 16384
34045 Main task has work to do, go find it
34046 got dsc Start(2)
34047 start 2
34048 Mar 07 18:20:04.182 INFO request completed, latency_us: 180, response_code: 204, uri: /start/cid/2, method: POST, req_id: fc79a6c4-c4f2-4693-b736-f4b1a90e9b5e, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
34049 Replay: started 2, returned:Ok(())
34050 ----------------------------------------------------------------
34051 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34052 States: Active Active Offline
34053 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34054 9754 Acked Write 8 Done Done Sent false
34055 9755 Acked Read 6 Done Done Sent false
34056 9756 Acked Write 2 Done Done Sent false
34057 9757 Acked Flush 0 Done Done Sent false
34058 9758 Acked Read 6 Done Done Sent false
34059 9759 Acked Read 10 Done Done Sent false
34060 9760 Acked Write 3 Done Done Sent false
34061 9761 Acked Read 5 Done Done Sent false
34062 9762 Acked Read 5 Done Done Sent false
34063 9763 Acked Read 10 Done Done Sent false
34064 9764 Acked Read 1 Done Done Sent false
34065 9765 Acked Read 10 Done Done Sent false
34066 9766 Acked Read 2 Done Done Sent false
34067 9767 Acked Flush 0 Done Done Sent false
34068 9768 Acked Read 2 Done Done Sent false
34069 9769 Acked Flush 0 Done Done Sent false
34070 9770 Acked Write 1 Done Done Sent false
34071 9771 Acked Read 9 Done Done Sent false
34072 9772 Acked Write 9 Done Done Sent false
34073 9773 Acked Write 8 Done Done Sent false
34074 9774 Acked Write 5 Done Done Sent false
34075 9775 Acked Read 10 Done Done Sent false
34076 9776 Acked Write 9 Done Done Sent false
34077 9777 Acked Read 7 Done Done Sent false
34078 9778 Acked Write 2 Done Done Sent false
34079 9779 Acked Write 4 Done Done Sent false
34080 9780 Acked Read 5 Done Done Sent false
34081 9781 Acked Write 10 Done Done Sent false
34082 9782 Acked Write 4 Done Done Sent false
34083 9783 Acked Write 10 Done Done Sent false
34084 9784 Acked Read 5 Done Done Sent false
34085 9785 Acked Flush 0 Done Done Sent false
34086 9786 Acked Read 3 Done Done Sent false
34087 9787 Acked Read 8 Done Done Sent false
34088 9788 Acked Read 7 Done Done Sent false
34089 9789 Acked Read 6 Done Done Sent false
34090 9790 Acked Write 1 Done Done Sent false
34091 9791 Acked Write 4 Done Done Sent false
34092 9792 Acked Write 9 Done Done Sent false
34093 9793 Acked Read 9 Done Done Sent false
34094 9794 Acked Write 2 Done Done Sent false
34095 9795 Acked Write 7 Done Done Sent false
34096 9796 Acked Write 10 Done Done Sent false
34097 9797 Acked Read 1 Done Done Sent false
34098 9798 Acked Read 4 Done Done Sent false
34099 9799 Acked Flush 0 Done Done Sent false
34100 9800 Acked Write 1 Done Done Sent false
34101 9801 Acked Read 9 Done Done Sent false
34102 9802 Acked Read 3 Done Done Sent false
34103 9803 Acked Write 3 Done Done Sent false
34104 9804 Acked Write 6 Done Done Sent false
34105 9805 Acked Write 8 Done Done Sent false
34106 9806 Acked Read 3 Done Done Sent false
34107 9807 Acked Write 6 Done Done Sent false
34108 9808 Acked Write 6 Done Done Sent false
34109 9809 Acked Read 7 Done Done Sent false
34110 9810 Acked Read 8 Done Done Sent false
34111 9811 Acked Write 9 Done Done Sent false
34112 9812 Acked Read 6 Done Done Sent false
34113 9813 Acked Write 7 Done Done Sent false
34114 9814 Acked Write 9 Done Done Sent false
34115 9815 Acked Read 8 Done Done Sent false
34116 9816 Acked Flush 0 Done Done Sent false
34117 9817 Acked Write 6 Done Done Sent false
34118 9818 Acked Write 4 Done Done Sent false
34119 9819 Acked Write 1 Done Done Sent false
34120 9820 Acked Read 10 Done Done Sent false
34121 9821 Acked Write 8 Done Done Sent false
34122 9822 Acked Read 1 Done Done Sent false
34123 9823 Acked Read 7 Done Done Sent false
34124 9824 Acked Read 7 Done Done Sent false
34125 9825 Acked Read 10 Done Done Sent false
34126 9826 Acked Write 9 Done Done Sent false
34127 9827 Acked Write 9 Done Done Sent false
34128 9828 Acked Read 4 Done Done Sent false
34129 9829 Acked Read 5 Done Done Sent false
34130 9830 Acked Write 9 Done Done Sent false
34131 9831 Acked Write 1 Done Done Sent false
34132 9832 Acked Read 1 Done Done Sent false
34133 9833 Acked Write 5 Done Done Sent false
34134 9834 Acked Read 1 Done Done Sent false
34135 9835 Acked Write 10 Done Done Sent false
34136 9836 Acked Read 6 Done Done Sent false
34137 9837 Acked Read 6 Done Done Sent false
34138 9838 Acked Flush 0 Done Done Sent false
34139 9839 Acked Read 7 Done Done Sent false
34140 9840 Acked Flush 0 Done Done Sent false
34141 9841 Acked Read 9 Done Done Sent false
34142 9842 Acked Read 5 Done Done Sent false
34143 9843 Acked Write 5 Done Done Sent false
34144 9844 Acked Read 3 Done Done Sent false
34145 9845 Acked Read 2 Done Done Sent false
34146 9846 Acked Flush 0 Done Done Sent false
34147 9847 Acked Write 3 Done Done Sent false
34148 9848 Acked Read 10 Done Done Sent false
34149 9849 Acked Read 5 Done Done Sent false
34150 9850 Acked Read 7 Done Done Sent false
34151 9851 Acked Read 7 Done Done Sent false
34152 9852 Acked Read 9 Done Done Sent false
34153 9853 Acked Write 1 Done Done Sent false
34154 9854 Acked Write 10 Done Done Sent false
34155 9855 Acked Flush 0 Done Done Sent false
34156 9856 Acked Read 5 Done Done Sent false
34157 9857 Acked Write 6 Done Done Sent false
34158 9858 Acked Flush 0 Done Done Sent false
34159 9859 Acked Write 3 Done Done Sent false
34160 9860 Acked Read 3 Done Done Sent false
34161 9861 Acked Read 8 Done Done Sent false
34162 9862 Acked Read 4 Done Done Sent false
34163 9863 Acked Read 9 Done Done Sent false
34164 9864 Acked Read 10 Done Done Sent false
34165 9865 Acked Write 5 Done Done Sent false
34166 9866 Acked Write 8 Done Done Sent false
34167 9867 Acked Read 4 Done Done Sent false
34168 9868 Acked Flush 0 Done Done Sent false
34169 9869 Acked Read 8 Done Done Sent false
34170 9870 Acked Read 4 Done Done Sent false
34171 9871 Acked Read 2 Done Done Sent false
34172 9872 Acked Read 8 Done Done Sent false
34173 9873 Acked Write 1 Done Done Sent false
34174 9874 Acked Read 4 Done Done Sent false
34175 9875 Acked Read 6 Done Done Sent false
34176 9876 Acked Write 2 Done Done Sent false
34177 9877 Acked Write 6 Done Done Sent false
34178 9878 Acked Write 6 Done Done Sent false
34179 9879 Acked Write 10 Done Done Sent false
34180 9880 Acked Write 10 Done Done Sent false
34181 9881 Acked Write 7 Done Done Sent false
34182 9882 Acked Write 8 Done Done Sent false
34183 9883 Acked Read 2 Done Done Sent false
34184 9884 Acked Write 6 Done Done Sent false
34185 9885 Acked Write 10 Done Done Sent false
34186 9886 Acked Read 8 Done Done Sent false
34187 9887 Acked Write 6 Done Done Sent false
34188 9888 Acked Read 7 Done Done Sent false
34189 9889 Acked Write 10 Done Done Sent false
34190 9890 Acked Read 6 Done Done Sent false
34191 9891 Acked Flush 0 Done Done Sent false
34192 9892 Acked Write 2 Done Done Sent false
34193 9893 Acked Read 4 Done Done Sent false
34194 9894 Acked Read 1 Done Done Sent false
34195 9895 Acked Read 1 Done Done Sent false
34196 9896 Acked Read 7 Done Done Sent false
34197 9897 Acked Read 10 Done Done Sent false
34198 9898 Acked Read 2 Done Done Sent false
34199 9899 Acked Read 6 Done Done Sent false
34200 9900 Acked Read 1 Done Done Sent false
34201 9901 Acked Write 9 Done Done Sent false
34202 9902 Acked Read 9 Done Done Sent false
34203 9903 Acked Read 6 Done Done Sent false
34204 9904 Acked Read 1 Done Done Sent false
34205 9905 Acked Write 5 Done Done Sent false
34206 9906 Acked Read 4 Done Done Sent false
34207 9907 Acked Read 5 Done Done Sent false
34208 9908 Acked Read 10 Done Done Sent false
34209 9909 Acked Read 2 Done Done Sent false
34210 9910 Acked Write 2 Done Done Sent false
34211 9911 Acked Read 7 Done Done Sent false
34212 9912 Acked Read 2 Done Done Sent false
34213 9913 Acked Read 10 Done Done Sent false
34214 9914 Acked Write 6 Done Done Sent false
34215 9915 Acked Read 9 Done Done Sent false
34216 9916 Acked Write 5 Done Done Sent false
34217 9917 Acked Read 2 Done Done Sent false
34218 9918 Acked Write 10 Done Done Sent false
34219 9919 Acked Write 8 Done Done Sent false
34220 9920 Acked Read 1 Done Done Sent false
34221 9921 Acked Write 3 Done Done Sent false
34222 9922 Acked Read 9 Done Done Sent false
34223 9923 Acked Write 9 Done Done Sent false
34224 9924 Acked Read 5 Done Done Sent false
34225 9925 Acked Write 3 Done Done Sent false
34226 9926 Acked Read 5 Done Done Sent false
34227 9927 Acked Write 5 Done Done Sent false
34228 9928 Acked Read 4 Done Done Sent false
34229 9929 Acked Flush 0 Done Done Sent false
34230 9930 Acked Write 7 Done Done Sent false
34231 9931 Acked Read 2 Done Done Sent false
34232 9932 Acked Read 2 Done Done Sent false
34233 9933 Acked Flush 0 Done Done Sent false
34234 9934 Acked Write 6 Done Done Sent false
34235 9935 Acked Write 5 Done Done Sent false
34236 9936 Acked Write 10 Done Done Sent false
34237 9937 Acked Write 7 Done Done Sent false
34238 9938 Acked Flush 0 Done Done Sent false
34239 9939 Acked Read 5 Done Done Sent false
34240 9940 Acked Write 4 Done Done Sent false
34241 9941 Acked Write 8 Done Done Sent false
34242 9942 Acked Read 7 Done Done Sent false
34243 9943 Acked Flush 0 Done Done Sent false
34244 9944 Acked Read 6 Done Done Sent false
34245 9945 Acked Read 10 Done Done Sent false
34246 9946 Acked Write 6 Done Done Sent false
34247 9947 Acked Read 1 Done Done Sent false
34248 9948 Acked Flush 0 Done Done Sent false
34249 9949 Acked Write 4 Done Done Sent false
34250 9950 Acked Write 6 Done Done Sent false
34251 9951 Acked Read 7 Done Done Sent false
34252 9952 Acked Flush 0 Done Done Sent false
34253 9953 Acked Read 9 Done Done Sent false
34254 9954 Acked Read 5 Done Done Sent false
34255 9955 Acked Read 1 Done Done Sent false
34256 9956 Acked Write 2 Done Done Sent false
34257 9957 Acked Read 5 Done Done Sent false
34258 9958 Acked Write 10 Done Done Sent false
34259 9959 Acked Write 8 Done Done Sent false
34260 9960 Acked Write 6 Done Done Sent false
34261 9961 Acked Write 4 Done Done Sent false
34262 9962 Acked Read 8 Done Done Sent false
34263 9963 Acked Write 9 Done Done Sent false
34264 9964 Acked Read 8 Done Done Sent false
34265 9965 Acked Write 10 Done Done Sent false
34266 9966 Acked Read 4 Done Done Sent false
34267 9967 Acked Read 9 Done Done Sent false
34268 9968 Acked Write 6 Done Done Sent false
34269 9969 Acked Read 8 Done Done Sent false
34270 9970 Acked Read 4 Done Done Sent false
34271 9971 Acked Read 8 Done Done Sent false
34272 9972 Acked Write 6 Done Done Sent false
34273 9973 Acked Read 4 Done Done Sent false
34274 9974 Acked Write 3 Done Done Sent false
34275 9975 Acked Read 8 Done Done Sent false
34276 9976 Acked Read 6 Done Done Sent false
34277 9977 Acked Write 7 Done Done Sent false
34278 9978 Acked Write 5 Done Done Sent false
34279 9979 Acked Write 5 Done Done Sent false
34280 9980 Acked Write 7 Done Done Sent false
34281 9981 Acked Read 6 Done Done Sent false
34282 9982 Acked Flush 0 Done Done Sent false
34283 9983 Acked Write 8 Done Done Sent false
34284 9984 Acked Write 10 Done Done Sent false
34285 9985 Acked Read 3 Done Done Sent false
34286 9986 Acked Read 7 Done Done Sent false
34287 9987 Acked Write 2 Done Done Sent false
34288 9988 Acked Write 6 Done Done Sent false
34289 9989 Acked Write 7 Done Done Sent false
34290 9990 Acked Read 3 Done Done Sent false
34291 9991 Acked Write 2 Done Done Sent false
34292 9992 Acked Write 7 Done Done Sent false
34293 9993 Acked Read 4 Done Done Sent false
34294 9994 Acked Read 6 Done Done Sent false
34295 9995 Acked Write 3 Done Done Sent false
34296 9996 Acked Read 2 Done Done Sent false
34297 9997 Acked Read 3 Done Done Sent false
34298 9998 Acked Read 6 Done Done Sent false
34299 9999 Acked Read 9 Done Done Sent false
34300 10000 Acked Read 10 Done Done Sent false
34301 10001 Acked Write 1 Done Done Sent false
34302 10002 Acked Read 4 Done Done Sent false
34303 10003 Acked Write 1 Done Done Sent false
34304 10004 Acked Write 2 Done Done Sent false
34305 10005 Acked Read 9 Done Done Sent false
34306 10006 Acked Write 8 Done Done Sent false
34307 10007 Acked Write 5 Done Done Sent false
34308 10008 Acked Read 2 Done Done Sent false
34309 10009 Acked Write 10 Done Done Sent false
34310 10010 Acked Write 7 Done Done Sent false
34311 10011 Acked Write 10 Done Done Sent false
34312 10012 Acked Read 9 Done Done Sent false
34313 10013 Acked Read 3 Done Done Sent false
34314 10014 Acked Read 7 Done Done Sent false
34315 10015 Acked Read 3 Done Done Sent false
34316 10016 Acked Read 1 Done Done Sent false
34317 10017 Acked Write 10 Done Done Sent false
34318 10018 Acked Write 6 Done Done Sent false
34319 10019 Acked Read 7 Done Done Sent false
34320 10020 Acked Flush 0 Done Done Sent false
34321 10021 Acked Read 10 Done Done Sent false
34322 10022 Acked Write 7 Done Done Sent false
34323 10023 Acked Write 5 Done Done Sent false
34324 10024 Acked Read 8 Done Done Sent false
34325 10025 Acked Write 2 Done Done Sent false
34326 10026 Acked Read 7 Done Done Sent false
34327 10027 Acked Write 7 Done Done Sent false
34328 10028 Acked Read 1 Done Done Sent false
34329 10029 Acked Write 4 Done Done Sent false
34330 10030 Acked Write 8 Done Done Sent false
34331 10031 Acked Write 4 Done Done Sent false
34332 10032 Acked Read 10 Done Done Sent false
34333 10033 Acked Read 8 Done Done Sent false
34334 10034 Acked Flush 0 Done Done Sent false
34335 10035 Acked Read 2 Done Done Sent false
34336 10036 Acked Read 6 Done Done Sent false
34337 10037 Acked Write 1 Done Done Sent false
34338 10038 Acked Read 2 Done Done Sent false
34339 10039 Acked Read 7 Done Done Sent false
34340 10040 Acked Flush 0 Done Done Sent false
34341 10041 Acked Read 8 Done Done Sent false
34342 10042 Acked Read 1 Done Done Sent false
34343 10043 Acked Flush 0 Done Done Sent false
34344 10044 Acked Write 10 Done Done Sent false
34345 10045 Acked Flush 0 Done Done Sent false
34346 10046 Acked Read 5 Done Done Sent false
34347 10047 Acked Write 9 Done Done Sent false
34348 10048 Acked Read 3 Done Done Sent false
34349 10049 Acked Write 7 Done Done Sent false
34350 10050 Acked Write 1 Done Done Sent false
34351 10051 Acked Read 8 Done Done Sent false
34352 10052 Acked Write 10 Done Done Sent false
34353 10053 Acked Read 10 Done Done Sent false
34354 10054 Acked Read 4 Done Done Sent false
34355 STATES DS:0 DS:1 DS:2 TOTAL
34356 Sent 0 0 301 301
34357 Done 301 301 0 602
34358 Skipped 0 0 0 0
34359 Error 0 0 0 0
34360 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9753))
34361 Downstairs last five completed: 9753 9752 9751 9750 9749
34362 Upstairs last five completed: 10054 10053 10052 10051 10050
34363 CLIENT: Up:0 ds:301 act:2
343642025-03-07T18:20:04.696ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
343652025-03-07T18:20:05.697ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
343662025-03-07T18:20:07.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
343672025-03-07T18:20:07.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34368 ----------------------------------------------------------------
34369 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34370 States: Active Active Offline
34371 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34372 9754 Acked Write 8 Done Done Sent false
34373 9755 Acked Read 6 Done Done Sent false
34374 9756 Acked Write 2 Done Done Sent false
34375 9757 Acked Flush 0 Done Done Sent false
34376 9758 Acked Read 6 Done Done Sent false
34377 9759 Acked Read 10 Done Done Sent false
34378 9760 Acked Write 3 Done Done Sent false
34379 9761 Acked Read 5 Done Done Sent false
34380 9762 Acked Read 5 Done Done Sent false
34381 9763 Acked Read 10 Done Done Sent false
34382 9764 Acked Read 1 Done Done Sent false
34383 9765 Acked Read 10 Done Done Sent false
34384 9766 Acked Read 2 Done Done Sent false
34385 9767 Acked Flush 0 Done Done Sent false
34386 9768 Acked Read 2 Done Done Sent false
34387 9769 Acked Flush 0 Done Done Sent false
34388 9770 Acked Write 1 Done Done Sent false
34389 9771 Acked Read 9 Done Done Sent false
34390 9772 Acked Write 9 Done Done Sent false
34391 9773 Acked Write 8 Done Done Sent false
34392 9774 Acked Write 5 Done Done Sent false
34393 9775 Acked Read 10 Done Done Sent false
34394 9776 Acked Write 9 Done Done Sent false
34395 9777 Acked Read 7 Done Done Sent false
34396 9778 Acked Write 2 Done Done Sent false
34397 9779 Acked Write 4 Done Done Sent false
34398 9780 Acked Read 5 Done Done Sent false
34399 9781 Acked Write 10 Done Done Sent false
34400 9782 Acked Write 4 Done Done Sent false
34401 9783 Acked Write 10 Done Done Sent false
34402 9784 Acked Read 5 Done Done Sent false
34403 9785 Acked Flush 0 Done Done Sent false
34404 9786 Acked Read 3 Done Done Sent false
34405 9787 Acked Read 8 Done Done Sent false
34406 9788 Acked Read 7 Done Done Sent false
34407 9789 Acked Read 6 Done Done Sent false
34408 9790 Acked Write 1 Done Done Sent false
34409 9791 Acked Write 4 Done Done Sent false
34410 9792 Acked Write 9 Done Done Sent false
34411 9793 Acked Read 9 Done Done Sent false
34412 9794 Acked Write 2 Done Done Sent false
34413 9795 Acked Write 7 Done Done Sent false
34414 9796 Acked Write 10 Done Done Sent false
34415 9797 Acked Read 1 Done Done Sent false
34416 9798 Acked Read 4 Done Done Sent false
34417 9799 Acked Flush 0 Done Done Sent false
34418 9800 Acked Write 1 Done Done Sent false
34419 9801 Acked Read 9 Done Done Sent false
34420 9802 Acked Read 3 Done Done Sent false
34421 9803 Acked Write 3 Done Done Sent false
34422 9804 Acked Write 6 Done Done Sent false
34423 9805 Acked Write 8 Done Done Sent false
34424 9806 Acked Read 3 Done Done Sent false
34425 9807 Acked Write 6 Done Done Sent false
34426 9808 Acked Write 6 Done Done Sent false
34427 9809 Acked Read 7 Done Done Sent false
34428 9810 Acked Read 8 Done Done Sent false
34429 9811 Acked Write 9 Done Done Sent false
34430 9812 Acked Read 6 Done Done Sent false
34431 9813 Acked Write 7 Done Done Sent false
34432 9814 Acked Write 9 Done Done Sent false
34433 9815 Acked Read 8 Done Done Sent false
34434 9816 Acked Flush 0 Done Done Sent false
34435 9817 Acked Write 6 Done Done Sent false
34436 9818 Acked Write 4 Done Done Sent false
34437 9819 Acked Write 1 Done Done Sent false
34438 9820 Acked Read 10 Done Done Sent false
34439 9821 Acked Write 8 Done Done Sent false
34440 9822 Acked Read 1 Done Done Sent false
34441 9823 Acked Read 7 Done Done Sent false
34442 9824 Acked Read 7 Done Done Sent false
34443 9825 Acked Read 10 Done Done Sent false
34444 9826 Acked Write 9 Done Done Sent false
34445 9827 Acked Write 9 Done Done Sent false
34446 9828 Acked Read 4 Done Done Sent false
34447 9829 Acked Read 5 Done Done Sent false
34448 9830 Acked Write 9 Done Done Sent false
34449 9831 Acked Write 1 Done Done Sent false
34450 9832 Acked Read 1 Done Done Sent false
34451 9833 Acked Write 5 Done Done Sent false
34452 9834 Acked Read 1 Done Done Sent false
34453 9835 Acked Write 10 Done Done Sent false
34454 9836 Acked Read 6 Done Done Sent false
34455 9837 Acked Read 6 Done Done Sent false
34456 9838 Acked Flush 0 Done Done Sent false
34457 9839 Acked Read 7 Done Done Sent false
34458 9840 Acked Flush 0 Done Done Sent false
34459 9841 Acked Read 9 Done Done Sent false
34460 9842 Acked Read 5 Done Done Sent false
34461 9843 Acked Write 5 Done Done Sent false
34462 9844 Acked Read 3 Done Done Sent false
34463 9845 Acked Read 2 Done Done Sent false
34464 9846 Acked Flush 0 Done Done Sent false
34465 9847 Acked Write 3 Done Done Sent false
34466 9848 Acked Read 10 Done Done Sent false
34467 9849 Acked Read 5 Done Done Sent false
34468 9850 Acked Read 7 Done Done Sent false
34469 9851 Acked Read 7 Done Done Sent false
34470 9852 Acked Read 9 Done Done Sent false
34471 9853 Acked Write 1 Done Done Sent false
34472 9854 Acked Write 10 Done Done Sent false
34473 9855 Acked Flush 0 Done Done Sent false
34474 9856 Acked Read 5 Done Done Sent false
34475 9857 Acked Write 6 Done Done Sent false
34476 9858 Acked Flush 0 Done Done Sent false
34477 9859 Acked Write 3 Done Done Sent false
34478 9860 Acked Read 3 Done Done Sent false
34479 9861 Acked Read 8 Done Done Sent false
34480 9862 Acked Read 4 Done Done Sent false
34481 9863 Acked Read 9 Done Done Sent false
34482 9864 Acked Read 10 Done Done Sent false
34483 9865 Acked Write 5 Done Done Sent false
34484 9866 Acked Write 8 Done Done Sent false
34485 9867 Acked Read 4 Done Done Sent false
34486 9868 Acked Flush 0 Done Done Sent false
34487 9869 Acked Read 8 Done Done Sent false
34488 9870 Acked Read 4 Done Done Sent false
34489 9871 Acked Read 2 Done Done Sent false
34490 9872 Acked Read 8 Done Done Sent false
34491 9873 Acked Write 1 Done Done Sent false
34492 9874 Acked Read 4 Done Done Sent false
34493 9875 Acked Read 6 Done Done Sent false
34494 9876 Acked Write 2 Done Done Sent false
34495 9877 Acked Write 6 Done Done Sent false
34496 9878 Acked Write 6 Done Done Sent false
34497 9879 Acked Write 10 Done Done Sent false
34498 9880 Acked Write 10 Done Done Sent false
34499 9881 Acked Write 7 Done Done Sent false
34500 9882 Acked Write 8 Done Done Sent false
34501 9883 Acked Read 2 Done Done Sent false
34502 9884 Acked Write 6 Done Done Sent false
34503 9885 Acked Write 10 Done Done Sent false
34504 9886 Acked Read 8 Done Done Sent false
34505 9887 Acked Write 6 Done Done Sent false
34506 9888 Acked Read 7 Done Done Sent false
34507 9889 Acked Write 10 Done Done Sent false
34508 9890 Acked Read 6 Done Done Sent false
34509 9891 Acked Flush 0 Done Done Sent false
34510 9892 Acked Write 2 Done Done Sent false
34511 9893 Acked Read 4 Done Done Sent false
34512 9894 Acked Read 1 Done Done Sent false
34513 9895 Acked Read 1 Done Done Sent false
34514 9896 Acked Read 7 Done Done Sent false
34515 9897 Acked Read 10 Done Done Sent false
34516 9898 Acked Read 2 Done Done Sent false
34517 9899 Acked Read 6 Done Done Sent false
34518 9900 Acked Read 1 Done Done Sent false
34519 9901 Acked Write 9 Done Done Sent false
34520 9902 Acked Read 9 Done Done Sent false
34521 9903 Acked Read 6 Done Done Sent false
34522 9904 Acked Read 1 Done Done Sent false
34523 9905 Acked Write 5 Done Done Sent false
34524 9906 Acked Read 4 Done Done Sent false
34525 9907 Acked Read 5 Done Done Sent false
34526 9908 Acked Read 10 Done Done Sent false
34527 9909 Acked Read 2 Done Done Sent false
34528 9910 Acked Write 2 Done Done Sent false
34529 9911 Acked Read 7 Done Done Sent false
34530 9912 Acked Read 2 Done Done Sent false
34531 9913 Acked Read 10 Done Done Sent false
34532 9914 Acked Write 6 Done Done Sent false
34533 9915 Acked Read 9 Done Done Sent false
34534 9916 Acked Write 5 Done Done Sent false
34535 9917 Acked Read 2 Done Done Sent false
34536 9918 Acked Write 10 Done Done Sent false
34537 9919 Acked Write 8 Done Done Sent false
34538 9920 Acked Read 1 Done Done Sent false
34539 9921 Acked Write 3 Done Done Sent false
34540 9922 Acked Read 9 Done Done Sent false
34541 9923 Acked Write 9 Done Done Sent false
34542 9924 Acked Read 5 Done Done Sent false
34543 9925 Acked Write 3 Done Done Sent false
34544 9926 Acked Read 5 Done Done Sent false
34545 9927 Acked Write 5 Done Done Sent false
34546 9928 Acked Read 4 Done Done Sent false
34547 9929 Acked Flush 0 Done Done Sent false
34548 9930 Acked Write 7 Done Done Sent false
34549 9931 Acked Read 2 Done Done Sent false
34550 9932 Acked Read 2 Done Done Sent false
34551 9933 Acked Flush 0 Done Done Sent false
34552 9934 Acked Write 6 Done Done Sent false
34553 9935 Acked Write 5 Done Done Sent false
34554 9936 Acked Write 10 Done Done Sent false
34555 9937 Acked Write 7 Done Done Sent false
34556 9938 Acked Flush 0 Done Done Sent false
34557 9939 Acked Read 5 Done Done Sent false
34558 9940 Acked Write 4 Done Done Sent false
34559 9941 Acked Write 8 Done Done Sent false
34560 9942 Acked Read 7 Done Done Sent false
34561 9943 Acked Flush 0 Done Done Sent false
34562 9944 Acked Read 6 Done Done Sent false
34563 9945 Acked Read 10 Done Done Sent false
34564 9946 Acked Write 6 Done Done Sent false
34565 9947 Acked Read 1 Done Done Sent false
34566 9948 Acked Flush 0 Done Done Sent false
34567 9949 Acked Write 4 Done Done Sent false
34568 9950 Acked Write 6 Done Done Sent false
34569 9951 Acked Read 7 Done Done Sent false
34570 9952 Acked Flush 0 Done Done Sent false
34571 9953 Acked Read 9 Done Done Sent false
34572 9954 Acked Read 5 Done Done Sent false
34573 9955 Acked Read 1 Done Done Sent false
34574 9956 Acked Write 2 Done Done Sent false
34575 9957 Acked Read 5 Done Done Sent false
34576 9958 Acked Write 10 Done Done Sent false
34577 9959 Acked Write 8 Done Done Sent false
34578 9960 Acked Write 6 Done Done Sent false
34579 9961 Acked Write 4 Done Done Sent false
34580 9962 Acked Read 8 Done Done Sent false
34581 9963 Acked Write 9 Done Done Sent false
34582 9964 Acked Read 8 Done Done Sent false
34583 9965 Acked Write 10 Done Done Sent false
34584 9966 Acked Read 4 Done Done Sent false
34585 9967 Acked Read 9 Done Done Sent false
34586 9968 Acked Write 6 Done Done Sent false
34587 9969 Acked Read 8 Done Done Sent false
34588 9970 Acked Read 4 Done Done Sent false
34589 9971 Acked Read 8 Done Done Sent false
34590 9972 Acked Write 6 Done Done Sent false
34591 9973 Acked Read 4 Done Done Sent false
34592 9974 Acked Write 3 Done Done Sent false
34593 9975 Acked Read 8 Done Done Sent false
34594 9976 Acked Read 6 Done Done Sent false
34595 9977 Acked Write 7 Done Done Sent false
34596 9978 Acked Write 5 Done Done Sent false
34597 9979 Acked Write 5 Done Done Sent false
34598 9980 Acked Write 7 Done Done Sent false
34599 9981 Acked Read 6 Done Done Sent false
34600 9982 Acked Flush 0 Done Done Sent false
34601 9983 Acked Write 8 Done Done Sent false
34602 9984 Acked Write 10 Done Done Sent false
34603 9985 Acked Read 3 Done Done Sent false
34604 9986 Acked Read 7 Done Done Sent false
34605 9987 Acked Write 2 Done Done Sent false
34606 9988 Acked Write 6 Done Done Sent false
34607 9989 Acked Write 7 Done Done Sent false
34608 9990 Acked Read 3 Done Done Sent false
34609 9991 Acked Write 2 Done Done Sent false
34610 9992 Acked Write 7 Done Done Sent false
34611 9993 Acked Read 4 Done Done Sent false
34612 9994 Acked Read 6 Done Done Sent false
34613 9995 Acked Write 3 Done Done Sent false
34614 9996 Acked Read 2 Done Done Sent false
34615 9997 Acked Read 3 Done Done Sent false
34616 9998 Acked Read 6 Done Done Sent false
34617 9999 Acked Read 9 Done Done Sent false
34618 10000 Acked Read 10 Done Done Sent false
34619 10001 Acked Write 1 Done Done Sent false
34620 10002 Acked Read 4 Done Done Sent false
34621 10003 Acked Write 1 Done Done Sent false
34622 10004 Acked Write 2 Done Done Sent false
34623 10005 Acked Read 9 Done Done Sent false
34624 10006 Acked Write 8 Done Done Sent false
34625 10007 Acked Write 5 Done Done Sent false
34626 10008 Acked Read 2 Done Done Sent false
34627 10009 Acked Write 10 Done Done Sent false
34628 10010 Acked Write 7 Done Done Sent false
34629 10011 Acked Write 10 Done Done Sent false
34630 10012 Acked Read 9 Done Done Sent false
34631 10013 Acked Read 3 Done Done Sent false
34632 10014 Acked Read 7 Done Done Sent false
34633 10015 Acked Read 3 Done Done Sent false
34634 10016 Acked Read 1 Done Done Sent false
34635 10017 Acked Write 10 Done Done Sent false
34636 10018 Acked Write 6 Done Done Sent false
34637 10019 Acked Read 7 Done Done Sent false
34638 10020 Acked Flush 0 Done Done Sent false
34639 10021 Acked Read 10 Done Done Sent false
34640 10022 Acked Write 7 Done Done Sent false
34641 10023 Acked Write 5 Done Done Sent false
34642 10024 Acked Read 8 Done Done Sent false
34643 10025 Acked Write 2 Done Done Sent false
34644 10026 Acked Read 7 Done Done Sent false
34645 10027 Acked Write 7 Done Done Sent false
34646 10028 Acked Read 1 Done Done Sent false
34647 10029 Acked Write 4 Done Done Sent false
34648 10030 Acked Write 8 Done Done Sent false
34649 10031 Acked Write 4 Done Done Sent false
34650 10032 Acked Read 10 Done Done Sent false
34651 10033 Acked Read 8 Done Done Sent false
34652 10034 Acked Flush 0 Done Done Sent false
34653 10035 Acked Read 2 Done Done Sent false
34654 10036 Acked Read 6 Done Done Sent false
34655 10037 Acked Write 1 Done Done Sent false
34656 10038 Acked Read 2 Done Done Sent false
34657 10039 Acked Read 7 Done Done Sent false
34658 10040 Acked Flush 0 Done Done Sent false
34659 10041 Acked Read 8 Done Done Sent false
34660 10042 Acked Read 1 Done Done Sent false
34661 10043 Acked Flush 0 Done Done Sent false
34662 10044 Acked Write 10 Done Done Sent false
34663 10045 Acked Flush 0 Done Done Sent false
34664 10046 Acked Read 5 Done Done Sent false
34665 10047 Acked Write 9 Done Done Sent false
34666 10048 Acked Read 3 Done Done Sent false
34667 10049 Acked Write 7 Done Done Sent false
34668 10050 Acked Write 1 Done Done Sent false
34669 10051 Acked Read 8 Done Done Sent false
34670 10052 Acked Write 10 Done Done Sent false
34671 10053 Acked Read 10 Done Done Sent false
34672 10054 Acked Read 4 Done Done Sent false
34673 STATES DS:0 DS:1 DS:2 TOTAL
34674 Sent 0 0 301 301
34675 Done 301 301 0 602
34676 Skipped 0 0 0 0
34677 Error 0 0 0 0
34678 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9753))
34679 Downstairs last five completed: 9753 9752 9751 9750 9749
34680 Upstairs last five completed: 10054 10053 10052 10051 10050
34681 CLIENT: Up:0 ds:301 act:2
346822025-03-07T18:20:08.702ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346832025-03-07T18:20:09.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346842025-03-07T18:20:09.927ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346852025-03-07T18:20:09.927ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346862025-03-07T18:20:09.927ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346872025-03-07T18:20:09.927ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346882025-03-07T18:20:09.927ZINFOcrucible: client task is exiting = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34689 [8830] got start action
34690 [8830] I am going to restart
34691 Starting downstairs at port 8830
34692 Make output file at "/tmp/dsc/downstairs-8830.txt"
34693 [8830][2] reports Starting
34694 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1238)
34695 [8830][2] reports Running
346962025-03-07T18:20:10.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
346972025-03-07T18:20:12.166ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
34698 ----------------------------------------------------------------
34699 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34700 States: Active Active Offline
34701 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34702 9754 Acked Write 8 Done Done Sent false
34703 9755 Acked Read 6 Done Done Sent false
34704 9756 Acked Write 2 Done Done Sent false
34705 9757 Acked Flush 0 Done Done Sent false
34706 9758 Acked Read 6 Done Done Sent false
34707 9759 Acked Read 10 Done Done Sent false
34708 9760 Acked Write 3 Done Done Sent false
34709 9761 Acked Read 5 Done Done Sent false
34710 9762 Acked Read 5 Done Done Sent false
34711 9763 Acked Read 10 Done Done Sent false
34712 9764 Acked Read 1 Done Done Sent false
34713 9765 Acked Read 10 Done Done Sent false
34714 9766 Acked Read 2 Done Done Sent false
34715 9767 Acked Flush 0 Done Done Sent false
34716 9768 Acked Read 2 Done Done Sent false
34717 9769 Acked Flush 0 Done Done Sent false
34718 9770 Acked Write 1 Done Done Sent false
34719 9771 Acked Read 9 Done Done Sent false
34720 9772 Acked Write 9 Done Done Sent false
34721 9773 Acked Write 8 Done Done Sent false
34722 9774 Acked Write 5 Done Done Sent false
34723 9775 Acked Read 10 Done Done Sent false
34724 9776 Acked Write 9 Done Done Sent false
34725 9777 Acked Read 7 Done Done Sent false
34726 9778 Acked Write 2 Done Done Sent false
34727 9779 Acked Write 4 Done Done Sent false
34728 9780 Acked Read 5 Done Done Sent false
34729 9781 Acked Write 10 Done Done Sent false
34730 9782 Acked Write 4 Done Done Sent false
34731 9783 Acked Write 10 Done Done Sent false
34732 9784 Acked Read 5 Done Done Sent false
34733 9785 Acked Flush 0 Done Done Sent false
34734 9786 Acked Read 3 Done Done Sent false
34735 9787 Acked Read 8 Done Done Sent false
34736 9788 Acked Read 7 Done Done Sent false
34737 9789 Acked Read 6 Done Done Sent false
34738 9790 Acked Write 1 Done Done Sent false
34739 9791 Acked Write 4 Done Done Sent false
34740 9792 Acked Write 9 Done Done Sent false
34741 9793 Acked Read 9 Done Done Sent false
34742 9794 Acked Write 2 Done Done Sent false
34743 9795 Acked Write 7 Done Done Sent false
34744 9796 Acked Write 10 Done Done Sent false
34745 9797 Acked Read 1 Done Done Sent false
34746 9798 Acked Read 4 Done Done Sent false
34747 9799 Acked Flush 0 Done Done Sent false
34748 9800 Acked Write 1 Done Done Sent false
34749 9801 Acked Read 9 Done Done Sent false
34750 9802 Acked Read 3 Done Done Sent false
34751 9803 Acked Write 3 Done Done Sent false
34752 9804 Acked Write 6 Done Done Sent false
34753 9805 Acked Write 8 Done Done Sent false
34754 9806 Acked Read 3 Done Done Sent false
34755 9807 Acked Write 6 Done Done Sent false
34756 9808 Acked Write 6 Done Done Sent false
34757 9809 Acked Read 7 Done Done Sent false
34758 9810 Acked Read 8 Done Done Sent false
34759 9811 Acked Write 9 Done Done Sent false
34760 9812 Acked Read 6 Done Done Sent false
34761 9813 Acked Write 7 Done Done Sent false
34762 9814 Acked Write 9 Done Done Sent false
34763 9815 Acked Read 8 Done Done Sent false
34764 9816 Acked Flush 0 Done Done Sent false
34765 9817 Acked Write 6 Done Done Sent false
34766 9818 Acked Write 4 Done Done Sent false
34767 9819 Acked Write 1 Done Done Sent false
34768 9820 Acked Read 10 Done Done Sent false
34769 9821 Acked Write 8 Done Done Sent false
34770 9822 Acked Read 1 Done Done Sent false
34771 9823 Acked Read 7 Done Done Sent false
34772 9824 Acked Read 7 Done Done Sent false
34773 9825 Acked Read 10 Done Done Sent false
34774 9826 Acked Write 9 Done Done Sent false
34775 9827 Acked Write 9 Done Done Sent false
34776 9828 Acked Read 4 Done Done Sent false
34777 9829 Acked Read 5 Done Done Sent false
34778 9830 Acked Write 9 Done Done Sent false
34779 9831 Acked Write 1 Done Done Sent false
34780 9832 Acked Read 1 Done Done Sent false
34781 9833 Acked Write 5 Done Done Sent false
34782 9834 Acked Read 1 Done Done Sent false
34783 9835 Acked Write 10 Done Done Sent false
34784 9836 Acked Read 6 Done Done Sent false
34785 9837 Acked Read 6 Done Done Sent false
34786 9838 Acked Flush 0 Done Done Sent false
34787 9839 Acked Read 7 Done Done Sent false
34788 9840 Acked Flush 0 Done Done Sent false
34789 9841 Acked Read 9 Done Done Sent false
34790 9842 Acked Read 5 Done Done Sent false
34791 9843 Acked Write 5 Done Done Sent false
34792 9844 Acked Read 3 Done Done Sent false
34793 9845 Acked Read 2 Done Done Sent false
34794 9846 Acked Flush 0 Done Done Sent false
34795 9847 Acked Write 3 Done Done Sent false
34796 9848 Acked Read 10 Done Done Sent false
34797 9849 Acked Read 5 Done Done Sent false
34798 9850 Acked Read 7 Done Done Sent false
34799 9851 Acked Read 7 Done Done Sent false
34800 9852 Acked Read 9 Done Done Sent false
34801 9853 Acked Write 1 Done Done Sent false
34802 9854 Acked Write 10 Done Done Sent false
34803 9855 Acked Flush 0 Done Done Sent false
34804 9856 Acked Read 5 Done Done Sent false
34805 9857 Acked Write 6 Done Done Sent false
34806 9858 Acked Flush 0 Done Done Sent false
34807 9859 Acked Write 3 Done Done Sent false
34808 9860 Acked Read 3 Done Done Sent false
34809 9861 Acked Read 8 Done Done Sent false
34810 9862 Acked Read 4 Done Done Sent false
34811 9863 Acked Read 9 Done Done Sent false
34812 9864 Acked Read 10 Done Done Sent false
34813 9865 Acked Write 5 Done Done Sent false
34814 9866 Acked Write 8 Done Done Sent false
34815 9867 Acked Read 4 Done Done Sent false
34816 9868 Acked Flush 0 Done Done Sent false
34817 9869 Acked Read 8 Done Done Sent false
34818 9870 Acked Read 4 Done Done Sent false
34819 9871 Acked Read 2 Done Done Sent false
34820 9872 Acked Read 8 Done Done Sent false
34821 9873 Acked Write 1 Done Done Sent false
34822 9874 Acked Read 4 Done Done Sent false
34823 9875 Acked Read 6 Done Done Sent false
34824 9876 Acked Write 2 Done Done Sent false
34825 9877 Acked Write 6 Done Done Sent false
34826 9878 Acked Write 6 Done Done Sent false
34827 9879 Acked Write 10 Done Done Sent false
34828 9880 Acked Write 10 Done Done Sent false
34829 9881 Acked Write 7 Done Done Sent false
34830 9882 Acked Write 8 Done Done Sent false
34831 9883 Acked Read 2 Done Done Sent false
34832 9884 Acked Write 6 Done Done Sent false
34833 9885 Acked Write 10 Done Done Sent false
34834 9886 Acked Read 8 Done Done Sent false
34835 9887 Acked Write 6 Done Done Sent false
34836 9888 Acked Read 7 Done Done Sent false
34837 9889 Acked Write 10 Done Done Sent false
34838 9890 Acked Read 6 Done Done Sent false
34839 9891 Acked Flush 0 Done Done Sent false
34840 9892 Acked Write 2 Done Done Sent false
34841 9893 Acked Read 4 Done Done Sent false
34842 9894 Acked Read 1 Done Done Sent false
34843 9895 Acked Read 1 Done Done Sent false
34844 9896 Acked Read 7 Done Done Sent false
34845 9897 Acked Read 10 Done Done Sent false
34846 9898 Acked Read 2 Done Done Sent false
34847 9899 Acked Read 6 Done Done Sent false
34848 9900 Acked Read 1 Done Done Sent false
34849 9901 Acked Write 9 Done Done Sent false
34850 9902 Acked Read 9 Done Done Sent false
34851 9903 Acked Read 6 Done Done Sent false
34852 9904 Acked Read 1 Done Done Sent false
34853 9905 Acked Write 5 Done Done Sent false
34854 9906 Acked Read 4 Done Done Sent false
34855 9907 Acked Read 5 Done Done Sent false
34856 9908 Acked Read 10 Done Done Sent false
34857 9909 Acked Read 2 Done Done Sent false
34858 9910 Acked Write 2 Done Done Sent false
34859 9911 Acked Read 7 Done Done Sent false
34860 9912 Acked Read 2 Done Done Sent false
34861 9913 Acked Read 10 Done Done Sent false
34862 9914 Acked Write 6 Done Done Sent false
34863 9915 Acked Read 9 Done Done Sent false
34864 9916 Acked Write 5 Done Done Sent false
34865 9917 Acked Read 2 Done Done Sent false
34866 9918 Acked Write 10 Done Done Sent false
34867 9919 Acked Write 8 Done Done Sent false
34868 9920 Acked Read 1 Done Done Sent false
34869 9921 Acked Write 3 Done Done Sent false
34870 9922 Acked Read 9 Done Done Sent false
34871 9923 Acked Write 9 Done Done Sent false
34872 9924 Acked Read 5 Done Done Sent false
34873 9925 Acked Write 3 Done Done Sent false
34874 9926 Acked Read 5 Done Done Sent false
34875 9927 Acked Write 5 Done Done Sent false
34876 9928 Acked Read 4 Done Done Sent false
34877 9929 Acked Flush 0 Done Done Sent false
34878 9930 Acked Write 7 Done Done Sent false
34879 9931 Acked Read 2 Done Done Sent false
34880 9932 Acked Read 2 Done Done Sent false
34881 9933 Acked Flush 0 Done Done Sent false
34882 9934 Acked Write 6 Done Done Sent false
34883 9935 Acked Write 5 Done Done Sent false
34884 9936 Acked Write 10 Done Done Sent false
34885 9937 Acked Write 7 Done Done Sent false
34886 9938 Acked Flush 0 Done Done Sent false
34887 9939 Acked Read 5 Done Done Sent false
34888 9940 Acked Write 4 Done Done Sent false
34889 9941 Acked Write 8 Done Done Sent false
34890 9942 Acked Read 7 Done Done Sent false
34891 9943 Acked Flush 0 Done Done Sent false
34892 9944 Acked Read 6 Done Done Sent false
34893 9945 Acked Read 10 Done Done Sent false
34894 9946 Acked Write 6 Done Done Sent false
34895 9947 Acked Read 1 Done Done Sent false
34896 9948 Acked Flush 0 Done Done Sent false
34897 9949 Acked Write 4 Done Done Sent false
34898 9950 Acked Write 6 Done Done Sent false
34899 9951 Acked Read 7 Done Done Sent false
34900 9952 Acked Flush 0 Done Done Sent false
34901 9953 Acked Read 9 Done Done Sent false
34902 9954 Acked Read 5 Done Done Sent false
34903 9955 Acked Read 1 Done Done Sent false
34904 9956 Acked Write 2 Done Done Sent false
34905 9957 Acked Read 5 Done Done Sent false
34906 9958 Acked Write 10 Done Done Sent false
34907 9959 Acked Write 8 Done Done Sent false
34908 9960 Acked Write 6 Done Done Sent false
34909 9961 Acked Write 4 Done Done Sent false
34910 9962 Acked Read 8 Done Done Sent false
34911 9963 Acked Write 9 Done Done Sent false
34912 9964 Acked Read 8 Done Done Sent false
34913 9965 Acked Write 10 Done Done Sent false
34914 9966 Acked Read 4 Done Done Sent false
34915 9967 Acked Read 9 Done Done Sent false
34916 9968 Acked Write 6 Done Done Sent false
34917 9969 Acked Read 8 Done Done Sent false
34918 9970 Acked Read 4 Done Done Sent false
34919 9971 Acked Read 8 Done Done Sent false
34920 9972 Acked Write 6 Done Done Sent false
34921 9973 Acked Read 4 Done Done Sent false
34922 9974 Acked Write 3 Done Done Sent false
34923 9975 Acked Read 8 Done Done Sent false
34924 9976 Acked Read 6 Done Done Sent false
34925 9977 Acked Write 7 Done Done Sent false
34926 9978 Acked Write 5 Done Done Sent false
34927 9979 Acked Write 5 Done Done Sent false
34928 9980 Acked Write 7 Done Done Sent false
34929 9981 Acked Read 6 Done Done Sent false
34930 9982 Acked Flush 0 Done Done Sent false
34931 9983 Acked Write 8 Done Done Sent false
34932 9984 Acked Write 10 Done Done Sent false
34933 9985 Acked Read 3 Done Done Sent false
34934 9986 Acked Read 7 Done Done Sent false
34935 9987 Acked Write 2 Done Done Sent false
34936 9988 Acked Write 6 Done Done Sent false
34937 9989 Acked Write 7 Done Done Sent false
34938 9990 Acked Read 3 Done Done Sent false
34939 9991 Acked Write 2 Done Done Sent false
34940 9992 Acked Write 7 Done Done Sent false
34941 9993 Acked Read 4 Done Done Sent false
34942 9994 Acked Read 6 Done Done Sent false
34943 9995 Acked Write 3 Done Done Sent false
34944 9996 Acked Read 2 Done Done Sent false
34945 9997 Acked Read 3 Done Done Sent false
34946 9998 Acked Read 6 Done Done Sent false
34947 9999 Acked Read 9 Done Done Sent false
34948 10000 Acked Read 10 Done Done Sent false
34949 10001 Acked Write 1 Done Done Sent false
34950 10002 Acked Read 4 Done Done Sent false
34951 10003 Acked Write 1 Done Done Sent false
34952 10004 Acked Write 2 Done Done Sent false
34953 10005 Acked Read 9 Done Done Sent false
34954 10006 Acked Write 8 Done Done Sent false
34955 10007 Acked Write 5 Done Done Sent false
34956 10008 Acked Read 2 Done Done Sent false
34957 10009 Acked Write 10 Done Done Sent false
34958 10010 Acked Write 7 Done Done Sent false
34959 10011 Acked Write 10 Done Done Sent false
34960 10012 Acked Read 9 Done Done Sent false
34961 10013 Acked Read 3 Done Done Sent false
34962 10014 Acked Read 7 Done Done Sent false
34963 10015 Acked Read 3 Done Done Sent false
34964 10016 Acked Read 1 Done Done Sent false
34965 10017 Acked Write 10 Done Done Sent false
34966 10018 Acked Write 6 Done Done Sent false
34967 10019 Acked Read 7 Done Done Sent false
34968 10020 Acked Flush 0 Done Done Sent false
34969 10021 Acked Read 10 Done Done Sent false
34970 10022 Acked Write 7 Done Done Sent false
34971 10023 Acked Write 5 Done Done Sent false
34972 10024 Acked Read 8 Done Done Sent false
34973 10025 Acked Write 2 Done Done Sent false
34974 10026 Acked Read 7 Done Done Sent false
34975 10027 Acked Write 7 Done Done Sent false
34976 10028 Acked Read 1 Done Done Sent false
34977 10029 Acked Write 4 Done Done Sent false
34978 10030 Acked Write 8 Done Done Sent false
34979 10031 Acked Write 4 Done Done Sent false
34980 10032 Acked Read 10 Done Done Sent false
34981 10033 Acked Read 8 Done Done Sent false
34982 10034 Acked Flush 0 Done Done Sent false
34983 10035 Acked Read 2 Done Done Sent false
34984 10036 Acked Read 6 Done Done Sent false
34985 10037 Acked Write 1 Done Done Sent false
34986 10038 Acked Read 2 Done Done Sent false
34987 10039 Acked Read 7 Done Done Sent false
34988 10040 Acked Flush 0 Done Done Sent false
34989 10041 Acked Read 8 Done Done Sent false
34990 10042 Acked Read 1 Done Done Sent false
34991 10043 Acked Flush 0 Done Done Sent false
34992 10044 Acked Write 10 Done Done Sent false
34993 10045 Acked Flush 0 Done Done Sent false
34994 10046 Acked Read 5 Done Done Sent false
34995 10047 Acked Write 9 Done Done Sent false
34996 10048 Acked Read 3 Done Done Sent false
34997 10049 Acked Write 7 Done Done Sent false
34998 10050 Acked Write 1 Done Done Sent false
34999 10051 Acked Read 8 Done Done Sent false
35000 10052 Acked Write 10 Done Done Sent false
35001 10053 Acked Read 10 Done Done Sent false
35002 10054 Acked Read 4 Done Done Sent false
35003 STATES DS:0 DS:1 DS:2 TOTAL
35004 Sent 0 0 301 301
35005 Done 301 301 0 602
35006 Skipped 0 0 0 0
35007 Error 0 0 0 0
35008 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9753))
35009 Downstairs last five completed: 9753 9752 9751 9750 9749
35010 Upstairs last five completed: 10054 10053 10052 10051 10050
35011 CLIENT: Up:0 ds:301 act:2
350122025-03-07T18:20:12.706ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
350132025-03-07T18:20:13.708ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
350142025-03-07T18:20:14.711ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
350152025-03-07T18:20:15.713ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
35016 ----------------------------------------------------------------
35017 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35018 States: Active Active Offline
35019 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35020 9754 Acked Write 8 Done Done Sent false
35021 9755 Acked Read 6 Done Done Sent false
35022 9756 Acked Write 2 Done Done Sent false
35023 9757 Acked Flush 0 Done Done Sent false
35024 9758 Acked Read 6 Done Done Sent false
35025 9759 Acked Read 10 Done Done Sent false
35026 9760 Acked Write 3 Done Done Sent false
35027 9761 Acked Read 5 Done Done Sent false
35028 9762 Acked Read 5 Done Done Sent false
35029 9763 Acked Read 10 Done Done Sent false
35030 9764 Acked Read 1 Done Done Sent false
35031 9765 Acked Read 10 Done Done Sent false
35032 9766 Acked Read 2 Done Done Sent false
35033 9767 Acked Flush 0 Done Done Sent false
35034 9768 Acked Read 2 Done Done Sent false
35035 9769 Acked Flush 0 Done Done Sent false
35036 9770 Acked Write 1 Done Done Sent false
35037 9771 Acked Read 9 Done Done Sent false
35038 9772 Acked Write 9 Done Done Sent false
35039 9773 Acked Write 8 Done Done Sent false
35040 9774 Acked Write 5 Done Done Sent false
35041 9775 Acked Read 10 Done Done Sent false
35042 9776 Acked Write 9 Done Done Sent false
35043 9777 Acked Read 7 Done Done Sent false
35044 9778 Acked Write 2 Done Done Sent false
35045 9779 Acked Write 4 Done Done Sent false
35046 9780 Acked Read 5 Done Done Sent false
35047 9781 Acked Write 10 Done Done Sent false
35048 9782 Acked Write 4 Done Done Sent false
35049 9783 Acked Write 10 Done Done Sent false
35050 9784 Acked Read 5 Done Done Sent false
35051 9785 Acked Flush 0 Done Done Sent false
35052 9786 Acked Read 3 Done Done Sent false
35053 9787 Acked Read 8 Done Done Sent false
35054 9788 Acked Read 7 Done Done Sent false
35055 9789 Acked Read 6 Done Done Sent false
35056 9790 Acked Write 1 Done Done Sent false
35057 9791 Acked Write 4 Done Done Sent false
35058 9792 Acked Write 9 Done Done Sent false
35059 9793 Acked Read 9 Done Done Sent false
35060 9794 Acked Write 2 Done Done Sent false
35061 9795 Acked Write 7 Done Done Sent false
35062 9796 Acked Write 10 Done Done Sent false
35063 9797 Acked Read 1 Done Done Sent false
35064 9798 Acked Read 4 Done Done Sent false
35065 9799 Acked Flush 0 Done Done Sent false
35066 9800 Acked Write 1 Done Done Sent false
35067 9801 Acked Read 9 Done Done Sent false
35068 9802 Acked Read 3 Done Done Sent false
35069 9803 Acked Write 3 Done Done Sent false
35070 9804 Acked Write 6 Done Done Sent false
35071 9805 Acked Write 8 Done Done Sent false
35072 9806 Acked Read 3 Done Done Sent false
35073 9807 Acked Write 6 Done Done Sent false
35074 9808 Acked Write 6 Done Done Sent false
35075 9809 Acked Read 7 Done Done Sent false
35076 9810 Acked Read 8 Done Done Sent false
35077 9811 Acked Write 9 Done Done Sent false
35078 9812 Acked Read 6 Done Done Sent false
35079 9813 Acked Write 7 Done Done Sent false
35080 9814 Acked Write 9 Done Done Sent false
35081 9815 Acked Read 8 Done Done Sent false
35082 9816 Acked Flush 0 Done Done Sent false
35083 9817 Acked Write 6 Done Done Sent false
35084 9818 Acked Write 4 Done Done Sent false
35085 9819 Acked Write 1 Done Done Sent false
35086 9820 Acked Read 10 Done Done Sent false
35087 9821 Acked Write 8 Done Done Sent false
35088 9822 Acked Read 1 Done Done Sent false
35089 9823 Acked Read 7 Done Done Sent false
35090 9824 Acked Read 7 Done Done Sent false
35091 9825 Acked Read 10 Done Done Sent false
35092 9826 Acked Write 9 Done Done Sent false
35093 9827 Acked Write 9 Done Done Sent false
35094 9828 Acked Read 4 Done Done Sent false
35095 9829 Acked Read 5 Done Done Sent false
35096 9830 Acked Write 9 Done Done Sent false
35097 9831 Acked Write 1 Done Done Sent false
35098 9832 Acked Read 1 Done Done Sent false
35099 9833 Acked Write 5 Done Done Sent false
35100 9834 Acked Read 1 Done Done Sent false
35101 9835 Acked Write 10 Done Done Sent false
35102 9836 Acked Read 6 Done Done Sent false
35103 9837 Acked Read 6 Done Done Sent false
35104 9838 Acked Flush 0 Done Done Sent false
35105 9839 Acked Read 7 Done Done Sent false
35106 9840 Acked Flush 0 Done Done Sent false
35107 9841 Acked Read 9 Done Done Sent false
35108 9842 Acked Read 5 Done Done Sent false
35109 9843 Acked Write 5 Done Done Sent false
35110 9844 Acked Read 3 Done Done Sent false
35111 9845 Acked Read 2 Done Done Sent false
35112 9846 Acked Flush 0 Done Done Sent false
35113 9847 Acked Write 3 Done Done Sent false
35114 9848 Acked Read 10 Done Done Sent false
35115 9849 Acked Read 5 Done Done Sent false
35116 9850 Acked Read 7 Done Done Sent false
35117 9851 Acked Read 7 Done Done Sent false
35118 9852 Acked Read 9 Done Done Sent false
35119 9853 Acked Write 1 Done Done Sent false
35120 9854 Acked Write 10 Done Done Sent false
35121 9855 Acked Flush 0 Done Done Sent false
35122 9856 Acked Read 5 Done Done Sent false
35123 9857 Acked Write 6 Done Done Sent false
35124 9858 Acked Flush 0 Done Done Sent false
35125 9859 Acked Write 3 Done Done Sent false
35126 9860 Acked Read 3 Done Done Sent false
35127 9861 Acked Read 8 Done Done Sent false
35128 9862 Acked Read 4 Done Done Sent false
35129 9863 Acked Read 9 Done Done Sent false
35130 9864 Acked Read 10 Done Done Sent false
35131 9865 Acked Write 5 Done Done Sent false
35132 9866 Acked Write 8 Done Done Sent false
35133 9867 Acked Read 4 Done Done Sent false
35134 9868 Acked Flush 0 Done Done Sent false
35135 9869 Acked Read 8 Done Done Sent false
35136 9870 Acked Read 4 Done Done Sent false
35137 9871 Acked Read 2 Done Done Sent false
35138 9872 Acked Read 8 Done Done Sent false
35139 9873 Acked Write 1 Done Done Sent false
35140 9874 Acked Read 4 Done Done Sent false
35141 9875 Acked Read 6 Done Done Sent false
35142 9876 Acked Write 2 Done Done Sent false
35143 9877 Acked Write 6 Done Done Sent false
35144 9878 Acked Write 6 Done Done Sent false
35145 9879 Acked Write 10 Done Done Sent false
35146 9880 Acked Write 10 Done Done Sent false
35147 9881 Acked Write 7 Done Done Sent false
35148 9882 Acked Write 8 Done Done Sent false
35149 9883 Acked Read 2 Done Done Sent false
35150 9884 Acked Write 6 Done Done Sent false
35151 9885 Acked Write 10 Done Done Sent false
35152 9886 Acked Read 8 Done Done Sent false
35153 9887 Acked Write 6 Done Done Sent false
35154 9888 Acked Read 7 Done Done Sent false
35155 9889 Acked Write 10 Done Done Sent false
35156 9890 Acked Read 6 Done Done Sent false
35157 9891 Acked Flush 0 Done Done Sent false
35158 9892 Acked Write 2 Done Done Sent false
35159 9893 Acked Read 4 Done Done Sent false
35160 9894 Acked Read 1 Done Done Sent false
35161 9895 Acked Read 1 Done Done Sent false
35162 9896 Acked Read 7 Done Done Sent false
35163 9897 Acked Read 10 Done Done Sent false
35164 9898 Acked Read 2 Done Done Sent false
35165 9899 Acked Read 6 Done Done Sent false
35166 9900 Acked Read 1 Done Done Sent false
35167 9901 Acked Write 9 Done Done Sent false
35168 9902 Acked Read 9 Done Done Sent false
35169 9903 Acked Read 6 Done Done Sent false
35170 9904 Acked Read 1 Done Done Sent false
35171 9905 Acked Write 5 Done Done Sent false
35172 9906 Acked Read 4 Done Done Sent false
35173 9907 Acked Read 5 Done Done Sent false
35174 9908 Acked Read 10 Done Done Sent false
35175 9909 Acked Read 2 Done Done Sent false
35176 9910 Acked Write 2 Done Done Sent false
35177 9911 Acked Read 7 Done Done Sent false
35178 9912 Acked Read 2 Done Done Sent false
35179 9913 Acked Read 10 Done Done Sent false
35180 9914 Acked Write 6 Done Done Sent false
35181 9915 Acked Read 9 Done Done Sent false
35182 9916 Acked Write 5 Done Done Sent false
35183 9917 Acked Read 2 Done Done Sent false
35184 9918 Acked Write 10 Done Done Sent false
35185 9919 Acked Write 8 Done Done Sent false
35186 9920 Acked Read 1 Done Done Sent false
35187 9921 Acked Write 3 Done Done Sent false
35188 9922 Acked Read 9 Done Done Sent false
35189 9923 Acked Write 9 Done Done Sent false
35190 9924 Acked Read 5 Done Done Sent false
35191 9925 Acked Write 3 Done Done Sent false
35192 9926 Acked Read 5 Done Done Sent false
35193 9927 Acked Write 5 Done Done Sent false
35194 9928 Acked Read 4 Done Done Sent false
35195 9929 Acked Flush 0 Done Done Sent false
35196 9930 Acked Write 7 Done Done Sent false
35197 9931 Acked Read 2 Done Done Sent false
35198 9932 Acked Read 2 Done Done Sent false
35199 9933 Acked Flush 0 Done Done Sent false
35200 9934 Acked Write 6 Done Done Sent false
35201 9935 Acked Write 5 Done Done Sent false
35202 9936 Acked Write 10 Done Done Sent false
35203 9937 Acked Write 7 Done Done Sent false
35204 9938 Acked Flush 0 Done Done Sent false
35205 9939 Acked Read 5 Done Done Sent false
35206 9940 Acked Write 4 Done Done Sent false
35207 9941 Acked Write 8 Done Done Sent false
35208 9942 Acked Read 7 Done Done Sent false
35209 9943 Acked Flush 0 Done Done Sent false
35210 9944 Acked Read 6 Done Done Sent false
35211 9945 Acked Read 10 Done Done Sent false
35212 9946 Acked Write 6 Done Done Sent false
35213 9947 Acked Read 1 Done Done Sent false
35214 9948 Acked Flush 0 Done Done Sent false
35215 9949 Acked Write 4 Done Done Sent false
35216 9950 Acked Write 6 Done Done Sent false
35217 9951 Acked Read 7 Done Done Sent false
35218 9952 Acked Flush 0 Done Done Sent false
35219 9953 Acked Read 9 Done Done Sent false
35220 9954 Acked Read 5 Done Done Sent false
35221 9955 Acked Read 1 Done Done Sent false
35222 9956 Acked Write 2 Done Done Sent false
35223 9957 Acked Read 5 Done Done Sent false
35224 9958 Acked Write 10 Done Done Sent false
35225 9959 Acked Write 8 Done Done Sent false
35226 9960 Acked Write 6 Done Done Sent false
35227 9961 Acked Write 4 Done Done Sent false
35228 9962 Acked Read 8 Done Done Sent false
35229 9963 Acked Write 9 Done Done Sent false
35230 9964 Acked Read 8 Done Done Sent false
35231 9965 Acked Write 10 Done Done Sent false
35232 9966 Acked Read 4 Done Done Sent false
35233 9967 Acked Read 9 Done Done Sent false
35234 9968 Acked Write 6 Done Done Sent false
35235 9969 Acked Read 8 Done Done Sent false
35236 9970 Acked Read 4 Done Done Sent false
35237 9971 Acked Read 8 Done Done Sent false
35238 9972 Acked Write 6 Done Done Sent false
35239 9973 Acked Read 4 Done Done Sent false
35240 9974 Acked Write 3 Done Done Sent false
35241 9975 Acked Read 8 Done Done Sent false
35242 9976 Acked Read 6 Done Done Sent false
35243 9977 Acked Write 7 Done Done Sent false
35244 9978 Acked Write 5 Done Done Sent false
35245 9979 Acked Write 5 Done Done Sent false
35246 9980 Acked Write 7 Done Done Sent false
35247 9981 Acked Read 6 Done Done Sent false
35248 9982 Acked Flush 0 Done Done Sent false
35249 9983 Acked Write 8 Done Done Sent false
35250 9984 Acked Write 10 Done Done Sent false
35251 9985 Acked Read 3 Done Done Sent false
35252 9986 Acked Read 7 Done Done Sent false
35253 9987 Acked Write 2 Done Done Sent false
35254 9988 Acked Write 6 Done Done Sent false
35255 9989 Acked Write 7 Done Done Sent false
35256 9990 Acked Read 3 Done Done Sent false
35257 9991 Acked Write 2 Done Done Sent false
35258 9992 Acked Write 7 Done Done Sent false
35259 9993 Acked Read 4 Done Done Sent false
35260 9994 Acked Read 6 Done Done Sent false
35261 9995 Acked Write 3 Done Done Sent false
35262 9996 Acked Read 2 Done Done Sent false
35263 9997 Acked Read 3 Done Done Sent false
35264 9998 Acked Read 6 Done Done Sent false
35265 9999 Acked Read 9 Done Done Sent false
35266 10000 Acked Read 10 Done Done Sent false
35267 10001 Acked Write 1 Done Done Sent false
35268 10002 Acked Read 4 Done Done Sent false
35269 10003 Acked Write 1 Done Done Sent false
35270 10004 Acked Write 2 Done Done Sent false
35271 10005 Acked Read 9 Done Done Sent false
35272 10006 Acked Write 8 Done Done Sent false
35273 10007 Acked Write 5 Done Done Sent false
35274 10008 Acked Read 2 Done Done Sent false
35275 10009 Acked Write 10 Done Done Sent false
35276 10010 Acked Write 7 Done Done Sent false
35277 10011 Acked Write 10 Done Done Sent false
35278 10012 Acked Read 9 Done Done Sent false
35279 10013 Acked Read 3 Done Done Sent false
35280 10014 Acked Read 7 Done Done Sent false
35281 10015 Acked Read 3 Done Done Sent false
35282 10016 Acked Read 1 Done Done Sent false
35283 10017 Acked Write 10 Done Done Sent false
35284 10018 Acked Write 6 Done Done Sent false
35285 10019 Acked Read 7 Done Done Sent false
35286 10020 Acked Flush 0 Done Done Sent false
35287 10021 Acked Read 10 Done Done Sent false
35288 10022 Acked Write 7 Done Done Sent false
35289 10023 Acked Write 5 Done Done Sent false
35290 10024 Acked Read 8 Done Done Sent false
35291 10025 Acked Write 2 Done Done Sent false
35292 10026 Acked Read 7 Done Done Sent false
35293 10027 Acked Write 7 Done Done Sent false
35294 10028 Acked Read 1 Done Done Sent false
35295 10029 Acked Write 4 Done Done Sent false
35296 10030 Acked Write 8 Done Done Sent false
35297 10031 Acked Write 4 Done Done Sent false
35298 10032 Acked Read 10 Done Done Sent false
35299 10033 Acked Read 8 Done Done Sent false
35300 10034 Acked Flush 0 Done Done Sent false
35301 10035 Acked Read 2 Done Done Sent false
35302 10036 Acked Read 6 Done Done Sent false
35303 10037 Acked Write 1 Done Done Sent false
35304 10038 Acked Read 2 Done Done Sent false
35305 10039 Acked Read 7 Done Done Sent false
35306 10040 Acked Flush 0 Done Done Sent false
35307 10041 Acked Read 8 Done Done Sent false
35308 10042 Acked Read 1 Done Done Sent false
35309 10043 Acked Flush 0 Done Done Sent false
35310 10044 Acked Write 10 Done Done Sent false
35311 10045 Acked Flush 0 Done Done Sent false
35312 10046 Acked Read 5 Done Done Sent false
35313 10047 Acked Write 9 Done Done Sent false
35314 10048 Acked Read 3 Done Done Sent false
35315 10049 Acked Write 7 Done Done Sent false
35316 10050 Acked Write 1 Done Done Sent false
35317 10051 Acked Read 8 Done Done Sent false
35318 10052 Acked Write 10 Done Done Sent false
35319 10053 Acked Read 10 Done Done Sent false
35320 10054 Acked Read 4 Done Done Sent false
35321 STATES DS:0 DS:1 DS:2 TOTAL
35322 Sent 0 0 301 301
35323 Done 301 301 0 602
35324 Skipped 0 0 0 0
35325 Error 0 0 0 0
35326 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(9753))
35327 Downstairs last five completed: 9753 9752 9751 9750 9749
35328 Upstairs last five completed: 10054 10053 10052 10051 10050
35329 CLIENT: Up:0 ds:301 act:2
353302025-03-07T18:20:17.167ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353312025-03-07T18:20:17.717ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353322025-03-07T18:20:18.719ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353332025-03-07T18:20:19.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353342025-03-07T18:20:19.928ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353352025-03-07T18:20:19.929ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33951) = io task client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353362025-03-07T18:20:19.930ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353372025-03-07T18:20:19.930ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353382025-03-07T18:20:19.930ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353392025-03-07T18:20:19.930ZINFOcrucible: send last flush ID to this DS: Some(JobId(9753)) = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353402025-03-07T18:20:19.930ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353412025-03-07T18:20:19.930ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9753)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353422025-03-07T18:20:19.930ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9753)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
35343 ----------------------------------------------------------------
35344 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
35345 States: Active Active Active
35346 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35347 10046 Acked Read 5 Done Done Done true
35348 10047 Acked Write 9 Done Done Done true
35349 10048 Acked Read 3 Done Done Done true
35350 10049 Acked Write 7 Done Done Done true
35351 10050 Acked Write 1 Done Done Done true
35352 10051 Acked Read 8 Done Done Done true
35353 10052 Acked Write 10 Done Done Done true
35354 10053 Acked Read 10 Done Done Done true
35355 10054 Acked Read 4 Done Done Done true
35356 STATES DS:0 DS:1 DS:2 TOTAL
35357 Sent 0 0 0 0
35358 Done 9 9 9 27
35359 Skipped 0 0 0 0
35360 Error 0 0 0 0
35361 Last Flush: Some(JobId(10045)) Some(JobId(10045)) Some(JobId(10045))
35362 Downstairs last five completed: 10045 10044 10043 10042 10041
35363 Upstairs last five completed: 10054 10053 10052 10051 10050
35364 CLIENT: Up:0 ds:9 act:3
35365 ----------------------------------------------------------------
35366 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35367 Downstairs last five completed: 10055 10054 10053 10052 10051
35368 Upstairs last five completed: 10055 10054 10053 10052 10051
35369 CLIENT: Up:0 ds:0 act:3
35370 Replay: All jobs finished
35371 Main task has work to do, go find it
35372 got dsc Stop(1)
35373 stop 1
35374 Mar 07 18:20:24.194 INFO request completed, latency_us[8820] Got stop action so:false kr:false
35375 : 376, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4f1e671e-ab9f-4521-a1e0-52b228bb31c1, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
353762025-03-07T18:20:24.195ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353772025-03-07T18:20:24.195ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
353782025-03-07T18:20:24.195ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
35379 Mar 07 18:20:24.195 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/1, method: GET{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:20:24.195646058Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1208,"":"io task","client":"1","":"downstairs","session_id":"5e9909a1-dc0d-4821-9383-88a904f8ac68"}
35380 , req_id: 2c70b398-1d42-473b-994d-8494cb221c90, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
35381 [8820] Exited with: ExitStatus(unix_wait_status(9))
35382 [8820][1] reports Exit
35383 Mar 07 18:20:28.198 INFO request completed, latency_us: 302, response_code: 200, uri: /state/cid/1, method: GET, req_id: 76d00b41-e857-4f62-8e06-fce22fb8f361, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
35384 001/300 Flush
35385 002/300 Write block 4887 len 28672 data: 7 7 5 5 5 5 4
35386 003/300 Read block 2834 len 24576
35387 004/300 Write block 4596 len 36864 data: 5 7 6 6 7 8 8 7 6
35388 005/300 Write block 1598 len 36864 data: 5 6 7 8 8 8 7 6 4
35389 006/300 Read block 4407 len 28672
35390 007/300 Read block 1626 len 40960
35391 008/300 Read block 1727 len 36864
35392 009/300 Write block 4528 len 36864 data: 3 2 4 4 5 7 7 8 9
35393 010/300 Read block 1679 len 24576
35394 011/300 Write block 2219 len 8192 data: 5 6
35395 012/300 Write block 3050 len 40960 data: 5 5 7 8 9 10 9 10 9 10
35396 013/300 Read block 4492 len 24576
35397 014/300 Flush
35398 015/300 Write block 1404 len 28672 data: 8 9 7 6 6 6 6
35399 016/300 Read block 1772 len 24576
35400 017/300 Flush
35401 018/300 Write block 3074 len 8192 data: 6 5
35402 019/300 Read block 3121 len 24576
35403 020/300 Write block 3690 len 32768 data: 5 5 6 7 6 6 7 4
35404 021/300 Read block 3249 len 32768
35405 022/300 Read block 4409 len 16384
35406 023/300 Read block 4461 len 4096
35407 024/300 Read block 4830 len 16384
35408 025/300 Flush
35409 026/300 Write block 1729 len 32768 data: 6 4 5 5 6 6 6 9
35410 027/300 Read block 4541 len 32768
35411 028/300 Write block 1945 len 28672 data: 5 5 5 5 6 6 6
35412 029/300 Read block 4707 len 24576
35413 030/300 Read block 3091 len 36864
35414 031/300 Flush
35415 032/300 Read block 1717 len 32768
35416 033/300 Read block 423 len 8192
35417 034/300 Read block 978 len 20480
35418 035/300 Write block 2238 len 36864 data: 6 6 6 7 7 8 9 6 6
35419 036/300 Read block 4141 len 16384
35420 037/300 Read block 2399 len 20480
35421 038/300 Write block 670 len 28672 data: 7 7 9 8 6 6 9
35422 039/300 Read block 1988 len 32768
35423 040/300 Read block 2458 len 16384
35424 041/300 Write block 19 len 32768 data: 7 5 5 7 7 6 6 6
35425 042/300 Flush
35426 043/300 Write block 142 len 28672 data: 2 2 5 6 7 7 8
35427 044/300 Flush
35428 045/300 Write block 4152 len 24576 data: 3 3 3 4 6 7
35429 046/300 Write block 4357 len 4096 data: 4
35430 047/300 Read block 4683 len 8192
35431 048/300 Read block 2161 len 20480
35432 049/300 Read block 3502 len 16384
35433 050/300 Flush
35434 051/300 Write block 2408 len 24576 data: 5 6 7 6 5 4
35435 052/300 Write block 1695 len 16384 data: 8 8 8 9
35436 053/300 Read block 1022 len 40960
35437 054/300 Write block 4606 len 20480 data: 6 7 8 9 10
35438 055/300 Read block 3316 len 16384
35439 056/300 Write block 1481 len 40960 data: 6 6 6 6 4 6 8 8 7 6
35440 057/300 Read block 704 len 12288
35441 058/300 Write block 2915 len 20480 data: 7 7 7 8 7
35442 059/300 Read block 2768 len 20480
35443 060/300 Write block 305 len 20480 data: 10 10 11 9 8
35444 061/300 Write block 1771 len 40960 data: 6 7 7 7 8 8 9 9 8 7
35445 062/300 Write block 2074 len 20480 data: 4 4 4 5 4
35446 063/300 Write block 1011 len 20480 data: 8 7 8 9 8
35447 064/300 Write block 4765 len 16384 data: 7 7 7 8
35448 065/300 Read block 4987 len 4096
35449 066/300 Write block 4347 len 20480 data: 8 8 8 8 8
35450 067/300 Write block 2649 len 28672 data: 5 6 6 6 7 8 10
35451 068/300 Write block 1742 len 36864 data: 8 8 7 9 8 8 8 10 9
35452 069/300 Write block 3025 len 36864 data: 5 6 5 5 3 5 5 5 6
35453 070/300 Read block 4024 len 8192
35454 071/300 Write block 754 len 4096 data: 5
35455 072/300 Write block 233 len 32768 data: 7 5 6 6 5 4 5 5
35456 073/300 Read block 63 len 12288
35457 074/300 Write block 1243 len 16384 data: 4 6 6 6
35458 075/300 Write block 4597 len 12288 data: 8 7 7
35459 076/300 Read block 1824 len 24576
35460 077/300 Flush
35461 078/300 Read block 4796 len 20480
35462 079/300 Read block 1861 len 4096
35463 080/300 Write block 1508 len 4096 data: 4
35464 081/300 Write block 1487 len 24576 data: 9 9 8 7 7 5
35465 082/300 Flush
35466 083/300 Read block 1325 len 12288
35467 084/300 Read block 1491 len 24576
35468 085/300 Read block 800 len 40960
35469 086/300 Write block 1022 len 8192 data: 6 7
35470 087/300 Read block 1474 len 28672
35471 088/300 Read block 2919 len 28672
35472 089/300 Write block 517 len 24576 data: 4 4 3 2 4 4
35473 090/300 Read block 3696 len 32768
35474 091/300 Write block 728 len 28672 data: 6 7 7 9 8 9 7
35475 092/300 Read block 2342 len 24576
35476 093/300 Write block 4859 len 36864 data: 6 4 3 5 8 8 9 10 7
35477 094/300 Read block 4660 len 28672
35478 095/300 Read block 4480 len 12288
35479 096/300 Read block 1173 len 36864
35480 097/300 Read block 2665 len 8192
35481 098/300 Read block 3487 len 20480
35482 099/300 Write block 1625 len 36864 data: 10 10 8 8 9 7 5 3 4
35483 100/300 Write block 4687 len 40960 data: 6 8 7 6 8 9 8 9 10 10
35484 101/300 Read block 505 len 36864
35485 102/300 Write block 807 len 16384 data: 7 5 5 4
35486 103/300 Write block 1363 len 40960 data: 9 8 7 8 7 5 4 4 4 4
35487 104/300 Read block 3892 len 20480
35488 105/300 Read block 879 len 8192
35489 106/300 Flush
35490 107/300 Read block 3604 len 40960
35491 108/300 Flush
35492 109/300 Flush
35493 110/300 Write block 1912 len 32768 data: 8 7 6 8 8 11 11 11
35494 111/300 Read block 3178 len 16384
35495 112/300 Write block 45 len 16384 data: 7 9 7 7
35496 113/300 Read block 933 len 12288
35497 114/300 Read block 1685 len 20480
35498 115/300 Write block 3740 len 32768 data: 4 5 5 6 8 6 6 6
35499 116/300 Flush
35500 117/300 Read block 2132 len 12288
35501 118/300 Write block 2278 len 24576 data: 4 6 6 7 7 7
35502 119/300 Read block 1953 len 28672
35503 120/300 Flush
35504 121/300 Write block 3111 len 40960 data: 7 6 8 8 8 8 8 6 6 6
35505 122/300 Flush
35506 123/300 Read block 4472 len 20480
35507 124/300 Read block 3469 len 12288
35508 125/300 Flush
35509 126/300 Write block 34 len 28672 data: 7 7 4 3 3 3 2
35510 127/300 Read block 1750 len 28672
35511 128/300 Read block 721 len 40960
35512 129/300 Flush
35513 130/300 Write block 2398 len 28672 data: 4 4 4 4 4 3 3
35514 131/300 Read block 2309 len 20480
35515 132/300 Read block 4599 len 24576
35516 133/300 Write block 2472 len 20480 data: 7 4 4 4 4
35517 134/300 Write block 3274 len 16384 data: 3 3 5 6
35518 135/300 Read block 4384 len 40960
35519 136/300 Read block 3864 len 4096
35520 137/300 Write block 2071 len 24576 data: 4 4 4 5 5 5
35521 138/300 Write block 4605 len 16384 data: 5 7 8 9
35522 139/300 Read block 2859 len 24576
35523 140/300 Write block 3348 len 40960 data: 7 7 8 8 9 8 6 6 7 8
35524 141/300 Read block 3174 len 4096
35525 142/300 Read block 520 len 8192
35526 143/300 Read block 403 len 28672
35527 144/300 Read block 2971 len 40960
35528 145/300 Read block 2456 len 24576
35529 146/300 Read block 1217 len 40960
35530 147/300 Read block 3371 len 12288
35531 148/300 Write block 3526 len 8192 data: 6 6
35532 149/300 Read block 837 len 40960
35533 150/300 Write block 2437 len 12288 data: 6 5 5
35534 151/300 Read block 1695 len 16384
35535 152/300 Read block 4459 len 16384
35536 153/300 Read block 1775 len 36864
35537 154/300 Read block 4200 len 12288
35538 155/300 Write block 239 len 36864 data: 6 6 4 5 7 9 8 7 5
35539 156/300 Write block 1649 len 40960 data: 8 8 6 5 5 6 5 6 7 7
35540 157/300 Read block 4242 len 16384
35541 158/300 Write block 3293 len 36864 data: 9 9 8 8 8 6 4 4 5
35542 159/300 Write block 4777 len 4096 data: 7
35543 160/300 Read block 2173 len 8192
35544 161/300 Read block 3411 len 28672
35545 162/300 Read block 3536 len 40960
35546 163/300 Write block 1683 len 16384 data: 4 4 5 7
35547 164/300 Read block 3623 len 24576
35548 165/300 Read block 2747 len 4096
35549 166/300 Read block 1947 len 24576
35550 167/300 Write block 2157 len 8192 data: 5 4
35551 168/300 Write block 3447 len 8192 data: 8 8
35552 169/300 Read block 3641 len 28672
35553 170/300 Flush
35554 171/300 Write block 1231 len 12288 data: 4 4 4
35555 172/300 Write block 1431 len 32768 data: 5 6 6 9 9 9 8 8
35556 173/300 Read block 748 len 4096
35557 174/300 Write block 1678 len 12288 data: 4 3 4
35558 175/300 Flush
35559 176/300 Write block 634 len 16384 data: 8 7 8 7
35560 177/300 Flush
35561 178/300 Write block 751 len 36864 data: 6 4 5 6 6 7 7 9 9
35562 179/300 Read block 3292 len 8192
35563 180/300 Write block 116 len 8192 data: 6 6
35564 181/300 Read block 1127 len 4096
35565 182/300 Read block 3199 len 32768
35566 183/300 Read block 1105 len 24576
35567 184/300 Write block 1056 len 12288 data: 4 3 4
35568 185/300 Write block 975 len 20480 data: 4 4 6 6 6
35569 186/300 Write block 1740 len 28672 data: 6 7 9 9 8 10 9
35570 187/300 Flush
35571 188/300 Flush
35572 189/300 Read block 1012 len 8192
35573 190/300 Read block 3897 len 4096
35574 191/300 Read block 1386 len 8192
35575 192/300 Write block 511 len 28672 data: 5 5 5 6 6 5 5
35576 193/300 Write block 1623 len 8192 data: 7 9
35577 194/300 Read block 333 len 36864
35578 195/300 Write block 1081 len 16384 data: 9 9 8 7
35579 196/300 Write block 4199 len 12288 data: 7 7 7
35580 197/300 Read block 1817 len 20480
35581 198/300 Read block 1738 len 8192
35582 199/300 Read block 3462 len 20480
35583 200/300 Read block 405 len 16384
35584 201/300 Read block 605 len 40960
35585 202/300 Write block 4979 len 40960 data: 9 9 7 8 7 8 8 6 6 4
35586 203/300 Write block 3929 len 8192 data: 8 7
35587 204/300 Read block 330 len 32768
35588 205/300 Flush
35589 206/300 Write block 4177 len 24576 data: 9 8 7 8 7 6
35590 207/300 Write block 3613 len 40960 data: 7 6 6 5 5 5 4 5 3 4
35591 208/300 Read block 1646 len 4096
35592 209/300 Write block 1782 len 8192 data: 7 7
35593 210/300 Read block 1220 len 4096
35594 211/300 Write block 3237 len 40960 data: 7 4 3 5 7 7 7 7 6 6
35595 212/300 Write block 1188 len 24576 data: 7 8 7 7 6 5
35596 213/300 Read block 1600 len 40960
35597 214/300 Flush
35598 215/300 Write block 1545 len 36864 data: 6 7 8 7 6 6 5 6 6
35599 216/300 Read block 730 len 8192
35600 217/300 Read block 3889 len 20480
35601 218/300 Write block 3347 len 24576 data: 5 8 8 9 9 10
35602 219/300 Write block 1302 len 24576 data: 5 4 4 4 5 7
35603 220/300 Read block 1293 len 40960
35604 221/300 Write block 706 len 16384 data: 5 4 3 3
35605 222/300 Write block 4003 len 24576 data: 5 5 5 5 5 5
35606 223/300 Write block 3529 len 4096 data: 7
35607 224/300 Read block 2607 len 12288
35608 225/300 Write block 135 len 24576 data: 6 6 6 5 4 3
35609 226/300 Flush
35610 227/300 Write block 3224 len 28672 data: 4 3 2 2 2 2 3
35611 228/300 Read block 3958 len 20480
35612 229/300 Flush
35613 230/300 Write block 792 len 16384 data: 6 4 4 3
35614 231/300 Write block 1012 len 12288 data: 8 9 10
35615 232/300 Read block 2537 len 12288
35616 233/300 Write block 4146 len 40960 data: 2 3 3 3 3 3 4 4 4 5
35617 234/300 Read block 2302 len 8192
35618 235/300 Read block 2567 len 12288
35619 236/300 Write block 3695 len 16384 data: 7 8 5 4
35620 237/300 Read block 4018 len 16384
35621 238/300 Write block 4145 len 24576 data: 2 3 4 4 4 4
35622 239/300 Read block 2759 len 12288
35623 240/300 Write block 949 len 28672 data: 4 4 4 5 5 6 5
35624 241/300 Flush
35625 242/300 Read block 3645 len 36864
35626 243/300 Write block 1976 len 28672 data: 4 5 4 4 3 5 4
35627 244/300 Read block 2881 len 20480
35628 245/300 Read block 3441 len 8192
35629 246/300 Write block 114 len 12288 data: 6 6 7
35630 247/300 Read block 1351 len 16384
35631 248/300 Read block 1171 len 32768
35632 249/300 Write block 3520 len 4096 data: 4
35633 250/300 Read block 3816 len 16384
35634 251/300 Read block 1040 len 32768
35635 252/300 Flush
35636 253/300 Write block 1566 len 40960 data: 9 13 15 14 11 9 9 9 7 5
35637 254/300 Read block 4953 len 4096
35638 255/300 Write block 787 len 20480 data: 8 9 9 8 7
35639 256/300 Read block 605 len 40960
35640 257/300 Read block 1437 len 12288
35641 258/300 Read block 366 len 40960
35642 259/300 Read block 3288 len 32768
35643 260/300 Read block 1641 len 32768
35644 261/300 Write block 2863 len 4096 data: 7
35645 262/300 Read block 4032 len 32768
35646 263/300 Read block 457 len 16384
35647 264/300 Flush
35648 265/300 Write block 336 len 12288 data: 3 4 5
35649 266/300 Write block 4960 len 40960 data: 10 10 7 8 8 9 6 4 4 5
35650 267/300 Write block 619 len 16384 data: 7 7 7 9
35651 268/300 Read block 2232 len 28672
35652 269/300 Flush
35653 270/300 Write block 217 len 40960 data: 6 6 7 8 8 9 8 6 6 7
35654 271/300 Read block 201 len 28672
35655 272/300 Read block 4155 len 12288
35656 273/300 Read block 2802 len 28672
35657 274/300 Read block 736 len 24576
35658 275/300 Write block 3085 len 36864 data: 9 9 8 7 5 5 6 6 5
35659 276/300 Write block 4698 len 28672 data: 9 9 9 8 9 9 10
35660 277/300 Flush
35661 278/300 Write block 4275 len 20480 data: 7 7 8 8 9
35662 279/300 Flush
35663 280/300 Write block 1651 len 28672 data: 7 6 6 7 6 7 8
35664 281/300 Write block 4640 len 16384 data: 7 6 4 5
35665 282/300 Read block 4607 len 4096
35666 283/300 Read block 2295 len 24576
35667 284/300 Write block 807 len 20480 data: 8 6 6 5 5
35668 285/300 Read block 4210 len 20480
35669 286/300 Write block 561 len 40960 data: 6 6 7 8 8 6 7 6 5 4
35670 287/300 Read block 3450 len 24576
35671 288/300 Write block 4696 len 40960 data: 11 9 10 10 10 9 10 10 11 10
35672 289/300 Read block 1514 len 40960
35673 290/300 Read block 177 len 20480
35674 291/300 Flush
35675 292/300 Write block 4570 len 16384 data: 4 3 3 3
35676 293/300 Read block 1063 len 32768
35677 294/300 Read block 2573 len 16384
35678 295/300 Write block 2179 len 36864 data: 4 5 6 6 6 6 6 7 8
35679 296/300 Write block 1763 len 40960 data: 2 2 3 6 6 5 5 5 7 8
35680 297/300 Flush
35681 298/300 Read block 1546 len 32768
35682 299/300 Write block 601 len 28672 data: 7 9 10 10 9 8 7
35683 300/300 Read block 4338 len 4096
35684 301/300 Read block 3898 len 28672
35685 Main task has work to do, go find it
35686 got dsc Start(1)
35687 start 1
35688 Mar 07 18:20:28.447 INFO request completed, latency_us: 184, response_code: 204, uri: /start/cid/1, method: POST, req_id: ba13732d-5352-448f-b6e3-fcb0c5626c0d, remote_addr: 127.0.0.1:64322, local_addr: 127.0.0.1:9998
35689 Replay: started 1, returned:Ok(())
35690 ----------------------------------------------------------------
35691 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35692 States: Active Offline Active
35693 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35694 10056 Acked Flush 0 Done Sent Done false
35695 10057 Acked Write 7 Done Sent Done false
35696 10058 Acked Read 6 Done Sent Done false
35697 10059 Acked Write 9 Done Sent Done false
35698 10060 Acked Write 9 Done Sent Done false
35699 10061 Acked Read 7 Done Sent Done false
35700 10062 Acked Read 10 Done Sent Done false
35701 10063 Acked Read 9 Done Sent Done false
35702 10064 Acked Write 9 Done Sent Done false
35703 10065 Acked Read 6 Done Sent Done false
35704 10066 Acked Write 2 Done Sent Done false
35705 10067 Acked Write 10 Done Sent Done false
35706 10068 Acked Read 6 Done Sent Done false
35707 10069 Acked Flush 0 Done Sent Done false
35708 10070 Acked Write 7 Done Sent Done false
35709 10071 Acked Read 6 Done Sent Done false
35710 10072 Acked Flush 0 Done Sent Done false
35711 10073 Acked Write 2 Done Sent Done false
35712 10074 Acked Read 6 Done Sent Done false
35713 10075 Acked Write 8 Done Sent Done false
35714 10076 Acked Read 8 Done Sent Done false
35715 10077 Acked Read 4 Done Sent Done false
35716 10078 Acked Read 1 Done Sent Done false
35717 10079 Acked Read 4 Done Sent Done false
35718 10080 Acked Flush 0 Done Sent Done false
35719 10081 Acked Write 8 Done Sent Done false
35720 10082 Acked Read 8 Done Sent Done false
35721 10083 Acked Write 7 Done Sent Done false
35722 10084 Acked Read 6 Done Sent Done false
35723 10085 Acked Read 9 Done Sent Done false
35724 10086 Acked Flush 0 Done Sent Done false
35725 10087 Acked Read 8 Done Sent Done false
35726 10088 Acked Read 2 Done Sent Done false
35727 10089 Acked Read 5 Done Sent Done false
35728 10090 Acked Write 9 Done Sent Done false
35729 10091 Acked Read 4 Done Sent Done false
35730 10092 Acked Read 5 Done Sent Done false
35731 10093 Acked Write 7 Done Sent Done false
35732 10094 Acked Read 8 Done Sent Done false
35733 10095 Acked Read 4 Done Sent Done false
35734 10096 Acked Write 8 Done Sent Done false
35735 10097 Acked Flush 0 Done Sent Done false
35736 10098 Acked Write 7 Done Sent Done false
35737 10099 Acked Flush 0 Done Sent Done false
35738 10100 Acked Write 6 Done Sent Done false
35739 10101 Acked Write 1 Done Sent Done false
35740 10102 Acked Read 2 Done Sent Done false
35741 10103 Acked Read 5 Done Sent Done false
35742 10104 Acked Read 4 Done Sent Done false
35743 10105 Acked Flush 0 Done Sent Done false
35744 10106 Acked Write 6 Done Sent Done false
35745 10107 Acked Write 4 Done Sent Done false
35746 10108 Acked Read 10 Done Sent Done false
35747 10109 Acked Write 5 Done Sent Done false
35748 10110 Acked Read 4 Done Sent Done false
35749 10111 Acked Write 10 Done Sent Done false
35750 10112 Acked Read 3 Done Sent Done false
35751 10113 Acked Write 5 Done Sent Done false
35752 10114 Acked Read 5 Done Sent Done false
35753 10115 Acked Write 5 Done Sent Done false
35754 10116 Acked Write 10 Done Sent Done false
35755 10117 Acked Write 5 Done Sent Done false
35756 10118 Acked Write 5 Done Sent Done false
35757 10119 Acked Write 4 Done Sent Done false
35758 10120 Acked Read 1 Done Sent Done false
35759 10121 Acked Write 5 Done Sent Done false
35760 10122 Acked Write 7 Done Sent Done false
35761 10123 Acked Write 9 Done Sent Done false
35762 10124 Acked Write 9 Done Sent Done false
35763 10125 Acked Read 2 Done Sent Done false
35764 10126 Acked Write 1 Done Sent Done false
35765 10127 Acked Write 8 Done Sent Done false
35766 10128 Acked Read 3 Done Sent Done false
35767 10129 Acked Write 4 Done Sent Done false
35768 10130 Acked Write 3 Done Sent Done false
35769 10131 Acked Read 6 Done Sent Done false
35770 10132 Acked Flush 0 Done Sent Done false
35771 10133 Acked Read 5 Done Sent Done false
35772 10134 Acked Read 1 Done Sent Done false
35773 10135 Acked Write 1 Done Sent Done false
35774 10136 Acked Write 6 Done Sent Done false
35775 10137 Acked Flush 0 Done Sent Done false
35776 10138 Acked Read 3 Done Sent Done false
35777 10139 Acked Read 6 Done Sent Done false
35778 10140 Acked Read 10 Done Sent Done false
35779 10141 Acked Write 2 Done Sent Done false
35780 10142 Acked Read 7 Done Sent Done false
35781 10143 Acked Read 7 Done Sent Done false
35782 10144 Acked Write 6 Done Sent Done false
35783 10145 Acked Read 8 Done Sent Done false
35784 10146 Acked Write 7 Done Sent Done false
35785 10147 Acked Read 6 Done Sent Done false
35786 10148 Acked Write 9 Done Sent Done false
35787 10149 Acked Read 7 Done Sent Done false
35788 10150 Acked Read 3 Done Sent Done false
35789 10151 Acked Read 9 Done Sent Done false
35790 10152 Acked Read 2 Done Sent Done false
35791 10153 Acked Read 5 Done Sent Done false
35792 10154 Acked Write 9 Done Sent Done false
35793 10155 Acked Write 10 Done Sent Done false
35794 10156 Acked Read 9 Done Sent Done false
35795 10157 Acked Write 4 Done Sent Done false
35796 10158 Acked Write 10 Done Sent Done false
35797 10159 Acked Read 5 Done Sent Done false
35798 10160 Acked Read 2 Done Sent Done false
35799 10161 Acked Flush 0 Done Sent Done false
35800 10162 Acked Read 10 Done Sent Done false
35801 10163 Acked Flush 0 Done Sent Done false
35802 10164 Acked Flush 0 Done Sent Done false
35803 10165 Acked Write 8 Done Sent Done false
35804 10166 Acked Read 4 Done Sent Done false
35805 10167 Acked Write 4 Done Sent Done false
35806 10168 Acked Read 3 Done Sent Done false
35807 10169 Acked Read 5 Done Sent Done false
35808 10170 Acked Write 8 Done Sent Done false
35809 10171 Acked Flush 0 Done Sent Done false
35810 10172 Acked Read 3 Done Sent Done false
35811 10173 Acked Write 6 Done Sent Done false
35812 10174 Acked Read 7 Done Sent Done false
35813 10175 Acked Flush 0 Done Sent Done false
35814 10176 Acked Write 10 Done Sent Done false
35815 10177 Acked Flush 0 Done Sent Done false
35816 10178 Acked Read 5 Done Sent Done false
35817 10179 Acked Read 3 Done Sent Done false
35818 10180 Acked Flush 0 Done Sent Done false
35819 10181 Acked Write 7 Done Sent Done false
35820 10182 Acked Read 7 Done Sent Done false
35821 10183 Acked Read 10 Done Sent Done false
35822 10184 Acked Flush 0 Done Sent Done false
35823 10185 Acked Write 7 Done Sent Done false
35824 10186 Acked Read 5 Done Sent Done false
35825 10187 Acked Read 6 Done Sent Done false
35826 10188 Acked Write 5 Done Sent Done false
35827 10189 Acked Write 4 Done Sent Done false
35828 10190 Acked Read 10 Done Sent Done false
35829 10191 Acked Read 1 Done Sent Done false
35830 10192 Acked Write 6 Done Sent Done false
35831 10193 Acked Write 4 Done Sent Done false
35832 10194 Acked Read 6 Done Sent Done false
35833 10195 Acked Write 10 Done Sent Done false
35834 10196 Acked Read 1 Done Sent Done false
35835 10197 Acked Read 2 Done Sent Done false
35836 10198 Acked Read 7 Done Sent Done false
35837 10199 Acked Read 10 Done Sent Done false
35838 10200 Acked Read 6 Done Sent Done false
35839 10201 Acked Read 10 Done Sent Done false
35840 10202 Acked Read 3 Done Sent Done false
35841 10203 Acked Write 2 Done Sent Done false
35842 10204 Acked Read 10 Done Sent Done false
35843 10205 Acked Write 3 Done Sent Done false
35844 10206 Acked Read 4 Done Sent Done false
35845 10207 Acked Read 4 Done Sent Done false
35846 10208 Acked Read 9 Done Sent Done false
35847 10209 Acked Read 3 Done Sent Done false
35848 10210 Acked Write 9 Done Sent Done false
35849 10211 Acked Write 10 Done Sent Done false
35850 10212 Acked Read 4 Done Sent Done false
35851 10213 Acked Write 9 Done Sent Done false
35852 10214 Acked Write 1 Done Sent Done false
35853 10215 Acked Read 2 Done Sent Done false
35854 10216 Acked Read 7 Done Sent Done false
35855 10217 Acked Read 10 Done Sent Done false
35856 10218 Acked Write 4 Done Sent Done false
35857 10219 Acked Read 6 Done Sent Done false
35858 10220 Acked Read 1 Done Sent Done false
35859 10221 Acked Read 6 Done Sent Done false
35860 10222 Acked Write 2 Done Sent Done false
35861 10223 Acked Write 2 Done Sent Done false
35862 10224 Acked Read 7 Done Sent Done false
35863 10225 Acked Flush 0 Done Sent Done false
35864 10226 Acked Write 3 Done Sent Done false
35865 10227 Acked Write 8 Done Sent Done false
35866 10228 Acked Read 1 Done Sent Done false
35867 10229 Acked Write 3 Done Sent Done false
35868 10230 Acked Flush 0 Done Sent Done false
35869 10231 Acked Write 4 Done Sent Done false
35870 10232 Acked Flush 0 Done Sent Done false
35871 10233 Acked Write 9 Done Sent Done false
35872 10234 Acked Read 2 Done Sent Done false
35873 10235 Acked Write 2 Done Sent Done false
35874 10236 Acked Read 1 Done Sent Done false
35875 10237 Acked Read 8 Done Sent Done false
35876 10238 Acked Read 6 Done Sent Done false
35877 10239 Acked Write 3 Done Sent Done false
35878 10240 Acked Write 5 Done Sent Done false
35879 10241 Acked Write 7 Done Sent Done false
35880 10242 Acked Flush 0 Done Sent Done false
35881 10243 Acked Flush 0 Done Sent Done false
35882 10244 Acked Read 2 Done Sent Done false
35883 10245 Acked Read 1 Done Sent Done false
35884 10246 Acked Read 2 Done Sent Done false
35885 10247 Acked Write 7 Done Sent Done false
35886 10248 Acked Write 2 Done Sent Done false
35887 10249 Acked Read 9 Done Sent Done false
35888 10250 Acked Write 4 Done Sent Done false
35889 10251 Acked Write 3 Done Sent Done false
35890 10252 Acked Read 5 Done Sent Done false
35891 10253 Acked Read 2 Done Sent Done false
35892 10254 Acked Read 5 Done Sent Done false
35893 10255 Acked Read 4 Done Sent Done false
35894 10256 Acked Read 10 Done Sent Done false
35895 10257 Acked Write 10 Done Sent Done false
35896 10258 Acked Write 2 Done Sent Done false
35897 10259 Acked Read 8 Done Sent Done false
35898 10260 Acked Flush 0 Done Sent Done false
35899 10261 Acked Write 6 Done Sent Done false
35900 10262 Acked Write 10 Done Sent Done false
35901 10263 Acked Read 1 Done Sent Done false
35902 10264 Acked Write 2 Done Sent Done false
35903 10265 Acked Read 1 Done Sent Done false
35904 10266 Acked Write 10 Done Sent Done false
35905 10267 Acked Write 6 Done Sent Done false
35906 10268 Acked Read 10 Done Sent Done false
35907 10269 Acked Flush 0 Done Sent Done false
35908 10270 Acked Write 9 Done Sent Done false
35909 10271 Acked Read 2 Done Sent Done false
35910 10272 Acked Read 5 Done Sent Done false
35911 10273 Acked Write 6 Done Sent Done false
35912 10274 Acked Write 6 Done Sent Done false
35913 10275 Acked Read 10 Done Sent Done false
35914 10276 Acked Write 4 Done Sent Done false
35915 10277 Acked Write 6 Done Sent Done false
35916 10278 Acked Write 1 Done Sent Done false
35917 10279 Acked Read 3 Done Sent Done false
35918 10280 Acked Write 6 Done Sent Done false
35919 10281 Acked Flush 0 Done Sent Done false
35920 10282 Acked Write 7 Done Sent Done false
35921 10283 Acked Read 5 Done Sent Done false
35922 10284 Acked Flush 0 Done Sent Done false
35923 10285 Acked Write 4 Done Sent Done false
35924 10286 Acked Write 3 Done Sent Done false
35925 10287 Acked Read 3 Done Sent Done false
35926 10288 Acked Write 10 Done Sent Done false
35927 10289 Acked Read 2 Done Sent Done false
35928 10290 Acked Read 3 Done Sent Done false
35929 10291 Acked Write 4 Done Sent Done false
35930 10292 Acked Read 4 Done Sent Done false
35931 10293 Acked Write 6 Done Sent Done false
35932 10294 Acked Read 3 Done Sent Done false
35933 10295 Acked Write 7 Done Sent Done false
35934 10296 Acked Flush 0 Done Sent Done false
35935 10297 Acked Read 9 Done Sent Done false
35936 10298 Acked Write 7 Done Sent Done false
35937 10299 Acked Read 5 Done Sent Done false
35938 10300 Acked Read 2 Done Sent Done false
35939 10301 Acked Write 3 Done Sent Done false
35940 10302 Acked Read 4 Done Sent Done false
35941 10303 Acked Read 8 Done Sent Done false
35942 10304 Acked Write 1 Done Sent Done false
35943 10305 Acked Read 4 Done Sent Done false
35944 10306 Acked Read 8 Done Sent Done false
35945 10307 Acked Flush 0 Done Sent Done false
35946 10308 Acked Write 10 Done Sent Done false
35947 10309 Acked Read 1 Done Sent Done false
35948 10310 Acked Write 5 Done Sent Done false
35949 10311 Acked Read 10 Done Sent Done false
35950 10312 Acked Read 3 Done Sent Done false
35951 10313 Acked Read 10 Done Sent Done false
35952 10314 Acked Read 8 Done Sent Done false
35953 10315 Acked Read 8 Done Sent Done false
35954 10316 Acked Write 1 Done Sent Done false
35955 10317 Acked Read 8 Done Sent Done false
35956 10318 Acked Read 4 Done Sent Done false
35957 10319 Acked Flush 0 Done Sent Done false
35958 10320 Acked Write 3 Done Sent Done false
35959 10321 Acked Write 10 Done Sent Done false
35960 10322 Acked Write 4 Done Sent Done false
35961 10323 Acked Read 7 Done Sent Done false
35962 10324 Acked Flush 0 Done Sent Done false
35963 10325 Acked Write 10 Done Sent Done false
35964 10326 Acked Read 7 Done Sent Done false
35965 10327 Acked Read 3 Done Sent Done false
35966 10328 Acked Read 7 Done Sent Done false
35967 10329 Acked Read 6 Done Sent Done false
35968 10330 Acked Write 9 Done Sent Done false
35969 10331 Acked Write 7 Done Sent Done false
35970 10332 Acked Flush 0 Done Sent Done false
35971 10333 Acked Write 5 Done Sent Done false
35972 10334 Acked Flush 0 Done Sent Done false
35973 10335 Acked Write 7 Done Sent Done false
35974 10336 Acked Write 4 Done Sent Done false
35975 10337 Acked Read 1 Done Sent Done false
35976 10338 Acked Read 6 Done Sent Done false
35977 10339 Acked Write 5 Done Sent Done false
35978 10340 Acked Read 5 Done Sent Done false
35979 10341 Acked Write 10 Done Sent Done false
35980 10342 Acked Read 6 Done Sent Done false
35981 10343 Acked Write 10 Done Sent Done false
35982 10344 Acked Read 10 Done Sent Done false
35983 10345 Acked Read 5 Done Sent Done false
35984 10346 Acked Flush 0 Done Sent Done false
35985 10347 Acked Write 4 Done Sent Done false
35986 10348 Acked Read 8 Done Sent Done false
35987 10349 Acked Read 4 Done Sent Done false
35988 10350 Acked Write 9 Done Sent Done false
35989 10351 Acked Write 10 Done Sent Done false
35990 10352 Acked Flush 0 Done Sent Done false
35991 10353 Acked Read 8 Done Sent Done false
35992 10354 Acked Write 7 Done Sent Done false
35993 10355 Acked Read 1 Done Sent Done false
35994 10356 Acked Read 7 Done Sent Done false
35995 STATES DS:0 DS:1 DS:2 TOTAL
35996 Sent 0 301 0 301
35997 Done 301 0 301 602
35998 Skipped 0 0 0 0
35999 Error 0 0 0 0
36000 Last Flush: Some(JobId(10352)) Some(JobId(10055)) Some(JobId(10352))
36001 Downstairs last five completed: 10055 10054 10053 10052 10051
36002 Upstairs last five completed: 10356 10355 10354 10353 10352
36003 CLIENT: Up:0 ds:301 act:2
360042025-03-07T18:20:28.950ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
360052025-03-07T18:20:29.739ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
360062025-03-07T18:20:30.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
360072025-03-07T18:20:31.743ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
36008 [8820] got start action
36009 [8820] I am going to restart
36010 Starting downstairs at port 8820
36011 Make output file at "/tmp/dsc/downstairs-8820.txt"
36012 [8820][1] reports Starting
36013 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1239)
36014 [8820][1] reports Running
36015 ----------------------------------------------------------------
36016 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36017 States: Active Offline Active
36018 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36019 10056 Acked Flush 0 Done Sent Done false
36020 10057 Acked Write 7 Done Sent Done false
36021 10058 Acked Read 6 Done Sent Done false
36022 10059 Acked Write 9 Done Sent Done false
36023 10060 Acked Write 9 Done Sent Done false
36024 10061 Acked Read 7 Done Sent Done false
36025 10062 Acked Read 10 Done Sent Done false
36026 10063 Acked Read 9 Done Sent Done false
36027 10064 Acked Write 9 Done Sent Done false
36028 10065 Acked Read 6 Done Sent Done false
36029 10066 Acked Write 2 Done Sent Done false
36030 10067 Acked Write 10 Done Sent Done false
36031 10068 Acked Read 6 Done Sent Done false
36032 10069 Acked Flush 0 Done Sent Done false
36033 10070 Acked Write 7 Done Sent Done false
36034 10071 Acked Read 6 Done Sent Done false
36035 10072 Acked Flush 0 Done Sent Done false
36036 10073 Acked Write 2 Done Sent Done false
36037 10074 Acked Read 6 Done Sent Done false
36038 10075 Acked Write 8 Done Sent Done false
36039 10076 Acked Read 8 Done Sent Done false
36040 10077 Acked Read 4 Done Sent Done false
36041 10078 Acked Read 1 Done Sent Done false
36042 10079 Acked Read 4 Done Sent Done false
36043 10080 Acked Flush 0 Done Sent Done false
36044 10081 Acked Write 8 Done Sent Done false
36045 10082 Acked Read 8 Done Sent Done false
36046 10083 Acked Write 7 Done Sent Done false
36047 10084 Acked Read 6 Done Sent Done false
36048 10085 Acked Read 9 Done Sent Done false
36049 10086 Acked Flush 0 Done Sent Done false
36050 10087 Acked Read 8 Done Sent Done false
36051 10088 Acked Read 2 Done Sent Done false
36052 10089 Acked Read 5 Done Sent Done false
36053 10090 Acked Write 9 Done Sent Done false
36054 10091 Acked Read 4 Done Sent Done false
36055 10092 Acked Read 5 Done Sent Done false
36056 10093 Acked Write 7 Done Sent Done false
36057 10094 Acked Read 8 Done Sent Done false
36058 10095 Acked Read 4 Done Sent Done false
36059 10096 Acked Write 8 Done Sent Done false
36060 10097 Acked Flush 0 Done Sent Done false
36061 10098 Acked Write 7 Done Sent Done false
36062 10099 Acked Flush 0 Done Sent Done false
36063 10100 Acked Write 6 Done Sent Done false
36064 10101 Acked Write 1 Done Sent Done false
36065 10102 Acked Read 2 Done Sent Done false
36066 10103 Acked Read 5 Done Sent Done false
36067 10104 Acked Read 4 Done Sent Done false
36068 10105 Acked Flush 0 Done Sent Done false
36069 10106 Acked Write 6 Done Sent Done false
36070 10107 Acked Write 4 Done Sent Done false
36071 10108 Acked Read 10 Done Sent Done false
36072 10109 Acked Write 5 Done Sent Done false
36073 10110 Acked Read 4 Done Sent Done false
36074 10111 Acked Write 10 Done Sent Done false
36075 10112 Acked Read 3 Done Sent Done false
36076 10113 Acked Write 5 Done Sent Done false
36077 10114 Acked Read 5 Done Sent Done false
36078 10115 Acked Write 5 Done Sent Done false
36079 10116 Acked Write 10 Done Sent Done false
36080 10117 Acked Write 5 Done Sent Done false
36081 10118 Acked Write 5 Done Sent Done false
36082 10119 Acked Write 4 Done Sent Done false
36083 10120 Acked Read 1 Done Sent Done false
36084 10121 Acked Write 5 Done Sent Done false
36085 10122 Acked Write 7 Done Sent Done false
36086 10123 Acked Write 9 Done Sent Done false
36087 10124 Acked Write 9 Done Sent Done false
36088 10125 Acked Read 2 Done Sent Done false
36089 10126 Acked Write 1 Done Sent Done false
36090 10127 Acked Write 8 Done Sent Done false
36091 10128 Acked Read 3 Done Sent Done false
36092 10129 Acked Write 4 Done Sent Done false
36093 10130 Acked Write 3 Done Sent Done false
36094 10131 Acked Read 6 Done Sent Done false
36095 10132 Acked Flush 0 Done Sent Done false
36096 10133 Acked Read 5 Done Sent Done false
36097 10134 Acked Read 1 Done Sent Done false
36098 10135 Acked Write 1 Done Sent Done false
36099 10136 Acked Write 6 Done Sent Done false
36100 10137 Acked Flush 0 Done Sent Done false
36101 10138 Acked Read 3 Done Sent Done false
36102 10139 Acked Read 6 Done Sent Done false
36103 10140 Acked Read 10 Done Sent Done false
36104 10141 Acked Write 2 Done Sent Done false
36105 10142 Acked Read 7 Done Sent Done false
36106 10143 Acked Read 7 Done Sent Done false
36107 10144 Acked Write 6 Done Sent Done false
36108 10145 Acked Read 8 Done Sent Done false
36109 10146 Acked Write 7 Done Sent Done false
36110 10147 Acked Read 6 Done Sent Done false
36111 10148 Acked Write 9 Done Sent Done false
36112 10149 Acked Read 7 Done Sent Done false
36113 10150 Acked Read 3 Done Sent Done false
36114 10151 Acked Read 9 Done Sent Done false
36115 10152 Acked Read 2 Done Sent Done false
36116 10153 Acked Read 5 Done Sent Done false
36117 10154 Acked Write 9 Done Sent Done false
36118 10155 Acked Write 10 Done Sent Done false
36119 10156 Acked Read 9 Done Sent Done false
36120 10157 Acked Write 4 Done Sent Done false
36121 10158 Acked Write 10 Done Sent Done false
36122 10159 Acked Read 5 Done Sent Done false
36123 10160 Acked Read 2 Done Sent Done false
36124 10161 Acked Flush 0 Done Sent Done false
36125 10162 Acked Read 10 Done Sent Done false
36126 10163 Acked Flush 0 Done Sent Done false
36127 10164 Acked Flush 0 Done Sent Done false
36128 10165 Acked Write 8 Done Sent Done false
36129 10166 Acked Read 4 Done Sent Done false
36130 10167 Acked Write 4 Done Sent Done false
36131 10168 Acked Read 3 Done Sent Done false
36132 10169 Acked Read 5 Done Sent Done false
36133 10170 Acked Write 8 Done Sent Done false
36134 10171 Acked Flush 0 Done Sent Done false
36135 10172 Acked Read 3 Done Sent Done false
36136 10173 Acked Write 6 Done Sent Done false
36137 10174 Acked Read 7 Done Sent Done false
36138 10175 Acked Flush 0 Done Sent Done false
36139 10176 Acked Write 10 Done Sent Done false
36140 10177 Acked Flush 0 Done Sent Done false
36141 10178 Acked Read 5 Done Sent Done false
36142 10179 Acked Read 3 Done Sent Done false
36143 10180 Acked Flush 0 Done Sent Done false
36144 10181 Acked Write 7 Done Sent Done false
36145 10182 Acked Read 7 Done Sent Done false
36146 10183 Acked Read 10 Done Sent Done false
36147 10184 Acked Flush 0 Done Sent Done false
36148 10185 Acked Write 7 Done Sent Done false
36149 10186 Acked Read 5 Done Sent Done false
36150 10187 Acked Read 6 Done Sent Done false
36151 10188 Acked Write 5 Done Sent Done false
36152 10189 Acked Write 4 Done Sent Done false
36153 10190 Acked Read 10 Done Sent Done false
36154 10191 Acked Read 1 Done Sent Done false
36155 10192 Acked Write 6 Done Sent Done false
36156 10193 Acked Write 4 Done Sent Done false
36157 10194 Acked Read 6 Done Sent Done false
36158 10195 Acked Write 10 Done Sent Done false
36159 10196 Acked Read 1 Done Sent Done false
36160 10197 Acked Read 2 Done Sent Done false
36161 10198 Acked Read 7 Done Sent Done false
36162 10199 Acked Read 10 Done Sent Done false
36163 10200 Acked Read 6 Done Sent Done false
36164 10201 Acked Read 10 Done Sent Done false
36165 10202 Acked Read 3 Done Sent Done false
36166 10203 Acked Write 2 Done Sent Done false
36167 10204 Acked Read 10 Done Sent Done false
36168 10205 Acked Write 3 Done Sent Done false
36169 10206 Acked Read 4 Done Sent Done false
36170 10207 Acked Read 4 Done Sent Done false
36171 10208 Acked Read 9 Done Sent Done false
36172 10209 Acked Read 3 Done Sent Done false
36173 10210 Acked Write 9 Done Sent Done false
36174 10211 Acked Write 10 Done Sent Done false
36175 10212 Acked Read 4 Done Sent Done false
36176 10213 Acked Write 9 Done Sent Done false
36177 10214 Acked Write 1 Done Sent Done false
36178 10215 Acked Read 2 Done Sent Done false
36179 10216 Acked Read 7 Done Sent Done false
36180 10217 Acked Read 10 Done Sent Done false
36181 10218 Acked Write 4 Done Sent Done false
36182 10219 Acked Read 6 Done Sent Done false
36183 10220 Acked Read 1 Done Sent Done false
36184 10221 Acked Read 6 Done Sent Done false
36185 10222 Acked Write 2 Done Sent Done false
36186 10223 Acked Write 2 Done Sent Done false
36187 10224 Acked Read 7 Done Sent Done false
36188 10225 Acked Flush 0 Done Sent Done false
36189 10226 Acked Write 3 Done Sent Done false
36190 10227 Acked Write 8 Done Sent Done false
36191 10228 Acked Read 1 Done Sent Done false
36192 10229 Acked Write 3 Done Sent Done false
36193 10230 Acked Flush 0 Done Sent Done false
36194 10231 Acked Write 4 Done Sent Done false
36195 10232 Acked Flush 0 Done Sent Done false
36196 10233 Acked Write 9 Done Sent Done false
36197 10234 Acked Read 2 Done Sent Done false
36198 10235 Acked Write 2 Done Sent Done false
36199 10236 Acked Read 1 Done Sent Done false
36200 10237 Acked Read 8 Done Sent Done false
36201 10238 Acked Read 6 Done Sent Done false
36202 10239 Acked Write 3 Done Sent Done false
36203 10240 Acked Write 5 Done Sent Done false
36204 10241 Acked Write 7 Done Sent Done false
36205 10242 Acked Flush 0 Done Sent Done false
36206 10243 Acked Flush 0 Done Sent Done false
36207 10244 Acked Read 2 Done Sent Done false
36208 10245 Acked Read 1 Done Sent Done false
36209 10246 Acked Read 2 Done Sent Done false
36210 10247 Acked Write 7 Done Sent Done false
36211 10248 Acked Write 2 Done Sent Done false
36212 10249 Acked Read 9 Done Sent Done false
36213 10250 Acked Write 4 Done Sent Done false
36214 10251 Acked Write 3 Done Sent Done false
36215 10252 Acked Read 5 Done Sent Done false
36216 10253 Acked Read 2 Done Sent Done false
36217 10254 Acked Read 5 Done Sent Done false
36218 10255 Acked Read 4 Done Sent Done false
36219 10256 Acked Read 10 Done Sent Done false
36220 10257 Acked Write 10 Done Sent Done false
36221 10258 Acked Write 2 Done Sent Done false
36222 10259 Acked Read 8 Done Sent Done false
36223 10260 Acked Flush 0 Done Sent Done false
36224 10261 Acked Write 6 Done Sent Done false
36225 10262 Acked Write 10 Done Sent Done false
36226 10263 Acked Read 1 Done Sent Done false
36227 10264 Acked Write 2 Done Sent Done false
36228 10265 Acked Read 1 Done Sent Done false
36229 10266 Acked Write 10 Done Sent Done false
36230 10267 Acked Write 6 Done Sent Done false
36231 10268 Acked Read 10 Done Sent Done false
36232 10269 Acked Flush 0 Done Sent Done false
36233 10270 Acked Write 9 Done Sent Done false
36234 10271 Acked Read 2 Done Sent Done false
36235 10272 Acked Read 5 Done Sent Done false
36236 10273 Acked Write 6 Done Sent Done false
36237 10274 Acked Write 6 Done Sent Done false
36238 10275 Acked Read 10 Done Sent Done false
36239 10276 Acked Write 4 Done Sent Done false
36240 10277 Acked Write 6 Done Sent Done false
36241 10278 Acked Write 1 Done Sent Done false
36242 10279 Acked Read 3 Done Sent Done false
36243 10280 Acked Write 6 Done Sent Done false
36244 10281 Acked Flush 0 Done Sent Done false
36245 10282 Acked Write 7 Done Sent Done false
36246 10283 Acked Read 5 Done Sent Done false
36247 10284 Acked Flush 0 Done Sent Done false
36248 10285 Acked Write 4 Done Sent Done false
36249 10286 Acked Write 3 Done Sent Done false
36250 10287 Acked Read 3 Done Sent Done false
36251 10288 Acked Write 10 Done Sent Done false
36252 10289 Acked Read 2 Done Sent Done false
36253 10290 Acked Read 3 Done Sent Done false
36254 10291 Acked Write 4 Done Sent Done false
36255 10292 Acked Read 4 Done Sent Done false
36256 10293 Acked Write 6 Done Sent Done false
36257 10294 Acked Read 3 Done Sent Done false
36258 10295 Acked Write 7 Done Sent Done false
36259 10296 Acked Flush 0 Done Sent Done false
36260 10297 Acked Read 9 Done Sent Done false
36261 10298 Acked Write 7 Done Sent Done false
36262 10299 Acked Read 5 Done Sent Done false
36263 10300 Acked Read 2 Done Sent Done false
36264 10301 Acked Write 3 Done Sent Done false
36265 10302 Acked Read 4 Done Sent Done false
36266 10303 Acked Read 8 Done Sent Done false
36267 10304 Acked Write 1 Done Sent Done false
36268 10305 Acked Read 4 Done Sent Done false
36269 10306 Acked Read 8 Done Sent Done false
36270 10307 Acked Flush 0 Done Sent Done false
36271 10308 Acked Write 10 Done Sent Done false
36272 10309 Acked Read 1 Done Sent Done false
36273 10310 Acked Write 5 Done Sent Done false
36274 10311 Acked Read 10 Done Sent Done false
36275 10312 Acked Read 3 Done Sent Done false
36276 10313 Acked Read 10 Done Sent Done false
36277 10314 Acked Read 8 Done Sent Done false
36278 10315 Acked Read 8 Done Sent Done false
36279 10316 Acked Write 1 Done Sent Done false
36280 10317 Acked Read 8 Done Sent Done false
36281 10318 Acked Read 4 Done Sent Done false
36282 10319 Acked Flush 0 Done Sent Done false
36283 10320 Acked Write 3 Done Sent Done false
36284 10321 Acked Write 10 Done Sent Done false
36285 10322 Acked Write 4 Done Sent Done false
36286 10323 Acked Read 7 Done Sent Done false
36287 10324 Acked Flush 0 Done Sent Done false
36288 10325 Acked Write 10 Done Sent Done false
36289 10326 Acked Read 7 Done Sent Done false
36290 10327 Acked Read 3 Done Sent Done false
36291 10328 Acked Read 7 Done Sent Done false
36292 10329 Acked Read 6 Done Sent Done false
36293 10330 Acked Write 9 Done Sent Done false
36294 10331 Acked Write 7 Done Sent Done false
36295 10332 Acked Flush 0 Done Sent Done false
36296 10333 Acked Write 5 Done Sent Done false
36297 10334 Acked Flush 0 Done Sent Done false
36298 10335 Acked Write 7 Done Sent Done false
36299 10336 Acked Write 4 Done Sent Done false
36300 10337 Acked Read 1 Done Sent Done false
36301 10338 Acked Read 6 Done Sent Done false
36302 10339 Acked Write 5 Done Sent Done false
36303 10340 Acked Read 5 Done Sent Done false
36304 10341 Acked Write 10 Done Sent Done false
36305 10342 Acked Read 6 Done Sent Done false
36306 10343 Acked Write 10 Done Sent Done false
36307 10344 Acked Read 10 Done Sent Done false
36308 10345 Acked Read 5 Done Sent Done false
36309 10346 Acked Flush 0 Done Sent Done false
36310 10347 Acked Write 4 Done Sent Done false
36311 10348 Acked Read 8 Done Sent Done false
36312 10349 Acked Read 4 Done Sent Done false
36313 10350 Acked Write 9 Done Sent Done false
36314 10351 Acked Write 10 Done Sent Done false
36315 10352 Acked Flush 0 Done Sent Done false
36316 10353 Acked Read 8 Done Sent Done false
36317 10354 Acked Write 7 Done Sent Done false
36318 10355 Acked Read 1 Done Sent Done false
36319 10356 Acked Read 7 Done Sent Done false
36320 STATES DS:0 DS:1 DS:2 TOTAL
36321 Sent 0 301 0 301
36322 Done 301 0 301 602
36323 Skipped 0 0 0 0
36324 Error 0 0 0 0
36325 Last Flush: Some(JobId(10352)) Some(JobId(10055)) Some(JobId(10352))
36326 Downstairs last five completed: 10055 10054 10053 10052 10051
36327 Upstairs last five completed: 10356 10355 10354 10353 10352
36328 CLIENT: Up:0 ds:301 act:2
363292025-03-07T18:20:32.952ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363302025-03-07T18:20:33.746ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363312025-03-07T18:20:34.197ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363322025-03-07T18:20:34.197ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57666) = io task client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363332025-03-07T18:20:34.198ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363342025-03-07T18:20:34.198ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363352025-03-07T18:20:34.198ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363362025-03-07T18:20:34.198ZINFOcrucible: send last flush ID to this DS: Some(JobId(10055)) = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363372025-03-07T18:20:34.198ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363382025-03-07T18:20:34.198ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10055)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
363392025-03-07T18:20:34.199ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10055)) = downstairs session_id = 5e9909a1-dc0d-4821-9383-88a904f8ac68
36340 ----------------------------------------------------------------
36341 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36342 Downstairs last five completed: 10357 10356 10355 10354 10353
36343 Upstairs last five completed: 10357 10356 10355 10354 10353
36344 CLIENT: Up:0 ds:0 act:3
36345 Replay: All jobs finished
36346 Test replay has completed
36347 Wrote out file "/tmp/test_replay_verify.log"
36348 CLIENT: Tests done. All submitted work has been ACK'd
36349 ----------------------------------------------------------------
36350 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36351 Downstairs last five completed: 10357 10356 10355 10354 10353
36352 Upstairs last five completed: 10357 10356 10355 10354 10353
36353 CLIENT: Up:0 ds:0 act:3
36354 CLIENT: All jobs finished, all DS active.
36355 Replays:30 time: 8:00
36356 Do final verify
36357 Mar 07 18:20:36.525 INFO accepted connection, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
36358 Mar 07 18:20:36.526 INFO request completed, latency_us: 217, response_code: 200, uri: /regioninfo, method: GET, req_id: d51d2621-4579-4534-9d1b-1707fc93f4fa, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
363592025-03-07T18:20:36.527ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
36360 Mar 07 18:20:36.527 INFO request completed, latency_us: 90, response_code: 200, uri: /regioncount, method: GET, req_id: c59b4dc5-aa5f-4f9b-b25b-aeafb87a9205, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
363612025-03-07T18:20:36.527ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
36362 Mar 07 18:20:36.528 INFO request completed, latency_us: 125, response_code: 200, uri: /port/cid/0, method: GET, req_id: b702deaa-667b-431d-8b10-8d7bf5526fac, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
36363 Mar 07 18:20:36.528 INFO request completed, latency_us: 90, response_code: 200, uri: /port/cid/1, method: GET, req_id: df3fbf4a-a335-4468-bf87-ed38ba9ca509, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
36364 Mar 07 18:20:36.529 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/2, method: GET, req_id: e06f0d6c-f699-4c7c-938c-a8a2c0a84f7a, remote_addr: 127.0.0.1:63947, local_addr: 127.0.0.1:9998
363652025-03-07T18:20:36.529ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
363662025-03-07T18:20:36.529ZINFOcrucible: Upstairs starts
363672025-03-07T18:20:36.529ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "6c903058314cb23845d8ec9d1b70e497f207c60e", git_commit_timestamp: "2025-03-07T17:56:28.000000000Z", git_branch: "renovate/dropshot-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
363682025-03-07T18:20:36.529ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
363692025-03-07T18:20:36.529ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
363702025-03-07T18:20:36.530ZINFOcrucible: Crucible 54ed9bb8-1f1c-4ccf-9a7d-192a832dad37 has session id: b3645106-821c-4b45-8b59-1e6859ba08fa session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363712025-03-07T18:20:36.530ZINFOcrucible: Upstairs opts: Upstairs UUID: 54ed9bb8-1f1c-4ccf-9a7d-192a832dad37, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363722025-03-07T18:20:36.530ZINFOcrucible: Crucible stats registered with UUID: 54ed9bb8-1f1c-4ccf-9a7d-192a832dad37 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363732025-03-07T18:20:36.530ZWARNcrucible: could not find Downstairs address for Nexus session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363742025-03-07T18:20:36.531ZINFOcrucible: Downstairs in volume = 3 task = crutest
363752025-03-07T18:20:36.531ZINFOcrucible: The guest has requested activation with gen:3
363762025-03-07T18:20:36.531ZINFOcrucible: 54ed9bb8-1f1c-4ccf-9a7d-192a832dad37 active request set session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363772025-03-07T18:20:36.531ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363782025-03-07T18:20:36.531ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363792025-03-07T18:20:36.531ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363802025-03-07T18:20:36.531ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363812025-03-07T18:20:36.531ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363822025-03-07T18:20:36.531ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363832025-03-07T18:20:36.531ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363842025-03-07T18:20:36.531ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363852025-03-07T18:20:36.531ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363862025-03-07T18:20:36.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58752) = io task client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363872025-03-07T18:20:36.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58695) = io task client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363882025-03-07T18:20:36.531ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49041) = io task client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363892025-03-07T18:20:36.532ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363902025-03-07T18:20:36.532ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363912025-03-07T18:20:36.532ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363922025-03-07T18:20:36.532ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363932025-03-07T18:20:36.532ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363942025-03-07T18:20:36.532ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363952025-03-07T18:20:36.533ZINFOcrucible: Waiting for more clients to be ready session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363962025-03-07T18:20:36.533ZINFOcrucible: Waiting for more clients to be ready session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363972025-03-07T18:20:36.533ZINFOcrucible: [0]R flush_numbers[0..12]: [954, 965, 968, 967, 931, 970, 972, 966, 970, 964, 964, 961] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363982025-03-07T18:20:36.533ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
363992025-03-07T18:20:36.533ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364002025-03-07T18:20:36.533ZINFOcrucible: [1]R flush_numbers[0..12]: [954, 965, 968, 967, 931, 970, 972, 966, 970, 964, 964, 961] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364012025-03-07T18:20:36.533ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364022025-03-07T18:20:36.533ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364032025-03-07T18:20:36.533ZINFOcrucible: [2]R flush_numbers[0..12]: [954, 965, 968, 967, 931, 970, 972, 966, 970, 964, 964, 961] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364042025-03-07T18:20:36.533ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364052025-03-07T18:20:36.533ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364062025-03-07T18:20:36.533ZINFOcrucible: Max found gen is 3 = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364072025-03-07T18:20:36.533ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
36408 {Wait for a query_work_queue command to finish before sending IO
36409 "msg":"Next flush: 973","v":0,"name":"crucible","level":30,"time":"2025-03-07T18:20:36.53395939Z","hostname":"w-01JNRWVQ30VS9MDGSCARQBP37R","pid":1244,"":"downstairs","":"downstairs","session_id":"b3645106-821c-4b45-8b59-1e6859ba08fa"}
364102025-03-07T18:20:36.533ZINFOcrucible: All extents match = downstairs session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364112025-03-07T18:20:36.533ZINFOcrucible: No downstairs reconciliation required session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364122025-03-07T18:20:36.534ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364132025-03-07T18:20:36.534ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364142025-03-07T18:20:36.534ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364152025-03-07T18:20:36.534ZINFOcrucible: All required reconciliation work is completed session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364162025-03-07T18:20:36.534ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364172025-03-07T18:20:36.534ZINFOcrucible: 54ed9bb8-1f1c-4ccf-9a7d-192a832dad37 is now active with session: b3645106-821c-4b45-8b59-1e6859ba08fa session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364182025-03-07T18:20:36.534ZINFOcrucible: Set Active after no reconciliation session_id = b3645106-821c-4b45-8b59-1e6859ba08fa
364192025-03-07T18:20:36.534ZINFOcrucible: The guest has finished waiting for activation with:3
36420 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
36421 Loading write count information from file "/tmp/test_replay_verify.log"
36422 Read and Verify all blocks (0..5000 range:false)
36423 Wrote out file "/tmp/test_replay_verify.log"
36424 Verify test completed
36425 Wrote out file "/tmp/test_replay_verify.log"
36426 CLIENT: Tests done. All submitted work has been ACK'd
36427 ----------------------------------------------------------------
36428 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
36429 States: Active Active Active
36430 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36431 1000 Acked Read 256 Done Done Done false
36432 1001 Acked Read 256 Done Done Done false
36433 1002 Acked Read 256 Done Done Done false
36434 1003 Acked Read 256 Done Done Done false
36435 1004 Acked Read 256 Done Done Done false
36436 1005 Acked Read 256 Done Done Done false
36437 1006 Acked Read 256 Done Done Done false
36438 1007 Acked Read 256 Done Done Done false
36439 1008 Acked Read 256 Done Done Done false
36440 1009 Acked Read 256 Done Done Done false
36441 1010 Acked Read 256 Done Done Done false
36442 1011 Acked Read 256 Done Done Done false
36443 1012 Acked Read 256 Done Done Done false
36444 1013 Acked Read 256 Done Done Done false
36445 1014 Acked Read 256 Done Done Done false
36446 1015 Acked Read 256 Done Done Done false
36447 1016 Acked Read 256 Done Done Done false
36448 1017 Acked Read 256 Done Done Done false
36449 1018 Acked Read 136 Done Done Done false
36450 1019 Acked Read 256 Done Done Done false
36451 STATES DS:0 DS:1 DS:2 TOTAL
36452 Sent 0 0 0 0
36453 Done 20 20 20 60
36454 Skipped 0 0 0 0
36455 Error 0 0 0 0
36456 Last Flush: None None None
36457 Downstairs last five completed:
36458 Upstairs last five completed: 1019 1018 1017 1016 1015
36459 CLIENT: Up:0 ds:20 act:3
36460 ----------------------------------------------------------------
36461 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
36462 Downstairs last five completed: 1020 1019 1018 1017 1016
36463 Upstairs last five completed: 1020 1019 1018 1017 1016
36464 CLIENT: Up:0 ds:0 act:3
36465 CLIENT: All crucible jobs finished, exiting program
36466 Mar 07 18:20:40.989 INFO accepted connection, remote_addr: 127.0.0.1:44744, local_addr: 127.0.0.1:9998
36467 Main task has work to do, go find it
36468 got dsc Shutdown
36469 Shutdown
36470 Mar 07 18:20:40.990 INFO request completed, latency_us: 168Shut it down
36471 , response_code: 204, uri: /shutdown, method: POST, req_id: f9702c2c-ddc7-4152-b9a2-10da8b6a4664, remote_addr: 127.0[8810] Disable keep_running
36472 .[8810] Got stop action so:false kr:false
36473 0.1:44744, local_addr: 127.0.0.1:9998
36474 [8820] Disable keep_running
36475 [8820] Got stop action so:false kr:false
36476 [8830] Disable keep_running
36477 [8830] Got stop action so:false kr:false
36478 [8820] Exited with: ExitStatus(unix_wait_status(9))
36479 [8830] Exited with: ExitStatus(unix_wait_status(9))
36480 [8820][1] reports Exit
36481 [8830][2] reports Exit
36482 [8810] Exited with: ExitStatus(unix_wait_status(9))
36483 [8810][0] reports Exit
36484 March 7, 2025 at 06:20:46 PM UTC Test ends with 0