Using existing output directory "/tmp/dsc" Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 16384, extent_count: 1600 }) start ds: 8810 start ds: 8820 start ds: 8830 start access at:127.0.0.1:9998 Starting downstairs at port 8810 Starting downstairs at port 8820 Make output file at "/tmp/dsc/downstairs-8810.txt" Make output file at "/tmp/dsc/downstairs-8820.txt" [8810][0] initial start wait reports Starting [8820][1] initial start wait reports Starting Starting downstairs at port 8830 Make output file at "/tmp/dsc/downstairs-8830.txt" [8830][2] initial start wait reports Starting Downstairs /var/tmp/dsc/8810 port 8810 PID:Some(1490) [8810][0] initial start wait reports Running Downstairs /var/tmp/dsc/8830 port 8830 PID:Some(1491) [8830][2] initial start wait reports Running Downstairs /var/tmp/dsc/8820 port 8820 PID:Some(1492) [8820][1] initial start wait reports Running All downstairs are running Nov 07 14:05:53.175 INFO listening, local_addr: 127.0.0.1:9998 Control access at:127.0.0.1:9998 Nov 07 14:05:58.213 INFO accepted connection, remote_addr: 127.0.0.1:48941, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc DisableRestartAll disable restart on all: 3 Nov 07 14:05:58.213 INFO request completed, latency_us: 100, response_code: 204, [8820] Disable keep_running uri: /disablerestart/all, method: POST, req_id: d149917e-313f-4997-ac76-636dee0137a1[8830] Disable keep_running , [8810] Disable keep_running remote_addr: 127.0.0.1:48941, local_addr: 127.0.0.1:9998 Nov 07 14:07:45.316 INFO accepted connection, remote_addr: 127.0.0.1:64709, local_addr: 127.0.0.1:9998 Main task has work to do, go find it got dsc Shutdown Shutdown Nov 07 14:07:45.317 INFO request completedShut it down , [8810] Disable keep_running latency_us[8820] Disable keep_running : 50, response_code: 204, uri:[8820] Got stop action so:false kr:false /shutdown, method: POST, req_id: 8ef99a00-4172-4e41-8085-36102f75e475, remote_addr:[8830] Disable keep_running [8830] Got stop action so:false kr:false 127.0[8810] Got stop action so:false kr:false .0.1:64709, local_addr: 127.0.0.1:9998 [8830] Exited with: ExitStatus(unix_wait_status(9)) [8830][2] reports Exit [8820] Exited with: ExitStatus(unix_wait_status(9)) [8820][1] reports Exit [8810] Exited with: ExitStatus(unix_wait_status(9)) [8810][0] reports Exit 5} {"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"311c3ca2c97d568407ae06780269d96fcfdd3a7c\",\n git_commit_timestamp: \"2024-11-07T13:47:04.000000000Z\",\n git_branch: \"renovate/tokio-1.x-lockfile\",\n rustc_semver: \"1.82.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"f6e511eec7342f59a25f7c0534f1dbea00d01b14\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: false,\n opt_level: 3,\n}","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.264576586Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495} {"msg":"Upstairs <-> Downstairs Message Version: 12","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.264582095Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495} {"msg":"Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 16384, shift: 12 }, extent_count: 1600, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.264587135Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495} {"msg":"Crucible 1c249640-644d-4350-9262-60763f6573b4 has session id: 40eaacc5-c5c0-45a1-8ad5-46cc9052dbba","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.26491312Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Upstairs opts: Upstairs UUID: 1c249640-644d-4350-9262-60763f6573b4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.264927488Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Crucible stats registered with UUID: 1c249640-644d-4350-9262-60763f6573b4","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.264936417Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Downstairs in volume = 3","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294802381Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"task":"crutest"} {"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.29481278Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495} {"msg":"1c249640-644d-4350-9262-60763f6573b4 active request set","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294830497Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294839646Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294849165Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294854944Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294860313Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294865483Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294870782Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294876621Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294932314Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294945372Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_connection connected from Ok(127.0.0.1:39340)","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294954511Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_connection connected from Ok(127.0.0.1:34148)","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.294962939Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_connection connected from Ok(127.0.0.1:41387)","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295004294Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"io task","client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295352296Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295365185Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295378463Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295386422Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295438675Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295450233Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295594913Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295606452Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.295679672Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.302185446Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.302195875Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.302200764Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.302996606Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303007804Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Waiting for more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303015303Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.30347576Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303490988Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303500187Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303508116Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303516315Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303524164Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303532333Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303540042Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303548071Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.30355568Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303563399Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303571178Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303578817Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303586345Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303594474Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"":"downstairs","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} Wait for a query_work_queue command to finish before sending IO {"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303614441Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.30362166Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"0","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.30362831Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"1","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303634689Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"client":"2","":"downstairs","session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303641328Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303647037Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"1c249640-644d-4350-9262-60763f6573b4 is now active with session: 40eaacc5-c5c0-45a1-8ad5-46cc9052dbba","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303652746Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303658136Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495,"session_id":"40eaacc5-c5c0-45a1-8ad5-46cc9052dbba"} {"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-11-07T14:05:58.303663605Z","hostname":"w-01JC3F11MM58CEYWKPEJC90EFX","pid":1495} Region: sv:1 bs:4096 ts:107374182400 tb:26214400 max_io:256 or 1048576 Fill test CLIENT: Tests done. All submitted work has been ACK'd ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:444 States: Active Active Active JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY 103159 Acked Write 256 Done Done Done false 103160 Acked Write 256 Done Done Done false 103161 Acked Write 256 Done Done Done false 103162 Acked Write 256 Done Done Done false 103163 Acked Write 256 Done Done Done false 103164 Acked Write 256 Done Done Done false 103165 Acked Write 256 Done Done Done false 103166 Acked Write 256 Done Done Done false 103167 Acked Write 256 Done Done Done false 103168 Acked Write 256 Done Done Done false 103169 Acked Write 256 Done Done Done false 103170 Acked Write 256 Done Done Done false 103171 Acked Write 256 Done Done Done false 103172 Acked Write 256 Done Done Done false 103173 Acked Write 256 Done Done Done false 103174 Acked Write 256 Done Done Done false 103175 Acked Write 256 Done Done Done false 103176 Acked Write 256 Done Done Done false 103177 Acked Write 256 Done Done Done false 103178 Acked Write 256 Done Done Done false 103179 Acked Write 256 Done Done Done false 103180 Acked Write 256 Done Done Done false 103181 Acked Write 256 Done Done Done false 103182 Acked Write 256 Done Done Done false 103183 Acked Write 256 Done Done Done false 103184 Acked Write 256 Done Done Done false 103185 Acked Write 256 Done Done Done false 103186 Acked Write 256 Done Done Done false 103187 Acked Write 256 Done Done Done false 103188 Acked Write 256 Done Done Done false 103189 Acked Write 256 Done Done Done false 103190 Acked Write 256 Done Done Done false 103191 Acked Write 256 Done Done Done false 103192 Acked Write 256 Done Done Done false 103193 Acked Write 256 Done Done Done false 103194 Acked Write 256 Done Done Done false 103195 Acked Write 256 Done Done Done false 103196 Acked Write 256 Done Done Done false 103197 Acked Write 256 Done Done Done false 103198 Acked Write 256 Done Done Done false 103199 Acked Write 256 Done Done Done false 103200 Acked Write 256 Done Done Done false 103201 Acked Write 256 Done Done Done false 103202 Acked Write 256 Done Done Done false 103203 Acked Write 256 Done Done Done false 103204 Acked Write 256 Done Done Done false 103205 Acked Write 256 Done Done Done false 103206 Acked Write 256 Done Done Done false 103207 Acked Write 256 Done Done Done false 103208 Acked Write 256 Done Done Done false 103209 Acked Write 256 Done Done Done false 103210 Acked Write 256 Done Done Done false 103211 Acked Write 256 Done Done Done false 103212 Acked Write 256 Done Done Done false 103213 Acked Write 256 Done Done Done false 103214 Acked Write 256 Done Done Done false 103215 Acked Write 256 Done Done Done false 103216 Acked Write 256 Done Done Done false 103217 Acked Write 256 Done Done Done false 103218 Acked Write 256 Done Done Done false 103219 Acked Write 256 Done Done Done false 103220 Acked Write 256 Done Done Done false 103221 Acked Write 256 Done Done Done false 103222 Acked Write 256 Done Done Done false 103223 Acked Write 256 Done Done Done false 103224 Acked Write 256 Done Done Done false 103225 Acked Write 256 Done Done Done false 103226 Acked Write 256 Done Done Done false 103227 Acked Write 256 Done Done Done false 103228 Acked Write 256 Done Done Done false 103229 Acked Write 256 Done Done Done false 103230 Acked Write 256 Done Done Done false 103231 Acked Write 256 Done Done Done false 103232 Acked Write 256 Done Done Done false 103233 Acked Write 256 Done Done Done false 103234 Acked Write 256 Done Done Done false 103235 Acked Write 256 Done Done Done false 103236 Acked Write 256 Done Done Done false 103237 Acked Write 256 Done Done Done false 103238 Acked Write 256 Done Done Done false 103239 Acked Write 256 Done Done Done false 103240 Acked Write 256 Done Done Done false 103241 Acked Write 256 Done Done Done false 103242 Acked Write 256 Done Done Done false 103243 Acked Write 256 Done Done Done false 103244 Acked Write 256 Done Done Done false 103245 Acked Write 256 Done Done Done false 103246 Acked Write 256 Done Done Done false 103247 Acked Write 256 Done Done Done false 103248 Acked Write 256 Done Done Done false 103249 Acked Write 256 Done Done Done false 103250 Acked Write 256 Done Done Done false 103251 Acked Write 256 Done Done Done false 103252 Acked Write 256 Done Done Done false 103253 Acked Write 256 Done Done Done false 103254 Acked Write 256 Done Done Done false 103255 Acked Write 256 Done Done Done false 103256 Acked Write 256 Done Done Done false 103257 Acked Write 256 Done Done Done false 103258 Acked Write 256 Done Done Done false 103259 Acked Write 256 Done Done Done false 103260 Acked Write 256 Done Done Done false 103261 Acked Write 256 Done Done Done false 103262 Acked Write 256 Done Done Done false 103263 Acked Write 256 Done Done Done false 103264 Acked Write 256 Done Done Done false 103265 Acked Write 256 Done Done Done false 103266 Acked Write 256 Done Done Done false 103267 Acked Write 256 Done Done Done false 103268 Acked Write 256 Done Done Done false 103269 Acked Write 256 Done Done Done false 103270 Acked Write 256 Done Done Done false 103271 Acked Write 256 Done Done Done false 103272 Acked Write 256 Done Done Done false 103273 Acked Write 256 Done Done Done false 103274 Acked Write 256 Done Done Done false 103275 Acked Write 256 Done Done Done false 103276 Acked Write 256 Done Done Done false 103277 Acked Write 256 Done Done Done false 103278 Acked Write 256 Done Done Done false 103279 Acked Write 256 Done Done Done false 103280 Acked Write 256 Done Done Done false 103281 Acked Write 256 Done Done Done false 103282 Acked Write 256 Done Done Done false 103283 Acked Write 256 Done Done Done false 103284 Acked Write 256 Done Done Done false 103285 Acked Write 256 Done Done Done false 103286 Acked Write 256 Done Done Done false 103287 Acked Write 256 Done Done Done false 103288 Acked Write 256 Done Done Done false 103289 Acked Write 256 Done Done Done false 103290 Acked Write 256 Done Done Done false 103291 Acked Write 256 Done Done Done false 103292 Acked Write 256 Done Done Done false 103293 Acked Write 256 Done Done Done false 103294 Acked Write 256 Done Done Done false 103295 Acked Write 256 Done Done Done false 103296 Acked Write 256 Done Done Done false 103297 Acked Write 256 Done Done Done false 103298 Acked Write 256 Done Done Done false 103299 Acked Write 256 Done Done Done false 103300 Acked Write 256 Done Done Done false 103301 Acked Write 256 Done Done Done false 103302 Acked Write 256 Done Done Done false 103303 Acked Write 256 Done Done Done false 103304 Acked Write 256 Done Done Done false 103305 Acked Write 256 Done Done Done false 103306 Acked Write 256 Done Done Done false 103307 Acked Write 256 Done Done Done false 103308 Acked Write 256 Done Done Done false 103309 Acked Write 256 Done Done Done false 103310 Acked Write 256 Done Done Done false 103311 Acked Write 256 Done Done Done false 103312 Acked Write 256 Done Done Done false 103313 Acked Write 256 Done Done Done false 103314 Acked Write 256 Done Done Done false 103315 Acked Write 256 Done Done Done false 103316 Acked Write 256 Done Done Done false 103317 Acked Write 256 Done Done Done false 103318 Acked Write 256 Done Done Done false 103319 Acked Write 256 Done Done Done false 103320 Acked Write 256 Done Done Done false 103321 Acked Write 256 Done Done Done false 103322 Acked Write 256 Done Done Done false 103323 Acked Write 256 Done Done Done false 103324 Acked Write 256 Done Done Done false 103325 Acked Write 256 Done Done Done false 103326 Acked Write 256 Done Done Done false 103327 Acked Write 256 Done Done Done false 103328 Acked Write 256 Done Done Done false 103329 Acked Write 256 Done Done Done false 103330 Acked Write 256 Done Done Done false 103331 Acked Write 256 Done Done Done false 103332 Acked Write 256 Done Done Done false 103333 Acked Write 256 Done Done Done false 103334 Acked Write 256 Done Done Done false 103335 Acked Write 256 Done Done Done false 103336 Acked Write 256 Done Done Done false 103337 Acked Write 256 Done Done Done false 103338 Acked Write 256 Done Done Done false 103339 Acked Write 256 Done Done Done false 103340 Acked Write 256 Done Done Done false 103341 Acked Write 256 Done Done Done false 103342 Acked Write 256 Done Done Done false 103343 Acked Write 256 Done Done Done false 103344 Acked Write 256 Done Done Done false 103345 Acked Write 256 Done Done Done false 103346 Acked Write 256 Done Done Done false 103347 Acked Write 256 Done Done Done false 103348 Acked Write 256 Done Done Done false 103349 Acked Write 256 Done Done Done false 103350 Acked Write 256 Done Done Done false 103351 Acked Write 256 Done Done Done false 103352 Acked Write 256 Done Done Done false 103353 Acked Write 256 Done Done Done false 103354 Acked Write 256 Done Done Done false 103355 Acked Write 256 Done Done Done false 103356 Acked Write 256 Done Done Done false 103357 Acked Write 256 Done Done Done false 103358 Acked Write 256 Done Done Done false 103359 Acked Write 256 Done Done Done false 103360 Acked Write 256 Done Done Done false 103361 Acked Write 256 Done Done Done false 103362 Acked Write 256 Done Done Done false 103363 Acked Write 256 Done Done Done false 103364 Acked Write 256 Done Done Done false 103365 Acked Write 256 Done Done Done false 103366 Acked Write 256 Done Done Done false 103367 Acked Write 256 Done Done Done false 103368 Acked Write 256 Done Done Done false 103369 Acked Write 256 Done Done Done false 103370 Acked Write 256 Done Done Done false 103371 Acked Write 256 Done Done Done false 103372 Acked Write 256 Done Done Done false 103373 Acked Write 256 Done Done Done false 103374 Acked Write 256 Done Done Done false 103375 Acked Write 256 Done Done Done false 103376 Acked Write 256 Done Done Done false 103377 Acked Write 256 Done Done Done false 103378 Acked Write 256 Done Done Done false 103379 Acked Write 256 Done Done Done false 103380 Acked Write 256 Done Done Done false 103381 Acked Write 256 Done Done Done false 103382 Acked Write 256 Done Done Done false 103383 Acked Write 256 Done Done Done false 103384 Acked Write 256 Done Done Done false 103385 Acked Write 256 Done Done Done false 103386 Acked Write 256 Done Done Done false 103387 Acked Write 256 Done Done Done false 103388 Acked Write 256 Done Done Done false 103389 Acked Write 256 Done Done Done false 103390 Acked Write 256 Done Done Done false 103391 Acked Write 256 Done Done Done false 103392 Acked Write 256 Done Done Done false 103393 Acked Write 256 Done Done Done false 103394 Acked Write 256 Done Done Done false 103395 Acked Write 256 Done Done Done false 103396 Acked Write 256 Done Done Done false 103397 Acked Write 256 Done Done Done false 103398 Acked Write 256 Done Done Done false 103399 Acked Write 256 Done Done Done false 103400 Acked Write 256 Done Done Done false 103401 Acked Write 256 Done Done Done false 103402 Acked Write 256 Done Done Done false 103403 Acked Write 256 Done Done Done false 103404 Acked Write 256 Done Done Done false 103405 Acked Write 256 Done Done Done false 103406 Acked Write 256 Done Done Done false 103407 Acked Write 256 Done Done Done false 103408 Acked Write 256 Done Done Done false 103409 Acked Write 256 Done Done Done false 103410 Acked Write 256 Done Done Done false 103411 Acked Write 256 Done Done Done false 103412 Acked Write 256 Done Done Done false 103413 Acked Write 256 Done Done Done false 103414 Acked Write 256 Done Done Done false 103415 Acked Write 256 Done Done Done false 103416 Acked Write 256 Done Done Done false 103417 Acked Write 256 Done Done Done false 103418 Acked Write 256 Done Done Done false 103419 Acked Write 256 Done Done Done false 103420 Acked Write 256 Done Done Done false 103421 Acked Write 256 Done Done Done false 103422 Acked Write 256 Done Done Done false 103423 Acked Write 256 Done Done Done false 103424 Acked Write 256 Done Done Done false 103425 Acked Write 256 Done Done Done false 103426 Acked Write 256 Done Done Done false 103427 Acked Write 256 Done Done Done false 103428 Acked Write 256 Done Done Done false 103429 Acked Write 256 Done Done Done false 103430 Acked Write 256 Done Done Done false 103431 Acked Write 256 Done Done Done false 103432 Acked Write 256 Done Done Done false 103433 Acked Write 256 Done Done Done false 103434 Acked Write 256 Done Done Done false 103435 Acked Write 256 Done Done Done false 103436 Acked Write 256 Done Done Done false 103437 Acked Write 256 Done Done Done false 103438 Acked Write 256 Done Done Done false 103439 Acked Write 256 Done Done Done false 103440 Acked Write 256 Done Done Done false 103441 Acked Write 256 Done Done Done false 103442 Acked Write 256 Done Done Done false 103443 Acked Write 256 Done Done Done false 103444 Acked Write 256 Done Done Done false 103445 Acked Write 256 Done Done Done false 103446 Acked Write 256 Done Done Done false 103447 Acked Write 256 Done Done Done false 103448 Acked Write 256 Done Done Done false 103449 Acked Write 256 Done Done Done false 103450 Acked Write 256 Done Done Done false 103451 Acked Write 256 Done Done Done false 103452 Acked Write 256 Done Done Done false 103453 Acked Write 256 Done Done Done false 103454 Acked Write 256 Done Done Done false 103455 Acked Write 256 Done Done Done false 103456 Acked Write 256 Done Done Done false 103457 Acked Write 256 Done Done Done false 103458 Acked Write 256 Done Done Done false 103459 Acked Write 256 Done Done Done false 103460 Acked Write 256 Done Done Done false 103461 Acked Write 256 Done Done Done false 103462 Acked Write 256 Done Done Done false 103463 Acked Write 256 Done Done Done false 103464 Acked Write 256 Done Done Done false 103465 Acked Write 256 Done Done Done false 103466 Acked Write 256 Done Done Done false 103467 Acked Write 256 Done Done Done false 103468 Acked Write 256 Done Done Done false 103469 Acked Write 256 Done Done Done false 103470 Acked Write 256 Done Done Done false 103471 Acked Write 256 Done Done Done false 103472 Acked Write 256 Done Done Done false 103473 Acked Write 256 Done Done Done false 103474 Acked Write 256 Done Done Done false 103475 Acked Write 256 Done Done Done false 103476 Acked Write 256 Done Done Done false 103477 Acked Write 256 Done Done Done false 103478 Acked Write 256 Done Done Done false 103479 Acked Write 256 Done Done Done false 103480 Acked Write 256 Done Done Done false 103481 Acked Write 256 Done Done Done false 103482 Acked Write 256 Done Done Done false 103483 Acked Write 256 Done Done Done false 103484 Acked Write 256 Done Done Done false 103485 Acked Write 256 Done Done Done false 103486 Acked Write 256 Done Done Done false 103487 Acked Write 256 Done Done Done false 103488 Acked Write 256 Done Done Done false 103489 Acked Write 256 Done Done Done false 103490 Acked Write 256 Done Done Done false 103491 Acked Write 256 Done Done Done false 103492 Acked Write 256 Done Done Done false 103493 Acked Write 256 Done Done Done false 103494 Acked Write 256 Done Done Done false 103495 Acked Write 256 Done Done Done false 103496 Acked Write 256 Done Done Done false 103497 Acked Write 256 Done Done Done false 103498 Acked Write 256 Done Done Done false 103499 Acked Write 256 Done Done Done false 103500 Acked Write 256 Done Done Done false 103501 Acked Write 256 Done Done Done false 103502 Acked Write 256 Done Done Done false 103503 Acked Write 256 Done Done Done false 103504 Acked Write 256 Done Done Done false 103505 Acked Write 256 Done Done Done false 103506 Acked Write 256 Done Done Done false 103507 Acked Write 256 Done Done Done false 103508 Acked Write 256 Done Done Done false 103509 Acked Write 256 Done Done Done false 103510 Acked Write 256 Done Done Done false 103511 Acked Write 256 Done Done Done false 103512 Acked Write 256 Done Done Done false 103513 Acked Write 256 Done Done Done false 103514 Acked Write 256 Done Done Done false 103515 Acked Write 256 Done Done Done false 103516 Acked Write 256 Done Done Done false 103517 Acked Write 256 Done Done Done false 103518 Acked Write 256 Done Done Done false 103519 Acked Write 256 Done Done Done false 103520 Acked Write 256 Done Done Done false 103521 Acked Write 256 Done Done Done false 103522 Acked Write 256 Done Done Done false 103523 Acked Write 256 Done Done Done false 103524 Acked Write 256 Done Done Done false 103525 Acked Write 256 Done Done Done false 103526 Acked Write 256 Done Done Done false 103527 Acked Write 256 Done Done Done false 103528 Acked Write 256 Done Done Done false 103529 Acked Write 256 Done Done Done false 103530 Acked Write 256 Done Done Done false 103531 Acked Write 256 Done Done Done false 103532 Acked Write 256 Done Done Done false 103533 Acked Write 256 Done Done Done false 103534 Acked Write 256 Done Done Done false 103535 Acked Write 256 Done Done Done false 103536 Acked Write 256 Done Done Done false 103537 Acked Write 256 Done Done Done false 103538 Acked Write 256 Done Done Done false 103539 Acked Write 256 Done Done Done false 103540 Acked Write 256 Done Done Done false 103541 Acked Write 256 Done Done Done false 103542 Acked Write 256 Done Done Done false 103543 Acked Write 256 Done Done Done false 103544 Acked Write 256 Done Done Done false 103545 Acked Write 256 Done Done Done false 103546 Acked Write 256 Done Done Done false 103547 Acked Write 256 Done Done Done false 103548 Acked Write 256 Done Done Done false 103549 Acked Write 256 Done Done Done false 103550 Acked Write 256 Done Done Done false 103551 Acked Write 256 Done Done Done false 103552 Acked Write 256 Done Done Done false 103553 Acked Write 256 Done Done Done false 103554 Acked Write 256 Done Done Done false 103555 Acked Write 256 Done Done Done false 103556 Acked Write 256 Done Done Done false 103557 Acked Write 256 Done Done Done false 103558 Acked Write 256 Done Done Done false 103559 Acked Write 256 Done Done Done false 103560 Acked Write 256 Done Done Done false 103561 Acked Write 256 Done Done Done false 103562 Acked Write 256 Done Done Done false 103563 Acked Write 256 Done Done Done false 103564 Acked Write 256 Done Done Done false 103565 Acked Write 256 Done Done Done false 103566 Acked Write 256 Done Done Done false 103567 Acked Write 256 Done Done Done false 103568 Acked Write 256 Done Done Done false 103569 Acked Write 256 Done Done Done false 103570 Acked Write 256 Done Done Done false 103571 Acked Write 256 Done Done Done false 103572 Acked Write 256 Done Done Done false 103573 Acked Write 256 Done Done Done false 103574 Acked Write 256 Done Done Done false 103575 Acked Write 256 Done Done Done false 103576 Acked Write 256 Done Done Done false 103577 Acked Write 256 Done Done Done false 103578 Acked Write 256 Done Done Done false 103579 Acked Write 256 Done Done Done false 103580 Acked Write 256 Done Done Done false 103581 Acked Write 256 Done Done Done false 103582 Acked Write 256 Done Done Done false 103583 Acked Write 256 Done Done Done false 103584 Acked Write 256 Done Done Done false 103585 Acked Write 256 Done Done Done false 103586 Acked Write 256 Done Done Done false 103587 Acked Write 256 Done Done Done false 103588 Acked Write 256 Done Done Done false 103589 Acked Write 256 Done Done Done false 103590 Acked Write 256 Done Done Done false 103591 Acked Write 256 Done Done Done false 103592 Acked Write 256 Done Done Done false 103593 Acked Write 256 Done Done Done false 103594 Acked Write 256 Done Done Done false 103595 Acked Write 256 Done Done Done false 103596 Acked Write 256 Done Done Done false 103597 Acked Write 256 Done Done Done false 103598 Acked Write 256 Done Done Done false 103599 Acked Write 256 Done Done Done false 103600 Acked Write 256 Done Done Done false 103601 Acked Write 256 Done Done Done false 103602 Acked Flush 0 Done Done Sent false STATES DS:0 DS:1 DS:2 TOTAL Sent 0 0 1 1 Done 444 444 443 1331 Skipped 0 0 0 0 Error 0 0 0 0 Last Flush: 103602 103602 103158 Downstairs last five completed: 103158 103157 103156 103155 103154 Upstairs last five completed: 103602 103601 103600 103599 103598 CLIENT: Up:0 ds:444 act:3 ---------------------------------------------------------------- Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0 Downstairs last five completed: 103602 103601 103600 103599 103598 Upstairs last five completed: 103602 103601 103600 103599 103598 CLIENT: Up:0 ds:0 act:3 CLIENT: All crucible jobs finished, exiting program