Max level shown:
1 [ Aug 1 05:13:26 Disabled. ]
2 [ Aug 1 05:13:26 Rereading configuration. ]
3 [ Aug 1 05:13:30 Enabled. ]
4 [ Aug 1 05:13:31 Rereading configuration. ]
5 [ Aug 1 05:13:34 Executing start method ("/opt/oxide/pantry/bin/crucible-pantry run -l [fd00:1122:3344:101::11]:17000"). ]
6 note: configured to log to "/dev/stdout"
72024-08-01T05:13:34.521ZINFOcrucible-pantry (dropshot): listening local_addr = [fd00:1122:3344:101::11]:17000
82024-08-01T05:13:34.542ZINFOcrucible-pantry: listen IP: [fd00:1122:3344:101::11]:17000
92024-08-01T05:14:55.358ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::11]:17000 remote_addr = [fd00:1122:3344:101::a]:41360
102024-08-01T05:14:55.385ZINFOcrucible-pantry (datafile): no entry exists for volume 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce, constructing...
112024-08-01T05:14:55.395ZINFOcrucible-pantry (datafile): Upstairs starts
122024-08-01T05:14:55.399ZINFOcrucible-pantry (datafile): Crucible Version: BuildInfo { version: "0.0.1", git_sha: "e58ca3693cb9ce0438947beba10e97ee38a0966b", git_commit_timestamp: "2024-07-18T20:38:05.000000000Z", git_branch: "main", rustc_semver: "1.76.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "07dca489ac2d933c78d3c5158e3f43beefeb02ce", cargo_triple: "x86_64-unknown-illumos", debug: false, opt_level: 3, }
132024-08-01T05:14:55.399ZINFOcrucible-pantry (datafile): Upstairs <-> Downstairs Message Version: 9
142024-08-01T05:14:55.399ZINFOcrucible-pantry (datafile): Using region definition RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 32, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }
152024-08-01T05:14:55.403ZINFOcrucible-pantry (datafile): Crucible 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce has session id: 09c6dc67-443a-4fab-b867-cefd83950c06 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
162024-08-01T05:14:55.405ZINFOcrucible-pantry (datafile): Upstairs opts: Upstairs UUID: 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce, Targets: [[fd00:1122:3344:101::19]:19000, [fd00:1122:3344:101::1a]:19000, [fd00:1122:3344:101::17]:19000], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
172024-08-01T05:14:55.405ZINFOcrucible-pantry (datafile): Crucible stats registered with UUID: 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
182024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): volume 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce constructed ok
192024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): The guest has requested activation
202024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce active request set session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
212024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
222024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): client set_active_request while in New; waiting... = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
232024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
242024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): client set_active_request while in New; waiting... = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
252024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
262024-08-01T05:14:55.492ZINFOcrucible-pantry (datafile): client set_active_request while in New; waiting... = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
272024-08-01T05:14:55.495ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::17]:19000 = io task client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
282024-08-01T05:14:55.495ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::1a]:19000 = io task client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
292024-08-01T05:14:55.495ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::19]:19000 = io task client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
302024-08-01T05:14:55.497ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::11]:44420) = io task client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
312024-08-01T05:14:55.500ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::11]:58723) = io task client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
322024-08-01T05:14:55.500ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::11]:51778) = io task client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
332024-08-01T05:14:55.501ZINFOcrucible-pantry (datafile): ds_transition from New to WaitActive = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
342024-08-01T05:14:55.501ZINFOcrucible-pantry (datafile): [0] Transition from New to WaitActive = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
352024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::19]:19000) has region UUID e0c1ec56-40a1-490d-a438-0b1dc954662d = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
362024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): ds_transition from New to WaitActive = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
372024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [1] Transition from New to WaitActive = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
382024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): ds_transition from New to WaitActive = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
392024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [2] Transition from New to WaitActive = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
402024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::1a]:19000) has region UUID 4e04b02b-b088-4545-a442-b55812ca3190 = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
412024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::17]:19000) has region UUID a54e6f3f-6c3f-4530-86ae-c5fdd6d4bc67 = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
422024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
432024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
442024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Waiting for more clients to be ready session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
452024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
462024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
472024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Waiting for more clients to be ready session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
482024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
492024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
502024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
512024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
522024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
532024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
542024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
552024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
562024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
572024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
582024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
592024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Max found gen is 1 = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
602024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Generation requested: 1 >= found:1 = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
612024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Next flush: 1 = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
622024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): All extents match = downstairs session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
632024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): No downstairs reconciliation required session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
642024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 0 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
652024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 1 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
662024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 2 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
672024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): All required reconciliation work is completed session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
682024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Set Downstairs and Upstairs active after reconciliation session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
692024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce is now active with session: 09c6dc67-443a-4fab-b867-cefd83950c06 session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
702024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): Set Active after no reconciliation session_id = 09c6dc67-443a-4fab-b867-cefd83950c06
712024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): The guest has finished waiting for activation
722024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): volume 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce activated ok
732024-08-01T05:14:55.522ZINFOcrucible-pantry (datafile): volume 5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce constructed and inserted ok
742024-08-01T05:14:55.522ZINFOcrucible-pantry (dropshot): request completed latency_us = 143740 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:41360 req_id = a9d8f130-23fd-4e37-bf62-7c3f0d571085 response_code = 200 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce
752024-08-01T05:14:56.991ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::11]:17000 remote_addr = [fd00:1122:3344:101::a]:43870
762024-08-01T05:14:56.997ZINFOcrucible-pantry (dropshot): request completed latency_us = 5240 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 3a666f18-4bd3-41b9-9caa-c9107f6bb98e response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
772024-08-01T05:14:57.098ZINFOcrucible-pantry (dropshot): request completed latency_us = 3106 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = b06cb4de-9c74-4a3a-b17d-a3f7fcc251eb response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
782024-08-01T05:14:57.590ZINFOcrucible-pantry (dropshot): request completed latency_us = 2962 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 4a303fc7-5c63-4d27-93ed-17e9e9d5db1e response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
792024-08-01T05:14:58.490ZINFOcrucible-pantry (dropshot): request completed latency_us = 2740 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 1df30d56-9a22-4868-8a75-d07d5b8e08d8 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
802024-08-01T05:14:59.626ZINFOcrucible-pantry (dropshot): request completed latency_us = 2677 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 44ccc072-9183-4279-a224-65518fde3d4c response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
812024-08-01T05:14:59.738ZINFOcrucible-pantry (dropshot): request completed latency_us = 61144 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 4630b514-ebbd-4102-b01c-842450ce23d5 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
822024-08-01T05:14:59.802ZINFOcrucible-pantry (dropshot): request completed latency_us = 3547 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = 3ebd96e2-5ff5-4d09-840b-bd5800eee8f4 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
832024-08-01T05:14:59.929ZINFOcrucible-pantry (dropshot): request completed latency_us = 3004 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = ec5aa288-1b5b-4776-8997-7dc3a6ca28b6 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
842024-08-01T05:15:00.563ZINFOcrucible-pantry (dropshot): request completed latency_us = 3552 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = d504a1fc-e38d-419f-9f89-f65ab8a5a718 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
852024-08-01T05:15:01.508ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::11]:17000 remote_addr = [fd00:1122:3344:101::a]:60712
862024-08-01T05:15:01.511ZINFOcrucible-pantry (dropshot): request completed latency_us = 2947 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:60712 req_id = 04f6742c-8dcb-4afa-ba46-eb1ea5b300aa response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
872024-08-01T05:15:01.528ZINFOcrucible-pantry (dropshot): request completed latency_us = 119754 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = fcd1add1-18b7-4ebf-95d0-6871ecd55a13 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
882024-08-01T05:15:01.767ZINFOcrucible-pantry (dropshot): request completed latency_us = 3267 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = b7aab0f9-0335-4cbb-89aa-c17e8416d5b6 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write
892024-08-01T05:15:01.882ZINFOcrucible-pantry (dropshot): request completed latency_us = 3516 local_addr = [fd00:1122:3344:101::11]:17000 method = POST remote_addr = [fd00:1122:3344:101::a]:43870 req_id = c6461df1-bce0-4c47-9706-8e98b6a84b57 response_code = 204 uri = /crucible/pantry/0/volume/5cb82e1e-011c-44b8-a2e0-1e0ecf68c6ce/bulk-write