Max level shown:
12025-03-07T22:45:01.873ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::a]:53169
22025-03-07T22:45:01.873ZINFOcrucible-pantry (datafile): detach removing entry for volume ac2306e4-d356-4e16-a9f4-324d4a447dee
32025-03-07T22:45:01.873ZINFOcrucible-pantry (datafile): detaching volume ac2306e4-d356-4e16-a9f4-324d4a447dee
42025-03-07T22:45:01.873ZINFOcrucible-pantry (datafile): Request to deactivate this guest session_id = b559f318-7cf0-40ce-b73f-02e4936db512
52025-03-07T22:45:01.873ZINFOcrucible-pantry (datafile): checking for deactivation session_id = b559f318-7cf0-40ce-b73f-02e4936db512
62025-03-07T22:45:01.873ZINFOcrucible-pantry (datafile): [0] deactivate, no work so YES = downstairs session_id = b559f318-7cf0-40ce-b73f-02e4936db512
72025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): halting IO task due to Deactivated = downstairs client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
82025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): deactivated client 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
92025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): [1] deactivate, no work so YES = downstairs session_id = b559f318-7cf0-40ce-b73f-02e4936db512
102025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): halting IO task due to Deactivated = downstairs client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
112025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): deactivated client 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
122025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): [2] deactivate, no work so YES = downstairs session_id = b559f318-7cf0-40ce-b73f-02e4936db512
132025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): halting IO task due to Deactivated = downstairs client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
142025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): deactivated client 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
152025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): not ready to deactivate due to state Stopping(Deactivated) = downstairs client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
162025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client stopping due to Ok(Deactivated) = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
172025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client stopping due to Ok(Deactivated) = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
182025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(RequestedStop) = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
192025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client stopping due to Ok(Deactivated) = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
202025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): downstairs task for 2 stopped due to RequestedStop session_id = b559f318-7cf0-40ce-b73f-02e4936db512
212025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(RequestedStop) = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
222025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): checking for deactivation session_id = b559f318-7cf0-40ce-b73f-02e4936db512
232025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): not ready to deactivate due to state Stopping(Deactivated) = downstairs client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
242025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(RequestedStop) = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
252025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): downstairs task for 1 stopped due to RequestedStop session_id = b559f318-7cf0-40ce-b73f-02e4936db512
262025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): checking for deactivation session_id = b559f318-7cf0-40ce-b73f-02e4936db512
272025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
282025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): not ready to deactivate due to state Stopping(Deactivated) = downstairs client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
292025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): downstairs task for 0 stopped due to RequestedStop session_id = b559f318-7cf0-40ce-b73f-02e4936db512
302025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): checking for deactivation session_id = b559f318-7cf0-40ce-b73f-02e4936db512
312025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): ready to deactivate from state Connecting { state: Start { auto_promote: false }, mode: New } = downstairs client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
322025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): ready to deactivate from state Connecting { state: Start { auto_promote: false }, mode: New } = downstairs client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
332025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): ready to deactivate from state Connecting { state: Start { auto_promote: false }, mode: New } = downstairs client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
342025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): All DS in the proper state! -> INIT session_id = b559f318-7cf0-40ce-b73f-02e4936db512
352025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
362025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
372025-03-07T22:45:01.875ZINFOcrucible-pantry (dropshot): request completed latency_us = 2545 local_addr = [fd00:1122:3344:101::10]:17000 method = DELETE remote_addr = [fd00:1122:3344:101::a]:53169 req_id = de365572-4295-4573-977a-a11bbb016dff response_code = 204 uri = /crucible/pantry/0/volume/ac2306e4-d356-4e16-a9f4-324d4a447dee
382025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): Guest handle has been dropped
392025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client IO task stopped during sleep: Err(RecvError(())) = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
402025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(QueueClosed) = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
412025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task could not reply to main task; shutting down? = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
422025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 0 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
432025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client IO task stopped during sleep: Err(RecvError(())) = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
442025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(QueueClosed) = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
452025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task could not reply to main task; shutting down? = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
462025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 1 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
472025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client IO task stopped during sleep: Err(RecvError(())) = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
482025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task is sending Done(QueueClosed) = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
492025-03-07T22:45:01.875ZWARNcrucible-pantry (datafile): client task could not reply to main task; shutting down? = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
502025-03-07T22:45:01.875ZINFOcrucible-pantry (datafile): client task is exiting = io task client = 2 session_id = b559f318-7cf0-40ce-b73f-02e4936db512
512025-03-07T22:45:01.895ZINFOcrucible-pantry (datafile): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] job = notify session_id = b559f318-7cf0-40ce-b73f-02e4936db512
522025-03-07T22:45:01.950ZERROcrucible-pantry (datafile): one of the notify channels was closed! job = notify session_id = b559f318-7cf0-40ce-b73f-02e4936db512
532025-03-07T22:45:01.950ZINFOcrucible-pantry (datafile): notify_task exiting job = notify session_id = b559f318-7cf0-40ce-b73f-02e4936db512
542025-03-07T22:45:21.766ZINFOcrucible-pantry (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:40673 req_id = 267576af-4691-487c-a9f6-9febd2d2db32 response_code = 200 uri = /crucible/pantry/0
552025-03-07T22:45:21.774ZINFOcrucible-pantry (dropshot): request completed latency_us = 16 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:46327 req_id = 21112be2-dd7e-4f23-bd05-297070e60ff3 response_code = 200 uri = /crucible/pantry/0
562025-03-07T22:45:21.999ZINFOcrucible-pantry (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:57672 req_id = 7b5e87fc-a517-4c58-b756-3bc526fec6e2 response_code = 200 uri = /crucible/pantry/0
572025-03-07T22:45:22.017ZINFOcrucible-pantry (dropshot): request completed latency_us = 21 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:42789 req_id = 5b15a3a5-5e3b-4e84-90a4-1a0955ccd140 response_code = 200 uri = /crucible/pantry/0
582025-03-07T22:45:22.173ZINFOcrucible-pantry (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:43416 req_id = 4e6a3ca3-abde-4aac-a3f4-e42ec6688986 response_code = 200 uri = /crucible/pantry/0
592025-03-07T22:45:22.194ZINFOcrucible-pantry (dropshot): request completed latency_us = 16 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49453 req_id = 9529f249-4521-4691-92df-e18d8bb16ae0 response_code = 200 uri = /crucible/pantry/0
602025-03-07T22:45:23.686ZINFOcrucible-pantry (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:45472 req_id = 55e9172c-7b76-452a-a0a2-34358b1770f7 response_code = 200 uri = /crucible/pantry/0
612025-03-07T22:45:24.203ZINFOcrucible-pantry (dropshot): request completed latency_us = 36 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:47137 req_id = 74c27bc9-7f57-4e1d-b8d8-aff630a64333 response_code = 200 uri = /crucible/pantry/0
622025-03-07T22:45:24.208ZINFOcrucible-pantry (dropshot): request completed latency_us = 20 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:34818 req_id = db1e82cc-28ef-4519-9d2f-98daf167a1a4 response_code = 200 uri = /crucible/pantry/0
632025-03-07T22:45:51.776ZINFOcrucible-pantry (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:40673 req_id = 31fdd312-bcf9-4527-87f5-7204af36f84b response_code = 200 uri = /crucible/pantry/0
642025-03-07T22:45:51.789ZINFOcrucible-pantry (dropshot): request completed latency_us = 11 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:46327 req_id = 02f67b99-2c26-49d8-8e00-e7f9c445fb65 response_code = 200 uri = /crucible/pantry/0
652025-03-07T22:45:52.018ZINFOcrucible-pantry (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:57672 req_id = cc3e83b5-1cb7-4b56-865f-c768d6ac8801 response_code = 200 uri = /crucible/pantry/0
662025-03-07T22:45:52.029ZINFOcrucible-pantry (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:42789 req_id = 22bca513-feb3-42da-96c5-1aefe4dbe7b4 response_code = 200 uri = /crucible/pantry/0
672025-03-07T22:45:52.194ZINFOcrucible-pantry (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:43416 req_id = e4b36d1b-62ae-4e60-957c-7086c2282867 response_code = 200 uri = /crucible/pantry/0
682025-03-07T22:45:52.196ZINFOcrucible-pantry (dropshot): request completed latency_us = 15 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49453 req_id = 6061e106-85fd-4691-a2e7-77d4385d0d58 response_code = 200 uri = /crucible/pantry/0
692025-03-07T22:45:53.708ZINFOcrucible-pantry (dropshot): request completed latency_us = 38 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:45472 req_id = 57e165f8-166d-4c15-9dcf-161170aab9a0 response_code = 200 uri = /crucible/pantry/0
702025-03-07T22:45:54.212ZINFOcrucible-pantry (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:34818 req_id = 5dbd3216-dfc1-4d54-8fb3-8eb7944dfe8d response_code = 200 uri = /crucible/pantry/0
712025-03-07T22:45:54.220ZINFOcrucible-pantry (dropshot): request completed latency_us = 21 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:47137 req_id = 19f80f9a-7bb1-4ba3-979e-e2a7b11d1edb response_code = 200 uri = /crucible/pantry/0
722025-03-07T22:46:21.784ZINFOcrucible-pantry (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:40673 req_id = d70d466f-caf7-4038-b9f1-b66ef09d94fb response_code = 200 uri = /crucible/pantry/0
732025-03-07T22:46:21.798ZINFOcrucible-pantry (dropshot): request completed latency_us = 17 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:46327 req_id = 21b67661-474b-4b4a-98d2-99eb93e9644c response_code = 200 uri = /crucible/pantry/0
742025-03-07T22:46:22.030ZINFOcrucible-pantry (dropshot): request completed latency_us = 38 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:57672 req_id = 6eb27aa4-1b98-4b8e-b3ff-bd12d96cde55 response_code = 200 uri = /crucible/pantry/0
752025-03-07T22:46:22.035ZINFOcrucible-pantry (dropshot): request completed latency_us = 17 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:42789 req_id = ebe3a34a-eeef-4385-99d2-fb4868565993 response_code = 200 uri = /crucible/pantry/0
762025-03-07T22:46:22.202ZINFOcrucible-pantry (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49453 req_id = 5804e74b-daeb-4f56-acd1-94d6d40d8ddf response_code = 200 uri = /crucible/pantry/0
772025-03-07T22:46:22.211ZINFOcrucible-pantry (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:43416 req_id = 08ce7278-617f-40c1-b4e3-7fb7bfdb7907 response_code = 200 uri = /crucible/pantry/0
782025-03-07T22:46:23.726ZINFOcrucible-pantry (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:45472 req_id = 24e4f4bf-d90c-42f3-806f-a4507a095fa0 response_code = 200 uri = /crucible/pantry/0
792025-03-07T22:46:24.214ZINFOcrucible-pantry (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:34818 req_id = 96aa96ed-9d39-4792-8cbd-67c07e4e0313 response_code = 200 uri = /crucible/pantry/0
802025-03-07T22:46:24.232ZINFOcrucible-pantry (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:47137 req_id = 0ca63c35-801e-4784-9aa5-a8607d78dc37 response_code = 200 uri = /crucible/pantry/0
812025-03-07T22:46:51.797ZINFOcrucible-pantry (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:40673 req_id = 03ac5b51-a84d-4059-be6d-81a22d36f4c5 response_code = 200 uri = /crucible/pantry/0
822025-03-07T22:46:51.818ZINFOcrucible-pantry (dropshot): request completed latency_us = 10 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:46327 req_id = 97075a21-1800-4c23-b1ed-de21c5b4e338 response_code = 200 uri = /crucible/pantry/0
832025-03-07T22:46:52.034ZINFOcrucible-pantry (dropshot): request completed latency_us = 42 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:57672 req_id = ffcb8f2a-82cb-499f-bdc1-cad555fbda92 response_code = 200 uri = /crucible/pantry/0
842025-03-07T22:46:52.053ZINFOcrucible-pantry (dropshot): request completed latency_us = 15 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:42789 req_id = 8331f8e1-57fc-422f-9734-59fe254dd9e4 response_code = 200 uri = /crucible/pantry/0
852025-03-07T22:46:52.214ZINFOcrucible-pantry (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49453 req_id = 79d3aed4-52ce-42d8-8270-8aa1867f7579 response_code = 200 uri = /crucible/pantry/0
862025-03-07T22:46:52.215ZINFOcrucible-pantry (dropshot): request completed latency_us = 11 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:43416 req_id = a5544b0a-4af3-41ca-a282-bcbcd445ac5b response_code = 200 uri = /crucible/pantry/0
872025-03-07T22:46:53.740ZINFOcrucible-pantry (dropshot): request completed latency_us = 40 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:45472 req_id = b71d1a5c-5fa7-4fdd-a0ce-aba14a186a7a response_code = 200 uri = /crucible/pantry/0