Max level shown:
1 [ Mar 2 08:14:18 Disabled. ]
2 [ Mar 2 08:14:18 Rereading configuration. ]
3 [ Mar 2 08:14:21 Enabled. ]
4 [ Mar 2 08:14:21 Rereading configuration. ]
5 [ Mar 2 08:14:25 Executing start method ("/opt/oxide/pantry/bin/crucible-pantry run -l [fd00:1122:3344:101::10]:17000"). ]
6 note: configured to log to "/dev/stdout"
72026-03-02T08:14:25.627ZINFOcrucible-pantry: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
82026-03-02T08:14:25.652ZINFOcrucible-pantry (dropshot): listening local_addr = [fd00:1122:3344:101::10]:17000
92026-03-02T08:14:25.738ZINFOcrucible-pantry: listen IP: [fd00:1122:3344:101::10]:17000
102026-03-02T08:14:32.270ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::b]:49438
112026-03-02T08:14:32.293ZINFOcrucible-pantry (dropshot): request completed latency_us = 10765 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49438 req_id = 3637c402-feb8-4aa9-b5c0-4ef94938602e response_code = 200 uri = /crucible/pantry/0
122026-03-02T08:14:32.398ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::a]:54620
132026-03-02T08:14:32.400ZINFOcrucible-pantry (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:54620 req_id = 55d84fcb-9561-4cfe-8c3d-78946fffd17c response_code = 200 uri = /crucible/pantry/0
142026-03-02T08:14:32.500ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::b]:63957
152026-03-02T08:14:32.500ZINFOcrucible-pantry (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:63957 req_id = 7f621b44-837b-4d7b-98bc-08e3b6fa574c response_code = 200 uri = /crucible/pantry/0
162026-03-02T08:14:32.501ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::b]:37789
172026-03-02T08:14:32.501ZINFOcrucible-pantry (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:37789 req_id = 392ecc1f-a991-4322-a220-3f2a1666a6ae response_code = 200 uri = /crucible/pantry/0
182026-03-02T08:14:32.604ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::a]:37778
192026-03-02T08:14:32.604ZINFOcrucible-pantry (dropshot): request completed latency_us = 31 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:37778 req_id = 43975724-ee42-49bf-b97a-8273ffce0a5c response_code = 200 uri = /crucible/pantry/0
202026-03-02T08:14:32.610ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::a]:39782
212026-03-02T08:14:32.610ZINFOcrucible-pantry (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:39782 req_id = b0898dce-dad4-420a-b838-9e68646d8284 response_code = 200 uri = /crucible/pantry/0
222026-03-02T08:14:32.611ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::a]:33956
232026-03-02T08:14:32.611ZINFOcrucible-pantry (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::a]:33956 req_id = 2b36a44c-708d-4ebc-bab9-e88bf01dae7f response_code = 200 uri = /crucible/pantry/0
242026-03-02T08:14:35.923ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::c]:34556
252026-03-02T08:14:35.923ZINFOcrucible-pantry (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:34556 req_id = a6d164c2-0731-4bca-98e9-5f92491f1f61 response_code = 200 uri = /crucible/pantry/0
262026-03-02T08:14:36.127ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::c]:33274
272026-03-02T08:14:36.127ZINFOcrucible-pantry (dropshot): request completed latency_us = 38 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:33274 req_id = 54db574a-7e08-4077-ba88-d45e94b6703a response_code = 200 uri = /crucible/pantry/0
282026-03-02T08:14:36.228ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::c]:53167
292026-03-02T08:14:36.228ZINFOcrucible-pantry (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::c]:53167 req_id = 73421515-cbc1-42b2-9073-bfd40afd692b response_code = 200 uri = /crucible/pantry/0
302026-03-02T08:14:57.074ZINFOcrucible-pantry (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::10]:17000 method = GET remote_addr = [fd00:1122:3344:101::b]:49438 req_id = 19345289-a020-4c92-9c6f-8c52314d4f26 response_code = 200 uri = /crucible/pantry/0
312026-03-02T08:14:57.196ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::b]:36902
322026-03-02T08:14:57.213ZINFOcrucible-pantry (datafile): no entry exists for volume 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d, constructing...
332026-03-02T08:14:57.226ZINFOcrucible-pantry (datafile): Upstairs starts
342026-03-02T08:14:57.226ZINFOcrucible-pantry (datafile): Crucible Version: BuildInfo { version: "0.0.1", git_sha: "7103cd3a3d7b0112d2949dd135db06fef0c156bb", git_commit_timestamp: "2025-11-10T22:24:53.000000000Z", git_branch: "main", rustc_semver: "1.90.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "1159e78c4747b02ef996e55082b704c09b970588", cargo_triple: "x86_64-unknown-illumos", debug: false, opt_level: 3, }
352026-03-02T08:14:57.226ZINFOcrucible-pantry (datafile): Upstairs <-> Downstairs Message Version: 13
362026-03-02T08:14:57.226ZINFOcrucible-pantry (datafile): Using region definition RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 32, uuid: 00000000-0000-0000-0000-000000000000, encrypted: true, database_read_version: 1, database_write_version: 1 }
372026-03-02T08:14:57.248ZINFOcrucible-pantry (datafile): Crucible 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d has session id: a2ca6d30-c868-44f8-8421-d5f485cf053f session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
382026-03-02T08:14:57.248ZINFOcrucible-pantry (datafile): Upstairs opts: Upstairs UUID: 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d, Targets: [[fd00:1122:3344:101::1a]:19000, [fd00:1122:3344:101::19]:19000, [fd00:1122:3344:101::13]:19000], lossy: false, flush_timeout: None, key populated: true, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
392026-03-02T08:14:57.248ZINFOcrucible-pantry (datafile): Crucible stats registered with UUID: 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
402026-03-02T08:14:57.251ZINFOcrucible-pantry (datafile): notify_task started job = notify session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
412026-03-02T08:14:57.251ZINFOcrucible-pantry (datafile): client is waiting for oneshot = io task client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
422026-03-02T08:14:57.251ZINFOcrucible-pantry (datafile): client is waiting for oneshot = io task client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
432026-03-02T08:14:57.253ZINFOcrucible-pantry (datafile): client is waiting for oneshot = io task client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
442026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): volume 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d constructed ok
452026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): The guest has requested activation
462026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d active request set session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
472026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
482026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
492026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): sending connect oneshot to client = downstairs client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
502026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::13]:19000 = io task client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
512026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::1a]:19000 = io task client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
522026-03-02T08:14:57.256ZINFOcrucible-pantry (datafile): connecting to [fd00:1122:3344:101::19]:19000 = io task client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
532026-03-02T08:14:57.257ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::10]:65495) = io task client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
542026-03-02T08:14:57.257ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::10]:56060) = io task client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
552026-03-02T08:14:57.257ZINFOcrucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:101::10]:62301) = io task client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
562026-03-02T08:14:57.260ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::1a]:19000) has region UUID 15c919d5-745c-473c-9ea1-09704c3597b6 = downstairs client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
572026-03-02T08:14:57.260ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::19]:19000) has region UUID fc4683d6-c09e-4837-9f9f-bba8d161e733 = downstairs client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
582026-03-02T08:14:57.260ZINFOcrucible-pantry (datafile): Waiting for more clients to be ready session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
592026-03-02T08:14:57.260ZINFOcrucible-pantry (datafile): Waiting for more clients to be ready session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
602026-03-02T08:14:57.261ZINFOcrucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:101::13]:19000) has region UUID 20cce076-f849-43c4-8fb8-1e0b08e9c34f = downstairs client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
612026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
622026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
632026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
642026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
652026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
662026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
672026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
682026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
692026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
702026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Max found gen is 1 = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
712026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Generation requested: 1 >= found:1 = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
722026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Next flush: 1 = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
732026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): All extents match = downstairs session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
742026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): No downstairs reconciliation required session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
752026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 0 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
762026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 1 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
772026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Transition from WaitQuorum to Active = downstairs client = 2 session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
782026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): All required reconciliation work is completed session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
792026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Set Downstairs and Upstairs active after reconciliation session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
802026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d is now active with session: a2ca6d30-c868-44f8-8421-d5f485cf053f session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
812026-03-02T08:14:57.262ZINFOcrucible-pantry (datafile): Set Active after no reconciliation session_id = a2ca6d30-c868-44f8-8421-d5f485cf053f
822026-03-02T08:14:57.264ZINFOcrucible-pantry (datafile): The guest has finished waiting for activation
832026-03-02T08:14:57.264ZINFOcrucible-pantry (datafile): Activated sub_volume 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d
842026-03-02T08:14:57.264ZINFOcrucible-pantry (datafile): Activated sub_volume 997df5aa-7e8b-41a8-bd68-02818bc832ae
852026-03-02T08:14:57.264ZINFOcrucible-pantry (datafile): volume 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d activated ok
862026-03-02T08:14:57.264ZINFOcrucible-pantry (datafile): volume 5957159d-5afb-4d5c-9f1a-23fcc0f21e3d constructed and inserted ok
872026-03-02T08:14:57.264ZINFOcrucible-pantry (dropshot): request completed latency_us = 67924 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:36902 req_id = e4f0914b-0bad-41cf-8155-6807db26444a response_code = 200 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d
882026-03-02T08:14:57.988ZINFOcrucible-pantry (dropshot): accepted connection local_addr = [fd00:1122:3344:101::10]:17000 remote_addr = [fd00:1122:3344:101::b]:39943
892026-03-02T08:14:57.993ZINFOcrucible-pantry (dropshot): request completed latency_us = 5354 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = 08d00058-bb7f-43d0-ae15-f2ab8c8baaba response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write
902026-03-02T08:14:58.561ZINFOcrucible-pantry (dropshot): request completed latency_us = 1594 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = f2a65873-ef03-49e1-afa4-ea1c8da246e5 response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write
912026-03-02T08:14:59.086ZINFOcrucible-pantry (dropshot): request completed latency_us = 1429 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = e982331f-573e-4d55-912b-235d08059a9f response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write
922026-03-02T08:14:59.597ZINFOcrucible-pantry (dropshot): request completed latency_us = 1045 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = 3f673ffb-228a-4c38-9543-fb156210df5e response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write
932026-03-02T08:15:00.095ZINFOcrucible-pantry (dropshot): request completed latency_us = 1326 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = 3ba55950-2347-4c59-8576-0f133a93bc64 response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write
942026-03-02T08:15:00.607ZINFOcrucible-pantry (dropshot): request completed latency_us = 1091 local_addr = [fd00:1122:3344:101::10]:17000 method = POST remote_addr = [fd00:1122:3344:101::b]:39943 req_id = f39c4a80-6833-4564-9933-b5f89f251d5a response_code = 204 uri = /crucible/pantry/0/volume/5957159d-5afb-4d5c-9f1a-23fcc0f21e3d/bulk-write