Max level shown:
1 [ Dec 24 02:31:14 Enabled. ]
2 [ Dec 24 02:31:14 Rereading configuration. ]
3 [ Dec 24 02:31:16 Rereading configuration. ]
4 [ Dec 24 02:31:18 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-12-24T02:31:18.903ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-12-24T02:31:18.903ZINFOcrucible-agent: dataset: "oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible"
82025-12-24T02:31:18.903ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
92025-12-24T02:31:18.903ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-12-24T02:31:18.920ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-12-24T02:31:18.991ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
122025-12-24T02:32:30.006ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:60553
132025-12-24T02:32:30.031ZINFOcrucible-agent (datafile): region cdcf79f2-ce9a-45d4-aeae-23755422b34d state: Requested
142025-12-24T02:32:30.033ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-12-24T02:32:30.033ZINFOcrucible-agent (dropshot): request completed latency_us = 13196 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60553 req_id = c913be9c-1ad4-437c-8aed-4763a8ee0085 response_code = 200 uri = /crucible/0/regions
162025-12-24T02:32:30.044ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
172025-12-24T02:32:30.044ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
182025-12-24T02:32:30.093ZINFOcrucible-agent (worker): creating region Region { id: RegionId("cdcf79f2-ce9a-45d4-aeae-23755422b34d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d" region = cdcf79f2-ce9a-45d4-aeae-23755422b34d
192025-12-24T02:32:30.096ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "cdcf79f2-ce9a-45d4-aeae-23755422b34d" "--data" "/data/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = cdcf79f2-ce9a-45d4-aeae-23755422b34d
202025-12-24T02:32:30.207ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60553 req_id = 7e542fac-5917-4293-8812-405894a06828 response_code = 200 uri = /crucible/0/regions
212025-12-24T02:32:30.430ZINFOcrucible-agent (worker): region files created ok region = cdcf79f2-ce9a-45d4-aeae-23755422b34d
222025-12-24T02:32:30.430ZINFOcrucible-agent (datafile): region cdcf79f2-ce9a-45d4-aeae-23755422b34d state: Requested -> Created
232025-12-24T02:32:30.430ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-12-24T02:32:30.431ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-cdcf79f2-ce9a-45d4-aeae-23755422b34d
252025-12-24T02:32:30.436ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-cdcf79f2-ce9a-45d4-aeae-23755422b34d
262025-12-24T02:32:30.449ZINFOcrucible-agent (worker): creating config property group
272025-12-24T02:32:30.452ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-cdcf79f2-ce9a-45d4-aeae-23755422b34d
282025-12-24T02:32:30.453ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
292025-12-24T02:32:30.453ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-12-24T02:32:30.453ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
312025-12-24T02:32:30.453ZINFOcrucible-agent (worker): commit
322025-12-24T02:32:30.457ZINFOcrucible-agent (worker): ok!
332025-12-24T02:32:30.461ZINFOcrucible-agent (worker): SMF ok!
342025-12-24T02:32:30.864ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60553 req_id = a5b729e8-c8f6-4ca1-b520-e712315cafba response_code = 200 uri = /crucible/0/regions
352025-12-24T02:32:31.042ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:42651
362025-12-24T02:32:31.042ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42651 req_id = 885a07a9-1e35-4a57-af62-d41ed350c484 response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
372025-12-24T02:32:31.179ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:46553
382025-12-24T02:32:31.179ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:46553 req_id = 2f77b425-a44e-4886-b0c5-8158b554812a response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
392025-12-24T02:32:31.216ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:49180
402025-12-24T02:32:31.216ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49180 req_id = 9dd526fe-4442-486c-8101-2580b48b2d52 response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
412025-12-24T02:32:31.572ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60553 req_id = 30ed3dd4-669f-4f22-8d08-72350d9c2812 response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
422025-12-24T02:32:31.581ZINFOcrucible-agent (datafile): path is "/data/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d"
432025-12-24T02:32:31.592ZINFOcrucible-agent (datafile): dataset is oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
442025-12-24T02:32:31.600ZINFOcrucible-agent (datafile): region cdcf79f2-ce9a-45d4-aeae-23755422b34d state: Created -> Tombstoned
452025-12-24T02:32:31.601ZINFOcrucible-agent (worker): applying SMF actions before removal...
462025-12-24T02:32:31.601ZINFOcrucible-agent (dropshot): request completed latency_us = 19791 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:60553 req_id = f0ea3279-b653-4358-96db-f6ed387bdd45 response_code = 204 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
472025-12-24T02:32:31.601ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-cdcf79f2-ce9a-45d4-aeae-23755422b34d (instance states: (Some(Online), None))
482025-12-24T02:32:31.605ZINFOcrucible-agent (worker): SMF ok!
492025-12-24T02:32:31.607ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60553 req_id = 084b7fd1-cced-464a-b173-63ce016410e4 response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
502025-12-24T02:32:31.615ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d" region = cdcf79f2-ce9a-45d4-aeae-23755422b34d
512025-12-24T02:32:31.691ZINFOcrucible-agent (datafile): region cdcf79f2-ce9a-45d4-aeae-23755422b34d state: Tombstoned -> Destroyed
522025-12-24T02:32:31.718ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:60553 req_id = 80251421-3514-4b10-9589-25b87b9e1073 response_code = 200 uri = /crucible/0/regions/cdcf79f2-ce9a-45d4-aeae-23755422b34d
532025-12-24T02:44:10.776ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::b]:44541
542025-12-24T02:44:10.777ZINFOcrucible-agent (datafile): region 9d7acfa6-8037-428d-a7b3-3f779f6c4dd8 state: Requested
552025-12-24T02:44:10.777ZINFOcrucible-agent (dropshot): request completed latency_us = 866 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:44541 req_id = ce7809d2-afb0-448c-a17f-70a4060ca7a3 response_code = 200 uri = /crucible/0/regions
562025-12-24T02:44:10.777ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
572025-12-24T02:44:10.788ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
582025-12-24T02:44:10.788ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
592025-12-24T02:44:10.837ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9d7acfa6-8037-428d-a7b3-3f779f6c4dd8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9d7acfa6-8037-428d-a7b3-3f779f6c4dd8" region = 9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
602025-12-24T02:44:10.837ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9d7acfa6-8037-428d-a7b3-3f779f6c4dd8" "--data" "/data/regions/9d7acfa6-8037-428d-a7b3-3f779f6c4dd8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
612025-12-24T02:44:10.922ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:44541 req_id = 699f9f6b-ddda-4cfb-9039-453776667997 response_code = 200 uri = /crucible/0/regions
622025-12-24T02:44:11.082ZINFOcrucible-agent (worker): region files created ok region = 9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
632025-12-24T02:44:11.082ZINFOcrucible-agent (datafile): region 9d7acfa6-8037-428d-a7b3-3f779f6c4dd8 state: Requested -> Created
642025-12-24T02:44:11.082ZINFOcrucible-agent (worker): applying SMF actions post create...
652025-12-24T02:44:11.083ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
662025-12-24T02:44:11.093ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
672025-12-24T02:44:11.113ZINFOcrucible-agent (worker): creating config property group
682025-12-24T02:44:11.116ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
692025-12-24T02:44:11.116ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9d7acfa6-8037-428d-a7b3-3f779f6c4dd8
702025-12-24T02:44:11.116ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
712025-12-24T02:44:11.116ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
722025-12-24T02:44:11.117ZINFOcrucible-agent (worker): commit
732025-12-24T02:44:11.120ZINFOcrucible-agent (worker): ok!
742025-12-24T02:44:11.124ZINFOcrucible-agent (worker): SMF ok!
752025-12-24T02:44:11.187ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:44541 req_id = 03e56bca-9d72-489d-a5b2-ed1314733133 response_code = 200 uri = /crucible/0/regions
762025-12-24T02:44:17.788ZINFOcrucible-agent (datafile): region ba5341c8-8d9f-42e4-98a5-162d07e4f72d state: Requested
772025-12-24T02:44:17.789ZINFOcrucible-agent (dropshot): request completed latency_us = 259 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:44541 req_id = e8682fee-8ec7-469f-8051-78164223d28e response_code = 200 uri = /crucible/0/regions
782025-12-24T02:44:17.789ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
792025-12-24T02:44:17.796ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
802025-12-24T02:44:17.796ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
812025-12-24T02:44:17.850ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ba5341c8-8d9f-42e4-98a5-162d07e4f72d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d" region = ba5341c8-8d9f-42e4-98a5-162d07e4f72d
822025-12-24T02:44:17.850ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ba5341c8-8d9f-42e4-98a5-162d07e4f72d" "--data" "/data/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ba5341c8-8d9f-42e4-98a5-162d07e4f72d
832025-12-24T02:44:17.922ZINFOcrucible-agent (worker): region files created ok region = ba5341c8-8d9f-42e4-98a5-162d07e4f72d
842025-12-24T02:44:17.922ZINFOcrucible-agent (datafile): region ba5341c8-8d9f-42e4-98a5-162d07e4f72d state: Requested -> Created
852025-12-24T02:44:17.922ZINFOcrucible-agent (worker): applying SMF actions post create...
862025-12-24T02:44:17.927ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ba5341c8-8d9f-42e4-98a5-162d07e4f72d
872025-12-24T02:44:17.935ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ba5341c8-8d9f-42e4-98a5-162d07e4f72d
882025-12-24T02:44:17.944ZINFOcrucible-agent (worker): creating config property group
892025-12-24T02:44:17.946ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ba5341c8-8d9f-42e4-98a5-162d07e4f72d
902025-12-24T02:44:17.947ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
912025-12-24T02:44:17.947ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
922025-12-24T02:44:17.947ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
932025-12-24T02:44:17.947ZINFOcrucible-agent (worker): commit
942025-12-24T02:44:17.950ZINFOcrucible-agent (worker): ok!
952025-12-24T02:44:17.954ZINFOcrucible-agent (worker): SMF ok!
962025-12-24T02:44:17.963ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:44541 req_id = ad25409d-1301-4972-b227-999c6cec9a09 response_code = 200 uri = /crucible/0/regions
972025-12-24T02:44:44.947ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:44541 req_id = bef902aa-7e6e-4d33-ba97-89ad56116c8f response_code = 200 uri = /crucible/0/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
982025-12-24T02:44:44.950ZINFOcrucible-agent (datafile): path is "/data/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d"
992025-12-24T02:44:44.961ZINFOcrucible-agent (datafile): dataset is oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
1002025-12-24T02:44:44.968ZINFOcrucible-agent (datafile): region ba5341c8-8d9f-42e4-98a5-162d07e4f72d state: Created -> Tombstoned
1012025-12-24T02:44:44.969ZINFOcrucible-agent (dropshot): request completed latency_us = 19064 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:44541 req_id = 5ec8936f-1b6f-435d-857c-d3df4a18f04d response_code = 204 uri = /crucible/0/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
1022025-12-24T02:44:44.969ZINFOcrucible-agent (worker): applying SMF actions before removal...
1032025-12-24T02:44:44.970ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ba5341c8-8d9f-42e4-98a5-162d07e4f72d (instance states: (Some(Online), None))
1042025-12-24T02:44:44.971ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:44541 req_id = ece6a0f2-a88d-46a2-90c4-68d54cb7cc08 response_code = 200 uri = /crucible/0/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d
1052025-12-24T02:44:44.984ZINFOcrucible-agent (worker): SMF ok!
1062025-12-24T02:44:44.991ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_60ea9ba7-6df6-4ad8-805e-f955f3ddbf2a/crucible/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d" region = ba5341c8-8d9f-42e4-98a5-162d07e4f72d
1072025-12-24T02:44:45.042ZINFOcrucible-agent (datafile): region ba5341c8-8d9f-42e4-98a5-162d07e4f72d state: Tombstoned -> Destroyed
1082025-12-24T02:44:45.091ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:44541 req_id = b06490bd-6cb5-4286-a8c8-898f054da4f1 response_code = 200 uri = /crucible/0/regions/ba5341c8-8d9f-42e4-98a5-162d07e4f72d