Max level shown:
1 [ Dec 24 01:44:59 Enabled. ]
2 [ Dec 24 01:44:59 Rereading configuration. ]
3 [ Dec 24 01:45:05 Rereading configuration. ]
4 [ Dec 24 01:45:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-12-24T01:45:14.117ZINFOcrucible-agent: Crucible Version: Crucible Version: 0.0.1 Commit SHA: 7103cd3a3d7b0112d2949dd135db06fef0c156bb Commit timestamp: 2025-11-10T22:24:53.000000000Z branch: main rustc: 1.90.0 stable x86_64-unknown-illumos Cargo: x86_64-unknown-illumos Debug: false Opt level: 3
72025-12-24T01:45:14.117ZINFOcrucible-agent: dataset: "oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible"
82025-12-24T01:45:14.117ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
92025-12-24T01:45:14.117ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
102025-12-24T01:45:14.129ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
112025-12-24T01:45:14.200ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
122025-12-24T01:45:39.891ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:34711
132025-12-24T01:45:39.925ZINFOcrucible-agent (datafile): region add5107a-1c64-48be-81c0-d952fb3c9571 state: Requested
142025-12-24T01:45:39.927ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-12-24T01:45:39.927ZINFOcrucible-agent (dropshot): request completed latency_us = 13925 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:34711 req_id = 9c910b61-e16e-4b27-bfce-5ae5954b3ba4 response_code = 200 uri = /crucible/0/regions
162025-12-24T01:45:39.936ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571
172025-12-24T01:45:39.936ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571
182025-12-24T01:45:39.982ZINFOcrucible-agent (worker): creating region Region { id: RegionId("add5107a-1c64-48be-81c0-d952fb3c9571"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/add5107a-1c64-48be-81c0-d952fb3c9571" region = add5107a-1c64-48be-81c0-d952fb3c9571
192025-12-24T01:45:39.985ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "add5107a-1c64-48be-81c0-d952fb3c9571" "--data" "/data/regions/add5107a-1c64-48be-81c0-d952fb3c9571" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = add5107a-1c64-48be-81c0-d952fb3c9571
202025-12-24T01:45:40.085ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:34711 req_id = f3b69cf4-a21b-481d-9643-9eddaa2c9c16 response_code = 200 uri = /crucible/0/regions
212025-12-24T01:45:40.345ZINFOcrucible-agent (worker): region files created ok region = add5107a-1c64-48be-81c0-d952fb3c9571
222025-12-24T01:45:40.345ZINFOcrucible-agent (datafile): region add5107a-1c64-48be-81c0-d952fb3c9571 state: Requested -> Created
232025-12-24T01:45:40.345ZINFOcrucible-agent (worker): applying SMF actions post create...
242025-12-24T01:45:40.346ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-add5107a-1c64-48be-81c0-d952fb3c9571
252025-12-24T01:45:40.352ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-add5107a-1c64-48be-81c0-d952fb3c9571
262025-12-24T01:45:40.363ZINFOcrucible-agent (worker): creating config property group
272025-12-24T01:45:40.366ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-add5107a-1c64-48be-81c0-d952fb3c9571
282025-12-24T01:45:40.368ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/add5107a-1c64-48be-81c0-d952fb3c9571
292025-12-24T01:45:40.368ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
302025-12-24T01:45:40.368ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
312025-12-24T01:45:40.368ZINFOcrucible-agent (worker): commit
322025-12-24T01:45:40.372ZINFOcrucible-agent (worker): ok!
332025-12-24T01:45:40.377ZINFOcrucible-agent (worker): SMF ok!
342025-12-24T01:45:40.837ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:34711 req_id = e0004024-0d3d-43c7-a1c9-b21a1c93a04f response_code = 200 uri = /crucible/0/regions
352025-12-24T01:56:51.132ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:40259
362025-12-24T01:56:51.133ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40259 req_id = 36426afd-d14a-4aa2-8615-32b86baf3dd9 response_code = 200 uri = /crucible/0/regions/add5107a-1c64-48be-81c0-d952fb3c9571
372025-12-24T01:56:51.140ZINFOcrucible-agent (datafile): path is "/data/regions/add5107a-1c64-48be-81c0-d952fb3c9571"
382025-12-24T01:56:51.151ZINFOcrucible-agent (datafile): dataset is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571
392025-12-24T01:56:51.160ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571@a083a91b-ac2a-44fe-98b8-278d2c0684b3
402025-12-24T01:56:51.161ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571", "a083a91b-ac2a-44fe-98b8-278d2c0684b3"]
412025-12-24T01:56:51.174ZINFOcrucible-agent (dropshot): request completed latency_us = 33155 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40259 req_id = 96e3aa6f-b52b-4771-8289-a3760dbd9390 response_code = 200 uri = /crucible/0/regions/add5107a-1c64-48be-81c0-d952fb3c9571/snapshots/a083a91b-ac2a-44fe-98b8-278d2c0684b3
422025-12-24T01:56:51.185ZINFOcrucible-agent (datafile): path is "/data/regions/add5107a-1c64-48be-81c0-d952fb3c9571"
432025-12-24T01:56:51.194ZINFOcrucible-agent (datafile): dataset is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571
442025-12-24T01:56:51.202ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571@a083a91b-ac2a-44fe-98b8-278d2c0684b3
452025-12-24T01:56:51.202ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571", "a083a91b-ac2a-44fe-98b8-278d2c0684b3"]
462025-12-24T01:56:51.212ZINFOcrucible-agent (datafile): requesting running snapshot add5107a-1c64-48be-81c0-d952fb3c9571-a083a91b-ac2a-44fe-98b8-278d2c0684b3 state: Requested
472025-12-24T01:56:51.213ZINFOcrucible-agent (dropshot): request completed latency_us = 28612 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40259 req_id = 585d2da0-cfa1-4bbb-b094-185a6b1bf789 response_code = 200 uri = /crucible/0/regions/add5107a-1c64-48be-81c0-d952fb3c9571/snapshots/a083a91b-ac2a-44fe-98b8-278d2c0684b3/run
482025-12-24T01:56:51.213ZINFOcrucible-agent (worker): applying SMF actions for region add5107a-1c64-48be-81c0-d952fb3c9571 running snapshot a083a91b-ac2a-44fe-98b8-278d2c0684b3 (state Requested)...
492025-12-24T01:56:51.234ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-add5107a-1c64-48be-81c0-d952fb3c9571-a083a91b-ac2a-44fe-98b8-278d2c0684b3
502025-12-24T01:56:51.246ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-add5107a-1c64-48be-81c0-d952fb3c9571-a083a91b-ac2a-44fe-98b8-278d2c0684b3
512025-12-24T01:56:51.265ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-add5107a-1c64-48be-81c0-d952fb3c9571-a083a91b-ac2a-44fe-98b8-278d2c0684b3
522025-12-24T01:56:51.265ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/add5107a-1c64-48be-81c0-d952fb3c9571/.zfs/snapshot/a083a91b-ac2a-44fe-98b8-278d2c0684b3
532025-12-24T01:56:51.265ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542025-12-24T01:56:51.266ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552025-12-24T01:56:51.266ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
562025-12-24T01:56:51.266ZINFOcrucible-agent (worker): commit
572025-12-24T01:56:51.269ZINFOcrucible-agent (worker): ok!
582025-12-24T01:56:51.274ZINFOcrucible-agent (worker): SMF ok!
592025-12-24T01:56:51.274ZINFOcrucible-agent (datafile): region add5107a-1c64-48be-81c0-d952fb3c9571 running snapshot a083a91b-ac2a-44fe-98b8-278d2c0684b3 state: Requested -> Created
602025-12-24T01:56:51.543ZINFOcrucible-agent (datafile): path is "/data/regions/add5107a-1c64-48be-81c0-d952fb3c9571"
612025-12-24T01:56:51.553ZINFOcrucible-agent (datafile): dataset is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571
622025-12-24T01:56:51.562ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571@a083a91b-ac2a-44fe-98b8-278d2c0684b3
632025-12-24T01:56:51.562ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_bec007fc-64d7-4e2b-b4b6-a99862c23ee6/crucible/regions/add5107a-1c64-48be-81c0-d952fb3c9571", "a083a91b-ac2a-44fe-98b8-278d2c0684b3"]
642025-12-24T01:56:51.569ZINFOcrucible-agent (dropshot): request completed latency_us = 26066 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40259 req_id = c88d7f3d-c9f8-4354-b8ee-7e69302040bd response_code = 200 uri = /crucible/0/regions/add5107a-1c64-48be-81c0-d952fb3c9571/snapshots/a083a91b-ac2a-44fe-98b8-278d2c0684b3/run