Max level shown:
1 [ Mar 25 22:49:25 Enabled. ]
2 [ Mar 25 22:49:25 Rereading configuration. ]
3 [ Mar 25 22:49:28 Rereading configuration. ]
4 [ Mar 25 22:49:33 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible -l [fd00:1122:3344:101::13]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-03-25T22:49:33.724ZINFOcrucible-agent: dataset: "oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible"
72025-03-25T22:49:33.728ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::13]:32345
82025-03-25T22:49:33.728ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-03-25T22:49:33.748ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-03-25T22:49:33.838ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::13]:32345
112025-03-25T22:49:52.405ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:33774
122025-03-25T22:49:52.425ZINFOcrucible-agent (datafile): region 21a1331d-4287-4eaa-b22b-046a3b309328 state: Requested
132025-03-25T22:49:52.428ZINFOcrucible-agent (dropshot): request completed latency_us = 7784 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33774 req_id = 5c144f6a-e204-4722-8b92-7e4546b55467 response_code = 200 uri = /crucible/0/regions
142025-03-25T22:49:52.429ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-03-25T22:49:52.439ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328
162025-03-25T22:49:52.439ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328
172025-03-25T22:49:52.547ZINFOcrucible-agent (worker): creating region Region { id: RegionId("21a1331d-4287-4eaa-b22b-046a3b309328"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/21a1331d-4287-4eaa-b22b-046a3b309328" region = 21a1331d-4287-4eaa-b22b-046a3b309328
182025-03-25T22:49:52.549ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "21a1331d-4287-4eaa-b22b-046a3b309328" "--data" "/data/regions/21a1331d-4287-4eaa-b22b-046a3b309328" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 21a1331d-4287-4eaa-b22b-046a3b309328
192025-03-25T22:49:52.735ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33774 req_id = a8d883cb-6221-4057-a656-99ed858c6f83 response_code = 200 uri = /crucible/0/regions
202025-03-25T22:49:52.736ZINFOcrucible-agent (worker): region files created ok region = 21a1331d-4287-4eaa-b22b-046a3b309328
212025-03-25T22:49:52.736ZINFOcrucible-agent (datafile): region 21a1331d-4287-4eaa-b22b-046a3b309328 state: Requested -> Created
222025-03-25T22:49:52.736ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-03-25T22:49:52.737ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-21a1331d-4287-4eaa-b22b-046a3b309328
242025-03-25T22:49:52.744ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-21a1331d-4287-4eaa-b22b-046a3b309328
252025-03-25T22:49:52.760ZINFOcrucible-agent (worker): creating config property group
262025-03-25T22:49:52.765ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-21a1331d-4287-4eaa-b22b-046a3b309328
272025-03-25T22:49:52.766ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/21a1331d-4287-4eaa-b22b-046a3b309328
282025-03-25T22:49:52.766ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-03-25T22:49:52.766ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
302025-03-25T22:49:52.766ZINFOcrucible-agent (worker): commit
312025-03-25T22:49:52.776ZINFOcrucible-agent (worker): ok!
322025-03-25T22:49:52.781ZINFOcrucible-agent (worker): SMF ok!
332025-03-25T22:49:53.414ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:33774 req_id = aec4f82a-bc9a-407f-b20f-e4878de429dc response_code = 200 uri = /crucible/0/regions
342025-03-25T22:59:55.130ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:45481
352025-03-25T22:59:55.133ZINFOcrucible-agent (dropshot): request completed latency_us = 3000 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:45481 req_id = 9c1bd491-bdd1-40f7-b60d-48ae3903d89b response_code = 200 uri = /crucible/0/regions/21a1331d-4287-4eaa-b22b-046a3b309328
362025-03-25T22:59:55.139ZINFOcrucible-agent (datafile): path is "/data/regions/21a1331d-4287-4eaa-b22b-046a3b309328"
372025-03-25T22:59:55.152ZINFOcrucible-agent (datafile): dataset is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328
382025-03-25T22:59:55.161ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328@2622e9be-a829-465b-8f08-acde5d239349
392025-03-25T22:59:55.161ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328", "2622e9be-a829-465b-8f08-acde5d239349"]
402025-03-25T22:59:55.172ZINFOcrucible-agent (dropshot): request completed latency_us = 35003 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:45481 req_id = 40a21fa4-c943-4900-a16a-ba062fb20ff6 response_code = 200 uri = /crucible/0/regions/21a1331d-4287-4eaa-b22b-046a3b309328/snapshots/2622e9be-a829-465b-8f08-acde5d239349
412025-03-25T22:59:55.175ZINFOcrucible-agent (datafile): path is "/data/regions/21a1331d-4287-4eaa-b22b-046a3b309328"
422025-03-25T22:59:55.183ZINFOcrucible-agent (datafile): dataset is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328
432025-03-25T22:59:55.191ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328@2622e9be-a829-465b-8f08-acde5d239349
442025-03-25T22:59:55.191ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328", "2622e9be-a829-465b-8f08-acde5d239349"]
452025-03-25T22:59:55.200ZINFOcrucible-agent (datafile): requesting running snapshot 21a1331d-4287-4eaa-b22b-046a3b309328-2622e9be-a829-465b-8f08-acde5d239349 state: Requested
462025-03-25T22:59:55.201ZINFOcrucible-agent (worker): applying SMF actions for region 21a1331d-4287-4eaa-b22b-046a3b309328 running snapshot 2622e9be-a829-465b-8f08-acde5d239349 (state Requested)...
472025-03-25T22:59:55.201ZINFOcrucible-agent (dropshot): request completed latency_us = 25639 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45481 req_id = 53f71b49-9d83-4d19-b2fe-1439c2190672 response_code = 200 uri = /crucible/0/regions/21a1331d-4287-4eaa-b22b-046a3b309328/snapshots/2622e9be-a829-465b-8f08-acde5d239349/run
482025-03-25T22:59:55.209ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-21a1331d-4287-4eaa-b22b-046a3b309328-2622e9be-a829-465b-8f08-acde5d239349
492025-03-25T22:59:55.213ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-21a1331d-4287-4eaa-b22b-046a3b309328-2622e9be-a829-465b-8f08-acde5d239349
502025-03-25T22:59:55.225ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-21a1331d-4287-4eaa-b22b-046a3b309328-2622e9be-a829-465b-8f08-acde5d239349
512025-03-25T22:59:55.226ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/21a1331d-4287-4eaa-b22b-046a3b309328/.zfs/snapshot/2622e9be-a829-465b-8f08-acde5d239349
522025-03-25T22:59:55.226ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532025-03-25T22:59:55.226ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542025-03-25T22:59:55.226ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
552025-03-25T22:59:55.226ZINFOcrucible-agent (worker): commit
562025-03-25T22:59:55.230ZINFOcrucible-agent (worker): ok!
572025-03-25T22:59:55.234ZINFOcrucible-agent (worker): SMF ok!
582025-03-25T22:59:55.234ZINFOcrucible-agent (datafile): region 21a1331d-4287-4eaa-b22b-046a3b309328 running snapshot 2622e9be-a829-465b-8f08-acde5d239349 state: Requested -> Created
592025-03-25T22:59:55.504ZINFOcrucible-agent (datafile): path is "/data/regions/21a1331d-4287-4eaa-b22b-046a3b309328"
602025-03-25T22:59:55.513ZINFOcrucible-agent (datafile): dataset is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328
612025-03-25T22:59:55.521ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328@2622e9be-a829-465b-8f08-acde5d239349
622025-03-25T22:59:55.521ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f62b90ae-a5ff-44f8-95be-c523217ebda4/crucible/regions/21a1331d-4287-4eaa-b22b-046a3b309328", "2622e9be-a829-465b-8f08-acde5d239349"]
632025-03-25T22:59:55.531ZINFOcrucible-agent (dropshot): request completed latency_us = 27420 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:45481 req_id = ec62248d-9617-4f12-8fdd-2d397e569b0c response_code = 200 uri = /crucible/0/regions/21a1331d-4287-4eaa-b22b-046a3b309328/snapshots/2622e9be-a829-465b-8f08-acde5d239349/run