Max level shown:
1 [ Jun 3 00:50:58 Enabled. ]
2 [ Jun 3 00:50:58 Rereading configuration. ]
3 [ Jun 3 00:51:00 Rereading configuration. ]
4 [ Jun 3 00:51:04 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible -l [fd00:1122:3344:101::16]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-06-03T00:51:04.667ZINFOcrucible-agent: dataset: "oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible"
72025-06-03T00:51:04.670ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::16]:32345
82025-06-03T00:51:04.670ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-06-03T00:51:04.688ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-06-03T00:51:04.783ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::16]:32345
112025-06-03T00:51:44.817ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::c]:49286
122025-06-03T00:51:44.838ZINFOcrucible-agent (datafile): region 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc state: Requested
132025-06-03T00:51:44.843ZINFOcrucible-agent (dropshot): request completed latency_us = 18184 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 3c220294-fbb8-4567-95a3-33b3de763e9f response_code = 200 uri = /crucible/0/regions
142025-06-03T00:51:44.845ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-06-03T00:51:44.856ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
162025-06-03T00:51:44.856ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
172025-06-03T00:51:44.913ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6478eb12-2e9d-486c-bd3d-6ef8281ae1fc"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc" region = 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
182025-06-03T00:51:44.913ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6478eb12-2e9d-486c-bd3d-6ef8281ae1fc" "--data" "/data/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
192025-06-03T00:51:45.034ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49286 req_id = f4979f99-d43e-4577-a116-1de695930bbd response_code = 200 uri = /crucible/0/regions
202025-06-03T00:51:45.148ZINFOcrucible-agent (worker): region files created ok region = 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
212025-06-03T00:51:45.148ZINFOcrucible-agent (datafile): region 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc state: Requested -> Created
222025-06-03T00:51:45.149ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-06-03T00:51:45.149ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
242025-06-03T00:51:45.155ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
252025-06-03T00:51:45.167ZINFOcrucible-agent (worker): creating config property group
262025-06-03T00:51:45.170ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
272025-06-03T00:51:45.171ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
282025-06-03T00:51:45.171ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-06-03T00:51:45.171ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
302025-06-03T00:51:45.171ZINFOcrucible-agent (worker): commit
312025-06-03T00:51:45.176ZINFOcrucible-agent (worker): ok!
322025-06-03T00:51:45.180ZINFOcrucible-agent (worker): SMF ok!
332025-06-03T00:51:45.337ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 7e77ccaf-c6d1-4657-aa0d-bd4e50de69d7 response_code = 200 uri = /crucible/0/regions
342025-06-03T00:51:45.812ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 3fff1bb7-b9a3-46f6-8691-54ac91f9cfc6 response_code = 200 uri = /crucible/0/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
352025-06-03T00:51:45.815ZINFOcrucible-agent (datafile): path is "/data/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc"
362025-06-03T00:51:45.826ZINFOcrucible-agent (datafile): dataset is oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
372025-06-03T00:51:45.837ZINFOcrucible-agent (datafile): region 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc state: Created -> Tombstoned
382025-06-03T00:51:45.837ZINFOcrucible-agent (worker): applying SMF actions before removal...
392025-06-03T00:51:45.837ZINFOcrucible-agent (dropshot): request completed latency_us = 22442 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 9adfeb29-b716-44e9-8dd4-a05347f797f3 response_code = 204 uri = /crucible/0/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
402025-06-03T00:51:45.838ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6478eb12-2e9d-486c-bd3d-6ef8281ae1fc (instance states: (Some(Online), None))
412025-06-03T00:51:45.840ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 150d7d89-f453-4db9-bfaa-845ea38ce400 response_code = 200 uri = /crucible/0/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
422025-06-03T00:51:45.842ZINFOcrucible-agent (worker): SMF ok!
432025-06-03T00:51:45.854ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc" region = 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
442025-06-03T00:51:45.899ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49286 req_id = 9f7485b1-c072-4e5d-a029-61e589c959dc response_code = 200 uri = /crucible/0/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
452025-06-03T00:51:45.918ZINFOcrucible-agent (datafile): region 6478eb12-2e9d-486c-bd3d-6ef8281ae1fc state: Tombstoned -> Destroyed
462025-06-03T00:51:45.973ZINFOcrucible-agent (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49286 req_id = d34344ac-3c92-4d59-982f-2c382bd1b39e response_code = 200 uri = /crucible/0/regions/6478eb12-2e9d-486c-bd3d-6ef8281ae1fc
472025-06-03T00:51:48.150ZINFOcrucible-agent (datafile): region 4202bcf6-7046-4caa-a228-3d5606cd020a state: Requested
482025-06-03T00:51:48.150ZINFOcrucible-agent (dropshot): request completed latency_us = 223 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49286 req_id = cd68660a-41ef-44fb-92e1-348eb2a6fcc5 response_code = 200 uri = /crucible/0/regions
492025-06-03T00:51:48.150ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502025-06-03T00:51:48.159ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/4202bcf6-7046-4caa-a228-3d5606cd020a
512025-06-03T00:51:48.159ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6ac47f75-a094-456e-b5e2-e83658a04ab2/crucible/regions/4202bcf6-7046-4caa-a228-3d5606cd020a
522025-06-03T00:51:48.208ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4202bcf6-7046-4caa-a228-3d5606cd020a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4202bcf6-7046-4caa-a228-3d5606cd020a" region = 4202bcf6-7046-4caa-a228-3d5606cd020a
532025-06-03T00:51:48.211ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4202bcf6-7046-4caa-a228-3d5606cd020a" "--data" "/data/regions/4202bcf6-7046-4caa-a228-3d5606cd020a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4202bcf6-7046-4caa-a228-3d5606cd020a
542025-06-03T00:51:48.304ZINFOcrucible-agent (worker): region files created ok region = 4202bcf6-7046-4caa-a228-3d5606cd020a
552025-06-03T00:51:48.304ZINFOcrucible-agent (datafile): region 4202bcf6-7046-4caa-a228-3d5606cd020a state: Requested -> Created
562025-06-03T00:51:48.304ZINFOcrucible-agent (worker): applying SMF actions post create...
572025-06-03T00:51:48.305ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4202bcf6-7046-4caa-a228-3d5606cd020a
582025-06-03T00:51:48.308ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4202bcf6-7046-4caa-a228-3d5606cd020a
592025-06-03T00:51:48.317ZINFOcrucible-agent (worker): creating config property group
602025-06-03T00:51:48.323ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4202bcf6-7046-4caa-a228-3d5606cd020a
612025-06-03T00:51:48.323ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4202bcf6-7046-4caa-a228-3d5606cd020a
622025-06-03T00:51:48.323ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632025-06-03T00:51:48.323ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
642025-06-03T00:51:48.323ZINFOcrucible-agent (worker): commit
652025-06-03T00:51:48.327ZINFOcrucible-agent (worker): ok!
662025-06-03T00:51:48.331ZINFOcrucible-agent (worker): SMF ok!
672025-06-03T00:51:48.355ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49286 req_id = dcf72136-94f0-4298-8eb2-f6a0412a2144 response_code = 200 uri = /crucible/0/regions