What to test
In an ideal world, any software developed would be accompanied by 100% test coverage validating all aspects of functionality and interaction with other software. However, due to pressures of research, having enough time to build a perfect test suite isn't always realistic. A parsimonious application of the Pareto principle will go a long way towards improving overall software quality without adding too much testing burden. To apply this principle, spend some time in a thought experiment to determine answers to questions such as:
Once answers to these questions are known, spend time developing tests to validate key features, avoiding major negatives, and ensuring software performs adequately. 
How to test
As mentioned previously, a combination of unit level and acceptance style testing - where tests are conducted to verify software meets requirements - is often best suited to the smaller scale and ad-hoc nature of research software.