Assignment 3

Using Program Analysis Tools

For this project, you will gain experience using both static and dynamic analysis tools on real world software.

You will work in groups of two or three students, and the requirements will scale with the group size.

For software that you analyze should be an open source project of some sort. Any analyzed project should contain at least 5000 lines of code and must include an established test suite and automated test process. You are also free to analyze two different projects, one with the dynamic analysis tools and one with the static analysis tools. Once again, you are free to consider different projects listed on www.sourceforge.net, www.ohloh.net, www.gnu.org, or other major collections of open source software. If you have questions about the suitability of a particular project, please ask.

Once again, you should identify and consider:

  1. Identification of the open-source project.
  2. Identification of the supporting organization.
  3. Size of the code base.
  4. Build time to compile and link an executable from source code.
  5. Execution time for the test suite.

As a group, you will apply one of the dynamic analysis tools we examined in class to an open source project and report on the results. You will then apply one of the static analysis tools we examined in class to an open source project and report on the results. As a group, you will then contrast the nature of the results, effort required, and possible benefits from using the different tools.

Recall that dynamic analysis tools analyze a single execution at a time. Thus, it is common practice to integrate the dynamic analysis tools with the test process for a project in order to detect potential bugs over a set of predetermined executions. In order to apply either Valgrind or the Clang sanitizers to a given open source project, you shall integrate the dynamic analysis tool into the existing test suite and automated testing infrastructure for the project and run the analysis over every execution in the test suite. Your write-up for the project should include the challenges you faced during this process, as well as your approaches for overcoming them. You should also report any errors indicated by the analysis. For groups of k members, the groups should also explain whether k of the errors found were real bugs or not. If fewer than k bugs were found, then all errors should be explained.

In contrast, static analysis tools try to consider all possible executions of a program, so they need not be integrated with a test suite. As a group, you should run either the Clang Static Analyzer or FindBugs over a project of your choice in order to identify potential bugs. Your write-up for the project should again include the challenges you faced during this process, as well as your approaches for overcoming them. You should also report any errors indicated by the analysis. For groups of k members, the groups should also explain whether 2k of the errors found were real bugs or not. If fewer than 2k bugs were found, then all errors should be explained.

As a group, you should reflect on the challenges faced, effort required, and either potential or recieved benefits of the tools you used for the projects you examined. You should form and justify an opinion as to which was more useful for the project(s) you examined.

The assignment is due 11:59 pm on Monday, March 24, 2014. It is worth 1/3 of your overall assignment grade.