Assignment 3

Using Program Analysis Tools

For this project, you will gain experience using both static and dynamic analysis tools on real world software.

You will work in groups of two or three students, and the requirements will scale with the group size.

Overview

As a group, you will apply one of the dynamic analysis tools we examined in class to an open source project and report on the results. You will then apply one of the static analysis tools we examined in class to an open source project (or a large project of your own) and report on the results. As a group, you will then contrast the nature of the results, effort required, and possible benefits from using the different tools.

Note that not all dynamic or static tools will work with all programs, so you may analyze different programs using the two approaches.

Dynamic Analysis

Possibly dynamic analysis tools for this assignment include:

Recall that dynamic analysis tools analyze a single execution at a time. Thus, it is common practice to integrate the dynamic analysis tools with the test process for a project in order to detect potential bugs over a set of predetermined executions. In order to apply your selected tool to a given open source project, you shall integrate the dynamic analysis tool into the existing test suite and automated testing infrastructure for the project and run the analysis over every execution in the test suite. If you are interested in using an unlisted dynamic analysis tool, ask to make sure that it is appropriate. Your write-up for the assignment should include the challenges you faced during this process, as well as your approaches for overcoming them. You should also report any errors indicated by the analysis. For groups of k members, the groups should also explain whether k of the errors found were real bugs or not. If fewer than k bugs were found, then all errors should be explained.

Static Analysis

Possible static analysis tools for this assignment include:

In contrast to dynamic tools, static analysis tools try to consider all possible executions of a program, so they need not be integrated with a test suite. As a group, you shall use one of the static analysis tools on a project of your choice in order to identify potential bugs. If you are interested in using an unlisted static analysis tool, ask the instructor to ensure that it is appropriate. Your write-up for the assignment should again include the challenges you faced during this process, as well as your approaches for overcoming them. You should also report any errors indicated by the analysis. For groups of k members, the groups should also explain whether 2k of the errors found were real bugs or not. If fewer than 2k bugs were found, then all errors should be explained.

Selecting Projects to Analyze

For software that you analyze should be an open source project of some sort. Any analyzed project should contain at least 5000 lines of code and must include an established test suite and automated test process. You are also free to analyze two different projects, one with the dynamic analysis tools and one with the static analysis tools. Once again, you are free to consider different projects listed on www.sourceforge.net, www.ohloh.net, www.gnu.org, or other major collections of open source software. If you have questions about the suitability of a particular project, please ask.

Once again, you should identify and consider:

  1. Identification of the open-source project.
  2. Identification of the supporting organization.
  3. Size of the code base.
  4. Build time to compile and link an executable from source code.
  5. Execution time for the test suite.

Again, include this information in your report.

Submission

As a group, you should reflect on the challenges faced, effort required, and either potential or recieved benefits of the tools you used for the projects you examined. What are the strengths and weaknesses of static analysis? What are the strengths and weaknesses of dynamic analysis? Are these reflected in your results? Why or why not? How? You should form and justify an opinion as to which was more useful for the project(s) you examined.

The assignment is due 11:59 pm on Wednesday, October 29, 2014. It is worth 1/3 of your overall assignment grade.