Assignment 0: High-stakes testing

Due date: Mon Jan 12 11:59 pm - Hard deadline: Mon Jan 12 11:59 pm

Assignment by Julie Zelenski

Learning goals

Completing this assignment will give you valuable practice with

  1. working in the unix environment (navigating the filesystem, executing programs, editing files, using Mercurial and Valgrind)
  2. devising strategies for testing and creating test inputs
  3. writing a high-quality bug report

Your assignment

Before we put you to work coding your first C program, this small exercise will acclimate you to the unix environment and bring out your inner software testing ninja. The assigned task is to make observations of a program's behavior relative to its specification and determine which tasks it does correctly and which it bungles. The program you'll be testing will be your implementation task as the next assignment so your testing efforts now will set you up nicely for testing your own version next week.

The compiled program you are given is a good-faith attempt at meeting the requirements described in the Assignment 1 Reassemble writeup. The program is mostly correct, but has been maliciously seeded with four known flaws, one in each of these categories:

Your job is to relentlessly torture the program into exposing its four problems and then write up a concise bug report on each.

A plan of attack

Usage and robustness



General notes

Readme and custom test files

The project directory contains two text files readme.txt and custom_tests where you are to document the results of your testing efforts. Edit readme.txt to include a concise bug report for each category. A bug report should identify the underlying pattern of the bug and document how to reproduce it with a sequence of steps and/or a minimal input. The pattern is best stated as a sentence that lists the specific condition(s) that trigger the bug and the undesirable outcome, e.g. "On any input file whose reassembled result is longer than 1000, the program goes into an infinite loop" or "Valgrind reports an invalid read error. It happens on all inputs and the count of errors is roughly equal to number of fragments." Assume the audience is the original software developer to whom you are reporting the key facts that allow the developer to reproduce the problem. Document the specific, minimal trigger that reproduces the bad behavior without additional long-winded commentary. Do not reiterate information of which the developer is already well-aware (e.g. no need to repeat program specification, make arguments for why a crash is a bad thing, and so on).

Each bug report can (and should) be documented in 1-3 sentences. The shell command wc -w readme.txt will count the total number of words in your readme file. Four solid and concise reports can total under 250 words altogether -- aim for your readme to be that tight and earn the gratitude and approbation of your grader :-)

The custom_tests file is for use with sanity check and should be edited to list your test cases (where possible). When you have created your own fragment files to reproduce your bugs, be sure to add them to your repository (using hg add filename) so they will be included with your submission. You can refer to these files in your readme.txt as the reproducible test cases. Note that custom sanity check does not support testing under Valgrind, so your memory test need not be listed in custom_tests.

The text files are managed under Mercurial revision control. After making edits, make a commit to record a snapshot for your revision history. After your final commit, use the submit script to send your work to us for grading. Please note that commit and submit are two different steps and both are essential!


The bug report for each category is worth 6 points, for an assignment total of 24 points. For full credit, we require a concise and accurate bug report in the readme.txt file and where possible/appropriate, a reproducible minimized test case added to custom_tests and submitted with the repo. If you can't nail down a particular bug, you can earn partial credit for describing your efforts and what you learned thus far. For example, providing one example input to trigger the bug but without identifying the broader underlying pattern is worth about half credit.

Getting started

The assign0 project contains a compiled reassemble program and the two text files readme.txt and custom_tests. The project is distributed as a Mercurial repository. Clone the starter project from your cs107 repository using the command

hg clone /afs/ir/class/cs107/repos/assign0/$USER assign0

The $USER shell variable automatically expands to your sunet id. If you find there is not a repo for your sunet, this means you were not registered for the course when the repos were distributed, please register asap and send email to cs107@cs asking us to create your repo.

The directory /afs/ir/class/cs107/samples/assign0 contains a correct (non-buggy!) reassemble solution and some sample fragment files. You can access these via their full path (e.g. /afs/ir/class/cs107/samples/assign0/reassemble_soln), but it is a bit unwieldy. Your repo has slink, which is a symbolic link to this shared samples directory. You can use it to more easily refer to those files (e.g. slink/reassemble_soln) and avoid retyping the full path. However, since it is a link it has some peculiar qualities--- you do not have permissions to edit within the samples directory and that if you cd into slink, the parent (..) directory will not refer to your repo, but to the true parent instead.

Once you have cloned, immediately make and commit a simple change to verify that your environment is properly configured:

  1. Edit the readme.txt file to add your name, save changes
  2. Commit that change

Now you're good to start bug hunting! If your commit is not successful, resolve the issue before going any further: Mercurial guide, forum/email, office hours, ...


Your final step is to submit your work for grading. Remember that submit is distinct from commit. You commit as you are working to take snapshots of your progress and submit when finished to send your work to the staff to be graded. We recommend you do a trial submit well in advance of the deadline to familiarize yourself with the process and allow time to work through any snags. You can replace that submission with a subsequent one if desired.

You may not use any late days on this assignment. The deadline is absolutely firm and no late submissions will be accepted. Don't miss this chance to snap up some quick points and start your quarter off right!

Frequently asked questions about assign0

When I try to run the reassemble program in my directory, it says "command not found". What can I do?

This classic question has plagued every newcomer to Unix since the beginning of the epoch. The solution is in our frequently asked questions about unix.

Am I responsible for identifying the root flaw in the code is or suggesting how the developer might fix it?

No. As a black-box outsider, you aren't privy to the necessary information to make this kind of judgment. When looking for the bug, it may be helpful to speculate about possible bugs and how they would surface (e.g. an off-by-one loop bound) but the bug report you submit needs only to identify the observed pattern and provide the reproducible test input.