For project 1, you will demonstrate your mastery of shell scripting by developing a framework for automatically testing a piece of software. You will then use this framework to analyze faults in a real-world piece of code and identify patterns in the output.
Testing is a critical aspect of the software development process. Often, this involves running a program against a set of inputs and comparing the output with known results. Because there are often many tests, automation of this running and testing can significantly reduce the time needed to identify problems in the code.
For this project, you are being asked to work with a program you have not written. This means that some subset of your time will be spent figuring out how to use the program. Further, this assignment is intentially underspecified; you will need to infer some of the requirements for achieving full marks through reasoning about expectations and feedback from the autograder.
Write a bash script named run_tests.sh that:
tests/inputs
tests/outputs, and
TEST {#}: PASS when the comparision matches
TEST {#}: FAIL when the comparision does not match
{#} is the number of the test (without curly braces)
stdout
You may reasonably assume that:
tests directory is located in the same directory as your
script
You may not assume that:
You are provided with some sample program and data to use for testing your script.
There are two directories in this archive, bin and tests.
The bin directory contains a single, compiled program that is compiled for
Ubuntu 20.04 (x86-64). This is the operating system running on your
cs-linuxlab server.
The program, days_to_years, takes a single file as input.
This file contains a single integer, representing a number of days from
1980 and outputs the year in which that day falls. For example, running the following
will produce the given output:
bin/days_to_years
Usage: bin/days_to_years dayfile
dayfile is a text file containing a single number
echo 0 > days_0.txt
bin/days_to_years days_0.txt
1980
echo 375 > days_375.txt
bin/days_to_years days_375.txt
1981
This code is taken from a commercial music player produced from 2006–2011. At midnight on December 31, 2008, many of these music players froze as a result of this code.
The tests directory contains a number of input files along with the
associated correct output for each input. These tests follow the directory
structure specified above.
You must also write a short two-paragraph report reflecting on your experience creating a shell script to test an unknown piece of software.
The first paragraph should address your design decisions and implemenation strategies for this assignment. You might consider addressing how your script handles all of the requirements (or if you fail to meet some of the requirements, why that might be). If there were particular aspects of this script that took you a while to perfect, these might be good things to discuss.
The second paragraph should analyze the provided sample program. Discuss which inputs fail. Do you notice any commonalities between the inputs that fail? Are there certain failure modes that you observe? If you have a hypothesis about what is going wrong, share your thoughts here.
Submit this report either as a ASCII text file or a PDF. Any other format may cause you to lose points on this assignment.
Nearly all of this assignment can be completed by composing tools we have discussed and used together in class. However, you might find that certain aspects of this project require you to research additional tools or settings.
The instructor solution for this project is about 20 lines of code. You may be able to complete the assignment with fewer (or you may need significantly more). Instead of targeting a specific line count, instead consider the quality of yoru script. How well can you reuse what you write for other projects? How much of it makes sense?
Further, while the solution is not particularly long, each line may require deep thought. Do not wait to begin this assignment.
It is NOT in your best interest to wait until the last minute to submit your project to Gradescope. The autograded portion of the assignment will give you (minimal) feedback each time you submit.
By submitting early an often, you receive feedback that can help guide the next steps of developing your script. Try to write a script that will always successfully run. You can incrementally refine this to create the valid output. For example, you might start by iterating over all of the tests, printing bogus output. Does Gradescope say your output has an incorrect number of lines? Next, you might try to make the output be formatted correctly, or you might try to actually use the test input without crashing.
In general, incremental programming is a good habit to develop. This will help you both in your classes, but also in future jobs. Students who submit incrementally on assignments tend to receive higher grades because they give themselves time to fix errors that are exposed by the autograder and also receive more feedback on each assignment.
You must submit the following files to Gradescope:
readme.txt or readme.pdf: your README file
references.txt: your file of citations
run_tests.sh: your shell script
You can submit these by dragging the files into the submission window on the Gradescope interface.
P1 Grading (out of 40 points):