# Writing a CodeCheck Problem

## Crash Course

Why use CodeCheck and not a fancier system? There are two advantages:

1. It is very easy to turn a sample program into an assignment. Read on to see how.
2. You don't need an account. Your students don't need an account. You don't need to ask anyone for permission. You don't need to install anything. Just upload your annotated program, and give the resulting URL to your students.

Suppose you have a sample program for your students, such as the classic “rainfall” problem: Given a sequence of numbers, find the average, but ignore negative values (which must have been measurement errors), and stop when you reach the sentinel 9999.

```public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
}
else
{
return sum / count;
}
}

public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}```

Wouldn't your students learn much more if they wrote the program instead of looking at your solution? Sure they would. With CodeCheck, you can produce a simple assignment quickly.

What do you want to focus on with this exercise? Presumably loops and corner cases, not the syntax of `public static void main`. If you want your students to fill in the entire method, do this:

```public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
//HIDE
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
}
else
{
return sum / count;
}
//EDIT . . .
}

public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}```

Visit this link and paste in the file. You will get an URL such as this one: http://codecheck.it/files/20032817325la004mxsdozekzv7jpe6nmsd. Give it to your students. Right now, check out that link so that you see what the students see—in particular, how the `//HIDE` and `//EDIT` leave a hole.

Maybe you want your students to focus on the logic, not the statement syntax? Not a bad idea—that way you can give them more problems to work on in their limited time. Then you need to be craftier about the holes that you punch out. Look at this student view and compare it with the markup:

```public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
//HIDE
while (rainfall[i] != 9999)
//EDIT while (. . .)
{
//HIDE
if (rainfall[i] >= 0)
//EDIT if (. . .)
{
//HIDE
sum += rainfall[i];
count++;
//EDIT . . .
}
i++;
}
//HIDE
if (count == 0)
//EDIT if (. . .)
{
return 0;
}
else
{
return sum / count;
}
}

public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}```

The reporting is pretty basic with this approach. You can do better by using `//CALL` pseudocomments for test cases:

```public class Rainfall
{
//CALL new double[] { 1, 2, 3, 4, 5, 9999 }
//CALL new double[] { 1, 2, 3, 4, 5, 9999, 6, 7 }
//CALL new double[] { 10, 9999 }
//CALL new double[] { 10, 0, 9999 }
//CALL new double[] { -1, -2, -3, 9999 }
//CALL new double[] { 15, 0, -53, 5, 2, 9999 }
//CALL new double[] { 9999 }
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
//HIDE
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
return count == 0 ? 0 : sum / count;
//EDIT . . .
}
}```

Try it out here. Paste in

```      return 0;
```
to see what happens with a wrong solution, or
```      while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
return sum / count;
```
for an almost correct solution.

That's the crash course. Try CodeCheck with your own sample programs, an read the following for all the details.

## Authoring Overview

CodeCheck is a “convention over configuration” unit test library for evaluating student programs. The emphasis is on minimizing the authoring work of an instructor.

CodeCheck supports Java, C++, Scala, Python, JavaScript, Scheme, Standard ML, and Haskell. Ask if you want support for another language.

The author of a problem provides

• A solution with markers to denote the parts that the student should complete
• An optional HTML problem description in a file `index.html`
• Optionally, files that are used for testing the student work (input files, unit tests, library files)

To upload a CodeCheck problem, visit this link. You can either paste the files into the form or upload a zip file.

When the problem is uploaded, you can preview it and see a report where CodeCheck compiles and runs the solution. If you spot any errors, you can edit the problem.

Once everything works, give the public URL to your students.

Hang on to the edit URL in case you need to edit your problem.

## Marking Up the Solution

Write the solution and test it locally. Then hide a part of the solution, by using `//HIDE` and `//EDIT` pseudocomments:

```public class Numbers
{
public static int square(int n)
{
//HIDE
return n * n;
//EDIT return . . .;
}
}```

The code between `//HIDE` and `//EDIT` is hidden. Everything else is read-only. The contents after the `//EDIT` is placed into an editable text area.

An `//EDIT` area can span multiple line, and you can have multiple `//EDIT` areas:

```public class Numbers
{
public static int square(int n)
{
//HIDE
return n * n;
//EDIT
}

public static int cube(int n)
{
//HIDE
return n * n * n;
//EDIT int result = . . .;
//EDIT return . . .;
}
}```

If you want to hide the entire solution and have the student only see a blank text area, put `//HIDE` on top and `//EDIT` on the bottom.

## Non-Editable Files

Any file that doesn't contain `//EDIT` is displayed to the student but can't be edited. This is for helper files that the student needs to read.

Any student file that starts with `//HIDE` is completely hidden from the student. Use this for utility or library files that are necessary for compiling/running but that the student should not see

## What You Can Test

You can make four kinds of checks:

1. Run the program. You supply the inputs. Both the student submission and the solution are compiled and run with the given inputs, and the outputs are compared.

Here is an example in Java. See below for C++ and Python examples.

Numbers.java
```//IN 3\n-3\n0\n
import java.util.Scanner;
public class Numbers
{
public static void main(String[] args)
{
//HIDE
Scanner in = new Scanner(System.in);
boolean done = false;
while (!done)
{
//EDIT . . .
System.out.println("Enter a number, 0 to quit");
//HIDE
int n = in.nextInt();
if (n == 0) done = true;
else
{
int square =  n * n;
//EDIT . . .
System.out.println("The square is " + square);
//HIDE
}
}
//EDIT . . .
}
}```
2. There can be more than one `IN` pseudocomment. Each of them causes a program run where the contents is fed to standard input (`System.in` in Java or `cin` in C++). The contents is a single line with Java escape sequences: `\n` is a newline, `\\` a backslash, and so on. Unicode escapes `\uxxxx` are supported.

If the inputs are long, don't use the `IN` pseudocomments, but instead supply input files named `test.in`, `test2.in,``test3.in`, and so on.

test.in
```3
-3
0```

Tip: It is a good idea to give the students the statements for prompts and outputs, as in the preceding example.

3. Unit tests.

CodeCheck supports a simple style of doing unit testing that works with any language, and that doesn't require any library. (With Java, JUnit is also supported.)

The tester file name ends in `Tester`, `Tester2`, `Tester3`, and so on. Each tester writes output consisting of pairs of lines, each of the form

```Optional label, colon, and space, followed by value
Expected: value```

For example:

```Cube volume: 27
Expected: 27
Surface area: 9
Expected: 54```

CodeCheck checks that the values of each pair match.

Example:

NumbersTester.java
```public class NumbersTester
{
public static void main(String[] args)
{
Numbers nums = new Numbers();
System.out.println(nums.square(3));
System.out.println("Expected: 9");
System.out.println(nums.square(-3));
System.out.println("Expected: 9");
}
}```
Numbers.java
```public class Numbers
{
public int square(int n)
{
//HIDE
return n * n;
//EDIT ...
}
}```

Normally, the instructor writes the tester. But if you add a `Tester` file that has an `EDIT` area, then that means that you expect the student to write it. In that case, you need to decide whether the student needs to supply the exact same tests as in your solution (in which case you need to give the student specific instructions to replicate the output) or whether arbitrary tests are ok (in which case you mark your solution with the `SAMPLE` pseudocomment).

You can also supply JUnit test cases in the student directory that end in `Test`, `Test1`, `Test2`, and so on. Those are executed with JUnit in the usual way.

4. Test a single method. Supply pseudocomments `//CALL` in the lines immediately above the method.

Example:

Numbers.java
```public class Numbers
{
//CALL 3, 4
//CALL -3, 3
//CALL 3, 0
public double average(int x, int y)
{
//HIDE
return 0.5 * (x + y);
//EDIT // Compute the average of x and y
}
}```

The pseudocomment `//CALL` must be present in exactly one solution file. It cannot be in student file.

5. Substitute variables. This is useful very early in a CS1 course when input and methods haven't yet been introduced. The file is rewritten with different settings for the variables.

Example:

Numbers.java
```public class Numbers
{
public static void main(String[] args)
{
int x = 3; //SUB 5; 8
int y = 4; //SUB 6; 4
//HIDE
double average = 0.5 * (x + y);
//EDIT // Compute the average of x and y
//EDIT // and assign it to a variable average
System.out.println(average);
}
}```

The student program is run three times, with `x` set to 3, 5, 8, and `y` set to 4, 6, 4.

The pseudocomment `//SUB` must be present in exactly one solution file. It cannot be in a student file. Values are separated by semicolons so that you can supply expressions with commas.

## Command-Line Arguments

When running a program, you can optionally supply command-line arguments with an `//ARGS` pseudocomment. You can supply mutliple `//ARGS` lines. The student program and solution are run for each of them, and their outputs are compared.

Example:

values.txt
```3
-3```
values2.txt
`0`
Numbers.java
```//ARGS values.txt
//ARGS values2.txt

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

public class Numbers
{
public static void main(String[] args) throws FileNotFoundException
{
Scanner in = new Scanner(new File(args));
//HIDE
while (in.hasNextInt())
{
int n = in.nextInt();
System.out.println(n * n);
}
// EDIT ...
}
}
```

## File Output

If you expect the student to write output to a file (i.e. not `System.out`), or to multiple files, provide their names with an `//OUT` pseudocomment. Names are separated by white space.

Then the student and solution programs are run to produce these files, and the results are compared. If an output file is an image, image comparison is automatically used.

Example:

values.txt
```3
-4
0```
Numbers.java
```//OUT evens.txt odd.txt
import java.io.*;
import java.util.*;

public class Numbers
{
public static void main(String[] args) throws IOException
{
Scanner in = new Scanner(new File("values.txt"));
PrintWriter odds = new PrintWriter("odds.txt");
PrintWriter evens = new PrintWriter("evens.txt");
//HIDE
while (in.hasNextInt())
{
int n = in.nextInt();
if (n % 2 == 0) evens.println(n); else odds.println(n);
}
//EDIT ...
}
}```

## Checkstyle

With Java problems, you can optionally run Checkstyle on the submitted files.

Put a file `checkstyle.xml` into the student directory, with whatever checkstyle checks you want. Here is one that just checks for javadoc and variable naming conventions.

```<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
<module name="Checker">
<module name="TreeWalker">
<property name="tabWidth" value="4"/>

<property name="scope" value="public"/>
<property name="allowMissingThrowsTags" value="true"/>
</module>

<!-- Checks for Naming Conventions.                  -->
<!-- See http://checkstyle.sf.net/config_naming.html -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName">
<property name="format" value="^[A-Z](_?[A-Z0-9]+)*\$"/>
</module>
<module name="LocalVariableName"/>
<module name="MemberName"/>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
</module>
</module>
```

## How Comparisons Work

The comparison rules take care of the biggest annoyances that students face: white space, roundoff, and letter case.

When the actual and expected outputs are compared, corresponding lines are compared.

By default, white space within a line is not significant. For example,

`Hello, World!`

and

`   Hello,  World!`

match. If you want to take white space into account. use the pseudocomment `//IGNORESPACE false`.

If corresponding tokens within a line are numbers, they compare identical if they are within the given tolerance (1E-6 by default). For example,

`Area: 1.9999999999996`

and

`Area: 2.0`

match, as do

`Area: 2`

and

`Area: 2.0`

If you want to use a different tolerance, supply a pseudocomment such as `//TOLERANCE 0.01`.

If tokens are not numbers,  the comparison is case-insensitive by default. For example,

`Hello, World!`

and

`HELLO, WORLD!`

match. Use the pseudocomment `//IGNORECASE false` if you want comparison to be case-sensitive.

## Checking for Required and Forbidden Strings

Suppose you want to insist that an assignment is done with a `for` loop. You can test for it with

`//REQUIRED for`

or, better, something like

`//REQUIRED for\s*\([^;]*;[^;]*;.*\)`

Conversely, you may want to forbid something, for example disallow calling the `contains` method. You can test that with

`//FORBIDDEN \.contains\s*\(`

If these tests fail, the student gets zero points and the program is not compiled.

## JAR Files

Just toss them into the `student` directory, and they will be added to the class path when the code is compiled and run. Thanks to Dustin Hoffman for this enhancement.

## Graphics Programs

If you want to check simple graphics programs in Java that draw graphics on a `JComponent`, add this `JFrame` class to your project. Make sure to `import javax.swing.*` and not `javax.swing.JFrame` so that the bogus frame class gets used. The painting is simply captured in a file. (The CodeCheck server is “headless”—there are no frames and windows.) As long as you don't use buttons, sliders, menus, timers, and so on, this works fine.

Alternatively, check out the Simple Java Graphics library that I wrote for the Udacity CS046 course.

For Python, use this EZGraphics file that implements Rance Necaise's EZGraphics package, ready for image capture with CodeCheck.

For C++, add the files in this archive. Then you get functions to read an image into a 2D vector of grayscale values, and to save the processed pixels to a file. That way, students can do the usual image manipulation—see this example.

## Parametric Problems

If you add a file `param.js` to your problem, you turn on parametric processing.

The `param.js` file can contain arbitrary JavaScript code. You can define an array of two strings called `delimiters` in which you define the left and the right delimiter of expressions that are evaluated and substituted. A good choice might be

```var delimiters = ["{{", "}}"]
```

The default is:

```var delimiters = ["⦃", "⦄"]
```

We initialize a random number generator from a cookie, so that each student gets the same problem when retrying. This random number generator is used in the following functions that are available for you:

Helper Functions for Parametric Problems
Function Explanation
`randInt(a, b)` A random integer ≥ a and ≤ b
`randFloat(a, b)` A random floating-point number ≥ a and ≤ b
`randCodePoint(a, b)` A random Unicode code point ≥ a and ≤ b. The arguments can be integers or strings of length 1. For example, `randCodePoint('a', 'z')` is a random lowercase letter.
`randSelect(a, b, c, ...)` Randomly selects one of the arguments.
`randWord()` Randomly selects a word from a list of about 3000 common English words.
`randString(len, a, b)` Randomly makes a string of the given length with characters ≥ a and ≤ b.
`randIntArray(len, a, b)` Fills an array of the given length with random integers ≥ a and ≤ b.
`randIntArray2(rows, columns, a, b)` Fills a two-dimensional array of the given dimensions with random integers ≥ a and ≤ b.
`randFloatArray(len, a, b)` Fills an array of the given length with random floating-point numbers ≥ a and ≤ b.
`randFloatArray2(rows, columns, a, b)` Fills a two-dimensional array of the given dimensions with random floating-point numbers ≥ a and ≤ b.
`randWordArray(len)` Fills an array of the given length with random English words.

For example, consider this `param.js` file:

```delimiters = ['{{', '}}']
num = randInt(1, 10)
name = ['', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine']
```

This `Numbers.java` file produces one of nine problems, to count the number of ones, twos, ..., nines.

```public class Numbers
{
/**
Count how many {{name[num]}}s are in the given integer n.
*/
//CALL 12345
//CALL 6789
//CALL {{num}}{{num}}0{{num}}
public static int {{name[num]}}s(int n)
{
int result = 0;
//HIDE
while (n > 0)
{
if ((n % 10) == {{num}})
result++;
n /= 10;
}
//EDIT . . .
return result;
}
}```

## Other Languages

### C++

The file extension must be .cpp. For example:

numbersTester.cpp
```#include <iostream>

using namespace std;

int square(int n);

main()
{
cout << square(3) << endl;
cout << "Expected: 9" << endl;

cout << square(-3) << endl;
cout << "Expected: 9" << endl;
}
```
numbers.cpp
```int square(int n)
{
//HIDE
return n * n;
//EDIT ...
}
```

or

average.cpp
```//CALL 3, 4
//CALL -3, 3
//CALL 3, 0
double average(int x, int y)
{
//HIDE
return 0.5 * (x + y);
//EDIT ...
}
```

When you use `CALL`, you are limited to functions returning `int`, `double`, `float`, `char`, `bool`, `string`, `char*`, or `vector` thereof, such as `vector<string>` or `vector<vector<double>>`.

When using `CALL`, the entire solution file (after any `#include` and `using` statements) is placed into a separate namespace.

### Python

Use `##` instead of `//` for pseudocomments. For example:

numbersRunner.py
```from numbers import square

def main() :
for i in range(0, 10) :
print(square(i))

main()
```
numbers.py
```def square(n) :
##HIDE
return n * n
##EDIT return . . .
```

Note the use of `main` so that CodeCheck can tell which is the main file.

Caution: Don't use `...` to indicate that students should fill in code. That's valid Python—an “ellipsis” object. Instead, use `. . .` with spaces.

With an “expected” style unit test, name your tester `xxxTester.py`. No `main` is required.

numbersTester.py
```from numbers import square

print(square(3))
print("Expected: 9");
print(square(-3))
print("Expected: 9");
print(square(0))
print("Expected: 0");
```
numbers.py
```def square(n) :
##HIDE
return n * n
##EDIT return . . .
```

No `main` is required with `CALL` either:

avg.py
```##CALL 3, 4
##CALL -3, 3
##CALL 3, 0
def average(x, y) :
##HIDE
return (x + y) / 2
##EDIT return . . .
```

Caution: Don't use the same name for the module and the functions in the module when using `##CALL`. It results in a malformed tester.

Python unit tests are supported. The unit test filename must end in `Test`, `Test1`, `Test2`, and so on. For example,

numbers.py
```def square(n) :
##HIDE
return n * n
##EDIT return . . .
```
NumbersTest.py
```import unittest, numbers

class NumbersTest(unittest.TestCase):

def testNonNegativeSquares(self):
for n in range(100):
self.assertEqual(n * n, numbers.square(n))

def testNegativeSquares(self):
for n in range(1, 100):
self.assertEqual(n * n, numbers.square(-n))
```

### JavaScript

CodeCheck uses the Node.js runtime that comes with Ubuntu 18.04 (currently 8.10.0.)

Here is an example for `CALL`:

average.js
```//CALL 3, 4
//CALL 3, 3
//CALL 0, -1
function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}```

And an example for `SUB`:

average.js
```function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}
const arg1 = 3 //SUB 3; 0
const arg2 = 4 //SUB 3; -1
console.log(average(arg1, arg2));
```

If you want to run a program with more than one file, have the name of the main file end in `Runner.js`, and use Node.js modules to include the others.

You also need to use modules to run “Expected” style testers. (The tester must be in a separate file so that CodeCheck doesn't assume that the student should write the tester.)

average.js
```function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}

exports.average = average```
averageTester.js
```const average = require("./average.js").average;
console.log(average(3, 4));
console.log("Expected: 3.5");
console.log(average(3, 3));
console.log("Expected: 3");
console.log(average(0, -1));
console.log("Expected: -0.5");
```

### Scheme

CodeCheck supports Racket. The file extension must be .rkt. The most useful problem type is `CALL`. For example:

PosNums.rkt
```#lang racket
(provide positive-nums-only)
;;CALL '(1 2 -4 90 -4)
;;CALL '(-4 -3 0)
(define (positive-nums-only lst)
;;HIDE
(cond [(null? lst) lst]
[(> (car lst) 0) (cons (car lst) (positive-nums-only (cdr lst)))]
[else (positive-nums-only (cdr lst))])
;;EDIT ...
)
```

Racket unit tests (but not RackUnit tests) are also supported:

RevTest1.rkt
```#lang racket
(require test-engine/racket-tests)
(require "Rev.rkt")
(check-expect (reverse-list '(1 2 3)) '(3 2 1))
(check-expect (reverse-list '(1)) '(2))
(check-expect (reverse-list '()) '())
(test)
```

Finally, you can run programs and provide inputs:

Rev.rkt
```;;IN (1 2 3 4 5)
;;IN ()
#lang racket

(define (reverse-list lst)
(foldl
;;HIDE
cons '() lst)
;;EDIT ...
)

(writeln (reverse-list arg)))
```

The heuristic is that a file that isn't a unit test and that doesn't contain `(provide ...)` should be run. Also, a file that provides a `main` function is run.

### Standard ML

CodeCheck uses PolyML.

The file extension must be .sml. The most useful problem type is `CALL`. For example:

factorial.sml
```(* Complete the function below to compute the factorial of a nonnegative argument. *)
(*HIDE*)
(*CALL 3 *)
(*CALL 0 *)
fun fact 0 = 1
| fact n = n * fact(n - 1)
(*EDIT fun fact ... *)
```

Separate multiple arguments with spaces:

```(*CALL 3 4 *)
(*CALL 0 2 *)
fun sum a b = a + b
```

Provide parentheses with tuple arguments:

factorial.sml
```(*CALL (3, 4) *)
(*CALL (0, 2) *)
fun sum (a, b) = a + b
```

Substitutions also work:

factorial.sml
```(*HIDE*)
fun fact 0 = 1
| fact n = n * fact(n - 1)
(*EDIT fun fact ... *)

fun main() =
let
val arg = 3; (*SUB 0; 1; 6; 20 *)
val actual = fact arg
in
print (PolyML.makestring (actual))
end;
```

But that can be done more elegantly like this:

factorial.sml
```(*HIDE*)
fun fact 0 = 1
| fact n = n * fact(n - 1)
(*EDIT fun fact ... *)

fun main() =
print (PolyML.makestring (map fact [0, 1, 2, 6, 20]));
```

You can also use “Expected” style testers. Note that the tester must be a separate file.

factorial.sml
```(*HIDE*)
fun fact 0 = 1
| fact n = n * fact(n - 1)
(*EDIT fun fact ... *)
```
factorialTester.sml
```use "factorial.sml";

fun main() =
let
val actual = map fact [0, 1, 2, 6, 20]
in
print ((PolyML.makestring (actual)) ^ "\nExpected: [1, 1, 2, 720, 2432902008176640000]\n")
end;
```

Here is how you use `CALL` tests:

Max.hs
```module Max where
--CALL []
--CALL [1, 2, 3]
--CALL 
maxNum :: [Integer] -> Integer
--HIDE
maxNum [] = error "empty list"
maxNum (x:[]) = x
maxNum (x:xs) = if x > largest then x else largest
where largest = maxNum xs
--EDIT
```

You can also run arbitrary programs:

Max.hs
```maxNum :: [Integer] -> Integer
--HIDE
maxNum [] = error "empty list"
maxNum (x:[]) = x
maxNum (x:xs) = if x > largest then x else largest
where largest = maxNum xs
--EDIT

main :: IO ()
main = do
print \$ maxNum [1,2,3]
print \$ maxNum 
print \$ maxNum []
```

## Summary of File Conventions

 `test.in`, `test[1-9].in` Standard input to be fed into running programs `*Tester.{java,cpp,py,...}`, `*Tester[1-9].{java,cpp,py,...}` An “Expected” style unit test `*Test.{java,py,...}`, `*Test[1-9].{java,py,...}` A unit test with the unit test framework supported with the given language `index.html` The problem description `checkstyle.xml` A CheckStyle configuration `*.jar` A JAR file with library code `param.js` Parameters for substitution in the problem description and source files

 `CALL` Call method with the supplied arguments `SUB` Substitute the given values for a variable `ID` The problem ID in your course. This is how the download JAR file will be named. Default: The name of the lexicographically first Java file in the solution directory. For example, if the solution has `Cat.java` and `Dog.java`, the file will be `Cat.jar`. If you don't like that, put something like `//ID hw2a` into one of the solution files, and the download will be named `hw2a.jar`. `HIDE` Don't show the portion between `HIDE` and the next `EDIT`. If this is the first line in a file without `EDIT`, don't show the file at all. `EDIT` Marks a block that should be editable in a solution. If present, the rest of the solution is not editable. Can optionally be followed by text that is placed into the editable text area. `SAMPLE` Program run is only a sample. Only capture and display student output; don't compare against solution `ARGS` The command line arguments to pass `OUT` The names of files (space separated) that the program is supposed to produce and that should be checked against the solution `TIMEOUT` The timeout for this codecheck run in milliseconds (default 30000) `MAXOUTPUTLEN` The maximum length of the output for this codecheck run (default 100000) `TOLERANCE` The tolerance to use when comparing numbers (default 1E-6) `IGNORECASE IGNORESPACE` True by default, set to `false` if you want comparisons to be case sensitive or space sensitive `REQUIRED FORBIDDEN` A regular expression that must be, or cannot be, present in the student's file. If there is a second comment line below it, its contents is displayed if the student's program fails the test. `NOSCORE` No score is reported.