Writing a CodeCheck Problem

Authoring Overview

CodeCheck is a “convention over configuration” unit test library for evaluating student programs. The emphasis is on minimizing the authoring work of an instructor.

CodeCheck supports Java, C++, Scala, and Python 3. More languages may be added in the future.

The author of a problem provides

CodeCheck expects a zip file with the following contents for each problem:

      index.html (optional)
      Solution file(s) 
      Source files for helper or tester classes (if needed)
      Input files (if needed)

Upload the ZIP file to this link. Then give the resulting URL to your students.

Marking Up the Solution

The solution file must start with the line

//SOLUTION

By default, the entire solution is hidden. The student only sees a blank text area to fill in.

You can hide only a part of the solution, by using //HIDE and //SHOW pseudocomments:

//SOLUTION
public class Numbers
{
   public static int square(int n)
   {
      //HIDE
      return n * n;
      //SHOW
   }
}

In this case, the student sees everything but the body of the square method.

If you want the student to see something instead of the hidden material, such as an ellipsis, a code fragment, or a comment, place it after the //SHOW. For example,

//SOLUTION
public class Numbers
{
   public static int square(int n)
   {
      //HIDE
      return n * n;
      //SHOW return . . .;
   }
}

You can provide multiple lines:

//SOLUTION
public class Numbers
{
   public static int square(int n)
   {
      //HIDE
      return n * n;
      //SHOW // Complete this method
      //SHOW . . .
   }
}

You can hide as many segments of the solution as you like:

//SOLUTION
public class Numbers
{
   public static int square(int n)
   {
      //HIDE
      return n * n;
      //SHOW . . .
   }

   public static int cube(int n)
   {
      //HIDE
      return n * n * n;
      //SHOW . . .
   }
}

What You Can Test

You can make four kinds of checks:

  1. Run the program. You supply the inputs. Both the student submission and the solution are compiled and run with the given inputs, and the outputs are compared.

    Here is an example in Java. See below for C++ and Python examples.

    Numbers.java
    //SOLUTION
    //IN 3\n-3\n0\n
    import java.util.Scanner;
    public class Numbers
    {
       public static void main(String[] args)
       {
          //HIDE
          Scanner in = new Scanner(System.in);
          boolean done = false;
          while (!done)
          {
             //SHOW . . .
             System.out.println("Enter a number, 0 to quit");
             //HIDE
             int n = in.nextInt();
             if (n == 0) done = true;
             else
             {
                 int square =  n * n;
                 //SHOW . . .
                 System.out.println("The square is " + square);
                 //HIDE
             }
          }
          //SHOW
       }
    }
  2. There can be more than one IN pseudocomment. Each of them causes a program run where the contents is fed to standard input (System.in in Java or cin in C++). The contents is a single line with Java escape sequences: \n is a newline, \\ a backslash, and so on. Unicode escapes \uxxxx are supported.

    If the inputs are long, don't use the IN pseudocomments, but instead supply input files named test.in, test2.in,test3.in, and so on.

    test.in
    3
    -3
    0

    Tip: It is a good idea to give the students the statements for prompts and outputs, as in the preceding example.

  3. Unit tests.

    CodeCheck supports a simple style of doing unit testing that works with any language, and that doesn't require any library. (With Java, JUnit is also supported.)

    The tester file name ends in Tester, Tester2, Tester3, and so on. Each tester writes output consisting of pairs of lines, each of the form

    Optional label, colon, and space, followed by value
    Expected: value

    For example:

    Cube volume: 27
    Expected: 27
    Surface area: 9
    Expected: 54

    CodeCheck checks that the values of each pair match.

    Example:

    NumbersTester.java
    public class NumbersTester
    {
       public static void main(String[] args)
       {
          Numbers nums = new Numbers();
          System.out.println(nums.square(3));
          System.out.println("Expected: 9");
          System.out.println(nums.square(-3));
          System.out.println("Expected: 9");
       }
    }
    Numbers.java
    //SOLUTION
    public class Numbers
    {
       public int square(int n) { return n * n; }
    }

    Normally, the instructor writes the tester. But if you add a Tester file that is marked as a solution, then that means that you expect the student to write it. In that case, you need to decide whether the student needs to supply the exact same tests as in your solution (in which case you need to give the student specific instructions to replicate the output) or whether arbitrary tests are ok (in which case you mark your solution with the SAMPLE pseudocomment).

    You can also supply JUnit test cases in the student directory that end in Test, Test1, Test2, and so on. Those are executed with JUnit in the usual way.

  4. Test a single method. Supply pseudocomments //CALL in the lines immediately above the method.

    Example:

    Numbers.java
    //SOLUTION
    public class Numbers
    {
    //CALL 3, 4
    //CALL -3, 3
    //CALL 3, 0
       public double average(int x, int y)
       {
          //HIDE
          return 0.5 * (x + y);
          //SHOW // Compute the average of x and y
       }
    }

    The pseudocomment //CALL must be present in exactly one solution file. It cannot be in student file.

  5. Substitute variables. This is useful very early in a CS1 course when input and methods haven't yet been introduced. The file is rewritten with different settings for the variables.

    Example:

    Numbers.java
    //SOLUTION
    public class Numbers
    {
       public static void main(String[] args)
       {
          int x = 3; //SUB 5; 8
          int y = 4; //SUB 6; 4
          //HIDE
          double average = 0.5 * (x + y);
          //SHOW // Compute the average of x and y
          //SHOW // and assign it to a variable average
          System.out.println(average);
       }
    }

    The student program is run three times, with x set to 3, 5, 8, and y set to 4, 6, 4.

    The pseudocomment //SUB must be present in exactly one solution file. It cannot be in a student file. Values are separated by semicolons so that you can supply expressions with commas.

Command-Line Arguments

When running a program, you can optionally supply command-line arguments with an //ARGS pseudocomment. You can supply mutliple //ARGS lines. The student program and solution are run for each of them, and their outputs are compared.

Example:

values.txt
3
-3
values2.txt
0
Numbers.java
//SOLUTION
//ARGS values.txt
//ARGS values2.txt

import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;

public class Numbers
{
   public static void main(String[] args) throws FileNotFoundException
   {
      Scanner in = new Scanner(new File(args[0]));      
      while (in.hasNextInt())
      {
         int n = in.nextInt();
         System.out.println(n * n);
      }
   }
}

File Output

If you expect the student to write output to a file (i.e. not System.out), or to multiple files, provide their names with an //OUT pseudocomment. Names are separated by white space.

Then the student and solution programs are run to produce these files, and the results are compared. If an output file is an image, image comparison is automatically used.

Example:

values.txt
3
-4
0
Numbers.java
//SOLUTION
//OUT evens.txt odd.txt
import java.io.*;
import java.util.*;

public class Numbers
{
   public static void main(String[] args) throws IOException
   {
      Scanner in = new Scanner(new File("values.txt"));
      PrintWriter odds = new PrintWriter("odds.txt");
      PrintWriter evens = new PrintWriter("evens.txt");
      while (in.hasNextInt())
      {
         int n = in.nextInt();
         if (n % 2 == 0) evens.println(n); else odds.println(n);
      }
   }
}

Separate Student and Solution Directories

If it is too tedious to use HIDE and SHOW pseudocomments in a solution, then you can use an alternate problem layout. Make subdirectories student and solution. Put the solution files into the solution directory, without using a SOLUTION pseudocomment. Put all other files in the student directory, as well as versions of the solution that you want students to complete.

Example:

solution/Numbers.java
public class Numbers
{
   //CALL 3
   //CALL 0
   //CALL -1
   public int square(int n) { return n * n; }
}
student/Numbers.java
public class Numbers
{
   /*
      Provide a method called square that receives an integer argument
      and that returns the square of the argument.
   */
   . . .
}

Multiple Levels

You can set up multiple levels of student checks, for example for draft and final submissions. Then add directories student, student2, student3, and so on. If you don't want to use HIDE and SHOW pseudocomments, provide matching solution directories (solution, solution2, solution3, and so on). When checking up to level l, all directories studentN and all directories solutionN with N < l are merged. (student is considered the same as student1 and solution the same as solution1.) Later files can wipe out earlier ones.

Example:

student/NumbersTester.java
// This tests the draft
public class NumbersTester
{
   public static void main(String[] args)
   {
      Numbers nums = new Numbers();
      System.out.println(nums.square(3));
      System.out.println("Expected: 0");
      System.out.println(nums.square(-3));
      System.out.println("Expected: 0");
   }
}
solution/Numbers.java
// This is the draft
public class Numbers
{
   public int square(int n) { return 0; }
}
student2/NumbersTester.java
// This tests the final version
public class NumbersTester
{
   public static void main(String[] args)
   {
      Numbers nums = new Numbers();
      System.out.println(nums.square(3));
      System.out.println("Expected: 9");
      System.out.println(nums.square(-3));
      System.out.println("Expected: 9");
   }
}
solution2/Numbers.java
// This is the final version
public class Numbers
{
   public int square(int n) { return n * n; }
}

C++ Problems

You can provide problems in C++. The file extension must be .cpp. For example:

numbersTester.cpp
#include <iostream>

using namespace std;

int square(int n);

main()
{
   cout << square(3) << endl;
   cout << "Expected: 9" << endl;

   cout << square(-3) << endl;
   cout << "Expected: 9" << endl;   
}
numbers.cpp
//SOLUTION
int square(int n)
{
   return n * n;
}

or

average.cpp
//SOLUTION
//CALL 3, 4
//CALL -3, 3
//CALL 3, 0
double average(int x, int y)
{
   return 0.5 * (x + y);
}

When you use CALL, you are limited to functions returning int, double, float, char, bool, string, char*, or vector thereof, such as vector<string> or vector<vector<double>>.

Python Problems

You can provide problems in Python. Then use ## instead of //. For example:

numbersTester.py
from numbers import square

print(square(3))
print("Expected: 9");
print(square(-3))
print("Expected: 9");
print(square(0))
print("Expected: 0");
numbers.py
##SOLUTION
def square(n) :
    return n * n

or

avg.py
##SOLUTION
##CALL 3, 4
##CALL -3, 3
##CALL 3, 0
def average(x, y) :
    ##HIDE
    return (x + y) / 2
    ##SHOW return . . .

Caution: Don't use ... to indicate that students should fill in code. That's valid Python—an “ellipsis” object. Instead, use . . . with spaces.

Caution: Don't use the same name for the module and the functions in the module when using ##CALL. It results in a malformed tester.

Python unit tests are supported. The unit test filename must end in Test, Test1, Test2, and so on. For example,

numbers.py
##SOLUTION
def square(n) :
    return n * n
NumbersTest.py
import unittest, numbers

class NumbersTest(unittest.TestCase):

    def testNonNegativeSquares(self):
        for n in range(100):
            self.assertEqual(n * n, numbers.square(n))

    def testNegativeSquares(self):
        for n in range(1, 100):
            self.assertEqual(n * n, numbers.square(-n))

Racket Problems

You can provide problems in Racket. The file extension must be .rkt. The most useful problem type is CALL. For example:

PosNums.scm
;;SOLUTION
#lang racket
(provide positive-nums-only)
;;CALL '(1 2 -4 90 -4)
;;CALL '(-4 -3 0)
(define (positive-nums-only lst)
;;HIDE
  (cond [(null? lst) lst]
        [(> (car lst) 0) (cons (car lst) (positive-nums-only (cdr lst)))]
        [else (positive-nums-only (cdr lst))])
;;SHOW ...
)

Racket unit tests (but not RackUnit tests) are also supported:

RevTest1.rkt
#lang racket
(require test-engine/racket-tests)
(require "Rev.rkt")
(check-expect (reverse-list '(1 2 3)) '(3 2 1))
(check-expect (reverse-list '(1)) '(2))
(check-expect (reverse-list '()) '())
(test)

Finally, you can run programs and provide inputs:

Rev.rkt
;;SOLUTION
;;IN (1 2 3 4 5)
;;IN ()
#lang racket

(define (reverse-list lst)
  (foldl
  ;;HIDE
  cons '() lst)
  ;;SHOW ...
  )

(let ((arg (read (current-input-port))))
     (writeln (reverse-list arg)))

The heuristic is that a file that isn't a unit test and that doesn't contain (provide ...) should be run. Also, a file that provides a main function is run.

Checkstyle

With Java problems, you can optionally run Checkstyle on the submitted files.

Put a file checkstyle.xml into the student directory, with whatever checkstyle checks you want. Here is one that just checks for javadoc and variable naming conventions.

<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
    "-//Puppy Crawl//DTD Check Configuration 1.3//EN"
    "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
<module name="Checker">
    <module name="TreeWalker">
        <property name="tabWidth" value="4"/>

        <!-- Checks for Javadoc comments.                    -->
        <!-- See http://checkstyle.sf.net/config_javadoc.html -->
        <module name="JavadocMethod">
           <property name="scope" value="public"/>
           <property name="allowMissingThrowsTags" value="true"/>
        </module>
        <module name="JavadocType"/>

        <!-- Checks for Naming Conventions.                  -->
        <!-- See http://checkstyle.sf.net/config_naming.html -->
        <module name="ConstantName"/>
        <module name="LocalFinalVariableName">
           <property name="format" value="^[A-Z](_?[A-Z0-9]+)*$"/>
        </module>
        <module name="LocalVariableName"/>
        <module name="MemberName"/>
        <module name="MethodName"/>
        <module name="PackageName"/>
        <module name="ParameterName"/>
        <module name="StaticVariableName"/>
        <module name="TypeName"/>
    </module>
</module>

How Comparisons Work

The comparison rules take care of the biggest annoyances that students face: white space, roundoff, and letter case.

When the actual and expected outputs are compared, corresponding lines are compared.

By default, white space within a line is not significant. For example,

Hello, World!

and

   Hello,  World!

match. If you want to take white space into account. use the pseudocomment //IGNORESPACE false.

If corresponding tokens within a line are numbers, they compare identical if they are within the given tolerance (1E-6 by default). For example,

Area: 1.9999999999996

and

Area: 2.0

match, as do

Area: 2

and

Area: 2.0

If you want to use a different tolerance, supply a pseudocomment such as //TOLERANCE 0.01.

If tokens are not numbers,  the comparison is case-insensitive by default. For example,

Hello, World!

and

HELLO, WORLD!

match. Use the pseudocomment //IGNORECASE false if you want comparison to be case-sensitive.

Checking for Required and Forbidden Strings

Suppose you want to insist that an assignment is done with a for loop. You can test for it with

//REQUIRED for

or, better, something like

//REQUIRED for\s*\([^;]*;[^;]*;.*\)

Conversely, you may want to forbid something, for example disallow calling the contains method. You can test that with

//FORBIDDEN \.contains\s*\(

If these tests fail, the student gets zero points and the program is not compiled.

JAR Files

Just toss them into the student directory, and they will be added to the class path when the code is compiled and run. Thanks to Dustin Hoffman for this enhancement.

Graphics Programs

If you want to check simple graphics programs in Java that draw graphics on a JComponent, add this JFrame class to your project. Make sure to import javax.swing.* and not javax.swing.JFrame so that the bogus frame class gets used. The painting is simply captured in a file. (The CodeCheck server is “headless”—there are no frames and windows.) As long as you don't use buttons, sliders, menus, timers, and so on, this works fine.

Alternatively, check out the Simple Java Graphics library that I wrote for the Udacity CS046 course.

For Python, use this EZGraphics file that implements Rance Necaise's EZGraphics package, ready for image capture with CodeCheck.

For C++, add the files in this archive. Then you get functions to read an image into a 2D vector of grayscale values, and to save the processed pixels to a file. That way, students can do the usual image manipulation—see this example.

Glossary of Pseudocomments

CALL Call method with the supplied arguments
SUB Substitute the given values for a variable
ID The problem ID in your course. This is how the download JAR file will be named. Default: The name of the lexicographically first Java file in the solution directory. For example, if the solution has Cat.java and Dog.java, the file will be Cat.jar. If you don't like that, put something like //ID hw2a into one of the solution files, and the download will be named hw2a.jar.
HIDE If this is the first line in a file, then don't show this file to students. If it occurs in a solution file, don't show the portion between HIDE and the next SHOW.
SHOW Marks the end of a block that should be hidden in a solution. Can optionally be followed by text that is spliced into the student version.
SOLUTION Marks this file as a solution. If the file contains no HIDE statements, it is not shown to students. Otherwise, the blocks surrounded by HIDE and SHOW are hidden.
SAMPLE Program run is only a sample. Only capture and display student output; don't compare against solution
ARGS The command line arguments to pass
OUT The names of files (space separated) that the program is supposed to produce and that should be checked against the solution
TIMEOUT The timeout for this codecheck run in milliseconds (default 30000)
MAXOUTPUTLEN The maximum length of the output for this codecheck run (default 100000)
TOLERANCE The tolerance to use when comparing numbers (default 1E-6)
IGNORECASE
IGNORESPACE
True by default, set to false if you want comparisons to be case sensitive or space sensitive
REQUIRED
FORBIDDEN
A regular expression that must be, or cannot be, present in the student's file. If there is a second comment line below it, its contents is displayed if the student's program fails the test.
NOSCORE No score is reported.

Note: Pseudocomments can only occur in solution files or in student files that also have the HIDE pseudocomment.