Why use CodeCheck and not a fancier system? There are two advantages:
Suppose you have a sample program for your students, such as the classic “rainfall” problem: Find the average of a sequence of numbers, but ignore negative values (which must have been measurement errors), and stop when you reach the sentinel 9999.
public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
}
else
{
return sum / count;
}
}
public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}
Wouldn't your students learn much more if they wrote the program instead of looking at your solution? Sure they would. With CodeCheck, you can produce a simple assignment quickly.
What do you want to focus on with this exercise? Presumably loops and corner cases, not the syntax of public static void main. If you want your students to fill in the entire method, do this:
public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
//HIDE
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
}
else
{
return sum / count;
}
//EDIT . . .
}
public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}
Visit this link and paste in the file. You will get an URL such as this one: https://codecheck.io/files/20032817325la004mxsdozekzv7jpe6nmsd. Give it to your students. Right now, check out that link so that you see what the students see—in particular, how the //HIDE and //EDIT leave a hole.
Maybe you want your students to focus on the logic, not the statement syntax? Not a bad idea—that way you can give them more problems to work on in their limited time. Then you need to be craftier about the holes that you punch out. Look at this student view and compare it with the markup:
public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
//HIDE
while (rainfall[i] != 9999)
//EDIT while (. . .)
{
//HIDE
if (rainfall[i] >= 0)
//EDIT if (. . .)
{
//HIDE
sum += rainfall[i];
count++;
//EDIT . . .
}
i++;
}
//HIDE
if (count == 0)
//EDIT if (. . .)
{
return 0;
}
else
{
return sum / count;
}
}
public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}
The reporting is pretty basic with this approach. You can do better by using CALL pseudocomments for test cases:
public class Rainfall
{
//CALL new double[] { 1, 2, 3, 4, 5, 9999 }
//CALL new double[] { 1, 2, 3, 4, 5, 9999, 6, 7 }
//CALL new double[] { 10, 9999 }
//CALL new double[] { 10, 0, 9999 }
//CALL new double[] { -1, -2, -3, 9999 }
//CALL new double[] { 15, 0, -53, 5, 2, 9999 }
//CALL new double[] { 9999 }
public static double averageRainfall(double[] rainfall)
{
double sum = 0;
int i = 0;
int count = 0;
//HIDE
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
return count == 0 ? 0 : sum / count;
//EDIT . . .
}
}
Try it out here. Paste in
return 0;to see what happens with a wrong solution, or
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
return sum / count;
for an almost correct solution.
Finally, perhaps you want your students not to get bogged down with syntax errors? Then make a Parsons puzzle, where the students drag tiles with the code in the right order:
You achieve this by using the pseudocomments //TILE and //FIXED. Then the lines turn into draggable tiles and fixed areas, as shown in the picture. The tiles are scrambled.
public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
//TILE
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
}
else
{
return sum / count;
}
//FIXED
}
public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}
You can also add distractor tiles and put multiple lines of code on a tile. For details, see the section on Parsons puzzles.
That's the crash course. Try CodeCheck with your own sample programs, and read the following for all the details.
In a Parsons puzzle (as in this example), students are given code in scrambled order and are asked to rearrange the lines so that the code fulfills a given task. These puzzles are popular for teaching programming. Researchers have found that performance on solving Parsons puzzles correlates well with performance on code-writing exercises, while reducing cognitive load and frustration.
Why use CodeCheck instead of a simple generator such as https://www.codepuzzle.io/en? If you can only specify one correct solution, then you are restricted to very simple puzzles. Conversely, with tools that allow the author to enumerate multiple correct permutations, students may find valid solutions that the author had not considered. When using CodeCheck for Parsons problems, you do not specify in advance which permutations of tiles are acceptable. The code that the student constructs simply needs to pass the test cases.
The //TILE pseudocomment activates the puzzle editor for a file. A problem can use the puzzle editor for some files and the text editor on others.
In the puzzle editor, lines either belong to a fixed or a tile region. The file starts out with a fixed region. The //TILE pseudocomment switches to a tile region, and the //FIXED pseudocomment switches back.
Each fixed region is turned into a block that cannot be moved. In a tile region, each line becomes a draggable tile. Tiles are scrambled and must be rearranged. They can be dragged into the gaps between blocks or other tiles.
In this screenshot (click to enlarge), you can see the blocks in dark gray on the left and the tiles in light gray. At the outset, all tiles are placed in random order on the right. Here, some of them have been dragged to the left to build up a solution.
When the CodeCheck button is pushed, the code on the left is sent to the autograder, in the same way as with the text editor. Remaining tiles on the right are ignored.
To obtain a tile with multiple lines, prefix it with //TILE n, where n is the desired number of lines to include. For example:
//TILE 3
}
else
{
Subsequent tiles revert to single lines.
Be careful about indentation. If you only use spaces, and are consistent about indentation levels, or if you only use tabs, CodeCheck will parse the indentation levels correctly.
Here is the input for the problem in the screenshot. Note the fixed blocks at the top and bottom, and the three-line tile with else and braces. The tiles starting with if and while automatically include the opening brace on the next line.
public class Rainfall
{
public static double averageRainfall(double[] rainfall)
{
//TILE
double sum = 0;
int i = 0;
int count = 0;
while (rainfall[i] != 9999)
{
if (rainfall[i] >= 0)
{
sum += rainfall[i];
count++;
}
i++;
}
if (count == 0)
{
return 0;
//TILE 3
}
else
{
return sum / count;
}
//FIXED
}
public static void main(String[] args)
{
System.out.println(averageRainfall(new double[] { 1, 2, -3, -4, -5, 6, 9999 }));
System.out.println(averageRainfall(new double[] { -1, -2, -3, 9999, 100 }));
}
}
In a Parsons puzzle, you can add alternatives with the //OR pseudocomment. For example:
for (int i = 0; i < n; i++)
//OR for (int i = 0; i <= n; i++)
//OR for (int i = 1; i <= n; i++)
The alternatives are grouped with the preceding tile, and the group is scrambled. In this case, the alternatives are distractors—wrong choices that students should not make. Use them to make students think about typical choices—whether to start the loop with 0 or 1, or whether to use < or <=.
Note that the distractor code is located inside a comment so that it does not disturb compilation of the file.
Alternatives in a fixed region become global. They are not grouped with other tiles but placed in random positions.
Multi-line distractors are similar to multi-line tiles:
//OR 2
// if (x > 0)
// System.out.println("Done");
If you want to introduce gaps between fixed blocks to confound students, create multiple adjacent //FIXED regions.
A pseudocode puzzle has tiles that are marked with pseudocode. Students need to rearrange them, just like with regular Parsons puzzles.
A pseudocode tile has a different legend than the code that it emits:
words = ["Mary", "had", "a", "little", "lamb"]##PSEUDO initialize the array words
##TILE
result = ""##PSEUDO result = empty string
for w in words :##PSEUDO for every word w in words
if len(result) > 0 :##PSEUDO if result is not empty
result += ", ";##PSEUDO append a comma to result
result += w;##PSEUDO append w to result
print(result)##PSEUDO print result
The student sees only the pseudocode, displayed in a cursive font. The checker sees the code before the comment.
This works best with Python since you probably don't want students to bother with braces in pseudocode. If you really don't want to author the problem in Python, you could use
import java.util.*;//PSEUDO imports
public class Test { public static void main(String[] args) {//PSEUDO start of main
//TILE
var words = List.of("Mary", "had", "a", "little", "lamb");//PSEUDO initialize words
String result = "";//PSEUDO result = empty string
for (String w : words) {//PSEUDO for every word w in words
if (result.length() > 0) {//PSEUDO if result is not empty
result += ", ";//PSEUDO append a comma to result
}//PSEUDO end
result += w;//PSEUDO append w to result
}//PSEUDO end
System.out.println(words);//PSEUDO print words
//FIXED
}}//PSEUDO end of main
CodeCheck supports
Ask if you want support for another language.
CodeCheck is a “convention over configuration” autograder for evaluating student programs. The emphasis is on minimizing the authoring work of an instructor.
The author of a problem provides
index.htmlTo upload a CodeCheck problem, visit this link. You can either paste the files into the form or upload a zip file.
When the problem is uploaded, you can preview it and see a report where CodeCheck compiles and runs the solution. If you spot any errors, you can edit the problem.
Once everything works, give the public URL to your students.
Hang on to the private URL in case you need to edit your problem.
Write the solution and test it locally. Then hide a part of the solution, by using HIDE and EDIT pseudocomments:
public class Numbers
{
public static int square(int n)
{
//HIDE
return n * n;
//EDIT return . . .;
}
}
The code between //HIDE and //EDIT is hidden. Everything else is read-only. The contents after the //EDIT is placed into an editable text area.
An //EDIT area can span multiple line, and you can have multiple //EDIT areas:
public class Numbers
{
public static int square(int n)
{
//HIDE
return n * n;
//EDIT
}
public static int cube(int n)
{
//HIDE
return n * n * n;
//EDIT int result = . . .;
//EDIT return . . .;
}
}
If you want to hide the entire solution and have the student only see a blank text area, put //HIDE on top and //EDIT on the bottom.
Caution: You cannot use HIDE/EDIT to hide required utility code, because then the student submission would not compile. Hidden utility code needs to be in a separate file—see the next section.
Any file that doesn't contain //EDIT is displayed to the student but can't be edited. This is for helper files that the student needs to read.
Any file that starts with //HIDE is completely hidden from the student. Use this for utility or library files that are necessary for compiling/running but that the student should not see
You can make four kinds of checks:
You supply the inputs. Both the student submission and the solution are compiled and run with the given inputs, and the outputs are compared.
The student program is run for each //IN line. Use \n for a line break.
Here is an example in Java, which is run twice, with inputs
3 -3 0
and
1 10 100 1000 0
See below for C++ and Python examples.
//IN 3\n-3\n0\n
//IN 1\n10\n100\n1000\n0\n
import java.util.Scanner;
public class Numbers
{
public static void main(String[] args)
{
//HIDE
Scanner in = new Scanner(System.in);
boolean done = false;
while (!done)
{
//EDIT . . .
System.out.println("Enter a number, 0 to quit");
//HIDE
int n = in.nextInt();
if (n == 0) done = true;
else
{
int square = n * n;
//EDIT . . .
System.out.println("The square is " + square);
//HIDE
}
}
//EDIT . . .
}
}
There can be more than one IN pseudocomment. Each of them causes a program run where the contents is fed to standard input (System.in in Java or cin in C++). The contents is a single line with Java escape sequences: \n is a newline, \\ a backslash, and so on. Unicode escapes \uxxxx are supported.
If the inputs are long, don't use the IN pseudocomments, but instead supply input files named test.in, test2.in,test3.in, and so on.
3 -3 0
Tip: It is a good idea to give the students the statements for prompts and outputs, as in the preceding example.
CodeCheck supports a simple style of doing unit testing that works with any language, and that doesn't require any framework. (With some languages, a unit testing framework is also supported.)
The tester file name ends in Tester, Tester2, Tester3, and so on. Snake case suffixes _tester, _tester2 are ok too. Each tester writes output consisting of pairs of lines, each of the form
Optional label, colon, and space, followed by value Expected: value
For example:
Cube volume: 27 Expected: 27 Surface area: 9 Expected: 54
CodeCheck checks that the values of each pair match.
Example:
public class NumbersTester
{
public static void main(String[] args)
{
Numbers nums = new Numbers();
System.out.println(nums.square(3));
System.out.println("Expected: 9");
System.out.println(nums.square(-3));
System.out.println("Expected: 9");
}
}
public class Numbers
{
public int square(int n)
{
//HIDE
return n * n;
//EDIT ...
}
}
Normally, the instructor writes the tester. But if you add a Tester file that has an EDIT area, then that means that you expect the student to write it. (In that case, you need to give the student specific instructions to replicate the output of your solution.)
You can also supply unit test cases that end in Test, Test1, Test2, and so on. Snake case _test, _test2 suffixes are ok too. Those are executed with the unit test framework appropriate for the language (e.g. JUnit for Java).
Supply CALL pseudocomments in the lines immediately above the method.
Example:
public class Numbers
{
//CALL 3, 4
//CALL -3, 3
//CALL 3, 0
public double average(int x, int y)
{
//HIDE
return 0.5 * (x + y);
//EDIT // Compute the average of x and y
}
}
The CALL pseudocomments must be present in exactly one file.
This is useful very early in a CS1 course when input and methods haven't yet been introduced. The file is rewritten with different settings for the variables.
Example:
public class Numbers
{
public static void main(String[] args)
{
int x = 3; //SUB 5; 8
int y = 4; //SUB 6; 4
//HIDE
double average = 0.5 * (x + y);
//EDIT // Compute the average of x and y
//EDIT // and assign it to a variable average
System.out.println(average);
}
}
The student program is run three times, with x set to 3, 5, 8, and y set to 4, 6, 4.
The SUB pseudocomments must be present in exactly one solution file. Values are separated by semicolons so that you can supply expressions with commas.
The SUB pseudocomment must be placed at the first declaration of the variable in the file. That declaration should be in the non-editable part of the code. Pay particular attention in Python where the declaration and subsequent assignments are indistinguishable.
When running a program, you can optionally supply command-line arguments with an ARGS pseudocomment. You can supply mutliple ARGS lines. The student program and solution are run for each of them, and their outputs are compared.
Example:
3 -3
0
//ARGS values.txt
//ARGS values2.txt
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class Numbers
{
public static void main(String[] args) throws FileNotFoundException
{
Scanner in = new Scanner(new File(args[0]));
//HIDE
while (in.hasNextInt())
{
int n = in.nextInt();
System.out.println(n * n);
}
// EDIT ...
}
}
If you expect the student to write output to a file (i.e. not System.out), or to multiple files, provide their names with an OUT pseudocomment. Names are separated by white space.
Then the student and solution programs are run to produce these files, and the results are compared. If an output file is an image, image comparison is automatically used.
Example:
3 -4 0
//OUT evens.txt odd.txt
import java.io.*;
import java.util.*;
public class Numbers
{
public static void main(String[] args) throws IOException
{
Scanner in = new Scanner(new File("values.txt"));
PrintWriter odds = new PrintWriter("odds.txt");
PrintWriter evens = new PrintWriter("evens.txt");
//HIDE
while (in.hasNextInt())
{
int n = in.nextInt();
if (n % 2 == 0) evens.println(n); else odds.println(n);
}
//EDIT ...
}
}
The comparison rules take care of the biggest annoyances that students face: white space, roundoff, and letter case.
When the actual and expected outputs are compared, corresponding lines are compared.
By default, white space within a line is not significant. For example,
Hello, World!
and
Hello, World!
match. If you want to take white space into account. use the pseudocomment IGNORESPACE false.
If corresponding tokens within a line are numbers, they compare identical if they are within the given tolerance (1E-6 by default). For example,
Area: 1.9999999999996
and
Area: 2.0
match, as do
Area: 2
and
Area: 2.0
If you want to use a different tolerance, supply a pseudocomment such as TOLERANCE 0.01.
If tokens are not numbers, the comparison is case-insensitive by default. For example,
Hello, World!
and
HELLO, WORLD!
match. Use the pseudocomment IGNORECASE false if you want comparison to be case-sensitive.
Suppose you want to insist that an assignment is done with a for loop. You can test for it with
//REQUIRED for
or, better, something like
//REQUIRED for\s*\([^;]*;[^;]*;.*\)
Conversely, you may want to forbid something, for example disallow calling the contains method. You can test that with
//FORBIDDEN \.contains\s*\(
A comment following the REQUIRED or FORBIDDEN pseudocomment is displayed to the student when the test fails:
//FORBIDDEN \.contains\s*\( // Do not use the contains method!
If these tests fail, the student gets zero points and the program is not compiled.
You can hide a unit test from students by starting it with a line containing the
HIDDEN pseudocomment.
Example:
//HIDDEN
public class NumbersTester
{
public static void main(String[] args)
{
Numbers nums = new Numbers();
System.out.println(nums.square(3));
System.out.println("Expected: 9");
System.out.println(nums.square(-3));
System.out.println("Expected: 9");
}
}
The results of the unit test will be displayed, but not the code.
You can hide individual CALL arguments by using CALL HIDDEN:
//CALL HIDDEN 3, -3, 0
Use //IN HIDDEN to hide individual inputs:
//IN HIDDEN 3\n-3\n0
By default, CodeCheck displays interleaved console input and output with Python, Java, and C++. Here is a typical dialog:
How old are you? 42 Next year, you'll be 43.
Without interleaving, the output would be
How old are you? Next year, you'll be 43.
Interleaving is slow because CodeCheck has to guess that an output line is a prompt, by waiting for a short time before sending each input line. This can result in timeouts if a program has more than a few input lines. In particular, if you have a loop:
while (!done)
{
System.out.print("Enter an integer, 0 to quit: ");
input = in.nextInt();
if (input == 0) done = true;
else process(input);
}
then there is a short delay after each prompt. With many inputs, those delays add up.
Another problem is a program that doesn't prompt at all. CodeCheck can't tell whether the program is slow to ask for the next input, or whether there won't be a prompt. To be safe, it delays between all of its input lines.
In these situations, turn interleaving off with the INTERLEAVE false pseudocomment.
If you want to check simple graphics programs in Java that draw graphics on a JComponent, add this JFrame class to your project. Make sure to import javax.swing.* and not javax.swing.JFrame so that the bogus frame class gets used. The painting is simply captured in a file. (The CodeCheck server is “headless”—there are no frames and windows.) As long as you don't use buttons, sliders, menus, timers, and so on, this works fine.
Alternatively, check out the Simple Java Graphics library that I wrote for the Udacity CS046 course.
For Python, use this EZGraphics file that implements Rance Necaise's EZGraphics package, ready for image capture with CodeCheck.
For C++, add the files in this archive. Then you get functions to read an image into a 2D vector of grayscale values, and to save the processed pixels to a file. That way, students can do the usual image manipulation—see this example.
If you add a file param.js to your problem, you turn on parametric processing.
The param.js file can contain arbitrary JavaScript code. You can define an array of two strings called delimiters in which you define the left and the right delimiter of expressions that are evaluated and substituted. A good choice might be
var delimiters = ["{{", "}}"]
The default is:
var delimiters = ["⦃", "⦄"]
We initialize a random number generator from a cookie, so that each student gets the same problem when retrying. This random number generator is used in the following functions that are available for you:
| Function | Explanation |
|---|---|
randInt(a, b) |
A random integer ≥ a and ≤ b |
randFloat(a, b) |
A random floating-point number ≥ a and ≤ b |
randCodePoint(a, b) |
A random Unicode code point ≥ a and ≤ b. The arguments can be integers or strings of length 1. For example, randCodePoint('a', 'z') is a random lowercase letter. |
randSelect(a, b, c, ...) |
Randomly selects one of the arguments. |
randWord() |
Randomly selects a word from a list of about 3000 common English words. |
randString(len, a, b) |
Randomly makes a string of the given length with characters ≥ a and ≤ b. |
randIntArray(len, a, b) |
Fills an array of the given length with random integers ≥ a and ≤ b. |
randIntArray2(rows, columns, a, b) |
Fills a two-dimensional array of the given dimensions with random integers ≥ a and ≤ b. |
randFloatArray(len, a, b) |
Fills an array of the given length with random floating-point numbers ≥ a and ≤ b. |
randFloatArray2(rows, columns, a, b) |
Fills a two-dimensional array of the given dimensions with random floating-point numbers ≥ a and ≤ b. |
randWordArray(len) |
Fills an array of the given length with random English words. |
For example, consider this param.js file:
delimiters = ['{{', '}}']
num = randInt(1, 10)
name = ['', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine']
This Numbers.java file produces one of nine problems, to count the number of ones, twos, ..., nines.
public class Numbers
{
/**
Count how many {{name[num]}}s are in the given integer n.
*/
//CALL 12345
//CALL 6789
//CALL {{num}}{{num}}0{{num}}
public static int {{name[num]}}s(int n)
{
int result = 0;
//HIDE
while (n > 0)
{
if ((n % 10) == {{num}})
result++;
n /= 10;
}
//EDIT . . .
return result;
}
}
When a file with the name Input is included (with a capital I followed by four lowercase letters), then the problem is executed in demo mode, feeding the input to the program and displaying the output. No score is earned.
CodeCheck uses OpenJDK 11 (which comes with Ubuntu 20.04).
Unfortunately, packages are not currently supported. Use the default package.
Any JAR files that you include are added to the class path when the code is compiled and run. Thanks to Dustin Hoffman for this enhancement.
You can optionally run Checkstyle on the submitted files.
Put a file checkstyle.xml into the student directory, with whatever checkstyle checks you want. Here is one that just checks for javadoc and variable naming conventions.
<?xml version="1.0"?>
<!DOCTYPE module PUBLIC
"-//Puppy Crawl//DTD Check Configuration 1.3//EN"
"http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
<module name="Checker">
<module name="TreeWalker">
<property name="tabWidth" value="4"/>
<!-- Checks for Javadoc comments. -->
<module name="JavadocType"/>
<!-- Checks for Naming Conventions. -->
<module name="ConstantName"/>
<module name="LocalFinalVariableName">
<property name="format" value="^[A-Z](_?[A-Z0-9]+)*$"/>
</module>
<module name="LocalVariableName"/>
<module name="MemberName"/>
<module name="MethodName"/>
<module name="PackageName"/>
<module name="ParameterName"/>
<module name="StaticVariableName"/>
<module name="TypeName"/>
</module>
</module>
The file extension must be .cpp. For example:
#include <iostream>
using namespace std;
int square(int n);
main()
{
cout << square(3) << endl;
cout << "Expected: 9" << endl;
cout << square(-3) << endl;
cout << "Expected: 9" << endl;
}
int square(int n)
{
//HIDE
return n * n;
//EDIT ...
}
or
//CALL 3, 4
//CALL -3, 3
//CALL 3, 0
double average(int x, int y)
{
//HIDE
return 0.5 * (x + y);
//EDIT ...
}
When you use CALL, you are limited to functions returning int, double, float, char, bool, string, char*, or vector thereof, such as vector<string> or vector<vector<double>>.
When using CALL, the entire file (after any #include and using statements) is placed into a separate namespace.
Use ## instead of // for pseudocomments. For example:
from numbers import square
def main() :
for i in range(0, 10) :
print(square(i))
main()
def square(n) :
##HIDE
return n * n
##EDIT return . . .
Note the use of main so that CodeCheck can tell which is the main file.
Caution: Don't use ... to indicate that students should fill in code. That's valid Python—an “ellipsis” object. Instead, use . . . with spaces.
With an “expected” style unit test, name your tester xxxTester.py. No main is required.
from numbers import square
print(square(3))
print("Expected: 9");
print(square(-3))
print("Expected: 9");
print(square(0))
print("Expected: 0");
def square(n) :
##HIDE
return n * n
##EDIT return . . .
No main is required with CALL either:
##CALL 3, 4
##CALL -3, 3
##CALL 3, 0
def average(x, y) :
##HIDE
return (x + y) / 2
##EDIT return . . .
Caution: Don't use the same name for the module and the functions in the module when using ##CALL. It results in a malformed tester.
Python unit tests are supported. The unit test filename must end in Test, Test1, Test2, and so on. For example,
def square(n) :
##HIDE
return n * n
##EDIT return . . .
import unittest, numbers
class NumbersTest(unittest.TestCase):
def testNonNegativeSquares(self):
for n in range(100):
self.assertEqual(n * n, numbers.square(n))
def testNegativeSquares(self):
for n in range(1, 100):
self.assertEqual(n * n, numbers.square(-n))
CodeCheck uses Node.js runtime version 10 (which comes with Ubuntu 20.04).
Here is an example for CALL:
//CALL 3, 4
//CALL 3, 3
//CALL 0, -1
function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}And an example for SUB:
function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}
const arg1 = 3 //SUB 3; 0
const arg2 = 4 //SUB 3; -1
console.log(average(arg1, arg2));
If you want to run a program with more than one file, use Node.js modules to include the others. The main file is the one that doesn't contain exports.
You also need to use modules to run “Expected” style testers. (The tester must be in a separate file so that CodeCheck doesn't assume that the student should write the tester.)
function average(x, y)
{
//HIDE
return (x + y) / 2;
//EDIT ...
}
exports.average = average
const average = require("./average.js").average;
console.log(average(3, 4));
console.log("Expected: 3.5");
console.log(average(3, 3));
console.log("Expected: 3");
console.log(average(0, -1));
console.log("Expected: -0.5");
Because console input is not very natural with Node.js, I suggest that you stay away from the IN pseudocomment.
The file extension is .c. For example:
//IN 3\n-3\n0\n
#include <stdbool.h>
#include <stdio.h>
int main()
{
//HIDE
bool done = false;
while (!done)
{
//EDIT . . .
printf("Enter a number, 0 to quit: ");
//HIDE
int n;
scanf("%d", &n);
if (n == 0) done = true;
else
{
int square = n * n;
//EDIT . . .
printf("The square is %d\n", square);
//HIDE
}
//EDIT . . .
}
}
The name of a tester must end in Tester.c, or, if you have several, Tester1.c, Tester2.c, and so on:
#include <stdio.h>
extern int square(int);
int main()
{
printf("%d\n", square(3));
printf("Expected: 9\n");
printf("%d\n", square(-3));
printf("Expected: 9\n");
return 0;
}
int square(int n)
{
//HIDE
return n * n;
//EDIT ...
}
The CALL pseudocomment is not supported.
CodeCheck supports Racket. The file extension must be .rkt. The most useful problem type is CALL. For example:
#lang racket
(provide positive-nums-only)
;;CALL '(1 2 -4 90 -4)
;;CALL '(-4 -3 0)
(define (positive-nums-only lst)
;;HIDE
(cond [(null? lst) lst]
[(> (car lst) 0) (cons (car lst) (positive-nums-only (cdr lst)))]
[else (positive-nums-only (cdr lst))])
;;EDIT ...
)
Racket unit tests (but not RackUnit tests) are also supported:
#lang racket (require test-engine/racket-tests) (require "Rev.rkt") (check-expect (reverse-list '(1 2 3)) '(3 2 1)) (check-expect (reverse-list '(1)) '(2)) (check-expect (reverse-list '()) '()) (test)
Finally, you can run programs and provide inputs:
;;IN (1 2 3 4 5)
;;IN ()
#lang racket
(define (reverse-list lst)
(foldl
;;HIDE
cons '() lst)
;;EDIT ...
)
(let ((arg (read (current-input-port))))
(writeln (reverse-list arg)))
The heuristic is that a file that isn't a unit test and that doesn't contain (provide ...) should be run. Also, a file that provides a main function is run.
The version is 2.11 (which comes with Ubuntu 20.04).
In a CALL problem, the called function must be in an object whose name matches the file name.
//FORBIDDEN (^|\PL)for(\PL|$)
//Do not use "for"
//FORBIDDEN (^|\PL)while(\PL|$)
//Do not use "while"
//FORBIDDEN (^|\PL)var(\PL|$)
//Do not use "var"
//FORBIDDEN (^|\PL)def(\PL|$)
//Do not use "def"
object Problem {
/**
* Checks whether a sequence of integer values in increasing
* @aparam a the sequence (of length >= 1)
* @return true if a(i) < a(i+1) for all adjacent elements
*/
//CALL 1.to(100)
//CALL Vector(1, 4, 9, 16, 9, 36)
//CALL Vector(1, 4, 9, 16, 16, 36)
//CALL Vector(1)
val isIncreasing = (a: Seq[Int]) => a.dropRight(1).zip(a.drop(1)).forall(p => p._1 < p._2)
}
CodeCheck uses PolyML.
The file extension must be .sml. The most useful problem type is CALL. For example:
(* Complete the function below to compute the factorial of a nonnegative argument. *) (*HIDE*) (*CALL 3 *) (*CALL 0 *) fun fact 0 = 1 | fact n = n * fact(n - 1) (*EDIT fun fact ... *)
Separate multiple arguments with spaces:
(*CALL 3 4 *) (*CALL 0 2 *) fun sum a b = a + b
Provide parentheses with tuple arguments:
(*CALL (3, 4) *) (*CALL (0, 2) *) fun sum (a, b) = a + b
Substitutions also work:
(*HIDE*)
fun fact 0 = 1
| fact n = n * fact(n - 1)
(*EDIT fun fact ... *)
fun main() =
let
val arg = 3; (*SUB 0; 1; 6; 20 *)
val actual = fact arg
in
print (PolyML.makestring (actual))
end;
But that can be done more elegantly like this:
(*HIDE*) fun fact 0 = 1 | fact n = n * fact(n - 1) (*EDIT fun fact ... *) fun main() = print (PolyML.makestring (map fact [0, 1, 2, 6, 20]));
You can also use “Expected” style testers. Note that the tester must be a separate file.
(*HIDE*) fun fact 0 = 1 | fact n = n * fact(n - 1) (*EDIT fun fact ... *)
use "factorial.sml";
fun main() =
let
val actual = map fact [0, 1, 2, 6, 20]
in
print ((PolyML.makestring (actual)) ^ "\nExpected: [1, 1, 2, 720, 2432902008176640000]\n")
end;
The file extension is .sh, and pseudocomments start with ##. These tests are supported:
main.sh. tester.sh, tester1.sh, tester2.sh, and so on. A tester script invokes the student script, then echoes the expected output. Only use this if the student script output is a single line.CALL. Note that the arguments are separated by spaces.SUBUse ARGS to provide command line arguments. If the script is expected to produce a file, you can test its contents with the OUT directive.
You can add a premain.sh to create files and directories, set environment variables, and so on. Use ##HIDE if you don't want CodeCheck to display its contents. The file is prepended to the student's main.sh before execution.
The script is executed in a temporary directory that contains only the files that you supply and that the student submits.
If you don't want students to see those files, you can create a directory in premain.sh and change to it.
The temporary directory is named /tmp/codecheck/workXXXXXXXXXXXXXXXXXXXX, where the XXXXXXXXXXXXXXXXXXXX will differ between runs. If you need to compare absolute directory paths, remove the variable part.
The file extension is .rs. When using CALL and SUB, make sure the supplied values match the declared data type of the method or variable. No main function is required when using CALL.
Here is an example for CALL:
//CALL 3.0, 4.0
//CALL -3.0, 3.0
//CALL 3.0, 0.0
pub fn average(x: f32, y: f32) -> f32 {
//HIDE
return (x + y) / 2.0;
//EDIT
}
SUB also supports using arrays as substituted values. Here is an example for SUB:
fn main() {
let mut arr: [i32; 5]= [1,2,3,4,5]; //SUB [2,4,6,8,10]; [5,10,15,20,25]; [10,20,30,40,50]
// TODO: Double each element of arr
//HIDE
for n in 0..arr.len() {
arr[n] = arr[n] * 2;
}
//EDIT
println!("{:?}", arr);
}Caution: When using the “Expected” style testers, the function being tested must be declared public with the pub keyword. The tester file would not be able to access the function from the imported module otherwise.
An “Expected” style unit test file name must end in _tester.rs, _tester1.rs, _tester2.rs and so on.
For example:
mod numbers;
fn main() {
println!("{:?}", numbers::square(3));
println!("Expected: 9");
println!("{:?}", numbers::square(-3));
println!("Expected: 9");
println!("{:?}", numbers::square(0));
println!("Expected: 0");
}pub fn square(n: i32) -> i32 {
//HIDE
return n * n;
//EDIT
}Rust unit tests are supported. The unit test file name must end in _test.rs, _test1.rs, _test2.rs and so on. Unit test files do not require a main function. The function being tested also needs to be declared public. For example:
pub fn square(n: i32) -> i32 {
//HIDE
return n * n;
//EDIT
}
mod numbers;
#[cfg(test)]
mod test {
use super::numbers::square;
#[test]
fn test_non_negative_squares() {
for i in 0..100 {
assert_eq!(square(i), i * i);
}
}
#[test]
fn test_negative_squares() {
for i in -100..0 {
assert_eq!(square(i), i * i);
}
}
}
Here is how you use CALL tests:
module Max where --CALL [] --CALL [1, 2, 3] --CALL [4] maxNum :: [Integer] -> Integer --HIDE maxNum [] = error "empty list" maxNum (x:[]) = x maxNum (x:xs) = if x > largest then x else largest where largest = maxNum xs --EDIT
You can also run arbitrary programs:
maxNum :: [Integer] -> Integer --HIDE maxNum [] = error "empty list" maxNum (x:[]) = x maxNum (x:xs) = if x > largest then x else largest where largest = maxNum xs --EDIT main :: IO () main = do print $ maxNum [1,2,3] print $ maxNum [4] print $ maxNum []
test.in, test[1-9].in |
Standard input to be fed into running programs |
*Tester.{java,cpp,py,...}, *Tester[1-9].{java,cpp,py,...} |
An “Expected” style unit test |
*Test.{java,py,...}, *Test[1-9].{java,py,...} |
A unit test with the unit test framework supported with the given language |
index.html |
The problem description |
checkstyle.xml |
A CheckStyle configuration |
*.jar |
A JAR file with library code |
param.js |
Parameters for substitution in the problem description and source files |
CALL |
Call method with the supplied arguments |
SUB |
Substitute the given values for a variable |
ID |
The problem ID. This is how the download JAR file will be named. Default: The first file name. For example, if your solution has files Cat.java and Dog.java, the downloaded file will be Cat.jar. If you don't like that, put something like //ID hw2a into one of the solution files, and the download will be named hw2a.jar. |
HIDE |
Don't show the portion between HIDE and the next EDIT. If this is the first line in a file without EDIT, don't show the file at all. |
EDIT |
Marks a block that should be editable in a solution. If present, the rest of the solution is not editable. Can optionally be followed by text that is placed into the editable text area. |
HIDDEN |
By itself: Does not display the contents of this tester or unit test to the student. When following CALL or IN: Does not display the arguments or inputs to the student. |
ARGS |
The command line arguments to pass |
OUT |
The names of files (space separated) that the program is supposed to produce and that should be checked against the solution |
INTERLEAVE |
True by default, set to false to turn off interleaved input and output. Use this option to avoid timeouts with console I/O. |
TIMEOUT |
The timeout for this codecheck run in milliseconds (default 30000) |
MAXOUTPUTLEN |
The maximum length of the output for this codecheck run (default 100000) |
TOLERANCE |
The tolerance to use when comparing numbers (default 1E-6) |
IGNORECASE |
True by default, set to false if you want comparisons to be case sensitive or space sensitive |
REQUIRED |
A regular expression that must be, or cannot be, present in the student's file. If there is a second comment line below it, its contents is displayed if the student's program fails the test. |
TILE |
For Parsons puzzles. Turns the file into a set of draggable tiles. |