CSCI 141 - POTD and Pytest Primer

In this class, you will complete a daily Program of the Day (POTD) assignment to build your coding skills. These short programs are graded using automated tests - another program that runs your program and checks whether its behavior is correct. We provide you with this test program so you can check your work. This way, you will know when you have arrived at a correct and working solution to the POTD.

This guide walks you through:

Where to find the problem prompt, skeleton (starter) code, and test program for each POTD.
How to use the test program to check your solution for correctness.

Where to find the POTD

The prompt for each POTD is linked from the Schedule table on the course webpage; these are labeled P##, where ## is the number of the lecture when the program is assigned. The program is due before the following class (not counting exam days).

As an example, here’s the link to the first POTD prompt: P01. Throughout this document, we’ll use P01 as a running example, but the process applies for all POTDs.

You can find the skeleton code and test program for each POTD at this url:

https://github.com/csci141/POTD/tree/2520_wehrwein/skel.

This link can also be found in the Quick Links above the Schedule table on the course webpage. On this page, you’ll see a list of files. All the files that pertain to P01 have filenames with the prefix P01_. Usually, you’ll find a skeleton file (P01_quiz1.py) and a test program (P01_quiz1_test.py).

To download the files, click on the file you’d like to download; you’ll see a preview of the file in a pane. At the top right, click the “Download raw file” button to download the file to your computer.

Download the skeleton, test program, and any others with the corresponding prefix, to the lab computer or personal computer you’re working on. It’s strongly recommended to you set up a folder to hold all your code for this class. You may want to create a subfolder for each POTD. Move the files from your downloads folder (or wherever they landed) into your CSCI 141 folder.

How to run the Test Program

One-time Setup

You may need to install the pytest package, which our test code relies on. In Thonny:

Go to the ‘Tools’ menu and select ‘Install Packages’.
Type ‘pytest’ in the search bar at the top and click ‘Search on PyPi’.
Pytest should come up in the results - click on it and then on the ‘Install’ button. Now you have Pytest!

Running the Test Program

Make sure that both files (P01_quiz1.py and P01_quiz1_test.py) are in the same directory, and that there are no other python files in this directory. If they are not in the same folder or other python files are present in that folder, the test program will may run correctly.

Open the test program (P01_quiz1_test.py) in Thonny and press the friendly green run button to run the tests on your solution.

If your program passes the tests, you’ll see an output like this (there will be some other output prior to this that you can safely ignore):


P01_quiz1_test.py::test_quiz1 PASSED                                     [100%]

============================== 1 passed in 0.09s ===============================

What’s in a test program?

Consider this test program for P01:

#POTD 1 test
import pytest
import subprocess
import sys

basename = "P01_quiz1"


def test_quiz1():
    process = subprocess.Popen(
        [sys.executable, "P01_quiz1.py"],
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True
    )
    #When prompted for input, enter newline
    stdout, stderr = process.communicate(input="\n")

    #verify that the input prompt had 'What if' in the text string (capitalization and spacing matters!).
    assert "What is" in stdout

pytest.main(["P01_quiz1_test.py",  "-vv", "--showlocals", "-p", "no:faulthandler"])

There’s plenty of unfamiliar stuff here, and that’s fine; but notice the assert statement near the bottom: this indicates what the program is testing for. In this case, it is looking for the string "What is" to be printed to the screen as part of a user input request (the variable stdout holds a string with the program’s output).

Here we’re not testing the fully specified functionality of the program, but in general the tests will be more comprehensive. You should always make sure your code’s output matches the specified format exactly. If it doesn’t match exactly (e.g., in this case if you’d printed What's instead of What is), the program will fail tests.

Interpreting Test Failures

I rarely write working code on the first try. If your code isn’t passing the tests, the test program provides output that can be helpful in understanding why. Understanding this output can be intimidating at first, but it gets easier as you get used to knowing what to look for.

Let’s run with the example above, where I wrote a P01 solution that prints What's 4 * 6? instead of What is 4 * 6?. Running the test program, I see this output:

=================================== FAILURES ===================================
__________________________________ test_quiz1 __________________________________

    def test_quiz1():
        process = subprocess.Popen(
            [sys.executable, "P01_quiz1.py"],
            stdin=subprocess.PIPE,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            text=True
        )
    
        #When prompted for input, enter newline
        stdout, stderr = process.communicate(input="\n")
    
        #verify that the input prompt had 'What if' in the text string (capitalization and spacing matters!).
>       assert "What is" in stdout
E       assert 'What is' in "What's 4 * 6?\n4 * 6 is 24\n"

process    = <Popen: returncode: 0 args: ['/Applications/Thonny.app/Contents/Frameworks/P...>
stderr     = ''
stdout     = "What's 4 * 6?\n4 * 6 is 24\n"

P01_quiz1_test.py:22: AssertionError
=========================== short test summary info ============================
FAILED P01_quiz1_test.py::test_quiz1 - assert 'What is' in "What's 4 * 6?\n4 * 6 is 24\n"
============================== 1 failed in 0.10s ===============================

This is a lot of output! Let’s look carefully at a few parts:

The first section of the ouptut (after the header line test_quiz1) shows the code of the test program with an arrow (>) pointing at the line where the assertion failed, causing the test to fail. Here, it’s the line assert "What is" in stdout.
More details are provided on the line below: the line beginning with E shows the string that the program expected to find (What is) and the string it expected to find it in (in this case, the program’s output, stored in stdout):
```
"What's 4 * 6?\n4 * 6 is 24\n"
```
With experience, looking carefully at this will immediately clue you in to the bug: the output isn’t as expected.
On the line before the short test summary info line, it shows the test file and line number (P01_quiz1_test.py:22) where the error happened, as well - we can also use this to locate the assertion that failed on line 22 of that file.

Types of Errors

You may encounter a variety of errors that cause tests to fail. Here’s a little more detail about the most common ones.

When encountering an error, you may find it helpful to switch back to running your program “manually” with inputs similar to what the test program is using - this may make it easier to catch bugs. However, understanding what cases the test program is checking can be helpful in narrowing down the source of the error.

Assert Error: Failed to meet the requirements of the test

This is the vaguest error. This will frequently happen when your code does not meet the requirements. It could mean that your code is not returning or outputting the correct thing at all. The test program is case sensitive and cares about all commas and periods. Make sure that it has all of those. Review the exact output of the test program and check to see if your program’s output matches the exact output the program is looking for. This error is likely to occur because your code does not match the exact syntax of the output. Make sure to go and review the exact syntax of the output in the test program. There are POTDs where you will be expected to exactly match a program with newlines and placement of print statements.

Type Error: Code attempts to do an operation on a type that doesn’t support that operation

In this case, the program threw a type-related error while running. This is usually because you tried to do an operation that isn’t valid, such as divide a string by a number. The fix is usually making sure that you’ve done the proper type conversions before performing the operations.

Runtime Error: Code ran into a problem during its runtime

This is a more generic kind of error for other reasons your program may have thrown an error while running. The test program will output the error message, which can help you understand the reason for the error.

Indentation Error or Syntax error: The program is not syntactically correct

These errors means that you have some syntax-related error, such as incorrect indentation, a prohibited variable name, unmatched parentheses, etc. The test program will output the error message, which can help you understand the reason for the error.