In this study, we attempted to quantify indicators of novice programmer progress in the task of writing programs, and we evaluated the use of these indicators for identifying academically at-risk students. Over the course of nine weeks, students completed five different graded programming exercises in a computer lab. Using an instrumented version of BlueJ, an integrated development environment for Java, we collected novice compilations and explored the errors novices encountered, the locations of these errors, and the frequency with which novices compiled their programs. We identified which frequently encountered errors and which compilation behaviors were characteristic of at-risk students. Based on these findings, we developed linear regression models that allowed prediction of students' scores on a midterm exam. However, the models derived could not accurately predict the at-risk students. Although our goal of identifying at-risk students was not attained, we have gained insights regarding the compilation behavior of our students, which may help us identify students who are in need of intervention.