PeerWise (PW) is an online tool that allows students in a course to collaborate and learn by creating, sharing, answering and discussing multiple-choice questions (MCQs). Previous studies of PW at the introductory level have shown that students in computing courses like it, and report statistically significant learning gains in courses taught by the investigators at different institutions. However, we recently conducted three quasi-experimental studies of PW use in upper-division computing courses in the U.S. and failed to replicate these positive results. In this paper we consider various factors that may impact the effectiveness of PW, including instructor engagement, usage requirements and subject-matter issues. We also report several positive results from other STEM courses at the same institution, discuss methodological issues pertaining to our recent studies and propose approaches for further investigation.