Depressive symptoms are common among cancer patients. They are associated with a poorer prognosis and often go unnoticed. The Predicting Depression Project, conducted in collaboration with the Boussard Lab at the Stanford University School of Medicine, aimed to predict the onset of depressive symptoms in cancer patients. Natural Language Processing techniques were applied to identify depressive content in patient emails and physician notes. A multimodal model was developed to combine features from open text and structured data into a risk score for the onset of depressive symptoms within a month after starting cancer treatment. During this talk, we will focus on some of the distinctive challenges we encountered during this project, like the low inter-rater reliability of our labellers and the subjective nature of our outcome of interest.