Uganda suffered four Ebola and five Marburg virus outbreaks from 2000 to 2012 with significant health worker mortality. This paper describes findings from 41 interviews with health workers from three outbreaks. Interviewees frequently encountered stigma from their communities, sometimes accompanied by mistrust and violence. These difficulties were defined as ‘challenges of society.’ Health workers also suffered emotional trauma, depressive symptoms, and fear classified as ‘challenges of psyche.’