The local recurrence rate of phylodes tumors of the breast varies widely among different subtypes, and distant metastasis is associated with poor survival. This study aimed to identify factors that are predictive of local recurrence-free survival (LRFS), distant metastasis-free survival (DMFS), and overal survival (OS) in patients with phyllodes tumors of the breast. Clinical data of all patients with a phyllodes tumor of the breast (n = 192) treated at Sun Yat-sen University Cancer Center between March 1997 and December 2012 were reviewed. The Pearsonχ2 test was used to investigate the relationship between clinical features of patients and histotypes of tumors. Univariate and multivariate Cox regression analyses were performed to identify factors that are predictive of LRFS, DMFS, and OS. In total, 31 (16.1%) patients developed local recurrence, and 12 (6.3%) developed distant metastasis. For the patients who developed local recurrence, the median age at the diagnosis of primary tumor was 33 years (range, 17-56 years), and the median size of primary tumor was 6.0 cm (range, 0.8-18 cm). For patients who developed distant metastasis, the median age at the diagnosis of primary tumor was 46 years (range, 24-68 years), and the median size of primary tumor was 5.0 cm (range, 0.8-18 cm). In univariate analysis, age, size, hemorrhage, and margin status were found to be predictive factors for LRFS (P = 0.009, 0.024, 0.004, and 0.001, respectively), whereas histotype, epithelial hyperplasia, margin status, and local recurrence were predictors of DMFS (P = 0.001, 0.007, 0.007, and < 0.001, respectively). In multivariate analysis, independent prognostic factors for LRFS included age [hazard ratio (HR) = 3.045,P = 0.005], tumor size (HR = 2.668,P = 0.013), histotype (HR = 1.715,P = 0.017), and margin status (HR = 4.530,P< 0.001). Histotype (DMFS: HR = 4.409, P = 0.002; OS: HR = 4.194,P = 0.003) and margin status (DMFS: HR = 2.581,P = 0.013; OS: HR = 2.507, P = 0.020) were independent predictors of both DMFS and OS. In this cohort, younger age, a larger tumor size, a higher tumor grade, and positive margins were associated with lower rates of LRFS. Histotype and margin status were found to be independent predictors of DMFS and OS.