We demonstrate an all-fiber high-power linearly polarized supercontinuum source with polarization-maintaining photonic crystal fibers (PM-PCFs) as the nonlinear medium. The source exhibits an average output power of 3.8 W with a flat spectrum from 480 nm to 2100 nm at the $-$10 dB level, except for the residual pump peak. The polarization extinction ratio (PER) is measured to be greater than 20 dB at selected sample wavelength points (532 nm, 1064 nm and 1550 nm) at the highest pump power level and greater than 20 dB at all wavelengths from 800 nm to 1500 nm at the low pump power level. We also experimentally study the spectral properties when the pump light propagates along different axes of the PM-PCF. The results show that propagating parallel to the slow axis enables a broader spectrum in the PM-PCF in this case, probably due to matching of the dispersion properties with the pump light, which is qualitatively in accordance with the numerical simulation. To our best knowledge, this is the first demonstration of a watt-level linearly polarized supercontinuum source generated from PM-PCFs in an all-fiber structure.