A semantic unit based event detection scheme in soccer videos is proposed in this paper.The scheme can be characterized as a three-layer framework. At the lowest layer, low-level features including color, texture, edge, shape, and motion are extracted. High-level semantic events are defined at the highest layer. In order to connect low-level features and high-level semantics, we design and define some semantic units at the intermediate layer. A semantic unit is composed of a sequence of consecutives frames with the same cue that is deduced from low-level features. Based on semantic units, a Bayesian network is used to reason the probabilities of events. The experiments for shoot and card event detection in soccer videos show that the proposed method has an encouraging performance.