This study presents a danger estimation system to prevent accidents among infants. A video camera positioned above the infant’s crib captures video. The proposed system can monitor the behavior of infants aged zero to six months. If there is a change in behavior or any other unusual occurrence, the system alerts the person responsible to attend to the baby immediately. The proposed system operates in three phases, which are foreground color model (FC model) construction, infant detection, and degree of danger analysis. During FC model construction, the foreground color histogram is created iteratively; the background image does not have to be constructed first. A motion-history image (MHI) is also obtained based on the motion of the infant. The color and motion information supplied by the FC model and the MHI are combined to detect the infant, who is regarded as the foreground object in the input frame. Moreover, six features of infant behavior are extracted from the detected infant to measure the degree of danger faced by the infant, and the result is used to warn the baby-sitter if needed. Experimental results show that the proposed method is robust and efficient. This study presents a danger estimation system to prevent accidents among infants. A video camera positioned above the infant’s crib captures video. The proposed system can monitor the behavior of infants aged zero to six months. If there is a change in behavior or any other unusual occurrence, the system alerts the person alert to attend to the baby immediately. The proposed system operates in three phases, which are foreground color model (FC model) construction, infant detection, and degree of danger analysis. During FC model construction, the foreground The color histogram is created iteratively; the background image does not have to be constructed first. A motion-history image (MHI) is also obtained on the motion of the infant. The color and motion information supplied by the FC model and the MHI are combined to detect the infant, who is viewed as the foreground object in the input frame. Moreover, six features of infant behavior are extracted from the detected infant to measure the degree of danger faced by the infant, and the result is used to warn the baby-sitter if needed. Experimental results show that the proposed method is robust and efficient.