Analysis and prediction of structural instabilities in open pit mines are an important design and operational consideration for ensuring safety and productivity of the operation. Unstable wedges and blocks occurring at the surface of the pit walls may be identified through three-dimensional(3D) image analysis combined with the discrete fracture network(DFN) approach. Kinematic analysis based on polyhedral modelling can be used for first pass analysis but cannot capture composite failure mechanisms involving both structurally controlled and rock mass progressive failures. A methodology is proposed in this paper to overcome such limitations by coupling DFN models with geomechanical simulations based on the discrete element method(DEM). Further, high resolution photogrammetric data are used to identify valid model scenarios. An identified wedge failure that occurred in an Australian coal mine is used to validate the methodology. In this particular case, the failure surface was induced as a result of the rock mass progressive failure that developed from the toe of the structure inside the intact rock matrix. Analysis has been undertaken to determine in what scenarios the measured and predicted failure surfaces can be used to calibrate strength parameters in the model. Analysis and prediction of structural instabilities in open pit mines are an important design and operational consideration for ensuring safety and productivity of the operation. Unstable wedges and blocks occurring at the surface of the pit walls may be identified through three-dimensional (3D) image analysis combined with the discrete fracture network (DFN) approach. Kinematic analysis based on polyhedral modeling can be used for first pass analysis but can not capture composite failure mechanisms involving both structurally controlled and rock mass progressive failures. A methodology is proposed in this paper to overcome such limitations by coupling DFN models with geomechanical simulations based on the discrete element method (DEM). Further, high resolution photogrammetric data are used to identify valid model scenarios. An identified wedge failure that occurred in an Australian coal mine is used to validate the methodology. In this particular case, the failure surface was induced as a result of the rock mass progressive failure that developed from the toe of the structure inside the intact rock matrix. Analysis has been undertaken to determine in what scenarios the measured and predicted failure surfaces can be used to calibrate strength parameters in the model.