We propose a novel vertical Bell Laboratories layered space-time (V-BLAST) based two-way relaying protocol that comprises two user terminals and one multi-antenna demodulate-and-forward (DMF) relay station. We consider the situation when these two user terminals employ different modulation types and investigate its detection techniques. The key feature of the detection techniques is that we quantify decision reliability at the relay station by introducing a so-called normalized correlation coefficient (CORR), rather than having full confidence in demodulation at the relay station. With the assistance of the normalized correlation coefficient, we develop detection methods at both the relay station and user terminals. In particular, we propose a CORR-based ordering algorithm at the relay station which can be applied to detection for general V-BLAST architectures. Simulation results demonstrate the superiority of the CORR-based ordering algorithm over the conventional signal-to-noise based ordering algorithm. Finally, we demonstrate overall error-performance of the V-BLAST-based DMF two-way relaying protocol via simulations.