In this paper, we present a novel algorithm for simultaneously detecting shot boundaries and extracting key frames from video sequences or streams in real-time. Multivariate feature vectors are extracted from the video frames and arranged in a feature matrix. Singular value decomposition is then used to factorize the feature matrix and compute the significant singular vectors. The rank of the singular vectors is traced using a sliding window approach. By tracing the computed rank, we are able to determine shot boundaries and extract key frames of the video. Results of experiments conducted on soccer videos show that the algorithm is robust to a wide range of digital effects used during shot transition. Moreover, the algorithm is shown to run in real time making it suitable for online video summarization and multimedia networking applications.