With the development of PC, internet as well as mobile devices, we are facing a data exploding era. On one hand, more and more features can be collected to describe the data, making the size of the data descriptor larger and larger. On the other hand, the number of data itself explodes and can be collected from multiple resources. When the data becomes large scale, the traditional data analysis method may fail, suffering the curse of dimensionality and etc. In order to explore and analyze the large-scale data more accurately and more efficiently, based on the characteristic of the data, we propose several learning algorithms to mine the Heterogeneous data. To be specific, if the feature dimension is large, we propose several sparse learning based feature selection methods to select the key words from the text or to find the bio-marker from the gene expression data; if the number of data itself is huge, we proposed multi-view K-Means method to do the clustering to avoid the heavy graph construction burden; if the data is represented or collected by multiple resources, we propose graph based multi-modality model to do semi-supervised learning and clustering. In addition, if the number of classes is large, we provides a global solution to the low-rank regression and proves that the low-rank regression is equivalent to doing linear regression in LDA space. We empirically evaluate each of our proposed models on several benchmark data sets and our methods can consistently achieve superior results with the comparison of state-of-art methods. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com.bibliotheek.ehb.be/en-US/products/dissertations/individuals.shtml.]