A Sample-based Criterion for Unsupervised Learning of Complex Models beyond Maximum Likelihood and Density Estimation
DOI:
https://doi.org/10.18034/abcjar.v5i2.581Keywords:
Unsupervised learning, Maximum likelihood, Gaussian mixtures, Boltzmann machinesAbstract
Many unsupervised learning processes have the purpose of aligning two probability distributions. Recoding models like ICA and projection pursuit, as well as generative models like Gaussian mixtures and Boltzmann machines, can be seen in this perspective. For these types of models, we offer a new sample-based error measure that can be used even when maximum likelihood (ML) and probability density estimation-based formulations can't be used, such as when the posteriors are nonlinear or intractable. Furthermore, the challenges of approximating a density function are avoided by our sample-based error measure. We show that with an unconstrained model, (1) our technique converges on the correct solution as the number of samples increases to infinity, and (2) our approach's predicted answer in the generative framework is the ML solution. Finally, simulations of linear and nonlinear models on mixtures of Gaussians and ICA issues are used to evaluate our approach. Our method's applicability and generality are demonstrated by the experiments.
Downloads
References
Bengio, Y., Courville, A. C. and Vincent, P. 2012. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR.
Bynagari, N. B. (2014). Integrated Reasoning Engine for Code Clone Detection. ABC Journal of Advanced Research, 3(2), 143-152. https://doi.org/10.18034/abcjar.v3i2.575
Bynagari, N. B. (2015). Machine Learning and Artificial Intelligence in Online Fake Transaction Alerting. Engineering International, 3(2), 115-126. https://doi.org/10.18034/ei.v3i2.566
Bynagari, N. B. (2016). Industrial Application of Internet of Things. Asia Pacific Journal of Energy and Environment, 3(2), 75-82. https://doi.org/10.18034/apjee.v3i2.576
Dayan, P., Hinton, G. E., Neal, R. M. and Zemel. R. S. 1995. The Helmholtz machine. Neural Computation, 7(5):889-904.
Doersch, C., Gupta, A. and Efros. A. A. 2015. Unsupervised visual representation learning by context prediction. ICCV.
Donepudi, P. K. (2014a). Technology Growth in Shipping Industry: An Overview. American Journal of Trade and Policy, 1(3), 137-142. https://doi.org/10.18034/ajtp.v1i3.503
Donepudi, P. K. (2014b). Voice Search Technology: An Overview. Engineering International, 2(2), 91-102. https://doi.org/10.18034/ei.v2i2.502
Donepudi, P. K. (2015). Crossing Point of Artificial Intelligence in Cybersecurity. American Journal of Trade and Policy, 2(3), 121-128. https://doi.org/10.18034/ajtp.v2i3.493
Donepudi, P. K. (2016). Influence of Cloud Computing in Business: Are They Robust?. Asian Journal of Applied Science and Engineering, 5(3), 193-196. Retrieved from https://journals.abc.us.org/index.php/ajase/article/view/1181
Duda R. O. and Hart, P. E. 1973. Pattern Classification and Scene Analysis. Wiley.
Erdogmus D. and Principe J. C. 2000. Comparision of entropy and mean square error criteria in adaptive system training using higher order statistics. In P. Pajunen and J. Karhunen, editors, Proceedings of the Second International Workshop on Independent Component Analysis and Blind Signal Separation, Helsinki, Finland, pages 75-80. Otamedia, Espoo, Finland, ISBN: 951-22-5017-9, 2000.
Everitt. B. S. 1984. An introduction to latent variable models. Chapman and Hall.
Ghahramani Z. and Hinton G. E. 1996. The EM algorithm for mixtures offactor analyzers. Technical Report CRG-TR-96-1 , University of Toronto, Dept. ofComp. Science, 1996.
Ghahramani Z. and Hinton, G. E. 1998. Hierachical non-linear factor analysis and topographic maps. In M. I. Jordan, M. J. Kearns, and S. A. Solla, editors, Advances in Neural Information Processing Systems 10, pages 486- 492. MIT Press, 1998.
Gray A. and Moore A. W. 2001. 'N-body' problems in statistical learning. In T. K.Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, 2001. In this proceeding.
Hinton G. E. and Sejnowski T. J. 1999. Introduction. In G. E. Hinton and T. J. Sejnowski, editors, Unsupervised Learning: Foundations of Neural Computation, pages VII- XVI. The MIT Press, Cambridge, MA, London, England.
Hinton G. E. and. Sejnowski T. J. 1986. Learning and relearning in Boltzmann machines. In Parallel Distributed Processing, volume 1, pages 282- 317. MIT Press, 1986.
Hochreiter S. and Mozer M. C. 2000. An electric field approach to independent component analysis. In P. Pajunen and J. Karhunen, editors, Proceedings of the Second International Workshop on Independent Component Analysis and Blind Signal Separation, Helsinki, Finland, pages 45- 50. Otamedia, Finland, ISBN: 951-22-5017-9.
Huffman. D. 1952. A method for the construction of minimum-redundancy codes. Proceedings of the I.R.E., pp. 1098-1102.
Hyviirinen A. 1999. Survey on independent component analysis. Neural Computing Surveys, 2:94- 128.
Manavalan, M. (2014). Fast Model-based Protein Homology Discovery without Alignment. Asia Pacific Journal of Energy and Environment, 1(2), 169-184. https://doi.org/10.18034/apjee.v1i2.580
Manavalan, M. (2016). Biclustering of Omics Data using Rectified Factor Networks. International Journal of Reciprocal Symmetry and Physical Sciences, 3, 1–10. Retrieved from https://upright.pub/index.php/ijrsps/article/view/40
Manavalan, M., & Bynagari, N. B. (2015). A Single Long Short-Term Memory Network can Predict Rainfall-Runoff at Multiple Timescales. International Journal of Reciprocal Symmetry and Physical Sciences, 2, 1–7. Retrieved from https://upright.pub/index.php/ijrsps/article/view/39
Manavalan, M., & Bynagari, N. B. (2015). A Single Long Short-Term Memory Network can Predict Rainfall-Runoff at Multiple Timescales. International Journal of Reciprocal Symmetry and Physical Sciences, 2, 1–7. Retrieved from https://upright.pub/index.php/ijrsps/article/view/39
Manavalan, M., & Ganapathy, A. (2014). Reinforcement Learning in Robotics. Engineering International, 2(2), 113-124. https://doi.org/10.18034/ei.v2i2.572
Marques G. C. and Almeida L. B. 1999. Separation of nonlinear mixtures using pattern repulsion. In J.-F. Cardoso, C. Jutten, and P. Loubaton, editors, Proceedings of the First International Workshop on Independent Component Analysis and Signal Separation, Aussois, France, pages 277- 282.
Noroozi M. and Favaro, P. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. CoRR.
Principe J. C. and Xu D. Information-theoretic learning using Renyi's quadratic entropy. In J.-F. Cardoso, C. Jutten, and P. Loubaton, editors, Proceedings of the First International Workshop on Independent Component Analysis and Signal Separation, Aussois, France, pages 407-412.
Shannon C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J., vol 27, pp. 379-423, Jul. 1948.
Shannon, C. E. 1951. Prediction and entropy of printed english. Bell Syst. Tech. J., vol 30, pp. 35-4, Jan. 1951, 1951.
Zhao Y. and Atkeson C. G. 1996. Implementing projection pursuit learning. IEEE Transactions on Neural Networks, 7(2):362- 373.
--0--
Downloads
Published
Issue
Section
License
How to Cite
