Bounding Generalization Error in ReLU Networks

  This post will discuss bounds on generalization error of ReLU networks proposed in [1]. The theorem proved in this paper, depends on assumptions made on the structure of the networks under consideration. Before discussing these assumptions, let us look at some definitions and notations that will be necessary to understand what is happening in… Continue reading Bounding Generalization Error in ReLU Networks

Information Theoretic Perspective on Deep Neural Networks Through the Information Bottleneck Method

1. Introduction In this blog post we will investigate the well acknowledged theoretical problem of what deep neural networks are performing with the data from a theoretical perspective that would explain their surprising performance in a vast range of different machine learning tasks. Particularly, we will be focusing on a very recent and, seemingly controversial,… Continue reading Information Theoretic Perspective on Deep Neural Networks Through the Information Bottleneck Method

Wasserstein Generative Adversarial Networks

Arjovsky et al. [1] proposed a new framework to suppress the gradient vanishing issue in the traditional Generative Adversarial Networks (GAN) [2]. A distance measure, Earth-Mover (EM) or Wasserstein distance, is utilized to guarantee continuous and differentiable gradient of objective function during training. This blog is organized as follows. Section 1 introduces the background knowledge… Continue reading Wasserstein Generative Adversarial Networks