TechTalks from event: Other ICML 2012 Tutorials

  • Probabilistic Topic Models Authors: David M. Blei, Princeton University
    Much of my research is in topic models, which are a suite of algorithms to uncover the hidden thematic structure of a collection of documents. These algorithms help us develop new ways to search, browse and summarize large archives of texts.
  • Mirror Descent Algorithms for Large-Scale Convex Optimization Authors: Arkadi Nemirovski, Georgia Institute of Technology
    Mirror Descent is a technique for solving nonsmooth problems with convex structure, primarily, convex minimization and convex-concave saddle point problems. Mirror Descent utilizes first order information on the problem and is a far-reaching extension of the classical Subgradient Descent algorithm (N. Shor, 1967). This technique allows to adjust, to some extent, the algorithms to the geometry of the problem at hand and under favorable circumstances results in nearly dimension-independent and unimprovable in the large scale case convergence rates. As a result, in some important cases (e.g., when solving large-scale deterministic and stochastic convex problems on the domains like Euclidean/$\ell_1$/nuclear norm balls), Mirror Descent algorithms become the methods of choice when low and medium accuracy solutions are sought. In the tutorial, we outline the basic Mirror Descent theory for deterministic and stochastic convex minimization and convex-concave saddle point problems, including recent developments aimed at accelerating MD algorithms by utilizing problem's structure.
  • Deep Learning Tutorial: Representation Learning Authors: Yoshua Bengio, U. Montreal
    The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data.  Although domain knowledge can be used to help design representations, learning can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms. We view the ultimate goal of these algorithms as disentangling the unknown underlying factors of variation that explain the observed data.This tutorial reviews the basics of feature learning and deep learning, as well as recent work relating these subjects to probabilistic modeling and manifold learning.  An objective is to raise questions and issues about the appropriate objectives for learning good representations, for computing representations (i.e., inference), and the geometrical connections between representation learning, density estimation and manifold learning.

    Outline:
    1. Motivations and Scope
      • Feature / Representation learning
      • Distributed representations
      • Exploiting unlabeled data
      • Deep representations
      • Multi-task / Transfer learning
      • Invariance vs Disentangling
    2. Algorithms
      • Probabilistic models and RBM variants
      • Auto-encoder variants (sparse, denoising, contractive)
      • Explaining away, sparse coding and Predictive Sparse Decomposition
      • Deep variants
    3. Analysis, Issues and Practice
      • Tips and tricks
      • Partition function gradient
      • Inference
      • Mixing between modes
      • Geometry and probabilistic interpretations of auto-encoders
      • Open questions