Laplace Prior and Sparsity

Sparse parameter estimation can be achieved in the Bayesian framework by using the Laplace (double exponential) prior distribution. It is known that the MAP estimates using the Laplace prior are the same as those produced by applying the lasso algorithm that minimizes the usual sum of squared errors, with a bound on the sum of the absolute values of the coefficients.

The Bayesian Logistic Regression (BBR) and Bayesian Mutinomial Regression (BMR) software packages have implemented this idea and showed good performance in text classification.

Challenges in Statistical Machine Learning

John Lafferty and Larry Wasserman have recently published an article talking about the Challenges in Statistical Machine Learning. The following problems are considered current challenges that cut across both statistics and machine learning communities.

• Sparse Learning in High Dimensions


• Semi-Supervised Learning


• Relation between Computation and Risk


• Structured Prediction

e^x and e^ix

When x -> infinity, e^x -> infinity, but in contrast e^ix is always bounded to the unit circle according to Euler's formula. This clever trick is said to be an important step that finally led to the discovery of Fourier series.

Rank Correlation Coefficients

To measure the correlation between two rankings rather than two random variables, we can consider using the following non-parametric correlation coefficients.
Spearman's ρ
Kendall's τ

Hellinger metric

Hellinger metric is the square-root of Jensen–Shannon divergence, a symmetric version of Kullback-Leibler divergence. It looks a good choice when measuring the dissimilarity between two distributions (e.g., two multinomial distributions corresponding to a pair of documents), because it is known to be a legitimate distance metric.