NLP Terms

Bag of Words Model
TF-IDF
Vector Space Model
Standard Boolean Model

Why does mean normalization help in gradient descent?

Essentially, scaling the inputs (through mean normalization, or z-score) gives the error surface a more spherical shape, where it would otherwise be a very high curvature ellipse. Since gradient descent is curvature-ignorant, having an error surface with high curvature will mean that we take many steps which aren't necessarily in the optimal direction. When we scale the inputs, we reduce the curvature, which makes methods that ignore curvature (like gradient descent) work much better. When the error surface is circular (spherical), the gradient points right at the minimum, so learning is easy.

Feature Scaling

Three Types:


http://stats.stackexchange.com/questions/70553/how-to-verify-a-distribution-is-normalized/70555#70555


Centered:  X - mean

Standardized:  X - mean / sd
Standardize features by removing the mean and scaling to unit variance

Normalized: X - mean / max - min


http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.

https://en.wikipedia.org/wiki/Feature_scaling

Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the majority of classifiers calculate the distance between two points by the Euclidean distance. If one of the features has a broad range of values, the distance will be governed by this particular feature. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance.

Another reason why feature scaling is applied is that gradient descent converges much faster with feature scaling than without it^{[citation needed]}.

http://stats.stackexchange.com/questions/29781/when-conducting-multiple-regression-when-should-you-center-your-predictor-varia

http://stats.stackexchange.com/questions/86434/is-standardisation-before-lasso-really-necessary

Lasso regression puts constraints on the size of the coefficients associated to each variable. However, this value will depend on the magnitude of each variable. It is therefore necessary to center and reduce, or standardize, the variables.

The result of centering the variables means that there is no longer an intercept. This applies equally to ridge regression, by the way.

Another good explanation is this post: Need for centering and standardizing data in regression

Need Normalization Before PCA

Normalization is important in PCA since it is a variance maximizing exercise. It projects your original data onto directions which maximize the variance. The first plot below shows the amount of total variance explained in the different principal components wher we have not normalized the data. As you can see it seems like only component one explains all the variance in the data.

If you look at the second picture we have normalized the data first. Here it is clear that the other components contribute as well. The reason for this is because PCA seeks to maximize the variance of each component. And since the covariance matrix of this particular dataset is:

             Murder   Assault   UrbanPop      Rape
Murder    18.970465  291.0624   4.386204  22.99141
Assault  291.062367 6945.1657 312.275102 519.26906
UrbanPop   4.386204  312.2751 209.518776  55.76808
Rape      22.991412  519.2691  55.768082  87.72916

From this structure the PCA will of course select to project as much as possible in the direction of Assault since that variance is much greater. So for finding features usable for any kind of model a PCA without normalization would perform badly.

shareeditflag

Log & Exp function

e = 2.71828

Log Loss

Log-loss is an appropriate performance measure when you're model output is the probability of a binary outcome.

The log-loss measure considers confidence of the prediction when assessing how to penalize incorrect classification. For instance consider two predictions of an outcome P(Y=1|X), where the predictions are 0.51 and 0.99 respectively. In the former case the model is only slightly confident of the class prediction (assuming a 0.5 cutoff), while in the latter it is extremely confident. Since in our case both are wrong, the penalty will be more harsh for the more confident (but incorrect) prediction by employing a log-loss penalty.

I just installed GTX 970 into my Ubuntu box. Using the following command shows the card is connected to the system.

~$ lspci -nnk |grep NVIDIA 60:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) 60:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

But when I ran my Keras script using TensorFlow as the backend, it didn't seem to speed up the training - each epoch still takes a long time to run.

Is there any way to check if Keras is using GPUs?

You need to install the cuda lib for tensorflow to use GPUhttps://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation

If it is installed correctly, everytime you run tf or keras with tf backend, it will show something like "using Cuda library..."