Medium Post Read

https://towardsdatascience.com/plotting-text-and-image-vectors-using-t-sne-d0e43e55d89

Linear Regression

As such, linear regression was developed in the field of statistics and is studied as a model for understanding the relationship between input and output numerical variables, but has been borrowed bymachine learning. It is both a statistical algorithm and a machine learning algorithm

Regression Correlation vs Casuality

Multiple regression, like all statistical techniques based on correlation, has a severe limitation due to the fact that correlation doesn't prove causation.

Regression analysis is widely used for prediction and forecasting, where its use has substantial overlap with the field of machine learning. Regression analysis is also used to understand which among the independent variables are related to the dependent variable, and to explore the forms of these relationships. In restricted circumstances, regression analysis can be used to infer causal relationships between the independent and dependent variables. However this can lead to illusions or false relationships, so caution is advisable;[2] for example, correlation does not imply causation.

https://www.ma.utexas.edu/users/mks/statmistakes/causality.html

Age -> Shoe size
Age -> Reading score
Shoe size and reading score correlated

Not casuality variable, but can be actionable variable

For marketing, we can't find age due to privacy. But we can use shoe size to find kids with higher reading score
============================================
Conversion rate study

Search Engine vs Recommendation Engine

Recommendation engine is a search engine without user intent.

With a search engine, customers find products through an active search, assuming customers know what they want and how to describe it when forming their search query.

Recommendation engines proactively identify products that have a high probabalility of being something the consumer might want to buy. Each time customers 

Why SVD?

Singular value decomposition (SVD) is not the same as reducing the dimensionality of the data. It is a method of decomposing a matrix into other matrices that has lots of wonderful properties which I won't go into here. For more on SVD, see the Wikipedia page.

Reducing the dimensionality of your data is sometimes very useful. It may be that you have a lot more variables than observations; this is not uncommon in genomic work. It may be that we have several variables that are very highly correlated, e.g., when they are heavily influenced by a small number of underlying factors, and we wish to recover some approximation to the underlying factors. Dimensionality-reducing techniques such as principal component analysis, multidimensional scaling, and canonical variate analysis give us insights into the relationships between observations and/or variables that we might not be able to get any other way.

A concrete example: some years ago I was analyzing an employee satisfaction survey that had over 100 questions on it. Well, no manager is ever going to be able to look at 100+ questions worth of answers, even summarized, and do more than guess at what it all means, because who can tell how the answers are related and what is driving them, really? I performed a factor analysis on the data, for which I had over 10,000 observations, and came up with five very clear and readily interpretable factors which could be used to develop manager-specific scores (one for each factor) that would summarize the entirety of the 100+ question survey. A much better solution than the Excel spreadsheet dump that had been the prior method of reporting results!

Bag of Words

Conceptually, we can view bag-of-word model as a special case of the n-gram model, with n=1. 


TF & TF-IDF belong to Vector Space Model

Bag-of-words: For a given document, you extract only the unigram words (aka terms) to create an unordered list of words. No POS tag, no syntax, no semantics, no position, no bigrams, no trigrams. Only the unigram words themselves, making for a bunch of words to represent the document. Thus: Bag-of-words.


We do not consider the order of words in a document. Represented the same way: John is quicker than Mary Mary is quicker than John This is called a bag of words model

Vector Space Model doesn't consider ordering as well.

Random Forest Limitation

Random forest for bag-of-words? No.

Random forest is a very good, robust and versatile method, however it’s no mystery that for high-dimensional sparse data it’s not a best choice. And BoW representation is a perfect example of sparse and high-d.

We covered bag of words a few times before, for example in A bag of words and a nice little network. In that post, we used a neural network for classification, but the truth is that a linear model in all its glorious simplicity is usually the first choice. We’ll use logistic regression, for now leaving hyperparams at their default values.

Validation AUC for logistic regression is 92.8%, and it trains much faster than a random forest. If you’re going to remember only one thing from this article, remember to use a linear model for sparse high-dimensional data such as text as bag-of-words.

NLP Terms

Bag of Words Model
TF-IDF
Vector Space Model
Standard Boolean Model

Why does mean normalization help in gradient descent?

Essentially, scaling the inputs (through mean normalization, or z-score) gives the error surface a more spherical shape, where it would otherwise be a very high curvature ellipse. Since gradient descent is curvature-ignorant, having an error surface with high curvature will mean that we take many steps which aren't necessarily in the optimal direction. When we scale the inputs, we reduce the curvature, which makes methods that ignore curvature (like gradient descent) work much better. When the error surface is circular (spherical), the gradient points right at the minimum, so learning is easy.

Feature Scaling

Three Types:


http://stats.stackexchange.com/questions/70553/how-to-verify-a-distribution-is-normalized/70555#70555


Centered:  X - mean

Standardized:  X - mean / sd
Standardize features by removing the mean and scaling to unit variance

Normalized: X - mean / max - min


http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html
For instance many elements used in the objective function of a learning algorithm (such as the RBF kernel of Support Vector Machines or the L1 and L2 regularizers of linear models) assume that all features are centered around 0 and have variance in the same order. If a feature has a variance that is orders of magnitude larger that others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.

https://en.wikipedia.org/wiki/Feature_scaling

Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the majority of classifiers calculate the distance between two points by the Euclidean distance. If one of the features has a broad range of values, the distance will be governed by this particular feature. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance.

Another reason why feature scaling is applied is that gradient descent converges much faster with feature scaling than without it^{[citation needed]}.

http://stats.stackexchange.com/questions/29781/when-conducting-multiple-regression-when-should-you-center-your-predictor-varia

http://stats.stackexchange.com/questions/86434/is-standardisation-before-lasso-really-necessary

Lasso regression puts constraints on the size of the coefficients associated to each variable. However, this value will depend on the magnitude of each variable. It is therefore necessary to center and reduce, or standardize, the variables.

The result of centering the variables means that there is no longer an intercept. This applies equally to ridge regression, by the way.

Another good explanation is this post: Need for centering and standardizing data in regression

Need Normalization Before PCA

Normalization is important in PCA since it is a variance maximizing exercise. It projects your original data onto directions which maximize the variance. The first plot below shows the amount of total variance explained in the different principal components wher we have not normalized the data. As you can see it seems like only component one explains all the variance in the data.

If you look at the second picture we have normalized the data first. Here it is clear that the other components contribute as well. The reason for this is because PCA seeks to maximize the variance of each component. And since the covariance matrix of this particular dataset is:

             Murder   Assault   UrbanPop      Rape
Murder    18.970465  291.0624   4.386204  22.99141
Assault  291.062367 6945.1657 312.275102 519.26906
UrbanPop   4.386204  312.2751 209.518776  55.76808
Rape      22.991412  519.2691  55.768082  87.72916

From this structure the PCA will of course select to project as much as possible in the direction of Assault since that variance is much greater. So for finding features usable for any kind of model a PCA without normalization would perform badly.

shareeditflag

Log & Exp function

e = 2.71828

Log Loss

Log-loss is an appropriate performance measure when you're model output is the probability of a binary outcome.

The log-loss measure considers confidence of the prediction when assessing how to penalize incorrect classification. For instance consider two predictions of an outcome P(Y=1|X), where the predictions are 0.51 and 0.99 respectively. In the former case the model is only slightly confident of the class prediction (assuming a 0.5 cutoff), while in the latter it is extremely confident. Since in our case both are wrong, the penalty will be more harsh for the more confident (but incorrect) prediction by employing a log-loss penalty.

I just installed GTX 970 into my Ubuntu box. Using the following command shows the card is connected to the system.

~$ lspci -nnk |grep NVIDIA 60:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1) 60:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)

But when I ran my Keras script using TensorFlow as the backend, it didn't seem to speed up the training - each epoch still takes a long time to run.

Is there any way to check if Keras is using GPUs?

You need to install the cuda lib for tensorflow to use GPUhttps://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation

If it is installed correctly, everytime you run tf or keras with tf backend, it will show something like "using Cuda library..."

Key Steps Enable Keras with TensorFlow backend to Use GPUs

1. Install driver

2. Pip install tensorflow-gpu
https://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation
3. Install CUDA
https://www.tensorflow.org/get_started/os_setup#test_the_tensorflow_installation

PCA

Principal Component transformation is a method that stands for reduction of the dimension, that means try to reshape your space in a way that a great part of the total information can be explained with the first few axes. and those axes are actually a linear combination of the initial axes ( variables) , so yes PCA can catch the interaction between variables but only a little , cause to express the interaction between 2 ,3 4 ... variables there a lot of advanced ( aggregation for example) things to do then a simple linear combination between the 2,3 4 ... if you get my idea.

Gradient Boosting & Imbalanced Classes

Insensitive to small classes

Some Notes about Scikit-learn Logistic Regression

1. It doesn't use SGD, so there is no learning rate. It uses liblinear, newton-sg, etc. method to solve.
2. Use SGDClassifier for SGD training
3. C is used to control regularization effect. Set C to be very large will reduce regularization to zero
4. 

Ensemble - Blending & Stacking

Blending means the same thing as Stacking

Stacking, Blending and and Stacked Generalization are all the same thing with different names. It is a kind of ensemble learning.

In traditional ensemble learning, we have multiple classifiers trying to fit to a training set to approximate the target function. Since each classifier will have its own output, we will need to find a combining mechanism to combine the results. This can be through voting (majority wins), weighted voting (some classifier has more authority than the others), averaging the results, etc. This is the traditional way of ensemble learning.

In stacking, the combining mechanism is that the output of the classifiers (Level 0 classifiers) will be used as training data for another classifier (Level 1 classifier) to approximate the same target function. Basically, you let the Level 1 classifier to figure out the combining mechanism.

In practice, this works very well. In fact, it is most famously used in Netflix to achieve a very good score.

Ensemble - Stacking

Stacking is a way to ensemble multiple classification or regression model. There are many ways to ensemble models. Among most widely known are bagging or boosting.  Bagging allows multiple similar models with high variance are averaged to decrease variance. Boosting builds multiple incremental models to decrease the bias, while keeping variance small.

Stacking is a different paradigm however. The point of stacking is to explore a space of different models for the same problem. The idea is that you can attack a learning problem with different types of models which are capable to learn some part of the problem, but not the whole space of the problem. So you can build multiple different learners and you use them to build an intermediate prediction, one prediction for each learned model. Then you add a new model which learns from the intermediate predictions the same target. This final model is said to be stacked on the top of the others, hence the name. Thus you might improve your overall performance, and often you end up with a model which is better than any individual intermediate model. Notice however that it does not gives you any guarantee, as is often the case with any machine learning technique.