Posts

Pivot table in Python

create a pandas dataframe- import numpy as np import pandas as pd cars_data= pd.DataFrame([[10,5,3,'X'], [1,2,3,'Y'],[3,4,5,'X'], [10,5,3,'X'], [1,2,3,'Y'],[3,4,5,'X']], columns=['A','B','C','Type']) cars_data A B C Type 0 10 5 3 X 1 1 2 3 Y 2 3 4 5 X 3 10 5 3 X 4 1 2 3 Y 5 3 4 5 X create pivot_table a method in pandas # using pivot to convert rows to column pivot_car= cars_data.pivot_table(columns=['Type', 'A'], aggfunc= [np.mean, np.amax]) mean amax Type A B X 3 4 4 10 5 5 ...

Sort Python Dictionary by Values/Keys

A-  create a dictionary- items_count = {' are': 1, 'hi': 2, 'me': 3, 'you': 5} B- Sort by values of dict - sorted_x = sorted(items_count.items(), key=lambda kv: kv[1], reverse= True) Sorted_x [('you', 5), ('me', 3), ('hi', 2), ('are', 1)] #op sorted()  have a  key  parameter to specify a function to be called on each list element prior to making comparisons. by default ascending oreder, if reverse = True then reverse order. items_count.items()- dict_item data type that items_count as tuples. kv: kv[1] function that takes 2nd value from each tuple so that comparison happens for values not keys. kv:kv[0] would have sorted by keys of dictionary. C- convert list to dictionary again- import collections sorted_dict = collections.OrderedDict(sorted_x) OrderedDict([('you', 5), ('me', 3), ('hi', 2), ('are', 1)]) #op creates dictionary from list to have dictionary ...

SHAP (SHapley Additive exPlanations) for variable importance in Python

Image
it is a unified approach to explain the output of any machine learning model. pip install shap ( on anaconda prompt) import xgboost import shap # load JS visualization code to notebook shap.initjs() # train XGBoost model X,y = shap.datasets. boston () model = xgboost.train({ " learning_rate " : 0.01 }, xgboost.DMatrix(X, label = y), 100 ) # explain the model's predictions using SHAP values # (same syntax works for LightGBM, CatBoost, and scikit-learn models) explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(X) # visualize the first prediction's explanation (use matplotlib=True to avoid Javascript) shap.force_plot(explainer.expected_value, shap_values[ 0 ,:], X.iloc[ 0 ,:]) The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown...

Deep Learning with H2O in Python

Image
Step 1-  First of all , we need to install H2o package in Python. on anaconda prompt pip install h2o Step 2-  Initialize and start the cluster - h2o . init () from h2o.estimators.deeplearning import H2ODeepLearningEstimator Step 3-  load train and test data set- train = h2o . import_file ( "https://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv" ) Step 4-  Creating test and train data set using split- splits = train . split_frame ( ratios = [ 0.75 ], seed = 1234 ) Step 5-  Configuring the model- model = H2ODeepLearningEstimator(distribution = "AUTO",activation = "RectifierWithDropout",hidden = [32,32],input_dropout_ratio = 0.2,l1 = 1e-5,epochs = 10) Step 6-  train(fit the model)- model.train(x="sepal_len", y=["petal_len"], training_frame=splits[0]) Step 7-  predicting using trained model and creating a new column in test data- (splits[1]['predicted_sepal_len'])=...

Groupby with multiple functions and handling multiple index

Image
creating a data-set- import pandas as pd a='a' b='b' df= pd.DataFrame({'survived':[0,1,1,1,0,1],'class': [a,b,a,b,a,b]}) have a look of data-set- group by based on class - grouped_data= df.groupby(['class']).agg({'survived':['max', 'sum']}) have alook at grouped data , it has got multiple index now- we can delete on of the column index- grouped_data.columns=grouped_data.columns.get_level_values(1)

Words count in a sentence in Python

### counting number of words in Python- This requires use of counter from collection package, Also as split can be used only on strings, one needs to convert list into str. from collections import Counter sentence= ['Data science is growing field but buzz is also around and This is hyped'] splitted_sentence= sentence[0].split(" ")      # we will get str from list and then split would work, split doesn't work on list count_word= Counter() for i in splitted_sentence:     count_word[i]+= 1   # these are like dictionary, can be used to count words etc print(count_word)     output is- Counter({'is': 3, 'and': 1, 'hyped': 1, 'growing': 1, 'around': 1, 'field': 1, 'science': 1, 'buzz': 1, 'Data': 1, 'also': 1, 'but': 1, 'This': 1}) Thus we get the count of all the unique words. can you try list comprehension for the same?

List Comprehension in Python

list can be created in this was- list_variable= [ x for x in iterable] v=(1,4,9,16,25,36) now list comprehension can be use as- v=[x**2 for x in range(7)] Using for loop for getting even numbers in 10 natural numbers- numbers= range(10) newlist=[] for i in numbers:     if i%2 == 0:         newlist= new_list.append(i) print(newlist) Using list comprehension to do the same: newlist= [x for x in numbers if x%2 == 0] print(newlist) output in both the case- [0,2,4,6,8]