Posts

Sort Python Dictionary by Values/Keys

A-  create a dictionary- items_count = {' are': 1, 'hi': 2, 'me': 3, 'you': 5} B- Sort by values of dict - sorted_x = sorted(items_count.items(), key=lambda kv: kv[1], reverse= True) Sorted_x [('you', 5), ('me', 3), ('hi', 2), ('are', 1)] #op sorted()  have a  key  parameter to specify a function to be called on each list element prior to making comparisons. by default ascending oreder, if reverse = True then reverse order. items_count.items()- dict_item data type that items_count as tuples. kv: kv[1] function that takes 2nd value from each tuple so that comparison happens for values not keys. kv:kv[0] would have sorted by keys of dictionary. C- convert list to dictionary again- import collections sorted_dict = collections.OrderedDict(sorted_x) OrderedDict([('you', 5), ('me', 3), ('hi', 2), ('are', 1)]) #op creates dictionary from list to have dictionary ...

SHAP (SHapley Additive exPlanations) for variable importance in Python

Image
it is a unified approach to explain the output of any machine learning model. pip install shap ( on anaconda prompt) import xgboost import shap # load JS visualization code to notebook shap.initjs() # train XGBoost model X,y = shap.datasets. boston () model = xgboost.train({ " learning_rate " : 0.01 }, xgboost.DMatrix(X, label = y), 100 ) # explain the model's predictions using SHAP values # (same syntax works for LightGBM, CatBoost, and scikit-learn models) explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(X) # visualize the first prediction's explanation (use matplotlib=True to avoid Javascript) shap.force_plot(explainer.expected_value, shap_values[ 0 ,:], X.iloc[ 0 ,:]) The above explanation shows features each contributing to push the model output from the base value (the average model output over the training dataset we passed) to the model output. Features pushing the prediction higher are shown...

Deep Learning with H2O in Python

Image
Step 1-  First of all , we need to install H2o package in Python. on anaconda prompt pip install h2o Step 2-  Initialize and start the cluster - h2o . init () from h2o.estimators.deeplearning import H2ODeepLearningEstimator Step 3-  load train and test data set- train = h2o . import_file ( "https://h2o-public-test-data.s3.amazonaws.com/smalldata/iris/iris_wheader.csv" ) Step 4-  Creating test and train data set using split- splits = train . split_frame ( ratios = [ 0.75 ], seed = 1234 ) Step 5-  Configuring the model- model = H2ODeepLearningEstimator(distribution = "AUTO",activation = "RectifierWithDropout",hidden = [32,32],input_dropout_ratio = 0.2,l1 = 1e-5,epochs = 10) Step 6-  train(fit the model)- model.train(x="sepal_len", y=["petal_len"], training_frame=splits[0]) Step 7-  predicting using trained model and creating a new column in test data- (splits[1]['predicted_sepal_len'])=...

Groupby with multiple functions and handling multiple index

Image
creating a data-set- import pandas as pd a='a' b='b' df= pd.DataFrame({'survived':[0,1,1,1,0,1],'class': [a,b,a,b,a,b]}) have a look of data-set- group by based on class - grouped_data= df.groupby(['class']).agg({'survived':['max', 'sum']}) have alook at grouped data , it has got multiple index now- we can delete on of the column index- grouped_data.columns=grouped_data.columns.get_level_values(1)

Words count in a sentence in Python

### counting number of words in Python- This requires use of counter from collection package, Also as split can be used only on strings, one needs to convert list into str. from collections import Counter sentence= ['Data science is growing field but buzz is also around and This is hyped'] splitted_sentence= sentence[0].split(" ")      # we will get str from list and then split would work, split doesn't work on list count_word= Counter() for i in splitted_sentence:     count_word[i]+= 1   # these are like dictionary, can be used to count words etc print(count_word)     output is- Counter({'is': 3, 'and': 1, 'hyped': 1, 'growing': 1, 'around': 1, 'field': 1, 'science': 1, 'buzz': 1, 'Data': 1, 'also': 1, 'but': 1, 'This': 1}) Thus we get the count of all the unique words. can you try list comprehension for the same?

List Comprehension in Python

list can be created in this was- list_variable= [ x for x in iterable] v=(1,4,9,16,25,36) now list comprehension can be use as- v=[x**2 for x in range(7)] Using for loop for getting even numbers in 10 natural numbers- numbers= range(10) newlist=[] for i in numbers:     if i%2 == 0:         newlist= new_list.append(i) print(newlist) Using list comprehension to do the same: newlist= [x for x in numbers if x%2 == 0] print(newlist) output in both the case- [0,2,4,6,8]

How to save a model variable in Python using pickle library

import pickle # take user input to take the amount of data number_of_data = int(input('Enter the number of data : ')) data = [] # take input of the data for i in range(number_of_data):     raw = input('Enter data '+str(i)+' : ')     data.append(raw) # open a file, where you ant to store the data file = open('important', 'wb') # dump information to that file pickle.dump(data, file) # close the file file.close()