python pathos multiprocessing -
i trying use pathos multiprocessing simple function pass array, splits parallel processes , joins @ end.
here code ported on using multiprocessing function, , instead of importing multiprocessing, imported pathos.
import sys import gender_guesser.detector gender import time pathos import multiprocessing multiprocessing d = gender.detector() def guess_gender (name): n = name.title() # make first letter upper case , rest lower case g = d.get_gender(n) # guess gender return g ls = ['john','joe','amamda','derick','peter','ashley','john','joe','amamda','derick','peter','ashley'] t=time.time() results=[] def callback(x): results.append(x) pool = multiprocessing.pool(processes=multiprocessing.cpu_count()-1, maxtasksperchild=1) n in ls: print (n) pool.apply_async(guess_gender,args=[n],callback=callback) pool.close() pool.join() results = pd.concat(results) print(time.time()-t)
however, error:
valueerror: no objects concatenate
i not sure how extract results of output. documentation on pathos thin.
does have ideas or experience?
fyi running python in jupyter notebooks.
Comments
Post a Comment