python - pool.apply_async takes so long to finish, how to speed up? -
i call pool.apply_async() 14 cores.
import multiprocessing time import time import timeit informative_patients = informative_patients_2500_end[20:] pool = multiprocessing.pool(14) results = [] wlength = [20,30,50] start = time() fn in informative_patients: result = pool.apply_async(compute_features_test_set, args = (fn, wlength), callback=results.append) pool.close() pool.join() stop = timeit.default_timer() print stop - start
the problem finishes calling compute_features_test_set() function first 13 data in less 1 hour, takes more 1 hour finish last one. size of data 14 data-set same. tried putting pool.terminate() after pool.close() in case doesn't start pool , terminate pool without going inside loop. happen in same way , if use more cores , more data set, last 1 takes long finish. compute_features_test_set() function simple feature extraction code , works correctly. work on server linux red hat 6, python 2.7 , jupyter. computation time important me , question wrong here , how can fix computation done in reasonable time?
question: ... wrong here , how can fix it
couldn't catch multiprocessing
issue.
how this: "always last 1 takes long finish"?
using callback=results.append
instead of own function
?
edit question , show how timeit
one process time.
add python version question.
do following verify it's not data issue:
start = time() results.append( compute_features_test_set(<first informative_patients>, wlength ) stop = timeit.default_timer() print stop - start start = time() results.append( compute_features_test_set(<last informative_patients>, wlength ) stop = timeit.default_timer() print stop - start
compare 2 times get.
Comments
Post a Comment