python - pool.apply_async takes so long to finish, how to speed up? -


i call pool.apply_async() 14 cores.

   import multiprocessing     time import time    import timeit     informative_patients = informative_patients_2500_end[20:]    pool = multiprocessing.pool(14)    results = []    wlength = [20,30,50]     start = time()     fn in informative_patients:         result = pool.apply_async(compute_features_test_set, args = (fn,          wlength), callback=results.append)     pool.close()     pool.join()      stop = timeit.default_timer()     print stop - start  

the problem finishes calling compute_features_test_set() function first 13 data in less 1 hour, takes more 1 hour finish last one. size of data 14 data-set same. tried putting pool.terminate() after pool.close() in case doesn't start pool , terminate pool without going inside loop. happen in same way , if use more cores , more data set, last 1 takes long finish. compute_features_test_set() function simple feature extraction code , works correctly. work on server linux red hat 6, python 2.7 , jupyter. computation time important me , question wrong here , how can fix computation done in reasonable time?

question: ... wrong here , how can fix it

couldn't catch multiprocessing issue.
how this: "always last 1 takes long finish"?
using callback=results.append instead of own function?
edit question , show how timeit one process time.
add python version question.

do following verify it's not data issue:

   start = time()    results.append(        compute_features_test_set(<first informative_patients>, wlength        )    stop = timeit.default_timer()    print stop - start      start = time()    results.append(        compute_features_test_set(<last informative_patients>, wlength        )    stop = timeit.default_timer()    print stop - start  

compare 2 times get.


Comments

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

php - How can i create a user dashboard -

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -