python - ipyparallel's LoadBalancedView bloats memory, how can I avoid that? -


this issue may related https://github.com/ipython/ipyparallel/issues/207 not marked solved, yet.
opened issue here https://github.com/ipython/ipyparallel/issues/286

i want execute multiple tasks in parallel using python , ipyparallel in jupyter notebook , using 4 local engines executing ipcluster start in local console. besides 1 can use directview, use loadbalancedview map set of tasks. each task takes around 0.2 seconds (can vary though) , each task mysql query loads data , processes it.
working ~45000 tasks works fine, however, memory grows high. bad because want run experiment on 660000 tasks can't run anymore because bloats memory limit of 16 gb , memory swapping on local drive starts. however, when using directview memory grows relatively small , never full. need loadbalancedview.
when running minimal working example without database query happens (see below).

i not familiar ipyparallel library i've read logs , caches ipcontroler may cause this. still not sure if bug or if can change settings avoid problem.

running mwe

for python 3.5.3 environment running on windows 10 use following (recent) packages:

  • ipython 6.1.0
  • ipython_genutils 6.1.0
  • ipyparallel 6.0.2
  • jupyter 1.0.0
  • jupyter_client 4.4.0
  • jupyter_console 5.0.0
  • jupyter_core 4.2.0

i following example work loadbalancedview without immense memory growth (if possible @ all):

  • start ipcluster start on console

  • run jupyter notebook following 3 cells:

    <1st cell> import ipyparallel ipp rc = ipp.client() lview = rc.load_balanced_view()  <2nd cell> %%px --local import time  <3rd cell> def sleep_here(i):     time.sleep(0.2)     return 42  amr = lview.map_async(sleep_here, range(660000)) amr.wait_interactive()


Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more