javascript - Download entire webpage (html, image, JS) by Selenium Python -


i have download source code of website www.humkinar.pk in simple html form. content on site dynamically generated. have tried driver.page_source function of selenium not download page such image , javascript files left. how can download complete page. there better , easy solution in python available?

using selenium

i know question selenium, experience telling selenium recommended testing , not scraping. slow. multiple instances of headless browsers (chrome situation), result delaying much.

recommendation

python 2, 3

this trio lot , save bunch of time.

do not use parser of dryscrape, slow , buggy. situation, 1 can use beautifulsoup lxml parser. use dryscrape scrape javascript generated content, plain html , images.

if scraping lot of links simultaneously, highly recommend using threadpoolexecutor

edit #1

dryscrape + beautifulsoup usage (python 3+)

from dryscrape import start_xvfb dryscrape.session import session dryscrape.mixins import waittimeouterror bs4 import beautifulsoup  def new_session():     session = session()     session.set_attribute('auto_load_images', false)     session.set_header('user-agent', 'someuseragent')     return session   def session_reset(session):     return session.reset()   def session_visit(session, url, check):     session.visit(url)     # ensure market table visible first     if check:         try:             session.wait_for(lambda: session.at_css(                 'some#css.selector.here'))         except waittimeouterror:             pass     body = session.body()     session_reset(session)     return body  # start xvfb in case no x running (server) start_xvfb()  session = new_session() url = 'https://stackoverflow.com/questions/45796411/download-entire-webpage-html-image-js-by-selenium-python/45824047#45824047' check = false  body = session_visit(session, url, check) soup = beautifulsoup(body, 'lxml')  result = soup.find('div', {'id': 'answer-45824047'})  print(result)

Comments

Post a Comment

Popular posts from this blog

python - Operations inside variables -

this question has answer here: evaluating mathematical expression in string 12 answers i want make python script finds maximum possible output using 2 operations. problem says there syntax error @ 2nd last line, in b in there. how can fix it? x = 0 = int(input("1st number:")) c = int(input("2nd number:" )) e = int(input("3rd number:" )) in range(4): if == 0: b = "+" elif == 1: b = "-" elif == 2: b = "/" else: b = "*" j in range(4): if j == 0: d = "+" elif j == 1: d = "-" elif j == 2: d = "/" else: d = "*" k = b c d e print(k) if you're after variables e on single line, couldn...
Read more

Generic Map Parameter java -

i having lot of methods of type: public static list<eduusers> getdetails(class c, map<string, integer> params) { list<eduusers> details = null; session session = hibernateutil.getsessionfactory().opensession(); transaction tx = null; try { tx = session.begintransaction(); criteria cr = session.createcriteria(c); (map.entry<string, integer> entry : params.entryset()) { cr.add(restrictions.eq(entry.getkey(), entry.getvalue())); } details = cr.list(); tx.commit(); } catch (exception asd) { log.debug(asd.getmessage()); if (tx != null) { tx.commit(); } } { session.close(); } return details; } i trying have generic method them , have written far: public static <t> list<t> getdetails(class<t> c, class<t> b) { l...
Read more

arrays - What causes a java.lang.ArrayIndexOutOfBoundsException and how do I prevent it? -

what arrayindexoutofboundsexception mean , how rid of it? here code sample triggers exception: string[] name = {"tom", "dick", "harry"}; for(int = 0; i<=name.length; i++) { system.out.print(name[i] +'\n'); } your first port of call should documentation explains reasonably clearly: thrown indicate array has been accessed illegal index. index either negative or greater or equal size of array. so example: int[] array = new int[5]; int boom = array[10]; // throws exception as how avoid it... um, don't that. careful array indexes. one problem people run thinking arrays 1-indexed, e.g. int[] array = new int[5]; // ... populate array here ... (int index = 1; index <= array.length; index++) { system.out.println(array[index]); } that miss out first element (index 0) , throw exception when index 5. valid indexes here 0-4 inclusive. correct, idiomatic for statement here be: for (int index = 0; index <...
Read more