web scraping - Using python scrapy to extract link and text -


i new python , scrapy. extract information website http://www.vodafone.com.au/about/legal/critical-information-summary/plans including link document, name , valid to.

i tried code, not work. appreciated if explain , me.

here file vodafone.py

import scrapy  scrapy.linkextractor import linkextractor scrapy.spiders import rule, crawlspider vodafone_scraper.items import vodafonescraperitem   class vodafonespider(scrapy.spider):     name = 'vodafone'     allowed_domains = ['vodafone.com.au']     start_urls = ['http://www.vodafone.com.au/about/legal/critical-information-summary/plans']      def parse(self, response):         sel in response.xpath('//tbody/tr/td[1]/a'):             item = vodafonescraperitem()             item['link'] = sel.xpath('href').extract()             item['name'] = sel.xpath('text()').extract_first()              yield item

it doesn't work because page content generated dynamically javascript. elements try extract data not present in html source scrapy receives response (you can see when open page source code in browser).

you have 2 options:

  1. try if won't find api page uses. in browser's developer tools xhr requests on network tab. luckily, concrete page seems data requests http://www.vodafone.com.au/rest/cis?field:plancategory:equals=mobile%20plans&field:planfromdate:lessthaneq=20/08/2017. returns json can parse.
  2. the other option render page including javascript , parse then. recommend using splash it's seamless integration scrapy via scrapy-splash library.

Comments

Post a Comment

Popular posts from this blog

ubuntu - PHP script to find files of certain extensions in a directory, returns populated array when run in browser, but empty array when run from terminal -

i running os ububtu 16.04. ultimate goal able run php script cron job every minute. here php script located @ /var/www/html/tests/test/index.php : <?php $directorypath = $_server['document_root']."/tests/test/data/"; echo "directorypath: $directorypath<br><br>";//check $imagefilepathsarray = glob($directorypath . "*.{png,gif}", glob_brace); echo "<br><br>imagefilepathsarray: "; print_r($imagefilepathsarray);//check ?> when open php script in browser visiting url my-ip-address/tests/test/index.php , get: directorypath: /var/www/html/tests/test/data/ imagefilepathsarray:array( [0] => /var/www/html/tests/test/data/pic.png [1] => /var/www/html/tests/test/data/pic.gif ) before writing cron job in crontab file, need check if can run anywhere using terminal . opened ubuntu terminal @ /home/ location, , executed in terminal: sudo php /var/www/html/tests/test/index.php i got following out...
Read more

php - How can i create a user dashboard -

today have questions you, want create user dashboard site, include (obviously), users data, example -email -username -and others stuff how can information, , print them in html page? if can, how can these information via php sessions? shuld use session? example, if user login, , start new session ( session_start() ), how can data print in dashboard? should use php "project"? in advice, have day , thank time!
Read more

javascript - How to detect toggling of the fullscreen-toolbar in jQuery Mobile? -

i have jquery mobile page fixed full-screen toolbar, data-tap-toggle enabled. straight under toolbar have positioned banner should slide upwards when toolbar hiding , slide downward when toolbar showing. jquery mobile toggles toolbar applying , removing ui-fixed-hidden class - sadly, can't find in documentation of toolbar widget toggle , hide or show event that. how can detect when toolbar toggled, reposition banner? .banner { position: fixed; background-color: darkseagreen; top: 46px; min-height: 48px; width: 100%; text-align: center; line-height: 48px; } <!doctype html> <html> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no"> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"> <script src="https://code.jquery.com/jquery-1....
Read more