python - Scrapy on a JSON response -
i having problem scraping json response in scrapy. getting following error:
typeerror: json object must str, not 'response'
i have tried decode using following importing codecs , decoding response utf-8, isn't working reason. code:
import scrapy import json class srealitkybuyspiderspider(scrapy.spider): name = "srealitky_buy" allowed_domains = ["https://www.sreality.cz/"] start_urls = ['https://www.sreality.cz/api/cs/v2/estates?category_main_cb=1&category_type_cb=1&per_page=20®ion=praha'] def parse(self, response): jsonresponse = json.loads(response) print(jsonresponse)
i tried adding .text response when using json.loads, error "response has no attribute 'text'".
jsonresponse = json.loads(response.text)
does know wrong?
if @ type(response)
scrapy.http.response.text.textresponse
and response.body
give b'{"meta_description": "5192 realit v nab\\u00eddce prodej byt\\u016f, .... "collective"}}'
so need use
jsonresponse = json.loads(response.body) print(jsonresponse["meta_description"]) '5192 realit v nabídce prodej bytů, praha. vyberte si novou nemovitost na sreality.cz s hledáním na mapě velkými náhledy fotografií nabízených bytů.'
edit-1
if above doesn't work because of python version try below
jsonresponse = json.loads(response.body.decode("utf-8"))
Comments
Post a Comment