web scraping - Webscrapping an alternate version/hidden item from a webpage using python beautifulsoup -


i trying webscrap information (points scored, tackles made, time played, position, etc...) top14 rugby players website. each player info page : http://www.lnr.fr/rugby-top-14/joueurs/nicholas-abendanon each player, can info 2015-2016 season easily, need info 2014-2015 season.

problem is, when open corresponding link (http://www.lnr.fr/rugby-top-14/joueurs/nicholas-abendanon#season=14535) source code same , info program scrap 2015-2016 data. can't seem find way info previous seasons though appears on webpage. knows how solve ?

here code player gave example.

import bs4 lxml import html import requests import string import _pickle pickle bs4 import beautifulsoup  dic={} url_player='http://www.lnr.fr/rugby-top-14/joueurs/nicholas-abendanon' page = requests.get(url_player) html=page.content parsed_html = beautifulsoup(html) body=parsed_html.body saison14_15=body.find('a',attrs={'data-title':'saison 2014-2015'}) link=saison14_15['href'] url_season='http://www.lnr.fr/rugby-top-14/joueurs/nicholas-abendanon'+link page_season = requests.get(url_season) html_season=page_season.content parsed_html_season = beautifulsoup(html_season) body_season=parsed_html_season.body  dic['nom']=body_season.find('h1',attrs={'id':'page-title'}).text dic[body_season.find('span',attrs=     {'class':'title'}).text]=body_season.find('span',attrs={'class':'text'}).text info1=body_season.find('ul',attrs={'class':'infos-list small-bold'}) try:     item in info1.findall('li'):         dic[item.find('span',attrs={'class':'title'}).text]=item.find('span',attrs={'class':'text'}).text      info2=body_season.find('ul',attrs={'class':'fluid-block-grid-3 team-stats'})     if info2 not none :         item in info2.findall('li'):                dic[item.find('span',attrs={'class':'title'}).text]=item.find('span',attrs={'class':'text'}).text     info3=body_season.find('ul',attrs={'class':'number-list small-block-grid-2'})     if info3 not none :         item in info3.findall('li'):                dic[item.find('span',attrs={'class':'title'}).text]=item.find('span',attrs={'class':'text'}).text except:     dic=dic`

when choose 2014-2015 season, page makes ajax request to

http://www.lnr.fr/ajax_player_stats_detail?player=33249&compet_type=1&=undefined&season=14535&_filter_current_tab_id=panel-filter-season&ajax-target-selector=%23player_stats_detail_block

if switch 2015-2016, makes ajax request to

http://www.lnr.fr/ajax_player_stats_detail?player=33249&compet_type=1&=undefined&season=18505&_filter_current_tab_id=panel-filter-season&ajax-target-selector=%23player_stats_detail_block

each request returns chunk of html gets inserted page.

if can figure out parameters needed player , season, suggest request data directly (without loading parent page @ all).


Comments

Post a Comment

Popular posts from this blog

sublimetext3 - what keyboard shortcut is to comment/uncomment for this script tag in sublime -

Image
i have script tag in code, keyboard shortcut comment/uncomment script tag in sublime. after selecting script open , close tag. <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> i pressing ctrl + shift + / , getting this /*<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script>*/ also pressed ctrl + / , got this // <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> // </script> instead of <!-- <script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"> </script> --> script open , close tag in 2 lines not in 1 line , not have spaces too. for mac,high light lines want comment. hold 'command , / ' commented. for windows: high line or select lines ,then hold ctrl + / if not work: ...
Read more

java - No use of nillable="0" in SOAP Webservice -

i have received xsd 3rd party supplier generated java based system; used create soap endpoint receive data transfers. xsd not make use of nillable attribute defined within w3c/xsd namespace, example: <xs:complextype name="customer"> <xs:sequence> <xs:element minoccurs="1" maxoccurs="1" name="id" type="xs:integer"/> <xs:element minoccurs="0" maxoccurs="1" name="titleid" type="xs:integer"/> <xs:element minoccurs="0" maxoccurs="1" name="firstname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" name="familyname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" name="previousname" type="xs:string"/> <xs:element minoccurs="0" maxoccurs="1" nam...
Read more

ubuntu - Laravel 5.2 quickstart guide gives Not Found Error -

edit: think problem due not setting apache properly, these 2 links helped me. solution: http://laravel.io/forum/06-11-2014-not-found-but-route-exists https://www.digitalocean.com/community/tutorials/how-to-set-up-apache-virtual-hosts-on-ubuntu-14-04-lts end edit i'm following laravel 5.2 quickstart guide. https://laravel.com/docs/5.2/quickstart i'm running ubuntu 14.04.2 the website runs initially, when click on add task button run 404 not found error. as per guide run following commands setup complete quickstart project. git clone https://github.com/laravel/quickstart-basic quickstart cd quickstart composer install php artisan migrate additionally run following commands because i'm on ubuntu server: sudo chown -r www-data.www-data quickstart sudo chmod -r 755 quickstart sudo chmod 777 quickstart/storage the routes.php file looks like: <?php use app\task; use illuminate\http\request; route::group(['middleware' => ['web']]...
Read more