Notes for KR lab 1 - Redigeerimiste ajalugu

Tammet: /* Real estate portals */

2015-02-16T21:39:57Z

Real estate portals

←Vanem redaktsioon		Redaktsioon: 16. veebruar 2015, kell 23:39
62. rida:		62. rida:
	http://www.kv.ee/?act=search.simple&company_id=&page=1&orderby=ob&page_size=&deal_type=1&dt_select=1&county=0&parish=&price_min=&price_max=&price_type=1&rooms_min=&rooms_max=&area_min=&area_max=&floor_min=&floor_max=&keyword=mustam%C3%A4e+tee+183		http://www.kv.ee/?act=search.simple&company_id=&page=1&orderby=ob&page_size=&deal_type=1&dt_select=1&county=0&parish=&price_min=&price_max=&price_type=1&rooms_min=&rooms_max=&area_min=&area_max=&floor_min=&floor_max=&keyword=mustam%C3%A4e+tee+183

	See also https://www.ehr.ee/ with a query		See also https://www.ehr.ee/

			<!-- with a query

	http://www.ehr.ee/app/otsing?7-1.IBehaviorListener.0-detailedsearchpanel-tabsPanel-panel-searchForms-searchSimple-searchForm-searchFieldWithButton-ajaxSubmit&id5c_hf_0=&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AsearchParameter=mustam%C3%A4e+tee+122&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AajaxSubmit=1		http://www.ehr.ee/app/otsing?7-1.IBehaviorListener.0-detailedsearchpanel-tabsPanel-panel-searchForms-searchSimple-searchForm-searchFieldWithButton-ajaxSubmit&id5c_hf_0=&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AsearchParameter=mustam%C3%A4e+tee+122&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AajaxSubmit=1

			-->

	== Small example as a start ==		== Small example as a start ==

Tammet: /* Real estate portals */

2015-02-16T21:38:48Z

Real estate portals

←Vanem redaktsioon		Redaktsioon: 16. veebruar 2015, kell 23:38
61. rida:		61. rida:

	http://www.kv.ee/?act=search.simple&company_id=&page=1&orderby=ob&page_size=&deal_type=1&dt_select=1&county=0&parish=&price_min=&price_max=&price_type=1&rooms_min=&rooms_max=&area_min=&area_max=&floor_min=&floor_max=&keyword=mustam%C3%A4e+tee+183		http://www.kv.ee/?act=search.simple&company_id=&page=1&orderby=ob&page_size=&deal_type=1&dt_select=1&county=0&parish=&price_min=&price_max=&price_type=1&rooms_min=&rooms_max=&area_min=&area_max=&floor_min=&floor_max=&keyword=mustam%C3%A4e+tee+183

			See also https://www.ehr.ee/ with a query

			http://www.ehr.ee/app/otsing?7-1.IBehaviorListener.0-detailedsearchpanel-tabsPanel-panel-searchForms-searchSimple-searchForm-searchFieldWithButton-ajaxSubmit&id5c_hf_0=&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AsearchParameter=mustam%C3%A4e+tee+122&searchSimple%3AsearchForm%3AsearchFieldWithButton%3AajaxSubmit=1

	== Small example as a start ==		== Small example as a start ==

Tammet – 16. veebruar 2015, kell 20:40

2015-02-16T20:40:56Z

←Vanem redaktsioon		Redaktsioon: 16. veebruar 2015, kell 22:40
54. rida:		54. rida:

	A good idea is to decide upon a small list of highly relevant and common words and search the name together with these words!		A good idea is to decide upon a small list of highly relevant and common words and search the name together with these words!

			== Real estate portals ==

			A good idea in Estonia is to use http://www.kv.ee/ where you can simply
			search with a get query like this:

			http://www.kv.ee/?act=search.simple&company_id=&page=1&orderby=ob&page_size=&deal_type=1&dt_select=1&county=0&parish=&price_min=&price_max=&price_type=1&rooms_min=&rooms_max=&area_min=&area_max=&floor_min=&floor_max=&keyword=mustam%C3%A4e+tee+183

	== Small example as a start ==		== Small example as a start ==

Tammet: /* Searching */

2014-02-21T15:28:30Z

Searching

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:28
6. rida:		6. rida:
	https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8&start=0&q=Andrus+Ansip		https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8&start=0&q=Andrus+Ansip

	where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=0 says you want to get the links from link nr 8.		where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=0 says you want to get the links from link nr 0.

	In other words, start with start=0, then use start=8, start=16 etc until you still get results: the max nr of links you get is ca 100.		In other words, start with start=0, then use start=8, start=16 etc until you still get results: the max nr of links you get is ca 100.

Tammet: /* Searching */

2014-02-21T15:28:19Z

Searching

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:28
4. rida:		4. rida:
	deprecated one is easier to use: just open url like		deprecated one is easier to use: just open url like

	https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8&start=8&q=Andrus+Ansip		https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8&start=0&q=Andrus+Ansip

	where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=8 says you want to get the links from link nr 8.		where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=0 says you want to get the links from link nr 8.

	In other words, start with start=0, then use start=8, start=16 etc until you still get results: the max nr of links you get is ca 100.		In other words, start with start=0, then use start=8, start=16 etc until you still get results: the max nr of links you get is ca 100.

Tammet – 21. veebruar 2014, kell 15:26

2014-02-21T15:26:28Z

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:26
33. rida:		33. rida:

	After such elementary replacing you should probably just split the whole html into "words" separated by whitespace.		After such elementary replacing you should probably just split the whole html into "words" separated by whitespace.

			== Processing words on the page ==

	Now when you have obtained a word list (containing lots of garbage words) you want to associate words with their relevance number on the page:		Now when you have obtained a word list (containing lots of garbage words) you want to associate words with their relevance number on the page:
41. rida:		43. rida:
	words, you may well throw away all words with a very low relevance: we will not use		words, you may well throw away all words with a very low relevance: we will not use
	them anyway.		them anyway.

			== Aggregating words on different pages ==

	The next step from here is to start aggregating the relevance-weighted lists of words for each html source. Observe that some html pages themselves are more relevant than others. For example, wikipedia and facebook pages are probably highly relevant. Similarly, pages at the top of the search list are probably more relevant than the pages and the end of the search list.		The next step from here is to start aggregating the relevance-weighted lists of words for each html source. Observe that some html pages themselves are more relevant than others. For example, wikipedia and facebook pages are probably highly relevant. Similarly, pages at the top of the search list are probably more relevant than the pages and the end of the search list.
47. rida:		51. rida:
	the weighted word lists of separate pages into the common weighted word list for the whole phrase, using the page relevance number as an additional weight to be used while aggregating.		the weighted word lists of separate pages into the common weighted word list for the whole phrase, using the page relevance number as an additional weight to be used while aggregating.

			== What to do if you find very few interesting words? ==

			A good idea is to decide upon a small list of highly relevant and common words and search the name together with these words!

	== Small example as a start ==		== Small example as a start ==

Tammet – 21. veebruar 2014, kell 15:24

2014-02-21T15:24:02Z

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:24
22. rida:		22. rida:

	The small example below does not do any caching.		The small example below does not do any caching.

			== Processing html source ==

			In the end we are only interested about a small number of specific keywords. Thus there is no real need to throw away garbage words, html constructions etc: you can do this,
			but you may also skip it.

			A good idea is to
			* replace potential separators like < , > with whitespace
			* replace period . with a whitespace-period-whitespace like this " . "

			After such elementary replacing you should probably just split the whole html into "words" separated by whitespace.

			Now when you have obtained a word list (containing lots of garbage words) you want to associate words with their relevance number on the page:
			* the closer the word is to a searched phrase, the more relevant it is.
			* the more the word occurs on the page, the more relevant it is.

			Once you decide upon the concrete "relevance formula" and calculate relevances for the
			words, you may well throw away all words with a very low relevance: we will not use
			them anyway.

			The next step from here is to start aggregating the relevance-weighted lists of words for each html source. Observe that some html pages themselves are more relevant than others. For example, wikipedia and facebook pages are probably highly relevant. Similarly, pages at the top of the search list are probably more relevant than the pages and the end of the search list.

			Once you have decided upon the "page relevance" formula, you may want to aggregate
			the weighted word lists of separate pages into the common weighted word list for the whole phrase, using the page relevance number as an additional weight to be used while aggregating.


	== Small example as a start ==		== Small example as a start ==

Tammet: /* Searching */

2014-02-21T15:14:35Z

Searching

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:14
8. rida:		8. rida:
	where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=8 says you want to get the links from link nr 8.		where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=8 says you want to get the links from link nr 8.

	In other words, start with start=8, then use start=16, etc until you still get results: the max nr of links you get is ca 100.		In other words, start with start=0, then use start=8, start=16 etc until you still get results: the max nr of links you get is ca 100.

	The small example below implements such an iteration over search pages.		The small example below implements such an iteration over search pages.

			== Caching ==

			While caching the results is not strictly necessary, it is a good idea to store both
			* google search results (link lists)
			* html sources obtained from links
			to a file.

			Then - while experimenting and debugging - you do not need to search for each name more than once. A large number of searches will simply deplete your qoogle quota for some period. Also, reading all these files takes time.

			The small example below does not do any caching.

	== Small example as a start ==		== Small example as a start ==

Tammet: /* Searching */

2014-02-21T15:11:23Z

Searching

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 17:11
1. rida:		1. rida:
	== Searching ==		== Searching ==

	You do not - strictly speaking - need Google: could also use bing etc. There are two ways to get data from Google: using an		You do not - strictly speaking - need Google: could also use bing etc. There are two ways to get data from Google: using a new APi or an old deprecated API. The old
			deprecated one is easier to use: just open url like

			https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8&start=8&q=Andrus+Ansip

	~~https://ajax.googleapis.com/ajax/services/search/web?v=1.0&~~rsz=8"		where rsz=8 indicates that there are 8 links per one answer page (default 4, numbers larger than 8 not accepted) and start=8 says you want to get the links from link nr 8.
	~~url+="&start="+str~~(start)
	~~url+~~=~~"&q="~~
	~~url+="Andrus+Ansip"~~

			In other words, start with start=8, then use start=16, etc until you still get results: the max nr of links you get is ca 100.

			The small example below implements such an iteration over search pages.

	== Small example as a start ==		== Small example as a start ==

Tammet – 21. veebruar 2014, kell 14:21

2014-02-21T14:21:18Z

←Vanem redaktsioon		Redaktsioon: 21. veebruar 2014, kell 16:21
1. rida:		1. rida:
			== Searching ==

			You do not - strictly speaking - need Google: could also use bing etc. There are two ways to get data from Google: using an


			https://ajax.googleapis.com/ajax/services/search/web?v=1.0&rsz=8"
			url+="&start="+str(start)
			url+="&q="
			url+="Andrus+Ansip"



			== Small example as a start ==

	A very small example program in Python 3 to introduce scraping and Google search api: it does a google search for Andrus Ansip with 8 results per page and then goes through the result pages (each containing 8 links).		A very small example program in Python 3 to introduce scraping and Google search api: it does a google search for Andrus Ansip with 8 results per page and then goes through the result pages (each containing 8 links).