Teadmised first lab 2011
The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.
Your system should take as input a list of arbitrary investigated objects like "Tower Bridge", "Statue of Liberty", "Tallinna raekoda", "Olde hansa", "Tallinn University of Technology", "Andrus Ansip", "Ford focus", "inception" and use web search and web-scraping to obtain probabilistic data about the objects. We look for
- popularity (search hits)
- object type (building, church, tower, landmark, person, car, ...)
- creation year (building year, birthdate, year of introduction to market, ...)
- other often-used characteristic words/tags (steeple, national, medieval, academic, peaminister, thriller, ...)
The data should be augmented by a numeric confidence measure (0....1) indicating rough trust in the result. The confidence is mostly based on the percentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low).
In order to get the list of pages to web scrape you should do several searches, selecting from / using in parallel
- wikipedia search
- wolfram alpha search
- google or bing (or some other) web search
The results should be scraped. Use a different scraping method for semistructured data (wikipedia, wolfram alpha) and unstructured data (arbitrary web pages). For the latter it makes sense to split the page into sentences (just use period . as a separator) and look for sentences containing the investigated phrase. What other words does it contain? What numbers does it contain?
After scraping N pages for the same phrase, look for the most highly occurring type words, year numbers, other tags.
Additional helpful suggestions and recommendations will be added later during the course.