Teadmised first lab 2011

The main goal of the first lab is to web-scrape data from the public sources using search engines and simple text analysis/statistics.

Conditions

You can do the lab either alone or together with a fellow student. Three students per lab is not ok, though.

Technology - language, op system etc - is completely free.

What should the app do

Your system should take as input a list of arbitrary investigated objects like "Tower Bridge", "Statue of Liberty", "Tallinna raekoda", "Olde hansa", "Tallinn University of Technology", "Andrus Ansip", "Ford focus", "inception" and use web search and web-scraping to obtain probabilistic data about the objects. We look for

• popularity (search hits)
• object type (building, church, tower, landmark, person, car, ...)
• creation year (building year, birthdate, year of introduction to market, ...)
• other often-used characteristic words/tags (steeple, national, medieval, academic, peaminister, thriller, ...)

The data should be augmented by a numeric confidence measure (0....1) indicating rough trust in the result. The confidence is mostly based on the percentage of investigated pages/sources giving this result, as well as a perceived quality of the data source (like: wikipedia has high quality, arbitrary pages low).

In order to get the list of pages to web scrape you should do several searches, selecting from / using in parallel

• wikipedia search
• wolfram alpha search
• google or bing (or some other) web search

The results should be scraped. Use a different scraping method for semistructured data (wikipedia, wolfram alpha) and unstructured data (arbitrary web pages). For the latter it makes sense to split the page into sentences (just use period . as a separator) and look for sentences containing the investigated phrase. What other words does it contain? What numbers does it contain?

After scraping N pages for the same phrase, look for the most highly occurring type words, year numbers, other tags.

Hints and suggestions

While developing, it is easier to use a pre-stored search result/wikipedia page and pre-stored set of pages instead of doing full search and download each time.

However, the system should finally work with any phrases, meaning that the pre-loaded data cannot be normally used.

Additional helpful suggestions and recommendations will be added later during the course.