Andme-apide märkmed loengust

General

Programmable web is a great site giving pointers to a large set of open data apis people have created.

Use either or both:

• curl to try the examples out or use the small python code down below to ease experimenting

• https://www.getpostman.com/ web app for chrome

Background example

See sightsmap as an example of a system based on a collection of large datasets obtained from open apis.

Also read how the sightsmap system is built

Weather

Weather underground example http://api.wunderground.com/api/da9972c7aa341e74/conditions/q/CA/Tartu.json

EMHI example http://www.emhi.ee/ilma_andmed/xml/observations.php

See emhi docs

Panoramio photos

Example: http://www.panoramio.com/map/get_panoramas.php?set=full&size=original&from=0&to=99&minx=24.73&miny=59.428&maxx=24.76&maxy=59.445&mapfilter=false

See panoramio api docs

Foursquare

Example (replace MYTOKEN, you can make it on the docs site or by going to https://developer.foursquare.com/docs/explore#req=users/requests)

https://api.foursquare.com/v2/venues/search?ll=59.437,24.745&radius=1000&&oauth_token=MYTOKEN&v=20141001&limit=50&intent=browse

See foursquare dev docs

Google places

Example (replace MYKEY, you can make it on the docs site) https://maps.googleapis.com/maps/api/place/search/json?location=-33.8670522,151.1957362&radius=500&types=food&name=harbour&sensor=false&key=MYKEY

Twitter

Example url from old api, which is no longer functional: http://search.twitter.com/search.json?q=%40tallinn

See twitter dev docs and twitter search docs

API testing console: https://dev.twitter.com/rest/tools/console

Google search

Example (replace MYKEY, you can make it on the docs site) https://www.googleapis.com/customsearch/v1?key=MYKEY&q=lectures

See web search docs and custom-search docs

See also stackoverflow discussion on alternatives

Facebook graph api

https://developers.facebook.com/docs/graph-api

Quick example: https://graph.facebook.com/me?fields=id,name&access_token=MY_TOKEN

you can get MY_TOKEN from several places, for example: https://developers.facebook.com/tools/explorer?method=GET&path=me%3Ffields%3Did%2Cname&version=v2.1

Check also non-api scraping:

• http://www.pcworld.com/article/2056080/facebook-stalker-tool-uses-graph-search-for-powerful-data-mining.html
• https://github.com/milo2012/osintstalker

Python example for experimenting

#!/usr/bin/python

from math import *
import os
import urllib2
import json
import xml.dom.minidom

# NB! Replace YOURKEY and YOURTOKEN below with a key and token you make yourself
# on the corresponding api doc page

#url="http://api.wunderground.com/api/da9972c7aa341e74/conditions/q/CA/Tartu.json"
#url="http://www.panoramio.com/map/get_panoramas.php?set=full&size=original&from=0&to=99&minx=24.73&miny=59.428&maxx=24.76&maxy=59.445&mapfilter=false"
#url="https://api.foursquare.com/v2/venues/search?ll=59.437,24.745&radius=1000&&oauth_token=YOURTOKEN&v=20120103&limit=50&intent=browse"
#url="https://maps.googleapis.com/maps/api/place/search/json?location=-33.8670522,151.1957362&radius=500&sensor=false&key=YOURKEY"
url="http://search.twitter.com/search.json?q=tallinn"
#url="https://www.googleapis.com/customsearch/v1?key=YOURKEY&q=lectures"

def main():
  f=urllib2.urlopen(url)
  s=f.read()
  data=json.loads(s)
  #print data
  print json.dumps(data, sort_keys=True, indent=2)
  

def main2():
  req=urllib2.Request(url=url)
  rawresp=urllib2.urlopen(req)
  print "headers:\n"
  h=rawresp.info().headers
  for e in h: print e,
  s=rawresp.read()    
  data=json.loads(s)
  print "\ndata:\n"
  print json.dumps(data, sort_keys=True, indent=2)

  
main2()  

Microsoft Open Data web api standard

See

• Front page: http://www.odata.org
• First tutorial: http://www.odata.org/getting-started/basic-tutorial/
• Quick crucial docs: http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part2-url-conventions.html