KR 2019 homework part 1: overall task - Redigeerimiste ajalugu

Tammet: /* Task */

2019-02-20T11:49:13Z

Task

←Vanem redaktsioon		Redaktsioon: 20. veebruar 2019, kell 13:49
1. rida:		1. rida:

	== Task ==		== Task ==

			The end result of the lab is a presentation: ca five minutes during which you explain:
			* which systems you experimented with
			* which gave useful output from your input
			* examples of output from your input (show actual textual examples)

	The task of the first lab is to build and demonstrate a system which can		The task of the first lab is to build and demonstrate a system which can

Tammet: /* Task */

2019-02-20T11:41:24Z

Task

←Vanem redaktsioon		Redaktsioon: 20. veebruar 2019, kell 13:41
7. rida:		7. rida:
	** some NER-s do not give wiki URL-s, but just types. This is also useful.		** some NER-s do not give wiki URL-s, but just types. This is also useful.
	** for the second overall scenario (like wikidata) we do need wiki urls, though.		** for the second overall scenario (like wikidata) we do need wiki urls, though.
	* run and experiment with a pre-trained word vectorization system like word2vec: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.		* run and experiment with a (recommendably) pre-trained word vectorization system like word2vec: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.
	* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.		* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.

Tammet: /* Task */

2019-02-20T11:40:47Z

Task

←Vanem redaktsioon		Redaktsioon: 20. veebruar 2019, kell 13:40
7. rida:		7. rida:
	** some NER-s do not give wiki URL-s, but just types. This is also useful.		** some NER-s do not give wiki URL-s, but just types. This is also useful.
	** for the second overall scenario (like wikidata) we do need wiki urls, though.		** for the second overall scenario (like wikidata) we do need wiki urls, though.
	* run and experiment with a pre-trained word2vec ~~system~~: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.		* run and experiment with a pre-trained word vectorization system like word2vec: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.
	* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.		* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.

Tammet: /* Task */

2019-02-20T11:33:12Z

Task

←Vanem redaktsioon		Redaktsioon: 20. veebruar 2019, kell 13:33
3. rida:		3. rida:

	The task of the first lab is to build and demonstrate a system which can		The task of the first lab is to build and demonstrate a system which can
	* tokenize text (~~input and question~~)		* tokenize text (split any text into words: trivial)
	* run a [https://en.wikipedia.org/wiki/Named-entity_recognition NER tool] on the text giving wikipedia url-s as output. See [https://gate.d5.mpi-inf.mpg.de/webaida/ webaida]. Notice that:		* run a [https://en.wikipedia.org/wiki/Named-entity_recognition NER tool] on the text giving wikipedia url-s as output. See [https://gate.d5.mpi-inf.mpg.de/webaida/ webaida]. Notice that:
	** some NER-s do not give wiki URL-s, but just types. This is also useful.		** some NER-s do not give wiki URL-s, but just types. This is also useful.

Tammet: /* Task */

2019-02-20T11:31:26Z

Task

←Vanem redaktsioon		Redaktsioon: 20. veebruar 2019, kell 13:31
3. rida:		3. rida:

	The task of the first lab is to build and demonstrate a system which can		The task of the first lab is to build and demonstrate a system which can
	* tokenize ~~the~~ text (input and question)		* tokenize text (input and question)
	* run a [https://en.wikipedia.org/wiki/Named-entity_recognition NER tool] on the text giving wikipedia url-s as output. See [https://gate.d5.mpi-inf.mpg.de/webaida/ webaida]. Notice that:		* run a [https://en.wikipedia.org/wiki/Named-entity_recognition NER tool] on the text giving wikipedia url-s as output. See [https://gate.d5.mpi-inf.mpg.de/webaida/ webaida]. Notice that:
	** some NER-s do not give wiki URL-s, but just types. This is also useful.		** some NER-s do not give wiki URL-s, but just types. This is also useful.

Tammet – 6. veebruar 2019, kell 08:56

2019-02-06T08:56:50Z

←Vanem redaktsioon		Redaktsioon: 6. veebruar 2019, kell 10:56
9. rida:		9. rida:
	* run and experiment with a pre-trained word2vec system: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.		* run and experiment with a pre-trained word2vec system: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.
	* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.		* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.

			You do not have to be wildly successful with all these subtasks to pass. You will, though, have to try out each of these subtasks and report on how it went (got ok results, got bad results, could not get it to work, ...)

	The second lab will build upon the results here. In particular, some choices in the second lab depend on what you managed to do with ner, word2vec and parsers.		The second lab will build upon the results here. In particular, some choices in the second lab depend on what you managed to do with ner, word2vec and parsers.

Tammet: /* Popular toolkits for NLP */

2019-02-06T08:54:22Z

Popular toolkits for NLP

←Vanem redaktsioon		Redaktsioon: 6. veebruar 2019, kell 10:54
25. rida:		25. rida:
	* [http://www.nltk.org/ NLTK: the main Python toolkit], see also [http://pixelmonkey.org/pub/nlp-training/ this tutorial]		* [http://www.nltk.org/ NLTK: the main Python toolkit], see also [http://pixelmonkey.org/pub/nlp-training/ this tutorial]
	* [https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html Google SyntaxNet] (see [https://github.com/tensorflow/models/tree/master/syntaxnet in github])		* [https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html Google SyntaxNet] (see [https://github.com/tensorflow/models/tree/master/syntaxnet in github])
	* [http://nlp.stanford.edu/software/corenlp.shtml CoreNLP: the main Stanford NLP tool] in the context of a larger set of [http://nlp.stanford.edu/software/ Stanford NLP toolkits] like [http://nlp.stanford.edu/software/CRF-NER.shtml stanford NER etc]		* [http://nlp.stanford.edu/software/corenlp.shtml CoreNLP: the main Stanford NLP tool] in the context of a larger set of [http://nlp.stanford.edu/software/ Stanford NLP toolkits] like [http://nlp.stanford.edu/software/CRF-NER.shtml stanford NER etc] [http://pythonprogramming.net/named-entity-recognition-nltk-python/ and this NER tutorial]
	[http://pythonprogramming.net/named-entity-recognition-nltk-python/ and this NER tutorial]
	* [http://www.clips.ua.ac.be/pattern Pattern toolkit for Python]		* [http://www.clips.ua.ac.be/pattern Pattern toolkit for Python]
	* [http://opennlp.apache.org/ opennlp]		* [http://opennlp.apache.org/ opennlp]

Tammet: /* Task */

2019-02-06T08:53:52Z

Task

←Vanem redaktsioon		Redaktsioon: 6. veebruar 2019, kell 10:53
9. rida:		9. rida:
	* run and experiment with a pre-trained word2vec system: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.		* run and experiment with a pre-trained word2vec system: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.
	* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.		* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.

			The second lab will build upon the results here. In particular, some choices in the second lab depend on what you managed to do with ner, word2vec and parsers.

	== Technology and tools ==		== Technology and tools ==

Tammet: /* Task */

2019-02-06T08:52:40Z

Task

←Vanem redaktsioon		Redaktsioon: 6. veebruar 2019, kell 10:52
7. rida:		7. rida:
	** some NER-s do not give wiki URL-s, but just types. This is also useful.		** some NER-s do not give wiki URL-s, but just types. This is also useful.
	** for the second overall scenario (like wikidata) we do need wiki urls, though.		** for the second overall scenario (like wikidata) we do need wiki urls, though.
	* run a parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.		* run and experiment with a pre-trained word2vec system: as an example, [https://medium.com/@martinpella/how-to-use-pre-trained-word-embeddings-in-pytorch-71ca59249f76 see this tutorial]. We will come back to that.
			* run and experiment with parser tool on the text. There are several ways to parse, some more useful for our purposes than others. We will come back to that.

	== Technology and tools ==		== Technology and tools ==

Tammet – 6. veebruar 2019, kell 08:48

2019-02-06T08:48:29Z

←Vanem redaktsioon		Redaktsioon: 6. veebruar 2019, kell 10:48
14. rida:		14. rida:

	You are free to use NLP tools and API-s and datasets, but your program should drive them from the beginning to end (ie your program takes input files, calls tools, modifies and prints output). It is better to use the tools than not to use them!		You are free to use NLP tools and API-s and datasets, but your program should drive them from the beginning to end (ie your program takes input files, calls tools, modifies and prints output). It is better to use the tools than not to use them!

			== Popular toolkits for NLP ==

			The recommendation is to take the first (spacy) or the second (nltk) toolkit.

			* [https://spacy.io Spacy toolkit for Python]
			* [http://www.nltk.org/ NLTK: the main Python toolkit], see also [http://pixelmonkey.org/pub/nlp-training/ this tutorial]
			* [https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html Google SyntaxNet] (see [https://github.com/tensorflow/models/tree/master/syntaxnet in github])
			* [http://nlp.stanford.edu/software/corenlp.shtml CoreNLP: the main Stanford NLP tool] in the context of a larger set of [http://nlp.stanford.edu/software/ Stanford NLP toolkits] like [http://nlp.stanford.edu/software/CRF-NER.shtml stanford NER etc]
			[http://pythonprogramming.net/named-entity-recognition-nltk-python/ and this NER tutorial]
			* [http://www.clips.ua.ac.be/pattern Pattern toolkit for Python]
			* [http://opennlp.apache.org/ opennlp]
			* [https://pypi.python.org/pypi/PyNLPl/ PyNLP for Python]
			* [http://williamjturkel.net/2013/06/30/named-entity-recognition-with-command-line-tools-in-linux/ NER tutorial for Linux] in the context of [http://williamjturkel.net/category/linux-unix/ a larger practical tutorial]

			=== Web APIs ===

			* [https://cloud.google.com/natural-language/ Google cloud natural language API]
			* [http://new.opencalais.com opencalais] (free registration required)