Lehekülje Mapreduce katsetused Azure platvormil lähteteksti vaatamine

== MapReduce Azure platvormil ==

Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-introduction "Getting started" artikkel]

Katsetustes kasutame shelli, Pythonit ja stream-I/O-d, mis ei eelda mingite pakettide ja arenduskeskkondade installimist.

Microsofti [https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-streaming-python Pythoni tutorial]

[https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/FileSystemShell.html hdfs käsud]

=== Katsetus: word count ===

<code>mapper.py</code>

<pre>
#!/usr/bin/env python

# Use the sys module
import sys

# 'file' in this case is STDIN
def read_input(file):
    # Split each line into words
    for line in file:
        yield line.split()

def main(separator='\t'):
    # Read the data using read_input
    data = read_input(sys.stdin)
    # Process each words returned from read_input
    for words in data:
        # Process each word
        for word in words:
            # Write to STDOUT
            print('%s%s%d' % (word, separator, 1))

if __name__ == "__main__":
    main()
</pre>

<code>reducer.py</code>

<pre>
#!/usr/bin/env python

# import modules
from itertools import groupby
from operator import itemgetter
import sys

# 'file' in this case is STDIN
def read_mapper_output(file, separator='\t'):
    # Go through each line
    for line in file:
        # Strip out the separator character
        yield line.rstrip().split(separator, 1)

def main(separator='\t'):
    # Read the data using read_mapper_output
    data = read_mapper_output(sys.stdin, separator=separator)
    # Group words and counts into 'group'
    #   Since MapReduce is a distributed process, each word
    #   may have multiple counts. 'group' will have all counts
    #   which can be retrieved using the word as the key.
    for current_word, group in groupby(data, itemgetter(0)):
        try:
            # For each word, pull the count(s) for the word
            #   from 'group' and create a total count
            total_count = sum(int(count) for current_word, count in group)
            # Write to stdout
            print("%s%s%d" % (current_word, separator, total_count))
        except ValueError:
            # Count was not a number, so do nothing
            pass

if __name__ == "__main__":
    main()
</pre>


Mingid sisendfailid on juba olemas:

  hdfs dfs -ls /example/data/

Tõmbame nende pealt käima, <code>/example</code> on kirjutatav storage

  yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar -files mapper.py,reducer.py -mapper mapper.py -reducer reducer.py -input wasbs:///example/data/gutenberg/davinci.txt -output wasbs:///example/wordcountout

Väljund

  hdfs dfs -text /example/wordcountout/part-00000 | head -50

=== Katsetus: sort ===

<code>mapper.py</code>

<pre>

</pre>

<code>reducer.py</code>

<pre>

</pre>


=== Katsetus: otsing ===

<code>mapper.py</code>

<pre>

</pre>

<code>reducer.py</code>

<pre>

</pre>