Big Data Cluster for teaching and research

• 2 namenodes (on c100: primary, on c101 secondary namenode)
• 18 datanodes c101--c118
• an administrative server h1 as
  • Cloudera Manager server
  • backup of administrative data
• a ZFS file server lbdnfs01 for /home with 300TB of storage space

• two XeonE5-2650v4 CPUs with 24 cores (total of 48 cores per node, 864 total worker cores)
• 256GB RAM (total of 4.5TB memory available to worker nodes of the whole cluster)
• four hard disks, each with a capacity of 4TB (total of 16TB per node, 288TB total for worker nodes)

Apart from two extra Ethernet devices for external connections on h1 and on c100, all nodes have the same hardware configurations. All ethernet connections (external and inter-node) support a speed of 10Gb/s.

HDFS configuration

• current version: Hadoop 3
• block size: 128 MiB
• default replication factor: 3

Course instructors can request accounts for TU students by providing the following information:

• a CSV file in the TISS format:

  "0123456";"Lastname";"Firstname";"e0123456@student.tuwien.ac.at"

• a unique name for their class (such as ADBS19)
• an expiry date for the accounts

import pyspark
import random
sc = pyspark.SparkContext(appName="Pi")
num_samples = 10000
def inside(p):     
  x, y = random.random(), random.random()
  return x*x + y*y < 1
count = sc.parallelize(range(0, num_samples)).filter(inside).count()
pi = 4 * count / num_samples
print(pi)
sc.stop()