Big Data: Architecture & infrastructure

Introduction to Big Data

1. Trainer and participants presentations
   

2. Big Data principles and challenges
   

3. The key numbers
   

4. The challenges (ROI / Confidentiality)
   

5. Big Data Architecture Concept

Big Data technologies

1. Market tools (MapR, Watson, Oracle Exalead, Microsoft, Teradata, Talend, Qlik Sense, AWS...)

2. Storage methods

3. Hot data

4. Lukewarm data

5. Archiving

Languages:

Python, Matlab, R, …

State of the art of Big Data

1. The main actors
   

2. The different professions of Big Data
   

Hardware Architecture

1. Storage
   

2. CPU/GPU
   

3. Memory
   

4. Network
   

5. Distributed Systems
   

Technical Architecture

1. Data Quality: use of an Open Data flow operation with Talend Data Quality
   

2. Data storage: Hadoop (HDFS), NoSQL (Cassandra, MongoDB)
   

3. Data dissemination
   

   1. Streaming with Apache Kafka, Amazon AWS Kinesis
      

4. Real-time or deferred processing
   

   1. Apache Spark
      

   2. ELK
      

      1. ElasticSearch
         

      2. Logstash
         

      3. Kibana
         

5. Distributed architectures: Hadoop clustering
   

6. Platform supervision: App Dynamix, Ambari
   

Application Architecture (case study)

1. The different stages of data management
   

2. Ingestion from the source
   

3. The transformation
   

4. Restitution / visualization
   

5. Predictive analysis
   

Workshop: 360 customer analysis

Business applications (use cases)

1. Text Analytics
   

2. Fraud detection
   

3. Customer targeting
   

Risks of the Big Data project

1. Business risks related to the framing of the scope
   

2. Strategic risks and lack of sponsorship
   

3. The KPIs
   

4. The maturity of market solutions
   

Safety, ethics & legal issues

1. Ensure data protection
   

   1. The anonymization of data
      

   2. Integrity check
      

   3. Encryption of data
      

2. What is the blockchain?
   

3. Use case: Bitcoin
   

Conclusion (global debrief)