Big Data Professional Programme

Course 1 · 12 Hours

Big Data Foundations, Data Lakes & Modern Storage

Develop the architectural and storage foundations required to understand modern Big Data platforms. Participants explore how distributed systems scale and how large datasets are organised across HDFS, Parquet, data lakes and S3-compatible object storage.

Key Areas

• Big Data characteristics and business drivers
• Scale-up versus scale-out architectures
• Distributed computing, partitioning and fault tolerance
• Batch, streaming and interactive processing
• Data warehouses, data lakes and lakehouse architectures
• Roles of Spark, Kafka, distributed storage and analytical databases
• Common scalability and failure challenges
• HDFS architecture, blocks, replication and fault tolerance
• Organising and accessing large datasets in HDFS
• Apache Parquet and efficient analytical storage
• Dataset partitioning and the small-files problem
• S3-compatible object storage and bucket/object concepts
• HDFS versus object-storage architectures
• Designing scalable data-lake storage layouts
• Introduction to Apache Iceberg and lakehouse storage

Explore Course 1 →

Course 2 · 12 Hours

Distributed Data Processing with Apache Spark

Build distributed data-processing pipelines with Apache Spark, transforming large datasets stored across HDFS and S3-compatible object storage into usable analytical data products.

Key Areas

• Spark architecture: drivers, executors, jobs, stages and tasks
• DataFrames, Spark SQL and schema management
• RDD concepts and lineage as Spark foundations
• Transformations, aggregations, joins and window operations
• Reading and writing CSV, JSON and Parquet across HDFS and S3-compatible storage
• Partitioning, shuffles, caching and persistence
• Building reusable PySpark processing pipelines
• Submitting and monitoring distributed Spark applications

Explore Course 2 →

Course 3 · 12 Hours

Real-Time Data Engineering with Kafka & Spark Streaming

Build reliable event-streaming and real-time analytics pipelines using Apache Kafka and Spark Structured Streaming, from event ingestion and replay through time-based processing and downstream analytical storage.

Apache Kafka

• Event-driven and streaming data architectures
• Kafka brokers, topics, partitions and replication
• Producers, consumers and consumer groups
• Message keys, partitioning and ordering
• Offsets, retention and replay
• Delivery semantics and reliability
• Event serialisation using JSON and schema-based formats
• Monitoring Kafka throughput and consumer lag
• Failure recovery and resilient streaming design

Spark Structured Streaming

• Structured Streaming architecture and streaming DataFrames
• Reading Kafka streams with Spark
• Event time and processing time
• Windowed aggregations
• Watermarking and late-arriving data
• Stateful stream processing
• Checkpointing and recovery
• Streaming data quality and validation
• Writing processed streams to downstream storage
• Monitoring and troubleshooting streaming applications

Explore Course 3 →

Course 4 · 12 Hours

Production Data Engineering & Analytical Serving

Complete the pathway by building analytical serving layers, orchestrating production workflows and integrating distributed storage, Spark, Kafka, stream processing and analytical databases into a complete production-style Big Data platform.

PostgreSQL & TimescaleDB

• Role of the analytical serving layer in a Big Data platform
• Relational and time-series data modelling
• PostgreSQL and TimescaleDB architecture
• Designing tables for high-volume time-series data
• Indexing and time-based partitioning
• Hypertables and continuous aggregates
• Loading batch and streaming analytical results
• Time-range queries and analytical aggregations
• Query performance and data-retention strategies
• Connecting analytical data to dashboards and applications

Production Data Engineering

• Pipeline orchestration and scheduling with Apache Airflow
• Data quality and validation
• Monitoring, logging and alerting
• Security and access control
• Performance and capacity considerations
• Failure detection, recovery and troubleshooting
• Data governance and retention
• Integrating batch, streaming, storage and serving components

Explore Course 4 →

By the end of the programme, participants will be able to:

• Explain Big Data architecture, distributed-systems principles and components of modern data platforms.
• Design and organise large-scale datasets using distributed storage, Parquet, data lakes and lakehouse architectures.
• Build and optimise batch-processing and data-engineering pipelines using Apache Spark.
• Design reliable event-streaming architectures using Apache Kafka.
• Build real-time processing pipelines using Kafka and Spark Structured Streaming.
• Manage historical and streaming data using modern table and storage architectures, including Apache Iceberg and S3-compatible object storage.
• Store, query and serve derived analytical results using PostgreSQL/TimescaleDB for time-series and analytical workloads.
• Apply data quality, security, governance, monitoring and performance practices across Big Data pipelines.
• Design, troubleshoot and operate a complete integrated Big Data platform in a production-like environment.