Innosoft Gulf · Big Data · Data Engineering Pathway

Big Data Foundations, Data Lakes & Modern Storage

Distributed Architecture, HDFS, Apache Parquet, S3-Compatible Object Storage and Apache Iceberg

Big Data Professional Programme · Course 1 of 4

Course Description

This hands-on course introduces the architectural foundations of modern Big Data platforms and explains how distributed systems scale beyond a single machine. Participants explore how storage, processing and streaming components work together and how large datasets are organised for scalable analytics.

The course begins with Big Data and distributed-system architecture and progresses to Hadoop HDFS, Apache Parquet, S3-compatible object storage, data-lake architectures and an introduction to Apache Iceberg and lakehouse storage.

Participants work with substantial real-world datasets and complete practical projects covering both Big Data architecture design and modern data-lake storage.

Course Information

Duration	12 instructor-led hours
Format	4 sessions × 3 hours
Delivery	Instructor-led training in person at Dubai Knowledge Park or live online
Course Fee	AED 3,500 VAT inclusive

Who Should Attend

Professionals and graduates seeking to develop practical skills in Big Data and Data Engineering, including those transitioning from software development, IT, database administration, analytics, or other technical backgrounds. Prior experience with Big Data technologies is not required, although basic familiarity with SQL and a programming language such as Python or Scala is recommended.

Recommended Prerequisites

Basic familiarity with a programming language, preferably Python.
Working knowledge of SQL and relational data concepts.
Comfort using a command-line environment is beneficial.
No prior experience with Spark, Kafka, Hadoop or other Big Data technologies is required.

Learning Outcomes

By the end of this course, participants will be able to:

Explain Big Data architecture, distributed-systems principles and components of modern data platforms.
Design and organise large-scale datasets using distributed storage, Parquet, data lakes and lakehouse architectures.

Hands-On Lab Environment

Participants work on Innosoft Gulf’s multi-node Big Data infrastructure using substantial real-world datasets. The environment provides hands-on access to distributed storage, Apache Spark, Kafka, object storage, streaming and analytical database services, allowing participants to build, observe and troubleshoot complete Big Data pipelines in a production-like setting.

Course Curriculum

Module 1

Big Data Foundations and Distributed Architecture

This module introduces how modern Big Data platforms scale beyond a single machine and how distributed storage, processing and streaming components work together to handle large and continuously growing datasets.

Key Topics

Big Data characteristics and business drivers
Scale-up versus scale-out architectures
Distributed computing, partitioning and fault tolerance
Batch, streaming and interactive processing
Data warehouses, data lakes and lakehouse architectures
Roles of Spark, Kafka, distributed storage and analytical databases
Common scalability and failure challenges

Lab Environment Overview

Participants are introduced to Innosoft Gulf’s multi-node Big Data environment and see how storage, distributed processing, streaming and analytical services fit together within a production-style architecture.

Practical Project — Design a Big Data Architecture

Participants analyse a high-volume data scenario, such as live market data, and design an end-to-end architecture for ingesting, storing, processing and serving the data using the lab environment as a reference.

Outcome: A practical Big Data architecture that participants can explain and defend.

Module 2

Distributed Storage, Data Lakes and Object Storage

Participants explore how large datasets are stored and organised for scalable analytics, beginning with HDFS and progressing to modern S3-compatible object storage and data-lake architectures.

Key Topics

HDFS architecture, blocks, replication and fault tolerance
Organising and accessing large datasets in HDFS
Apache Parquet and efficient analytical storage
Dataset partitioning and the small-files problem
S3-compatible object storage and bucket/object concepts
HDFS versus object-storage architectures
Designing scalable data-lake storage layouts
Introduction to Apache Iceberg and lakehouse storage

Practical Project — Build a Modern Data Lake

Participants work with a substantial real-world dataset currently stored in HDFS, organise and optimise it using Parquet and partitioning, and then explore how the same dataset can be stored using S3-compatible object storage.

Outcome: A practical understanding of how to organise large datasets across both distributed file systems and modern object-storage architectures.

Part of the Big Data Professional Programme

This is Course 1 of 4 in the 48-hour Big Data Professional Programme. Participants may take this course independently or continue through the complete professional pathway.

Next Course: Distributed Data Processing with Apache Spark →

View the Full Big Data Professional Programme →

Assessment & Programme Certification

Assessment in this course includes hands-on exercises and the practical projects described above. Across the full programme, assessment also includes the final integrated Big Data platform project.

Participants who successfully complete the programme and meet the assessment requirements will receive an Innosoft Gulf Certificate of Completion, eligible for attestation by Dubai’s Knowledge and Human Development Authority (KHDA).

Ready to Join This Course?

Big Data Foundations, Data Lakes & Modern Storage

Apply Now View Training Calendar

Instructor-led training in person at Dubai Knowledge Park or live online

Innosoft Gulf · Dubai Knowledge Park · Big Data Foundations, Data Lakes & Modern Storage