Credit Points 10

Description Data Engineers design systems that gather, store and present data for critical decision-making and further analysis. While there are several software tools to set up data systems, the dynamic nature of the data landscape necessitates that Data Engineers possess an in-depth understanding of fundamental concepts and agile processes. In this subject, students will acquire a foundational understanding of key data engineering concepts, enabling them to design and construct robust data systems. Every facet of the data engineering lifecycle will be meticulously explored, providing students with hands-on exposure to state of the art data software systems. Additionally, students will delve into essential topics such as data security, ethics, and data ownership.

School Computer, Data & Math Sciences

Student Contribution Band HECS Band 2 10cp

Check your fees via the Fees page.

Level Postgraduate Coursework Level 7 subject

Pre-requisite(s) COMP 7024

Assumed Knowledge

Basic programming knowledge is required.

Learning Outcomes

1. Set-up appropriate software for the ingestion of data.
2. Compute and allocate appropriate storage for data.
3. Write scripts to query and transform data.
4. Provide appropriate interfaces for data access.
5. Identify ethical and security issues of storing and accessing data.

Subject Content

Data Engineering Foundations

• Data Engineering Described
• Data Engineering Lifecycle
• Designing Good Data Architecture
• Choosing Technologies

Data Engineering Lifecycle

• Data Generation Source Systems
• Storage Methods
• Ingestion
• Queries, Modelling and Transformation
• Serving data analytics, machine learning, reverse ETL

Data Governance

• Security, Privacy, Ethics, Data Sovereignty (IGA)

Future of Data Engineering