Discover actionable steps to maintain healthy data pipelines to promote data observability within your teams with this essential guide to elevating data engineering practices
Key Features
- Learn how to monitor your data pipelines in a scalable way
- Apply real-life use cases and projects to gain hands-on experience in implementing data observability
- Instil trust in your pipelines among data producers and consumers alike
- Purchase of the print or Kindle book includes a free PDF eBook
Book Description
In the age of information, strategic management of data is critical to organizational success. The constant challenge lies in maintaining data accuracy and preventing data pipelines from breaking. Data Observability for Data Engineering is your definitive guide to implementing data observability successfully in your organization.
This book unveils the power of data observability, a fusion of techniques and methods that allow you to monitor and validate the health of your data. You'll see how it builds on data quality monitoring and understand its significance from the data engineering perspective. Once you're familiar with the techniques and elements of data observability, you'll get hands-on with a practical Python project to reinforce what you've learned. Toward the end of the book, you'll apply your expertise to explore diverse use cases and experiment with projects to seamlessly implement data observability in your organization.
Equipped with the mastery of data observability intricacies, you'll be able to make your organization future-ready and resilient and never worry about the quality of your data pipelines again.
What you will learn
- Implement a data observability approach to enhance the quality of data pipelines
- Collect and analyze key metrics through coding examples
- Apply monkey patching in a Python module
- Manage the costs and risks associated with your data pipeline
- Understand the main techniques for collecting observability metrics
- Implement monitoring techniques for analytics pipelines in production
- Build and maintain a statistics engine continuously
Who this book is for
This book is for data engineers, data architects, data analysts, and data scientists who have encountered issues with broken data pipelines or dashboards. Organizations seeking to adopt data observability practices and managers responsible for data quality and processes will find this book especially useful to increase the confidence of data consumers and raise awareness among producers regarding their data pipelines.
Table of Contents
- Fundamentals of Data Quality Monitoring
- Fundamentals of Data Observability
- Data Observability techniques
- Data Observability elements
- Defining rules on indicators
- Root cause analysis
- Optimizing data pipelines
- Introducing and changing culture in the team
- Data observability checklist
- Use Cases
Michele Pinto is the Head of Engineering at Kensu. With over 15 years of experience, Michele has a great knack for understanding how data observability and data engineering are closely linked. He started his career as a software engineer and has worked since then in various roles, such as big data engineer, big data architect, head of data and until recently he was a Head of Engineering. He has a great community presence and believes in giving back to the community. He has also been a teacher for Digital Product Management Master TAG Innovation School in Milan, Italy. His collaboration on the book has been prompt, swift, eager, and very invested.
Sammy El Khammal works at Kensu. He started off as a field engineer and worked his way up to the position of product manager. In the past, he has also worked with Mercedes as their Business Development Analyst - Intern. He has also been an O'Reilly teacher for 3 workshops on data quality, lineage monitoring, and data observability. During that time, he provided some brilliant insights, very responsive behaviour, and immense talent and determination.
