Verwandte Artikel zu Demand-based Data Stream Gathering, Processing, and...

Demand-based Data Stream Gathering, Processing, and Transmission: Efficient Solutions for Real-Time Data Analytics in the Internet of Things - Softcover

Traub, Jonas

 
9783752671254: Demand-based Data Stream Gathering, Processing, and Transmission: Efficient Solutions for Real-Time Data Analytics in the Internet of Things
Softcover
ISBN 10: 3752671254 ISBN 13: 9783752671254
Verlag: BoD – Books on Demand, 2021

Inhaltsangabe

This book presents an end-to-end architecture for demand-based data stream gathering, processing, and transmission. The Internet of Things (IoT) consists of billions of devices which form a cloud of network connected sensor nodes. These sensor nodes supply a vast number of data streams with massive amounts of sensor data. Real-time sensor data enables diverse applications including traffic-aware navigation, machine monitoring, and home automation. Current stream processing pipelines are demand-oblivious, which means that they gather, transmit, and process as much data as possible. In contrast, a demand-based processing pipeline uses requirement specifications of data consumers, such as failure tolerances and latency limitations, to save resources. Our solution unifies the way applications express their data demands, i.e., their requirements with respect to their input streams. This unification allows for multiplexing the data demands of all concurrently running applications. On sensor nodes, we schedule sensor reads based on the data demands of all applications, which saves up to 87% in sensor reads and data transfers in our experiments with real-world sensor data. Our demand-based control layer optimizes the data acquisition from thousands of sensors. We introduce time coherence as a fundamental data characteristic. Time coherence is the delay between the first and the last sensor read that contribute values to a tuple. A large scale parameter exploration shows that our solution scales to large numbers of sensors and operates reliably under varying latency and coherence constraints. On stream analysis systems, we tackle the problem of efficient window aggregation. We contribute a general aggregation technique, which adapts to four key workload characteristics: Stream (dis)order, aggregation types, window types, and window measures. Our experiments show that our solution outperforms alternative solutions by an order of magnitude in throughput, which prevents expensive system scale-out. We further derive data demands from visualization needs of applications and make these data demands available to streaming systems such as Apache Flink. This enables streaming systems to pre-process data with respect to changing visualization needs. Experiments show that our solution reliably prevents overloads when data rates increase.

Die Inhaltsangabe kann sich auf eine andere Ausgabe dieses Titels beziehen.

Über die Autorin bzw. den Autor

Dr. Jonas Traub is an expert for databases, streaming systems, and the Internet of Things, passionate to provide efficient software solutions. He received his doctoral degree from Technische Universität Berlin where he worked as a researcher in the Database Systems and Information Management Group lead by Prof. Dr. Volker Markl. He continuously publishes at top-tier journals and conferences in the field of database systems and stream processing such as VLDB, ACM SIGMOD, IEEE ICDE, EDBT, ACM CIKM and ACM SoCC.

Von der hinteren Coverseite

This book presents an end-to-end architecture for demand-based data stream gathering, processing, and transmission. The Internet of Things (IoT) consists of billions of devices which form a cloud of network connected sensor nodes. These sensor nodes supply a vast number of data streams with massive amounts of sensor data. Real-time sensor data enables diverse applications including traffic-aware navigation, machine monitoring, and home automation. Current stream processing pipelines are demand-oblivious, which means that they gather, transmit, and process as much data as possible. In contrast, a demand-based processing pipeline uses requirement specifications of data consumers, such as failure tolerances and latency limitations, to save resources. Our solution unifies the way applications express their data demands, i.e., their requirements with respect to their input streams. This unification allows for multiplexing the data demands of all concurrently running applications. On sensor nodes, we schedule sensor reads based on the data demands of all applications, which saves up to 87% in sensor reads and data transfers in our experiments with real-world sensor data. Our demand-based control layer optimizes the data acquisition from thousands of sensors. We introduce time coherence as a fundamental data characteristic. Time coherence is the delay between the first and the last sensor read that contribute values to a tuple. A large scale parameter exploration shows that our solution scales to large numbers of sensors and operates reliably under varying latency and coherence constraints. On stream analysis systems, we tackle the problem of efficient window aggregation. We contribute a general aggregation technique, which adapts to four key workload characteristics: Stream (dis)order, aggregation types, window types, and window measures. Our experiments show that our solution outperforms alternative solutions by an order of magnitude in throughput, which prevents expensive system scale-out. We further derive data demands from visualization needs of applications and make these data demands available to streaming systems such as Apache Flink. This enables streaming systems to pre-process data with respect to changing visualization needs. Experiments show that our solution reliably prevents overloads when data rates increase.

„Über diesen Titel“ kann sich auf eine andere Ausgabe dieses Titels beziehen.

Verlag
BoD – Books on Demand
Erscheinungsdatum
2021
Sprache
Englisch
ISBN 10 
3752671254
ISBN 13 
9783752671254
Einband
Taschenbuch
Auflage
1
Anzahl der Seiten
206
Kontakt zum Hersteller
Books on Demand GmbH
bod@bod.de

Überseering 33
Hamburg
22297
Deutschland