zoom_in Click to enlarge

Google Cloud Dataflow

▲ ▼

language

description Google Cloud Dataflow Overview

Google Cloud Dataflow is a fully managed, serverless service for stream and batch data processing. Built on the Apache Beam model, it allows developers to write code once and execute it on either streaming or batch pipelines. Dataflow excels at auto-scaling, dynamically adjusting resources based on the incoming data volume, which eliminates the need for manual cluster management. It is deeply integrated with the Google Cloud ecosystem, making it the most seamless choice for teams already using BigQuery, Pub/Sub, and Cloud Storage.

Its 'exactly-once' processing guarantee is highly reliable and easy to implement.

help Google Cloud Dataflow FAQ

What is Google Cloud Dataflow used for?

Dataflow runs managed pipelines that transform, aggregate, and move data in either batch or real time. Common destinations and sources include Pub/Sub, BigQuery, Cloud Storage, Kafka, and database systems.

Is Google Cloud Dataflow based on Apache Beam?

Yes, Dataflow uses the Apache Beam programming model for unified batch and streaming pipelines. A Beam pipeline can therefore express processing logic that is portable across supported Beam runners rather than being tied only to one execution style.

How does Dataflow scale a streaming pipeline?

Dataflow can adjust worker capacity as the workload changes, subject to the limits and settings configured for the job. Streaming costs can include worker vCPU and memory as well as Streaming Engine resources or processed data.

How is Google Cloud Dataflow priced?

Dataflow uses usage-based billing rather than a simple fixed software license. Google lists charges for resources such as vCPU, memory, batch shuffle data, and streaming resources, with current examples including $0.069 per vCPU hour for a standard batch or streaming worker vCPU.

swap_horiz

Looking for Google Cloud Dataflow alternatives? Compare top competitors ranked & scored

arrow_forward