Redpanda Unveils General Availability of Apache Iceberg Topics for Real-Time Data Querying in Enterprise Environments
In a significant leap forward for enterprise data management, Redpanda, the real-time data platform designed for the agentic enterprise, has announced the general availability of Apache Iceberg Topics. This groundbreaking feature empowers organizations to query streaming data as Iceberg tables without the need for complex ETL/ELT pipelines. Redpanda is the first company to deliver this functionality in a “Bring Your Own Cloud” (BYOC) environment, supporting all three major cloud providers—AWS, GCP, and Azure. With Apache Iceberg Topics, businesses can seamlessly integrate real-time operational insights with historical data within their existing data lakehouse environments, laying the foundation for AI agents and autonomous decision-making.
Bridging Real-Time and Batch Data for Next-Gen AI Applications
“Iceberg has become a groundbreaking technology for organizations seeking to unify their analytical and operational data substrates,” said Alex Gallego, founder and CEO of Redpanda. “At Redpanda, we’ve been committed to equipping enterprise data teams with cutting-edge tools that eliminate complexity and deliver exceptional data experiences. Iceberg will play a pivotal role in the future of agentic AI, serving as the backbone to bring streaming data into next-generation, industry-defining real-time and AI applications.”
Apache Iceberg has emerged as a cornerstone of modern data architectures, addressing critical limitations of traditional data lakes. By standardizing metadata to define groups of files as tables, Iceberg provides data lakes with the same benefits as data warehouses—transactional integrity, portability, and performance—while maintaining flexibility to use multiple query engines and data engineering tools. This makes it an ideal solution for enterprises looking to scale their data operations efficiently.
How Redpanda’s Iceberg Topics Works
Redpanda’s Iceberg Topics simplifies the process of creating Iceberg tables from Redpanda topics with a straightforward topic property setting. These tables are automatically registered with major Iceberg REST catalogs, including Snowflake Open Catalog, AWS Glue, Nessie, and others. Additionally, Redpanda offers advanced features such as:
- Custom Partitioning: Users can define custom partitioning schemes to optimize query performance.
- Built-In Dead Letter Queues: Invalid data is automatically filtered and set aside for targeted reprocessing.
- Automatic Schema Translation: Redpanda supports schema evolution and full fidelity across formats like Protobuf and Avro.
- SQL Accessibility: Analysts can leverage familiar SQL queries to analyze streaming data directly within their data lakehouse environments.
This seamless integration eliminates the need for complex data pipelines, significantly reducing the time to insight while enhancing operational efficiency.
Real-World Impact: Simplifying Streaming Data Infrastructure
Industry leaders are already recognizing the value of Redpanda’s Iceberg Topics. Mohsin Hussain, CTO at LiveRamp, shared his perspective: “As LiveRamp continues to innovate and scale solutions for our clients, Redpanda has been instrumental in strengthening our streaming data infrastructure. With the new Iceberg integration, we anticipate faster analytics, simplified data governance, and improved interoperability. This lays the groundwork for more intelligent, AI-driven customer experiences and delivers greater value to our customers.”
Key Benefits of Redpanda’s Iceberg Topics
- Simplified Data Access: Create Iceberg tables directly from Redpanda topics, enabling seamless access to streaming data for offline analytics.
- Elimination of Data Movement: Access data from analytics platforms without costly or complex data migrations.
- Enhanced Efficiency: Bypass custom data engineering jobs and Kafka Connect, reducing both time to insight and infrastructure costs.
- Schema Evolution and Discoverability: Leverage Iceberg’s robust schema management and secure, centralized REST data catalogs for improved data governance.
- Real-Time to Offline Integration: Blend real-time and historical data for richer analytics insights.
- Flexible Schema Support: Automatically derive Iceberg schemas from Redpanda’s schema registry or utilize default schemas for semi-structured views of data.
- Custom Partitioning: Define custom partitioning schemes to optimize query performance.
- Dead Letter Queue: Built-in dead letter queues ensure invalid data is filtered and available for reprocessing.
- SQL Accessibility: Enable analysts to use standard SQL queries to analyze streaming data within their data lakehouse environments.
Accelerating Innovation with Public Beta and Enterprise Availability
The beta version of Iceberg Topics was launched in December 2024, and the feature is now generally available to all Redpanda Enterprise Edition customers. Additionally, it is accessible as a public beta for Redpanda BYOC customers. Organizations interested in learning more can visit Redpanda’s website and register for an upcoming Tech Talk to explore the full potential of this innovative solution.
Redpanda at the Iceberg Summit
Redpanda will be showcasing its capabilities as a gold sponsor at the Iceberg Summit on Tuesday, April 8, at the Hyatt Regency SF Downtown. Attendees are invited to visit the Redpanda booth to learn more about Apache Iceberg Topics and attend the Redpanda session. This event provides an excellent opportunity for data professionals to gain insights into how Redpanda is transforming real-time data management and empowering enterprises to build smarter, AI-driven applications.
Empowering Enterprises with Unified Data Experiences
With the general availability of Apache Iceberg Topics, Redpanda is revolutionizing how organizations manage and query streaming data. By eliminating the barriers between real-time and batch data, Redpanda enables businesses to unlock deeper insights, streamline operations, and accelerate innovation. As enterprises increasingly adopt AI agents and autonomous decision-making systems, Redpanda’s Iceberg Topics provides the technological foundation needed to thrive in today’s data-driven world.
For organizations seeking to enhance their data infrastructure with a scalable, flexible, and efficient solution, Redpanda’s Iceberg Topics represents a game-changing advancement. Visit Redpanda’s website today to learn more and take the first step toward transforming your data strategy.
