Ahmet Yavuz Demir’s Post

This article goes into all those configurations regarding setting up a Spark catalog for Apache Iceberg and breaks them down so you can configure any catalog you need. https://lnkd.in/ePkm2KvS

Configuring an Apache Spark session to work with different Apache Iceberg catalogs can sometimes include a lot of options. In this article, Developer Advocate, Alex Merced helps breakdown the relevant Spark configurations to configure all types of catalogs with for Apache Iceberg. https://bit.ly/3OYfqVK #ApacheIceberg #ApacheSpark

If you are using apache iceberg, i recommend you read This blog post. Great insights!!!

I was wondering, how many people are using Apache Iceberg along with Apache Spark in production? I understand the advantages of using Iceberg with respect to Delta Lake, it gives you the ability to maintain a history and granular control over the table schemas, that is, if you are going to change the schema very often, and you are going to insert data into those tables, Iceberg will allow you better control over the maintenance of the previous information, so it may be interesting to delegate the management of the schemas to Iceberg and the ACID transactional hard work to delta lake. The fact is that I have never seen or had the need to change the schema of a table so often, since it is something that when done has to be super justified, it has to go through many permissions, blablabla. How many people use Apache Iceberg and Apache Spark Delta Lake in production?

Check out this Dremio article about configuring your Apache Iceberg catalog with Apache Spark.

Ballista (Rust) vs Apache Spark. A Tale of Woe. Sometimes it seems like the Data Engineering landscape is starting to shoot off into infinity. With the rise of Rust, new tools like DuckDB, Polars, and whatever else, things do seem to shifting at a fundamental level. It seems like there is someone at the base of a titering rock with a crowbar, picking and […] The post Ballista (Rust) vs Apache Spark. A Tale of Woe. appeared first on Confessions of a Data Guy.

Here are interesting recent updates about Apache Iceberg ✔️ 7 best practices for a successful Apache Iceberg implementation The Iceberg table format brings data warehouse characteristics to cloud object storage – including consistent SQL behavior, hidden partitioning and schema evolution. However, as with any new technology, there are new techniques you'll need to master in order to succeed. In this webinar Dan Weeks, Tabular CTO and Apache Iceberg PMC member, will cover the most important practices you need to develop to ensure your Iceberg deployment exceeds your expectations for performance, cost, security and simple operation. Dan will cover best practices across several areas including: ▪️ Selecting a Catalog ▪️ Ingesting Data ▪️ Connecting Compute ▪️ Maintaining Tables ▪️ Optimizing Performance ▪️ Enforcing Security ▪️ Data Privacy and Compliance Link - https://lnkd.in/gHStsp-m ✔️ PyIceberg 0.6.0: Write support - Write to Apache Iceberg from Python This blog gives an introduction to the Python API to write an Iceberg table, looking at how this is different from writing just Parquet files, and what’s next. PyIceberg writes an in-memory Arrow table to an Iceberg table on your object store. Arrow was chosen because it is a popular in-memory format, which makes it easy to integrate into the Python data science ecosystem. Link - https://lnkd.in/gSX44w6E #apacheiceberg 👉 I publish posts on #dataengineering, #datastreaming, #analytics, #realtimeanalytics and #architecture regularly. Follow me or see my articles on https://jaehyeon.me/.

Looking to supercharge your Apache Spark applications? The secret lies in understanding and configuring the partition size. In my latest article, I delve into the factors that influence partitioning Discover how to determine the ideal partition size based on factors like dataset size, maxPartitionBytes, and openCostInBytes. Unlock the power of optimized workload distribution, parallelism, and resource utilization. Don't miss out on this essential read to take your Apache Spark projects to new heights! #apachespark #bigdata #dataanalytics #dataprocessing https://lnkd.in/dZJU3qu6

🧊 💫 A nifty How-To spotted on Medium: "Ingesting Apache Iceberg tables into Apache Druid" https://bit.ly/3u4uBoa #ApacheIceberg #Iceberg #ApacheDruid #Druid