#Apache iceberg spark series#

_prod.catalog-impl = com.my.custom.CatalogImpl In the examples above, hive_prod and hadoop_prod can be used to prefix database and table names that will be loaded from those catalogs. Whether to enable catalog cache, default value is trueĪdditional properties can be found in common catalog configuration.Ĭatalog names are used in SQL queries to identify a table. Metastore connect URI default from hive-site.xml The default current namespace for the catalog The underlying Iceberg catalog implementation. The underlying Iceberg catalog implementation, HiveCatalog, HadoopCatalog or left unset if using a custom catalog Common configuration properties for Hive and Hadoop are: Property

.SparkSessionCatalog adds support for Iceberg tables to Spark’s built-in catalog, and delegates to the built-in catalog for non-Iceberg tablesīoth catalogs are configured using properties nested under the catalog name.

.SparkCatalog supports a Hive Metastore or a Hadoop warehouse as a catalog.

To load non-Iceberg tables in the same Hive metastore, use a session catalog.Ī catalog is created and named by adding a property .(catalog-name) with an implementation class for its value.

The Hive-based catalog only loads Iceberg tables. _prod.warehouse = hdfs://nn:8020/warehouse/path so we need to collect all the event_time partitions and re-ingest them from TableA for .hadoop_prod = .SparkCatalog but as we allow late data processing during that time frame we could have processed 03-2023 data as well. we need to fix data of Table B and Table C between 08-30-2023 and 09-07-2023 as we know some bug happened after 08-30. Caveat is with needs of reprocessing any arbitrary partition data upto this last 6 months. we still process them and write to correct event_time based partitions in all tables. Issue is, Many events are coming very late, upto 6 months old. In that case, partitioning them by event_time make sense.

#Apache iceberg spark series#

I want TableA, TableB and TableC to be efficiently queriably by event_time eventhough it's a time series data. Pipeline B, C reads from TableA and writes to Table B and Table C respectively. Pipeline A writes processed events to Table A however, it's not consumer friendly query.

however, that is harder to maintain.Ģ) I can partitioned by ingestion_time and during querying I can just use big enough time range to accommodate for late data. one partitioned by event_time and another by ingestion_time. Some solutions I thought of are: 1) I can create two sets of table for each schema. but if data in only partitioned by event_time than querying the detla live table like select * from table where ingestion_time between date1 and date2 will scan entire table because it's not partitioned by ingestion_time. For reprocessing, it makes more sense to reprocess on store_time or ingestion_time. However, another use case is for data reprocessing in event of any failure, data inconsistency. I can partition my detla table on event_time for this use case. I can see that event_time will be most useful for querying as there can be lot of late arrival data and event_time will give updated value every time it's queried. this table can be ingested and processed by downstream data pipeline or it can be directly used by end consumer to do data analysis.Įvery event has publish_time, event_time and at some point ingestions_time and store_time. I am using spark-streaming to ingest live event streams every 5 minutes and append into delta or apache iceberg table.