Meta tags:
Headings (most frequently used words):
evolution, schema, partition, sort, order, correctness, features, get, started, community, asf,
Text of the page (most frequently used words):
flink (197), configuration (84), api (70), amazon (66), java (61), apache (52), spark (50), queries (50), integrations (47), started (45), tables (45), writes (44), ddl (44), getting (44), evolution (40), views (36), aws (32), hive (31), the (30), and (30), quickstart (27), catalog (27), partitioning (27), table (25), data (25), migration (24), branching (23), tagging (23), javadoc (22), custom (22), nessie (22), jdbc (22), dell (22), connector (22), structured (22), streaming (22), procedures (22), schemas (22), reliability (22), performance (22), maintenance (22), introduction (22), trino (21), starrocks (21), presto (21), impala (21), dremio (21), clickhouse (21), doris (21), emr (21), athena (21), iceberg (20), partition (20), metrics (19), reporting (19), column (18), snowflake (17), pyiceberg (17), actions (17), google (15), spec (14), order (14), bigquery (14), daft (14), icebergrust (13), kafka (13), connect (13), druid (12), redshift (12), firehose (12), catalogs (12), that (11), new (10), storage (10), concepts (10), sort (9), for (9), you (9), risingwave (9), schema (8), not (8), third (8), party (8), are (7), can (7), existing (7), field (7), layout (7), struct (7), change (7), estuary (7), amoro (7), with (6), when (6), values (6), columns (6), delta (6), lake (6), overview (6), community (5), docs (5), sql (5), update (5), does (5), files (5), changes (5), nested (5), iceberggo (5), ecs (5), dynamodb (5), glue (5), tablemaintenance (5), project (4), supports (4), category (4), sampletable (4), example (4), used (4), add (4), from (4), each (4), other (4), correctness (4), tinybird (4), redpanda (4), olake (4), memiiso (4), debezium (4), firebolt (4), duckdb (4), databend (4), bladepipe (4), software (3), foundation (3), license (3), security (3), asf (3), support (3), blog (3), updating (3), following (3), updated (3), evolve (3), written (3), write (3), latest (3), metadata (3), need (3), because (3), track (3), name (3), fields (3), map (3), previous (3), home (3), specification (3), logo (2), either (2), trademarks (2), sponsorship (2), events (2), talks (2), get (2), rest (2), engine (2), hidden (2), through (2), its (2), statement (2), see (2), more (2), details (2), alter (2), commit (2), nullorder (2), replacesortorder (2), provides (2), code (2), could (2), sorted (2), nulls (2), old (2), earlier (2), remains (2), unchanged (2), updatespec (2), remove (2), uses (2), don (2), specific (2), instead (2), partitioned (2), using (2), separately (2), this (2), formats (2), cannot (2), delete (2), without (2), changing (2), which (2), violates (2), never (2), any (2), dropping (2), rewriting (2), rewritten (2), rename (2), even (2), like (2), contents (2), releases (2), implementations (2), hivecatalog (2), hadoopcatalog (2), properties (2), file (2), encryption (2), nightly (2), feather, registered, copyright, 2025, licensed, under, version, thanks, guidelines, contribute, issues, mailing, lists, open, multiple, language, apis, compute, advanced, filtering, optimistic, concurrency, serializable, isolation, features, back, top, null_first, dec, nulls_last, asc, create, ascending, last, descending, first, similar, also, engines, always, choose, unsorted, sorting, prohibitively, expensive, removefield, bucket, addfield, places, into, buckets, operation, eagerly, rewrite, fast, select, automatically, prunes, out, contain, matching, 2008, month, starting, 2009, day, both, layouts, able, coexist, same, versions, kept, start, writing, split, planning, where, plans, filter, derives, here, visual, representation, contrived, reference, directly, position, names, inadvertently, reused, unique, ids, assigned, mistake, associated, added, read, another, guarantees, independent, free, side, effects, note, keys, adding, would, equality, updates, perform, reorder, widen, type, key, value, list, element, drop, moving, daily, hourly, requires, dependent, partitions, must, some, cases, simple, renaming, supported, cause, problems, just, structures, volume, require, costly, distractions, migrating, place, implementation, status, udf, aes, gcm, stream, puffin, view, terms, vendors, sponsors, privacy, how, release, benchmarks, developer, snapshot, testing, multi, contributing, starburst, stackable, sail, ryft, nimtable, microsoft, onelake, fluss, lakekeeper, biglake, metastore, datahub, boring, polaris, gravitino, rust, python, archive, initializing, search, skip, content,
Text of the page (random words):
snowflake impala doris druid kafka connect integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust iceberggo 1 8 0 1 8 0 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft risingwave clickhouse presto dremio starrocks amazon athena amazon emr amazon data firehose amazon redshift google bigquery snowflake impala doris druid kafka connect integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust iceberggo 1 7 2 1 7 2 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft clickhouse presto dremio starrocks amazon athena amazon emr amazon data firehose amazon redshift google bigquery snowflake impala doris druid kafka connect integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 7 1 1 7 1 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft clickhouse presto dremio starrocks amazon athena amazon emr amazon data firehose amazon redshift google bigquery snowflake impala doris druid kafka connect integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 7 0 1 7 0 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft clickhouse presto dremio starrocks amazon athena amazon emr amazon data firehose amazon redshift google bigquery snowflake impala doris druid kafka connect integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 6 1 1 6 1 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft clickhouse presto dremio starrocks amazon athena amazon emr google bigquery snowflake impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 6 0 1 6 0 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino daft clickhouse presto dremio starrocks amazon athena amazon emr google bigquery snowflake impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 5 2 1 5 2 introduction tables tables branching and tagging configuration evolution maintenance partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr snowflake impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 5 1 1 5 1 introduction tables tables branching and tagging configuration evolution maintenance partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr snowflake impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 5 0 1 5 0 introduction tables tables branching and tagging configuration evolution maintenance partitioning performance reliability schemas views views configuration spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr snowflake impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog javadoc pyiceberg icebergrust 1 4 3 1 4 3 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog migration migration overview hive migration delta lake migration javadoc pyiceberg 1 4 2 1 4 2 introduction tables tables branching and tagging configuration evolution evolution table of contents schema evolution correctness partition evolution sort order evolution maintenance metrics reporting partitioning performance reliability schemas spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog migration migration overview hive migration delta lake migration javadoc pyiceberg 1 4 1 1 4 1 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog migration migration overview hive migration delta lake migration javadoc pyiceberg 1 4 0 1 4 0 introduction tables tables branching and tagging configuration evolution maintenance metrics reporting partitioning performance reliability schemas spark spark getting started configuration ddl procedures queries structured streaming writes flink flink flink getting started flink connector flink ddl flink queries flink writes flink actions flink configuration hive trino clickhouse presto dremio starrocks amazon athena amazon emr impala doris integrations integrations aws dell jdbc nessie api api java quickstart java api java custom catalog migration migration overview hive migration delta lake migration javadoc pyiceberg archive other implementations other implementations python rust go c third party third party catalogs catalogs apache gravitino apache polaris boring catalog datahub google biglake metastore lakekeeper integrations integrations amazon athena amazon data firehose amazon emr amazon redshift apache amoro apache doris apache druid apache fluss bladepipe clickhouse daft databend dremio duckdb estuary firebolt google bigquery impala memiiso debezium microsoft onelake nimtable olake presto redpanda risingwave ryft sail snowflake stackable starburst starrocks tinybird trino releases project project contributing multi engine support developer snapshot testing benchmarks security how to release asf asf sponsorship events privacy license security sponsors community community community talks vendors blog specification specification terms rest catalog spec table spec view spec puffin spec aes gcm stream spec udf spec implementation status table of contents schema evolution correctness partition evolution sort order evolution home docs java previous 1 4 2 tables evolution iceberg supports in place table evolution you can evolve a table schema just like sql even in nested structures or change partition layout when data volume changes iceberg does not require costly distractions like rewriting table data or migrating to a new table for example hive table partitioning cannot change so moving from a daily partition layout to an hourly partition layout requires a new table and because queries are dependent on partitions queries must be rewritten for the new table in some cases even changes as simple as renaming a column are either not supported or can cause data correctness problems schema evolution iceberg supports the following schema evolution changes add add a new column to the table or to a nested struct drop remove an existing column from the table or a nested struct rename rename an existing column or field in a nested struct update widen the type of a column struct field map key map value or list element reorder change the order of columns or fields in a nested struct iceberg schema updates are metadata changes so no data files need to be rewritten to perform the update note that map keys do not support adding or dropping struct fields that would change equality correctness iceberg guarantees that schema evolution changes are independent and free of side effects without rewriting files added columns never read existing values from another column dropping a column or field does not change the values in any other column updating a column or field does not change values in any other column changing the order of columns or fields in a struct does not change the values associated with a column or field name iceberg uses unique ids to track each column in a table when you add a column it is assigned a new id so existing data is never used by mistake formats that track columns by name can inadvertently un delete a column if a name is reused which violates 1 formats that track columns by position cannot delete columns without changing the names that are used for each column which violates 2 partition evolution iceberg table partitioning can be updated in an existing table because queries do not reference partition values directly when you evolve a partition spec the old data written with an earlier spec remains unchanged new data is written using the new spec in a new layout metadata for each of the partition versions is kept separately because of this when you start writing queries you get split planning this is where each partition layout plans files separately using the filter it derives for that specific partition layout here s a visual representation of a contrived example the data for 2008 is partitioned by month starting from 2009 the table is updated so that the data is instead partitioned by day both partitioning layouts are able to coexist in the same table iceberg uses hidden partitioning so you don t need to write queries for a specific partition layout to be fast instead you can write queries that select the data you need and iceberg automatically prunes out files that don t contain matching data partition evolution is a metadata operation and does not eagerly rewrite files iceberg s java table api provides updatespec api to update partition spec for example the following code could be used to update the partition spec to add a new partition field that places id column values into 8 buckets and remove an existing partition field category table sampletable sampletable updatespec addfield bucket id 8 removefield category commit spark supports updating partition spec through its alter table sql statement see more details in spark sql sort order evolution similar to partition spec iceberg sort order can also be updated in an existing table when you evolve a sort order the old data written with an earlier order remains unchanged engines can always choose to write data in the latest sort order or unsorted when sorting is prohibitively expensive iceberg s java table api provides replacesortorder api to update sort order for example the following code could be used to create a new sort order with id column sorted in ascending order with nulls last and category column sorted in descending order with nulls first table sampletable sampletable replacesortorder asc id nullorder nulls_last dec category nullorder null_first commit spark supports updating sort order through its alter table sql statement see more details in spark sql back to top features schema evolution hidden partitioning partition evolution serializable isolation branching and tagging optimistic concurrency advanced filtering compute engine integrations rest catalog multiple language apis get started spark quickstart hive quickstart open table spec docs blog talks community support mailing lists iceberg events issues contribute guidelines asf apache software foundation thanks sponsorship security license apache iceberg iceberg apache the apache feather logo and the apache iceberg project logo are either registered trademarks or trademarks of the apache software foundation copyright 2025 the apache software foundation licensed under the apache license ve...
|
