Documentation
Hive 4.2.0 - REST Catalog Integration
Oct 31, 2025REST Catalog Integration
Table of Contents
Hive + Gravitino + Keycloak
- The code for this setup is located in the Hive repository in
packaging/src/docker/thirdparties/gravitinofolder. - It contains a docker-compose-based setup integrating Apache Hive, Gravitino Iceberg REST server, and Keycloak for OAuth2 authentication. It allows Hive to use an Iceberg REST catalog secured via Keycloak.
Architecture Overview
This diagram illustrates the key docker-compose components and their interactions in this setup:
Administration Manual
Jul 24, 2025HCatalog Manual
Jul 24, 2025Language Manual
Jul 24, 2025User Manual
Jul 24, 2025WebHCat (Templeton) Manual
Jul 24, 2025Apache Hive : ChangeLog
Dec 12, 2024Apache Hive : ChangeLog
Release 4.0.0 - 2024-03-29
NEW FEATURES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-27850 | Iceberg: Major QB Compaction | Major | Iceberg integration | Dmitriy Fingerman | Dmitriy Fingerman |
| HIVE-26222 | Native GeoSpatial Support in Hive | Major | Hive, HiveServer2 | mahesh kumar behera | Ayush Saxena |
| HIVE-27980 | Hive Iceberg Compaction: add support for OPTIMIZE TABLE syntax | Major | . | Dmitriy Fingerman | Dmitriy Fingerman |
| HIVE-26435 | Add method for collecting HMS meta summary | Major | . | Ruyi Zheng | Hongdan Zhu |
IMPROVEMENTS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-26666 | Filter out compactions by id to minimise expense of db operations | Major | . | KIRTI RUGE | KIRTI RUGE |
| HIVE-27563 | Add typeof UDF | Minor | UDF | John Sherman | John Sherman |
| HIVE-27575 | MASK_HASH UDF generate a log per row | Major | UDF | Shohei Okumiya | Shohei Okumiya |
| HIVE-27396 | Use -strict argument for Thrift code generation to prevent compatibility issues | Major | Build Infrastructure | Joe McDonnell | Joe McDonnell |
| HIVE-27554 | Validate URL used by SSO workflow for JDBC connection | Minor | JDBC | Henri Biestro | Henri Biestro |
| HIVE-27516 | Add programatically added DAG scoped properties to DAG Configuration | Major | . | László Bodor | László Bodor |
| HIVE-27586 | Parse dates from strings ignoring trailing (potentialy) invalid chars | Major | HiveServer2 | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27578 | Refactor genJoinRelNode to use genAllRexNode instead of genAllExprNodeDesc | Major | . | Soumyakanti Das | Soumyakanti Das |
| HIVE-27600 | Reduce filesystem calls in OrcFileMergeOperator | Minor | Hive | Yi Zhang | Yi Zhang |
| HIVE-25576 | Configurable datetime formatter for unix_timestamp, from_unixtime | Major | . | Ashish Sharma | Stamatis Zampetakis |
| HIVE-27585 | Upgrade kryo serialization lib to latest version | Minor | Serializers/Deserializers | Suprith Chandrashekharachar | Suprith Chandrashekharachar |
| HIVE-27595 | Improve efficiency in the filtering hooks | Minor | HiveServer2 | Naveen Gangam | Henri Biestro |
| HIVE-27645 | Clean test cases by refactoring assertFalse(equals()) using assertNotEquals & @Test(excepted) using assertThrows | Minor | . | Taher Ghaleb | Taher Ghaleb |
| HIVE-27599 | ConvertAstToSearchArg improvement with output and typo in comments | Major | Server Infrastructure | xy | xy |
| HIVE-27627 | Iceberg: Insert into/overwrite partition support | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27525 | Ease the write permissions on external table during create table operation | Major | Standalone Metastore | Sai Hemanth Gantasala | Riju Trivedi |
| HIVE-27665 | Change Filter Parser on HMS to allow backticks | Major | Standalone Metastore | Steve Carlin | Steve Carlin |
| HIVE-27461 | HiveMetaStoreAuthorizer should log the root cause of checkPrivileges | Major | Hive | Wechar | Wechar |
| HIVE-27661 | Auth mode inferred from the Authorization header | Major | HiveServer2 | Gergely Farkas | Gergely Farkas |
| HIVE-27309 | Large number of partitions and small files causes OOM in query coordinator | Major | Iceberg integration | Rajesh Balamohan | Dmitriy Fingerman |
| HIVE-27656 | Upgrade jansi.version to 2.4.0 | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-27673 | Configurable datetime formatter for date_format | Major | HiveServer2 | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27664 | AlterTableSetLocationAnalyzer threw a confusing exception “Cannot connect to namenode” | Major | . | xiongyinke | xiongyinke |
| HIVE-27564 | Add log for ZooKeeperTokenStore | Major | . | lvyanquan | lvyanquan |
| HIVE-27694 | Include HiveIcebergSerDe in default list of serdes using HMS | Minor | Standalone Metastore | Naveen Gangam | Naveen Gangam |
| HIVE-27696 | Docker build from source should include iceberg profile | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-21100 | Allow flattening of table subdirectories resulted when using TEZ engine and UNION clause | Major | . | George Pachitariu | Ayush Saxena |
| HIVE-27646 | Iceberg: Retry query when concurrent write queries fail due to conflicting writes | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-27672 | Iceberg: Truncate partition support | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27729 | Iceberg: Check Iceberg type in AlterTableExecuteAnalyzer | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27406 | CompactionTxnHandler cleanup | Major | Hive | László Végh | László Végh |
| HIVE-27752 | Remove DagUtils duplicate class | Minor | . | László Bodor | Akshat Mathur |
| HIVE-27757 | Upgrade hadoop to 3.3.6 | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27723 | Prevent localizing the same original file more than once if symlinks are present | Major | . | László Bodor | László Bodor |
| HIVE-27771 | Iceberg: Allow expire snapshot by time range | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27755 | Quote identifiers in SQL emitted by SchemaTool for MySQL | Major | Standalone Metastore | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27793 | Iceberg: Support setting current snapshot with SnapshotRef | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27799 | Bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2 | Major | . | Ayush Saxena | |
| HIVE-27182 | tez_union_with_udf.q with TestMiniTezCliDriver is flaky | Major | . | Ayush Saxena | Stamatis Zampetakis |
| HIVE-27686 | Use ORC 1.8.5. | Major | . | Zoltán Rátkai | Zoltán Rátkai |
| HIVE-27802 | Simplify TestTezSessionState.testSymlinkedLocalFilesAreLocalizedOnce | Major | . | László Bodor | László Bodor |
| HIVE-27786 | Iceberg: Eliminate engine.hive.enabled table property | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27346 | Getting exception for wildcard (*) search for database and table name | Major | . | Hongdan Zhu | Hongdan Zhu |
| HIVE-27731 | Perform metadata delete when only static filters are present | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27826 | Upgrade to Parquet 1.13.1 | Major | Parquet | Butao Zhang | Butao Zhang |
| HIVE-27780 | Implement direct SQL for get_all_functions | Major | Standalone Metastore | Butao Zhang | Butao Zhang |
| HIVE-27825 | Better error message for an empty quoted identifier | Minor | Parser | Shohei Okumiya | Shohei Okumiya |
| HIVE-24815 | Remove “IDXS” Table from Metastore Schema | Major | Metastore, Standalone Metastore | Hunter Logan | Riju Trivedi |
| HIVE-27779 | Iceberg: Drop partition support | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27789 | Iceberg: Add a way to expire snapshots with retain last | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27593 | Iceberg: Keep iceberg properties in sync with hms properties | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27864 | Update plugin for SBOM generation to 2.7.10 | Major | . | Vinod Anandan | Vinod Anandan |
| HIVE-27762 | Don’t fall back to jdo query in ObjectStore if direct sql throws unrecoverable exception | Major | Hive | Wechar | Wechar |
| HIVE-27819 | Iceberg: Upgrade iceberg version to 1.4.2 | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27869 | Iceberg: Select on HadoopTable fails at HiveIcebergStorageHandler#canProvideColStats | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27828 | Iceberg integration: enable copy on write update when split update is on | Major | Iceberg integration | Krisztian Kasa | Krisztian Kasa |
| HIVE-27877 | Bump org.apache.avro:avro from 1.11.1 to 1.11.3 | Major | . | Ayush Saxena | |
| HIVE-27889 | Bump org.apache.santuario:xmlsec from 2.3.0 to 2.3.4 | Major | . | Ayush Saxena | |
| HIVE-27871 | Fix some formatting problems is YarnQueueHelper | Major | . | László Bodor | Mahesh Raju Somalaraju |
| HIVE-27896 | Remove common-lang usage | Major | . | Cheng Pan | Cheng Pan |
| HIVE-27906 | Iceberg: Implement Delete Orphan Files | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27912 | Include Iceberg module in nightly builds | Major | . | Butao Zhang | Butao Zhang |
| HIVE-27903 | Iceberg: Implement Expire Snapshot with default table properties | Major | Hive | JK Pasimuthu | Ayush Saxena |
| HIVE-27633 | HMS: MTable to Table process reduces view related SQL | Minor | Metastore | dzcxzl | dzcxzl |
| HIVE-26630 | Introduce copy-on-write DML implementations for Iceberg tables | Major | . | Ádám Szita | Denys Kuzmenko |
| HIVE-27918 | Iceberg: Push transforms for clustering during table writes | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27894 | Enhance HMS Handler Logs for all ‘get_partition’ functions. | Major | . | Shivangi Jha | Shivangi Jha |
| HIVE-27905 | Some GenericUDFs wrongly cast ObjectInspectors | Major | . | Shohei Okumiya | Shohei Okumiya |
| HIVE-27935 | Add qtest for Avro invalid schema and field names | Major | . | Akshat Mathur | Akshat Mathur |
| HIVE-27856 | Change the default value of hive.optimize.cte.materialize.threshold to -1 | Major | . | Seonggon Namgung | Seonggon Namgung |
| HIVE-27936 | Disable flaky test testBootstrapAcidTablesDuringIncrementalWithConcurrentWrites | Major | Testing Infrastructure | Butao Zhang | Butao Zhang |
| HIVE-27803 | Bump org.apache.avro:avro from 1.11.1 to 1.11.3 | Major | . | Ayush Saxena | |
| HIVE-27934 | Fix incorrect description about the execution framework in README.md | Minor | . | Butao Zhang | Butao Zhang |
| HIVE-27690 | Handle casting NULL literal to complex type | Major | . | Krisztian Kasa | Krisztian Kasa |
| HIVE-27919 | Constant reduction in CBO does not work for FROM_UNIXTIME, DATE_ADD, DATE_SUB, TO_UNIX_TIMESTAMP | Major | CBO | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-23558 | Remove compute_stats UDAF | Major | Statistics | Jesús Camacho Rodríguez | Butao Zhang |
| HIVE-27961 | Beeline will print duplicate stats info when hive.tez.exec.print.summary is true | Minor | . | Butao Zhang | Butao Zhang |
| HIVE-27530 | Implement direct SQL for alter partitions to improve performance | Major | . | Wechar | Wechar |
| HIVE-27925 | HiveConf: unify ConfVars enum and use underscore for better readability | Major | . | László Bodor | Kokila N |
| HIVE-27969 | Add verbose logging for schematool and metastore service for Docker container | Major | . | Akshat Mathur | Akshat Mathur |
| HIVE-27978 | Tests in hive-unit module are not running again | Major | . | László Bodor | László Bodor |
| HIVE-27911 | Drop database query failing with Invalid ACL Exception | Major | . | KIRTI RUGE | KIRTI RUGE |
| HIVE-27937 | Clarifying comments and xml configs around tez container size | Major | . | László Bodor | László Bodor |
| HIVE-27977 | Fix ordering flakiness in TestHplSqlViaBeeLine | Major | . | László Bodor | László Bodor |
| HIVE-27023 | Add setting to disable hive session being started during Hive Cli startup | Minor | . | Alagappan Maruthappan | Alagappan Maruthappan |
| HIVE-27827 | Improve performance of direct SQL implement for getPartitionsByFilter | Major | . | Wechar | Wechar |
| HIVE-27994 | Optimize renaming the partitioned table | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-27979 | HMS alter_partitions log adds table name | Minor | Standalone Metastore | dzcxzl | dzcxzl |
| HIVE-28008 | ParquetFileReader is not closed in ParquetHiveSerDe.readSchema | Minor | Hive | Michal Lorek | Michal Lorek |
| HIVE-27991 | Utilise FanoutWriters when inserting records in an Iceberg table when the records are unsorted | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27984 | Support backward compatibility of hms thrift struct about column stats | Major | Standalone Metastore | Butao Zhang | Butao Zhang |
| HIVE-27481 | TxnHandler cleanup | Major | Hive | László Végh | László Végh |
| HIVE-28038 | Disable fallback to jdo for DeadlineException | Major | Hive | Wechar | Wechar |
| HIVE-27958 | Refactor DirectSqlUpdatePart class | Major | Hive | Wechar | Wechar |
| HIVE-27992 | Upgrade to tez 0.10.3 | Major | . | László Bodor | László Bodor |
| HIVE-26445 | Use tez.local.mode.without.network for qtests | Major | . | László Bodor | László Bodor |
| HIVE-27022 | Split removeDuplicateCompletedTxnComponents away from AcidHouseKeeper and onto a separate timer | Major | . | Jason Phelps | Taraka Rama Rao Lethavadla |
| HIVE-28071 | Sync jetty version across modules | Minor | . | Raghav Aggarwal | Raghav Aggarwal |
| HIVE-28015 | Iceberg: Add identifier-field-ids support in Hive | Major | Iceberg integration | Denys Kuzmenko | Butao Zhang |
| HIVE-28081 | Code refine on ClearDanglingScratchDir::removeLocalTmpFiles | Trivial | . | Butao Zhang | Butao Zhang |
| HIVE-28064 | Add cause to ParseException for diagnosability purposes | Major | Parser | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-28056 | Bump com.jayway.jsonpath:json-path from 2.8.0 to 2.9.0 | Major | . | László Bodor | László Bodor |
| HIVE-28078 | TestTxnDbUtil should generate csv files when we query the metastore database | Major | Standalone Metastore | Zsolt Miskolczi | Zsolt Miskolczi |
| HIVE-27692 | Explore removing the always task from embedded HMS | Major | Standalone Metastore | Zhihua Deng | Zhihua Deng |
| HIVE-27405 | Throw out the detail error Invalid partition name to the clients | Major | HiveServer2 | Taraka Rama Rao Lethavadla | Taraka Rama Rao Lethavadla |
| HIVE-27845 | Upgrade protobuf to 3.24.4 to fix CVEs | Major | . | Akshat Mathur | tanishqchugh |
| HIVE-28080 | Propagate statistics from a source table to the materialized CTE | Major | Query Planning | Shohei Okumiya | Shohei Okumiya |
| HIVE-28090 | correct desc of hive.metastore.disallow.incompatible.col.type.changes | Major | . | zzzzming95 | zzzzming95 |
| HIVE-28083 | Enable HMS client/query cache for EXPLAIN queries | Minor | Hive | Soumyakanti Das | Soumyakanti Das |
| HIVE-25972 | HIVE_VECTORIZATION_USE_ROW_DESERIALIZE in hiveconf.java imply default value is false，in fact the default value is ’true’ | Major | Configuration, Hive | lkl | Kokila N |
| HIVE-28093 | Re-execute DAG in case of NoCurrentDAGException | Major | . | László Bodor | László Bodor |
| HIVE-28051 | LLAP: cleanup local folders on startup and periodically | Major | . | László Bodor | László Bodor |
BUG FIXES:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-26837 | CTLT with hive.create.as.external.legacy as true creates managed table instead of external table | Major | HiveServer2 | Ramesh Kumar Thangarajan | Ramesh Kumar Thangarajan |
| HIVE-27562 | Iceberg: Fetching virtual columns failing | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27487 | NPE in Hive JDBC storage handler | Major | JDBC storage handler | Zhihua Deng | Zhihua Deng |
| HIVE-27582 | Do not cache HBase table input format in FetchOperator | Major | . | Ganesha Shreedhara | Ganesha Shreedhara |
| HIVE-27304 | Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer. | Major | Hive | Sai Hemanth Gantasala | Sai Hemanth Gantasala |
| HIVE-24771 | Fix hang of TransactionalKafkaWriterTest | Major | . | Zoltan Haindrich | Kokila N |
| HIVE-27631 | Fix CCE when set fs.hdfs.impl other than DistributedFileSystem | Major | Hive | Baolong Mao | Baolong Mao |
| HIVE-22961 | Drop function in Hive should not send request for drop database to Ranger plugin. | Major | Hive | Sam An | Riju Trivedi |
| HIVE-27463 | Non-daemon thread prevents HMS from exiting when failed to start thrift server | Minor | . | Zhihua Deng | Zhihua Deng |
| HIVE-27536 | Merge task must be invoked after optimisation for external CTAS queries | Major | . | Sourabh Badhya | Sourabh Badhya |
| HIVE-27566 | Fix some yarn cluster options for tests | Major | . | László Bodor | László Bodor |
| HIVE-27632 | ClassCast Exception in Vectorization converting decimal64 to decimal | Major | Vectorization | Riju Trivedi | Stephen Carlin |
| HIVE-27539 | Drop renamed external table fails when hive.metastore.try.direct.sql.ddl is disabled | Major | . | Venugopal Reddy K | Venugopal Reddy K |
| HIVE-21213 | Acid table bootstrap replication needs to handle directory created by compaction with txn id | Major | Hive, HiveServer2, repl | mahesh kumar behera | mahesh kumar behera |
| HIVE-27667 | Fix get partitions with max_parts | Major | Metastore | Yuming Wang | Yuming Wang |
| HIVE-27657 | Change hive.fetch.task.conversion.threshold default value | Major | . | Mayank Kunwar | Mayank Kunwar |
| HIVE-26961 | Fix improper replication metric count when hive.repl.filter.transactions is set to true. | Major | . | Rakshith C | Rakshith C |
| HIVE-27642 | StartMiniHS2Cluster fails to run due to missing JDBC driver with Postgres | Major | HiveServer2 | Zoltán Rátkai | Zoltán Rátkai |
| HIVE-27643 | Exclude compaction queries from ranger policies | Critical | . | László Végh | László Végh |
| HIVE-17350 | metrics errors when retrying HS2 startup | Major | . | Sergey Shelukhin | Mayank Kunwar |
| HIVE-27648 | CREATE TABLE with CHECK constraint fails with SemanticException | Major | Hive | Soumyakanti Das | Krisztian Kasa |
| HIVE-24606 | Multi-stage materialized CTEs can lose intermediate data | Major | Query Planning | Shohei Okumiya | Shohei Okumiya |
| HIVE-27138 | MapJoinOperator throws NPE when computing OuterJoin with filter expressions on small table | Blocker | . | Seonggon Namgung | Seonggon Namgung |
| HIVE-27675 | Support keystore/truststore types for hive to zookeeper integration points | Major | HiveServer2, JDBC, Standalone Metastore | Naveen Gangam | Naveen Gangam |
| HIVE-27730 | Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.4 | Major | . | Ayush Saxena | |
| HIVE-27738 | SchemaTool picks incorrect schema script after 4.0.0-beta-1 release | Major | . | KIRTI RUGE | KIRTI RUGE |
| HIVE-27649 | Support ORDER BY clause in subqueries with set operators | Major | Parser | Nicolas Richard | Nicolas Richard |
| HIVE-27760 | WHERE condition on DATE type partitioning column leads to wrong results | Major | HiveServer2 | Dayakar M | Dayakar M |
| HIVE-27728 | Changed behavior for alter table rename partition from legacy tables | Major | . | Naveen Gangam | Zhihua Deng |
| HIVE-27733 | Intermittent ConcurrentModificationException in HiveServer2 | Major | HiveServer2 | Henri Biestro | Henri Biestro |
| HIVE-27764 | Authentication does not work behind Knox gateway because the “WWW-Authenticate: Negotiate” response header is missing | Major | HiveServer2 | Gergely Farkas | Gergely Farkas |
| HIVE-27695 | Intermittent OOM when running TestMiniTezCliDriver | Major | Test | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-26828 | Fix OOM for hybridgrace_hashjoin_2.q | Major | Test, Tez | Alessandro Solimando | Stamatis Zampetakis |
| HIVE-27798 | Correct configuration item in hive-site.xml in docker. | Major | . | 易霖威 | 易霖威 |
| HIVE-27772 | UNIX_TIMESTAMP should return NULL when date fields are out of bounds | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-27682 | AlterTableAlterPartitionOperation cannot change the type if the column has default partition | Minor | HiveServer2 | Zhihua Deng | Zhihua Deng |
| HIVE-27777 | CBO fails on multi insert overwrites with common group expression | Major | HiveServer2 | Steve Carlin | Steve Carlin |
| HIVE-27651 | Upgrade hbase version | Major | . | Ayush Saxena | Butao Zhang |
| HIVE-27324 | Hive query with NOT IN condition is giving incorrect results when the sub query table contains the null value. | Major | Hive | Shobika Selvaraj | Diksha |
| HIVE-27114 | Provide a configurable filter for removing useless properties in Partition objects from listPartitions HMS Calls | Major | . | Naresh P R | Zhihua Deng |
| HIVE-27113 | Increasing default for hive.thrift.client.max.message.size to 2 GB | Major | Hive | Riju Trivedi | Riju Trivedi |
| HIVE-27846 | Tests under hive-unit module are not running | Major | Testing Infrastructure | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27849 | Replication tests using ivy fail after the upgrade to 2.5.2 | Major | . | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27866 | JDBC: HttpRequestInterceptorBase should not add an empty “Cookie:” header to the request if no custom cookies have been specified | Major | JDBC | Gergely Farkas | Gergely Farkas |
| HIVE-27862 | Map propertyContent to a wrong column in package.jdo | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-27679 | Ranger Yarn Queue policies are not applying correctly, rework done for HIVE-26352 | Major | . | Mahesh Raju Somalaraju | Mahesh Raju Somalaraju |
| HIVE-27865 | HMS in http mode shuts down silently with no errors | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-27885 | Cast decimal from string with space without digits before dot returns NULL | Major | . | Naresh P R | Naresh P R |
| HIVE-27867 | Incremental materialized view throws NPE whew Iceberg source table is empty | Major | . | Krisztian Kasa | Krisztian Kasa |
| HIVE-27662 | Incorrect parsing of nested complex types containing map during vectorized text processing | Major | Vectorization | Raghav Aggarwal | Raghav Aggarwal |
| HIVE-27890 | Tez Progress bar is not displayed in Beeline upon setting session level execution engine to Tez | Major | Beeline | Shivangi Jha | Shivangi Jha |
| HIVE-27713 | Iceberg: metadata location overrides can cause data breach | Major | Authorization, Iceberg integration | Janos Kovacs | Ayush Saxena |
| HIVE-27797 | Transactions that got timed out are not getting logged as ‘ABORTED’ in NOTIFICATION_LOG | Major | repl, Transactions | Taraka Rama Rao Lethavadla | Taraka Rama Rao Lethavadla |
| HIVE-27093 | Fix NPE in initialize() of Partition class | Critical | . | Wechar | Wechar |
| HIVE-27893 | Add a range validator in hive.metastore.batch.retrieve.max to only have values greater than 0 | Major | . | Vikram Ahuja | Vikram Ahuja |
| HIVE-27240 | NPE on Hive Hook Proto Log Writer | Critical | Hive | Shubham Sharma | Shubham Sharma |
| HIVE-27658 | Error resolving join keys during conversion to dynamic partition hashjoin | Major | Query Planning | xiaojunxiang | Stamatis Zampetakis |
| HIVE-27555 | Upgrade issues with Kudu table on backend db | Critical | . | Zhihua Deng | Zhihua Deng |
| HIVE-24730 | Shims classes override values from hive-site.xml and tez-site.xml silently | Major | . | László Bodor | László Bodor |
| HIVE-27446 | Exception when rebuild materialized view incrementally in presence of delete operations | Major | CBO, Materialized views | Krisztian Kasa | Krisztian Kasa |
| HIVE-27801 | Exists subquery rewrite results in a wrong plan | Critical | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27943 | NPE in VectorMapJoinCommonOperator.setUpHashTable when running query with join on date | Major | Query Processor | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27930 | Insert/Load overwrite table partition does not clean up directory before overwriting | Major | . | Kiran Velumuri | Kiran Velumuri |
| HIVE-27892 | Hive “insert overwrite table” for multiple partition table issue | Major | . | Mayank Kunwar | Mayank Kunwar |
| HIVE-24219 | Disable flaky TestStreaming | Major | . | Peter Varga | Stamatis Zampetakis |
| HIVE-27952 | Hive fails to create SslContextFactory when KeyStore has multiple certificates | Major | . | Seonggon Namgung | Seonggon Namgung |
| HIVE-27876 | Incorrect query results on tables with ClusterBy & SortBy | Major | . | Naresh P R | Ramesh Kumar Thangarajan |
| HIVE-27963 | Build failure when license-maven-plugin downloads bsd-license.php | Major | Hive | Akshat Mathur | Akshat Mathur |
| HIVE-27161 | MetaException when executing CTAS query in Druid storage handler | Critical | Druid integration | Stamatis Zampetakis | Krisztian Kasa |
| HIVE-25803 | URL Mapping appends default Fs scheme even for LOCAL DIRECTORY ops | Critical | Authorization, HiveServer2 | Soumitra Sulav | Ayush Saxena |
| HIVE-27804 | Implement batching in getPartition calls which returns partition list along with auth info | Major | . | Vikram Ahuja | Vikram Ahuja |
| HIVE-27967 | Iceberg: Fix dynamic runtime filtering | Major | Iceberg integration | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27966 | Disable flaky testFetchResultsOfLogWithOrientation | Major | Hive | Wechar | Wechar |
| HIVE-27916 | Increase tez.am.resource.memory.mb for TestIcebergCliDrver to 512MB | Major | Test | László Bodor | László Bodor |
| HIVE-27948 | Wrong results when using materialized views with non-deterministic/dynamic functions | Critical | CBO, Materialized views | Stamatis Zampetakis | Krisztian Kasa |
| HIVE-27988 | Do not convert FullOuterJoin with filter to MapJoin | Critical | HiveServer2 | Denys Kuzmenko | Seonggon Namgung |
| HIVE-27951 | hcatalog dynamic partitioning fails with partition already exist error when exist parent partitions path | Critical | HCatalog | Yi Zhang | Yi Zhang |
| HIVE-27974 | Fix flaky test - TestReplicationMetricCollector.testSuccessStageFailure | Major | HiveServer2 | Zsolt Miskolczi | Zsolt Miskolczi |
| HIVE-27989 | Wrong database name in MetaException from MetastoreDefaultTransformer.java | Minor | . | Riza Suminto | Butao Zhang |
| HIVE-27857 | Do not check write permission while dropping external table or partition | Major | Hive | Wechar | Wechar |
| HIVE-27492 | HPL/SQL built-in functions like sysdate not working | Major | hpl/sql | Dayakar M | Dayakar M |
| HIVE-27999 | Run Sonar analysis using Java 17 | Major | . | Wechar | Wechar |
| HIVE-21520 | Query “Submit plan” time reported is incorrect | Trivial | . | Rajesh Balamohan | Butao Zhang |
| HIVE-27749 | SchemaTool initSchema fails on Mariadb 10.2 | Critical | Standalone Metastore | Stamatis Zampetakis | Sourabh Badhya |
| HIVE-28013 | No space left on device when running precommit tests | Blocker | Testing Infrastructure | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27960 | Invalid function error when using custom udaf | Major | Hive | gaoxiong | gaoxiong |
| HIVE-27489 | HPL/SQL does not support table aliases on column names in loops | Major | hpl/sql | Dayakar M | Dayakar M |
| HIVE-28009 | Shared work optimizer ignores schema merge setting in case of virtual column difference | Major | Query Planning | Krisztian Kasa | Krisztian Kasa |
| HIVE-26713 | StringExpr ArrayIndexOutOfBoundsException with LIKE ‘%xxx%’ | Major | storage-api | Ryu Kobayashi | Ryu Kobayashi |
| HIVE-27942 | Missing aux jar errors during LLAP launch | Minor | Hive, llap | Shubham Sharma | Shubham Sharma |
| HIVE-27993 | Netty4 ShuffleHandler: should use 1 boss thread | Major | llap | Anmol Sundaram | Anmol Sundaram |
| HIVE-27914 | Fix the missing partitions judgement in drop_partitions_req | Major | . | Wechar | Wechar |
| HIVE-28017 | Add generated protobuf code | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-28000 | Hive QL : “not in” clause gives incorrect results when type coercion cannot take place. | Major | Hive | Anmol Sundaram | Anmol Sundaram |
| HIVE-27938 | Iceberg: Fix java.lang.ClassCastException during vectorized reads on partition columns | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-28004 | DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List | Blocker | Transactions | Butao Zhang | László Végh |
| HIVE-28053 | Incorrect shading configuration for beeline jar-with-dependencies | Major | Beeline | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-26818 | Beeline module misses transitive dependencies due to shading | Major | Beeline | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-28052 | Iceberg: Major QB Compaction fails with ClassNotFoundException: org.springframework.core.ErrorCoded | Major | . | Butao Zhang | Ayush Saxena |
| HIVE-28054 | SemanticException for join condition in subquery | Major | Hive | Soumyakanti Das | Soumyakanti Das |
| HIVE-28057 | Iceberg: Branches with non-lowercase characters can’t be accessed | Major | . | Attila Turoczy | Ayush Saxena |
| HIVE-28050 | Disable Incremental non aggregated materialized view rebuild in presence of delete operations | Major | Materialized views | Krisztian Kasa | Krisztian Kasa |
| HIVE-28065 | Upgrade Bouncy castle to bcprov-jdk18on 1.77 | Major | . | Araika Singh | Araika Singh |
| HIVE-27950 | STACK UDTF returns wrong results when # of argument is not a multiple of N | Major | . | Shohei Okumiya | Shohei Okumiya |
| HIVE-27778 | Alter table command gives error after computer stats is run with Impala | Major | . | Kokila N | Zhihua Deng |
| HIVE-27924 | Incremental rebuild goes wrong when inserts and deletes overlap between the source tables | Critical | Materialized views | Wenhao Li | Krisztian Kasa |
| HIVE-28021 | Iceberg: Attempting to create a table with a percent symbol fails | Minor | Iceberg integration | Tim Thorpe | Tim Thorpe |
| HIVE-28084 | Iceberg: COW fix for Merge operation | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27490 | HPL/SQL says it support default value for parameters but not considering them when no value is passed | Major | hpl/sql | Dayakar M | Dayakar M |
| HIVE-27775 | DirectSQL and JDO results are different when fetching partitions by timestamp in DST shift | Critical | Standalone Metastore | Stamatis Zampetakis | Zhihua Deng |
| HIVE-28073 | Upgrade jackson version to 2.16.1 | Major | . | Araika Singh | Araika Singh |
| HIVE-28102 | Iceberg: Invoke validateDataFilesExist for RowDelta operations | Major | . | Zoltán Borók-Nagy | Ayush Saxena |
| HIVE-28076 | Selecting data from a bucketed table with decimal column type throwing NPE. | Major | HiveServer2 | Dayakar M | Dayakar M |
| HIVE-28123 | Add Generated Protobuf code for 3.24.4 Upgrade | Major | . | Indhumathi Muthumurugesh | Indhumathi Muthumurugesh |
| HIVE-28034 | HiveConf: unify ConfVars enum and use underscore for better readability - pt2 leftovers | Major | . | László Bodor | Kokila N |
TESTS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-27431 | Clean invalid properties in test module | Minor | Test | Butao Zhang | Butao Zhang |
| HIVE-27747 | Generalize TestSchemaToolForMetastore to run on every supported DBMS | Major | Tests | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-27745 | Add unit test to ensure short version is inline with full version | Major | . | KIRTI RUGE | KIRTI RUGE |
| HIVE-28001 | Fix the flaky test TestLeaderElection | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-27556 | Add Unit Test for KafkaStorageHandlerInfo | Major | kafka integration, StorageHandler | Kokila N | Kokila N |
SUB-TASKS:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-27031 | Iceberg: Implement Copy-On-Write for Delete queries | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27523 | Implement array_union UDF in Hive | Major | . | Taraka Rama Rao Lethavadla | Taraka Rama Rao Lethavadla |
| HIVE-27277 | Set up github actions workflow to build and push docker image to docker hub | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-27630 | Iceberg: Fast forward branch | Major | . | Denys Kuzmenko | Ayush Saxena |
| HIVE-27654 | Iceberg: Cherry-Pick commit to a branch | Major | . | Ayush Saxena | Ayush Saxena |
| HIVE-27670 | Failed to build the image locally on Apple silicon | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-27322 | Iceberg: metadata location overrides can cause data breach - custom location to AuthZ | Blocker | Iceberg integration | Janos Kovacs | Ayush Saxena |
| HIVE-27716 | Precommit: Save log files for first 10 failures | Major | . | László Bodor | László Bodor |
| HIVE-27711 | Allow creating a branch from tag name | Major | . | Ayush Saxena | Butao Zhang |
| HIVE-27702 | Remove PowerMock from beeline and upgrade mockito to 4.11 | Major | HiveServer2 | Zsolt Miskolczi | Mayank Kunwar |
| HIVE-27736 | Remove PowerMock from itests-jmh and upgrade mockito | Major | . | Ayush Saxena | Zsolt Miskolczi |
| HIVE-27705 | Remove PowerMock from service (hive-service) and upgrade mockito to 4.11 | Major | HiveServer2 | Zsolt Miskolczi | KIRTI RUGE |
| HIVE-27701 | Remove PowerMock from llap-client and upgrade mockito to 4.11 | Major | HiveServer2 | Zsolt Miskolczi | Zsolt Miskolczi |
| HIVE-27704 | Remove PowerMock from jdbc-handler and upgrade mockito to 4.11 | Major | HiveServer2 | Zsolt Miskolczi | KIRTI RUGE |
| HIVE-26455 | Remove PowerMockito from hive-exec | Minor | Hive | Zsolt Miskolczi | Zsolt Miskolczi |
| HIVE-27399 | Add lateral view support for CBO | Major | HiveServer2 | Steve Carlin | Steve Carlin |
| HIVE-27597 | Implement JDBC Connector for HiveServer | Major | Hive | Naveen Gangam | Naveen Gangam |
| HIVE-27783 | Iceberg: Implement Copy-On-Write for Update queries | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27006 | ParallelEdgeFixer inserts misconfigured operator and does not connect it in Tez DAG | Major | . | Seonggon Namgung | Seonggon Namgung |
| HIVE-26621 | TPC-DS query 2 fails with java.lang.RuntimeException: cannot find field _col0 from [] | Major | . | Sungwoo Park | Seonggon Namgung |
| HIVE-27794 | Iceberg: Implement Copy-On-Write for Merge queries | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27269 | VectorizedMapJoin returns wrong result for TPC-DS query 97 | Critical | . | Seonggon Namgung | Seonggon Namgung |
| HIVE-27714 | Iceberg: metadata location overrides can cause data breach - handling default locations | Critical | Authorization, Iceberg integration | Janos Kovacs | Ayush Saxena |
| HIVE-27926 | Iceberg: Allow restricting Iceberg data file reads to table location | Blocker | Iceberg integration | Janos Kovacs | Ayush Saxena |
| HIVE-27955 | Missing Postgres driver when start services from Docker compose | Major | . | Zhihua Deng | Zhihua Deng |
| HIVE-28016 | Iceberg: NULL column values handling in COW mode | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27880 | Iceberg: Support creating a branch on an empty table | Major | Iceberg integration | Butao Zhang | Butao Zhang |
| HIVE-27929 | Run TPC-DS queries and validate results correctness | Major | . | Denys Kuzmenko | Simhadri Govindappa |
OTHER:
| JIRA | Summary | Priority | Component | Reporter | Contributor |
|---|---|---|---|---|---|
| HIVE-27589 | Iceberg: Branches of Merge/Update statements should be committed atomically | Major | . | Denys Kuzmenko | Simhadri Govindappa |
| HIVE-27638 | Preparing for 4.0.0-beta-2 development | Major | . | Stamatis Zampetakis | Stamatis Zampetakis |
| HIVE-22618 | Fix checkstyle violations for ParseUtils | Minor | Query Processor | Shohei Okumiya | Shohei Okumiya |
| HIVE-27558 | HBase table query does not push BETWEEN predicate to storage layer | Major | . | Denys Kuzmenko | Dayakar M |
| HIVE-27526 | Cleaner should honor compaction writeIdHwm | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-27687 | Logger variable should be static final as its creation takes more time in query compilation | Major | Hive | Ramesh Kumar Thangarajan | Ramesh Kumar Thangarajan |
| HIVE-27843 | Add QueryOperation to Hive proto logger for post execution hook information | Major | . | Ramesh Kumar Thangarajan | Ramesh Kumar Thangarajan |
| HIVE-27907 | Upgrade aws-java-sdk version in HIVE | Minor | . | Devaspati Krishnatri | Devaspati Krishnatri |
| HIVE-27824 | Upgrade ivy to 2.5.2 and htmlunit to 2.70.0 | Major | Hive | Devaspati Krishnatri | Devaspati Krishnatri |
| HIVE-28005 | Remove upgrade-acid module | Major | . | Ayush Saxena | Butao Zhang |
| HIVE-28030 | LLAP util code refactor | Major | . | Denys Kuzmenko | Denys Kuzmenko |
| HIVE-28020 | Iceberg: Upgrade iceberg version to 1.4.3 | Major | . | Simhadri Govindappa | Simhadri Govindappa |
| HIVE-28043 | Upgrade ZooKeeper to 3.8.3 | Major | . | Anmol Sundaram | Anmol Sundaram |
| HIVE-28063 | Drop PerfLogger#setPerfLogger method and unused fields/methods | Major | Hive, Standalone Metastore | Stamatis Zampetakis | Stamatis Zampetakis |
Hive 4.0 - Overview of Major Changes
Dec 12, 2024Hive 4.0 - Overview of Major Changes
Iceberg Integration
- Advanced Snapshot management
- Branches & Tags support
- DML (insert/update/delete/merge)
- COW & MOR modes
- Vectorised Reads & Writes
- Table migration command
- LOAD DATA statements support
- Partition-level operations support
- Improved statistics (column stats support)
Hive ACID
- Use sequences for TXN_ID generation (performance)
- Read-only transactions optimization
- Zero-wait readers
- Optimistic and Pessimistic concurrency control
- Lockless reads
Compaction
- Rebalance compaction (Hive ACID)
- Compaction requests prioritization (compaction pooling)
- Iceberg compaction (Major)
Hive Metastore
- API optimization (performance)
- Dynamic leader election
- External data sources support
- HMS support for Thrift over HTTP
- JWT authentication for Thrift over HTTP
- HMS metadata summary
- Use Zookeeper for service discovery
HiveServer2
- Support SAML 2.0/JWT authentication mode
- Support both Kerberos and LDAP auth methods in parallel
- Graceful shutdown
- Easy access to the operation log through web UI
Hive Replication
- Optimised Bootstrap Solution
- Support for Snapshot Based Replication for External Table
- Better Replication Tracking Metrics
- Support for Checkpointing during Replication
Security
- Authorizations in alter table/view, UDFs, and Views created from Apache Spark.
- Authorizations on tables created based on storage handlers.
- Critical CVE fixes of transitive dependencies.
Compiler
- Materialized view support for Iceberg tables
- Improvements to refresh materialized views
- Date/Timestamp fixes and improvements
- Anti join support
- Split update support
- Branch pruning
- Column histogram statistics support
- Calcite upgrade to 1.25
- HPL/SQL improvements
- Scheduled query support
Miscl.
- Support for ESRI GeopSpatial UDF’s
- Added support for Apache Ozone
- Support Hadoop-3.3.6
- Supports Tez 0.10.3
- Works with Aarch64 (ARM)
- New UDFs (Hive UDFs)
- Deprecated Hive on MR & Removed Hive on Spark
- Deprecated Hive CLI
Introduction to Apache Hive
Dec 12, 2024The Apache Hive™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage and queried using SQL syntax.
Built on top of Apache Hadoop™, Hive provides the following features:
- Tools to enable easy access to data via SQL, thus enabling data warehousing tasks such as extract/transform/load (ETL), reporting, and data analysis.
- A mechanism to impose structure on a variety of data formats
- Access to files stored either directly in Apache HDFS™ or in other data storage systems such as Apache HBase™
- Query execution via Apache Tez™ or MapReduce
- Procedural language with HPL-SQL
- Sub-second query retrieval via Hive LLAP, Apache YARN and Apache Slider.
Hive provides standard SQL functionality, including many of the later SQL:2003, SQL:2011, and SQL:2016 features for analytics.
Hive’s SQL can also be extended with user code via user defined functions (UDFs), user defined aggregates (UDAFs), and user defined table functions (UDTFs).