Advanced Analytics Forum

How to enable dynam...
 

[Solved] How to enable dynamic partition in hive?  

  RSS

Jitesh Reddy
(@hadoopuser)
New Member
Joined: 1 year ago
Posts: 4
16/08/2018 4:07 am  

Hi,

I am looking to do dynamic partition here in Hive.

Can you please tell me how to do it?

-Jitesh


Quote
HDFS Tutorial Team
(@ashuhadoop)
Member Admin
Joined: 3 years ago
Posts: 4
16/08/2018 4:11 am  

Hi Jitesh,

First of all thanks for asking the question. To enable the dynamic property in Hive, you need to enable the following 2 properties in Hive. You can run these 2 from Hive shell and then you will be able to create a dynamic partition.

set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;

Hope this will help. If you need more details on this, please check this link-

https://www.hdfstutorial.com/blog/capgemini-hadoop-interview-questions-and-answers/#14_What_is_Hive_Partition_and_have_you_used_hive_partition


ReplyQuote
Jitesh Reddy
(@hadoopuser)
New Member
Joined: 1 year ago
Posts: 4
16/08/2018 4:13 am  
Posted by: HDFS Tutorial Team

Hi Jitesh,

First of all thanks for asking the question. To enable the dynamic property in Hive, you need to enable the following 2 properties in Hive. You can run these 2 from Hive shell and then you will be able to create a dynamic partition.

set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;

Hope this will help. If you need more details on this, please check this link-

https://www.hdfstutorial.com/blog/capgemini-hadoop-interview-questions-and-answers/#14_What_is_Hive_Partition_and_have_you_used_hive_partition

Thanks so much 😀 


ReplyQuote
kaduswapnali456
(@kaduswapnali456)
New Member
Joined: 1 year ago
Posts: 1
07/12/2018 9:14 am  

Partitions are defined at the time of table creation using the PARTITIONED BY clause, with a list of column definitions for partitioning.

Syntax

CREATE [EXTERNAL] TABLE table_name (col_name_1 data_type_1, ....)

PARTITIONED BY (col_name_n data_type_n [COMMENT col_comment], ...);

CREATE [EXTERNAL] TABLE table_name (col_name_1 data_type_1, ....)

PARTITIONED BY (col_name_n data_type_n [COMMENT col_comment], ...);

As shown in syntax, we can also add comments to partitioned columns.

Advantages

  • Partitioning is used for distributing execution load horizontally.
  • As the data is stored as slices/parts, query response time is faster to process the small part of the data instead of looking for a search in the entire data set.
  • For example, In a large user table where the table is partitioned by country, then selecting users of country ‘IN’ will just scan one directory ‘country=IN’ instead of all the directories.

ReplyQuote
Share:

Please Login or Register