org.apache.hadoop.hive.ql.io
Class RCFile

java.lang.Object
  org.apache.hadoop.hive.ql.io.RCFile

public class RCFile
extends Object

RCFiles, short of Record Columnar File, are flat files consisting of binary key/value pairs, which shares much similarity with SequenceFile. RCFile stores columns of a table in a record columnar way. It first partitions rows horizontally into row splits. and then it vertically partitions each row split in a columnar way. RCFile first stores the meta data of a row split, as the key part of a record, and all the data of a row split as the value part. When writing, RCFile.Writer first holds records' value bytes in memory, and determines a row split if the raw bytes size of buffered records overflow a given parameterWriter.columnsBufferSize, which can be set like: conf.setInt(COLUMNS_BUFFER_SIZE_CONF_STR, 4 * 1024 * 1024) .

RCFile provides RCFile.Writer, RCFile.Reader and classes for writing, reading respectively.

RCFile stores columns of a table in a record columnar way. It first partitions rows horizontally into row splits. and then it vertically partitions each row split in a columnar way. RCFile first stores the meta data of a row split, as the key part of a record, and all the data of a row split as the value part.

RCFile compresses values in a more fine-grained manner then record level compression. However, It currently does not support compress the key part yet. The actual compression algorithm used to compress key and/or values can be specified by using the appropriate CompressionCodec.

The RCFile.Reader is used to read and explain the bytes of RCFile.

RCFile Formats

RC Header

• version - 3 bytes of magic header SEQ, followed by 1 byte of actual version number (e.g. SEQ4 or SEQ6)
• keyClassName -KeyBuffer's class name
• valueClassName - ValueBuffer's class name
• compression - A boolean which specifies if compression is turned on for keys/values in this file.
• blockCompression - always false. this field is kept for compatible with SequeceFile's format
• compression codec - CompressionCodec class which is used for compression of keys and/or values (if compression is enabled).
• metadata - SequenceFile.Metadata for this file.
• sync - A sync marker to denote end of the header.

RCFile Format

• Header
• Record
• Key part
  • Record length in bytes
  • Key length in bytes
  • Number_of_rows_in_this_record(vint)
  • Column_1_ondisk_length(vint)
  • Column_1_row_1_value_plain_length
  • Column_1_row_2_value_plain_length
  • ...
  • Column_2_ondisk_length(vint)
  • Column_2_row_1_value_plain_length
  • Column_2_row_2_value_plain_length
  • ...
• Value part
  • Compressed or plain data of [column_1_row_1_value, column_1_row_2_value,....]
  • Compressed or plain data of [column_2_row_1_value, column_2_row_2_value,....]

Nested Class Summary

static class	RCFile.Reader Read KeyBuffer/ValueBuffer pairs from a RCFile.
static class	RCFile.Writer Write KeyBuffer/ValueBuffer pairs to a RCFile.

Field Summary

static String	COLUMN_NUMBER_CONF_STR
static String	COLUMN_NUMBER_METADATA_STR
static String	RECORD_INTERVAL_CONF_STR
static int	SYNC_INTERVAL The number of bytes between sync points.

Constructor Summary

RCFile()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RECORD_INTERVAL_CONF_STR

public static String RECORD_INTERVAL_CONF_STR

COLUMN_NUMBER_METADATA_STR

public static String COLUMN_NUMBER_METADATA_STR

COLUMN_NUMBER_CONF_STR

public static String COLUMN_NUMBER_CONF_STR

SYNC_INTERVAL

public static final int SYNC_INTERVAL

The number of bytes between sync points.

See Also:

Constant Field Values

Constructor Detail

RCFile

public RCFile()