org.bytedeco.arrow_dataset

Class ParquetFileFragment

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.Fragment

org.bytedeco.arrow_dataset.FileFragment

org.bytedeco.arrow_dataset.ParquetFileFragment

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class ParquetFileFragment
extends FileFragment

\brief A FileFragment with parquet logic. ParquetFileFragment provides a lazy (with respect to IO) interface to scan parquet files. Any heavy IO calls are deferred to the Scan() method. The caller can provide an optional list of selected RowGroups to limit the number of scanned RowGroups, or to partition the scans across multiple threads. Metadata can be explicitly provided, enabling pushdown predicate benefits without the potentially heavy IO of loading Metadata from the file system. This can induce significant performance boost when scanning high latency file systems.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ParquetFileFragment(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
Status	EnsureCompleteMetadata()
Status	EnsureCompleteMetadata(FileReader reader) \brief Ensure this fragment's FileMetaData is in memory.
FileMetaData	metadata() \brief Return the FileMetaData associated with this fragment.
IntPointer	row_groups() \brief Return the RowGroups selected by this fragment.
FragmentResult	SplitByRowGroup(Expression predicate)
FragmentResult	Subset(Expression predicate) \brief Return fragment which selects a filtered subset of this fragment's RowGroups.
FragmentResult	Subset(int[] row_group_ids)
FragmentResult	Subset(IntBuffer row_group_ids)
FragmentResult	Subset(IntPointer row_group_ids)

Methods inherited from class org.bytedeco.arrow_dataset.FileFragment

CountRows, format, Scan, ScanBatchesAsync, source, ToString, type_name

Methods inherited from class org.bytedeco.arrow_dataset.Fragment

partition_expression, ReadPhysicalSchema

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetFileFragment

public ParquetFileFragment(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

SplitByRowGroup

@ByVal
public FragmentResult SplitByRowGroup(@ByVal
                                              Expression predicate)

row_groups

@StdVector
public IntPointer row_groups()

\brief Return the RowGroups selected by this fragment.

metadata

@SharedPtr
public FileMetaData metadata()

\brief Return the FileMetaData associated with this fragment.

EnsureCompleteMetadata

@ByVal
public Status EnsureCompleteMetadata(FileReader reader)

\brief Ensure this fragment's FileMetaData is in memory.

EnsureCompleteMetadata

@ByVal
public Status EnsureCompleteMetadata()

Subset

@ByVal
public FragmentResult Subset(@ByVal
                                     Expression predicate)

\brief Return fragment which selects a filtered subset of this fragment's RowGroups.

Subset

@ByVal
public FragmentResult Subset(@StdVector
                                     IntPointer row_group_ids)

Subset

@ByVal
public FragmentResult Subset(@StdVector
                                     IntBuffer row_group_ids)

Subset

@ByVal
public FragmentResult Subset(@StdVector
                                     int[] row_group_ids)