org.bytedeco.arrow_dataset

Class ParquetDatasetFactory

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.DatasetFactory

org.bytedeco.arrow_dataset.ParquetDatasetFactory

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class ParquetDatasetFactory
extends DatasetFactory

\brief Create FileSystemDataset from custom _metadata cache file. Dask and other systems will generate a cache metadata file by concatenating the RowGroupMetaData of multiple parquet files into a single parquet file that only contains metadata and no ColumnChunk data. ParquetDatasetFactory creates a FileSystemDataset composed of ParquetFileFragment where each fragment is pre-populated with the exact number of row groups and statistics for each columns.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ParquetDatasetFactory(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
DatasetResult	Finish(FinishOptions options) \brief Create a Dataset with the given options
SchemaVectorResult	InspectSchemas(InspectOptions options) \brief Get the schemas of the Fragments and Partitioning.
static DatasetFactoryResult	Make(BytePointer metadata_path, FileSystem filesystem, ParquetFileFormat format, ParquetFactoryOptions options)
static DatasetFactoryResult	Make(FileSource metadata, BytePointer base_path, FileSystem filesystem, ParquetFileFormat format, ParquetFactoryOptions options)
static DatasetFactoryResult	Make(FileSource metadata, String base_path, FileSystem filesystem, ParquetFileFormat format, ParquetFactoryOptions options) \brief Create a ParquetDatasetFactory from a metadata source.
static DatasetFactoryResult	Make(String metadata_path, FileSystem filesystem, ParquetFileFormat format, ParquetFactoryOptions options) \brief Create a ParquetDatasetFactory from a metadata path.

Methods inherited from class org.bytedeco.arrow_dataset.DatasetFactory

Finish, Finish, Inspect, Inspect, root_partition, SetRootPartition

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ParquetDatasetFactory

public ParquetDatasetFactory(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

Make

@ByVal
public static DatasetFactoryResult Make(@StdString
                                                String metadata_path,
                                                @SharedPtr
                                                FileSystem filesystem,
                                                @SharedPtr
                                                ParquetFileFormat format,
                                                @ByVal
                                                ParquetFactoryOptions options)

\brief Create a ParquetDatasetFactory from a metadata path. The metadata_path will be read from filesystem. Each RowGroup contained in the metadata file will be relative to dirname(metadata_path).

Parameters:

metadata_path - [in] path of the metadata parquet file

filesystem - [in] from which to open/read the path

format - [in] to read the file with.

options - [in] see ParquetFactoryOptions

Make

@ByVal
public static DatasetFactoryResult Make(@StdString
                                                BytePointer metadata_path,
                                                @SharedPtr
                                                FileSystem filesystem,
                                                @SharedPtr
                                                ParquetFileFormat format,
                                                @ByVal
                                                ParquetFactoryOptions options)

Make

@ByVal
public static DatasetFactoryResult Make(@Const @ByRef
                                                FileSource metadata,
                                                @StdString
                                                String base_path,
                                                @SharedPtr
                                                FileSystem filesystem,
                                                @SharedPtr
                                                ParquetFileFormat format,
                                                @ByVal
                                                ParquetFactoryOptions options)

\brief Create a ParquetDatasetFactory from a metadata source. Similar to the previous Make definition, but the metadata can be a Buffer and the base_path is explicited instead of inferred from the metadata path.

Parameters:

metadata - [in] source to open the metadata parquet file from

base_path - [in] used as the prefix of every parquet files referenced

filesystem - [in] from which to read the files referenced.

format - [in] to read the file with.

options - [in] see ParquetFactoryOptions

Make

@ByVal
public static DatasetFactoryResult Make(@Const @ByRef
                                                FileSource metadata,
                                                @StdString
                                                BytePointer base_path,
                                                @SharedPtr
                                                FileSystem filesystem,
                                                @SharedPtr
                                                ParquetFileFormat format,
                                                @ByVal
                                                ParquetFactoryOptions options)

InspectSchemas

@ByVal
public SchemaVectorResult InspectSchemas(@ByVal
                                                 InspectOptions options)

Description copied from class: DatasetFactory

\brief Get the schemas of the Fragments and Partitioning.

Overrides:

InspectSchemas in class DatasetFactory

Finish

@ByVal
public DatasetResult Finish(@ByVal
                                    FinishOptions options)

Description copied from class: DatasetFactory

\brief Create a Dataset with the given options

Overrides:

Finish in class DatasetFactory