org.bytedeco.arrow_dataset

Class ScannerBuilder

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.ScannerBuilder

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class ScannerBuilder
extends Pointer

\brief ScannerBuilder is a factory class to construct a Scanner. It is used to pass information, notably a potential filter expression and a subset of columns to materialize.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ScannerBuilder(Dataset dataset)
ScannerBuilder(Dataset dataset, ScanOptions scan_options)
ScannerBuilder(Pointer p) Pointer cast constructor.
ScannerBuilder(Schema schema, Fragment fragment, ScanOptions scan_options)

Method Summary

Modifier and Type	Method and Description
Status	BatchSize(long batch_size) \brief Set the maximum number of rows per RecordBatch.
Status	Filter(Expression filter) \brief Set the filter expression to return only rows matching the filter.
ScannerResult	Finish() \brief Return the constructed now-immutable Scanner object
Status	FragmentReadahead(int fragment_readahead) \brief Limit how many fragments the scanner will read at once Note: This is only enforced in "async" mode
Status	FragmentScanOptions(FragmentScanOptions fragment_scan_options) \brief Set fragment-specific scan options.
static ScannerBuilder	FromRecordBatchReader(RecordBatchReader reader) \brief Make a scanner from a record batch reader.
Status	Pool(MemoryPool pool) \brief Set the pool from which materialized and scanned arrays will be allocated.
Status	Project(Expression exprs, StringVector names) \brief Set expressions which will be evaluated to produce the materialized columns.
Status	Project(StringVector columns) \brief Set the subset of columns to materialize.
Schema	projected_schema()
Schema	schema()
Status	UseAsync()
Status	UseAsync(boolean use_async) \brief Indicate if the Scanner should run in experimental "async" mode This mode should have considerably better performance on high-latency or parallel filesystems but is still experimental
Status	UseThreads()
Status	UseThreads(boolean use_threads) \brief Indicate if the Scanner should make use of the available ThreadPool found in ScanOptions;

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ScannerBuilder

public ScannerBuilder(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

ScannerBuilder

public ScannerBuilder(@SharedPtr @ByVal
                      Dataset dataset)

ScannerBuilder

public ScannerBuilder(@SharedPtr @ByVal
                      Dataset dataset,
                      @SharedPtr
                      ScanOptions scan_options)

ScannerBuilder

public ScannerBuilder(@SharedPtr @ByVal
                      Schema schema,
                      @SharedPtr @ByVal
                      Fragment fragment,
                      @SharedPtr
                      ScanOptions scan_options)

Method Detail

FromRecordBatchReader

@SharedPtr
 @ByVal
public static ScannerBuilder FromRecordBatchReader(@SharedPtr
                                                                        RecordBatchReader reader)

\brief Make a scanner from a record batch reader. The resulting scanner can be scanned only once. This is intended to support writing data from streaming sources or other sources that can be iterated only once.

Project

@ByVal
public Status Project(@ByVal
                              StringVector columns)

\brief Set the subset of columns to materialize. Columns which are not referenced may not be read from fragments.

Parameters:

columns - [in] list of columns to project. Order and duplicates will be preserved.

Returns:

Failure if any column name does not exists in the dataset's Schema.

Project

@ByVal
public Status Project(@StdVector
                              Expression exprs,
                              @ByVal
                              StringVector names)

\brief Set expressions which will be evaluated to produce the materialized columns. Columns which are not referenced may not be read from fragments.

Parameters:

exprs - [in] expressions to evaluate to produce columns.

names - [in] list of names for the resulting columns.

Returns:

Failure if any referenced column does not exists in the dataset's Schema.

Filter

@ByVal
public Status Filter(@Const @ByRef
                             Expression filter)

\brief Set the filter expression to return only rows matching the filter. The predicate will be passed down to Sources and corresponding Fragments to exploit predicate pushdown if possible using partition information or Fragment internal metadata, e.g. Parquet statistics. Columns which are not referenced may not be read from fragments.

Parameters:

filter - [in] expression to filter rows with.

Returns:

Failure if any referenced columns does not exist in the dataset's Schema.

UseThreads

@ByVal
public Status UseThreads(@Cast(value="bool")
                                 boolean use_threads)

\brief Indicate if the Scanner should make use of the available ThreadPool found in ScanOptions;

UseThreads

@ByVal
public Status UseThreads()

FragmentReadahead

@ByVal
public Status FragmentReadahead(int fragment_readahead)

\brief Limit how many fragments the scanner will read at once Note: This is only enforced in "async" mode

UseAsync

@ByVal
public Status UseAsync(@Cast(value="bool")
                               boolean use_async)

\brief Indicate if the Scanner should run in experimental "async" mode This mode should have considerably better performance on high-latency or parallel filesystems but is still experimental

UseAsync

@ByVal
public Status UseAsync()

BatchSize

@ByVal
public Status BatchSize(@Cast(value="int64_t")
                                long batch_size)

\brief Set the maximum number of rows per RecordBatch.

Parameters:

batch_size - [in] the maximum number of rows.

Returns:

An error if the number for batch is not greater than 0. This option provides a control limiting the memory owned by any RecordBatch.

Pool

@ByVal
public Status Pool(MemoryPool pool)

\brief Set the pool from which materialized and scanned arrays will be allocated.

FragmentScanOptions

@ByVal
public Status FragmentScanOptions(@SharedPtr
                                          FragmentScanOptions fragment_scan_options)

\brief Set fragment-specific scan options.

Finish

@ByVal
public ScannerResult Finish()

\brief Return the constructed now-immutable Scanner object

schema

@Const
 @SharedPtr
 @ByRef
public Schema schema()

projected_schema

@Const
 @SharedPtr
 @ByRef
public Schema projected_schema()