org.bytedeco.arrow_dataset

Class ScanOptions

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.ScanOptions

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @Properties(inherit=arrow_dataset.class)
public class ScanOptions
extends Pointer

Scan-specific options, which can be changed between scans of the same dataset.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
ScanOptions() Default native constructor.
ScanOptions(long size) Native array allocator.
ScanOptions(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
int	batch_readahead() How many batches to read ahead within a file Set to 0 to disable batch readahead Note: May not be supported by all formats Note: May not be supported by all scanners Note: Will be ignored if use_threads is set to false
ScanOptions	batch_readahead(int setter)
long	batch_size() Maximum row count for scanned batches.
ScanOptions	batch_size(long setter)
Schema	dataset_schema() Schema with which batches will be read from fragments.
ScanOptions	dataset_schema(Schema setter)
Expression	filter() A row filter (which will be pushed down to partitioning/reading if supported).
ScanOptions	filter(Expression setter)
int	fragment_readahead() How many files to read ahead Set to 0 to disable fragment readahead Note: May not be enforced by all scanners Note: Will be ignored if use_threads is set to false
ScanOptions	fragment_readahead(int setter)
FragmentScanOptions	fragment_scan_options() Fragment-specific scan options.
ScanOptions	fragment_scan_options(FragmentScanOptions setter)
ScanOptions	getPointer(long i)
IOContext	io_context() IOContext for any IO tasks Note: The IOContext executor will be ignored if use_threads is set to false
ScanOptions	io_context(IOContext setter)
StringVector	MaterializedFields()
MemoryPool	pool() A pool from which materialized and scanned arrays will be allocated.
ScanOptions	pool(MemoryPool setter)
ScanOptions	position(long position)
Schema	projected_schema() Schema of projected record batches.
ScanOptions	projected_schema(Schema setter)
Expression	projection() A projection expression (which can add/remove/rename columns).
ScanOptions	projection(Expression setter)
TaskGroup	TaskGroup()
boolean	use_async() If true then an asycnhronous implementation of the scanner will be used.
ScanOptions	use_async(boolean setter)
boolean	use_threads() If true the scanner will scan in parallel Note: If true, this will use threads from both the cpu_executor and the io_context.executor Note: This must be true in order for any readahead to happen
ScanOptions	use_threads(boolean setter)

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ScanOptions

public ScanOptions()

Default native constructor.

ScanOptions

public ScanOptions(long size)

Native array allocator. Access with Pointer.position(long).

ScanOptions

public ScanOptions(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

position

public ScanOptions position(long position)

Overrides:

position in class Pointer

getPointer

public ScanOptions getPointer(long i)

Overrides:

getPointer in class Pointer

filter

@ByRef
public Expression filter()

A row filter (which will be pushed down to partitioning/reading if supported).

filter

public ScanOptions filter(Expression setter)

projection

@ByRef
public Expression projection()

A projection expression (which can add/remove/rename columns).

projection

public ScanOptions projection(Expression setter)

dataset_schema

@SharedPtr
 @ByRef
public Schema dataset_schema()

Schema with which batches will be read from fragments. This is also known as the "reader schema" it will be used (for example) in constructing CSV file readers to identify column types for parsing. Usually only a subset of its fields (see MaterializedFields) will be materialized during a scan.

dataset_schema

public ScanOptions dataset_schema(Schema setter)

projected_schema

@SharedPtr
 @ByRef
public Schema projected_schema()

Schema of projected record batches. This is independent of dataset_schema as its fields are derived from the projection. For example, let dataset_schema = {"a": int32, "b": int32, "id": utf8} projection = project({equal(field_ref("a"), field_ref("b"))}, {"a_plus_b"}) (no filter specified). In this case, the projected_schema would be {"a_plus_b": int32}

projected_schema

public ScanOptions projected_schema(Schema setter)

batch_size

@Cast(value="int64_t")
public long batch_size()

Maximum row count for scanned batches.

batch_size

public ScanOptions batch_size(long setter)

batch_readahead

public int batch_readahead()

How many batches to read ahead within a file Set to 0 to disable batch readahead Note: May not be supported by all formats Note: May not be supported by all scanners Note: Will be ignored if use_threads is set to false

batch_readahead

public ScanOptions batch_readahead(int setter)

fragment_readahead

public int fragment_readahead()

How many files to read ahead Set to 0 to disable fragment readahead Note: May not be enforced by all scanners Note: Will be ignored if use_threads is set to false

fragment_readahead

public ScanOptions fragment_readahead(int setter)

pool

public MemoryPool pool()

A pool from which materialized and scanned arrays will be allocated.

pool

public ScanOptions pool(MemoryPool setter)

io_context

@ByRef
public IOContext io_context()

IOContext for any IO tasks Note: The IOContext executor will be ignored if use_threads is set to false

io_context

public ScanOptions io_context(IOContext setter)

use_threads

@Cast(value="bool")
public boolean use_threads()

If true the scanner will scan in parallel Note: If true, this will use threads from both the cpu_executor and the io_context.executor Note: This must be true in order for any readahead to happen

use_threads

public ScanOptions use_threads(boolean setter)

use_async

@Cast(value="bool")
public boolean use_async()

If true then an asycnhronous implementation of the scanner will be used. This implementation is newer and generally performs better. However, it makes extensive use of threading and is still considered experimental

use_async

public ScanOptions use_async(boolean setter)

fragment_scan_options

@SharedPtr
public FragmentScanOptions fragment_scan_options()

Fragment-specific scan options.

fragment_scan_options

public ScanOptions fragment_scan_options(FragmentScanOptions setter)

MaterializedFields

@ByVal
public StringVector MaterializedFields()

TaskGroup

@SharedPtr
public TaskGroup TaskGroup()