org.bytedeco.arrow_dataset

Class Scanner

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.Scanner

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class Scanner
extends Pointer

\defgroup dataset-scanning Scanning API \{

\brief A scanner glues together several dataset classes to load in data. The dataset contains a collection of fragments and partitioning rules. The fragments identify independently loadable units of data (i.e. each fragment has a potentially unique schema and possibly even format. It should be possible to read fragments in parallel if desired). The fragment's format contains the logic necessary to actually create a task to load the fragment into memory. That task may or may not support parallel execution of its own. The scanner is then responsible for creating scan tasks from every fragment in the dataset and (potentially) sequencing the loaded record batches together. The scanner should not buffer the entire dataset in memory (unless asked) instead yielding record batches as soon as they are ready to scan. Various readahead properties control how much data is allowed to be scanned before pausing to let a slow consumer catchup. Today the scanner also handles projection & filtering although that may change in the future.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
Scanner(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
LongResult	CountRows() \brief Count rows matching a predicate.
Dataset	dataset() \brief Get the dataset that this scanner will scan
TableResult	Head(long num_rows) \brief Get the first N rows.
ScanOptions	options() \brief Get the options for this scan.
ScanTaskIteratorResult	Scan() Deprecated.
Status	Scan(arrow_dataset.TaggedRecordBatchVisitor visitor) \brief Apply a visitor to each RecordBatch as it is scanned.
TaggedRecordBatchIteratorResult	ScanBatches() \brief Scan the dataset into a stream of record batches.
TaggedRecordBatchGeneratorResult	ScanBatchesAsync()
EnumeratedRecordBatchIteratorResult	ScanBatchesUnordered() \brief Scan the dataset into a stream of record batches.
EnumeratedRecordBatchGeneratorResult	ScanBatchesUnorderedAsync()
TableResult	TakeRows(Array indices) \brief A convenience to synchronously load the given rows by index.
RecordBatchReaderSharedResult	ToRecordBatchReader() \brief Convert the Scanner to a RecordBatchReader so it can be easily used with APIs that expect a reader.
TableResult	ToTable() \brief Convert a Scanner into a Table.

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

Scanner

public Scanner(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

Scan

@Deprecated
 @ByVal
public ScanTaskIteratorResult Scan()

Deprecated.

\brief The Scan operator returns a stream of ScanTask. The caller is responsible to dispatch/schedule said tasks. Tasks should be safe to run in a concurrent fashion and outlive the iterator. Note: Not supported by the async scanner Planned for removal from the public API in ARROW-11782.

Scan

@ByVal
public Status Scan(@ByVal
                           arrow_dataset.TaggedRecordBatchVisitor visitor)

\brief Apply a visitor to each RecordBatch as it is scanned. If multiple threads are used (via use_threads), the visitor will be invoked from those threads and is responsible for any synchronization.

ToTable

@ByVal
public TableResult ToTable()

\brief Convert a Scanner into a Table. Use this convenience utility with care. This will serially materialize the Scan result in memory before creating the Table.

ScanBatches

@ByVal
public TaggedRecordBatchIteratorResult ScanBatches()

\brief Scan the dataset into a stream of record batches. Each batch is tagged with the fragment it originated from. The batches will arrive in order. The order of fragments is determined by the dataset. Note: The scanner will perform some readahead but will avoid materializing too much in memory (this is goverended by the readahead options and use_threads option). If the readahead queue fills up then I/O will pause until the calling thread catches up.

ScanBatchesAsync

@ByVal
public TaggedRecordBatchGeneratorResult ScanBatchesAsync()

ScanBatchesUnordered

@ByVal
public EnumeratedRecordBatchIteratorResult ScanBatchesUnordered()

\brief Scan the dataset into a stream of record batches. Unlike ScanBatches this method may allow record batches to be returned out of order. This allows for more efficient scanning: some fragments may be accessed more quickly than others (e.g. may be cached in RAM or just happen to get scheduled earlier by the I/O) To make up for the out-of-order iteration each batch is further tagged with positional information.

ScanBatchesUnorderedAsync

@ByVal
public EnumeratedRecordBatchGeneratorResult ScanBatchesUnorderedAsync()

TakeRows

@ByVal
public TableResult TakeRows(@Const @ByRef
                                    Array indices)

\brief A convenience to synchronously load the given rows by index. Will only consume as many batches as needed from ScanBatches().

Head

@ByVal
public TableResult Head(@Cast(value="int64_t")
                                long num_rows)

\brief Get the first N rows.

CountRows

@ByVal
public LongResult CountRows()

\brief Count rows matching a predicate. This method will push down the predicate and compute the result based on fragment metadata if possible.

ToRecordBatchReader

@ByVal
public RecordBatchReaderSharedResult ToRecordBatchReader()

\brief Convert the Scanner to a RecordBatchReader so it can be easily used with APIs that expect a reader.

options

@SharedPtr
public ScanOptions options()

\brief Get the options for this scan.

dataset

@Const
 @SharedPtr
 @ByRef
public Dataset dataset()

\brief Get the dataset that this scanner will scan