org.bytedeco.arrow_dataset

Class Fragment

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.Fragment

All Implemented Interfaces:

AutoCloseable

Direct Known Subclasses:

FileFragment, InMemoryFragment

@Namespace(value="arrow::dataset")
 @NoOffset
 @Properties(inherit=arrow_dataset.class)
public class Fragment
extends Pointer

\brief A granular piece of a Dataset, such as an individual file. A Fragment can be read/scanned separately from other fragments. It yields a collection of RecordBatches when scanned, encapsulated in one or more ScanTasks. Note that Fragments have well defined physical schemas which are reconciled by the Datasets which contain them; these physical schemas may differ from a parent Dataset's schema and the physical schemas of sibling Fragments.

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
Fragment(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
LongOptionalFuture	CountRows(Expression predicate, ScanOptions options) \brief Count the number of rows in this fragment matching the filter using metadata only.
Expression	partition_expression() \brief An expression which evaluates to true for all data viewed by this Fragment.
SchemaResult	ReadPhysicalSchema() \brief Return the physical schema of the Fragment.
ScanTaskIteratorResult	Scan(ScanOptions options) \brief Scan returns an iterator of ScanTasks, each of which yields RecordBatches from this Fragment.
RecordBatchGeneratorResult	ScanBatchesAsync(ScanOptions options) An asynchronous version of Scan
String	ToString()
String	type_name()

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

Fragment

public Fragment(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

ReadPhysicalSchema

@ByVal
public SchemaResult ReadPhysicalSchema()

\brief Return the physical schema of the Fragment. The physical schema is also called the writer schema. This method is blocking and may suffer from high latency filesystem. The schema is cached after being read once, or may be specified at construction.

Scan

@ByVal
public ScanTaskIteratorResult Scan(@SharedPtr
                                           ScanOptions options)

\brief Scan returns an iterator of ScanTasks, each of which yields RecordBatches from this Fragment. Note that batches yielded using this method will not be filtered and may not align with the Fragment's schema. In particular, note that columns referenced by the filter may be present in yielded batches even if they are not projected (so that they are available when a filter is applied). Additionally, explicitly projected columns may be absent if they were not present in this fragment. To receive a record batch stream which is fully filtered and projected, use Scanner.

ScanBatchesAsync

@ByVal
public RecordBatchGeneratorResult ScanBatchesAsync(@SharedPtr
                                                           ScanOptions options)

An asynchronous version of Scan

CountRows

@ByVal
public LongOptionalFuture CountRows(@ByVal
                                            Expression predicate,
                                            @SharedPtr
                                            ScanOptions options)

\brief Count the number of rows in this fragment matching the filter using metadata only. That is, this method may perform I/O, but will not load data. If this is not possible, resolve with an empty optional. The fragment can perform I/O (e.g. to read metadata) before it deciding whether it can satisfy the request.

type_name

@StdString
public String type_name()

ToString

@StdString
public String ToString()

partition_expression

@Const
 @ByRef
public Expression partition_expression()

\brief An expression which evaluates to true for all data viewed by this Fragment.