org.bytedeco.arrow_dataset

Class FileSystemFactoryOptions

java.lang.Object

org.bytedeco.javacpp.Pointer

org.bytedeco.arrow_dataset.FileSystemFactoryOptions

All Implemented Interfaces:

AutoCloseable

@Namespace(value="arrow::dataset")
 @Properties(inherit=arrow_dataset.class)
public class FileSystemFactoryOptions
extends Pointer

\ingroup dataset-filesystem

Nested Class Summary

Nested classes/interfaces inherited from class org.bytedeco.javacpp.Pointer

Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter

Field Summary

Fields inherited from class org.bytedeco.javacpp.Pointer

address, capacity, limit, position

Constructor Summary

Constructors

Constructor and Description
FileSystemFactoryOptions() Default native constructor.
FileSystemFactoryOptions(long size) Native array allocator.
FileSystemFactoryOptions(Pointer p) Pointer cast constructor.

Method Summary

Modifier and Type	Method and Description
boolean	exclude_invalid_files() Invalid files (via selector or explicitly) will be excluded by checking with the FileFormat::IsSupported method.
FileSystemFactoryOptions	exclude_invalid_files(boolean setter)
FileSystemFactoryOptions	getPointer(long i)
String	partition_base_dir() For the purposes of applying the partitioning, paths will be stripped of the partition_base_dir.
FileSystemFactoryOptions	partition_base_dir(String setter)
PartitioningOrFactory	partitioning() Either an explicit Partitioning or a PartitioningFactory to discover one.
FileSystemFactoryOptions	partitioning(PartitioningOrFactory setter)
FileSystemFactoryOptions	position(long position)
StringVector	selector_ignore_prefixes() When discovering from a Selector (and not from an explicit file list), ignore files and directories matching any of these prefixes.
FileSystemFactoryOptions	selector_ignore_prefixes(StringVector setter)

Methods inherited from class org.bytedeco.javacpp.Pointer

address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

FileSystemFactoryOptions

public FileSystemFactoryOptions()

Default native constructor.

FileSystemFactoryOptions

public FileSystemFactoryOptions(long size)

Native array allocator. Access with Pointer.position(long).

FileSystemFactoryOptions

public FileSystemFactoryOptions(Pointer p)

Pointer cast constructor. Invokes Pointer(Pointer).

Method Detail

position

public FileSystemFactoryOptions position(long position)

Overrides:

position in class Pointer

getPointer

public FileSystemFactoryOptions getPointer(long i)

Overrides:

getPointer in class Pointer

partitioning

@ByRef
public PartitioningOrFactory partitioning()

Either an explicit Partitioning or a PartitioningFactory to discover one. If a factory is provided, it will be used to infer a schema for partition fields based on file and directory paths then construct a Partitioning. The default is a Partitioning which will yield no partition information. The (explicit or discovered) partitioning will be applied to discovered files and the resulting partition information embedded in the Dataset.

partitioning

public FileSystemFactoryOptions partitioning(PartitioningOrFactory setter)

partition_base_dir

@StdString
public String partition_base_dir()

For the purposes of applying the partitioning, paths will be stripped of the partition_base_dir. Files not matching the partition_base_dir prefix will be skipped for partition discovery. The ignored files will still be part of the Dataset, but will not have partition information. Example: partition_base_dir = "/dataset"; - "/dataset/US/sales.csv" -> "US/sales.csv" will be given to the partitioning - "/home/john/late_sales.csv" -> Will be ignored for partition discovery. This is useful for partitioning which parses directory when ordering is important, e.g. DirectoryPartitioning.

partition_base_dir

public FileSystemFactoryOptions partition_base_dir(String setter)

exclude_invalid_files

@Cast(value="bool")
public boolean exclude_invalid_files()

Invalid files (via selector or explicitly) will be excluded by checking with the FileFormat::IsSupported method. This will incur IO for each files in a serial and single threaded fashion. Disabling this feature will skip the IO, but unsupported files may be present in the Dataset (resulting in an error at scan time).

exclude_invalid_files

public FileSystemFactoryOptions exclude_invalid_files(boolean setter)

selector_ignore_prefixes

@ByRef
public StringVector selector_ignore_prefixes()

When discovering from a Selector (and not from an explicit file list), ignore files and directories matching any of these prefixes. Example (with selector = "/dataset/**"): selector_ignore_prefixes = {"_", ".DS_STORE" }; - "/dataset/data.csv" -> not ignored - "/dataset/_metadata" -> ignored - "/dataset/.DS_STORE" -> ignored - "/dataset/_hidden/dat" -> ignored - "/dataset/nested/.DS_STORE" -> ignored

selector_ignore_prefixes

public FileSystemFactoryOptions selector_ignore_prefixes(StringVector setter)