@Namespace(value="tensorflow::ops") @NoOffset @Properties(inherit=tensorflow.class) public class UnicodeTranscode extends Pointer
errors
attribute sets the policy for how to deal with them. If the default
error-handling policy is used, invalid formatting will be substituted in the
output by the replacement_char
. If the errors policy is to ignore
, any
invalid encoding positions in the input are skipped and not included in the
output. If it set to strict
then any invalid formatting will result in an
InvalidArgument error.
This operation can be used with output_encoding = input_encoding
to enforce
correct formatting for inputs even if they are already in the desired encoding.
If the input is prefixed by a Byte Order Mark needed to determine encoding
(e.g. if the encoding is UTF-16 and the BOM indicates big-endian), then that
BOM will be consumed and not emitted into the output. If the input encoding
is marked with an explicit endianness (e.g. UTF-16-BE), then the BOM is
interpreted as a non-breaking-space and is preserved in the output (including
always for UTF-8).
The end result is that if the input is marked as an explicit endianness the
transcoding is faithful to all codepoints in the source. If it is not marked
with an explicit endianness, the BOM is not considered part of the string itself
but as metadata, and so is not preserved in the output.
Arguments:
* scope: A Scope object
* input: The text to be processed. Can have any shape.
* input_encoding: Text encoding of the input strings. This is any of the encodings supported
by ICU ucnv algorithmic converters. Examples: "UTF-16", "US ASCII", "UTF-8"
.
* output_encoding: The unicode encoding to use in the output. Must be one of
"UTF-8", "UTF-16-BE", "UTF-32-BE"
. Multi-byte encodings will be big-endian.
Optional attributes (see Attrs
):
* errors: Error handling policy when there is invalid formatting found in the input.
The value of 'strict' will cause the operation to produce a InvalidArgument
error on any invalid input formatting. A value of 'replace' (the default) will
cause the operation to replace any invalid formatting in the input with the
replacement_char
codepoint. A value of 'ignore' will cause the operation to
skip any invalid formatting in the input and produce no corresponding output
character.
* replacement_char: The replacement character codepoint to be used in place of any invalid
formatting in the input when errors='replace'
. Any valid unicode codepoint may
be used. The default value is the default unicode replacement character is
0xFFFD or U+65533.)
Note that for UTF-8, passing a replacement character expressible in 1 byte, such
as ' ', will preserve string alignment to the source since invalid bytes will be
replaced with a 1-byte replacement. For UTF-16-BE and UTF-16-LE, any 1 or 2 byte
replacement character will preserve byte alignment to the source.
* replace_control_characters: Whether to replace the C0 control characters (00-1F) with the
replacement_char
. Default is false.
Returns:
* Output
: A string tensor containing unicode text encoded using output_encoding
.Modifier and Type | Class and Description |
---|---|
static class |
UnicodeTranscode.Attrs
Optional attribute setters for UnicodeTranscode
|
Pointer.CustomDeallocator, Pointer.Deallocator, Pointer.NativeDeallocator, Pointer.ReferenceCounter
Constructor and Description |
---|
UnicodeTranscode(Pointer p)
Pointer cast constructor.
|
UnicodeTranscode(Scope scope,
Input input,
BytePointer input_encoding,
BytePointer output_encoding) |
UnicodeTranscode(Scope scope,
Input input,
BytePointer input_encoding,
BytePointer output_encoding,
UnicodeTranscode.Attrs attrs) |
UnicodeTranscode(Scope scope,
Input input,
String input_encoding,
String output_encoding) |
UnicodeTranscode(Scope scope,
Input input,
String input_encoding,
String output_encoding,
UnicodeTranscode.Attrs attrs) |
Modifier and Type | Method and Description |
---|---|
Input |
asInput() |
Output |
asOutput() |
static UnicodeTranscode.Attrs |
Errors(BytePointer x) |
static UnicodeTranscode.Attrs |
Errors(String x) |
Node |
node() |
Operation |
operation() |
UnicodeTranscode |
operation(Operation setter) |
Output |
output() |
UnicodeTranscode |
output(Output setter) |
static UnicodeTranscode.Attrs |
ReplaceControlCharacters(boolean x) |
static UnicodeTranscode.Attrs |
ReplacementChar(long x) |
address, asBuffer, asByteBuffer, availablePhysicalBytes, calloc, capacity, capacity, close, deallocate, deallocate, deallocateReferences, deallocator, deallocator, equals, fill, formatBytes, free, getDirectBufferAddress, getPointer, getPointer, getPointer, getPointer, hashCode, interruptDeallocatorThread, isNull, isNull, limit, limit, malloc, maxBytes, maxPhysicalBytes, memchr, memcmp, memcpy, memmove, memset, offsetAddress, offsetof, offsetof, parseBytes, physicalBytes, physicalBytesInaccurate, position, position, put, realloc, referenceCount, releaseReference, retainReference, setNull, sizeof, sizeof, toString, totalBytes, totalCount, totalPhysicalBytes, withDeallocator, zero
public UnicodeTranscode(Pointer p)
Pointer(Pointer)
.public UnicodeTranscode(@Const @ByRef Scope scope, @ByVal Input input, @tensorflow.StringPiece BytePointer input_encoding, @tensorflow.StringPiece BytePointer output_encoding)
public UnicodeTranscode(@Const @ByRef Scope scope, @ByVal Input input, @tensorflow.StringPiece String input_encoding, @tensorflow.StringPiece String output_encoding)
public UnicodeTranscode(@Const @ByRef Scope scope, @ByVal Input input, @tensorflow.StringPiece BytePointer input_encoding, @tensorflow.StringPiece BytePointer output_encoding, @Const @ByRef UnicodeTranscode.Attrs attrs)
public UnicodeTranscode(@Const @ByRef Scope scope, @ByVal Input input, @tensorflow.StringPiece String input_encoding, @tensorflow.StringPiece String output_encoding, @Const @ByRef UnicodeTranscode.Attrs attrs)
public Node node()
@ByVal public static UnicodeTranscode.Attrs Errors(@tensorflow.StringPiece BytePointer x)
@ByVal public static UnicodeTranscode.Attrs Errors(@tensorflow.StringPiece String x)
@ByVal public static UnicodeTranscode.Attrs ReplacementChar(@Cast(value="tensorflow::int64") long x)
@ByVal public static UnicodeTranscode.Attrs ReplaceControlCharacters(@Cast(value="bool") boolean x)
public UnicodeTranscode operation(Operation setter)
public UnicodeTranscode output(Output setter)
Copyright © 2022. All rights reserved.