org.grobid.core.utilities

Class UnicodeUtil

java.lang.Object

org.grobid.core.utilities.UnicodeUtil

public class UnicodeUtil
extends java.lang.Object

Class for holding static methods for processing related to unicode.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	bullet_chars
static java.lang.String	close_parenthesis
static java.lang.String	horizontal_low_lines_chars
static java.lang.String	my_whitespace_chars
static java.lang.String	new_line_chars
static java.lang.String	open_parenthesis
static java.lang.String	vertical_lines_chars
static java.lang.String	whitespace_chars

Constructor Summary

Constructors

Constructor and Description
UnicodeUtil()

Method Summary

Modifier and Type	Method and Description
static java.lang.String	normaliseText(java.lang.String text) Normalise the space, EOL and punctuation unicode characters.
static java.lang.String	normaliseTextAndRemoveSpaces(java.lang.String text) Unicode normalisation of the token text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

whitespace_chars

public static java.lang.String whitespace_chars

my_whitespace_chars

public static java.lang.String my_whitespace_chars

horizontal_low_lines_chars

public static java.lang.String horizontal_low_lines_chars

vertical_lines_chars

public static java.lang.String vertical_lines_chars

new_line_chars

public static java.lang.String new_line_chars

bullet_chars

public static java.lang.String bullet_chars

open_parenthesis

public static java.lang.String open_parenthesis

close_parenthesis

public static java.lang.String close_parenthesis

Constructor Detail

UnicodeUtil

public UnicodeUtil()

Method Detail

normaliseText

public static java.lang.String normaliseText(java.lang.String text)

Normalise the space, EOL and punctuation unicode characters. In particular all the characters which are treated as space in C++ (http://en.cppreference.com/w/cpp/string/byte/isspace) will be replace by the punctuation space character U+2008 so that the token can be used to generate a robust feature vector legible as Wapiti input.

Parameters:

text - to be normalised

Returns:

normalised string, legible for Wapiti feature generation

normaliseTextAndRemoveSpaces

public static java.lang.String normaliseTextAndRemoveSpaces(java.lang.String text)

Unicode normalisation of the token text. Works as the normaliseText(java.lang.String), but in addition also removes spaces

Parameters:

text - to be normalised

Returns:

normalised string, legible for Wapiti feature generation