org.grobid.core.analyzers

Class GrobidDefaultAnalyzer

java.lang.Object

org.grobid.core.analyzers.GrobidDefaultAnalyzer

All Implemented Interfaces:

Analyzer

public class GrobidDefaultAnalyzer
extends java.lang.Object
implements Analyzer

Default tokenizer adequate for all Indo-European languages.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	delimiters

Method Summary

Modifier and Type	Method and Description
static GrobidDefaultAnalyzer	getInstance()
java.lang.String	getName()
java.util.List<java.lang.String>	retokenize(java.util.List<java.lang.String> chunks)
java.util.List<java.lang.String>	tokenize(java.lang.String text)
java.util.List<java.lang.String>	tokenize(java.lang.String text, Language lang)
java.util.List<LayoutToken>	tokenizeWithLayoutToken(java.lang.String text)
java.util.List<LayoutToken>	tokenizeWithLayoutToken(java.lang.String text, Language language) Tokenize text returning list of LayoutTokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

delimiters

public static final java.lang.String delimiters

Method Detail

getInstance

public static GrobidDefaultAnalyzer getInstance()

getName

public java.lang.String getName()

Specified by:

getName in interface Analyzer

tokenize

public java.util.List<java.lang.String> tokenize(java.lang.String text)

Specified by:

tokenize in interface Analyzer

tokenize

public java.util.List<java.lang.String> tokenize(java.lang.String text,
                                                 Language lang)

Specified by:

tokenize in interface Analyzer

retokenize

public java.util.List<java.lang.String> retokenize(java.util.List<java.lang.String> chunks)

Specified by:

retokenize in interface Analyzer

tokenizeWithLayoutToken

public java.util.List<LayoutToken> tokenizeWithLayoutToken(java.lang.String text)

Specified by:

tokenizeWithLayoutToken in interface Analyzer

tokenizeWithLayoutToken

public java.util.List<LayoutToken> tokenizeWithLayoutToken(java.lang.String text,
                                                           Language language)

Tokenize text returning list of LayoutTokens.