Package studio.magemonkey.codex.util.eval.javaluator

Class Tokenizer

java.lang.Object

studio.magemonkey.codex.util.eval.javaluator.Tokenizer

public class Tokenizer
extends Object

A String tokenizer that accepts delimiters that are greater than one character.

Author:

Jean-Marc Astesana

See Also:

• License information

Field Summary

Fields

Modifier and Type	Field	Description
protected final Map<String,Constant>	constants
protected final Map<String,BracketPair>	expressionBrackets
protected final String	functionArgumentSeparator
protected final Map<String,BracketPair>	functionBrackets
protected final Map<String,Function>	functions
protected final Map<String,List<Operator>>	operators

Constructor Summary

Constructors

Constructor	Description
Tokenizer(List<String> delimiters, Map<String,Function> functions, Map<String,List<Operator>> operators, Map<String,Constant> constants, String functionArgumentSeparator, Map<String,BracketPair> functionBrackets, Map<String,BracketPair> expressionBrackets)	Constructor.

Method Summary

Modifier and Type	Method	Description
protected static Pattern	delimitersToRegexp(List<String> delimiters)
protected BracketPair	getBracketPair(String token)
protected Operator	guessOperator(Token previous, List<Operator> candidates)	When a token can be more than one operator (homonym operators), this method guesses the right operator.
boolean	isTrimTokens()	Tests whether this tokens trims the tokens returned by tokenize(String) method.
void	setTrimTokens(boolean trimTokens)	Sets the trimTokens attribute.
Collection<Token>	tokenize(String string)	Converts a string into tokens.
Token	toToken(Token previous, String strToken)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

functions

protected final Map<String,Function> functions

operators

protected final Map<String,List<Operator>> operators

constants

protected final Map<String,Constant> constants

functionArgumentSeparator

protected final String functionArgumentSeparator

functionBrackets

protected final Map<String,BracketPair> functionBrackets

expressionBrackets

protected final Map<String,BracketPair> expressionBrackets

Constructor Details

Tokenizer

public Tokenizer(List<String> delimiters,
 Map<String,Function> functions,
 Map<String,List<Operator>> operators,
 Map<String,Constant> constants,
 String functionArgumentSeparator,
 Map<String,BracketPair> functionBrackets,
 Map<String,BracketPair> expressionBrackets)

Constructor.
By default, this tokenizer trims all the tokens.

Parameters:

delimiters - the delimiters of the tokenizer, usually, the operators symbols, the brackets and the function argument separator are used as delimiter in the string.

functions -

operators -

constants -

functionArgumentSeparator -

functionBrackets -

expressionBrackets -

Method Details

isTrimTokens

public boolean isTrimTokens()

Tests whether this tokens trims the tokens returned by tokenize(String) method.

Returns:

true if tokens are trimmed.

setTrimTokens

public void setTrimTokens(boolean trimTokens)

Sets the trimTokens attribute.

Parameters:

trimTokens - true to have the tokens returned by tokenize(String) method trimmed.
Note that empty tokens are always omitted by this class.

delimitersToRegexp

protected static Pattern delimitersToRegexp(List<String> delimiters)

tokenize

public Collection<Token> tokenize(String string)

Converts a string into tokens.
Example: The result for the expression "-1+min(10,3)" evaluated for a DoubleEvaluator is an iterator on "-", "1", "+", "min", "(", "10", ",", "3", ")".

Parameters:

string - The string to be split into tokens

Returns:

The tokens

toToken

public Token toToken(Token previous,
 String strToken)

getBracketPair

protected BracketPair getBracketPair(String token)

guessOperator

protected Operator guessOperator(Token previous,
 List<Operator> candidates)

When a token can be more than one operator (homonym operators), this method guesses the right operator.
A very common case is the - sign in arithmetic computation which can be an unary or a binary operator, depending on what was the previous token.
Warning: maybe the arguments of this function are not enough to deal with all the cases. So, this part of the evaluation is in alpha state (method may change in the future).

Parameters:

previous - The last parsed tokens (the previous token in the infix expression we are evaluating).

candidates - The candidate tokens.

Returns:

A token