Java StringTokenizer class

StringTokenizer is a legacy class of java. util package. It is used to parse or separate out different parts of a string based on a delimiter. These parts are tokens, and this process is known as tokenization. For example, if we tokenize the program statement int salary = basic + da; using space as the delimiter file, tokens int, salary, basic, +, da; will be generated.

To tokenize a string, we need to input a string and a delimiter. The delimiter is a single character. However, if you want more than one delimiter, we can keep all of them within a string and pass that string as a parameter for the delimiter. For example, if we use " ;" as a delimiter string, the tokenization of the above string would generate the token da instead of da;. The following example code demonstrates this.

			

import java.util.StringTokenizer;
public class Token {
    public static void main(String args[])
    {
        String line = "int salary = basic + da;";
StringTokenizerst = new StringTokenizer(line, " ;"); 
        // space and semi-colon are delimiter
        while(st.hasMoreTokens()) {
System.out.println(st.nextToken());
        }
    }
}

				

Output:

int salary = basic + Da

Default delimiter

The default set of delimiters consists of the whitespace characters - space, tab, newline, and carriage return.

Constructors of StringTokenizer Class

Constructor Type Description
StringTokenizer(String str)

str is the string to be tokenized. The default delimiters are used to tokenize. Tokens are returned.

StringTokenizer(String str, String delimiters) str is the string to be tokenized. delimiters string is used to tokenize. Tokens are returned.
StringTokenizer(String str, String delimiters, booleandelimAsToken) str is the string to be tokenized. delimiters string is used to tokenize. If delimAsToken is true, delimiters are also returned along with tokens. If false, delimiters are not returned.

The following code example demonstrates the use of 3 different constructors:

			

import java.util.StringTokenizer;

public class Token {
    public static void main(String args[])
    {
        //  Example 1 to demonstrate use of default delimiter – whitespaces
        //  Returns 4 Tokens
        String line1 = "Testing of StringTokenizer Class";
StringTokenizerst = new StringTokenizer(line1);
        int count=1;
        while(st.hasMoreTokens()) {
System.out.println("Token"+count+++". "+st.nextToken());
        }

        //    Example 2 to demonstrate multiple delimiters
        //    Returns tokens delimited by comma and tab characters [3 Tokens]
        count=1;
        String line2 = "StringTokenizerClass,This class has 3 constructors\tThis class has 6 methods";
StringTokenizer st1 = new StringTokenizer(line2,",\t");
while(st1.hasMoreTokens()) {
System.out.println("Token"+count+++". "+st1.nextToken());
        }

        // Example 3:demonstrate multiple delimiter with delimAsToken flag = true
        // Returns 3 Tokens and 2 delimiters also as tokens [Total 5 Tokens]
        count=1;
        String line3 = "Training Material|Computer Programming-Java & Python";
        st1 = new StringTokenizer(line3,"|-",true);
while(st1.hasMoreTokens()) {
System.out.println("Token"+count+++". "+st1.nextToken());
        }
    }
}

				

Output:

Output of Example 1 Token1. Testing Token2. of Token3. StringTokenizer Token4. Class Output of Example 2 Token1. StringTokenizer Class Token2. This class has 3 constructors. Token3. This class has 6 methods Output of Example 3 Token1. Training Material Token2. | Token3. Computer Programming Token4. - Token5. Java & Python

StringTokenizerClass Methods

StringTokenizer class has six methods.

Method Description
booleanhasMoreTokens() It tests whether any more token is available in the string. Returns true if found, false otherwise.
String nextToken() It returns the next token from the StringTokenizer object. If no token is available, NoSuchElementException is raised. To avoid getting the exception, this method is called after testing has more tokens.
int countTokens() It returns the total number of tokens present from the current position. Returns 0 when there is none.
String nextToken(String delim) It returns the next token based on the delimiter.
booleanhasMoreElements() It is the same as hasMoreTokens() method.
Object nextElement() It is the same as nextToken() but its return type is Object.If no token is available, NoSuchElementException is raised.

The following example demonstrates the use of some of the methods.

			

import java.util.StringTokenizer;
public class Token {

    public static void main(String args[]) {
        String str = "He is my friend Sanjib";
StringTokenizerst = new StringTokenizer(str);
        while (st.countTokens()>=1)
        {
            if (st.hasMoreTokens())
System.out.println("Token Count = "+st.countTokens()+"  Token = "+st.nextToken());
        }
    }
}

				

Output:

Token Count = 5 Token = He Token Count = 4 Token = is Token Count = 3 Token = my Token Count = 2 Token = friend Token Count = 1 Token = Sanjib