7.8.14. TokenNgram — Groonga v16.0.2 documentation

Skip to main content
`Ctrl`+`K`

# 7.8.14. `TokenNgram`

## 7.8.14.1. Summary

`TokenNgram` can change define its behavior dynamically via its options. For example, we can use it as unigram, bigram, trigram on with changing `n` option value as below.

Uni-gram:

Execution example:

    tokenize --tokenizer 'TokenNgram("n", 1)' --string "Hello World"
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "H",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "e",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "l",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "l",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": " ",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "W",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "r",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "l",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "d",
    #       "position": 10,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Bi-gram:

Execution example:

    tokenize --tokenizer 'TokenNgram("n", 2)' --string "Hello World"
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "He",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "el",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ll",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "lo",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o ",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": " W",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "Wo",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "or",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "rl",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ld",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "d",
    #       "position": 10,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

Tri-gram:

Execution example:

    tokenize --tokenizer 'TokenNgram("n", 3)' --string "Hello World"
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "Hel",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ell",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "llo",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "lo ",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o W",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": " Wo",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "Wor",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "orl",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "rld",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ld",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "d",
    #       "position": 10,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

This tokenizer also has options other than the above.

## 7.8.14.2. Syntax

`TokenNgram` has optional parameter.

No options:

    TokenNgram

If we don’t use options, `TokenNgram` is the same behavior for `TokenBigram`.

Specify option:

    TokenNgram("n", true)

    TokenNgram("loose_symbol", true)

    TokenNgram("loose_blank", true)

    TokenNgram("remove_blank", true)

    TokenNgram("report_source_location", true)

    TokenNgram("unify_alphabet", true)

    TokenNgram("unify_symbol", true)

    TokenNgram("unify_digit", true)

Specify multiple options:

    TokenNgram("loose_symbol", true, "loose_blank", true)

We can also specify multiple options at the same time except the above examples.

## 7.8.14.3. Usage

## 7.8.14.4. Simple usage

Here is an example of `TokenNgram`.

If we use `TokenNgram` in the nothing of option. `TokenNgram` behavior is the same as `TokenBigram` as below:

- If no normalizer is used.

Execution example:

    tokenize --tokenizer TokenNgram --string "Hello World"
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "He",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "el",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ll",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "lo",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o ",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": " W",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "Wo",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "or",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "rl",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ld",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "d",
    #       "position": 10,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

- If normalizer is used.

Execution example:

    tokenize --tokenizer TokenNgram --string "Hello World" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "hello",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "world",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

## 7.8.14.5. Advanced usage

We can specify multiple options for `TokenNgram`.

For example, we can deal with variantsas as phone number by using `loose_symbol` and `loose_blank` as below.

We can search `0123(45)6789`, `0123-45-6789` and, `0123 45 6789` by using `0123456789` as below example.

Execution example:

    tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123(45)6789" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "0123",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "(",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "45",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": ")",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "6789",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "￰",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "0123456789",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]
    tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123-45-6789" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "0123",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "-",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "45",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "-",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "6789",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "￰",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "0123456789",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]
    tokenize --tokenizer 'TokenNgram("loose_symbol", true, "loose_blank", true)' --string "0123 45 6789" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "0123",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "45",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "6789",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "￰",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "0123456789",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

## 7.8.14.6. Parameters

### 7.8.14.6.1. Optional parameter

There are eight optional parameters.

#### 7.8.14.6.1.1. `n`

This option shows `N` of Ngram. For example, `n` is `3` for trigram.

#### 7.8.14.6.1.2. `ignore_blank`

It ignores white-spaces in continuous symbols and non-ASCII characters. For example, the keyword `日 本 語 ! ! !` is tokenized as `日本語!!!`.

#### 7.8.14.6.1.3. `loose_symbol`

Tokenize keywords including symbols, to be searched by both queries with/without symbols. For example, a keyword `090-1111-2222` will be found by any of `09011112222`, `090`, `1111`, `2222` and `090-1111-2222`.

#### 7.8.14.6.1.4. `loose_blank`

Tokenize keywords including blanks, to be searched by both queries with/without blanks. For example, a keyword `090 1111 2222` will be found by any of `09011112222`, `090`, `1111`, `2222` and `090 1111 2222`.

#### 7.8.14.6.1.5. `remove_blank`

Tokenize keywords including blanks, to be searched by queries without blanks. For example, a keyword `090 1111 2222` will be found by any of `09011112222`, `090`, `1111` or `2222`.

Note that the keyword won’t be found by a query including blanks like `090 1111 2222`.

#### 7.8.14.6.1.6. `report_source_location`

This option tells us a location of token of highlight target when we highlight token by `highlight_html`.

We only use this option when we want to highlight token by `highlight_html`.

When Groonga tokenize texts that target highlight, always used `NormalizerAuto` to normalizer until now. Therefore, if we use `NormalizerNFKC100` to normalizer, sometimes it can’t find the position of the highlight as below.

Execution example:

    table_create Entries TABLE_NO_KEY
    # [[0,1337566253.89858,0.000355720520019531],true]
    column_create Entries body COLUMN_SCALAR ShortText
    # [[0,1337566253.89858,0.000355720520019531],true]
    table_create Terms TABLE_PAT_KEY ShortText --normalizer 'NormalizerNFKC100'
    # [[0,1337566253.89858,0.000355720520019531],true]
    column_create Terms document_index COLUMN_INDEX|WITH_POSITION Entries body
    # [[0,1337566253.89858,0.000355720520019531],true]
    load --table Entries
    [
    {"body": "ア㌕Ａz"}
    ]
    # [[0,1337566253.89858,0.000355720520019531],1]
    select Entries \
      --match_columns body \
      --query 'グラム' \
      --output_columns 'highlight_html(body, Terms)'
    # [[0,1337566253.89858,0.000355720520019531],[[[0],[["highlight_html",null]]]]]

Because we use different normalizer to normalize token.

This option is used to reduce the shift of the position of the highlight as below.

Execution example:

    table_remove Terms
    # [[0,1337566253.89858,0.000355720520019531],true]
    table_create Terms TABLE_PAT_KEY ShortText \
      --default_tokenizer 'TokenNgram("report_source_location", true)' \
      --normalizer 'NormalizerNFKC100'
    # [[0,1337566253.89858,0.000355720520019531],true]
    column_create Terms document_index COLUMN_INDEX|WITH_POSITION Entries body
    # [[0,1337566253.89858,0.000355720520019531],true]
    select Entries \
      --match_columns body \
      --query 'グラム' \
      --output_columns 'highlight_html(body, Terms)'
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     [
    #       [
    #         1
    #       ],
    #       [
    #         [
    #           "highlight_html",
    #           null
    #         ]
    #       ],
    #       [
    #         "ア<span class=\"keyword\">㌕</span>Ａz"
    #       ]
    #     ]
    #   ]
    # ]

#### 7.8.14.6.1.7. `unify_alphabet`

If we set false, `TokenNgram` uses bigram tokenize method for ASCII character.

Default value of this option is true.

Execution example:

    tokenize --tokenizer 'TokenNgram("unify_alphabet", false)' --string "Hello World" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "he",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "el",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ll",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "lo",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "o",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "wo",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "or",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "rl",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "ld",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "d",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

#### 7.8.14.6.1.8. `unify_symbol`

If we set false, `TokenNgram` uses bigram tokenize method for symbols. `TokenNgram("unify_symbol", false)` is same behavior of `TokenBigramSplitSymbol`.

Default value of this option is true.

Execution example:

    tokenize --tokenizer 'TokenNgram("unify_symbol", false)' --string "___---" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "__",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "__",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "_-",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "--",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "--",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "-",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

#### 7.8.14.6.1.9. `unify_digit`

If we set false, `TokenNgram` uses bigram tokenize method for digits.

Default value of this option is true.

Execution example:

    tokenize --tokenizer 'TokenNgram("unify_digit", false)' --string "012345 6789" --normalizer NormalizerAuto
    # [
    #   [
    #     0,
    #     1337566253.89858,
    #     0.000355720520019531
    #   ],
    #   [
    #     {
    #       "value": "01",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "12",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "23",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "34",
    #       "position": 3,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "45",
    #       "position": 4,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "5",
    #       "position": 5,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "67",
    #       "position": 6,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "78",
    #       "position": 7,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "89",
    #       "position": 8,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     },
    #     {
    #       "value": "9",
    #       "position": 9,
    #       "force_prefix": false,
    #       "force_prefix_search": false
    #     }
    #   ]
    # ]

## 7.8.14.7. See also

- tokenize