12.9.9 MeCab Full-Text Parser Plugin
The built-in MySQL full-text parser uses the white space between words as a delimiter to determine where words begin and end, which is a limitation when working with ideographic languages that do not use word delimiters. To address this limitation for Japanese, MySQL provides a MeCab full-text parser plugin. The MeCab full-text parser plugin is supported for use with
MySQL also provides an ngram full-text parser plugin that supports Japanese. For more information, see Section 12.9.8, “ngram Full-Text Parser”.
The MeCab full-text parser plugin is a full-text parser plugin for Japanese that tokenizes a sequence of text into meaningful words. For example, MeCab tokenizes “データベース管理” (“Database Management”) into “データベース” (“Database”) and “管理” (“Management”). By comparison, the ngram full-text parser tokenizes text into a contiguous sequence of
n characters, where
n represents a number between 1 and 10.
In addition to tokenizing text into meaningful words, MeCab indexes are typically smaller than ngram indexes, and MeCab full-text searches are generally faster. One drawback is that it may take longer for the MeCab full-text parser to tokenize documents, compared to the ngram full-text parser.
The full-text search syntax described in Section 12.9, “Full-Text Search Functions” applies to the MeCab parser plugin. Differences in parsing behavior are described in this section. Full-text related configuration options are also applicable.
For additional information about the MeCab parser, refer to the MeCab: Yet Another Part-of-Speech and Morphological Analyzer project on Github.
The MeCab parser plugin requires
On supported Fedora, Debian and Ubuntu platforms (except Ubuntu 12.04 where the system
mecab version is too old), MySQL dynamically links to the system
mecab installation if it is installed to the default location. On other supported Unix-like platforms,
libmecab.so is statically linked in
libpluginmecab.so, which is located in the MySQL plugin directory.
mecab-ipadic is included in MySQL binaries and is located in
You can install
mecab-ipadic using a native package management utility (on Fedora, Debian, and Ubuntu), or you can build
mecab-ipadic from source. For information about installing
mecab-ipadic using a native package management utility, see Installing MeCab From a Binary Distribution (Optional). If you want to build
mecab-ipadic from source, see Building MeCab From Source (Optional).
libmecab.dll is found in the MySQL
mecab-ipadic is located in
To install and configure the MeCab parser plugin, perform the following steps:
In the MySQL configuration file, set the
mecab_rc_fileconfiguration option to the location of the
mecabrcconfiguration file, which is the configuration file for MeCab. If you are using the MeCab package distributed with MySQL, the
mecabrcfile is located in
looseprefix is an option modifier. The
mecab_rc_fileoption is not recognized by MySQL until the MeCaB parser plugin is installed but it must be set before attempting to install the MeCaB parser plugin. The
looseprefix allows you restart MySQL without encountering an error due to an unrecognized variable.
If you use your own MeCab installation, or build MeCab from source, the location of the
mecabrcconfiguration file may differ.
For information about the MySQL configuration file and its location, see Section 18.104.22.168, “Using Option Files”.
Also in the MySQL configuration file, set the minimum token size to 1 or 2, which are the values recommended for use with the MeCab parser. For
InnoDBtables, minimum token size is defined by the
innodb_ft_min_token_sizeconfiguration option, which has a default value of 3. For
MyISAMtables, minimum token size is defined by
ft_min_word_len, which has a default value of 4.
mecabrcconfiguration file to specify the dictionary you want to use. The
mecab-ipadicpackage distributed with MySQL binaries includes three dictionaries (
mecabrcconfiguration file packaged with MySQL contains and entry similar to the following:
dicdir = /path/to/mysql/lib/mecab/lib/mecab/dic/ipadic_euc-jp
To use the
ipadic_utf-8dictionary, for example, modify the entry as follows:
If you are using your own MeCab installation or have built MeCab from source, the default
dicdirentry in the
mecabrcfile will differ, as will the dictionaries and their location.Note
After the MeCab parser plugin is installed, you can use the
mecab_charsetstatus variable to view the character set used with MeCab. The three MeCab dictionaries provided with the MySQL binary support the following character sets.
ipadic_euc-jpdictionary supports the
ipadic_sjisdictionary supports the
ipadic_utf-8dictionary supports the
mecab_charsetonly reports the first supported character set. For example, the
ipadic_utf-8dictionary supports both
utf8when this dictionary is in use.
Install the MeCab parser plugin:
The MeCab parser plugin is installed using
INSTALL PLUGINsyntax. The plugin name is
mecab, and the shared library name is
libpluginmecab.so. For additional information about installing plugins, see Section 5.5.1, “Installing and Uninstalling Plugins”.
INSTALL PLUGIN mecab SONAME 'libpluginmecab.so';
Once installed, the MeCab parser plugin loads at every normal MySQL restart.
Verify that the MeCab parser plugin is loaded using the
mysql> SHOW PLUGINS;
mecabplugin should appear in the list of plugins.
This example demonstrates creating a table with a
FULLTEXT index, inserting sample data, and viewing tokenized data in the
mysql> USE test; mysql> CREATE TABLE articles ( id INT UNSIGNED AUTO_INCREMENT NOT NULL PRIMARY KEY, title VARCHAR(200), body TEXT, FULLTEXT (title,body) WITH PARSER mecab ) ENGINE=InnoDB CHARACTER SET utf8; mysql> SET NAMES utf8; mysql> INSERT INTO articles (title,body) VALUES ('データベース管理','このチュートリアルでは、私はどのようにデータベースを管理する方法を紹介します'), ('データベースアプリケーション開発','データベースアプリケーションを開発することを学ぶ'); mysql> SET GLOBAL innodb_ft_aux_table="test/articles"; mysql> SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_INDEX_CACHE ORDER BY doc_id, position;
CREATE TABLE articles ( id INT UNSIGNED AUTO_INCREMENT NOT NULL PRIMARY KEY, title VARCHAR(200), body TEXT ) ENGINE=InnoDB CHARACTER SET utf8; ALTER TABLE articles ADD FULLTEXT INDEX ft_index (title,body) WITH PARSER mecab; # Or: CREATE FULLTEXT INDEX ft_index ON articles (title,body) WITH PARSER mecab;
The MeCab parser uses spaces as separators in query strings. For example, the MeCab parser tokenizes データベース管理 as データベース and 管理.
By default, the MeCab parser uses the default stopword list, which contains a short list of English stopwords. For a stopword list applicable to Japanese, you must create your own. For information about creating stopword lists, see Section 12.9.4, “Full-Text Stopwords”.
For natural language mode search, the search term is converted to a union of tokens. For example, データベース管理 is converted to データベース 管理.
SELECT COUNT(*) FROM articles WHERE MATCH(title,body) AGAINST('データベース管理' IN NATURAL LANGUAGE MODE);
For boolean mode search, the search term is converted to a search phrase. For example, データベース管理 is converted to データベース 管理.
SELECT COUNT(*) FROM articles WHERE MATCH(title,body) AGAINST('データベース管理' IN BOOLEAN MODE);
Wildcard search terms are not tokenized. A search on データベース管理* is performed on the prefix, データベース管理.
SELECT COUNT(*) FROM articles WHERE MATCH(title,body) AGAINST('データベース*' IN BOOLEAN MODE);
Phrases are tokenized. For example, データベース管理 is tokenized as データベース 管理.
SELECT COUNT(*) FROM articles WHERE MATCH(title,body) AGAINST('"データベース管理"' IN BOOLEAN MODE);
This section describes how to install
mecab-ipadic from a binary distribution using a native package management utility. For example, on Fedora, you can use Yum to perform the installation:
On Debian or Ubuntu, you can perform an APT installation:
apt-get install mecab apt-get install mecab-ipadic
If you want to build
mecab-ipadic from source, basic installation steps are provided below. For additional information, refer to the MeCab documentation.
Download the tar.gz packages for
mecab-ipadicfrom http://taku910.github.io/mecab/#download . As of February, 2016, the latest available packages are
tar zxfv mecab-0.996.tar cd mecab-0.996 ./configure make make check su make install
tar zxfv mecab-ipadic-2.7.0-20070801.tar cd mecab-ipadic-2.7.0-20070801 ./configure make su make install
If you defined a custom installation directory, set
WITH_MECABto the custom directory. For example: