Chapter 14. Pluggable Storage Engine Architecture

Table of Contents

14.1. Introduction to Pluggable Storage Engine Support
14.2. Overview of MySQL Storage Engine Architecture
14.3. The Common MySQL Database Server Layer
14.4. Choosing a Storage Engine
14.5. Assigning Storage Engines to Tables
14.6. Storage Engines and Transactions
14.7. Plugging in a Storage Engine
14.8. Unplugging a Storage Engine
14.9. Security Implications of Pluggable Storage

14.1. Introduction to Pluggable Storage Engine Support

With MySQL 5.1, MySQL AB has introduced a new pluggable storage engine architecture that allows storage engines to be loaded into and unloaded from a running MySQL server.

This chapter describes the pluggable storage engine architecture and provides an overview of the various storage engines provided with MySQL along with instructions on how to assign storage engines to tables and plug and unplug storage engines.

14.2. Overview of MySQL Storage Engine Architecture

The MySQL pluggable storage engine architecture allows a database professional to select a specialized storage engine for a particular application need while being completely shielded from the need to manage any specific application coding requirements. The MySQL server architecture isolates the application programmer and DBA from all of the low-level implementation details at the storage level, providing a consistent and easy application model and API. Thus, although there are different capabilities across different storage engines, the application is shielded from these differences.

The MySQL pluggable storage engine architecture has the structure shown in the following figure:

Figure 14.1. The MySQL pluggable storage engine architecture

The MySQL pluggable storage engine
          architecture

The pluggable storage engine architecture provides a standard set of management and support services that are common among all underlying storage engines. The storage engines themselves are the components of the database server that actually perform actions on the underlying data that is maintained at the physical server level.

This efficient and modular architecture provides huge benefits for those wishing to specifically target a particular application need — such as data warehousing, transaction processing, or high availability situations — while enjoying the advantage of utilizing a set of interfaces and services that are independent of any one storage engine.

The application programmer and DBA interact with the MySQL database through Connector APIs and service layers that are above the storage engines. If application changes bring about requirements that demand the underlying storage engine change, or that one or more additional storage engines be added to support new needs, no significant coding or process changes are required to make things work. The MySQL server architecture shields the application from the underlying complexity of the storage engine by presenting a consistent and easy-to-use API that applies across storage engines.

14.3. The Common MySQL Database Server Layer

A MySQL pluggable storage engine is the component in the MySQL database server that is responsible for performing the actual data I/O operations for a database as well as enabling and enforcing certain feature sets that target a specific application need. A major benefit of using specific storage engines is that you are only delivered the features needed for a particular application, and therefore you have less system overhead in the database, with the end result being more efficient and higher database performance. This is one of the reasons that MySQL has always been known to have such high performance, matching or beating proprietary monolithic databases in industry standard benchmarks.

From a technical perspective, what are some of the unique supporting infrastructure components that are in a storage engine? Some of the key feature differentiations include:

  • Concurrency — some applications have more granular lock requirements (such as row-level locks) than others. Choosing the right locking strategy can reduce overhead and therefore improve overall performance. This area also includes support for capabilities such as multi-version concurrency control or “snapshot” read.

  • Transaction Support — Not every application needs transactions, but for those that do, there are very well defined requirements such as ACID compliance and more.

  • Referential Integrity — The need to have the server enforce relational database referential integrity through DDL defined foreign keys.

  • Physical Storage — This involves everything from the overall page size for tables and indexes as well as the format used for storing data to physical disk.

  • Index Support — Different application scenarios tend to benefit from different index strategies. Each storage engine generally has its own indexing methods, although some (such as B-tree indexes) are common to nearly all engines.

  • Memory Caches — Different applications respond better to some memory caching strategies than others, so although some memory caches are common to all storage engines (such as those used for user connections or MySQL's high-speed Query Cache), others are uniquely defined only when a particular storage engine is put in play.

  • Performance Aids — This includes multiple I/O threads for parallel operations, thread concurrency, database checkpointing, bulk insert handling, and more.

  • Miscellaneous Target Features — This may include support for geospatial operations, security restrictions for certain data manipulation operations, and other similar features.

Each set of the pluggable storage engine infrastructure components are designed to offer a selective set of benefits for a particular application. Conversely, avoiding a set of component features helps reduce unnecessary overhead. It stands to reason that understanding a particular application's set of requirements and selecting the proper MySQL storage engine can have a dramatic impact on overall system efficiency and performance.

14.4. Choosing a Storage Engine

The various storage engines provided with MySQL are designed with different use-cases in mind. To use the pluggable storage architecture effectively, it is good to have an idea of the benefits and drawbacks of the various storage engines. The following table provides an overview of some storage engines provided with MySQL:

FeatureMyISAMBDBMemoryInnoDBArchiveNDB
Storage limits256TBNoYes64TBNo384EB[4]
TransactionsNoYesNoYesNoYes
Locking granularityTablePageTableRowRowRow
MVCC (snapshot read)NoNoNoYesYesNo
Geospatial supportYesYes[1]NoYes[1]Yes[1]Yes[1]
B-tree indexesYesYesYesYesNoYes
Hash indexesNoNoYesNoNoYes
Full-text search indexesYesNoNoNoNoNo
Clustered indexesNoYesNoYesNoNo
Data cachesNoYesN/AYesNoYes
Index cachesYesYesN/AYesNoYes
Compressed dataYesNoNoNoYesNo
Encrypted data[2]YesYesYesYesYesYes
Cluster database supportNoNoNoNoNoYes
Replication support[3]YesYesYesYesYesYes
Foreign key supportNoNoNoYesNoNo
Backup / point-in-time recovery[3]YesYesYesYesYesYes
Query cache supportYesYesYesYesYesYes
Update statistics for data dictionaryYesYesYesYesYesYes
  • [1] Storage engine supports spatial data types but no indexing of such data

  • [2] Implemented in the server (via encryption functions), rather than in the storage engine

  • [3] Implemented in the server, rather than in the storage engine

  • [4] EB = exabyte (1024 * 1024 terabyte)

The following storage engines are the most commonly used:

  • MyISAM — The default MySQL pluggable storage engine and the one that is used the most in Web, data warehousing, and other application environments. Note that a MySQL server's default storage engine can easily be changed by setting the storage_engine configuration variable.

  • InnoDB — Used for transaction processing applications, and sports a number of features including ACID transaction support and foreign keys.

  • BDB — An alternative to InnoDB that supports COMMIT, ROLLBACK, and other transactional features.

  • Memory — Stores all data in RAM for extremely fast access in environments that require quick lookups of reference and other like data.

  • Merge — Allows a MySQL DBA or developer to logically group a series of identical MyISAM tables and reference them as one object. Good for VLDB environments such as data warehousing.

  • Archive — Provides the perfect solution for storing and retrieving large amounts of seldom-referenced historical, archived, or security audit information.

  • Federated — Offers the ability to link separate MySQL servers to create one logical database from many physical servers. Very good for distributed or data mart environments.

  • NDB — The Clustered database engine that is particularly suited for applications with high performance lookup needs that also require the highest possible degree of uptime and availability.

  • Other — Other storage engines include CSV (references comma-separated files as database tables), Blackhole (for temporarily disabling application input to the database), and an Example engine that helps jump start the process of creating custom pluggable storage engines.

It is important to remember that you are not restricted to using the same storage engine for an entire server or schema: you can use a different storage engine for each table in your schema.

For detailed information on the storage engines included with MySQL, see Chapter 15, Storage Engines and Table Types.

14.5. Assigning Storage Engines to Tables

Storage engines can be designated either when creating new tables or through the use of an ALTER TABLE statement.

To specify a storage engine when creating a table, use the ENGINE table option:

CREATE TABLE engineTest (id INT) ENGINE = MyISAM;

To change the storage engine of an existing table, use the ALTER TABLE statement:

ALTER TABLE engineTest ENGINE = ARCHIVE;

14.6. Storage Engines and Transactions

The following storage engines support transactions:

  • InnoDB — Supports transactions through MVCC, allows for COMMIT, ROLLBACK, and savepoints.

  • NDB — Supports transactions, allows for COMMIT and ROLLBACK.

  • BDB — Supports transactions, allows for COMMIT and ROLLBACK.

14.7. Plugging in a Storage Engine

Before a storage engine can be used, the storage engine plugin shared library must be loaded into MySQL using the INSTALL PLUGIN statement. For example, if the EXAMPLE engine plugin is named ha_example and the shared library is named ha_example.so, you load it with the following statement:

INSTALL PLUGIN ha_example SONAME 'ha_example.so';

The shared library must be located in the MySQL server plugin directory, the location of which is given by the plugin_dir system variable.

14.8. Unplugging a Storage Engine

To unplug a storage engine, use the UNINSTALL PLUGIN statement:

UNINSTALL PLUGIN ha_example;

If you unplug a storage engine that is needed by existing tables, those tables become accessible, but will still be present on disk (where applicable). Ensure that there are no tables using a storage engine before you unplug the storage engine.

14.9. Security Implications of Pluggable Storage

To install a pluggable storage engine, the plugin file must be located in the MySQL plugin directory, and the user issuing the INSTALL PLUGIN statement must have INSERT privileges for the mysql.plugin table.