Scalable BLOB Streaming Infrastructure for MySQL

The Scalable BLOB Streaming infrastructure for MySQL will transform MySQL into a scalable media server capable of streaming pictures, films, MP3 files and other binary and text objects (BLOBs) directly in and out of the database. On this site you will find all information relating to the ongoing activities of this project. The development is led by PrimeBase Technologies, www.primebase.org, an open source software company. PrimeBase Technologies, together with the support of MySQL AB, is also the main sponsor of the implementation of this technology.

Goals of the Project

These are the goals of the Scalable BLOB Streaming Project:

• Store BLOBs of any size in the database. Currently MySQL is limited to storing BLOBs that it can hold in memory.
  
• Stream BLOB data directly in and out of the database.
  
• Create a scalable back-end that can handle any throughput requirements.
  
• Provide an open system that can be used by all MySQL storage engines.
  
• Provide a standardized client API for applications, that extends the existing MySQL client/server API.
  
• Implement the system with minimal changes to the MySQL code base.
  
• A system designed to meet a broad range of customer needs.
  

Advantages of BLOBs in the database

There are a number of advantages of storing BLOBs in the database.

• Operations can be done in atomic transactions. This avoids "dangling" references that can occur when the BLOBs are stored in the file system.
  
• BLOBs subject to the normal database Backup.
  
• BLOB data can be replicated.
  
• BLOBs in the database will scale better. Most file systems perform badly when the number of files exceeds 2 million.
  

BLOB Streaming with MySQL

These are the most important characteristics of the design of the scalable BLOB Streaming infrastructure.

• A client-side extension to the MySQL client API. The API includes functions to create, read and write BLOBs.
  
• A BLOB Streaming storage engine will provide a Server-side API for engines that want to support BLOB streaming.
  
• BLOBs will be transported using a specialized stream-based protocol which connects directly to the BLOB streaming back-end.
  
• The stream based communication will run parallel to the standard MySQL client/server protocol.
  
• Scale-out will be achieved using a stand-alone Streaming Server, which registers itself with the BLOB Streaming engine.
  

BLOB Streaming Architecture

The following diagram illustrates the most important components of the Scalable BLOB Streaming infrastructure.