BLOB Streaming Release Notes ============================ ------- 0.5.04 Alpha - 2007-10-30 RN34: The JDBC parameters names are case-sensitive! In TestJDBC using 'EnableBlobStreaming=true' was ignored. 'enableBlobStreaming=true' must be used. RN33: The format of the URL genearated by MyBS has been changed. The format of the BLOB URLs is now as follows: '~*' '/' '-' '-' '-' Where is '_' or '~'. Examples: ~*test/_11-128-fbd590b-0, ~*test/~1-524-3dc45b09-0 In other words, the characters '>' has been replace by '*', '^' has been replace by '_' and ':' has been replace by '~'. The reason for this is that the characters '>' and '^' are not allowed in URLs, and must be URL-encoded. The character ':' is reserved, but allowed. NOTE: This change makes this version incompatible with previous versions of MyBS. If you have a table with BLOB URLs, you can upgrade the URLs as follows: UPDATE blob_table SET blob_col = REPLACE(REPLACE(blob_col, '~>', '~*'), '/:', '/~'); Replacing '^' is not necessary because BLOB URLs with '^' should not appear in tables. ------- 0.5.03 Alpha - 2007-10-17 RN32: BLOB field references are now disable by default. This is for security reasons. The system variable 'mybs_field_references' now deterimes if the BLOB field reference are permitted or not. To enable BLOB field reference use the command line option --mybs-field-references. It is also possible to set the variable at runtime as follows: set global mybs_field_references=TRUE; RN31: Included the location of an error in the stack trace that is passed through the BLOB streaming API. RN30: Implemented BLOB streaming for JDBC. The version used is Connector/J 5.0.7. Enable BLOB streaming by adding the following option to the JDBC URL: enableBlobStreaming=true. When set to true the driver will upload values set with setBinaryStream(), setAsciiStream() and setBlob() to the MyBS BLOB repository and insert a URL reference into the table, but only if a valid length is specified. For example, a value inserted using setBinaryStream() with a length of -1 will be not be uploaded to the repository. Instead the value will by inserted directly into the table, as if enableBlobStreaming were set to false. NOTE: When enableBlobStreaming=true you should only use setBinaryStream(), setAsciiStream() and setBlob() with a valid length on columns of type LONGBLOB. Streamable BLOBs are currently only recognised in LONGBLOB type columns. Selected values of type LONGBLOB are checked for valid BLOB URLs. Valid URLs are used to automatically download the BLOB data from the BLOB repository. Using a getBinaryStream(), getAsciiStream() or getBlob() you have direct access to the BLOB data stream from the repository. LIMITATION: This implementation is based on JDK 1.5.x, which only allows for lengths in setBinaryStream(), setAsciiStream() and setBlob() of type 'int'. This means that the maximum BLOB size that can currently permitted is 2GB. RN29: Changed the format of BLOB URLs: '~>' '/' '-' '-' '-' Examples: ~>test/^11-128-fbd590b-0, ~>test/:1-524-3dc45b09-0 The sequence '~>' is a prefix which helps to identify a value BLOB URL quickly. is ':' or '^'. BLOB URLs with type-char '^' are BLOBs that do not have a corresponding table (also called an unreferenced BLOB). In this case, the is a repository file ID, and is the repository file offset. BLOB URLs with type-char ':' are standard BLOB repository references with a corresponding table. On INSERT, an unreferenced BLOB URL is transformed into a standard BLOB repository reference. RN28: When uploading a BLOB it is now only necessary to specify the target database. ------- 0.5.02 Alpha - 2007-09-12 RN27: Threads are now handled in non-global "groups". The only group used is the connection handler group. RN26: Unused sockets where not closed correctly. This caused an "out of file handles" error. RN25: Fixed a bug in the buffer file reading which caused system table output to be incorrect. RN24: Handle the situation that a temporary log file is missing. RN23: Fixed a bug that caused the compactor to go into a busy loop. RN22: Handled incomplete records in the repository files. RN21: Fixed several problems with shuting down the PBXT and MyBS engines cleanly, when MySQL shutdown. For example, each engine requires its own copy of the streaming API code (or the code is inloaded when one of the engines is unloaded). It was also neccessary to ignore signals during the shutdown (finalize) phase of the connection handler threads. RN20: Fixed a bug which returned an open table to the pool twice. RN19: Added thread pooling to handle multiple connections without creating to many threads. ------- 0.5.01 Alpha - 2007-08-17 RN18 (2007-08-22): Null dates must be set to zero! RN17: (2007-08-21): Corrected the MyBS version number. RN16: "my_global.h" included in all sources (at the top) because it redefines off_t! RN15: "mysql_priv.h" and are not compatible under Fedora (ugh!). Removed all references to the C++ header file. RN14: Fixed the insertion and deleting of NULL references. RN13: Corrected compilation errors in BSDatabase.h and BSRepository.cc. ------- 0.5.00 Alpha - 2007-07-25 RN12: Made some corrections in order to compile with MySQL 5.1.20. RN11: Added supprt for MyBS repository system tables: mybs_repository and mybs_reference. The table are created as follows: CREATE TABLE mybs_repository ( Repository_id INT COMMENT 'The reppository file number', Repo_blob_offset BIGINT COMMENT 'The offset of the BLOB in the repository file', Blob_size BIGINT COMMENT 'The size of the BLOB in bytes', Head_size SMALLINT UNSIGNED COMMENT 'The size of the BLOB header - preceeds the BLOB data', Access_code INT COMMENT 'The 4-byte authorisation code required to access the BLOB - part of the BLOB URL', Creation_time TIMESTAMP COMMENT 'The time the BLOB was created', Last_ref_time TIMESTAMP COMMENT 'The last time the BLOB was referenced', Last_access_time TIMESTAMP COMMENT 'The last time the BLOB was accessed (read)', Content_type CHAR(128) COMMENT 'The content type of the BLOB - returned by HTTP GET calls', Blob_data LONGBLOB COMMENT 'The data of this BLOB' ) ENGINE=MyBS; CREATE TABLE mybs_reference ( Table_name CHAR(64) COMMENT 'The name of the referencing table', Blob_id BIGINT COMMENT 'The BLOB reference number - part of the BLOB URL', Column_name CHAR(64) COMMENT 'The column name of the referencing field', Row_condition VARCHAR(255) COMMENT 'This condition identifies the row in the table', Blob_url VARCHAR(200) COMMENT 'The BLOB URL for HTTP GET access', Repository_id INT COMMENT 'The repository file number of the BLOB', Repo_blob_offset BIGINT COMMENT 'The offset in the repository file', Blob_size BIGINT COMMENT 'The size of the BLOB in bytes', Deletion_time TIMESTAMP COMMENT 'The time the BLOB was deleted', Remove_in INT COMMENT 'The number of seconds before the reference/BLOB is removed perminently', Temp_log_id INT COMMENT 'Temporary log number of the referencing deletion entry', Temp_log_offset BIGINT COMMENT 'Temporary log offset of the referencing deletion entry' ) ENGINE=MyBS; These tables must currently be created manually in each database. The mybs_repository contains one entry for each BLOB in the repository. The mybs_reference contains one entry for each reference to a particular BLOB. RN10: The uploaded content type is stored with the BLOB this value is then returned on download. The content types are stored in a text file called 'bs-content-types.txt'. This file may only be edited with MyBS is not running. RN9: Added create, access and reference timestamps to the BLOBs in a repository. RN8: Implemente RENAME TABLE. When a table is renamed the BLOB references are preserved. RN7: Implemented DROP TABLE. When a table containing BLOB is dropped, the MyBS engine deletes all the references from that table, after the temporary BLOB timeout (mybs_temp_blob_timeout) has expired. This is done by creating a '.bsr' file per table, where the all the references of a particular table are stored. RN6: The MyBS compactor thread monitors the level of garbage in the repository files. As soon as the level exceeds the garbage level specified by the 'mybs_garbage_threshold' system variable (a value from 0-100, representing a percentage), the compactor begins to compact the repository file. The compactor thread does this by coping all valid BLOBs to a new repository file, and deleting the old repository file. Note that a repository file my be locked for writting by an open table handle. RN5: BLOBs that have been uploaded to the repository can be reference by inserting the reference into a PBXT table field of type LONGBLOB. For example: use test; CREATE TABLE notes_tab ( n_id INTEGER PRIMARY KEY, n_text LONGBLOB ) ENGINE=pbxt; curl -d "This is a BLOB streaming test." http://localhost:8080/test/notes_tab test/1-128-3dd49f08-0 INSERT notes_tab VALUES (1, "test/1-128-3dd49f08-0"); The BLOB will then be referenced from the table, and will not be deleted when the temporary BLOB timeout expires. NOTE: Required version of PBXT is 0.9.88. RN4: BLOB in the repository will be deleted after a certain amoount of time, if not referenced by a table. The value is set using the system variable 'mybs_temp_blob_timeout' in seconds. The delete information is stored in the "temporary logs", in the bs-logs subdirectory of a database. The maximum size of a temporary log is set using the system variable 'mybs_temp_log_threshold'. The size is a number of bytes, unless the value is followed by one of the following: Kb, MB or GB. Note: the maximum size is 4GB. RN3: Implement upload of BLOBs into the MyBS BLOB repository. For example: curl -d "This is a BLOB streaming test." http://localhost:8080/test/notes_tab test/1-128-3dd49f08-0 Upload supports both HTTP PUT and POST. The upload returns the BLOB URL ("test/1-128-3dd49f08-0" in this example). After upload, a BLOB can be downloaded as follows: curl http://localhost:8080/test/1-128-3dd49f08-0 The repository consists of a number of "repository files" stored in the bs-repository sub-directory. Each repository file contains a number of BLOBs. The maximum size of a repository file is determined by the 'mybs_repository_threshold' system variable. The size is a number of bytes, unless the value is followed by one of the following: Kb, MB or GB. Note: the maximum size is 256 Terabytes, which is also the maximum size of a single BLOB. Each database has its own repository. The maximum number of repository files per database repository is 16777215. ------- 0.1.00 Alpha - 2007-06-19 RN2: The port number of the MyBS engine can be set using the 'mybs_port' system variable. This variable should be set on the mysqld command line, or in the configuation file. RN1: Bugfix: On shutdown server prints the following error "Error in my_thread_global_end(): 1 threads didn't exit". This error was caused by a THD strcuture still reference by the engine (free moved to the finally block). Thanks to all those who have offered advice and comments. This is the first release of the BLOB Streaming engine for MySQL (MyBS). The MyBS engine requires a MySQL storage engine that is "streaming enabled". The first streaming enabled engine is PBXT 0.9.87 Beta, which can be downloaded from www.primebase.com/xt or http://sourceforge.net/projects/pbxt. The first version of the BLOB Streaming engine enables download of BLOB data that has been stored normally in the tables of the streaming enabled engine. For example: use test; CREATE TABLE notes_tab ( n_id INTEGER PRIMARY KEY, n_text BLOB ) ENGINE=pbxt; INSERT notes_tab VALUES (1, "This is a BLOB streaming test."); The URL: http://localhost:8080/test/notes_tab/n_text/n_id=1 will return the value "This is a BLOB streaming test." NOTE: The issue of security has not been addressed in this version.