Multi-tiered storage systems for broadcast and post production can be complex beasts. The combination of large file sizes and high throughput requirements has led, over the years, to most companies adopting a two or three tier storage approach using slow but cheap LTO data tape for the archive, a high bandwidth NAS or SAN as a working store and in the three tiered version smaller local stores for each workstation or playout server.
AltoFM screenshot graph
Managing all this is usually accomplished by a Media Asset Management or Automation system communicating with an archive manager which deals with the complexity of loading tapes located in a robotic system and storing and retrieving files.
As hard disc sizes have grown it has become economically feasible to move this third tier to disc storage, the issues then become the packing density (how much cooling can be applied to a given equipment volume) and the power consumption (disks use power, tapes don’t).
A recent and highly innovative storage system ALTO from Disk Archive Corporation aims to solve these issues by only powering the disks when they are accessed, this means the disks can be packed together achieving similar power and storage densities to tape based systems.
Since the disks are hermetically sealed with the the heads floating over the disk surface some of the mechanical issues associated with dirt and clogging of tape based systems are also avoided leading to very high reliability.
Front-Alto
AltoFM start copy
When one of our customers wanted to use an ALTO system as part of an existing setup we realised there was a need for a file manager application to interface between the Media Management and Automation layers and this new type of storage solution. We created ALTO File Manager to meet this need, providing a simple web based user interface, direct emulation of existing archive control protocols to enable easy integration into existing systems and the ability to combine an unlimited number of ALTO units into one large storage system.

Features

  • Simple linking to Automation and Media Management system using existing drivers and protocols
  • Can connect to an unlimited number of Alto shelves
  • Replication to multiple disks inside Alto
  • Manages the creation of a copy of content in a cloud storage system like Amazon S3
  • Disk Lifecycle Management using pre-conditioning and S.M.A.R.T. monitoring of disks
  • Web based user interface allows manual operation and monitoring
  • All file transfers are verified by MD5 checksums
  • Policy based replication and regeneration
  • Provides both Virtual File System (NFS, CIFS, FTP) and File Transfer API for maximum flexibility
  • Built in Web based documentation

Example Configurations

Type A System

Standalone File Manager runs on an Alto with optional expansion chassis. Provides all features but has no redundancy.

typeASystem

Type B System

File manager runs on an external server. This configuration can support up to 64 Alto main chassis each with expansion. This arrangement can provide greater throughput and by placing some of the Alto chassis remote from the installation offsite backup.

typeBSystem

Type C System

File manager runs in a dual redundant configuration on a pair of external servers. These servers do not need to be co-located. Provides high throughput and complete redundancy.

typeCSystem

How it works

Database

File Manager uses a database to provide the index and directory operations usually provided by a filesystem. When a file is opened for reading or writing the database is used to locate the file or allocate space for it if it is a new file. The appropriate disks are then powered up until no longer required. The database also stores an audit trail of all disk operations with their times. This together with the location data enables reports to be produced showing disk usage, spread of items, free storage etc.

Checksums

Because Alto disks are used as direct file stores (no raid configurations) most installs will be set up to make multiple copies of a file. File manager makes these multiple copies simultaneously as the file is written, at the same time an MD5 checksum is calculated for the data and stored in the database. Whenever a linear read is made of a file the checksum is checked, any error will place the disk offline and fail the disk operation. The retry will then be sourced from the second copy of the file.

Disk Regeneration

In File Manager systems set to make 2 copies of files a failed disk can be regenerated using the second copies of the files. This operation can be used either to regenerate the files onto the same disk after reformatting if the failure is considered temporary or onto a new disk.

Virtual File System

File Manager creates a virtual file system. The top level directories are the storage categories which determine storage policies (speed of transfer, number of copies kept etc). Below this any file hierarchy can be used enabling archiving of post production projects or integration with existing processes. Unlike most virtual file systems data access is directly to and from the Alto disks, there are no stub files and no intermediate raid store is used. This gives better system scaling and the assurance that a write that has been completed is secure and copied in the archive. The virtual file system can be shared by NFS, CIFS/SMB or FTP including FXP for media management systems. Checksums are used for all linear transfers to give data reassurance.

File Transfer API

File manager supports File transfer by API for control of transfers by external systems. It can simulate many popular archiving systems to simplify integration. In this mode full prioritised job control is available allowing a large batch of transfer requests to be handled in the most efficient way.

Web UI

A multi-user web UI can be used to monitor and trigger transfers and reconfigure the system. The UI allows the SMART data from the disks to be monitored.