Google showcases TensorStore ML open-source software library

The software library is integrable with the likes of Dask and Apache Beam.

Google has unveiled TensorStore, a C++ and Python open-source software library designed to decrease ML system development time.

The library, developed by Google Research, is designed to avoid issues relating to storing and manipulating data, providing users with an API capable of handling large datasets without requiring powerful devices.

TensorStore does not store in memory until requested, with Google’s AI team explaining “arbitrarily large underlying datasets can be loaded and manipulated without having to store the entire dataset in memory.”

One use case Google researchers suggested TensorStore could be used for is brain mapping, creating a detailed schematic of the function of different areas of the brain. Such a use case requires sizable storage and processing capabilities to handle petabyte-size datasets.

Google’s team used TensorStore to handle the h01 dataset, among the largest datasets for data on human brain tissue. h01 consists of roughly 1.4 petabytes of imaging data, which users can manipulate using TensorStore.

Google researchers demonstrated this capability by using the library to build a simple Python API and create a 3D image of a fly’s brain consisting of 56 trillion voxels, which are 3D pixels.

The TensorStore library supports several storage systems, including Google Cloud Storage, HTTP servers and local and network filesystems. It’s also integrable with parallel computing libraries including Apache Beam and Dask.

The newly launched AI tool can also support indexing features, including alignment, and virtual views using downsampling and data type conversion.