The software library is integrable with the likes of Dask and Apache Beam.

Ben Wodecki, Jr. Editor

September 27, 2022

1 Min Read

The software library is integrable with the likes of Dask and Apache Beam.

Google has unveiled TensorStore, a C++ and Python open-source software library designed to decrease ML system development time.

The library, developed by Google Research, is designed to avoid issues relating to storing and manipulating data, providing users with an API capable of handling large datasets without requiring powerful devices.

TensorStore does not store in memory until requested, with Google’s AI team explaining “arbitrarily large underlying datasets can be loaded and manipulated without having to store the entire dataset in memory.”

One use case Google researchers suggested TensorStore could be used for is brain mapping, creating a detailed schematic of the function of different areas of the brain. Such a use case requires sizable storage and processing capabilities to handle petabyte-size datasets.

Google’s team used TensorStore to handle the h01 dataset, among the largest datasets for data on human brain tissue. h01 consists of roughly 1.4 petabytes of imaging data, which users can manipulate using TensorStore.

ai-image3.gif

Google researchers demonstrated this capability by using the library to build a simple Python API and create a 3D image of a fly’s brain consisting of 56 trillion voxels, which are 3D pixels.

The TensorStore library supports several storage systems, including Google Cloud Storage, HTTP servers and local and network filesystems. It’s also integrable with parallel computing libraries including Apache Beam and Dask.

The newly launched AI tool can also support indexing features, including alignment, and virtual views using downsampling and data type conversion.

About the Author(s)

Ben Wodecki

Jr. Editor

Ben Wodecki is the Jr. Editor of AI Business, covering a wide range of AI content. Ben joined the team in March 2021 as assistant editor and was promoted to Jr. Editor. He has written for The New Statesman, Intellectual Property Magazine, and The Telegraph India, among others. He holds an MSc in Digital Journalism from Middlesex University.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like