Are Data Scientists the most dissatisfied teams in AI?

by Matthew Bishop, Core Scientific

1 October 2019

Data science is a complex, multidisciplinary field with a small pool of talent. Demand for that talent will only continue to grow, and attracting and retaining expertise is an increasing challenge for all companies focused on AI.

At the heart of the dissatisfaction Data Scientists experience is the lack of on-demand compute that is optimized for deep learning and model training. Put simply, it is just hard for Data Science teams to get the infrastructure to handle large data sets collocated with high-availability GPU compute – it is capital intensive and complex to deploy on-premises and expensive to scale-up in public clouds, even if available. The importance of AI to a company’s future is too critical to have Data Scientists doing ML by credit card, or running their own tech ops on self-managed environments!

Core Scientific has set themselves the task of making it fast to train and deploy AI models and equip Data Scientists to take on the most challenging AI problems. They are providing accelerated GPU compute tuned to the demands of deep learning and AI, combining NVIDIA DGX-2 and the latest flash storage solutions.

This new on-demand service from Core Scientific could be the start of helping business leaders make their companies a great destination for Data Scientists looking to make an impact and change the world through AI.

The Core Scientific Cloud for Data Scientists

To achieve this, Core Scientific has curated the best-in-class componentry across the hardware and software landscape to build the Cloud for Data Scientists. With the goal of equipping data scientists to take on the most advanced AI challenges, Core Scientific has partnered with the leaders in GPU hardware and software. Dedicated NVIDIA DGX-2 GPU Systems and the latest Pure Storage FlashBlade reference architecture solutions have enabled them to deliver an on-demand AI solution.

An important service is the Core Scientific aiLab, designed for customers to train large models. The aiLab provides Artificial Intelligence and Machine Learning tools built from the ground up and including the most common Data Science tools. It is a complete solution for deep learning customers that frequently train large models and struggle to complete their work on commodity hardware.

Focus on outcomes

Deploying AI infrastructure in a high-availability and fully redundant environment, with Kubernetes and JupyterHub for easy deployment and management, is a challenging prospect for many organizations. But this solution from Core Scientific means that data science organizations can focus on outcomes rather than managing a complex technical environment. Whether it is accelerating Spark workloads using GPU compute without needing any code rewrite, or delivering Postgres solutions using the fastest GPU database available, Core Scientific have built a network of AI apps that can provide a complete solution.

A further partnership with SFL Scientific means that customers of the Cloud for Data Scientists can easily access the skills and expertise needed to develop and train models significantly faster, and utilize much larger data sets, in optimized AI-ready environments. SFL Scientific is a data science consulting and professional services company, providing a broad range of solutions in data engineering, machine learning, and Artificial Intelligence. Together, the two organizations work with R&D and executive teams to build AI solutions, and solve challenges across every industry.  

Customers have enjoyed the performance benefits that come from this kind of advanced architecture, and the cost management advantages that come from a cloud hosted solution. Rami Safadi, Chief Data Officer at Jam City, is a customer: “At Jam City, we rely on AI to guide game design and transform our business. Core Scientific’s cloud has enhanced how we utilize data and allowed us to analyze billions of rows of data per day. We have seen a significant increase in speed, enabling us to train an entirely new set of winning AI business models.”


Matthew Bishop is the Chief Administrative Officer of Core Scientific. He has a history of leading large scale and global transformations in the technology sector, and leads the strategy and business development for this innovative Seattle based tech startup. Core Scientific is a leader in Artificial Intelligence and Blockchain technologies, delivering best-in-class infrastructure and software solutions.