Analytics-as-a-service, or analytics job service, is emerging as a new paradigm for enterprise data analytics. These services are motivated by the fact that setting up and running data analytics is a major hurdle for enterprises. Although platform as a service (PaaS), software as a service (SaaS), and more recently database as a service (DBaaS) have eased the pain of provisioning and scaling hardware and software infrastructures, users are still responsible for managing and tuning their servers. A job service mitigates this pain by offering server-less analytics capability that does not require users to provision and manage servers. Instead, the service provider takes care of managing and tuning a query engine that can scale instantly and on demand. Users can get started quickly using the all familiar SQL interface and pay only for the processing used for each query, in contrast to paying for the entire provisioned server infrastructure irrespective of the compute resources actually used.
At Microsoft, SCOPE is an analytics-as-a-service which is used for internal data analytics. SCOPE is deployed over hundreds of thousands of machines, running hundreds of thousands of production analytic jobs per day that are written by thousands of developers, processing several exabytes of data per day, and involving several hundred petabytes of I/O. SCOPE users are not required to manage or tune their hardware and software infrastructure, and they concentrate only on their processing logic. However, the shared nature of SCOPE job service across several users and teams leads to significant overlaps in partial computations, i.e., parts of the processing are duplicated across multiple jobs, thus generating redundant costs. The goal of CloudViews is to automatically detect and reuse overlapping computations in the SCOPE job service, while allowing users to write their jobs just as before, i.e., with zero changes to user scripts.
Publications
-
Sunny Gakhar, Joyce Cahoon, Wangchao Le, Xiangnan Li, Kaushik Ravichandran, Hiren Patel, Marc Friedman, Brandon Haynes, Shi Qiao, Alekh Jindal, Jyoti Leeka
Pipemizer: An Optimizer for Analytics Data Pipelines
VLDB 2022, Sydney, Australia. (Demo paper) -
Yiwen Zhu, Matteo Interlandi, Abhishek Roy, Krishnadhan Das, Hiren Patel, Malay Bag, Hitesh Sharma, Alekh Jindal
Phoebe: A Learning-based Checkpoint Optimizer
VLDB 2021 -
Alekh Jindal, Shi Qiao, Hiren Patel, Abhishek Roy, Jyoti Leeka, Brandon Haynes
Production Experiences from Computation Reuse at Microsoft
EDBT 2021 (Industry) -
Alekh Jindal
Applied Research Lessons from CloudViews Project
SIGMOD Record, September 2020 -
Abhishek Roy, Alekh Jindal, Hiren Patel, Ashit Gosalia, Subru Krishnan, Carlo Curino
SparkCruise: Handsfree Computation Reuse in Spark
VLDB 2019/PVLDB, Los Angeles, USA. (Demo paper) -
Alekh Jindal, Konstantinos Karanasos, Sriram Rao, Hiren Patel
Selecting Subexpressions to Materialize at Datacenter Scale
VLDB 2018/PVLDB, Rio de Janeiro, Brazil. -
Alekh Jindal, Shi Qiao, Hiren Patel, Jarod Yin, Jieming Di, Malay Bag, Marc Friedman, Yifung Lin, Konstantinos Karanasos, Sriram Rao
Computation Reuse in Analytics Job Service at Microsoft
SIGMOD 2018, Houston, USA.
Patents
-
Brandon Haynes, Rana Alotaibi, Anna Pavlenko, Yuanyuan Tian, Jyoti Leeka, Alekh Jindal
Machine learning accelerated semantic equivalence detection.
US Patent 12,436,950 -
Brandon Haynes, Jyoti Leeka, Anna Pavlenko, Rana Alotaibi, Alekh Jindal
Materialized view generation and provision based on queries having a semantically equivalent or containment relationship.
US Patent 12,561,321, US Patent App. US20260072913A1