SlideShare a Scribd company logo
1 of 60
Download to read offline
Big Data is changing abruptly, 
and where it is likely heading 
Big Data Spain 
2014-11-17 
bigdataspain.org 
Paco Nathan 
@pacoid
Some questions, and an emerging idea… 
The economics of datacenters has shifted toward 
warehouse scale, based on commodity hardware 
that includes multicore and large memory spaces. 
! 
Incumbent technologies for Big Data do not 
embrace those changes, however newer OSS 
projects have emerged to leverage them. 
2
Some questions, and an emerging idea… 
Also, the data requirements are changing abruptly 
as sensor data, microsatellites, and other IoT use 
cases boost data rates by orders of magnitude. 
! 
We have math available to address those industrial 
needs, but have software frameworks kept up? 
3
1. Containers 
4
If you’re not using them, you’re missing out… 
! 
See recent announcements by Google, Amazon, etc. 
! 
Containers are the tip of the iceberg: OS have 
evolved quietly, while practices near to customers 
have perhaps not kept pace 
5 
Containers
Key benefits based on Mesos case studies: 
• mixed workloads 
• higher utilization rates 
• lower latency for data products 
• significantly lower Ops overhead 
! 
Examples: Mesos @Twitter, eBay, Netflix, many 
startups, etc.; Borg/Omega @Google, 
Symphony @IBM, Autopilot @Microsoft, etc. 
! 
Major cloud users understand this shift; however 
much of the world still thinks in terms of VMware 
6 
Containers
We need much more than Docker, we need better 
definitions for what is to be used outside of the 
container as well – e.g., for microservices 
• Intro to Mesos: goo.gl/PbUfNe 
• https://mesosphere.com/ 
• A project to watch: Weave 
7 
Containers
8 
Containers 
One problem: how does an institution, such 
as a major bank, navigate this kind of change?
2. Clouds 
9
Speaking of clouds, forgive me for being biased… 
For nearly a decade, I’ve been talking with people 
about whether cloud economics makes sense? 
Consider: datacenter util rates in single-digits (~8%) 
10 
Clouds
Companies accustomed to cheap energy rates, 
with staff who thinking in terms of individual VMs… 
these will not find clouds to be a friendly territory. 
Not so much about software, as it is about 
sophisticated engineering practice. 
The change recalls the shift of BI ⇒ Data Science 
That was not simple… 
11 
Clouds
The majors understand this new world: Google, 
Amazon, Microsoft, IBM, Apple, etc. 
Sub-majors can create their own clouds: Facebook, 
Twitter, LinkedIn, etc. 
12 
Clouds 
What about industries approaching Big Data now 
which cannot afford to buy small armies of expert 
Ops people? How they must proceed becomes a 
key question.
Current research points to significant changes 
ahead, where machine learning plays a key role 
in the context of advanced cluster scheduling: 
13 
Clouds 
Improving Resource Efficiency 
with Apache Mesos 
Christina Delimitrou 
youtu.be/YpmElyi94AA 
Experiences with Quasar+Mesos showed: 
• 88% apps get >95% performance 
• ~10% overprovisioning instead of 500% 
• up to 70% cluster util at steady state 
• 23% shorter scenario completion
3. Abstract Algebra 
14
A question on StackOverflow in 2013 challenged 
Twitter’s use of Abstract Algebra libraries for their 
revenue apps… 
cs.stackexchange.com/questions/9648/what-use-are-groups- 
monoids-and-rings-in-database-computations 
The answers are enlightening: think of a kind of 
containerization for business logic – leading to 
exponential increase in performance, particularly 
in the context of streaming cases. 
15 
Abstract Algebra
Abstract Algebra 
A cheat-sheet: 
non-empty 
set 
Semigroup 
has an identity element 
Group 
has a binary associative 
operation, with closure 
each element has an inverse 
has two binary 
associative operations: 
addition and multiplication 
Ring 
Monoid
Add ALL the Things: 
Abstract Algebra Meets Analytics 
infoq.com/presentations/abstract-algebra- 
analytics 
Avi Bryant, Strange Loop (2013) 
• grouping doesn’t matter (associativity) 
• ordering doesn’t matter (commutativity) 
• zeros get ignored 
In other words, while partitioning data 
at scale is quite difficult, you can let the 
math allow your code to be flexible at 
scale 
Avi Bryant 
@avibryant 
Abstract Algebra
Algebra for Analytics 
speakerdeck.com/johnynek/ 
algebra-for-analytics 
Oscar Boykin, Strata SC (2014) 
• “Associativity allows parallelism 
in reducing” by letting you put 
the () where you want 
• “Lack of associativity increases 
latency exponentially” 
Oscar Boykin 
@posco 
Abstract Algebra
4. Functional Programming 
19
Functional Programming for Big Data 
Theory, eight decades ago: 
what can be computed? 
Haskell Curry 
haskell.org 
Alonso Church 
wikipedia.org 
Praxis, four decades ago: 
algebra for applicative systems 
John Backus 
acm.org 
David Turner 
wikipedia.org 
Reality, two decades ago: 
machine data from web apps 
Pattie Maes 
MIT Media Lab 
20
Functional Programming for Big Data 
circa 2002: 
mitigate risk of large distributed workloads lost 
due to disk failures on commodity hardware… 
Google File System 
Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung 
research.google.com/archive/gfs.html 
! 
MapReduce: Simplified Data Processing on Large Clusters 
Jeffrey Dean, Sanjay Ghemawat 
research.google.com/archive/mapreduce.html 
21
Functional Programming for Big Data 
DryadLINQ influenced a new class of workflow 
abstractions based on functional programming: 
Spark, Flink, Scalding, Scoobi, Scrunch, etc. 
• needed for leveraging the newer hardware: 
multicore, large memory spaces, etc. 
• significant code volume reduction ⇒ eng costs 
22
Functional Programming for Big Data 
2002 
2004 
MapReduce paper 
2002 
MapReduce @ Google 
2004 2006 2008 2010 2012 2014 
2006 
Hadoop @ Yahoo! 
2014 
Apache Spark top-level 
2010 
Spark paper 
2008 
Hadoop Summit 
23
Functional Programming for Big Data 
MapReduce 
Pregel Giraph 
Dremel Drill 
S4 Storm 
F1 
MillWheel 
General Batch Processing Specialized Systems: 
Impala 
GraphLab 
iterative, interactive, streaming, graph, etc. 
Tez 
MR doesn’t compose well for large applications, 
and so specialized systems emerged as workarounds 
24
Functional Programming for Big Data 
circa 2010: 
a unified engine for enterprise data workflows, 
based on commodity hardware a decade later… 
Spark: Cluster Computing with Working Sets 
Matei Zaharia, Mosharaf Chowdhury, 
Michael Franklin, Scott Shenker, Ion Stoica 
people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf 
! 
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for 
In-Memory Cluster Computing 
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, 
Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica 
usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf 
25
Functional Programming for Big Data 
One minor problem: where do we find 
many graduating students who are good at 
Scala, Clojure, F#, etc.? 
26
4. Databases 
27
+1 generations of computer scientists has 
grown up thinking that Data implies SQL… 
Excellent as a kind of declarative, functional 
language for describing data transformations 
However, in the long run we should move away 
from thinking in terms of SQL 
Computational Thinking as a rubric in 
general would be preferred 
28 
Databases
A step further is to move away from thinking of 
data as an engineering issue – it is about business 
So much of the US curricula for math is still stuck 
in the Cold War: years of calculus to identify the 
best people to build ICBMs 
We need more business people thinking in terms 
of math, beyond calculus, to contend with difficult 
problems in supply chain, maintenance schedules, 
energy needs, transportation routes, etc. 
29 
Databases
Databases 
1. starting with 
real-world data ⇒ 
2. leverage graph queries 
for representation ⇒ 
3. convert to sparse matrix, 
use FP and abstract algebra ⇒ 
4. achieve high-ROI parallelism at scale, 
mostly about optimization ⇒
6. Approximations 
31
Approximations 
19-20c. statistics emphasized defensibility 
in lieu of predictability, based on analytic 
variance and goodness-of-fit tests 
! 
That approach inherently led toward a 
manner of computational thinking based 
on batch windows 
! 
They missed a subtle point… 
32
21c. shift towards modeling based on probabilistic 
approximations: trade bounded errors for greatly 
reduced resource costs 
highlyscalable.wordpress.com/2012/05/01/ 
probabilistic-structures-web-analytics-data- 
mining/ 
33 
Approximations
21c. shift towards modeling based on probabil 
approximations: trade bounded errors for greatly 
reduced resource costs 
Twitter catch-phrase: 
“Hash, don’t sample” 
highlyscalable.wordpress.com/2012/05/01/ 
probabilistic-structures-web-analytics-data- 
mining/ 
34 
Approximations
a fascinating and relatively new area, pioneered 
by relatively few people – e.g., Philippe Flajolet 
provides approximation, with error bounds – 
in general uses significantly less resources 
(RAM, CPU, etc.) 
many algorithms can be constructed from 
combinations of read and write monoids 
aggregate different ranges by composing 
hashes, instead of repeating full-queries 
35 
Approximations
algorithm use case example 
Count-Min Sketch frequency summaries code 
HyperLogLog set cardinality code 
Bloom Filter set membership 
MinHash 
set similarity 
DSQ streaming quantiles 
SkipList ordered sequence search 
36 
Approximations
algorithm use case example 
Count-Min Sketch frequency summaries code 
HyperLogLog set cardinality code 
suggestion: consider these 
as your most quintessential 
collections data types at scale 
Bloom Filter set membership 
MinHash 
set similarity 
DSQ streaming quantiles 
SkipList ordered sequence search 
37 
Approximations
• sketch algorithms: trade bounded errors for 
orders of magnitude less required resources, 
e.g., fit more complex apps in memory 
• multicore + large memory spaces (off heap) are 
increasing the resources per node in a cluster 
• containers allow for finer-grain allocation of 
cluster resources and multi-tenancy 
• monoids, etc.: guarantees of associativity within 
the code allow for more effective distributed 
computing, e.g., partial aggregates 
• less resources must be spent sorting/windowing 
data prior to working with a data set 
• real-time apps, which don’t have the luxury of 
anticipating data partitions, can respond quickly 
38 
Approximations
Probabilistic Data Structures for Web 
Analytics and Data Mining 
Ilya Katsov (2012-05-01) 
A collection of links for streaming 
algorithms and data structures 
Debasish Ghosh 
Aggregate Knowledge blog (now Neustar) 
Timon Karnezos, Matt Curcio, et al. 
Probabilistic Data Structures and 
Breaking Down Big Sequence Data 
C. Titus Brown, O'Reilly (2010-11-10) 
Algebird 
Avi Bryant, Oscar Boykin, et al. Twitter (2012) 
Mining of Massive Datasets 
Jure Leskovec, Anand Rajaraman, 
Jeff Ullman, Cambridge (2011) 
39 
Approximations
7. Notebooks 
40
Spreadsheets represented a revolution in how 
to think about computing 
! 
What Fernando Perez, et al., have done with 
IPython is subtle and powerful 
! 
Technology is not just about writing code; it is 
about people using that code 
! 
Enterprise data workflows are moving toward 
cloud-based notebooks 
41 
Notebooks
Early emphasis on tools in Big Data now gives 
way to workflows: people, automation, process, 
integration, test, maintenance, scale 
Google had the brilliant approach of shared 
documents… 
Now we see cloud-based notebooks, which 
begin to displace web apps 
42 
Notebooks 
http://nbviewer.ipython.org/ 
http://jupyter.org/
43 
Notebooks 
nature.com/news/ipython-interactive-demo-7.21492
What is Spark?
What is Spark? 
Developed in 2009 at UC Berkeley AMPLab, then 
open sourced in 2010, Spark has since become 
one of the largest OSS communities in big data, 
with over 200 contributors in 50+ organizations 
spark.apache.org 
“Organizations that are looking at big data challenges – 
including collection, ETL, storage, exploration and analytics – 
should consider Spark for its in-memory performance and 
the breadth of its model. It supports advanced analytics 
solutions on Hadoop clusters, including the iterative model 
required for machine learning and graph analysis.” 
Gartner, Advanced Analytics and Data Science (2014)
What is Spark?
What is Spark? 
Spark Core is the general execution engine for the 
Spark platform that other functionality is built atop: 
! 
• in-memory computing capabilities deliver speed 
• general execution model supports wide variety 
of use cases 
• ease of development – native APIs in Java, Scala, 
Python (+ SQL, Clojure, R)
What is Spark? 
WordCount in 3 lines of Spark 
WordCount in 50+ lines of Java MR
What is Spark? 
Sustained exponential growth, as one of the most 
active Apache projects ohloh.net/orgs/apache
Big Data is changing abruptly, and where it is likely heading
• generalized patterns 
⇒ unified engine for many use cases 
• lazy evaluation of the lineage graph 
⇒ reduces wait states, better pipelining 
• generational differences in hardware 
⇒ off-heap use of large memory spaces 
• functional programming / ease of use 
⇒ reduction in cost to maintain large apps 
• lower overhead for starting jobs 
• less expensive shuffles 
51 
What is Spark?
What is Spark? 
Kafka + Spark + Cassandra 
datastax.com/documentation/datastax_enterprise/4.5/ 
datastax_enterprise/spark/sparkIntro.html 
http://helenaedelson.com/?p=991 
github.com/datastax/spark-cassandra-connector 
github.com/dibbhatt/kafka-spark-consumer 
unified compute 
data streams columnar key-value
databricks.com/blog/2014/11/05/spark-officially-sets- 
a-new-record-in-large-scale-sorting.html 
53 
What is Spark?
What is Spark? 
Databricks Cloud 
databricks.com/blog/2014/07/14/databricks-cloud-making- 
big-data-easy.html 
youtube.com/watch?v=dJQ5lV5Tldw#t=883
Resources 
55
certification: 
Apache Spark developer certificate program 
• http://oreilly.com/go/sparkcert 
• defined by Spark experts @Databricks 
• assessed by O’Reilly Media 
• establishes the bar for Spark expertise 
56
community: 
spark.apache.org/community.html 
video+slide archives: spark-summit.org 
events worldwide: goo.gl/2YqJZK 
resources: databricks.com/spark-training-resources 
workshops: databricks.com/spark-training 
new: MOOCs via edX and University of California 
57
books: 
Fast Data Processing 
with Spark 
Holden Karau 
Packt (2013) 
shop.oreilly.com/product/ 
9781782167068.do 
Spark in Action 
Chris Fregly 
Manning (2015*) 
sparkinaction.com/ 
Learning Spark 
Holden Karau, 
Andy Konwinski, 
Matei Zaharia 
O’Reilly (2015*) 
shop.oreilly.com/product/ 
0636920028512.do 
58
events: 
Strata EU 
Barcelona, Nov 19-21 
strataconf.com/strataeu2014 
Data Day Texas 
Austin, Jan 10 
datadaytexas.com 
Strata CA 
San Jose, Feb 18-20 
strataconf.com/strata2015 
Spark Summit East 
NYC, Mar 18-19 
spark-summit.org/east 
Strata EU 
London, May 5-7 
strataconf.com/big-data-conference-uk-2015 
Spark Summit 2015 
SF, Jun 15-17 
spark-summit.org 
59
presenter: 
monthly newsletter for updates, 
events, conf summaries, etc.: 
liber118.com/pxn/ 
Just Enough Math 
O’Reilly, 2014 
justenoughmath.com 
preview: youtu.be/TQ58cWgdCpA 
Enterprise Data Workflows 
with Cascading 
O’Reilly, 2013 
shop.oreilly.com/product/ 
0636920028536.do 60

More Related Content

What's hot

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
Tiny Batches, in the wine: Shiny New Bits in Spark StreamingTiny Batches, in the wine: Shiny New Bits in Spark Streaming
Tiny Batches, in the wine: Shiny New Bits in Spark StreamingPaco Nathan
 
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Big Data Spain
 
Graph Analytics in Spark
Graph Analytics in SparkGraph Analytics in Spark
Graph Analytics in SparkPaco Nathan
 
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Analytics - Best of the Worst : Anti-patterns & AntidotesBig Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Analytics - Best of the Worst : Anti-patterns & AntidotesKrishna Sankar
 
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and MoreStrata 2015 Data Preview: Spark, Data Visualization, YARN, and More
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and MorePaco Nathan
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
QCon São Paulo: Real-Time Analytics with Spark StreamingQCon São Paulo: Real-Time Analytics with Spark Streaming
QCon São Paulo: Real-Time Analytics with Spark StreamingPaco Nathan
 
Big Graph Analytics on Neo4j with Apache Spark
Big Graph Analytics on Neo4j with Apache SparkBig Graph Analytics on Neo4j with Apache Spark
Big Graph Analytics on Neo4j with Apache SparkKenny Bastani
 
Agile data science with scala
Agile data science with scalaAgile data science with scala
Agile data science with scalaAndy Petrella
 
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...Andy Petrella
 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data ScienceDataWorks Summit
 
Graph Data: a New Data Management Frontier
Graph Data: a New Data Management FrontierGraph Data: a New Data Management Frontier
Graph Data: a New Data Management FrontierDemai Ni
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapePaco Nathan
 
How Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapeHow Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapePaco Nathan
 
Scala: the unpredicted lingua franca for data science
Scala: the unpredicted lingua franca  for data scienceScala: the unpredicted lingua franca  for data science
Scala: the unpredicted lingua franca for data scienceAndy Petrella
 
Big Data Analysis and Industrial Approach using Spark
Big Data Analysis and Industrial Approach using SparkBig Data Analysis and Industrial Approach using Spark
Big Data Analysis and Industrial Approach using SparkJongwook Woo
 
High Performance Computing and Big Data
High Performance Computing and Big Data High Performance Computing and Big Data
High Performance Computing and Big Data Geoffrey Fox
 
Deep Learning with MXNet - Dmitry Larko
Deep Learning with MXNet - Dmitry LarkoDeep Learning with MXNet - Dmitry Larko
Deep Learning with MXNet - Dmitry LarkoSri Ambati
 
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...Mahantesh Angadi
 

What's hot (19)

Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
Tiny Batches, in the wine: Shiny New Bits in Spark StreamingTiny Batches, in the wine: Shiny New Bits in Spark Streaming
Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
 
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
Fishing Graphs in a Hadoop Data Lake by Jörg Schad and Max Neunhoeffer at Big...
 
Graph Analytics in Spark
Graph Analytics in SparkGraph Analytics in Spark
Graph Analytics in Spark
 
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Analytics - Best of the Worst : Anti-patterns & AntidotesBig Data Analytics - Best of the Worst : Anti-patterns & Antidotes
Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes
 
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and MoreStrata 2015 Data Preview: Spark, Data Visualization, YARN, and More
Strata 2015 Data Preview: Spark, Data Visualization, YARN, and More
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
QCon São Paulo: Real-Time Analytics with Spark StreamingQCon São Paulo: Real-Time Analytics with Spark Streaming
QCon São Paulo: Real-Time Analytics with Spark Streaming
 
Big Graph Analytics on Neo4j with Apache Spark
Big Graph Analytics on Neo4j with Apache SparkBig Graph Analytics on Neo4j with Apache Spark
Big Graph Analytics on Neo4j with Apache Spark
 
Agile data science with scala
Agile data science with scalaAgile data science with scala
Agile data science with scala
 
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...
Agile data science: Distributed, Interactive, Integrated, Semantic, Micro Ser...
 
The Future of Data Science
The Future of Data ScienceThe Future of Data Science
The Future of Data Science
 
Graph Data: a New Data Management Frontier
Graph Data: a New Data Management FrontierGraph Data: a New Data Management Frontier
Graph Data: a New Data Management Frontier
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscape
 
How Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapeHow Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscape
 
Spark streaming
Spark streamingSpark streaming
Spark streaming
 
Scala: the unpredicted lingua franca for data science
Scala: the unpredicted lingua franca  for data scienceScala: the unpredicted lingua franca  for data science
Scala: the unpredicted lingua franca for data science
 
Big Data Analysis and Industrial Approach using Spark
Big Data Analysis and Industrial Approach using SparkBig Data Analysis and Industrial Approach using Spark
Big Data Analysis and Industrial Approach using Spark
 
High Performance Computing and Big Data
High Performance Computing and Big Data High Performance Computing and Big Data
High Performance Computing and Big Data
 
Deep Learning with MXNet - Dmitry Larko
Deep Learning with MXNet - Dmitry LarkoDeep Learning with MXNet - Dmitry Larko
Deep Learning with MXNet - Dmitry Larko
 
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
Introduction and Overview of BigData, Hadoop, Distributed Computing - BigData...
 

Similar to Big Data is changing abruptly, and where it is likely heading

Trivento summercamp masterclass 9/9/2016
Trivento summercamp masterclass 9/9/2016Trivento summercamp masterclass 9/9/2016
Trivento summercamp masterclass 9/9/2016Stavros Kontopoulos
 
Data Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackData Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackAnant Corporation
 
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...Mihai Criveti
 
Big Data using NoSQL Technologies
Big Data using NoSQL TechnologiesBig Data using NoSQL Technologies
Big Data using NoSQL TechnologiesAmit Singh
 
Big Data
Big DataBig Data
Big DataNGDATA
 
Café da manhã - São Paulo - Use-cases and opportunities in BigData with Hadoop
Café da manhã - São Paulo - Use-cases and opportunities in BigData with HadoopCafé da manhã - São Paulo - Use-cases and opportunities in BigData with Hadoop
Café da manhã - São Paulo - Use-cases and opportunities in BigData with HadoopOCTO Technology
 
Trivento summercamp fast data 9/9/2016
Trivento summercamp fast data 9/9/2016Trivento summercamp fast data 9/9/2016
Trivento summercamp fast data 9/9/2016Stavros Kontopoulos
 
DCAF 2023 1 and 2.pdf
DCAF 2023 1 and 2.pdfDCAF 2023 1 and 2.pdf
DCAF 2023 1 and 2.pdfAlan Morrison
 
Big Data Session 1.pptx
Big Data Session 1.pptxBig Data Session 1.pptx
Big Data Session 1.pptxElsonPaul2
 
Data Science at Scale - The DevOps Approach
Data Science at Scale - The DevOps ApproachData Science at Scale - The DevOps Approach
Data Science at Scale - The DevOps ApproachMihai Criveti
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big DataRoi Blanco
 
Introduction to Cloud computing and Big Data-Hadoop
Introduction to Cloud computing and  Big Data-HadoopIntroduction to Cloud computing and  Big Data-Hadoop
Introduction to Cloud computing and Big Data-HadoopNagarjuna D.N
 
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloMini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloOCTO Technology
 
Cloud computing 13 principal enabling technologies
Cloud computing 13 principal  enabling technologiesCloud computing 13 principal  enabling technologies
Cloud computing 13 principal enabling technologiesVaibhav Khanna
 
Future of big data nick kabra speaker compendium march 2013
Future of big data nick kabra speaker compendium march 2013Future of big data nick kabra speaker compendium march 2013
Future of big data nick kabra speaker compendium march 2013nkabra
 

Similar to Big Data is changing abruptly, and where it is likely heading (20)

Big data business case
Big data   business caseBig data   business case
Big data business case
 
Trivento summercamp masterclass 9/9/2016
Trivento summercamp masterclass 9/9/2016Trivento summercamp masterclass 9/9/2016
Trivento summercamp masterclass 9/9/2016
 
Graph Realities
Graph RealitiesGraph Realities
Graph Realities
 
Data Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data StackData Engineer's Lunch #85: Designing a Modern Data Stack
Data Engineer's Lunch #85: Designing a Modern Data Stack
 
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
DevOps for Data Engineers - Automate Your Data Science Pipeline with Ansible,...
 
Cloud computingjun28
Cloud computingjun28Cloud computingjun28
Cloud computingjun28
 
Cloud computingjun28
Cloud computingjun28Cloud computingjun28
Cloud computingjun28
 
Big Data using NoSQL Technologies
Big Data using NoSQL TechnologiesBig Data using NoSQL Technologies
Big Data using NoSQL Technologies
 
Big Data
Big DataBig Data
Big Data
 
Café da manhã - São Paulo - Use-cases and opportunities in BigData with Hadoop
Café da manhã - São Paulo - Use-cases and opportunities in BigData with HadoopCafé da manhã - São Paulo - Use-cases and opportunities in BigData with Hadoop
Café da manhã - São Paulo - Use-cases and opportunities in BigData with Hadoop
 
Trivento summercamp fast data 9/9/2016
Trivento summercamp fast data 9/9/2016Trivento summercamp fast data 9/9/2016
Trivento summercamp fast data 9/9/2016
 
DCAF 2023 1 and 2.pdf
DCAF 2023 1 and 2.pdfDCAF 2023 1 and 2.pdf
DCAF 2023 1 and 2.pdf
 
Computer project
Computer projectComputer project
Computer project
 
Big Data Session 1.pptx
Big Data Session 1.pptxBig Data Session 1.pptx
Big Data Session 1.pptx
 
Data Science at Scale - The DevOps Approach
Data Science at Scale - The DevOps ApproachData Science at Scale - The DevOps Approach
Data Science at Scale - The DevOps Approach
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big Data
 
Introduction to Cloud computing and Big Data-Hadoop
Introduction to Cloud computing and  Big Data-HadoopIntroduction to Cloud computing and  Big Data-Hadoop
Introduction to Cloud computing and Big Data-Hadoop
 
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São PauloMini-course "Practices of the Web Giants" at Global Code - São Paulo
Mini-course "Practices of the Web Giants" at Global Code - São Paulo
 
Cloud computing 13 principal enabling technologies
Cloud computing 13 principal  enabling technologiesCloud computing 13 principal  enabling technologies
Cloud computing 13 principal enabling technologies
 
Future of big data nick kabra speaker compendium march 2013
Future of big data nick kabra speaker compendium march 2013Future of big data nick kabra speaker compendium march 2013
Future of big data nick kabra speaker compendium march 2013
 

More from Paco Nathan

Human in the loop: a design pattern for managing teams working with ML
Human in the loop: a design pattern for managing  teams working with MLHuman in the loop: a design pattern for managing  teams working with ML
Human in the loop: a design pattern for managing teams working with MLPaco Nathan
 
Human-in-the-loop: a design pattern for managing teams that leverage ML
Human-in-the-loop: a design pattern for managing teams that leverage MLHuman-in-the-loop: a design pattern for managing teams that leverage ML
Human-in-the-loop: a design pattern for managing teams that leverage MLPaco Nathan
 
Human-in-a-loop: a design pattern for managing teams which leverage ML
Human-in-a-loop: a design pattern for managing teams which leverage MLHuman-in-a-loop: a design pattern for managing teams which leverage ML
Human-in-a-loop: a design pattern for managing teams which leverage MLPaco Nathan
 
Humans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIHumans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIPaco Nathan
 
Humans in the loop: AI in open source and industry
Humans in the loop: AI in open source and industryHumans in the loop: AI in open source and industry
Humans in the loop: AI in open source and industryPaco Nathan
 
Computable Content
Computable ContentComputable Content
Computable ContentPaco Nathan
 
Computable Content: Lessons Learned
Computable Content: Lessons LearnedComputable Content: Lessons Learned
Computable Content: Lessons LearnedPaco Nathan
 
SF Python Meetup: TextRank in Python
SF Python Meetup: TextRank in PythonSF Python Meetup: TextRank in Python
SF Python Meetup: TextRank in PythonPaco Nathan
 
Data Science Reinvents Learning?
Data Science Reinvents Learning?Data Science Reinvents Learning?
Data Science Reinvents Learning?Paco Nathan
 
GraphX: Graph analytics for insights about developer communities
GraphX: Graph analytics for insights about developer communitiesGraphX: Graph analytics for insights about developer communities
GraphX: Graph analytics for insights about developer communitiesPaco Nathan
 
Apache Spark and the Emerging Technology Landscape for Big Data
Apache Spark and the Emerging Technology Landscape for Big DataApache Spark and the Emerging Technology Landscape for Big Data
Apache Spark and the Emerging Technology Landscape for Big DataPaco Nathan
 
A New Year in Data Science: ML Unpaused
A New Year in Data Science: ML UnpausedA New Year in Data Science: ML Unpaused
A New Year in Data Science: ML UnpausedPaco Nathan
 
Microservices, Containers, and Machine Learning
Microservices, Containers, and Machine LearningMicroservices, Containers, and Machine Learning
Microservices, Containers, and Machine LearningPaco Nathan
 
Databricks Meetup @ Los Angeles Apache Spark User Group
Databricks Meetup @ Los Angeles Apache Spark User GroupDatabricks Meetup @ Los Angeles Apache Spark User Group
Databricks Meetup @ Los Angeles Apache Spark User GroupPaco Nathan
 
What's new with Apache Spark?
What's new with Apache Spark?What's new with Apache Spark?
What's new with Apache Spark?Paco Nathan
 
Strata EU 2014: Spark Streaming Case Studies
Strata EU 2014: Spark Streaming Case StudiesStrata EU 2014: Spark Streaming Case Studies
Strata EU 2014: Spark Streaming Case StudiesPaco Nathan
 
Brief Intro to Apache Spark @ Stanford ICME
Brief Intro to Apache Spark @ Stanford ICMEBrief Intro to Apache Spark @ Stanford ICME
Brief Intro to Apache Spark @ Stanford ICMEPaco Nathan
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapePaco Nathan
 

More from Paco Nathan (18)

Human in the loop: a design pattern for managing teams working with ML
Human in the loop: a design pattern for managing  teams working with MLHuman in the loop: a design pattern for managing  teams working with ML
Human in the loop: a design pattern for managing teams working with ML
 
Human-in-the-loop: a design pattern for managing teams that leverage ML
Human-in-the-loop: a design pattern for managing teams that leverage MLHuman-in-the-loop: a design pattern for managing teams that leverage ML
Human-in-the-loop: a design pattern for managing teams that leverage ML
 
Human-in-a-loop: a design pattern for managing teams which leverage ML
Human-in-a-loop: a design pattern for managing teams which leverage MLHuman-in-a-loop: a design pattern for managing teams which leverage ML
Human-in-a-loop: a design pattern for managing teams which leverage ML
 
Humans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AIHumans in a loop: Jupyter notebooks as a front-end for AI
Humans in a loop: Jupyter notebooks as a front-end for AI
 
Humans in the loop: AI in open source and industry
Humans in the loop: AI in open source and industryHumans in the loop: AI in open source and industry
Humans in the loop: AI in open source and industry
 
Computable Content
Computable ContentComputable Content
Computable Content
 
Computable Content: Lessons Learned
Computable Content: Lessons LearnedComputable Content: Lessons Learned
Computable Content: Lessons Learned
 
SF Python Meetup: TextRank in Python
SF Python Meetup: TextRank in PythonSF Python Meetup: TextRank in Python
SF Python Meetup: TextRank in Python
 
Data Science Reinvents Learning?
Data Science Reinvents Learning?Data Science Reinvents Learning?
Data Science Reinvents Learning?
 
GraphX: Graph analytics for insights about developer communities
GraphX: Graph analytics for insights about developer communitiesGraphX: Graph analytics for insights about developer communities
GraphX: Graph analytics for insights about developer communities
 
Apache Spark and the Emerging Technology Landscape for Big Data
Apache Spark and the Emerging Technology Landscape for Big DataApache Spark and the Emerging Technology Landscape for Big Data
Apache Spark and the Emerging Technology Landscape for Big Data
 
A New Year in Data Science: ML Unpaused
A New Year in Data Science: ML UnpausedA New Year in Data Science: ML Unpaused
A New Year in Data Science: ML Unpaused
 
Microservices, Containers, and Machine Learning
Microservices, Containers, and Machine LearningMicroservices, Containers, and Machine Learning
Microservices, Containers, and Machine Learning
 
Databricks Meetup @ Los Angeles Apache Spark User Group
Databricks Meetup @ Los Angeles Apache Spark User GroupDatabricks Meetup @ Los Angeles Apache Spark User Group
Databricks Meetup @ Los Angeles Apache Spark User Group
 
What's new with Apache Spark?
What's new with Apache Spark?What's new with Apache Spark?
What's new with Apache Spark?
 
Strata EU 2014: Spark Streaming Case Studies
Strata EU 2014: Spark Streaming Case StudiesStrata EU 2014: Spark Streaming Case Studies
Strata EU 2014: Spark Streaming Case Studies
 
Brief Intro to Apache Spark @ Stanford ICME
Brief Intro to Apache Spark @ Stanford ICMEBrief Intro to Apache Spark @ Stanford ICME
Brief Intro to Apache Spark @ Stanford ICME
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscape
 

Recently uploaded

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 

Recently uploaded (20)

20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Big Data is changing abruptly, and where it is likely heading

  • 1. Big Data is changing abruptly, and where it is likely heading Big Data Spain 2014-11-17 bigdataspain.org Paco Nathan @pacoid
  • 2. Some questions, and an emerging idea… The economics of datacenters has shifted toward warehouse scale, based on commodity hardware that includes multicore and large memory spaces. ! Incumbent technologies for Big Data do not embrace those changes, however newer OSS projects have emerged to leverage them. 2
  • 3. Some questions, and an emerging idea… Also, the data requirements are changing abruptly as sensor data, microsatellites, and other IoT use cases boost data rates by orders of magnitude. ! We have math available to address those industrial needs, but have software frameworks kept up? 3
  • 5. If you’re not using them, you’re missing out… ! See recent announcements by Google, Amazon, etc. ! Containers are the tip of the iceberg: OS have evolved quietly, while practices near to customers have perhaps not kept pace 5 Containers
  • 6. Key benefits based on Mesos case studies: • mixed workloads • higher utilization rates • lower latency for data products • significantly lower Ops overhead ! Examples: Mesos @Twitter, eBay, Netflix, many startups, etc.; Borg/Omega @Google, Symphony @IBM, Autopilot @Microsoft, etc. ! Major cloud users understand this shift; however much of the world still thinks in terms of VMware 6 Containers
  • 7. We need much more than Docker, we need better definitions for what is to be used outside of the container as well – e.g., for microservices • Intro to Mesos: goo.gl/PbUfNe • https://mesosphere.com/ • A project to watch: Weave 7 Containers
  • 8. 8 Containers One problem: how does an institution, such as a major bank, navigate this kind of change?
  • 10. Speaking of clouds, forgive me for being biased… For nearly a decade, I’ve been talking with people about whether cloud economics makes sense? Consider: datacenter util rates in single-digits (~8%) 10 Clouds
  • 11. Companies accustomed to cheap energy rates, with staff who thinking in terms of individual VMs… these will not find clouds to be a friendly territory. Not so much about software, as it is about sophisticated engineering practice. The change recalls the shift of BI ⇒ Data Science That was not simple… 11 Clouds
  • 12. The majors understand this new world: Google, Amazon, Microsoft, IBM, Apple, etc. Sub-majors can create their own clouds: Facebook, Twitter, LinkedIn, etc. 12 Clouds What about industries approaching Big Data now which cannot afford to buy small armies of expert Ops people? How they must proceed becomes a key question.
  • 13. Current research points to significant changes ahead, where machine learning plays a key role in the context of advanced cluster scheduling: 13 Clouds Improving Resource Efficiency with Apache Mesos Christina Delimitrou youtu.be/YpmElyi94AA Experiences with Quasar+Mesos showed: • 88% apps get >95% performance • ~10% overprovisioning instead of 500% • up to 70% cluster util at steady state • 23% shorter scenario completion
  • 15. A question on StackOverflow in 2013 challenged Twitter’s use of Abstract Algebra libraries for their revenue apps… cs.stackexchange.com/questions/9648/what-use-are-groups- monoids-and-rings-in-database-computations The answers are enlightening: think of a kind of containerization for business logic – leading to exponential increase in performance, particularly in the context of streaming cases. 15 Abstract Algebra
  • 16. Abstract Algebra A cheat-sheet: non-empty set Semigroup has an identity element Group has a binary associative operation, with closure each element has an inverse has two binary associative operations: addition and multiplication Ring Monoid
  • 17. Add ALL the Things: Abstract Algebra Meets Analytics infoq.com/presentations/abstract-algebra- analytics Avi Bryant, Strange Loop (2013) • grouping doesn’t matter (associativity) • ordering doesn’t matter (commutativity) • zeros get ignored In other words, while partitioning data at scale is quite difficult, you can let the math allow your code to be flexible at scale Avi Bryant @avibryant Abstract Algebra
  • 18. Algebra for Analytics speakerdeck.com/johnynek/ algebra-for-analytics Oscar Boykin, Strata SC (2014) • “Associativity allows parallelism in reducing” by letting you put the () where you want • “Lack of associativity increases latency exponentially” Oscar Boykin @posco Abstract Algebra
  • 20. Functional Programming for Big Data Theory, eight decades ago: what can be computed? Haskell Curry haskell.org Alonso Church wikipedia.org Praxis, four decades ago: algebra for applicative systems John Backus acm.org David Turner wikipedia.org Reality, two decades ago: machine data from web apps Pattie Maes MIT Media Lab 20
  • 21. Functional Programming for Big Data circa 2002: mitigate risk of large distributed workloads lost due to disk failures on commodity hardware… Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html ! MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html 21
  • 22. Functional Programming for Big Data DryadLINQ influenced a new class of workflow abstractions based on functional programming: Spark, Flink, Scalding, Scoobi, Scrunch, etc. • needed for leveraging the newer hardware: multicore, large memory spaces, etc. • significant code volume reduction ⇒ eng costs 22
  • 23. Functional Programming for Big Data 2002 2004 MapReduce paper 2002 MapReduce @ Google 2004 2006 2008 2010 2012 2014 2006 Hadoop @ Yahoo! 2014 Apache Spark top-level 2010 Spark paper 2008 Hadoop Summit 23
  • 24. Functional Programming for Big Data MapReduce Pregel Giraph Dremel Drill S4 Storm F1 MillWheel General Batch Processing Specialized Systems: Impala GraphLab iterative, interactive, streaming, graph, etc. Tez MR doesn’t compose well for large applications, and so specialized systems emerged as workarounds 24
  • 25. Functional Programming for Big Data circa 2010: a unified engine for enterprise data workflows, based on commodity hardware a decade later… Spark: Cluster Computing with Working Sets Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf ! Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf 25
  • 26. Functional Programming for Big Data One minor problem: where do we find many graduating students who are good at Scala, Clojure, F#, etc.? 26
  • 28. +1 generations of computer scientists has grown up thinking that Data implies SQL… Excellent as a kind of declarative, functional language for describing data transformations However, in the long run we should move away from thinking in terms of SQL Computational Thinking as a rubric in general would be preferred 28 Databases
  • 29. A step further is to move away from thinking of data as an engineering issue – it is about business So much of the US curricula for math is still stuck in the Cold War: years of calculus to identify the best people to build ICBMs We need more business people thinking in terms of math, beyond calculus, to contend with difficult problems in supply chain, maintenance schedules, energy needs, transportation routes, etc. 29 Databases
  • 30. Databases 1. starting with real-world data ⇒ 2. leverage graph queries for representation ⇒ 3. convert to sparse matrix, use FP and abstract algebra ⇒ 4. achieve high-ROI parallelism at scale, mostly about optimization ⇒
  • 32. Approximations 19-20c. statistics emphasized defensibility in lieu of predictability, based on analytic variance and goodness-of-fit tests ! That approach inherently led toward a manner of computational thinking based on batch windows ! They missed a subtle point… 32
  • 33. 21c. shift towards modeling based on probabilistic approximations: trade bounded errors for greatly reduced resource costs highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 33 Approximations
  • 34. 21c. shift towards modeling based on probabil approximations: trade bounded errors for greatly reduced resource costs Twitter catch-phrase: “Hash, don’t sample” highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 34 Approximations
  • 35. a fascinating and relatively new area, pioneered by relatively few people – e.g., Philippe Flajolet provides approximation, with error bounds – in general uses significantly less resources (RAM, CPU, etc.) many algorithms can be constructed from combinations of read and write monoids aggregate different ranges by composing hashes, instead of repeating full-queries 35 Approximations
  • 36. algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 36 Approximations
  • 37. algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code suggestion: consider these as your most quintessential collections data types at scale Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 37 Approximations
  • 38. • sketch algorithms: trade bounded errors for orders of magnitude less required resources, e.g., fit more complex apps in memory • multicore + large memory spaces (off heap) are increasing the resources per node in a cluster • containers allow for finer-grain allocation of cluster resources and multi-tenancy • monoids, etc.: guarantees of associativity within the code allow for more effective distributed computing, e.g., partial aggregates • less resources must be spent sorting/windowing data prior to working with a data set • real-time apps, which don’t have the luxury of anticipating data partitions, can respond quickly 38 Approximations
  • 39. Probabilistic Data Structures for Web Analytics and Data Mining Ilya Katsov (2012-05-01) A collection of links for streaming algorithms and data structures Debasish Ghosh Aggregate Knowledge blog (now Neustar) Timon Karnezos, Matt Curcio, et al. Probabilistic Data Structures and Breaking Down Big Sequence Data C. Titus Brown, O'Reilly (2010-11-10) Algebird Avi Bryant, Oscar Boykin, et al. Twitter (2012) Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman, Cambridge (2011) 39 Approximations
  • 41. Spreadsheets represented a revolution in how to think about computing ! What Fernando Perez, et al., have done with IPython is subtle and powerful ! Technology is not just about writing code; it is about people using that code ! Enterprise data workflows are moving toward cloud-based notebooks 41 Notebooks
  • 42. Early emphasis on tools in Big Data now gives way to workflows: people, automation, process, integration, test, maintenance, scale Google had the brilliant approach of shared documents… Now we see cloud-based notebooks, which begin to displace web apps 42 Notebooks http://nbviewer.ipython.org/ http://jupyter.org/
  • 45. What is Spark? Developed in 2009 at UC Berkeley AMPLab, then open sourced in 2010, Spark has since become one of the largest OSS communities in big data, with over 200 contributors in 50+ organizations spark.apache.org “Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model. It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.” Gartner, Advanced Analytics and Data Science (2014)
  • 47. What is Spark? Spark Core is the general execution engine for the Spark platform that other functionality is built atop: ! • in-memory computing capabilities deliver speed • general execution model supports wide variety of use cases • ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R)
  • 48. What is Spark? WordCount in 3 lines of Spark WordCount in 50+ lines of Java MR
  • 49. What is Spark? Sustained exponential growth, as one of the most active Apache projects ohloh.net/orgs/apache
  • 51. • generalized patterns ⇒ unified engine for many use cases • lazy evaluation of the lineage graph ⇒ reduces wait states, better pipelining • generational differences in hardware ⇒ off-heap use of large memory spaces • functional programming / ease of use ⇒ reduction in cost to maintain large apps • lower overhead for starting jobs • less expensive shuffles 51 What is Spark?
  • 52. What is Spark? Kafka + Spark + Cassandra datastax.com/documentation/datastax_enterprise/4.5/ datastax_enterprise/spark/sparkIntro.html http://helenaedelson.com/?p=991 github.com/datastax/spark-cassandra-connector github.com/dibbhatt/kafka-spark-consumer unified compute data streams columnar key-value
  • 54. What is Spark? Databricks Cloud databricks.com/blog/2014/07/14/databricks-cloud-making- big-data-easy.html youtube.com/watch?v=dJQ5lV5Tldw#t=883
  • 56. certification: Apache Spark developer certificate program • http://oreilly.com/go/sparkcert • defined by Spark experts @Databricks • assessed by O’Reilly Media • establishes the bar for Spark expertise 56
  • 57. community: spark.apache.org/community.html video+slide archives: spark-summit.org events worldwide: goo.gl/2YqJZK resources: databricks.com/spark-training-resources workshops: databricks.com/spark-training new: MOOCs via edX and University of California 57
  • 58. books: Fast Data Processing with Spark Holden Karau Packt (2013) shop.oreilly.com/product/ 9781782167068.do Spark in Action Chris Fregly Manning (2015*) sparkinaction.com/ Learning Spark Holden Karau, Andy Konwinski, Matei Zaharia O’Reilly (2015*) shop.oreilly.com/product/ 0636920028512.do 58
  • 59. events: Strata EU Barcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east Strata EU London, May 5-7 strataconf.com/big-data-conference-uk-2015 Spark Summit 2015 SF, Jun 15-17 spark-summit.org 59
  • 60. presenter: monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/ Just Enough Math O’Reilly, 2014 justenoughmath.com preview: youtu.be/TQ58cWgdCpA Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do 60