Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions Architectures - Cloud and big data.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Big Data Architectures

## Big Data and Analytics Overview
- [Four V's](http://www.ibmbigdatahub.com/infographic/four-vs-big-data)
+ Volume - Scale of data
+ Variety - Different forms of data
+ Velocity - Analysis of streaming data
+ Veracity - Uncertainty of data
- Additional V's
+ Visualization
+ Value
+ Variability
- Considerations
- Data batching vs. streaming
- Embedded models or interfaces
- API or RPC or REST
- Deployed trained models (batch) vs. [real-time/streaming/online machine learning](https://en.wikipedia.org/wiki/Online_machine_learning)
- Reliable quality data
- Solution elements
+ Data sources: real time measurement, events, IoT, and so on
+ Raw data extraction, collection, and ingestion
+ Data preparation, processing, cleaning, normalization, and standardization
+ Data transformation and aggregation
+ Data storage
+ Data comsumption, visualization, and analysis
+ Prediction and prescription
+ Asynchronous messaging
+ Actionable insights
+ Deployed model monitoring and performance measurement
- Data Analytics
+ Descriptive analytics - What happened and why?
+ Predictive analytics - What is the probability of something happening?
+ Prescriptive analytics - What specific recommendations will drive business decisions and help achieve business goals (i.e., what to do if 'X' happens)

## AWS
- [Solution Development Guides](https://aws.amazon.com/solutions)
+ Reference Architectures
* [Web Application Hosting](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_web_01.pdf)
* [Batch Processing](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_batch_03.pdf)
* [Large Scale Computing and Large Data Sets](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_largescale_05.pdf)
* [Time Series Processing](http://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_timeseriesprocessing_16.pdf)
+ By Application
* [Websites](https://aws.amazon.com/websites/)
* [Backup and Recovery](https://aws.amazon.com/backup-recovery/)
* [Archiving](https://aws.amazon.com/archive/)
* [DevOps](https://aws.amazon.com/devops/)
* [Big Data](https://aws.amazon.com/big-data/)
* [High Performance Computing](https://aws.amazon.com/hpc/)
* [Internet of Things](https://aws.amazon.com/iot/)
* [Business Applications](https://aws.amazon.com/business-applications/)
* [Content Delivery](https://aws.amazon.com/content-delivery/)
* [Mobile Services](https://aws.amazon.com/mobile/)
* [Scientific Computing](https://aws.amazon.com/government-education/scientific-computing1/)
* [E-commerce](https://aws.amazon.com/ecommerce-applications/)
+ By Industry Sector
* [Financial Services](https://aws.amazon.com/financial-services/)
* [Digital Marketing](https://aws.amazon.com/digital-marketing/)
* [Media and Entertainment](https://aws.amazon.com/digital-media/)
* [Gaming](https://aws.amazon.com/game-hosting/)
* [Enterprise IT](https://aws.amazon.com/enterprise/)
* [Healthcare & Life Sciences](https://aws.amazon.com/health/)
* [Government](https://aws.amazon.com/government-education/government/)
* [Nonprofit](https://aws.amazon.com/government-education/nonprofits/)
* [Education](https://aws.amazon.com/education/)
- [Partner Solutions](https://aws.amazon.com/solutions/partners/)
+ [Big Data](https://aws.amazon.com/big-data/partner-solutions/)
+ [Storage](https://aws.amazon.com/backup-recovery/partner-solutions/)
+ [DevOps](https://aws.amazon.com/devops/partner-solutions/)
- [Case Studies](https://aws.amazon.com/solutions/case-studies/)
+ [Analytics](https://aws.amazon.com/solutions/case-studies/analytics/)
+ [Big Data](https://aws.amazon.com/solutions/case-studies/big-data/)
+ [Enterprise](https://aws.amazon.com/solutions/case-studies/enterprise-it/)
+ [Startups](https://aws.amazon.com/solutions/case-studies/start-ups/)
+ [Web Apps](https://aws.amazon.com/solutions/case-studies/web-mobile-social/)

## Google Cloud Platform
- [Big data reference architecture diagram](https://cloud.google.com/images/products/big-data/big-data-diagram.png)
- [Solution Development Guides](https://cloud.google.com/solutions/)
+ [Media](https://cloud.google.com/solutions/media/)
+ [Mobile Applications](https://cloud.google.com/solutions/mobile/#development_guides)
+ [Big Data](https://cloud.google.com/solutions/big-data/#development_guides)
+ [Financial Services](https://cloud.google.com/solutions/financial-services/#development_guides)
+ [Gaming](https://cloud.google.com/solutions/gaming/#development_guides)
+ [Retail & Commerce](https://cloud.google.com/solutions/commerce/#development_guides)
+ [Internet of Things](https://cloud.google.com/solutions/iot/#development_guides)
+ [Websites and Web Apps](https://cloud.google.com/solutions/websites/#development_guides)
+ [Development & Test](https://cloud.google.com/solutions/dev-test/#development_guides)


## General
- [Lambda Architecture](http://lambda-architecture.net/)
- [AWS Architecture Center](https://aws.amazon.com/architecture/?nc1=f_cc)
- [AWS Big Data Partner Solutions](https://aws.amazon.com/big-data/partner-solutions/)
- [GCP Architecture](https://cloud.google.com/docs/tutorials#architecture)
- [Introduction to big data classification and architecture](http://www.ibm.com/developerworks/library/bd-archpatterns1/)
- [An Enterprise Architect’s Guide to Big Data](http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf)
- [BIG DATA REFERENCE ARCHITECTURE](https://thinkbiganalytics.com/leading_big_data_technologies/big-data-reference-architecture/)
- [Getting Started with Big Data Architecture](http://blog.cloudera.com/blog/2014/09/getting-started-with-big-data-architecture/)
- [BIG DATA: Architectures and Technologies](https://www.sei.cmu.edu/go/big-data/)
- [Big Data Architecture](http://bigdata.teradata.com/US/Big-Ideas/Big-Data-Architecture/)
- [Big Data Analytics Architecture](http://www.thebigdatainsightgroup.com/site/sites/default/files/Teradata's%20-%20Big%20Data%20Architecture%20-%20Putting%20all%20your%20eggs%20in%20one%20basket.pdf)
- [What is Streaming Data?](https://aws.amazon.com/streaming-data/)
Loading