How to use components of Microsoft Azure for data solutions
Microsoft Azure cloud platform provides a set of tools and technologies to address a holistic data engineering, analytics, and data management solution for enterprises. The components for data solutions are integrated to provide end-to-end data solutions for an enterprise.
It is a collaborative platform for data engineers, data scientists, and business users. The following provides an overview of these components and the purpose they serve.
Azure Data Factory
Azure Data Factory is a cloud service for ETL /ELT (Extract-Transform-Load / Extract-Load-Transform). The data sources for pulling data encompass both on-premises and cloud, and can be from disparate sources across relational, non-relational, structured, unstructured, or semi-structured data. Azure data factory allows creating workflows (also called pipelines) to automate the process of pulling the data and storing in cloud in a centralized data store.
Azure Blob Storage
Once data is ingested thru Azure Data Factory, it can be stored in Azure Blob Storage. The storage capacity is extremely large and can store unstructured data including log files, text, image, and video.
Use Azure Databricks to clean data and prepare to run analytics and machine learning to derive insights. The underlying Apache Spark based platform has inbuilt machine learning algorithms library (MLLib) to perform classification, clustering, and regression among others. Spark Core API allows using languages like R, Scala, SQL, and Python for data exploration, preparation, and analytics.
Azure SQL Data Warehouse
This enterprise data warehouse can store large amount of data (in petabytes) and process complex queries and run analytics on the data. Processing is much faster than traditional on-premises data warehouses. Needless to say, the cost of implementations along with maintenance for traditional data warehouses has been very high.
Azure SQL Data Warehouse solves this problem. Integration with Azure Data Factory, Blob Storage, Databricks, and other components in Azure Data Solutions helps to easily and efficiently move data across these components.
Azure Power BI
This cloud-based business intelligence and analytics service provides sophisticated visualization, reports, and dashboards on the analyzed data.
