In IT terms, the concept of Big Data has been around for a very long time. But when it was first coined in the ancient past (around 2005) no one had any real idea of just how much raw data we\u2019d end up talking about. A report by Statista published in June 2021 states that in 2020, the total amount of data created, captured, copied, and consumed globally reached 64.2 zettabytes, and predicts this figure will exceed 180 zettabytes by 2025 (one zettabyte being equal to a billion terabytes).<\/p>\r\n\r\n\r\n\r\n
But as we all know, data is only as useful as the insights you can extract from it. Within organisations, data is often stored in different forms and locations, so lacks the context and meaning required for decision-making. To make sense of it, you need to connect data sources and processing services, then move the data to a central location where it can be extracted, transformed and loaded back into platforms \u2013 the familiar ETL process.<\/p>\r\n\r\n\r\n\r\n
Until fairly recently, this meant custom-building integrations solutions. Then in 2016, Microsoft launched its cloud-based data integration service, Azure Data Factory (ADF). This allowed data engineers to create \u2018pipelines\u2019, time-sliced on a specified schedule, for connecting and collecting data, transforming and enriching it, then publishing it in the required form.<\/p>\r\n\r\n\r\n\r\n
The first version of ADF was widely known in the industry \u2013 and quietly acknowledged as such by Microsoft itself \u2013 to be a \u2018first run\u2019 at the concept. Many organisations were understandably hesitant about adopting it, knowing (or at least assuming) it would be superseded. And now, it has been.<\/p>\r\n\r\n\r\n\r\n
ADF version 2 isn\u2019t just an update of v1: it\u2019s a totally different animal.<\/p>\r\n\r\n\r\n\r\n
We\u2019ve been working with ADF since its inception and v2 does everything its predecessor does \u2013 just much, much better. In our view, these are the key benefits and, if you\u2019ve been holding back, why it could now be the right solution for your organisation.<\/p>\r\n\r\n\r\n\r\n
In its latest iteration (V2) ADF is a fully cloud-based ETL and Data Integration service, ideal for users and businesses working with big data in a multi-platform environment. It\u2019s highly scalable and cost-effective \u2013 and being code-free within Microsoft\u2019s public cloud, it\u2019s simple to set up and use. Like V1, it\u2019s based around the central principle of Pipelines; units of \u2018work\u2019 that can be logically grouped together into Activities and then executed by scheduled or event-based Triggers. Both inputs and outputs are referred to as Datasets \u2013 \u2018containers\u2019 of data which can be traditional databases or data warehouses. Data Flow, Data Movement and Activity Dispatch form the integration infrastructure, acting as a \u2018bridge\u2019 between Activities and linked services.<\/p>\r\n\r\n\r\n\r\n
ADF really comes into its own in businesses producing individual datasets from different, discrete platforms across a number of departments. Extracting data from these multiple platforms, Transforming it into a meaningful form and then Loading it onto a platform can provide far more powerful, complete and cohesive analytics and insights, increasing efficiency and driving growth.<\/p>\r\n\r\n\r\n\r\n
We\u2019ve worked with both cloud-native clients and those making the transition to cloud-based working to deploy and realise the benefits of ADF v2, at all scales and across a wide range of sectors. To find out more and see how ADF could work for you, please get in touch.<\/p>\r\n","protected":false},"excerpt":{"rendered":"
In IT terms, the concept of Big Data has been around for a very long time. But when it was first coined in the ancient past (around 2005) no one had any real idea of just how much raw data…<\/p>\n","protected":false},"author":24,"featured_media":11947,"comment_status":"closed","ping_status":"open","sticky":false,"template":"full-width-single-post.blade.php","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-11946","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-archive"],"acf":[],"yoast_head":"\n