How to solve your data engineering roadblock: Key takeaways from Apex Assembly NYC

At the latest APEX Assembly, our CEO, Ju-kay Kwek, took part in a fireside chat in which he described how Switchboard built an end-to-end data-driven solution that helped the US’ largest digital and print publisher realize a seven-figure revenue.

But our approach to data unification doesn’t stop at publishing – Switchboard helps companies across a wide range of sectors harness their ever-increasing data sets. Let’s take a look at the key takeaways of the session to find out how.

Say ‘yes’ to business

Dotdash Meredith’s portfolio includes well-known publications across a number of verticals such as health, beauty, food and drink, finance, travel and technology. In total, the company was generating about two billion rows of data every day across all of its assets, which could amount to 100 terabytes per month. Its acquisition of Time Inc (and subsequent 200% increase in data volume), as well as the sale of its radio and TV divisions, added to the complexity of the company’s needs.

In the modern commercial environment, we believe there is a need for a dynamic approach: say ‘yes’ to business changes and manage the processes as needed. Data should be organized around a company’s needs, rather than the other way around. However, this dynamic approach does come with challenges: 1) large and disparate data; and 2) governance and security.

• Large and disparate data: Data must be defined, robust, and repeatable at source. Although Dotdash Meredith had already standardized their operations in cloud data warehouses, their data was spread over several of these resources, including Snowflake, BigQuery, and AWS. Their analysts knew how to plug in BI tools, such as Tableau, but ultimately, they didn’t trust the data they were using. As the company attempted to unify its data, each new business team brought more data silos. Since their data architecture was so disparate and complex, mastering it required a high degree of data engineering and IT.

• Maintain governance and security: Naturally, security is paramount. Regulations must be respected and data governance upheld. But securing distributed data architectures became increasingly difficult with each data source that was added. Since the business analytics team needed to be hands-on with the data, there was a risk of creating ‘shadow IT’ (using computer systems that lack approval or security vetting). Therefore, any solution would require transparency for the IT team, in addition to data compliance.

Like many data-driven organizations, there is a reliance on Revenue Operations teams, who are highly skilled in data analytics and manipulating spreadsheets. But it’s not their job to write code or manage large datasets. So, both the scale of data involved and the security compliance put pressure on the company’s already limited engineering resources. Switchboard was able to provide this expertise, working with Revenue Operations and the business teams to create a bespoke solution for their needs.

Automating big data

Rather than attempting to build their own CRM system, today we are seeing more and more teams use a ready-made cloud-based platform. In the same vein, Switchboard is essentially a cloud-hosted enterprise platform for data automation and operations. Dotdash Meredith’s Revenue Operations team implemented Switchboard to bring all of their disparate data sources under one roof.

Switchboard has baked in large-scale data automation techniques which automate continuous and real-time data processing, such as ingestion, aggregation, and normalization. Crucially, our platform enables the business analytics team to control this process (while lifting the burden from the technical team), functioning as a management, automation, and governance solution for all.

Automation – a win-win for Rev Ops

Switchboard’s solution provided Dotdash Meredith with a means to generate a phenomenal uplift in revenue, but what about the timescale? We estimate that building their own solution would have taken about a year to complete. In contrast, the Switchboard implementation and onboarding took approximately eight weeks, and didn’t require hiring any data engineers.

WIth Switchboard, the road ahead became clear: not only did the relevant teams get access to trustworthy data – but they were able to start using it to drive serious revenue from the outset.