Accessing CDN logs has become much more straightforward in recent years, with most major CDNs offering programmatic access to their log data. This enables streaming operators to integrate CDN logs into platforms like New Relic or Splunk for advanced analysis. However, the immense size of CDN logs can make ingestion and processing time-consuming. In multi-CDN environments, this challenge grows even more complex due to the lack of standardisation in CDN log formats. Each CDN structures its logs differently, creating additional hurdles for operators trying to diagnose delivery-related issues. Despite these challenges, CDN log data is a goldmine of insights into streaming performance and viewer QoE, making it essential for operators to develop strategies to streamline and maximise its usage.
More data, more complexity: challenges in optimising CDN logs
Including more CDN log data might seem like it would lead to greater insights and a better understanding of streaming performance. However, challenges such as normalisation, standardisation, and data volume can make that assumption incorrect. In multi-CDN architectures, more data often increases complexity and delays the time needed to extract meaningful insights. For real-time events, such as live sports, any delay in data analysis can result in higher viewer dissatisfaction and even lead to churn. Operators need a way to deal with a set of core challenges related to making the most effective and efficient use of CDN log data:
Ingestion
First and foremost, CDN logs must be accessible programmatically. By leveraging API ingestion, operators can integrate CDN log data into their operational workflows through automation. Operators should never have to manually request logs. However, during live events, ingestion can be delayed due to stress on the CDN network, including API rate limiting, as logs are generated and delivered in real time.
Normalisation
When pulling log data from multiple CDNs, normalisation is essential before the data can be used in dashboards or other tools. This process, though, can be time-consuming, especially for large datasets. While less of a technical hurdle and more of an operational one, normalisation must also be automated. Instead of processing data after ingestion, logs should be normalised to an internal data dictionary before reaching internal storage systems.
Enrichment
Log data, by itself, is interesting but not the whole story. In many cases, KPIs (which can be a combination of multiple data points) need to be computed. But carrying out those computations through some business logic layer that’s part of a visualisation dashboard can add delays to seeing the data. Enrichment can add a lot of value and increase the usefulness of data but it can’t add delays to being able to act on the data.
Storage
In a multi-CDN architecture, storage challenges escalate quickly. The issue isn’t just storing vast amounts of data; it’s also about effectively using the stored data. As data volume grows, analysis slows down due to the increasing time required for indexing. Operators must balance having data readily available in real-time while maintaining long-term storage for historical insights.
Querying
As with indexing, querying becomes more complex as the data pool expands. Operators want granular insights into CDN delivery and performance issues. But if queries take minutes instead of seconds, they cannot address viewer QoE issues in real-time, ultimately impacting the viewing experience.
Visualisation
The ultimate goal of any data workflow for a streaming operator is actionable insights presented in a dashboard. However, delays in normalisation, indexing, and querying cascade into visualisations. Dashboards may populate minutes after data ingestion instead of seconds, preventing operations teams from solving issues in real-time.
Every challenge outlined above ties back to cost. As data grows, so do storage expenses. Longer query times increase CPU usage, driving up processing costs. Most critically, failing to resolve QoE issues in real-time—due to delays in working with CDN log data—can lead to viewer churn, which carries the most significant financial impact on the business.
Making CDN log data analysis real-time: strategies for efficiency and cost optimisation
Thankfully, there are effective strategies to solve the challenges that add time and cost when handling large volumes of CDN log data.
Strategy #1: Data subsets
First, operators should focus on data subsets. Key fields that contribute to a single metric should be organised into a separate table. For example, if a specific metric indicates a potential performance issue, the faster an operator can identify the problem, the sooner it can be addressed—before it affects viewer QoE. Using data subsets like this helps improve indexing and query performance, allowing graphs and dashboards to load more quickly.
Strategy #2: Data sampling
Next, operators must recognise that not all data is necessary for real-time analysis. Continuing with the data subset concept, suppose an operator creates a subset based on buffer ratio, which is directly related to potential CDN issues. The operator collects real-time data for the subset while storing the rest of the data in long-term storage for later analysis. The subset only represents every 10th session, allowing for faster analysis in real time. When the metric exceeds a predefined threshold, the system automatically switches to pulling all relevant sessions, giving the operator deeper insight and enabling immediate action.
Strategy #3: De-couple data storage and reporting
Operators should separate data storage from reporting (visualisation). Often, dashboards are directly connected to the full CDN log data pool, but this can be inefficient. By embracing data subsets, the CDN log storage can be separated from the reporting layer, with dashboards drawing from smaller, real-time data pools. This approach reduces costs by minimising compute cycles required for indexing and ensures that active storage—more expensive than long-term storage—holds only the most relevant data.
Strategy #4: Create a real-time CDN data pool
Ultimately, streaming operators should aim to create a real-time CDN data pool. This pool is a subset of the complete logs, aggregating data from multiple vendors and focusing only on the key data points needed for operational metrics. Initially, the data pool functions as a sample, only fully normalising records when necessary for further analysis. The entire process should be automated, optimised for performance, and cost-effective.
To make this strategy work, you’ll need to take a different approach to ingestion. Even with API ingestion, you might think of just importing logs into individual tables. But, when analysing data across CDNs, such as for the sake of comparing performance, individual tables need to be connected via complex JOIN statements which can impact query performance. By ingesting all CDN logs into a single table (appending an identifier, through the API call for example, to separate out sources), analysis can happen much quicker and more simply.
Strategy #5: Split log data into individual columns and compute at storage
To make CDN data as useful as possible, it’s key to break it apart into individual columns. CDN log lines include valuable context in the form of URIs, which provide information about paths, query parameters, and more. In this example, you could extract parameters from a URI query string into separate fields. By dividing elements, you can increase the speed at which queries can run (as you aren’t having to parse arrays of text or long text strings). And there’s no need to wait to compute specific KPIs. Just like dividing elements from log lines into individual columns, you can create columns for computing values which makes it easier than to compare KPIs in multi-source views in visualisation dashboards.
CDN log handling doesn’t have to be as difficult (or costly) as it is
CDN logs are essential for streaming operations and, fortunately, they are programmatically accessible, enabling automated ingestion into operational workflows. However, using large volumes of CDN log data in real-time presents several challenges. Despite these obstacles, there are effective strategies to enhance query and indexing performance while reducing storage costs.
To learn more, watch the workshop Processing Video CDN Logs at Scale in a Cost-Effective Way where streaming operators and leading vendors share insights into technologies you can implement today to optimise how you handle and utilise CDN log data.