Generation data is a critical component for traders within short term power markets in Europe. When plant operators and TSOs fail to transmit complete data on load, outages or other critical information traders face significant challenges. It creates problems for market participants when trying to forecast prices and supply & demand that hinder decision making processes.

Under the Transparency Regulation and associated ENTSO-E rules, TSOs are required to report data on unavailability of generation / production units (planned outages, forced outages) as part of “generation” datasets. However, on the ENTSO-E Transparency Platform, in the “Actual Generation per Production Type” view, many entries are tagged “n/e” (not existing / not available) at certain hours for certain production types / bidding zones leaving consumers of this data guessing at the values. 

During a breakdown, fallback or default, zero reporting happens as well - which leaves market participants scrambling to fill these missing values. An incident occurred in 2024 where there was a known data outage incident affecting Germany: Amprion (a German TSO) reportedly could not provide solar generation data for certain hours, and temporarily published 0 values in lieu of the true data. 

Consider another real example below collected by Kpler. Data for French power demand between April 13 and April 15, as published by the French TSO and reported by ENTSO-E contained missing values indicated in the redline below.

Without a reliable mechanism to detect and correct these gaps, market participants are stuck working with flawed inputs that can distort outcomes and delay action. 

Kpler Power delivers forecasts for intraday and short term availability and generation, in addition to prices, for European markets. The product scrapes over 100 public time series to produce downstream forecasting analytics. While most other data & analytics providers in the space simply provide data with gaps, Kpler has developed a special mechanism by which these gaps in data can be identified and missing values can be replaced with the most accurate value. Our users need continuous time series with short publication delays. 

With Kpler Platinum, a pipeline built by Kpler to identify outliers and missing values in time series, data with the most accurate replacement value is always provided for downstream forecasts so that customers never miss key information.

As part of Kpler’s offering, we provide forecasting engines for most time series available. The platinum pipeline uses those engines to impute data where missing or not reliable. Let’s take a look at how the feature was built to demonstrate how it is currently solving real problems for Kpler Power customers. 

Methodology

The high level Platinum pipeline is captured in the diagram below:

1. Detector

The detection step makes use of adtk pipelines. ADTK stands for Anomaly Detection Tool Kit — it’s an open-source Python library designed for unsupervised anomaly detection in time series data.

An ADTK pipeline usually refers to a sequence of steps one builds using ADTK’s components to automatically:

1. Preprocess time series
   
   • Cleaning, resampling, and filling missing values.
     
   • Applying rolling statistics or transformations.
     
2. Detect anomalies
   
   • Using built-in detectors such as:
     
     • ThresholdAD (flag values above/below thresholds)
       
     • LevelShiftAD (detect mean shifts)
       
     • VolatilityShiftAD (detect variance shifts)
       
     • AutoregressionAD (spot deviations from AR models)
       
     • QuantileAD (flag outliers based on quantiles).
       
3. Postprocess results
   
   • Filtering out spurious anomalies, combining multiple detectors, or applying rules to reduce false positives.
     
4. Integrate into workflows
   
   • Results can feed into data quality pipelines (like detecting missing or unreliable plant transmission data, as in your Kpler use case), or into monitoring dashboards and forecasting systems.

This adtk pipeline enables Kpler to chain complex detection algorithms. We have one pipeline per time series, unlocking a great level of modularity and ensuring that each time series is treated independently. A detection pipeline might look like the below:

Machine Learning Models

Kpler uses machine learning models to predict each series that we correct, meaning we can forecast any value in the past that we consider unreliable and impute with an insample forecast. These ML models are trained every day and rely on exogenous factors, such as the weather, to predict power demand. They don’t learn on outliers so they are able to predict realistic synthetic values in a sanitised way for past outlier data points.

A Concrete Example: Italian Solar Production

Since the beginning of 2025, Italian solar production reporting has been error-prone, as illustrated in the below graph. One can see that there are large spikes in the value for generation in the days following May 24. These do not represent actual load, rather they are erroneous figures from the reporting:

Based on our detection pipelines, Kpler is capable of detecting abnormal values like these and running our models to address these issues in order to expose clean data through the API and the frontend for our customers. One can see in the updated graph below that Kpler’s Platinum pipeline actively found and addressed this issue, with the corrected values indicated in red, preventing poor data from being consumed by any customers. 

Conclusion

European power markets demand precision, yet publicly reported data from ENTSO-E is often incomplete or delayed. These imperfections create significant downstream challenges for forecasting models that depend on continuous, high-quality inputs. Kpler addresses this structural problem with a systematic, technology-driven approach that ensures both accuracy and continuity.

At the core of this solution is the Platinum pipeline, which integrates anomaly detection techniques with advanced machine learning models. Using ADTK-based detection pipelines, Kpler can independently tailor anomaly detection for each time series, applying methods such as thresholding, volatility shifts, and autoregression checks. Once irregularities are detected, daily retrained ML models—enriched with exogenous variables such as weather—generate realistic and sanitized replacement values. This process guarantees that even historical outliers or missing data points are imputed with statistically robust forecasts rather than ad hoc estimates.

The result is a resilient framework capable of maintaining clean, reliable time series across more than 100 public datasets. Concrete examples, such as correcting French demand misreporting or stabilizing Italian solar generation data, illustrate how Kpler ensures that erroneous spikes or reporting gaps never propagate into the forecasts its customers rely upon. 

By embedding anomaly detection and machine learning directly into its data pipeline, Kpler Power ensures that clients access continuous, high-integrity datasets optimized for intraday and short-term forecasting. This not only safeguards the analytical process but also provides market participants with a consistent technical foundation for decision-making in fast-moving markets. Kpler’s Platinum feature has been successful in correcting over 70,000 data points across European load and generation in the last 10 years.

To learn more about how Kpler Power can help your business, click below to speak with one of our specialists.