“A person with a watch knows the time,
but a person with two watches is never sure“
Friends, I love measuring things. Being able to use a thermometer or a voltmeter or pH [not Ph!] meter to measure an apparently inaccessible quantity is like having a superpower. It enables one to perceive an otherwise invisible world.
But if one measures the same quantity with two different measuring instruments, they will – in general – disagree. And if that disagreement is significant, one’s initial insights from the measurement become clouded with doubt. And eventually one has to decide which one of the instruments is reporting closer to the true answer. This is both an everyday occurrence in metrology labs (and my house) and also a profound philosophical question about measurement.
This article describes how I am resolving my philosophical confusion with regard to measurements my 5 kW Vaillant Arotherm Plus heat pump. The installation reports its own estimate for the electricity it consumes and the heat it delivers to my house. But I also monitor the electricity consumed and heat delivered using separate measuring instruments installed as part of a Metering and Monitoring Service Package (MMSP).
In this article I will compare the readings from the Vaillant and the MMSP system and show that although qualitatively similar, they disagree significantly, and the Vaillant system displays many signs of being not very reliable. But it appears to have changed its behaviour in late 2023 and may be improving.
Data
The MMSP system monitors an electrical meter and a heat meter every 2 minutes and logs the data on line. I read these meters as part of my Saturday meter-reading ritual – and I log the readings which were effective at midnight on the Friday night/Saturday morning.
The Vaillant system stores its estimates for electricity consumed and heat delivered on line and I can access them as daily, weekly, monthly or annual summaries using the myVaillant app on my iPhone. I stepped back week-by-week to the start of my installation and recorded the Vaillant estimates by hand into a spreadsheet.
[Note added moments after publication: In fact, if I were not a complete idiot, I might have noticed that there is a button to download a whole year of data at time. Doh!]
Click on image for a larger version. Screenshots from the myVaillant iPhone app for three different weeks from December 2021 to March 2024. The three full screen images show weekly summaries of data for electricity consumed, environmental yield, total heat generated and the amount of domestic hot water generated. Further details are available from each measurand as shown in the two screenshots at the left. Notice that in all but one of the screenshots, every number ends in “.0”.
Because I typed in the Vaillant data by hand, it was immediately obvious to me that there was something very fishy about the data. For prolonged periods, the quantities recorded both per day and per week (kWh readings for electricity consumed, environmental yield and domestic hot water) were all exact integers i.e. the data read “XX.0” rather than “XX.1” or “XX.2” etc. Of the 134 weekly readings I recorded, 51% of the electricity readings and 36% of the heat readings ended “.0” when one would expect typically about 10%.
Nonetheless when plotted against the MMSP readings the Vaillant data appeared qualitatively similar. One would not expect the Vaillant data to be exactly the same as the MMSP data because the Vaillant data run from Sunday to Saturday night – a one day shift compared with the MMSP data.
Electricity Data
Click on image for a larger version. Electricity consumption used to run the heat pump (kWh/day averaged weekly) estimated by the MMSP installation and the Vaillant heat pump. 10 kWh/day corresponds to around 420 W of continuous power.
Looking at the electricity consumption data it’s not easy to spot a consistent difference between the two systems. But if one records the cumulative consumption, then the different weekly periods become irrelevant. When one does this it becomes apparent that the Vaillant data are consistently lower than the MMSP data by about 7.4%.
Click on image for a larger version. Cumulative electricity consumption used to run the heat pump (kWh) since installation estimated by the MMSP installation and the Vaillant heat pump. The Vaillant system underestimates total consumption by 7.4%.
This 7.4% error amounts to approximately 500 kWh over 134 weeks or 3.7 kWh/week or 0.23 kWh/day or an error of 22 watts of continuous power. This both (a) not very much but also (b) much greater than I can explain. My first suspicion was that the Vaillant may not be measuring the power used for the auxiliary hydraulic pump used to circulate water around my radiators. But looking at the difference data in detail, I see no evidence that there was change in October 2023 when I removed the hydraulic pump.
Heat Data
Click on image for a larger version. Heat delivered to my home by the heat pump (kWh/day averaged weekly) estimated by the MMSP installation and the Vaillant heat pump. 50 kWh/day corresponds to around 2 kW of heating power.
Looking at the heat delivered data one can see that the MMSP data for winter – when most heat is delivered – seem to be consistently higher than the equivalent Vaillant data. It seems as though this difference is less this winter of 2023/24. If one records the cumulative consumption, then it becomes apparent that the Vaillant data are consistently lower than the MMSP data by about 16.4%.
Click on image for a larger version. Cumulative heat delivered by the heat pump (kWh) since installation estimated by the MMSP installation and the Vaillant heat pump. The Vaillant system underestimates total heat delivered by 16.4%.
Seasonal Coefficient of Performance (SCOP)
Click on image for a larger version. Seasonal (52 week average) COP estimated from 12 months after installation and running continuously until March 2024 estimated by the MMSP installation and the Vaillant heat pump. The Vaillant system underestimates SCOP – but the amount of underestimation appears to be getting less since reducing the flow rate in November 2023.
Having compiled the weekly data, I calculated the Seasonal Coefficient Of Performance (SCOP) by finding the ratio of total heat delivered to the total electricity consumed over the preceding 52 weeks. I then calculated this as a running quantity versus time.
Considering first the MMSP data. The first SCOP calculation is about 3.53 covering the 12 month period from August 2021 to July 2022, and plotted on August 2022. As the winter of 2022/23 proceeds the SCOP falls to about 3.45 – probably because the winter of 2022/23 was colder than the preceding winter of 2021/22. And the SCOP is currently (March 2024) at a similar level. I have changed some operational settings during this period (details later) but my conclusion from the MMSP data is that heat pump is operating in a roughly similar manner to when it was installed in August 2021.
Moving on to the Vaillant data, we see a more complicated story. First of all, based on the anomalous “.0” values, which are especially prevalent in the early data, I conclude that the initial data is just wrong: it gives a SCOP of about 3.06 when the MMSP value was 3.53. This is a large error – and when I looked at this previously…
- Initial Heat Pump Review – after 1 year but before the myVaillant App
- myVaillant App Review
… I basically dismissed the Vaillant data as being ‘indicative’ at best.
However, looking at the graph above it’s clear that something happened in November 2023. As more recent data replaced older data, the Vaillant estimate for SCOP has risen consistently and is currently 3.32 compared with 3.43, the MMSP estimate. With this level of discrepancy, the Vaillant data are now good enough to be more than indicative: even borderline useful.
What happened in November 2023?
A couple of things changed in the Autumn of 2023. In October 2023, everyone’s favourite urban plumber Szymon Czaban removed the low-loss header from the system, and a few weeks later I reduced the flow rate through the system to 50% of it’s maximum value. One can see a small step-increase in SCOP in the MMSP data when I reduced the flow rate I think one or both of these changes might possibly be behind the change in quality of the Vaillant data.
With the low loss-header in place, the heat pump could circulate water rapidly – approaching 1,000 litres per hour – and there would be a very low difference in temperature between the ‘flow’ and ‘return’ temperature of the water circulated by the pump – typically just 1 °C or less. So if the heat meter – which measures the difference between the flow and return temperatures – had a small mis-calibration, then reducing the flow and consequently increasing the temperature difference -might well improve its accuracy.
However this is all just speculation. But the Vaillant results do seem to be changing.
Summary: Can I trust my Vaillant SCOP? And can you?
Comparing the MMSP and Vaillant data, I find that – reaching beyond my philosophical confusion – I just don’t believe the initial Vaillant data. The issue where every quantity measured every day ends in an exact number of kWh is deeply suspicious. It’s like finding someone with jam around their mouth swearing that they didn’t eat the doughnut. So I trust the MMSP readings over the Vaillant readings – and if the trend in the Vaillant data of improved agreement with the MMSP data continues, it may eventually prove useful.
But based on what I have seen on my installation, the Vaillant cannot be trusted to accurately report COP and SCOP. I wish it weren’t that way, but that’s what I see. However, I can’t speak for what you can trust in your installation. Of course I could sign up for open energy monitor (see Heat Pump Monitor for installed systems) and have a third system monitoring my installation. Mmmmm.
Protons for Breakfast 
March 13, 2024 at 10:42 pm |
Michael can’t you download the data from the top right down arrow button on the right hand app screen shot?
March 13, 2024 at 10:55 pm |
Guy
Thank you. It turns out that “Yes, I can”. More proof, if I needed it, that I am an idiot. I’ll put a note in the article about that.
Thanks
Michael
March 13, 2024 at 10:59 pm
Here to help Michael – I wonder how clean it is and whether it also suffers from the .0 problem!
March 13, 2024 at 11:12 pm |
Oddly I really did enjoy that post. The search for meaning or perhaps rather explanation is a feature of the human condition. We cast that net of meaning at an otherwise uncaring universe and hope to catch some fishes within. Either that or it was like an episode of Death in Paradise. Sometimes the search itself for truth is reward enough however frustrating that truth oftimes is an elusive creature. With best wishes.
March 13, 2024 at 11:23 pm |
Exactly! In the end just recording what I observed was all I could do. Perhaps it will make sense to someone else!
Best wishes: M
March 14, 2024 at 1:36 am |
Hmm… flow meters are notoriously prone to measurement bias. If I’m understanding the measurement system of the Valliant correctly, it’ll be performing a numerical integration of (T1(t)-T2(t))*F(t), where T1 and T2 are readouts from thermocouples and F is a readout from a flowmeter.
When T1-T2 is small, there may only be a few bits of signal, in which case the discretization error could be quite large — and could sometimes introduce quite a bit of bias into the integration — unless whoever programmed the firmware knew enough to “dither” the signal. Bias will arise, for example, when T1 and T2 are nearly constant and T1-T2 = 0.03 +/- 0.02 on a system that’s measuring T1 and T2 to the nearest 0.1. In that case, T1(t)-T2(t) would almost always be computed as 0, and after integration the foolishly-programmed instrument would report the time-averaged T1-T2 as 0. If however you integrate T1(t)-T2(t)+r(t), where r(t) is a pseudorandom noise of magnitude of 0.05, the long-term average will be computed without bias, and without much noise (if you’re at all clever in how you pick the period of the pseudorandom number generator). Well…. that’s pretty advanced signal-processing, so I think it *possible* there’s no dithering in either or both of the systems which are reporting on the total heat produced by your heatpump — in which case one or both might be biased (especially when T1-T2 is small enough to be near their limits of the *digital* resolution of their sensor’s output).
Significant bias is, I think, almost inevitable in a consumer-grade ammeter or a flowmeter, even they were carefully calibrated when the system was manufactured or installed.
I’d *guess* that both of your measurement systems are using a current transformer (CT) to measure current… and if they were properly designed and aren’t subject to wide fluctuations in temperature, external magnetic fields (oh, hmm, there are some motors nearby), or power factor (oh, hmm, there are some motors on the same circuit; and your lines company won’t always be delivering at PF=1.0) then perhaps they were accurate to within 3% at the time of installation? All to say that a 7% “disagreement” in the kWh as reported by your two systems wouldn’t surprise me.
As for measuring the flow of water, I really doubt either of your systems will have a magnetic flow sensor — because what consumer would want to pay the (not insignificant) additional price for a more accurate measurement of the waterflow through their system? Probably both are using paddle switches — so maybe 3% to 6% accurate at the time of manufacture… but have they ever been recalibrated? They’ll drift. https://jsgindustrial.co.nz/wp-content/uploads/2023/06/kobold-Catalogue.pdf#page=7
March 14, 2024 at 2:55 am |
So….is the flow still too high?
March 14, 2024 at 6:56 am |
I speak as an atheist who regards both gods and heat pumps as human creations. However I am content to believe that both for the moment exceed all measure of human understanding.You may have inadvertently acquired a household god that requires propriation I wish you luck.
March 30, 2024 at 8:14 pm |
There is an interesting article by Zarch who has made the same comparison between the myVaillant app and his Open Energy Monitor and Heat Meter setup. He found the reverse to be the case for his installation with the myVaillant app giving 4.7 and his Heat Meter Setup 4.14.
https://energy-stats.uk/how-to-measure-vaillant-arotherm-cop/
March 30, 2024 at 8:33 pm |
Good Evening.
Yes, I’m aware of Zarch’s work on this. I think what it indicates is that it is not possible to know whether or not to trust the Vaillant’s self-reported results. Other correspondents have written to say that they have tested the temperature sensors on the return and flow in the Vaillant and found shockingly large errors – more than a degree. I can’t see how the heat pump could work properly with these errors but the correspondent seemed very technically savvy.
So In short, I just don’t quite know what to make of the data! I wish my conclusion was different.
Best wishes
Michael
March 30, 2024 at 8:48 pm
I just wanted to say how grateful I am for all your information here and on YouTube. By providing your ‘simple rule of thumb’ for ASHP size, I was able to pick the right quote from the four that I obtained. The system works perfectly in sub-zero temperatures using all the original radiators and gave a COP (myVaillant!) of 4.2 over the winter months of October to March. Thanks again and keep up the good work!
March 30, 2024 at 8:53 pm
Fantastic! I am so happy you found the articles and videos helpful!
Wishing you many warm winters.
M
April 22, 2024 at 8:34 am |
Hi Michael,
Thanks for the intersting insights. I too have a 5kW AroTHERM plus heat pump (since February). Unfortunately my installer wasn’t interested in adding heat moitoring so I rely on the app. They did say it was fairly accurate 🙂
I am curious about your reasoning for reducing the flow rate. Mine runs constantly at 860l/hr and has low deltaT on flow/return. What did you reduce the flow to and why? Could be that the slight increase in COP is down to more accurate heat output readings, and not necessarily in ‘real’ efficiency?
Thanks,
Nick
April 22, 2024 at 9:33 am |
Nicholas, Good Morning.
The flow setting offers the possibility of “Auto” which seemed to always give close to maximum, or to set a % of maximum from 50% to 99%. I set it to 50% “just to see” what difference it made. I noted that the ΔT increase from ~1 °C to ~3 or 4 °C and that over a period of a dew days the COP improved systematically.
I’m afraid I didn’t complete the study and I didn’t draw any firm conclusions.
Of course if the heat meter isn’t working then… it could all be nonsense! Could it be due to an effect on the flow meter accuracy? Well: it’s possible. But the flow meter does have a specification which (if I recall correctly) is only around a percent or so, and (if I recall correctly) the effect was larger than that.
I’m afraid that many parameters could have been affected, from the internal operation of the heat pump to the metering to the operation of the radiators. Obviously I’ll keep looking and see whether the Vaillant system comes even more in line with the MMSP data.
Best wishes with your endeavours. I hope your home is warm.
M
April 22, 2024 at 10:00 am
Yes I’m pleased with the constantly warm house on weather comp mode. I might be brave and have a play with the flow rate. I wonder if it will affect cycling in these mild temperatures, but I guess the heat input rate needed is the same at a given outside temp.
Thanks,
Nick
April 22, 2024 at 11:28 am
Happy playing!
For my case, – and perhaps for your case given your comment – I think the cycling is inevitable because for much of the year the heat load is below 40% of the nominal 5 kW heat load.
May 23, 2024 at 12:36 pm |
Michael I’m looking for a decent monitoring system for my aroTHERM 5kW heating/cooling/sunamp hot water system. My installer is about to take back the emoncms system he’s lent me for a year! You refer to MMSP – is that an after market fitted system? I’m looking at open energy monitoring based systems which seem a little bewildering. Do you have any helpful suggestions? Happy to take this offline if you have any detailed ideas.
May 24, 2024 at 12:21 pm |
Guy
Good Afternoon. I don’t think the MMSP hardware from PassivLiving is available anymore. I think the monitoring system from Open Energy Monitor is currently the only game in town.
It’s much better than the MMSP system which provides data in a format which is very hard to process and analyse.
The Open Energy Monitor System is expensive: I haven’t looked in detail but I think an installation would come to over a thousand pounds. But unfortunately I don’t know of any alternative. Sorry.
Best wishes
Michael
May 24, 2024 at 12:35 pm
Thanks Michael I think you’re right about OEM – having looked around further. There’s an interesting ebus daemon based product https://ebusd.de which allows more extensive extraction of Vaillant data from the ebus which is much cheaper and might do all I really need. One must rely on Vaillant data though which you’ve questioned previously albeit there is hope it’s getting better. Frankly it’s poor that Vaillant have all the data and only release it through the MyValliant Pro platform offered to installer types only. I’ve lobbied Vaillant a lot in the past two years for more data and controls and they seem to be (reluctantly and unlikely as a result of my efforts!) improving the MyVaillant app but it’s still very basic IMO.
June 9, 2024 at 7:41 pm |
Hi Michael,
I am wondering what specific heat capacity Vaillant are using in their heat output calculation? As Vaillant generally expect Glycol to be present by default I was thinking they are using a lower specific heat capacity but what I have found is that the Vaillant heat output reported appears to be higher (around 8%) than what I am calculating using independant measurement and a lower specific heat capacity as I have Glycol in my system. Electrical data seems to correlate quite well.
John
June 10, 2024 at 8:21 am |
John, Good Morning,
My guess is that Vaillant assume that there is just water (with anti-corrosion fluid) circulating. Firstly, it must be quite hard to work out the precise concentration of glycol – and thus the associated heat capacity. I’m not sure if any of the installers I follow on twitter (who install a lot of Vaillants) use glycol. I am pretty sure that the excellent specifications are just for water.
I can’t speak for what’s happening in your system, but recently the Vaillant results have got closer to the results from the MMSP monitoring system. The graph below shows the results up to early May 2024
What happened last November? I had my low-loss header removed and something seemed to change quite abruptly. Now the SCOP results are within ±0.05 of each other.
So I agree that if glycol is used that could explain some of the difference, but I don’t think that’s what Vaillant assume.
Best wishes
Michael
June 10, 2024 at 8:38 pm
Hi Michael,
Vaillants general line for a long time on glycol is protection down to -15 Deg C for UK. However Vaillant are now accepting of no glycol in their systems and maintaining warranty.
I agree with you that I think their calculations are using specific heat capacity for water. It would be simpler for them and unlike some other manufacturers in the UI they do not ask if glycol is in the system unless they are doing something clever based on the system diagram selected. Protection down to -15 would adjust the heat out by about 10%.
January 5, 2025 at 12:49 pm |
Based on my observations the Vaillant system does not measure its electrical power consumption but rather empirically estimates it. I have come to this conclusion because whereas Vaillant’s power estimate might be constant during a given run of the compressor, my independent measurement shows the power increasing as the run progresses. I suspect that this may be due to increasing friction losses as the scroll compressor heats up; an effect Vaillant’s estimate does not appear to take into account.
January 5, 2025 at 8:06 pm |
Jan, Good Evening.
I really don’t know how the Vaillant estimates electricity use, but there are many oddities, such as the app recording only whole kWh of consumption.
Comparing the Vaillant against my monitoring system, I wrote about the inaccuracies previously
However these discrepancies have reduced in the last year for reasons I don’t understand.
It would seem bizarre to simulate something that can be measured so easily, but who knows what engineering subtleties are involved.
Have you published your data anywhere?
Best wishes
Michael
January 6, 2025 at 6:43 am
Yes, that’s the article I was replying to. No, my data is as yet unpublished.
Measuring things requires measurement hardware which (apart from temperature sensors which are indispensable in a heating system) Vaillant doesn’t seem to want to add to their products.
Electrical consumption isn’t the only quantity they estimate rather than measure: the flow rate also seems to be estimated. Obviously the GIGO principle applies.
Their temperature sensors aren’t the best either. I recently had cause to replace my exterior temperature sensor (because the DCF77 receiver in it wasn’t working properly), and there’s about a 1.5 degree C offset between the old and new sensors. Furthermore I have placed a DS18B20 sensor right next to it, and (apart from a small offset which is to be expected) the relationship between the readings from the two sensors isn’t linear.