Six aquarium Automatic Top Off (ATO) systems, also know as water level controllers, were put head to head in a detailed comparison of their most important aspects. A system engineering sensitivity analysis was done to find and reduce uncertainty until there was no doubt which one was the top pick. The Tunze Osmolator 3155 came out on top as the best aquarium ATO system.
Choosing the best ATO can be a difficult decision if you are unsure about which product features are the most important to you. Aquarium ATO systems are particularly critical to maintaining stability in a reef aquarium. If they fail in the on position and continuously pump fresh water into the tank, the salinity will rapidly fall and shock the sensitive corals. This can wipe out all the corals in a reef tank. There are several aquarium ATO parameters to keep in mind including cost, customer ratings, warranty, level sensor type, level sensor attachment type, reparability and others. Each person may value these parameters differently. Which ones are important to selecting an ATO, and which ones will make a difference when picking the best ATO will be shown.
A detailed trade study comparing 6 ATO candidates and 13 parameters for each was developed. This is a complex decision with two primary areas of uncertainty, cost and customer ratings. To determine if variability in cost or customer ratings would make a difference in the final decision, a sensitivity analysis was done. Monte Carlo simulations were run with 1000 iterations on each variable-parameter, candidate combination. Where the sensitivity analysis showed variable data would have made a difference in the final decision, more samples were collected to reduce the uncertainty until the parameter was modeled accurately enough that it did not make a difference in the final decision.
In the end the best choice was obvious. This repeatable decision process is transparent. The method for each step is described and information is traceable to its source. The sensitivity analysis focused time and effort on just the areas that made a difference. Confidence in this decision is high.
The evaluation framework was developed in Excel 2011 for the Mac. 6 Candidates were chosen and 16 evaluation parameters were selected; 3 were later determined as not important. Raw input values mostly came from the manufacture’s specifications. Parameters with many variable inputs like cost and customer ratings were fit using many samples to a distribution function.
An acceptable limit and importance (or value) was assigned to each parameter. The value function was applied to each input in three steps, thresholding, normalization and weighting. Each input was limited between the smallest and largest acceptable input, and the result was linearly normalized between 0 and 1. The normalized score was weighted by that parameter’s importance. Those results were summed up for each candidate in a total score. The highest total candidate score was the initial best choice.
Sensitivity analysis was then done to determine if a variable input’s distribution might change the decision. 1000 Monte Carlo simulation runs were used to estimate inputs for each candidate – variable parameter combination. Each run resulted in a slightly different total candidate score. Results for each candidate and variable parameter were displayed in a tornado diagram. Variable inputs that drive too much uncertainty in the final decision were reanalyzed to find more sample data and develop a more accurate model of the variable input distribution. When there was little uncertainty in the top candidate, the analysis was considered complete.
The goal was to find the best aquarium ATO for a reef aquarium. The available candidates were reduced to just six using several initial requirements.
- Electronically automated, not mechanical only
- Able to pump fresh water from a reserve tank, not gravity flow
- Off the shelf equipment, not a DIY system
Six ATO systems were found online that matched the initial requirements.
- JBJ ATO
- Tunze Osmolator 3155
- Elos Osmocontroller Digital
- Spectra Pure Ultra Precise Level Controller (UPLC)
- Aqua Medic SP 3000 Niveaumat
- Reef Fanatic Level Controller
Evaluation parameters were chosen from the available online information and what was considered valuable to the decision analysis. The 13 important parameters were choses and placed into four groups: Cost, Quality, Safety Features, and Sustainability.
- Customer rating
- Level sensor 1 type
- Sensor attachment type
- Level sensor 2 type
- Safety Features
- Float snail/crab guard (only for float type level sensors)
- Pump run dry safety feature
- Audible alarm
- Pump time limiter
- Instruction manual page length
- Replacement parts availability
- Customer service
A Candidate Parameter table, shown in Figure 1, was developed by collecting information online. Information came from several sources including: original manufacture’s descriptions, users manuals, drawings and pictures, web sites, Facebook pages, videos, online retail sales sites, and online customer reviews. Sample data in red had a single value. A distribution model was built for parameters with multiple values. In a few cases information on a specific parameter was not available. Later it was decided to not invest the time and effort to find that information because it was not important enough to affect the final decision. In other cases (as with customer rating) the sample inputs varied and that parameter was very important. In those cases, much time and effort was invested in gathering enough data to reduce the decision uncertainty through sensitivity analysis to a low level.
Figure 1 Candidate Parameter table with input data.
There were two manufactures that did not provide a pump with their system: JBJ and Reef Fanatic. In order to compare all the systems equally $90 was added for a paristaltic dosing pumps that would compare well to the other systems. After doing that, total system costs only varied by $80. In order to properly use cost in the analysis, a cost model was developed for each candidate. Cost data was gathered from 8 online retailers including: Amazon, Marine Depot, Bulk Reef Supply, Drs. Foster and Smith, Pet Solutions, Premium Aquatics, Aquacave and Petco. Candidates were not sold by every retailer so there were normally less than 8 cost samples. The mean and standard deviation was calculated for each, and fit to a normal distribution function.
Although reliability is an extremely important evaluation parameter, manufactures do not provide that information. It is possible to calculate reliability from component pieces. It involves getting the design, materials list, and a complete component parts list, down to the type of plastic used and the last resistor. The system reliability is then calculated from the reliability for each component. Again, manufactures do not provide proprietary information like that to the general public. Fortunately there are two parameters that are highly correlated to reliability: customer rating and warranty. Ratings and warranty information is readily available online. Customers give low ratings to ATOs that fail, and manufactures do not provide warranties for ATOs that would normally fail within the warranty time. That is why customer rating and warranty are weighted so high in the evaluation framework.
Customer Rating Model
Customer ratings varied by each customer’s experience. Each rating also varied for the same ATO between online review sites. In order to pick a single customer rating for each Monte Carlo run, a discrete distribution model was built. 429 ratings were collected from seven online aquarium product supply sites including: Amazon, Marine Depot, Bulk Reef Supply, Drs. Foster and Smith, Pet Solutions, Petco and Aquacave. Customer ratings do not follow a Normal distribution. When a product is new to the market or not a common choice, there are few customer ratings available. Therefore the model was initialized with a flat distribution assumption, a single vote in each of the five rating levels. Then each model was modified with ratings data as available. The resulting customer rating discrete distribution functions for each candidate are shown in Figure 2.
Figure 2 Customer ratings distribution for each of the six candidates: JBJ ATO, Tunze Osmolator 3155, Elos Osmocontroller Digital, Spectrapure UPLC, Aqua Medic SP 3000 Niveaumat, and Reef Fanatic Level Controler.
Value functions were used to limit, normalize and weight each parameter input so that each ATO could be compared equally.
As mentioned above, two of the manufactures did not include pumps with their systems: JBJ and Reef Fanatic. In order to compare all the systems equally $90 was added to the cost. This would pay for a paristaltic dosing pumps that would compare well to the other systems. Paying less than $150 or more than $270 for an ATO was considered undesirable. A candidate’s cost score was limited to at worst 0 for $270 or more, and at best 1 for $150 or less. Scores were linearly normalized between the high and low acceptable costs. An example graph of this cost normalization function is shown in Figure 3. Cost was considered extremely important in the decision. The normalized cost result was weighted at 22% of the total score in comparison to the other parameters as shown in Figure 4.
Figure 3 Cost parameter normalization function.
Figure 4 Weighting factors for each of the important parameters.
Customer Rating Value
ATOs with ratings of 1 or 2 stars were considered undesirable. A candidate’s customer ratings score was limited to at worst 0 for a 2-star or lower rating, and at best 1 for a 5-star rating. Scores were linearly normalized between 2-stars and 5-stars. Customer ratings were considered extremely important. The normalized customer rating result was weighted at 19% of the total score.
ATOs with warranty lengths of 1 year or less were considered undesirable. No candidate had better than a 5-year warranty. A candidate’s warranty score was limited to at worst 0 for 1 year or less, and at best 1 for 5 years. Scores were linearly normalized between the worst and best values. Warranty duration was considered very important and weighted at 14%.
Primary Level Sensor Type Value
The primary level sensor type or level sensor 1 type, was categorized into four types: none, float, pressure, and optical. These are ordered in increasing subjective quality. The None type was used as a place holder and was given a score of 0. Float level sensors are the cylindrical styrofoam floats switched with a magnetic reed switch. They were given a score of 1. Pressure level sensors are the airline and tube sensors, switched with a pressure sensor at the base unit. They were given a score of 2. Optical level sensors use an emitting infrared LED to reflect off of water and back to a photocell to detect the water level. They are switched by the photocell. They were considered to be the highest quality sensor and given a score of 3. The candidate’s score was normalized to at worst 0 for no sensor, at best 1 for an optical sensor, and linearly normalized between. The primary sensor type was considered very important and weighted at 10%.
Sensor Attachment Type Value
The sensor attachment type was categorized into four types: suction cup, bracket and magnetic. These were ordered in increasing subjective quality. The suction cup attachment method uses one to four rubber suction cups to attach to the inside of the sump at the water level. Suction cups have a tendency to release once algae or bacteria gets between them and the tank wall. As algae grow they produce oxygen, or as bacteria grow they produce carbon dioxide. The gas bubble then pushes the suction cup off the glass. Suction cups were given a score of 0. The bracket attachment method typically uses an adjustable plastic bracket connected to the top edge or ledge of the sump and extends down to the water level. They tend to be highly reliable, but quite bulky. They were given a score of 1. The magnetic attachment method uses two attracting magnets to attach the sensor. One is connected to the level sensor and held at the water level inside the sump. The other is place in the same spot outside the sump. Manufactures use one of two types of magnets. One is a very strong neodymium-iron-boron (NdFeB) magnet that can rust so it is incased within plastic. The second is weaker but inert ferrite magnet. Ferrites are ceramic oxide (glass like) materials and will not corrode in seawater. Magnetic holders will not release with algae growth and they are relatively small. They can cause float level sensors using a magnetic reed switch to malfunction if the level sensor and attachment combination are not designed correctly. The magnetic attachment method was given a score of 2. This parameter was considered important and weighted at 7%.
Replacement Parts Availability Value
The availability of replacement parts play a significant role in being able to quickly fix critical equipment in case of failure. The idea is to have the parts that tend to fail on the shelf. The option to keep parts readily available is valuable. Candidate’s that had parts available were given a score of 1. The others were scored at 0. This parameter was weighted at 4%.
Pump Run Dry Safety Value
The pump run-dry safety feature value was either available and given a score of 1, or not available and scored 0. This feature was implemented by the manufactures in different ways. Typically the second level sensor was placed into the freshwater storage container and would turn off the pump when it emptied. Other manufactures used a peristaltic dosing pump that can run dry without issue. I believe the peristaltic dosing pump is a better solution because of their added benefits. Peristaltic pumps will not back siphon and the run dry capability frees up the other level sensor to act as a back up for the primary sensor in the sump. This feature was weighted at 4%.
Audible Alarm Value
The audible alarm feature was either available with a score of 1, or not available with a score of 0. One of the most frequent ATO failure methods is when the primary sensor gets stuck on or does not properly turn off. To mitigate this, the second level sensor is often used as a backup to the primary sensor. The alarm feature would sound an audible alarm if the second level sensor indicated the water level in the sump was too high. This feature was weighted at 4%.
Pump Time Limiter Value
The pump time limiter feature was available with a score of 1, or not available with a score of 0. If the primary sensor gets stuck on or does not properly turn off, a pump timer can provide a safety. The timer can limit the amount of water pumped each day to at most that which would evaporate during the hottest summer day. This would give some extra time to notice and fix the problem. This feature was weighted at 4%.
Manual Page Length Value
It was surprising to see how poor some of the ATO instruction manuals were. Elos did not provide a copy of its manual to retailers or on its own web site. There was an online page with the sales copy including instructions so to be generous that was considered a 1-page manual. All of the others had a downloadable pdf instruction manual available. The quality of the manual was pretty much in line with its page length. Tunze had a 68-page manual in multiple languages. Those that took the time to write longer manuals provided excellent content describing the ATO installation, operation, maintenance, and troubleshooting approaches. Candidates scored at worst 0 with no pages and at best 1 with 40 or more pages. The score was linearly normalized between. This parameter was weighted at 3%.
Float Snail and Crab Guard Value
The float level sensor snail and hermit crab guard was only used on float level sensors. Having snails or crabs crawl onto a float sensor and cause it to get stuck is a common failure method with float sensors. This feature does not apply if the ATO design does not use float level sensors. It was therefore given a score of 0 if not provided, a score of 0.5 if provided, or a score of 1 if it was not applicable. This feature was weighted at 2%.
Customer service was categorized into four levels:
- Poor, with no contact info or hard to find
- Ok, with a toll free number
- Good, with everything in the previous level plus an email address and an easy to find online manufactures information
- Great, with everything in the previous level plus good social proof on the manufactures web site, twitter profile, or Facebook page
Candidates scored at worst 0 with poor customer service, at best 1 with great customer service, and linearly normalized in between. It was weighted at 2%.
Figure 5 shows ATO total scores using the variable parameter average values and the non-varying parameter manufacture values. Scores are between 0 and 1. A score of 1 would be perfect according to the value functions.
Figure 5 Overall scores for each candidate ATO.
The initial results show the Tunze Osmolator 3155 with the highest score. It scores over 16 points higher than the next ATO. The Elos Osmocontroler Digital, Spectrpure UPLC and JBJ ATO are grouped together second through fourth. A sensitivity analysis was run next to see if the parameter variability would make a difference in the final score.
A sensitivity analysis was run in excel on the two important variable parameters, cost and customer rating. The variable parameter distribution models allowed Monte Carlo simulation runs. While holding other parameters at initial conditions, 1000 points were simulated for each modeled parameter – candidate combination. The resulting Tornado diagrams are shown in Figure 6 through Figure 11. Each Tornado is centered at the average score, on the horizontal axis, between 0 and 1. This is the initial result shown above. The 10 percentile, average, and 90 percentile input values are shown on top of each horizontal bar. Blue bars show downside 10 percentile scores. Red bars show upside 90 percentile scores. Parameters are ordered top to bottom, with largest score range to least score range.
Figure 6 JBJ ATO sensitivity analysis Tornado diagram.
There were 164 customer ratings on the JBJ ATO with a 10-percentile of 3, average 4.4, and 90-percentile of 5. 60% of the customers rated it at 5-stars. The JBJ ATO had the fourth best overall total score of 0.48. Customer rating variability changed the total score from a 0.35 downside to 0.53 upside, a 0.18 swing. This is a typical customer rating swing.
There were 7 JBJ costs samples with a worst likely $188, average $175, and best likely $163. Cost variability changed the total score from a 0.46 downside to a 0.50 upside, a very small 0.03 swing. The small sample set and cost variability were not a deciding factor.
Figure 7 Tunze Osmolator 3155 sensitivity analysis Tornado diagram.
Customers consistently rated the Tunze Osmolator 3155 very high. There were 171 customer ratings with a 10-percentile of 4, average 4.7, and 90-percentile of 5. 75% rated the Tunze at 5-stars. The Tunze had the best overall total score at 0.69. Customer rating variability caused the total score to vary from a 0.65 downside to a 0.71 upside. That is the smallest 0.06 swing of all the candidates, meaning customers were constant in their ratings. The downside rating is still above all other candidate average ratings, and above all other other candidate upside ratings. That means this ATO is clearly far above the best customer rated ATO. Customer rating variability made no difference in the final decision.
There were 6 Tunze costs samples with a worst likely $209, average $204, and best likely $200. Cost variability changed the total score from a 0.68 downside to a 0.70 upside, a very small 0.02 swing. The small sample set and cost variability made no difference in the final decision.
Figure 8 Elos Osmocontroller Digital sensitivity analysis Tornado diagram.
There were 22 customer ratings on the Elos Osmocontroller Digital with a 10- percentile of 1, average 4.0, and 90-percentile of 5. 15% of the customers rated it at 1-star, not good. Elos had the third best total score of 0.53. Variability in customer rating drove total score between a 0.35 downside and 0.59 upside, a large 0.24 swing. The 1-star ratings drove this swing.
There were 3 Elos costs samples with a worst likely $201, average $199, and best likely $196. Cost variability changed total score from a 0.527 downside to a 0.535 upside, essentially no swing. The small sample size and cost variability is not a deciding factor.
Figure 9 Spectrapure Ultra Precise Level Controller sensitivity analysis Tornado diagram.
Customers rated the Spectrapure Ultra Precise Level Controller high. There were only 6 customer ratings, fewest of all candidates, with a 10-percentile worst of 2, average 4.1, and 90-percentile best of 5. 64% rated it at 5-stars. Spectrapure had the third best total score at 0.50. Customer rating variability causes the total score to vary from a 0.38 downside to a 0.56 upside, a 0.18 swing. Despite searching additional customer rating samples were not found. The small sample size may drive the overall position for second, third or forth, but not for first place.
There were 4 Spectrapure costs samples with a worst likely $250, average $244, and the best likely $237. Cost variability changed the total score from a 0.50 downside to a 0.51 updide, a very small 0.01 swing. The small sample size and cost variability made no difference in the final decision.
Figure 10 Aqua Medic SP 3000 Niveaumat sensitivity analysis Tornado diagram.
There were 25 customer reviews on the Aqua Medic SP 3000 Niveaumat with a 10-percentile of 1, average 3.0, and 90-percentile of 5. 33% of the customers rated the Aqua Medic at 1-star. Because of that the Aqua Medic came in last with a total score of 0.35. Customer rating variability caused the total score to vary between 0.23 and 0.47, a 0.24 swing.
There were 4 Aqua Medic costs samples with a worst likely $173, average $167, and best likely $161. Cost variability changed the total score from a 0.23 downside to a 0.47 upside, a 0.24 swing. The small sample size and variability was not a deciding factor.
Figure 11 Reef Fanatic Level Controller sensitivity analysis Tornado diagram.
There were 11 ratings on the Reef Fanatic Level Controller with a 10-percentile of 2, average 3.9, and 90-percentile of 5. 56% rated it at 5-stars. Reef Fanatic took fifth place with a total score of 0.40. Customer rating variability was the primary factor driving the total score from 0.28 to 0.46, a 0.18 swing. The small sample size may be a factor in only getting fifth place.
There were 3 cost samples with the worst likely at $199, average $182 and best likely at $164. Cost variability changed the total score from a 0.37 downside to a 0.42 upside, a small 0.05 swing. The small sample size was not a deciding factor.
Cost Performance Bubble Chart
Candidate best value was compared in a cost versus performance bubble chart shown in Figure 12. Candidate performance data are a summation of all non-cost parameters including customer rating. Cost and customer rating standard deviations were added and used for the bubble size.
Figure 12 Cost Performance bubble chart comparing JBJ ATO, Tunze Osmolator 3155, Elos Osmocontroller Digital, Spectrapure UPLC, Aqua Medic SP 3000 Niveaumat, and Reef Fanatic Level Controler.
There is often a knee in the cost performance curve where you find the best value. As more money is put into the product, performance quickly ramps up. After a while it takes disproportionately more money to squeeze out that last bit of performance. If you exclude the Tunze, the knee in the curve would be at the Elos. However, the Tunze is an outlier with excellent performance and not the highest cost. The bubble sizes show cost and performance variability. If the decision were between the JBJ, Aqua Medic, and Reef Fanatic, there would be some uncertainty because the bubbles overlap. However, the Tunze has a little variability, does not overlap with any other candidates, and is clearly the best value.
The Tunze Osmolator 3155 comes out on top as the best aquarium automatic top off system. The sensitivity analysis also shows that this is the best choice considering all likely variability and reasonable uncertainty in sample data. Tunze is far and above the best choice, with a distant following group of three: the Elos, Spectapure and JBJ.
If you found this information helpful please consider buying your ATO from my affiliate link. There is no additional cost to you when using the link, and it helps me out.