Allion Labs | Ryan Huang
With this article, our series of AP performance evaluations come to a close. Based on our tests, we will compare the “multi-user support”, “stability”, and “anti-interference” abilities of different wireless APs.
Figure 1: TR-398’s Six Main Test Areas
6.4.1 – Multiple STAs Performance Test
The purpose of this test is to simulate whether the overall transmission throughput of the wireless AP will be affected by long-distance devices when it transmits with devices with different signal strengths simultaneously.
Figure 2: 6.4.1 Test Results
A certain amount of degradation was observed in the overall transmission throughput of all wireless access points during this test. Consequently, none of the wireless APs met the test standards. Overall, however, Google Wi-Fi’s wireless AP performed best in 802.11n and 802.11ac tests; in 802.11ax-2.4GHz and 802.11ax-5GHz tests, Telstra’s and Eero’s wireless APs were the best performers, respectively.
Figure 3: 6.4.1 Test Standards
6.4.2 – Multiple Association/Disassociation Stability Test
In this test, we are simulating a situation where the status of the connection changes frequently, and observing if other connected devices are experiencing transmission problems.
Figure 4: 6.4.2 – 802.11n/ 802.11ac Test Results
In 802.11n, the device’s transmission throughput must be more than 99% of 4Mbps, and its packet loss rate must be less than 0.01%; meanwhile, in 802.11ac, the device must have a transmission throughput of more than 99% of 8Mbps, with a packet loss rate of no more than 0.01%. As can be seen from the 802.11n test results, all wireless APs pass the test. However, Eero’s wireless AP failed the 802.11ac test because it was transmitting too many packets.
Figure 5: 6.4.2 – 11ax Test Results
The 802.11ax-2.4GHz test requires that the transmission throughput of each device should not be less than 99% of 4Mbps, and the packet loss rate must be less than 0.01%; while the 802.11ax-5GHz test requires that the transmission throughput of each device should not be less than 99% of 8Mbps, and the packet loss rate must be less than 0.01%. As with 802.11n, all 802.11ax-2.4GHz wireless access points passed the test specifications. However, in the 802.11ax-5GHz test, the Eero wireless AP also failed to meet the test criteria due to too much packet loss.
6.4.3 Downlink MU-MIMO Performance Test
The purpose of this test item is to confirm the wireless AP’s Downlink MU-MIMO performance.
Figure 6: 6.4.3Test Results
Since this test item specifies that the wireless AP’s antennas must support 4×4, only two wireless APs from Google and Telstra were eligible to participate. Google wireless AP’s test result for 802.11ac is just 33%, so it cannot meet the test requirement of higher than 45%, whereas Telstra wireless AP’s test result is 59.79%, which met the criteria. Nonetheless, Telstra’s 802.11ax performance remains 45% lower than the requirement, regardless of whether it uses 2.4GHz or 5GHz.
Figure 7: 6.4.3 Test Criteria
6.5.1 Long Term Stability Test
This test confirms the wireless AP’s stability over an extended period of time.
Figure 8: 6.5.1 Test Results
In the 802.11n/ac test, all wireless APs can meet the test criteria. However, in 802.11ax, Eero’s wireless AP has a packet loss rate of 1.77797% at 2.4GHz, exceeding the test criteria of less than 0.01%. In spite of the fact that it exceeds the criteria, this is not noticeable to the average user.
Figure 9: 6.5.1 Test Criteria
6.5.2 – AP Coexistence Test
In this test item, we will test how well the wireless AP is able to deal with interference from other wireless APs in the environment.
Figure 10: 6.5.2 – 802.11n/ 802.11ac Test Results
From the above results, we can see that the wireless APs of Kogan and Telstra performed better in the 802.11n and 802.11ac tests, respectively, although some items still failed to pass the test criteria. The only wireless AP that passed the test criteria is Netcomm. Also, Vodafone’s and Google’s two wireless APs were most affected by adjacent frequency interference in the 802.11n and 802.11ac tests, and their throughput was reduced by 42% (Vodafone) and 56% (Google).
Figure 11: 6.5.2 – 802.11ax Test Results
It can be seen from the 802.11ax-2.4GHz test that the Eero wireless AP is the best performer, while the Netcomm wireless AP is the best performer in the 802.11ax-5GHz test. Nevertheless, none of the three wireless APs passed the test. In the adjacent frequency test of 802.11ax-2.4GHz, Telstra and Netcomm performed 10% higher than specified, and in the 802.11ax-5GHz test, both can meet the criteria. On the other hand, Eero’s wireless AP does not meet the criteria in all test scenarios of the different 802.11ax-5GHz test.
Figure 12: 6.5.2 Test Criteria
6.5.3 Automatic Channel Selection Test
This test checks the functionality of the Automatic Channel Selection function of the wireless AP.
Figure 13: 6.5.3 Test Results
As can be seen from the results, only Telstra and Netcomm wireless APs can operate properly in each wireless channel with Automatic Channel Selection. The other wireless APs did not pass the standard criteria since they used the same channels without avoiding interference caused by other wireless APs.
Since the TR-398 test was released by the Broadband Forum Association, many industry players have started using these test criteria to check whether their products meet the specification, and to conduct a comparative analysis between different brands and products based on this unified test content. Allion Laboratories adopts the octoBox test solution for the TR-398 test, as it provides consistent, reliable, accurate, and replicable test results, meanwhile overcoming the challenges associated with setting up the TR-398 test environment.
Take the Next Step
With over 30 years of IT testing experience, Allion Labs provides certification services, customized test services, professional market evaluations, and competitive product analysis reports for customers to improve product performance and user experience. Our company helps major manufacturers build brand reputations and take advantage of market opportunities in fierce product competition.
If you have any wireless performance or TR-398 testing needs, please let us know by email or contact form.