4 major challenges in IoT testing 2021/2022

Reveal, test and master

According to Gartner, the number of Internet of Things (IoT) devices will grow to more than 20 billion by 2026. Testing these devices - those like refrigerators that automatically place orders to the supermarket to self-driving cars - will be one of the biggest challenges for device manufacturers and integrators in the coming years.

Effective testing plays a critical role here. But which is the best approach? Below, we uncover four challenges that IoT testers will face in the coming year and explain how to overcome them.

The so-called Internet of Things represents a network between objects connected to the Internet, i.e., smart devices that are able to exchange information using a defined method and data schema. Testing in IoT requires fundamental knowledge of data traffic and protocols, hardware trade-offs, software coding, Big Data implications, security, user experience (UX), and means high demands from end users and regulators.

"Now get to the point...!" you're probably thinking to yourself. All right! Surely you have encountered these three areas in your everyday life: Smart Home, Smart Building and Smart City. Each of these three areas has the potential to significantly drive growth in IoT. Yet the IoT industry is currently plagued by a significant problem. Every year, the industry spends millions on recalling defective devices that are basically due to software errors. To avoid high losses as well as image damage, the adoption of IoT testing for quality assurance (QA) is indispensable. Nevertheless, there are a number of challenges that complicate IoT testing.

Challenge #1: Myriad of different IoT platforms.

Each IoT device has its own hardware and equally relies on robust software to control it. App software is also integrated into IoT devices, issuing commands to the device and analyzing data collected by the device, such as smartwatches from various manufacturers running Android operating systems.

But because there are so many variants of software and hardware for all sorts of devices, as well as different versions of firmware and operating systems, it's nearly impossible to test every possible combination of hardware and software. So to define a meaningful subset that can be effectively tested, gather information from your end users to understand what devices and software versions they use, and analyze it to determine the most common combinations.

Once you know which devices and operating systems need to be tested, you can focus the bulk of your testing on those combinations and perform smaller integrity tests on less common combinations.

Challenge #2: Enormous number of IoT communication protocols.

IoT devices today use many different so-called communications protocols to interact with controllers and each other. Protocols such as Message Queuing Telemetry Transport (MQTT), Extensible Messaging and Presence Protocol (XMPP), and Constrained Application Protocol (CoAP) are widely used, and each has its own advantages and disadvantages. Among the most popular, however, is clearly MQTT, which performs quite well in high-latency, low-bandwidth situations.

An API, usually based on JavaScript Object Notation (JSON) or XML, is often layered on top of the transport protocol to make it easier for developers to interact with the device. Software testers can use the protocols and APIs for automated testing, and the test design depends on which APIs and protocols are used. To be effective, the testing tool must be able to support these protocols and APIs.

Challenge #3: Vulnerability to hacking attacks on IoT devices.

In October, a huge distributed denial-of-service (DDOS) attack on Dyn's servers brought down giants like PayPal, Twitter, and Spotify. The hackers had infected thousands of vulnerable IoT devices with a malware that caused the devices to generate traffic.

With more than 70% of IoT devices currently vulnerable to security issues, testing for vulnerabilities is an extremely sensitive topic. However, IoT device testers should pay particular attention to the device's password policy. Here, it is important to ensure that minimum password requirements are built into the device and also implemented.

A recommended procedure here is the old familiar one: make a password change the first time the device is accessed. This should primarily be considered when developing automated tests.

Challenge No. 4: Fast data traffic and high loads

Interconnected IoT devices rely on fast communication and brisk data traffic, such as lossless data traffic in the face of changing broadcast towers while traveling. Consequently, network status can have a significant impact on device performance. Smart devices often experience network infrastructure issues, such as congested Wi-Fi channels, unreliable network hardware, and slow or unstable Internet connectivity. IoT devices and apps need to be tested under a wide variety of conditions to ensure they respond properly without data loss.

However, creating a network with a wide variety of conditions is still practiced far too infrequently, so many teams are using network virtualization to programmatically simulate different network conditions and responses. Device system metrics, such as CPU and memory, should be closely monitored during testing.

For example, an IoT device with low memory will operate extremely slowly and may not respond properly. Software on desktop or laptop computers is activated and tested by the user tapping, clicking or typing. IoT devices are often passive, so testers need to understand what devices are being used and how they behave, which often requires a close but flexible look at performance testing and performance monitoring tools.

Not to mention that IoT testing is encountering more and more challenges as time goes on, there are a number of different tools that enable testing of IoT systems to be more efficient and of higher quality. Here are some of them:

• The JTAG-Dongle tool works as a debugger in desktop apps and is used in step-by-step debugging of platform code.
• The Digital Storage Oscilloscope tool is used to check various events in timestamps, power supply faults, and signal integrity.
• The Wireshark tool is an open-source app used to monitor incoming and outgoing traffic.
• The software testing tool tcpdump is similar to Wireshark, but does not provide a GUI.
• Blazemeter is an extremely popular tool known for its use in API testing. The tool is built on top of JMeter. This open source tool is also not new, it has existed on the market for some time. The testing tools identify their own domain specific programming language on which tests can then be written and easily converted to JMeter format at any time. You can perform IoT testing via the MQTT JMeter plugin.
• SoC Specific is one of the top tools for monitoring raw packets.
• IoTIFY was developed with the Cloud Native approach. The testing tool is perfect for realistic Internet of Tools simulation. It manages more than load testing: device to device interaction in real time, GPS synchronizations and battery modeling are just some of its testing features.
• LoadRunner is an old but proven and widely used testing tool. It supports MQTT and CoAP. The tool is very versatile and therefore can perform a wide range of testing.

Another important player in IoT testing is integration testing. IoT is not successful until the Integration Testing plan is accurate and robust enough to detect defects in the system.

IOT testing can be a tough and challenging task, but equally incredibly exciting for the testing team, making it their mission to test such a complicated web of devices, protocols, hardware, operating systems, firmware.

Interested? Then take a look at our career page of andagon people GmbH. There you will find all job offers in the field of Software Testing. We are looking forward to meeting you.

If you have further questions about IoT Testing, please contact our andagon expert team on our official homepage.