Automation of testing using mac80211_hwsim and Orbit
We can use testing the robustness and functionality of the Linux wireless stack. Ideally we'd like to see automated tests run weekly to ensure there are no regressions for basic functionality. To test mac80211 and cfg80211 we can use and advance mac80211_hwsim as required without the need to actually use hardware. To test actual device drivers we can work with Orbit to use their existing hardware (or purchase new hardware) and infrastructure to automate routine tests. The idea is we'll have a set of standard tests the project developer will write in coordination with the Linux wireless development community. Each driver will run through the set of tests and we'll be able to determine whether or not each driver passes each individual test. We'll mark in green each passed test and with red each failed test. Ultimately it would be great to merge these efforts as part of the Linux Testing Project.
To get started start testing build the wireless-testing git tree and load the mac80211_hwsim driver. Then read the mac80211_hwsim documentation. Then start doing basic AP, STA testing as you would with two devices present on one system, one acting as an AP and another as a STA.
To start testing at Orbit read Orbit's documentation for developers and register for an account.
This work would involve becoming very familiar with 802.11 wireless technology in general, Linux wireless from a user perspective, and kernel side when enhancements are required. Recommended readings are the Oreilly book on 802.11 Networks.
Getting your hands wet
Students can start by becoming familiar with iw and using wpa_supplicant with the nl80211 driver. We eventually want to not depend on wireless extensions so the new cutting edge tests will all be based on the new nl80211 stuff which both iw and wpa_supplicant can use. iw is strictly nl80211 based. Of course we should still have things to test wireless-extensions but soon it may be possible to live completely without them for new cfg80211 based drivers.
Then you can look at starting to quickly create an AP with the different encryption options, and use a STA to associate to that AP quickly and easily as well. To test AP mode can use hostapd. Please see the wireles.kernel.org hostapd documentation to learn how to use the nl80211 driver with hostapd.
There are some old scripts which create an AP and STA quickly depending on your encryption preferences which you may feel free to use/modify/disregard. There's one script for AP and another script for STA. You can also find some other test material on the hostapd git tree:
In there check out the hostap/mac80211_hwsim/tests/ directory.
A starting point would be to extend this with simple scripts which let you test mac80211_hwsim to do certain individual tasks one at a time. For example getting association completed and transferring some amount of data using the different encryptions algorithms available would test the software encryption with mac80211. That would be a good first easy step to become familiar with what we want to accomplish with the project.
The idea is to build a list of tests which we can automate and run to test against the wireless-testing tree completely in software. Once these tests are in place we then use them to also test against individual drivers using real hardware. Since its not expected that students will have all the wireless hardware available we can leverage the kindness of folks at Orbit and use sandboxes there during a reservation to test the different hardware present there. Fortunately the list of cards available in the Sandboxes is pretty decent and can grow. Go read the Orbit developer's section. This is the hardware present so far:
- Atheros AR5002X Mini PCI 802.11a/b/g
- Intel PRO/Wireless 2915ABG Network Connection Mini PCI 802.11a/b/g
- GNURadio - USB (we won't be testing this)
- zd1211 - USB 802.11g
- RTL8185 - PCI 802.11bg
- ISL3890 - Mini PCI 802.11bg (prism54)
- Marvell Libertas 8388 802.11 USB
We may also have Intel 5000 11n and Atheros 11n as well. Surely this will depend on coordination with the Orbit folks and we cannot simply expect everything to be there but its a starting point.
Once we have a set of scripts we are happy with for mac80211_hwsim we can save an image for re-use at Orbit with these scripts and that image can be installed during a reserved time-slot for testing. Its advised the automation of testing to make use of Orbit's own automation infrastructure called OMF. These are used at Orbit for research purposes but just as they are used for research purposes it can also be used to automate tests. Some scripts may need to be ported to use OMF.
To start becoming familiar with the Orbit testbed and using OMF you can read Orbit's tutorials.
The goal is to have a nice list of tests which we can run on for mac80211, the driver infrastructure through mac80211_hwsim and also for each driver either through Orbit or manually so users can run them at home or so that the Linux Testing Project can use. In the end it would be nice to have a page online which we can go to and see some results of tests for each driver and also for [en/developers/Documentation/mac80211|mac80211]] internals.
- Luis R. Rodriguez