Automation of testing using mac80211_hwsim and Orbit
We can use testing the robustness and functionality of the Linux wireless stack. Ideally we'd like to see automated tests run weekly to ensure there are no regressions for basic functionality. To test mac80211 and cfg80211 we can use and advance mac80211_hwsim as required without the need to actually use hardware. To test actual device drivers we can work with Orbit to use their existing hardware (or purchase new hardware) and infrastructure to automate routine tests. The idea is we'll have a set of standard tests the project developer will write in coordination with the Linux wireless development community. Each driver will run through the set of tests and we'll be able to determine whether or not each driver passes each individual test. We'll mark in green each passed test and with red each failed test. Ultimately it would be great to merge these efforts as part of the Linux Testing Project.
- Automation of testing using mac80211_hwsim and Orbit
- Orbit account
- Getting started
- Similar work
- 2009 mentors and students
Familiarity with 802.11 helps but not required. For students unfamiliar with 802.11 technology a recommended reading is the O'Reilly 802.11 Wireless Networks: The Definitive Guide.
- General scripting with bash is required
- C programming is desirable but not required at all
- Register for an account here or use OpenID to log in and then subscribe to this wiki page to get e-mails of updates to this page.
IRC - you should know how to use IRC or learn. You can start by joining the Linux wireless irc channel.
- Be comfortable or willing to be come comfortable with discussions in public mailing list. We will be using the linux-wireless mailing lists on a regular basis.
If you are going to apply to this project for GSoC and need an Orbit account please be sure to include the fact that your account is being requested for this purpose in the comment section. Please at least mention GSoC 2009.
Students interested in this project should read this section and its links to get a better idea of what this project involves.
To get started start testing build the wireless-testing git tree and load the mac80211_hwsim driver. Then read the mac80211_hwsim documentation. Then start doing basic AP, STA testing as you would with two devices present on one system, one acting as an AP and another as a STA.
To start testing at Orbit read Orbit's documentation for developers and register for an account.
This work would involve becoming very familiar with 802.11 wireless technology in general, Linux wireless from a user perspective, and kernel side when enhancements are required. Recommended readings are the Oreilly book on 802.11 Networks.
Getting your hands wet
Students can start by becoming familiar with iw and using wpa_supplicant with the nl80211 driver. We eventually want to not depend on wireless extensions so the new cutting edge tests will all be based on the new nl80211 stuff which both iw and wpa_supplicant can use. iw is strictly nl80211 based. Of course we should still have things to test wireless-extensions but soon it may be possible to live completely without them for new cfg80211 based drivers.
Then you can look at starting to quickly create an AP with the different encryption options, and use a STA to associate to that AP quickly and easily as well. To test AP mode can use hostapd. Please see the wireles.kernel.org hostapd documentation to learn how to use the nl80211 driver with hostapd.
Existing scripts for testing
There are some old scripts which create an AP and STA quickly depending on your encryption preferences which you may feel free to use/modify/disregard. There's one script for AP and another script for STA. You can also find some other test material on the hostapd git tree:
In there check out the hostap/mac80211_hwsim/tests/ directory.
A starting point would be to extend this with simple scripts which let you test mac80211_hwsim to do certain individual tasks one at a time. For example getting association completed and transferring some amount of data using the different encryptions algorithms available would test the software encryption with mac80211. That would be a good first easy step to become familiar with what we want to accomplish with the project.
Orbit usage review
The idea is to build a list of tests which we can automate and run to test against the wireless-testing tree completely in software. Once these tests are in place we then use them to also test against individual drivers using real hardware. Since its not expected that students will have all the wireless hardware available we can leverage the kindness of folks at Orbit and use sandboxes there during a reservation to test the different hardware present there. Fortunately the list of cards available in the Sandboxes is pretty decent and can grow. Go read the Orbit developer's section. This is the hardware present so far:
- Atheros AR5002X Mini PCI 802.11a/b/g
- Intel PRO/Wireless 2915ABG Network Connection Mini PCI 802.11a/b/g
- GNURadio - USB (we won't be testing this)
- zd1211 - USB 802.11g
- RTL8185 - PCI 802.11bg
- ISL3890 - Mini PCI 802.11bg (prism54)
- Marvell Libertas 8388 802.11 USB
We may also have Intel 5000 11n and Atheros 11n as well. Surely this will depend on coordination with the Orbit folks and we cannot simply expect everything to be there but its a starting point.
Once we have a set of scripts we are happy with for mac80211_hwsim we can save an image for re-use at Orbit with these scripts and that image can be installed during a reserved time-slot for testing. Its advised the automation of testing to make use of Orbit's own automation infrastructure called OMF.
OMF is a is a Testbed Control, Measurement and Management Framework. It is used at Orbit for research purposes but just it is used for research purposes it can also be used to automate tests. Some scripts may need to be ported to use OMF.
Leverage of using compat-wireless
To avoid creating new images for every single new test we must we can make use of compat-wirelesss which gets updates from wireless-testing. The Orbit lab has an NFS server where the wireless-testing tree is kept and updated daily. This directory is available through an automount NFS directory. You can simply update compat-wireless locally on the baseline image by running the admin-update.sh script from compat-wireless. Then you can compile and install the latest modules.
To start becoming familiar with the Orbit testbed and using OMF you can read Orbit's tutorials.
End goal review
The goal is to have a nice list of tests which we can run on for mac80211, the driver infrastructure through mac80211_hwsim and also for each driver either through Orbit or manually so users can run them at home or so that the Linux Testing Project can use. In the end it would be nice to have a page online which we can go to and see some results of tests for each driver and also for mac80211 internals.
A successful application should include a list of tests that are planned to be implemented. These will may range from trivial "check that functionality X works" to more complicated verifications of features like fragmentation (though verifying fragmentation is possibly too hard for this project).
Additional possible goals
- IBSS mode simulation (Beacon transmission with competition for "air time")
- IEEE 802.11a and 802.11n modes - allow packet injection (hwsim_mon_xmit)
- mac time setting and signal strength simulation with optional packet dropping (mac80211_hwsim_tx, mac80211_hwsim_beacon_tx)
Wi-Fi Alliance tests:
Add tests of the Wi-Fi Alliance certification - http://www.wi-fi.org/. Please note these tests are only available to members so they are private but we can use this to get an idea of what considerations to use for the test framework API.
Central test result database:
- central database for collecting test results
web application to show and compare submitted test results (maybe like http://www.kerneloops.org/)
UNH seems to have done some work on this. We should contact them to be aware of this effort and see if they have ideas or want to help with writing tests.
2009 mentors and students
GSoC only allows one official mentor but we also have "virtual mentors". And of course there is also the community which you are expected to also rely on for use for review of your work and ideas.
- Georgy Berdyshev
- Qasim Javed