-
Notifications
You must be signed in to change notification settings - Fork 6
Unit Testing faradayio Manually
Bryce Salmi edited this page Jan 25, 2018
·
1 revision
We use pytest to unit test the faradayio code. This runs as much of our custom code as possible through specific tests designed to ensure functions and classes operate exactly as intended. Any code that results in an unexpected change should cause a test to fail. This is however, only true if the test is thorough and complete.
Our project uses Travis CI to automatically unit test code when a Pull Request is submitted. You should however test your code locally before uploading it to the server. To show you how this is done the follow example will step through running pytest to unit test faradayio.
- You are using Python 3.5.2 or newer
- A virtual environment
.venvis to be created inside thefaradayiogit repository root folder - You are operating in the virtual environment
Since we are running a test on faradayio which implements a TUN/TAP adapter you will see that we must run the code as a sudo privileged user. There are a few ways around this but for now we're going to assume a standard installation of Linux.
Running a test is as simple as the following command (which is run specifically from the virtual environment):
(.venv)$ sudo .venv/bin/python3 -m pytest
This results in the following output. Note that we don't see all the tests run. Running pytest without specific options to be verbose will only display summary data.
(.venv)$ sudo .venv/bin/python3 -m pytest
[sudo] password for bryce:
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.5.2, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: /home/bryce/Documents/git/faradayio, inifile:
plugins: asyncio-0.8.0
collected 37 items
tests/test_serial.py ................................. [ 89%]
tests/test_tun.py .... [100%]
======================================================================================== 37 passed in 0.45 seconds =========================================================================================
(.venv) bryce@bryce-ubuntu:~/Documents/git/faradayio$
That's it. These tests passed! Your PR should look like this prior to submitting it and as much of your code should be added to the unit tests as possible.
Now let's look at running pytest verbosely so we can see how many tests and what tests are being run in the 450ms it takes to test (at this time). To do this all we need is the -v option.
(.venv) bryce@bryce-ubuntu:~/Documents/git/faradayio$ sudo .venv/bin/python3 -m pytest -v
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.5.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /home/bryce/Documents/git/faradayio/.venv/bin/python3
cachedir: .cache
rootdir: /home/bryce/Documents/git/faradayio, inifile:
collected 37 items
tests/test_serial.py::test_socketOne PASSED [ 2%]
tests/test_serial.py::test_serialParamaterizedSynchSend[] PASSED [ 5%]
tests/test_serial.py::test_serialParamaterizedSynchSend[abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ] PASSED [ 8%]
tests/test_serial.py::test_serialParamaterizedSynchSend[ABCDEFGHIJKLMNOPQRSTUVWXYZ] PASSED [ 10%]
tests/test_serial.py::test_serialParamaterizedSynchSend[abcdefghijklmnopqrstuvwxyz] PASSED [ 13%]
tests/test_serial.py::test_serialParamaterizedSynchSend[0123456789] PASSED [ 16%]
tests/test_serial.py::test_serialParamaterizedSynchSend[0123456789abcdefABCDEF] PASSED [ 18%]
tests/test_serial.py::test_serialParamaterizedSynchSend[0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
] PASSED [ 21%]
tests/test_serial.py::test_serialParamaterizedSynchSend[01234567] PASSED [ 24%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xc0] PASSED [ 27%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xc0\xc0] PASSED [ 29%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdb] PASSED [ 32%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdb\xdb] PASSED [ 35%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdd] PASSED [ 37%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdd\xdd] PASSED [ 40%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdc] PASSED [ 43%]
tests/test_serial.py::test_serialParamaterizedSynchSend[\xdc\xdc] PASSED [ 45%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[] PASSED [ 48%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ] PASSED [ 51%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[ABCDEFGHIJKLMNOPQRSTUVWXYZ] PASSED [ 54%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[abcdefghijklmnopqrstuvwxyz] PASSED [ 56%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[0123456789] PASSED [ 59%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[0123456789abcdefABCDEF] PASSED [ 62%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
] PASSED [ 64%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[01234567] PASSED [ 67%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xc0] PASSED [ 70%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xc0\xc0] PASSED [ 72%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdb] PASSED [ 75%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdb\xdb] PASSED [ 78%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdd] PASSED [ 81%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdd\xdd] PASSED [ 83%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdc] PASSED [ 86%]
tests/test_serial.py::test_serialParamaterizedSynchReceive[\xdc\xdc] PASSED [ 89%]
tests/test_tun.py::test_tunSetup PASSED [ 91%]
tests/test_tun.py::test_tunSend PASSED [ 94%]
tests/test_tun.py::test_tunSlipSend PASSED [ 97%]
tests/test_tun.py::test_serialToTUN PASSED [100%]
======================================================================================== 37 passed in 0.46 seconds =========================================================================================
(.venv) bryce@bryce-ubuntu:~/Documents/git/faradayio$
Many tests were run in a short duration of time. The sliblib tests even documented what input data was sent during the test (various ASCII characters). Our TUN/TAP testing simply shows a PASSED or FAILED result though various assertions are made during each test.
Running the unit test code without sudo privileges will result in failures similar to the ones documented below. It may not be obvious until you realize the common trait of what is failing. Let's run the pytest command without sudo!
(.venv) bryce@bryce-ubuntu:~/Documents/git/faradayio$ .venv/bin/python3 -m pytest
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.5.2, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/bryce/Documents/git/faradayio, inifile:
collected 37 items
tests/test_serial.py ................................. [ 89%]
tests/test_tun.py FTUN brought down...
FFF [100%]
================================================================================================= FAILURES =================================================================================================
______________________________________________________________________________________________ test_tunSetup _______________________________________________________________________________________________
def test_tunSetup():
"""Setup a Faraday TUN and check initialized values"""
> faradayTUN = faraday.TunnelServer()
tests/test_tun.py:12:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <faradayio.faraday.TunnelServer object at 0x7f75b13c8f60>, addr = '10.0.0.1', dstaddr = '10.0.0.2', netmask = '255.255.255.0', mtu = 1500, name = 'Faraday'
def __init__(self, addr='10.0.0.1',
dstaddr='10.0.0.2',
netmask='255.255.255.0',
mtu=1500,
name="Faraday"):
self._tun = pytun.TunTapDevice(name=name)
> self._tun.addr = addr
E pytun.Error: [Errno 1] Operation not permitted
faradayio/faraday.py:93: Error
_______________________________________________________________________________________________ test_tunSend _______________________________________________________________________________________________
def test_tunSend():
"""
Start a TUN adapter and send a message through it. This tests sending ascii
over a socket connection through the TUN while using pytun to receive the
data and check that the IP payload is valid with scapy.
"""
# Start a TUN adapter
> faradayTUN = faraday.TunnelServer()
tests/test_tun.py:28:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <faradayio.faraday.TunnelServer object at 0x7f75b13d01d0>, addr = '10.0.0.1', dstaddr = '10.0.0.2', netmask = '255.255.255.0', mtu = 1500, name = 'Faraday'
def __init__(self, addr='10.0.0.1',
dstaddr='10.0.0.2',
netmask='255.255.255.0',
mtu=1500,
name="Faraday"):
> self._tun = pytun.TunTapDevice(name=name)
E pytun.Error: [Errno 16] Device or resource busy
faradayio/faraday.py:92: Error
_____________________________________________________________________________________________ test_tunSlipSend _____________________________________________________________________________________________
def test_tunSlipSend():
"""
Test SLIP data sent over the TUN adapter and serial port.
Start a TUN adapter and send data over it while a thread runs to receive
data sent over the tunnel and promptly send it over a serial port which is
running a serial loopback test. Ensures data at the end of the loopback
test is valid when received over serial. This test does not cover serial
to TUN/IP nor IP to TUN data validation.
"""
# Create a test serial port
serialPort = SerialTestClass()
# Configure the TUN adapter and socket port we aim to use to send data on
sourceHost = '10.0.0.1' # IP of the TUN adapter
sourcePort = 9998
destHost = '10.0.0.2'
destPort = 9999
# Start the monitor
TUNMonitor = faraday.Monitor(serialPort=serialPort,
name="Faraday",
addr=sourceHost,
> dstaddr=destHost)
tests/test_tun.py:88:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
faradayio/faraday.py:112: in __init__
self._TUN = TunnelServer(name=name, addr=addr, dstaddr=dstaddr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <faradayio.faraday.TunnelServer object at 0x7f75b13707f0>, addr = '10.0.0.1', dstaddr = '10.0.0.2', netmask = '255.255.255.0', mtu = 1500, name = 'Faraday'
def __init__(self, addr='10.0.0.1',
dstaddr='10.0.0.2',
netmask='255.255.255.0',
mtu=1500,
name="Faraday"):
self._tun = pytun.TunTapDevice(name=name)
> self._tun.addr = addr
E pytun.Error: [Errno 1] Operation not permitted
faradayio/faraday.py:93: Error
_____________________________________________________________________________________________ test_serialToTUN _____________________________________________________________________________________________
def test_serialToTUN():
"""
Test serial port to TUN link. Don't need a serial port but just assume that
an IP packet was received from the serial port and properly decoded with
SLIP. Send it to the TUN and verify that the IP:PORT receives the message.
"""
# Create a test serial port for TUN Monitor class. Won't be used.
serialPort = SerialTestClass()
# Configure TUN IP:PORT and IP Packet source IP:PORT parameters for test
sourceAddress = '10.0.0.2'
sourcePort = 9998
tunAddress = '10.0.0.1'
tunPort = 9999
# Start a TUN Monitor class
TUNMonitor = faraday.Monitor(serialPort=serialPort,
name="Faraday",
addr=tunAddress,
> dstaddr=sourceAddress)
tests/test_tun.py:149:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
faradayio/faraday.py:112: in __init__
self._TUN = TunnelServer(name=name, addr=addr, dstaddr=dstaddr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <faradayio.faraday.TunnelServer object at 0x7f75b1380208>, addr = '10.0.0.1', dstaddr = '10.0.0.2', netmask = '255.255.255.0', mtu = 1500, name = 'Faraday'
def __init__(self, addr='10.0.0.1',
dstaddr='10.0.0.2',
netmask='255.255.255.0',
mtu=1500,
name="Faraday"):
> self._tun = pytun.TunTapDevice(name=name)
E pytun.Error: [Errno 16] Device or resource busy
faradayio/faraday.py:92: Error
=================================================================================== 4 failed, 33 passed in 0.51 seconds ====================================================================================
Exception ignored in: <bound method TunnelServer.__del__ of <faradayio.faraday.TunnelServer object at 0x7f75b13d01d0>>
Traceback (most recent call last):
File "/home/bryce/Documents/git/faradayio/faradayio/faraday.py", line 101, in __del__
self._tun.down()
AttributeError: 'TunnelServer' object has no attribute '_tun'
TUN brought down...
Exception ignored in: <bound method TunnelServer.__del__ of <faradayio.faraday.TunnelServer object at 0x7f75b1380208>>
Traceback (most recent call last):
File "/home/bryce/Documents/git/faradayio/faradayio/faraday.py", line 101, in __del__
AttributeError: 'TunnelServer' object has no attribute '_tun'
(.venv) bryce@bryce-ubuntu:~/Documents/git/faradayio$
Notice how all the TUN/TAP tests are failing. This should be a good reason to check that the test was run with sudo! That covers the basics of running a unit test locally.
A FaradayRF project - Open Source Digital Radio