Cumulus Linux: Devops and Python Scripting

As I mentioned in a prior blog, Cumulus Linux on Dell Networking Switches, one benefit of running Cumulus Linux on network switches is that you can run your typical Linux commands that you’re used to running for also configuring and managing your network switches. Additionally, you can leverage the many robust Linux applications and integrate easily with a Linux environment; this will appeal to developers (especially those in devops roles) as it will allow for new innovative approaches for automation, troubleshooting, and developing/deploying network applications/services.

In this short blog, I’ll demonstrate how a simple Python script that monitors for and takes action upon a network event can be run natively on a Dell S6000 switch running Cumulus Linux. The lab setup is the same as what was configured in a prior blog, Cumulus Linux on Dell Switches: Spine-Leaf Architecture & OSPF on Unnumbered Interfaces. 2 x Dell S6000-ON and 2 x Dell S4810-ON switches [Cumulus Linux 2.1.1] are utilized as shown below.

Cumulus Linux on Dell Switches: Spine-Leaf Architecture with OSPF

As you can see, 10.11.0.2 is an IP address on a loopback interface of S4810-Cumulus-1 directly below S6K-Cumulus-1. OSPFv2 is being utilized over point-to-point links with unnumbered interfaces as setup via quagga in the prior blog.

I’ll write a Python script on S6K-Cumulus-1 and activate the script to run in the background even when the shell is closed. The Python script will send continuous pings to the 10.11.0.2 IP address on the S4810-ON switch directly below it. We can imagine for practicality sake that in a real environment this IP address may belong to a critical application or database server we want to monitor. If a ping is missed, based on whatever threshold we set, a log file is generated describing the issue and an e-mail alert is sent with a description of the connectivity issue along with the attached log file.

The cool thing is that Python 2.7.x is pretty much installed by default on all Linux builds and Cumulus Linux is no exception, so a network/devops engineer can quickly write this script and kick it off with standard Linux commands without mucking around much. Below is the complete Python script for monitoring an end-node, generating a log file upon a missed ping, and generating an e-mail with a description of the issue and attached generated log file. For ease of simplicity and understanding, error-checking is left out of the code.

The Python script is shown in two separate screenshots. In the first screenshot directly below, a Boolean variable, ping_success, is defined with a default value of True. A continuous loop that loops through the code which sends three pings to the 10.11.0.1 IP address out interface swp31 is implemented. If even one ping reply is not received, the Boolean variable, ping-success, is set to False, which causes the conditional check of the while loop to fail, and, in return, the script exits the loop.

Python Script for Monitoring End-Node via 'ping' (1 of 2)

Once the while loop is exited, it is known that a ping to the respective IP address has failed. As such, a log file is generated and stored locally. Additionally, an e-mail alert is sent alerting the admin of the connectivity issue; the generated log file is also attached to this e-mail for additional information. Note, in this case, Google’s Gmail smtp server is leveraged to actually send the e-mail; this can also easily be done locally if desired.

Python Script for Monitoring End-Node via 'ping' (2 of 2)

Next, I make the script executable with the chmod +x monitor_server.py command. Now, the script can be run in the background and persist even after closing the shell as shown below.

Running Python Script in Background (Persists after Shell Close)

Observe, the process ID given for the job running in the background is 12402. Using the ps aux command, I can see all processes running. Below, I show a snapshot of the 12402 process running. Note, also the process with the process ID of 12430, ping 10.11.0.1 -I swp31 -c 3; this process was initiated within the Python script using the Python subprocess library.

Snapshot of Running Processes Output from 'ps aux' Command

I can simulate a connectivity issue by shutting down the swp31 interface being utilized by the Python script to ping the 10.11.0.1 IP address. To accomplish this, I use the ifdown swp31 command. Once the respective ICMP reply from issuing the ping command is not received, a log file called monitoring_server.log is generated describing the issue encountered and an e-mail describing the issue with the log file attached is sent to the respective admin e-mail account.

Log File Generated via Python Script Once Connectivity Issues are Detected

Below is confirmation that the e-mail generated via the Python script was sent.

Generated and Sent E-mail via Python Script

In this short blog, I demonstrated how Python scripting can be utilized natively on Dell switches running Cumulus Linux. In Cumulus Linux, just like in any Linux build, there are a vast variety of languages, utilities, and programs available that can be incredibly useful in a devops like environment or any environment desiring to take advantage of the vast Linux and open source ecosystem. By using Cumulus Linux as the network OS, it is possible to leverage Linux and open source tools across both the compute and network infrastructure.

Follow me on Twitter: @Humair_Ahmed