Ondatra The Golang Openconfig testing framework. ·

Intro

The git repo used for this blog.

Ondatra is a framework from the openconfig working group. The framework itself has a few really good selling points / bulletin points.

Uses the same Golang test framework for Go unit testing. ie go test xx -v
Provides the availability to run unit tests against a KNE topology.
Provides the availability to run unit tests against real hardware.
Can Merge, append or remove config from devices during nit tests.
Can test openconfig paths appending, removing or modifying devices YANG trees.
Can use gNOI to devices to do things like pinging destinations, rotating certs or code upgrades.
Can add traffic generators dynamically and leverage OTG to craft packets anywhere throughout the network.

I looked around and did not find a ton of blogs or general outsider info on this topic. This is something that if you are highly into go, openconfig and kuberntes this is something that you will really enjoy.

This is a difficult thing imho to get started in for most as its not entirely a simple thing. So my goal here is to provide somewhat of a summary with a github repo and a few examples.

Disclaimer

This is entirely using KNE with arista cEOS. I will not cover the topology entirely but will put it within the repo on how I have this environment running. I am not gong to cover service proviles or OTG in this blog post. I am going to do the unit tests the hardway essentially.

This post is going to use KNEbind/kinit in real life you can either use this or you can present to ondatra in the file format it knows with their very own proto this just makes it a bit easier.

Workflow

Since I live in pretend land / Demo land fwiw I imagine most would make their workflow as follows.

Create a change.
Push said change to KNE environment.
Run unit tests.
Deploy if passes to real environment.

How Ondatra works?

The idea here is that you provide standard go unit tests so imagine a world in your repo where you store a folder called tests/ and within tests you have testing files called interfaces_test.go, bgp_test.go etc. Where in each of the testing files you write basic unit tests.

Those unit tests would be the same type of unit tests you would leverage golang for. Basic unit tests.

However, the interesting part about the ondatra go module. This go module provides an API wit specific helpers like the following I am using.

This picture was taken from the nanog 85 preso on KNE. gnoioverall

You can ues these Go based API’s against a testbed. The testbed basically provides what are called DUTs and ATEs.

Duts Devices Under Test
ATEs Automated test equipment

In this environment I am using DUT’s. DUT’s are going to be in my examples cEOS pods using KNE. The information on DUT’s can be found within each one of their methods here I will get to this later on but with KNE it has to expose services for the DUTs for example, it needs gnmi, gnoi etc.

Before we get into it here is a example of what really needs to be provided via KNE/Kubernetes per device.

kubectl get services -n ceos service-r1 -o json | jq .spec.ports 
[
  {
    "name": "api443",
    "nodePort": 30266,
    "port": 443,
    "protocol": "TCP",
    "targetPort": 443
  },
  {
    "name": "gnmi6030",
    "nodePort": 32302,
    "port": 6030,
    "protocol": "TCP",
    "targetPort": 6030
  },
  {
    "name": "ssh22",
    "nodePort": 31982,
    "port": 22,
    "protocol": "TCP",
    "targetPort": 22
  }
]

Getting start on the environment.

This environment runs within KNE. The environment has 2 cEOS devices each running back to back connections with exposed gNMI, api and gNOI services.

knepic

Each of the pods then has a service for gNMI and HTTP.

For example the kubernetes output for services.

➜  blog git:(master) ✗ kubectl get pods -n ceos
NAME              READY   STATUS    RESTARTS   AGE
r1                1/1     Running   0          20h
r2                1/1     Running   0          20h

and the services.

 kubectl get services -n ceos | grep service-r
service-r1                      LoadBalancer   10.96.120.76    192.168.32.105   443:30266/TCP,6030:32302/TCP,22:31982/TCP   20h
service-r2                      LoadBalancer   10.96.184.27    192.168.32.106   443:32275/TCP,6030:30093/TCP,22:32312/TCP   20h

This effectively means that within our KNE environment we have 2 devices connected and exposing ports for 443,6030 and 22.

Running KNE.

Everything once again is located within the git repo

Create the KIND cluster with the cluster.yaml file

kind create cluster –config cluster.yaml

Create meshnet daemonset.

Assuming you have the meshnet manifests locally.

kubectl apply -k meshnet-cni/manifests/base

Apply ceos operator

https://github.com/aristanetworks/arista-ceoslab-operator

Apply metalblb

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

IPaddress pool for kind..

You might need to change the address pool IP’s. I used the 192.168.32.x because that is what my docker interface used.

kubectl apply -f metallb-config.yaml

Move ceos images

You need to copy the ceos image to the kind kubernetes cluster.

ceoslab 4.31.0F ecc52bdfdf25 7 weeks ago 2.42GB kind load docker-image ceoslab:4.31.0F

Apply KNE Binary

You will need to get the kne binary and compile it.

git pull kne && cd kne_cli && go build.. then move file here ./kne_cli create ceos.pb.txt

kubectl get pods -n ceos
NAME              READY   STATUS    RESTARTS   AGE
r1                1/1     Running   0          26m
r2                1/1     Running   0          26m

At this point we should be ready to go.

Taking a look at the code.

Now that we have our environment up and running we now need to write some code for ondatra.

Lets take a look at what is relevent within the git repo.

 ondatra_test.go 
 testbed.textproto
 config.yaml

There are a lot of files within this environment. The ones of importance relate

ondatra_test.go - This will provide the unit tests. This is where we are going to spend most of our time. You will write actual go unit tests within this file. For example, can I use gnmi path x or can I use gNOI service y.

testbed.textproto - This is the testbed file. This tells ondatra what DUTs, ATEs and where they are cabled together. This essentially is the test bed.

config.yaml - This file is specifically for the KNEbind testing. It specifically tells ondatra where and which DUTs to use for the ondatra API.

Go unit test file.

Lets take a look at the ondatra_test.go file with some explanation

func TestMain(m *testing.M) {
	//Init the kinit to read the kne yaml file
	ondatra.RunTests(m, kinit.Init)
}

func Allduts(t *testing.T) []string {
	devs := ondatra.DUTs(t)
	var dutsslice []string
	for _, r := range devs {
		//Loop through the names of the duts and append it to dusslice
		dutsslice = append(dutsslice, r.Name())
	}
	return dutsslice
}

This code is pretty simplistic. The TestMAin file reads the KNEbind file. The all duts returns a slice of all the duts. In my case it would return [“r1”,“r2”] for example leveraging the Ondatra API.

Lets take a look at the Ondatra unit tests that are written one by one.


func TestAllConfigs(t *testing.T) {
	//Get the pwd
	path, err := os.Getwd()
	if err != nil {
		fmt.Println(err)
	}
	for _, i := range Allduts(t) {
		dut := ondatra.DUT(t, i)
		config, err := os.ReadFile(path + "/" + i + "-append.cfg")
		if err != nil {
			fmt.Print(err)
		}
		//Either config can be appended or wiped.
		//dut.Config().New().WithAristaText(string(config)).Push(t)
		dut.Config().New().WithAristaText(string(config)).Append(t)
	}
}

The TestALLConfigs function will open up the current working directory specificically the r1 and r2 files for r1-append.cfg / r2-append.cfg and try to specifically append config to the devices. As in live devices. This is a pretty easy test.


func TestGNMISystem(t *testing.T) {
	dut := ondatra.DUT(t, "r1")
	sys := gnmi.Lookup(t, dut, gnmi.OC().System().Hostname().State())
	configHostname, _ := sys.Val()
	if configHostname != "r1" {
		t.Errorf("Expected result to be %s, got %s", configHostname, "r1")
	}
}

func TestGNMILLDPNeighbors(t *testing.T) {
	dut := ondatra.DUT(t, "r1")
	lldp := gnmi.Lookup(t, dut, gnmi.OC().Lldp().State())
	if !lldp.IsPresent() {
		t.Fatalf("No LLDP for %v", dut)
	}
}

The TestGNMISytem is really interesting to most here. It will allow ondatra to test gNMI paths. The basic idea here is really amazing because it uses the ygnmi typing to check system hostname. If the system hostname for r1 for example is r1 then it is good. If something happens during a change where someone changes r1 to something else this would effectively fail.

The TestGNMILLDPNeighbors is another one. SPecifically if there is not a lldp neighbor this test will in fact fail.


func TestInterfaceEth1(t *testing.T) {
	InterfaceEth := gnmi.OC().Interface("Ethernet1").AdminStatus().State()
	dut := ondatra.DUT(t, "r1")
	Ethernet1 := gnmi.Lookup(t, dut, InterfaceEth)
	if !Ethernet1.IsPresent() {
		t.Fatalf("Check Interface %v is present", dut)
	}
	if result, ok := Ethernet1.Val(); ok && result == oc.Interface_AdminStatus_DOWN {
		t.Fatalf("Check Interface %v is up", dut)
	}
}

The TestInterfaceEth1 is just a quick example of gNMI ygnmi typing as well. This test says that is ethernet1 is NOT admin up then the test will fail.


func TestDestination(t *testing.T) {
	dut := ondatra.DUT(t, "r2")
	dest := gnoi.Execute(t, dut, system.NewPingOperation().Destination("8.8.8.8").Count(5))
	for _, d := range dest {
		if d.Received == 5 {
			if d.Received == 0 {
				t.Fatalf("Cannot ping destination")
			}
		}
	}
}

The last test is the gNOI test. Ondatra has the ability to run gNOI services on a DUT. So if R2 is unable to ping 8.8.8.8 this will fail.

Running the tests

go test -testbed=testbed.textproto -config=$PWD/config.yaml

This is exactly like running a go unit test. But we are pointing to the testbed which has the mappings and the config for KNEbind.

Here is a snippet of the results.

I0208 19:51:11.892527    5728 arista.go:307] r1 resetting config
I0208 19:51:14.136294    5728 arista.go:326] r1 - finshed resetting config
I0208 19:51:14.142103    5728 arista.go:276] r1 - pushing config
I0208 19:51:16.538082    5728 arista.go:300] r1 - finished config push
I0208 19:51:16.543638    5728 arista.go:307] r2 resetting config
I0208 19:51:18.279975    5728 arista.go:326] r2 - finshed resetting config
I0208 19:51:18.339838    5728 arista.go:276] r2 - pushing config
I0208 19:51:21.214160    5728 arista.go:300] r2 - finished config push

********************************************************************************

  Testbed Reservation Complete
  ID: e49b2964-bdea-4004-b74f-d127ebf94596

    r2:               r2
    port1:            Ethernet500
    port2:            Ethernet1
    r1:               r1
    port1:            Ethernet500
    port2:            Ethernet1

********************************************************************************

=== RUN   TestAllConfigs
    ondatra_test.go:46: 
        *** Appending config to r1...
        
        
I0208 19:51:21.221273    5728 arista.go:276] r1 - pushing config
I0208 19:51:22.198059    5728 arista.go:300] r1 - finished config push
    ondatra_test.go:46: 
        *** Appending config to r2...
        
        
I0208 19:51:22.269131    5728 arista.go:276] r2 - pushing config
I0208 19:51:22.734840    5728 arista.go:300] r2 - finished config push
--- PASS: TestAllConfigs (1.57s)
=== RUN   TestGNMISystem
--- PASS: TestGNMISystem (0.21s)
=== RUN   TestGNMILLDPNeighbors
--- PASS: TestGNMILLDPNeighbors (0.01s)
=== RUN   TestInterfaceEth1
--- PASS: TestInterfaceEth1 (0.00s)
=== RUN   TestDestination
    gnoi.go:35: 
        *** Fetching gNOI clients for r2...
        
        
--- PASS: TestDestination (5.22s)
PASS

*** Releasing the testbed...

ok      github.com/burnyd/ondatrablog   16.801s

Wrapping up.

This is really interesting if you are into CICD and want to write legitimate unit tests within Go if that is your thing. I plan in a future blog to cover a bit more to this like feature profiles and OTG. But as of right now this has a real developer like feel to unit tests and network devices.