Turbocharge you Infoblox API calls (Part 3)

In part 3 of Turbocharge your Infoblox RESTful API calls series, we discuss how to batch WAPI requests using the request body to speed up our automation scripts.

OVERVIEW

In this third article of the Turbocharge your Infoblox RESTful API calls series, we discuss synchronous programming and its drawbacks. In addition, we show how to leverage the Infoblox WAPI request object to effectively batch a large number of “requests” into a single WAPI call to improve performance. First, we should discuss what synchronous programming is, why it’s good, and what are its weaknesses.

The code used in development of this article series is accessible at https://github.com/ddiguru/turbocharge_infoblox_api

Synchronous programming

Synchronous code: Scripts are loaded and executed sequentially along with the other web page components. Each time a function is called, the program execution waits until the function is returned before executing the next line of code.

Pros of Synchronous programs

• easy to write, execute, and debug
• serial or step-wise execution
• simpler business logic
• predictable

Cons of Synchronous programs

• blocking nature – a function must complete before allowing the next task to begin
• slower performing

In the first article, we created and used a simple synchronous script which creates networks – 1,024 of them to be exact. See the code listing for wapi-network.py. If we were to graphically depict what was going on in our script, it would look something like the following:

Each numbered request represents the creation of an IPv4 Network by issuing a POST call to the WAPI network object. There is some I/O wait to this operation, as the server receives and processes the request. The script blocks and does not start to add the next IPv4 Network until it has received the success or failure response from the server. Once received, the script continues to process the next network and so on.

WAPI Request Object

How can we make our script run faster?

The Infoblox REST API has a special object, called the request object, that can be used to issue several “requests” in the body of a single WAPI POST call. We will essentially batch process the creation of 1,024 IPv4 Networks in a single WAPI POST request. The code from our original synchronous script is shown below:

    # create IPv4 Network object(s) one at a time
    for network in networks:
        payload = dict(network=network, comment='test-network')
        s.post(f'{url}/network', data=json.dumps(payload), verify=False)

To use the WAPI request object, we change the business logic of our script to the following:

    # add networks using the requests object
    network_requests = []
    for network in networks:
        network_object = dict(
            method="POST",
            object="network",
            data=dict(network=network, comment='test-network')
        )
        network_requests.append(network_object)

    s.post(f'{url}/request', data=json.dumps(network_requests), verify=False)

Instead of creating a network object one at a time, we build up a list network_requests consisting of WAPI network objects that we are calling a network_object. Once all 1,024 networks have been added to the list, the network_requests list is passed as data to a single POST request call to the WAPI request object. See the full code for the wapi-request.py script. Again, if we were to graphically depict batch processing using the WAPI request object, it would look like the following:

The figure above shows a single request for all the networks.

To test this we ran our wapi-network.py script against an empty Infoblox Grid database three times to establish our baseline. We then ran the wapi-request.py scrpt three times and compared the duration times for script execution. The results were astounding. The wapi-request.py script was almost 70% faster!! The table below summarizes our results:

While, the performance of using this method was significant, performance is not everything. Recall, we’re loading this data on an empty grid – we’re not likely to see any exceptions. But what if we did have an exception? Would the entire script fail? OR partially fail? Could we ascertain which record or at which point in the script the failure occurred? Using the WAPI request module to bulk load data is far more complicated than the synchronous code of our first script. Therefore, performance comes with a price. The WAPI request object is best for the following:

• raw speed
• predictable data insert or creation i.e., inserting data into a clean database or loading zone data into a newly created zone
• creating “transactions” involving read(s) and update(s)
• insert(s) and update(s) commonly referred to as upserts

Pro Tip: While we’re focusing on the raw performance gain of the Infoblox WAPI Request Object – please know that the WAPI Request object can be leveraged to do much more than simple insert action(s). The Infoblox Request object can be used to build the concept of a transaction where you fetch object(s), make update(s), followed by persisting the changes. The word “transaction” is another term for making atomic changes, updates etc. So, please do explore the power of the WAPI Request object type.

If you decide to leverage the pure speed of the WAPI request object in your scripts, be prepared to write more complicated exception handling routines to handle data errors. In part 4 of our Turbocharge your Infoblox RESTful API calls series, we’ll look at how asynchronous programming can be used to improve our automation scripts using the Infoblox WAPI.