Turbocharge your Infoblox RESTful API calls (part 4)

In part 4 of Turbocharge your Infoblox RESTful API calls series, we discuss what asynchronous programming is, and how it can be implemented to dramatically improve the speed of automation scripts.

OVERVIEW

In this fourth and final article of the Turbocharge your Infoblox RESTful API calls series, we explain what async programming is, its pros and cons, as well as demonstrate how how it can be leveraged to vastly improve the performance of scripts which make use of the Infoblox WAPI. In this article we will again take our original WAPI script which inserts 1,024 IPv4 network objects into the grid, and refactor it into a script which leverages async programming. For this, we will use the aiohttp and asyncio Python libraries to make async calls to the WAPI.

The code used in development of this article series is accessible at https://github.com/ddiguru/turbocharge_infoblox_api

Async Programming

Asynchronous programming is the opposite of synchronous programming. A synchronous program has a one-track mind. The flow of a synchronous script is step-wise. It performs each step (task, method or function) one at a time and completes it before moving on to the next step or chunk of data. In the last article, we used batch processing by stuffing numerous network objects into a single request WAPI call, this too is an example of a synchronous script.

An async program or script behaves differently. It still uses a single thread of execution at a time, but the difference is that the system does not have to necessarily wait for the step (task, method, or function) to be completed before moving on to the next one. An async script can effectively launch several tasks in succession without having to wait for responses before starting others. The program knows what to do when a previously launched task does finish running.

Pros

• async programs are typically written to be non-blocking
• super fast!

Cons

• more complex to write
• requires specific async versions of libraries, i.e. it is not supported in infoblox-client
• poorly written tasks can block, delay or starve outstanding tasks that need running

Since we’re loading 1,024 networks, we want to do so as quickly as possible. Our business logic is simple and very well suited for async programming, because we effectively want to “shove it in there”. Our script doesn’t have any dependencies that have to be solved between each network. It’s truly “best effort” – try to get them all inserted/created and maybe output to a log any that fail.

Recall the original synchronous script logic for adding networks was as follows:

    # create IPv4 Network object(s) one at a time
    for network in networks:
        payload = dict(network=str(network), comment='test-network')
        s.post(f'{url}/network', data=json.dumps(payload), verify=False)

We refactored the above script to use asynchronous logic for adding networks as follows:

    auth = aiohttp.BasicAuth(login=ibx_username, password=ibx_password)
    async with aiohttp.ClientSession(cookie_jar=aiohttp.CookieJar(unsafe=True)) as session:
        async with session.get(f'{url}/grid', auth=auth, ssl=False) as res:
            logger.debug(res.status)
            tasks = []
            sem = asyncio.Semaphore(16)
            for network in networks:
                task = asyncio.ensure_future(load_network(sem, session, network))
                tasks.append(task)

            responses = asyncio.gather(*tasks)
            await responses

Note: the call to load_network(sem, session, network) – we implement that quite simply with a simple function the following:

async def load_network(sem, session, network):
    async with sem:
        await fetch(session, network)

The load_network method call takes a semaphore, our session, and a network object as args and asynchronously calls another method called fetch. This is the non-blocking method will make the async call to the server.

async def fetch(session, network):
    payload = dict(network=str(network), comment='test-network')
    async with session.post(f'{url}/network', data=payload, ssl=False) as res:
        if res.status == 201:
            logger.info(f'successfully inserted network {str(network)}')
        else:
            logger.error(f'failed to load network {str(network)}')

The difference is that it’s an async function call. The function is called without waiting for the response. See the full listing for wapi-async.py

If we graphically depict the flow and operation of our refactored async script for creating network objects, it would look something like the following: