Turbocharge your Infoblox RESTful API calls (part 1)

In this multi-part series we provide four ways to vastly improve the performance and efficiency of your calls to the Infoblox REST API.

OVERVIEW

I’ve spent the last several weeks exploring ways to improve the performance of scripts that leverage the Infoblox NIOS WAPI to make customer DDI automation execute faster. In this multi-part series, we’ll detail four ways to turbocharge your scripts when making calls to the Infoblox WAPI. Our goals are to provide the following:

• Increase performance – i.e., reduce the time to execute scripts
• Achieve greater efficiency
• Improve business logic

This series will cover these topics:

1. Use HTTP Keepalive
2. Use multi-threading
3. Make bulk updates using the WAPI request object
4. Implement asynchronous calls

Developing this series of articles, we used the following platforms, language and supporting modules:

• Infoblox TE-v1425 w/ NIOS 8.5.2
  • Standalone Grid Master
  • Not running protocol services
  • No other members in the Grid
• Python 3.9.4 on Mac OS X
• requests 2.25.1
• aiohttp 2.7.4.post0

All of the code used in development of this article series is accessible at https://github.com/ddiguru/turbocharge_infoblox_api

HTTP Keepalive

While enabling HTTP Keepalive is not a programming technique per se, it can improve the performance and efficiency of your automation scripts while interacting with the Infoblox WAPI. What is HTTP Keepalive?

HTTP Keepalive, also known as, HTTP persistent connection, is an instruction to the server that allows a single TCP connection to remain open for multiple HTTP requests/responses between the client and server. Without HTTP Keepalive, the server will close the TCP connection (not session) after each request which can lead to higher page load times. Enabling HTTP Keepalive allows you to serve all web pages or API calls over a single connection which reduces CPU, memory and other resources on your server.

You might notice the following debug output when using the Python Requests module which indicates new connections are required to the Grid Master.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): 192.168.40.60:443
DEBUG:urllib3.connectionpool:https://192.168.40.60:443 "GET /wapi/v2.11/grid HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Resetting dropped connection: 192.168.40.60
DEBUG:urllib3.connectionpool:https://192.168.40.60:443 "GET /wapi/v2.11/grid HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Resetting dropped connection: 192.168.40.60

In the above log snippet, you’ll notice that the server is dropping the connection between each call to the grid object. Each time the server closes the connection, our script has to reestablish a new connection with the server. Establishing a TCP connection with a web server over TLS is expensive! It’s several packets just to rebuild the connection and it takes time — precious time!

HTTP Keepalive is not enabled by default on Infoblox NIOS, and you must explicitly enable it by using CLI command. HTTP Keepalive can be enabled on the Grid Master using the following CLI commands:

set httpd_client keepalive on


set httpd_client keepalivetime 5


set httpd_client maxrequest 2048

What does this look like when we’ve enabled HTTP Keepalive on the Infoblox Grid Master?

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): 192.168.40.59:443
DEBUG:urllib3.connectionpool:https://192.168.40.59:443 "GET /wapi/v2.12/grid HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://192.168.40.59:443 "GET /wapi/v2.12/grid HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://192.168.40.59:443 "GET /wapi/v2.12/grid HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://192.168.40.59:443 "GET /wapi/v2.12/grid HTTP/1.1" 200 None

Much better! We’re now able to make successive WAPI calls over the same socket connection without it being torn down between calls. This alone will give us improved performance and make our scripts more efficient. See the following Python script used for making these simple sequential calls. wapi-grid.py

So, what kind of performance gain will I see with HTTP Keepaliave? Your mileage will vary – I’ve seen anywhere from 18% to 40% improvement in the performance of my scripts. And that’s without making any changes to my scripts! The Python Requests module leverages Keepalive by default via the urllib3 module, as long as you use the requests.Session in your scripts. To test this, I wrote a simple script that creates 1,024 contiguous 24-bit networks from RFC1918 10.0.0.0/8. There is nothing fancy about this script — behind the scenes it’s making 1,024 POST requests to the WAPI network object one POST request per network that we create. You can view the script used in testing. wapi-network.py

I ran the script against an Infoblox TE-v1425 running NIOS 8.5.2 several times both with and without HTTP Keepalive and got the following results:

The amount of performance gain is dependent on the type of RESTful call you’re making (POST, PUT, GET, DELETE) and the type of WAPI objects you are working with. Additionally, the performance gain is directly tied to how proximal the client is from the Grid Master, i.e. the number of router hops that exist between client and the Grid Master. The farther away the client is (in router hops), the greater the performance improvement. This is just one example of the savings when you perform numerous POST requests on the network object. Try it yourself and see what you get!

Pro Tip #1 If you are enabling HTTP Keepalive in production, make sure you add this configuration to all Grid Members that will have the potential to be Grid Master. If you are running an HA pair Grid Master, you must add the above configuration change to both node1 and node2 of the HA pair! Additionally, a Grid Master Candidate or GMC could become Grid Master – so make sure you apply the same configuration to all nodes of your Grid Master Candidate(s) as well!

Pro Tip #2 If you didn’t already know – you can configure the Grid Master Candidate to be a WAPI read-only server. See the checkbox on the Member configuration of your Grid Master Candidate for Read-Only API access. This allows you to direct any WAPI calls that read or fetch (but not write, delete or update) data.

In the next article, we look at making our scripts run even faster over our Grid with Keepalive. We’ll explore the sequential blocking style of our original scripts and how we might harness concurrency to speed up our original script.