My fellow Infoblox IPv6 Center of Excellence (CoE) author and longtime friend, Cody Christman, wrote an article on “The True Cost of IPv4”. His article discussed the tangible costs of purchasing public IPv4 addresses on the open market and the hidden costs of the added complexity of using Network Address Translation (NAT). He points out how we often overlook the CAPEX and OPEX costs of “the way we have always done things” and take it for granted that IPv4 constraints are just “the cost of doing business”.
After reading his article I started to wonder if we could actually estimate some of these intangible costs. Would it be possible to quantify these “hidden costs” and estimate the “costs of doing business” with a limited supply of IPv4 addresses? Could we compare the business-as-usual acceptable operational costs of people’s time administering an IPv4 environment versus the cost of the IPv4 addressing asset? Let’s explore this thought-provoking exercise.
Cost of Public IPv4 Addresses
Most enterprise networks are constrained by the number of public IPv4 addresses they have at their disposal. We know the price of recent public IPv4 address transfers and expect that future purchases will be even higher. At NANOG 85 there was a panel, “Buying and Selling IPv4 Addresses,” that discussed the public IPv4 address transfer market. The panelists mentioned that the price per IPv4 address can vary widely depending on reputation cleanliness, address availability, timeliness of delivery, RIR approval, urgency of the buyer, reputation of buyer/seller, and of course the block size itself. Both Cody and this panel seem to agree that $50/public-IPv4-address may be more a reasonable general estimate for 2022.
Cost of Private RFC 1918 IPv4 Addresses
Now that we have an idea of the cost of public IPv4 addresses, let’s dive right in and attempt to estimate some of the hard-to-quantify costs. Even if the “back of the envelope” numbers are somewhat inaccurate, our goal is to try to develop an estimate and get within an order of magnitude of the true cost. We can refine the numbers as we learn more and have more empirical data, but let’s begin with a rough starting point.
We can agree that private RFC 1918 address space is not nearly as valuable as public IPv4 addresses. However, there is still a value to private addresses that is quantifiable because these addresses have some intrinsic value to the organization’s business operations. We can also add to this list the “Shared Address Space” 100.64.0.0/10 (RFC 6598), as that is being use in some networks and in IaaS public cloud virtual network infrastructures.
- 10.0.0.0/8 = 16,777,216 addresses
- 100.64.0.0/10 = 4,194,304 addresses
- 172.16.0.0/12 = 1,048,576 addresses
- 192.168.0.0/16 = 65,536 addresses
This makes a total of 22,085,632 unique private IPv4 addresses available for an organization to use internally.
Let’s assume that each private address is internally worth $1 each. They can’t be sold on the open market, but they do have value to an organization that is using them for private networks. This simple $1 valuation is just a guess but is about 2% the value of a public IPv4 address, which seems like a good place to start. That would make this entire private address pool internally valued at around $22M.
While most enterprises do not have a practice of apportioning “charge-backs” to business units using large amounts of private IPv4 addresses, let’s consider this idea for just a moment. Imagine if the cloud DevOps team asks the DDI team for a /16 of 10-net address space (65,536 addresses) for their new application project. Provided that amount of 10-net space is even available, at an internal cost of $1 each, that comes to $65,536.
Would the IT management team support charging that cloud team for their use of these limited resources? It is more commonplace that the DDI team would reluctantly give the cloud team the addresses because they don’t have a mechanism to charge their project for the use of this asset. What commonly occurs is that the cloud environment’s IPv4 addresses overlap the internal addresses, and a bi-directional NAT is put in place for the hybrid-cloud connectivity back to the corporate network. However, the DDI team could still insist that the cloud team create a business justification for why they aren’t using IPv6 instead.
OPEX Estimate of IPv4 Renumbering Activities
Now, let’s explore the operational costs of having to optimize the IPv4 address utilization in a private data center network. Imagine a scenario where a small server subnet only has 100 servers but has been assigned a /24 prefix to that VLAN. The network team has developed a plan to change the subnet mask from a /24 down to a /25 and utilize the other /25 for another network somewhere else in the data center that is desperate for addresses. Sound familiar? The operational costs for changing this subnet mask will impact the system administration team, the security administrators, and the network administrators. A Project Manager may also be employed to oversee the timely implementation of the effort.
First, the system administrators must change any server’s IP addresses that are in the upper part of the /24 (.128 to .254) down into the lower part of the /24 (.2 to .126, assuming the router is .1). When making these changes they also need to turn down the DNS TTL a day before the change occurs, update DNS, have the security team update firewall objects and policies, renumber the server, and perform app testing for each of those renumbered servers. Let’s estimate that this involves re-addressing 50 servers, and each server takes just one hour to prepare and one hour to perform the change and test. For argument’s sake, assume the cost of an hour of a system administrator’s time is $100. But this task will be at a time-and-a-half overtime rate since this can’t be done during business hours (so $150/hour). Adding up all of these costs: prep for 50 hours during the day (50 X $100 = $5,000), 50 hours during change windows (50 X $150 = $7,500). That brings this activity OPEX cost to $12,500.
Next, the system administrators must change the subnet mask on the remaining servers from a /24 to a /25. This task needs to be performed on 50 servers. Unlike the previous tasks, let’s assume that the sysadmins use software automation, and this won’t be a manual process. Let’s estimate $20/server which brings this OPEX costs to $1000.
Finally, the network administrator needs to change the upstream router(s) from a /24 to a /25 subnet mask and check that proxy-ARP is turned on. It takes the network admin just 2 hours to prepare for the change window, make the change, and test everything. Let’s assume an hour of network admin’s time is worth $100, but this after-hours change makes this OPEX cost come to $300.
The total for this subnet mask change comes to $13,800. That seems about right for several weeks of work.
OPEX Cost Estimate of Renumbering a Data Center Network
Let’s take this exercise a step further with yet another (not uncommon) example. Consider having to completely readdress an entire data center subnet. In this scenario, a company has been acquired, and due to the M&A activity, there are IPv4 address conflicts with the acquiring company. This forces the acquired company to re-address so the companies can be integrated. Let’s use that same data center network with 100 servers on it and all 100 need to be re-addressed.
The system admins will be preparing to change all the addresses of all the servers in the /24 to a different /24, and during the change windows updating DNS, firewall rules, app testing, etc. Let’s stick with 100 hours of preparations at $100/hour and 100 hours of change window time at $150/hour, which adds up to a total of $25,000.
Also, the network admins need to prepare for changing the routers, maybe change VLANs assigned to switch ports, check routing tables, and perform the cut-over and perform testing. Let’s estimate 4 hours to prepare at $100/hour and 4 hours to conduct the change, and test at the $150/hour overtime rate; this totals $1,000.
The total estimated OPEX cost for this activity is about $26,000. A typical large data center might have somewhere between 100 and 200 VLANs, each filled with servers, backup equipment, security tools, OOB management, storage, and a variety of other devices. Readdressing 200 networks at $26,000 would come to a total OPEX cost of $5.2M per data center. That is probably why most organizations opt for implementing NATs rather than take on a massive re-addressing project.
Furthermore, the real cost could be higher than this estimate because there could also be extensive application testing required, server backups that need to be tested, and this could involve rebooting systems multiple times to validate persistence of the new address. In addition, there are risks of roll-backs or having to attempt these changes multiple times due to unforeseen application issues. If the IT teams lack software configuration automation of these tasks, then these OPEX costs remain high.
Preparing to Sell Public IPv4 Address Assets
Imagine a university campus network that has a /16 of public IPv4 addresses. In this scenario, the “pointy haired” boss just got back from a business trip and on their flight read a seat-back-pocket-magazine article about how valuable public IPv4 addresses are. Now they want an estimate of the amount of time it would take to completely renumber the internal university network from public addresses to private 10-net space so they can sell the public IPv4 addresses. This would go a long way toward ensuring their promotion to CIO.
The university has 200 networks each with /24s of public addresses. We assume that data center networks are more difficult to renumber while end-user access networks use DHCP scopes and are easier to readdress. There are 40 data center networks and 160 campus wired and wireless access networks. Let’s use our data center readdressing estimates from above. The cost to readdress 40 data center networks with 100 servers each would come to $1,040,000.
To readdress user networks, a network engineer needs to change the DHCP scope and adjust the network configurations and then the desktop team will need to test and assist users rebooting desktops/phones, etc. For 160 end-user networks at 5 hours each and $150/hour after-hours overtime rate, this comes to $120,000 for this activity.
In this scenario, the cost to renumber from public addresses to 10-net address space is approximately $1,160,000.
The university could receive revenue from selling the public IPv4 /16 prefix at $50/address to a total of $3,276,800. If the $2M gain from this exercise went to the IT department’s budget, then this could make the effort a worthwhile endeavor. However, if the university uses the $2M gain for other purposes, then the IT department may not feel as compelled to take on this large project.
Cost of IPv4 “Squat-Space”
There are very large multi-national enterprises that have been desperate for additional private IPv4 addresses and have resorted to using IPv4 addresses that are assigned elsewhere. For example, some enterprises may have resorted to using addresses that were formally unallocated like 1.0.0.0/8 (which APNIC allocated back in 2010), or 2.0.0.0 (which RIPE NCC allocated in 2009), but now these are used by other entities. There are many large enterprises that have been using U.S. Department of Defense (DoD) addresses like 6.0.0.0/8, 7.0.0.0/8, 11.0.0.0/8, 21.0.0.0/8, etc., on their internal networks.
The internal value of these blocks is not as valuable as RFC 1918 addresses because they have the risk of potentially overlapping with other multi-national corporations doing the exact same “squatting.” However, the bigger financial risk is having to rapidly renumber once that DoD address is sold/transferred/used on the Internet. If you estimate a large enterprise is using 1,000 of /24s of squat-space, the OPEX cost of renumbering could be $26,000,000.
Even if the large enterprise’s IT team had the available time to rapidly renumber, what IPv4 address range would be used for the re-addressing? There isn’t enough 10-net address space to go around. The unfortunate but likely outcome would be multiple regions of overlapping 10-net address space with high-performance bi-directional NATs and lots of troubleshooting.
OPEX Cost Estimate of Troubleshooting Address Overlaps and NAT Issues
Let’s consider one final scenario when private IPv4 addresses overlap and troubleshooting ensues. Could we try to estimate how much time it takes to troubleshoot a single IPv4 address overlap on an access network? When two nodes are accidentally using the same IPv4 address, then both systems are negatively affected, and a trouble-ticket is generated and assigned to the network and systems team to solve. First, the IT team must track down the two systems, check their MAC addresses, log into both, troubleshoot, correct the problem by re-addressing one system, and perform testing to verify it is now working. This simple troubleshooting exercise could require 2 people (a system admin and a network admin) working 2 hours each at $100/hour, which comes to an OPEX total of $400. However, we are still not calculating the cost of lost productivity or downstream impacts of the service-affecting issue this might have caused. This further increases the total.
Now what would be the OPEX cost estimate of the time it takes to troubleshoot a bi-directional NAT address conflict? This could be due to more M&A activity or resulting from a business partner/vendor LAN-to-LAN VPN tunnel with a NAT at each end. This could be a more complex troubleshooting exercise requiring 3 people (a system admin, a network admin, and a security admin for troubleshooting the firewalls) at both companies for 5 hours each at $100/hour. This OPEX cost would come to a combined total of $3,000.
Firewalls with NAT rules are complex, difficult to understand, time consuming to troubleshoot, and have business risks of inadvertently creating an incident. There could be catastrophic consequences of a misconfiguration resulting in a security breach (security breach insurance policy), remediation, not to mention costs of credit monitoring for those whose PII is leaked. We can see that there’s an added administrative burden due to the risks, brittleness, and potential of human error due to IPv4 address constraints.
Summary
The initialism YMMV (Your Mileage May Vary) is appropriate in the examples above, and we can certainly debate these cost estimates for days. For most organizations, these costs may be different or the time it takes to perform tasks may vary. Do these estimates seem reasonable, or are they even a bit conservative? Regardless of the estimated accuracy, the time IT teams are spending renumbering IPv4 and dealing with IPv4 address overlaps and conflicts has a high cost. These activities are preventing IT teams from having the available time to move forward with IPv6 deployment.
Enterprise organizations will continue to use IPv4 for the foreseeable future, which means we can expect at least a decade more of these types of activities. Consider how many times the enterprise IT teams have had to perform one or more of these tasks in the past year. How many of these tasks will be performed in 2022, or will the organization expect to perform in 2023? Over the coming decade this re-addressing activity could add up to entire full-time-equivalent human resources–or much more!
While many smaller organizations haven’t exhausted their supply of public or private addresses, large enterprises are literally throwing good OPEX money at the problem of dealing with legacy IPv4 networks. IT practitioners are well aware that these activities take place after hours, on weekends, and are burdensome. IT people are seldom compensated for “going above and beyond” related to these events and this is never the project that the network engineer wants to be assigned. It is a thankless job, no fun, no learning, risky, non-differentiated heavy lifting, and at the end of the project, in the end, we still have an IPv4 network that could overlap again in the future.
Maybe next time an organization encounters an IPv4 re-addressing project they consider using IPv6 to help solve the problem. IPv6 has an abundance of address space, making each individual address essentially free. The supply of global IPv6 addresses means that these can be used internally, on the Internet, in the cloud, and private data centers without any overlaps or having to use NAT. The plentiful amount means never having to re-address an IPv6 /64 prefix. With IPv6 the OPEX costs of renumbering disappear. Imagine that!
