PyPaloCleaner – Part 1

Intro

During the lifetime of firewall, a configuration may hit limit of maximum address objects created and you as an administrator have to deal with it. How? Of course by cleaning up your firewall configuration.

What are the most common causes of exceeding configuration capacity? Duplicates? – maybe, unused objects? – probably, lazy admins ? – bingo!

So what lazy admin could do to clean his firewall configuration in order to free up some space or keep firewall ‘neat & tidy’? Well, perfect place to start would be address objects. Identify and remove duplicates, do the same with unused ones. You should do the same with other objects as well in order to remove unneeded ones and give you firewall some ‘fresh air’. You should take care of:

• duplicated & unused address objects
• duplicated & unused service objects
• duplicated & unused address groups
• duplicated & unused service groups

Tools for cleaning

When we speak about Palo Alto Firewall configuration cleanup there few (free) tools avaliable. Some of the requires from you to invest some time (sometimes alot of time) and patience, some requires you to set up another system and finally some requires from you some scripting skills.

• Manual Cleaning

Manual cleaning requires from you a lot of time and patience. It may be just ‘ok’ for very small configuration and one time clean up action.

• Expedition Tool

Expedition Tool should be your best pick. It can do many things for you. It can clean unused objects, duplicated objects, unused & shadowed rules. It also can create security rules based on traffic logs, make a cup of coffee for admin and many more! A fancy tool it is!

More about Expedition Tool can be found on https://live.paloaltonetworks.com/t5/expedition-articles/expedition-documentation/ta-p/215619

However there is a small problem with this tool – it may have some problems with big configuration files. I had a case with Panorama where configuration size was over 80 megabytes and Expedition Tool failed to perform unused objects clean up. Next tool I will discuss, performed this cleanup pretty well.

• Pan-OS-PHP (aka pan-configurator)

PAN-OS-PHP is a PHP library aimed at making PANOS config changes easy (and XML free ;), maintainable and allowing complex scenarios like rule merging, unused object tracking, conversion of checkpoint exclusion groups, massive rule editing, AppID conversion … . It will work seamlessly on PAN-OS local xml config file or PAN-OS API.

https://github.com/PaloAltoNetworks/pan-os-php

It is a powerful PHP library. If you know how to write PHP code, you can do a lot of magic with this tool. Even if you don’t know PHP (like me), you can still do many things becasue developer prepare few, ready to be used scripts for us -> https://github.com/PaloAltoNetworks/pan-os-php/tree/main/utils

• Pan-OS-Python and Pan-python

There are two python libraries for panos api handling, pan-os-python and pan-python. Pan-os-python is a package to help interact with Palo Alto Networks devices (including physical and virtualized Next-generation Firewalls and Panorama). The pan-os-python SDK is object oriented and mimics the traditional interaction with the device via the GUI or CLI/API.

As I am big python fan in this blog post I’ll be using pan-os-python library to show you how to create simple object cleaner for Palo Alto Networks firewall.

Test Use Cases

First use case: there are 12 address objects configured and 4 of them are duplicates.

Some of those address objects are used in security policy rules.

The objective is to develop a script which will identify all duplicates, replace them in security rules and finally delete them.

Second use case: some of address objects are members of the address groups:

And those groups are used in new security policy rule:

Building the script

Because pan-os-python is not a standard python library, it is mandatory to install it before usage. It can be installed

> pip install pan-os-python

Now we are ready to start scripting. Let’s build import section of the script. For both use cases we need to import following modules from panos library:

• firewall – will handle connection to the Palo Alto Networks firewall
• objects – will handle objects
• policies – will handle security rules

The first thing after import section is to create a python object representing our Palo Alto Networks firewall. To create an object, we need to create a connection to the firewall, and to create a connection we need login credentials. So let’s create few variables.

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

Login details section is pretty self-explanatory. It is optional, because you could pass credentials and firewall address directly to the object generator. But have credentials defined as variables is more flexible.

Now let’s generate our firewall object. To do this we will use Firewall class.

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

How does fw object looks like in iPython?

Right now we have import section created, login details assigned to the variables and a firewall object created. We are ready to retrieve all address objects (huge number – 12 😉 ) and security rules in order to be able to work on this data. First address objects, let’s represent address objects by a variable called original_objects.

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)

We are using refreshall() method from AddressObject class in order to get Address Objects from firewall’s candidate configuration. Method refreshall() was issued with two arguemnts fw and add=False. We pass fw object in order to get config from live device, add=False is passed because we don’t want to add our address objects to fw object. Address objects will be returned in the form of python list.

Address Objects are retrieved and assigned to the original_objects variable, now it is time to get security rules from the firewall. First we need to create rulebase object using Rulebase class and add to our firewall object. Then using refreshall() method called from SecurityRule class we will get all security rules from the firewall and assign them to the security_rules variable (type list).

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

Because both original_objects and security_rules variables are lists we can iterate over them. So let’s do this in iPython to print objects names:

…and Security Rules:

Let’s pick first address from the original_objects list and see how it is structured. I will assign it to the variable called obj1 and use about() method on this variable to see object’s structure.

Oh! it returns a dictionary, so it means we could pick object’s IP address like this: obj1.about()[‘value’]. But there is a simpler way to do this: obj.value

Duplicate objects are objects having the same value as other objects. To detect duplicates, we need to compare values of all objects, it doesn’t matter which object was created first, only unique objects must remain in the configuration.

Let’s start with creating some addtional lists and dictionaries.

• unique_objects – (list) will contain only unique address objects
• duplicates – (list) will contain duplicated address objects
• duplicates_replacement – (dictionary) will contain information which unique address object should be used to replace duplicate address object in the configuration

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}

We need to take object A and object B from unique_objects list and compare them. First let’s do an excercise: take object A and object B and print both values and names.

Because we are running for loop inside for loop, at some point of iteration, two the same objects will be assigned to variable a and b. Take a look at the sample output screenshot above. We want to avoid situation when Proxima object assigned to variable a will be compared to Proxima object assigned b will be compared and detected as a duplicate. In Palo Alto Networks firewall configuration, two address objects cannot have the same name. Let’s implement this rule in our exercise code:

The difference is visible – Proxima is in pair with Proxima, Peleus is not in pair with Peleus etc. But still, we print everything what is inside unique_objects. It is a time to limit output to objects with the same values. So we need to take a.value and compare it with b.value and if these are the same – print them.

The output is now much smaller and I’m able to caption it on the screenshot. We can see that only objects with the same values are printed. But there is still a problem, take a look at first and last line of the output. We have Proxima-Maelstrom and Maelstrom-Proxima pairs: we have detected duplicates in both pairs, but the objects in the pairs are the same, the order is just different. This happend because after first detection (1st line) we left duplicate in the unique_objects list. Time for a fix: if all our conditions in if statement are true, we will consider object b as a duplicate and we’ll remove it from unqiue_objects list.

Let’s remove our print function and caption detected duplicates in the duplicates list. Also it is time to update our duplicates_replacement dictionary. We need to add our duplicates do the firewall object as well – it is needed for object deletion. Having above modifications implemented, our code should looks like below:

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}

for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)

Run this code and see how our unique_objects, duplicates, duplicates_replacement variables looks like.

Fix For duplicates_replacement

Do you see the problem? It is in duplicates_replacement dictionary. We want to replace Nebula duplicate object with Odysseus object… which is also a duplicate and should be replaced by Clover object. Need to fix this problem. We could find unique object (in unique_objects list) that has the same value as our Nebula duplicate and put it as a replacement in duplicates_replacement. Other option is to check if ‘Odysseus’ is in duplicates list and if it’s True, find ‘Odysseus’ entry in duplicates_replacement, take the value ‘Clover‘ and copy it to ‘Nebula’ entry. We will implement the first option – validating in unique_objects list.

To implement first option we need to iterate over unique_objects list and then over duplicates_replacement dictionary. Inside this nested for loop we need to compare unique object value and name, with duplicate value and replacement name from duplicates_replacement. If values are the same but names are different we need to update ‘ReplaceWith’ key with value from unique object. So in python:

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}

for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)
    # Fix for a problem in duplicates_replacement
    for k, v in duplicates_replacement.items():
        if a.value == v['Value'] and a.name != v['ReplaceWith']:
            v['ReplaceWith'] = a.name

At this moment we have a script which is able to identify duplicated objects and replecement objects for duplicates. Now it is time to take a look at security rules and see if duplicates we identified are in use in security rules. We already have a variable containing list of security rules as objects.

All we have to do is to iterate over our security policy rules and check if a duplicate address exists in a source, a destination or in both places in a rule.

Before we’ll start iterating over security rulebase, let’s pick first rule from a list and see how it is structed in a python.

When we run about() method on a rule, we can see it returns dictionary formated output. Our main interest is in source and destination entries, it looks like corresponding values are lists. Let’s see..

Now let’s write some code that will check the rules and print out a message on a screen when duplicate usage is detected.

Perfect! Code is detecting duplicates in our rules and printing out messages. Now there are few elements to implent. We need to:

1. Remove a duplicate from a rule if replacement already exists in a source or a destination
2. Replace a duplicate object with a replacement object based on a information from duplicates_replacement.
3. Put modified rules in seperate list for latter apply process.

To implement element number 1 from the list we need to add if statement inside of our if duplicate.name in rule.source statement, which will check replacement object name already exits in rule source. Example:

The same applies to the destination part:

So right now piece of code responsible for duplicate detection in security rules should look like this:

for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)

Now let’s write a code responsible for swapping duplicated objects with replacement objects.

for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
            else:
                rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)
            else:
                rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']

Don’t forget about adding modified rules to a separate list. Create a list called modified_rules and if rule is not already in the list – add it.

modified_rules = []
for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
            else:
                rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)
            else:
                rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)

And that’s it! All we have to do is push our modifications to the firewall. But first let’s see what we developed so far:

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule

# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}

for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)
    # Fix for a problem in duplicates_replacement
    for k, v in duplicates_replacement.items():
        if a.value == v['Value'] and a.name != v['ReplaceWith']:
            v['ReplaceWith'] = a.name

# Check rules for duplicates usage
modified_rules = []
for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
            else:
                rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)
            else:
                rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)

The final part is to write few lines of code that will push our changes to the firewall’s candidate configuration. Before pushing our modification it is a good practice to check if there are any. If there are no changes and we will try to apply them we will get an error. To validate if there are modifications to apply, simply check length of the duplicates list and modified_rules list.

if len(modified_rules) > 0:
    Apply rule changes.
if len(duplicates) > 0:
    Remove duplicated address objects

To apply changes we will use bulk operations to limit number of API requests send to the firewall. If you want learn more about bulk operations follow this link: https://pan-os-python.readthedocs.io/en/latest/howto.html#optimize-with-bulk-operations

So let’s complete our apply section in python code.

if len(modified_rules) > 0:
    modified_rules[0].apply_similar()
    print("*** Applied Changes to the Security Rules. ***")
if len(duplicates) > 0:
    duplicates[0].delete_similar()
    print("*** Removed duplicated address objects. ***")

It is time to put everything together! We can implement small feature at this point and add a timer that will show us how long script was running in order to clean up our config.

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule
import datetime

start = datetime.datetime.now()
# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'

# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)

# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)

# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}

for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)
    # Fix for a problem in duplicates_replacement
    for k, v in duplicates_replacement.items():
        if a.value == v['Value'] and a.name != v['ReplaceWith']:
            v['ReplaceWith'] = a.name

# Check rules for duplicates usage
modified_rules = []
for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
            else:
                rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)
            else:
                rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)

# Apply Section
if len(modified_rules) > 0:
    modified_rules[0].apply_similar()
    print("*** Applied Changes to the Security Rules. ***")
if len(duplicates) > 0:
    # https://github.com/PaloAltoNetworks/pan-os-python/issues/510
    #duplicates[0].delete_similar()
    for duplicate in duplicates:
        duplicate.delete()
    print("*** Removed duplicated address objects. ***")

print("### Cleaning took: {} ###".format(datetime.datetime.now() - start))

Tests

Before running our script let’s log into the firewall and take a look at address objects and security rules once again.

Let’s run the script now:

And now let’s see the changes in the GUI.

We can see that insted of 12 address objects, we have 8 address objects. 4 duplicates were removed during the clean up proces performed by the script. Let’s see how security policy looks like:

In the first rule ‘ApisDioscuri’, ‘Maelstorm’ got replaced by ‘Proxima’ and in rule ‘NemoAchelois’, ‘Nebula’ got replaced by ‘Clover’.

Now you can run ‘Validate’ job on firewall and if all is ok ‘Commit’ candidate config to running conifg. Of course you can add those actions to the script as well!

Summary

The whole cleaning operation took only ~2 seconds! Our script is much faster than this guy:

This blog post got quite large, so I will continue with Address Groups in a next one.

If you are reading this sentence, you probably reached to the end of this post. Thank you for your attention! Leave a comment if you like 😉

Krzysztof.