PyPaloCleaner – Part 3

Table of contents

  1. Intro
  2. Case Study Overview
  3. Script Building
  4. Test
  5. Summary

This is a continuation of Part 2 you can find here:

PyPaloCleaner – Part 2

Intro

In the previous episode of the series (Part 2) we covered how to identify and replace duplicated objects. This time we will focus our attention on detecting duplicated address groups.

Case Study Overview

For the Part 3, I’ve slightly modified test configuration. In Address Groups section, three new groups appeared; ‘Apple‘, ‘Pizza’ and ‘BigBang’:

New Address Groups configuration

‘Apple‘ and ‘Banana‘ groups are duplicated. They contain different objects, but those objects have the same IP assigned, so after running the script both ‘Apple‘ and ‘Banana‘ will have the same object as a member.

‘BigBang’ and ‘Pizza‘ are groups containing other address group as a member, along with some address objects.

New Address Groups are used within Two new security Rules; ‘Jandoo’ and ‘Astraeus Cloud‘:

New Security Rules

Script Building

Here is the code developed in Part 1 & Part 2. I’ve highlighted lines 46, 47 and 48 – there, we will put our code developed in this blog post.

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule
import datetime
  
start = datetime.datetime.now()
# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'
  
# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)
  
# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)
original_groups = AddressGroup.refreshall(fw, add=False)
 
# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}
 
# Find Duplicated Address Objects
for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)
    # Fix for a problem in duplicates_replacement
    for k, v in duplicates_replacement.items():
        if a.value == v['Value'] and a.name != v['ReplaceWith']:
            v['ReplaceWith'] = a.name
 
# Find Duplicated Address Objects in Address Groups
for og in original_groups:
    for duplicate in duplicates:
        if duplicate.name in og.static_value:
            og.static_value[og.static_value.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            og.apply()
# Start of Part 3 Section Here

# End of Part 3 Section Here  
# Check rules for duplicates usage
modified_rules = []
for rule in security_rules:
    for duplicate in duplicates:
        if duplicate.name in rule.source:
            print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
                rule.source.remove(duplicate.name)
            else:
                rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)
        if duplicate.name in rule.destination:
            print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
            if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
                rule.destination.remove(duplicate.name)
            else:
                rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            if rule not in modified_rules:
                modified_rules.append(rule)
  
# Apply Section
if len(modified_rules) > 0:
    modified_rules[0].apply_similar()
    print("*** Applied Changes to the Security Rules. ***")
if len(duplicates) > 0:
    duplicates[0].delete_similar()
    print("*** Removed duplicated address objects. ***")
  
print("### Cleaning took: {} ###".format(datetime.datetime.now() - start))

Finding Duplicated Address Groups

Here we will detect all address groups with the same values. The mentioned values are basically the group members. We need to take two groups, compare their members and if group members are the same we need to remove duplicate from the list of original groups and put it into duplictates list.

Before running any comparision code, we need to have our ‘storage’ prepared, our list and dictionary were information about duplicates will be kept.

# Part 3
group_duplicates = []
group_duplicates_rep = {}

We are ready to take group_a and group_b from original_groups list do something with them. Maybe let’s begin with a simple print() that will print out group names to the screen. Let’s see it in action in ipython:

Address groups presented in pairs.

Now let’s embeed if statement into our for loop to printout only groups that have the same members. Before we build if statement logic, we need to check how a particular address group object is build in python. In ipython run:

Structure of the Address Group.

We know how the address group is structured so now we are able to build a logical statement that will check if the address groups have the same value.

First of all we need to make sure to not compare the same groups. For example: at the beginning of for loop runtime, Banana address group will be assigned to variable group_a and the same Banana address group will be assigned to another variable group_b. If this will happen our if statement must return False, because otherwise it will detect a duplicate – a false-positive duplicate. Because address group names must have unique values, we can use them to check if variables group_a and group_b are the same objects. First part of if statement must check if name of the group_a is diffrenet from name of the group_b. So the first part of if statement will be:

(group_a.name != group_b.name)

In the next part of the if statement logic we must check if group_a and group_b values (group members) are the same. We’ve seen already that address group has a static_value field. So we need to take group_a.static_value and compare it with group_b.static_value. The second part of if statement will be:

(group_a.static_value == group_b.static_value)

The last part of the if statement is to check if the group has been already detected as duplicate. Essentialy we need to check if group_b isn’t in group_duplicates list. So the third part of if statement will be:

(group_b not in group_duplicates)

If we combine above and put logical and between them, we will get:

(group_a.name != group_b.name) and (group_a.static_value == group_b.static_value) and (group_b not in group_duplicates)
Very True

If all 3 logical groups returns True, our if statement is also True – this means we have detected a duplicate. Let’s run python code and printout group names only when there is a duplicate in group_a and group_b pair:

Only duplicate names are printed to the screen.

That is better, our output is much smaller. We detected duplicates in Banana – Apple pair and in Apple – Banana pair… Yes, there are the same pairs, just in different configuration. To prevent this we need to remove a duplicate from a original_groups list. Adding one line of code will result with the following:

Fixed problem with doubled pairs

Perfect! Now all we have to do is remove print() function and replace it with 3 lines of code that will do the following:

  • add duplicate address groups to the group_duplicates lists
  • add information about the duplicate and the replacetent to group_duplicates_rep dictionary
  • add duplicate address group to the firewall object (this will let us remove duplicate from the firewall config)

Implementation of above actions in python should look like this:

Implementation in python

Below I will post Part 3 code we developed so far. It inclueds a fix for duplicates replacement that I covered already in Part 1.

# Part 3
group_duplicates = []
group_duplicates_rep = {}
for group_a in original_groups:
    for group_b in original_groups:
        if (
        (group_a.name != group_b.name) and \
        (group_a.static_value == group_b.static_value) and \
        (group_b not in group_duplicates)):
            group_duplicates.append(group_b)
            group_duplicates_rep[group_b.name] = {"Value": group_b.static_value, "ReplaceWith": group_a.name}
            fw.add(group_b)
            original_groups.remove(group_b)
    # Fix for a problem in duplicates_replacement
    for k, v in group_duplicates_rep.items():
        if (group_a.static_value == v['Value']) and (group_a.name != v['ReplaceWith']):
            v['ReplaceWith'] = group_a.name

Detecting Duplicated Groups As Other Group Members

The task here is to take our original_groups and check if duplicated groups, we identified previously, exists as the members. Again we need to iterate over original_groups and group_duplicates and check if duplicate.name is in original_groups.static_value. 

for og in original_groups:
    for duplicate in group_duplicates:
        if duplicate.name in og.static_value:
            # do something

Let’s run above code in ipython and if if statement will return true, we will print a message to a screen.

An Apple in a Pizza. Is a pineapple better?

Oh, we have something. When duplicate is detected we need to perform few actions (instead of just printing a message to a console):

  • replace duplicate group with a replacement object
  • add modified group to the firewall object
  • apply changes to the candidate config
  • remove modified (and applied) group from the firewall object

In python code it will look like

# Part 3
group_duplicates = []
group_duplicates_rep = {}
for group_a in original_groups:
    for group_b in original_groups:
        if (
        (group_a.name != group_b.name) and \
        (group_a.static_value == group_b.static_value) and \
        (group_b not in group_duplicates)):
            group_duplicates.append(group_b)
            group_duplicates_rep[group_b.name] = {"Value": group_b.static_value, "ReplaceWith": group_a.name}
            fw.add(group_b)
            original_groups.remove(group_b)
    # Fix for a problem in duplicates_replacement
    for k, v in group_duplicates_rep.items():
        if (group_a.static_value == v['Value']) and (group_a.name != v['ReplaceWith']):
            v['ReplaceWith'] = group_a.name

# Find Duplicated Address Groups in Address Groups
for og in original_groups:
   for duplicate in group_duplicates:
      if duplicate.name in og.static_value:
         og.static_value[og.static_value.index(duplicate.name)] = group_duplicates_rep[duplicate.name]['ReplaceWith']
         fw.add(og)
         og.apply()
         fw.remove(og)

Update Security Rules

So now that we have identified our duplicated address groups, we need to replace them in security policy rules. Because this action is very similar to the previously performed in the Part 1, I will simply put updated “# Check rules for duplicates usage” section below:

# Check rules for duplicates usage
modified_rules = []
for rule in security_rules:
   # Change duplicated address objects in the security rules
   for duplicate in duplicates:
      if duplicate.name in rule.source:
         print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
         if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
            rule.source.remove(duplicate.name)
         else:
            rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
      if duplicate.name in rule.destination:
         print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
         if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
            rule.destination.remove(duplicate.name)
         else:
            rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
   # Change duplicated address groups in the security rules
   for grp_duplicate in group_duplicates:
      if grp_duplicate.name in rule.source:
         print("Duplicate '{}' detected as a source in the rule '{}'".format(grp_duplicate.name, rule.name))
         if group_duplicates_rep[grp_duplicate.name]['ReplaceWith'] in rule.source:
            rule.source.remove(grp_duplicate)
         else:
            rule.source[rule.source.index(grp_duplicate.name)] = group_duplicates_rep[grp_duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
      if grp_duplicate.name in rule.destination:
         print("Duplicate '{}' detected as a destination in the rule '{}'".format(grp_duplicate.name, rule.name))
         if group_duplicates_rep[grp_duplicate.name]['ReplaceWith'] in rule.destination:
            rule.destination.remove(grp_duplicate)
         else:
            rule.destination[rule.destination.index(grp_duplicate.name)] = group_duplicates_rep[grp_duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)

Update Live Device

And finally we have to update our live device candidate configuration with modifications we introduced.

# Apply Section
if len(modified_rules) > 0:
    modified_rules[0].apply_similar()
    print("*** Applied Changes to the {} Security Rules. ***".format(len(modified_rules)))
if len(group_duplicates) > 0:
    group_duplicates[0].delete_similar()
    print("*** Removed {} duplicated address groups ***".format(len(group_duplicates)))
if len(duplicates) > 0:
    duplicates[0].delete_similar()
    print("*** Removed {} duplicated address objects. ***".format(len(duplicates)))

Test

Before running any tests, let’s post complete code we created so far, with highlighted lines we created/updated in this Part:

# Import Section
from panos.firewall import Firewall
from panos.objects import AddressObject, AddressGroup
from panos.policies import Rulebase, SecurityRule
import datetime
  
start = datetime.datetime.now()
# Login Details
HOSTNAME = '10.20.30.94'
USERNAME = 'bot'
PASSWORD = 'BotPassword123'
  
# Initiate firewall object
fw = Firewall(HOSTNAME, USERNAME, PASSWORD)
  
# Retrieve Data from the firewall
original_objects = AddressObject.refreshall(fw, add=False)
rulebase = fw.add(Rulebase())
security_rules = SecurityRule.refreshall(rulebase)
original_groups = AddressGroup.refreshall(fw, add=False)
 
# Cleaning Section
unique_objects = original_objects.copy()
duplicates = []
duplicates_replacement = {}
 
# Find Duplicated Address Objects
for a in unique_objects:
    for b in unique_objects:
        if (a.value == b.value) and (a.name != b.name) and (a not in duplicates):
            duplicates.append(b)
            duplicates_replacement[b.name] = {"Value": b.value, "ReplaceWith": a.name}
            fw.add(b)
            unique_objects.remove(b)
    # Fix for a problem in duplicates_replacement
    for k, v in duplicates_replacement.items():
        if a.value == v['Value'] and a.name != v['ReplaceWith']:
            v['ReplaceWith'] = a.name
 
# Find Duplicated Address Objects in Address Groups
for og in original_groups:
    for duplicate in duplicates:
        if duplicate.name in og.static_value:
            og.static_value[og.static_value.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
            og.apply()
# Start of Part 3 Section Here
# Part 3
group_duplicates = []
group_duplicates_rep = {}
for group_a in original_groups:
    for group_b in original_groups:
        if (
        (group_a.name != group_b.name) and \
        (group_a.static_value == group_b.static_value) and \
        (group_b not in group_duplicates)):
            group_duplicates.append(group_b)
            group_duplicates_rep[group_b.name] = {"Value": group_b.static_value, "ReplaceWith": group_a.name}
            fw.add(group_b)
            original_groups.remove(group_b)
    # Fix for a problem in duplicates_replacement
    for k, v in group_duplicates_rep.items():
        if (group_a.static_value == v['Value']) and (group_a.name != v['ReplaceWith']):
            v['ReplaceWith'] = group_a.name

# Find Duplicated Address Groups in Address Groups
for og in original_groups:
   for duplicate in group_duplicates:
      if duplicate.name in og.static_value:
         og.static_value[og.static_value.index(duplicate.name)] = group_duplicates_rep[duplicate.name]['ReplaceWith']
         fw.add(og)
         og.apply()
         fw.remove(og)
# End of Part 3 Section Here  

# Check rules for duplicates usage
modified_rules = []
for rule in security_rules:
   # Change duplicated address objects in the security rules
   for duplicate in duplicates:
      if duplicate.name in rule.source:
         print("Duplicate '{}' detected as a source in the rule '{}'".format(duplicate.name, rule.name))
         if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.source:
            rule.source.remove(duplicate.name)
         else:
            rule.source[rule.source.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
      if duplicate.name in rule.destination:
         print("Duplicate '{}' detected as a destination in the rule '{}'".format(duplicate.name, rule.name))
         if duplicates_replacement[duplicate.name]['ReplaceWith'] in rule.destination:
            rule.destination.remove(duplicate.name)
         else:
            rule.destination[rule.destination.index(duplicate.name)] = duplicates_replacement[duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
   # Change duplicated address groups in the security rules
   for grp_duplicate in group_duplicates:
      if grp_duplicate.name in rule.source:
         print("Duplicate '{}' detected as a source in the rule '{}'".format(grp_duplicate.name, rule.name))
         if group_duplicates_rep[grp_duplicate.name]['ReplaceWith'] in rule.source:
            rule.source.remove(grp_duplicate)
         else:
            rule.source[rule.source.index(grp_duplicate.name)] = group_duplicates_rep[grp_duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
      if grp_duplicate.name in rule.destination:
         print("Duplicate '{}' detected as a destination in the rule '{}'".format(grp_duplicate.name, rule.name))
         if group_duplicates_rep[grp_duplicate.name]['ReplaceWith'] in rule.destination:
            rule.destination.remove(grp_duplicate)
         else:
            rule.destination[rule.destination.index(grp_duplicate.name)] = group_duplicates_rep[grp_duplicate.name]['ReplaceWith']
         if rule not in modified_rules:
            modified_rules.append(rule)
  
# Apply Section
if len(modified_rules) > 0:
    modified_rules[0].apply_similar()
    print("*** Applied Changes to the {} Security Rules. ***".format(len(modified_rules)))
if len(group_duplicates) > 0:
    group_duplicates[0].delete_similar()
    print("*** Removed {} duplicated address groups ***".format(len(group_duplicates)))
if len(duplicates) > 0:
    duplicates[0].delete_similar()
    print("*** Removed {} duplicated address objects. ***".format(len(duplicates)))
  
print("### Cleaning took: {} ###".format(datetime.datetime.now() - start))

Firewall configuration is reverted to the running – so we are in initial state, ready for a test. Let’s run our code in ipython:

Code execution is a sucess!

Here are the results:

  • Apple group was removed from the configuration
  • Apple group was replaced by Banana group in Pizza Group
  • Apple group was replaced by Banana group in Astraeus Cloud
Apple group removed and replaced by Banana in Pizza 🙂
An Apple replaced with a Banana 😉

Summary

That’s it for this blog post. Not sure what will be in the next Part, maybe service objects & groups and NAT rules.

If you are reading this sentence, you probably reached to the end of this post. Thank you for your attention! Leave a comment if you like 😉

 Follow us on Twitter and Instagram.

Krzysztof.

Posted

in

,

by

Krzysztof

Tags:

, , , ,

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *