Automation by PowerShell, Bash Script, and Python

In the IT world, we all agree on the fact that automation is not a buzzword anymore; but how much we know and we can leverage this to the benefit of our manual tasks is a matter of importance. Therefore, at first, let us assume that we have understood the importance of automation. If so, the next stage would be finding a way to start automating our manual tasks if need be at all. As daunting as it sounds, knowing the fundamentals of your chosen tool for automation and programming helps you to do your job easier and quicker.

Let’s start automating

To make our life easier, I choose three key tools as the great means of automating boring or tedious tasks including PowerShell, Bash script, and Python. In the beginning, we go through the basics of PowerShell followed by the fundamentals of Bash script to get to know how to script and then automate using both tools. Lastly, Python and its abilities will be discussed how to be utilized for automation.

PowerShell

PowerShell, as the name indicates, is a powerful cross-platform (Version 6 and later) tool and framework by Microsoft that helps automate tasks and manage OS configurations with the help of a command-line shell and its native scripting language. The older versions of PowerShell, by default, are installed on Windows 7, Server 2008 R2, and Windows 8 whereas the latest cross-platform version (That is called PowerShell Core) is only installed on Windows 10. For Linux and macOS, however, it must be manually installed. Thankfully, PowerShell commands consist of not only the native commands called command lets (Cmdlets), corresponding aliases of the Cmdlets, also Legacy CMD commands, and a variety of basics Unix Shell commands in only one package. As an example, if you wish to print a list of directories in the current directory, you can issue Get-ChildIte as a native command. The good news is that to accomplish this task, you can also use the legacy CMD command for this purpose, dir, or ls for Linux users with the same result.

Before delving into PowerShell scripting, first, to become familiar with the most common and important Cmdlets, I kindly suggest reading my article, Essential Windows commands, and shortcuts for CMD, PowerShell.

To start writing our command-let or script, we can use any text editor from the simplest option, notepad, to the most advanced one that is Microsoft Visual Studio Code with the autocomplete feature for all the PowerShell’s commands. Thankfully, Microsoft provides us with a special tool for PowerShell scripting named Windows PowerShell Integrated Scripting Environment (ISE). This tool benefits from different sections including a Shell view, Text view and Commands view and can be accessed by typing ise in the command prompt as well as the start menu.

1. Preparation

After writing your script, the file’s format should be .ps1. To run this script, firstly, we need to set the execution policy of PowerShell, otherwise, this file cannot be run by just double-clicking like in the past we did on .bat files.

Execution policy in PowerShell has 2 core parameters:

-executionpolicy: It specifies a policy of scope in Windows environment whereas, for non-Windows computers, the default execution policy is set on unrestricted and can’t be changed as stated by Microsoft as of PowerShell v.6 onward. These policies can be seen in the next image.

-scope: It specifies the scope of each execution policy and can be seen in the following image by issuing get-executionpolicy –list based on their precedence.

For example, to set an execution policy on our local machine to require remote and external scripts to be signed by a certificate authority, this command should be used.

Set-executionpolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine

2. Variables

A variable in PowerShell like in all other programming languages is a location in memory with a name in which a value can be written to be later referenced and manipulated. Quite interesting, we can create a variable in which an output of a command(s) is saved. To create a variable, we only need to use $ followed by a string, and to feed this variable, an equal sign followed by our data that can be rather a comment, an arithmetic calculation, and/or a mere string.

$variable = string / 2 + 2 / get-service

However, PowerShell has many reserved variables that cannot be altered and their names cannot be used again. To see them, Get-variable has to be issued. Thankfully, variables in PowerShell are flexible to save all types of data including string, integer, Boolean unless you specify the data type of a variable as you can see in the following examples.

$variable = ‘ string $(1 + 2)‘ # the data is considered a string

$variable = “ string $(1 + 2)“ # the data is considered integer

[string]$variable

In PowerShell, there are also some environment variables by which an OS can hold necessary data like the computer’s hostname, users’ information, etc. For more information, try to use the following examples.

Get-childitem env: # list of all the environment variables

$env:COMPUTERNAME # read an environment variable that is the hostname of computer

$env:USERNAME # read the current username logged in

Another interesting fact about variables in PowerShell is that if we save the output of a command, we can then utilize and read any properties of a command. However, the question is how to find a property(s) of a command. Well, it is so easy that we just need to issue this: command | Get-Member.

Take a look at this example of variables and properties.

$services = get-service # Hold all the services installed on a computer in this variable

$services.Name # Load only the name (As a property of Get-service) of all the services

Using this feature in a condition:

If ($variable.property -eq ‘string‘ ) {

            Command #perform an action

}

3. Hello World

Now, it is time to start scripting with PowerShell. To do so, we need to be familiar with both PowerShell Cmdlets and the scripting structure. Therefore, in this article, we briefly go through different parts of PowerShell scripting format with helpful examples of Cmdlets to get the best of both worlds. At first, one fundamental command is how to remotely connect to different hosts in a network and then execute a command(s). For example, to install a Windows-based service, run Internet Explorer for several remote computers and copy a file remotely to our target hosts, we issue the following commands.

$servers=’host1’, ‘host2’ # holding the names of two systems

$servers = New-pssession –computername $servers # create two sessions for two remote hosts

$servers | foreeach {start iexplore http://site.com} #start running IE for each computer

$servers | foreach=object {copy-item .\name.txt –destination \\$_\C$\directory\… } #copy a file to different hosts

Invoke-command –session $ servers {install-windowsfeature program}# such as web-server (IIS)

4. Functions in PowerShell

The next step is to know how to create our function or command let in PowerShell so that our tasks can be effectively automated. The name of a function in PowerShell follows verb-noun format like Get-service and Get-help in such a way that verbs indicate the action a command performs while a noun is an entity an action is based on. For the sake of consistency between our commands and standard commands, it is also recommended by Microsoft to follow this format for our functions. To see a list of all standard verbs issue this command: Get-verb.

The format of a function in PowerShell is as follows:

Function verb-noun {

[cmdletbinding()] # Optional, but it gives a function more options like -verbose

Param (  

            [parameter(Mandatory=$true)] # To make our parameter mandatory to use

            [string[]] # [] = for multiple value // ‘’ to get input

            [switch] # optional, it works only as a parameter with no value

$targets=’ default value if any ’,  

)

$targets = read-host –prompt ‘plz enter an input’ # A way to input data and save it to a variable

$targets | foreeach {start iexplore http://site.com} #start running IE for each computer

Return $string

}

A parameter in a function allows a user to input data and feed the variable(s) inside a script as opposed to editing a script. Also, a parameter can manipulate the output of a function based on a user’s interest.

5. Conditional statement

Conditions in PowerShell like other programming languages include If statement and Switch.

The format of If statement is as follows.

$condition = $true

if ( $condition )

{

    Write-Output “The condition was true”

}

Elseif { } # otherwise do this action

Else { } # perform an action if none of the above is true

5.1 Operators in If statement:

There are plenty of operations in PowerShell that can be utilized and listed by Microsoft but the most important ones are collected here.

-eq = equal (case-insensitive equality)

-ne = not equal

-gt = greater than

-le = less than

If ($variable.property -eq ‘string‘ ) {

            Command #perform an action

}

5.2 Logical operations

-not / !

If (-not (test-path –path address –erroraction -stop)) { #test whether or not an location exists

            New-item –itemtype directory –path address

            Add-content –path address of a log file –value  “text” # writing to our log file

            Write-warning “string”

            Write-error “string”

-and

if ( ($age -gt 13) -and ($age -lt 55) )

5.3 Switch statement

To check multiple conditions, we can use a Switch statement. It is equivalent to a series of If statements, but it is simpler.

$input = Read-Host

switch (“$input”)

{

    1 {“one”; Break}

    2 {“two”; Break}

    3 {“three”; Break}

    4 {“four”; Break}

    Default {

        “No matches”

    }

}

6. Loop statement

For (i = 0; 1 –le / -ge 10; i++ )   # –le = less than / -ge = greater than

{ Perform an action }

foreach=object (variable in variable2) { }

# using this method when we do not know the number of iteration

Get-Process | ForEach-Object ProcessName # priting only the process name of each process

7. Error handling

7.1 Validation by If

$variable = 1

if ($variable > 1) {

    Write-Host “It is bigger than 1”

}

else {Write-Host “$variable is smaller than 1”}

7.2 Try, Catch, and Finally

This block in PowerShell benefits us to capture terminating errors within our function.

The Try block contains the code that is to execute, and catch any potential errors that occur.

The Catch block contains the code is to execute after a terminating error has happened. All the current errors will be also accessible via variable $_  ($Error) that is an array.

The finall block contains the code that should be run after the event has occurred whether an error or not.

Try {

            Command -ErrorAction Stop # to catch a terminating error

}

Catch {

            Throw

            Write-warning –Message “Ran into an issue!”

            Write-warning $Error[0] # Displaying the most resent error

Send-MailMessage -From it@test.com -To Admin@test.com -Subject “error!” -SmtpServer smtp.test.com

            Break

}

Finally { # Always run this in the end }

8. Run your script

For security reasons, when we run a PowerShell script file it doesn’t run unless we change this behavior in the Windows register. However, we have two options to run our script equipped with PowerShell commands. The first is to run it manually on the PowerShell terminal using ./script.ps1 and the second is to run your file automatically which makes more sense as the whole point of creating a script file is the automation of your manual tasks. To do the second way, just create a shortcut and add this code to the Target field of the shortcut:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command “& ‘C:\your-script-location.ps1 -noprofile'”

Now, once you run the shortcut, the script starts running as well.

Bash script

Bash is a command-line interface (CLI) based on UNIX shell that has been also the default user shell of Linux distributions since 1989 to enable users to interact with the services and functions of a Unix-based operating system. Designed to be run and manipulated by UNIX shell, Bash script is a scripting language to automate shell commands, file and text manipulation, and so forth that we can do manually using Bash terminal. To create our first script file, we just need to follow these important rules.

1. The file format of a Bash script is .sh and it must start with #!/bin/bash by which the interpreter to run the script is specified.

2.  To properly run the file after saving your Bash script file, we have to make it executable by issuing chmod +x your-script.sh and then type ./script.sh that would otherwise run by typing: bash your-script.sh.

1. Variables

For Bash script, contrary to PowerShell, we just need to write a string without $ and feed it with our value. Being said, when a variable is used, $ before the variable is inevitable. In the following, you can see different forms of variables that can be defined and used for a Bash script.

Name=”arash”

Variable=2+2

echo “Hello, $name!”

Interestingly, an output of a Bash command, whoami in this case, can be utilized like a variable.

echo “Hello, $(whoami)”

More surprisingly, an input from a keyboard can also be saved to a variable.

read name

echo “Hello, $name!”

Take two following scripts as two examples of using variables in Bash script.

1.1 A script that check the existence of a user      

#!/bin/bash

read -p “Enter a username to be checked:” user # read from keyboard and save it to variable

if grep –q -w “^$user*” /etc/passwd # -q in grep return true or false without printing

then

  echo “Username $user exists!” # print a variable

else

  echo “Username: $user does not exists.”

Fi

1.2 A script to create a user and given password

#!/bin/bash

Echo “Account for $username is being created”

Useradd $username

Echo “Password for $username is being set”

Passwd $username

Echo “User account: $username is ready for use”

To run the script the following command along with a username that passes a name to our positional variable, $username, should be used.

Sudo ./script.sh username

2. Conditional statements

Conditions in Bash script consist of If else and Case as collected in the following with examples.

2.1 If statement

If [ condition ] && [ condition ] # AND

If [ condition ] || [ condition ] # OR

If [ $string == ‘string’ ] # comparision

Then   

            Commands/actions

Else

            Commands/actions

Elif [ condition ] # optional

Then

            Commands/actions

fi

2.2 Removing Duplicate Lines from Files

#! /bin/sh

echo -n “Enter Filename-> “

read filename 

if [ -f “$filename” ];

then

  sort $filename | uniq | tee sorted.txt

else

  echo “No $filename in $pwd, please try again”

fi

2.3 Check If a File Exists

#!/bin/bash

filename=$1

if [ -f “$filename” ]; then

  echo “File exists”

else

  echo “File does not exist”

fi

2.4 Case statement

The Case statement is similar to the Switch statement in C. Instead, the Bash script checks the condition and controls the flow of the program based on different given patterns.

Case expression in # An expression can be a $variable

            Pattern) # A pattern can be a string or a pattern matching

                        Command/action

            ;;

            Pattern1 | pattern2 | pattern3) # multiple patterns

                        Command/action

                        break

            ;;

            *) # default case that matches anything

Esac

3. Loops

Loops in Bash script are For and While that give us a flexible choice to create our scripts with infinite and/or finite loops which are as follows.

For x in 1 2 3 4 / text1 text2 text3

For x in {star..end..incremen}

For x in {1..10} # a series of numbers

For x in $(command / ls *.txt) # giving the output of a command or variable

Do

            Commands

Done

for x in {1..30} = a loop with 30 iteration

do

sudo command

echo “done!”

done

While condition / true

While [ “$string” != “string” ] && [ “$string = “” ];  # and operation

Do

            Commands

Done

4. Function in Bash script

A function in Bash script can be created in two following ways.

function_name () {

  commands

  return number/string #optional

}

function function_name {

  commands

}

To call or execute a function, the name of a function must be individuallyin place somewhere in a bash script. Also, to print the return value that is optional in a function structure, echo $? Can be used.

5. Delay in Bash script

It is possible to pause an action/command or a part of our script in Bash script. It can be done using the sleep command and can be defined in seconds (s), minutes (m), hours (h), and days (d).

sleep NUMBER[SUFFIX]

for x in {1..5}

do

  echo “Hostname : $(hostname)”

  sleep 3s # three seconds

done

6. Run a bash script

Aside from running a Bash script locally on a Linux-based environment, Bash scripts can be run on both Linux and Windows-based machines remotely as well. To achieve this, on a Linux machine, we just need to follow the next command.

ssh user@ your-taget-machine -p port-number ‘bash -s’ < script.sh

Meanwhile, on a Windows-based computer, plink as a connection tool based on Putty should be used. To do so, Putty must be installed first, and then the following command can be run.

plink user@your-taget-machine -P port-number -m script.sh

6.1 Using PowerShell to run a bash script remotely

To make this process even easier, a permanent PowerShell alias can be created to run the Bash script remotely. To do so, first, create a profile in Powershell and then open the profile with notepad:

New-Item -Type file -Path $PROFILE -Force

notepad $PROFILE

Next, add these two commands in the text file and save the file. Now, the command can be run by using a short alias every time a PowerShell session is created. In the first command, a function is created with a name followed by the plink command and its options while the second command creates an alias with a name based on the function name we just created.

Function string2{plink user@ip -P port -m C:\your-script.sh}

New-Alias -Name string -Value string2

Linux/UNIX commands for automation

1. Cron job

The Cron daemon is a built-in UNIX time scheduler that can periodically run a file, script, and commands at a fixed time or date. To do its job, Cron reads a cron table called Crontab for shell commands in the root directory of the daemon with a special format. Each user is also able to have their cron table and schedule their jobs by using the following commands.

Crontab –e # Edit your crontab file

Crontab –u user # To create a cron table for a specific user

Crontab –l # To list all the cron jobs for the current user created before

The format of the crontab is straightforward and entails time and then command/the address of a file followed by output.

* * * * *  command  output

It should be noted that the output of a cron job by default is emailed to the location specified in the MAILTO variable. However, to turn off this option, you can add the following string, >/dev/null 2>&1, after the command field.

* * * * *  command to execute >/dev/null 2>&1

Concerning the field of time shown by five *, from the left that is “minute” to the last * that is “day of week” can be defined as shown in the next image and examples.

Symbols to defined time/date:

* matches anything

, allows us to define multiple time

– allows us to define a time range

/ allows us to defined step between each cron

Some common samples of cron jobs.

* * * * * command/script file: run an action every minute

0 * * * * command/ script file: run an action every hour

0 0 * * * command/ script file: run an action every day at 12:00 AM

0 1,13 * * * command/script file: run an action twice a day at 1 AM and 1 PM

0 */1 * * * sun command/script file: run an action every hour on each Sunday

@weekly command/script file: run once a week at midnight on Sunday morning

Additionally, defined patterns can also be utilized instead of specifying an exact time and duration that are as follows.

At command

at, as a UNIX-based command, is used to schedule a command to be run only once at a particular time in the future. To do its job, at must be followed by time based on the following format. After specifying the time and date (Optional) by pressing enter, a command has to be given to run in the future based on the given time.

at HH:MM 000000 (00 day, 00 month, 00 year)

To run a job once in one hour from now, we can use the following command.

at now +1 hour

To run a file at a specific time, this option can be used.

At time –f file

To list all the jobs, this command should be issued.

Atq or at –l

To delete a specific job, use the next command.

At – r job ID or atrm job ID

Automation by Python

Python a high-level interpreted programming language can be a great tool for automation purposes.
Python codes can be written in every text editor and run following the installation of Python on an Operating system. Among other tools we have discussed thus far, Python helps us write cross-platform scripts to control various OS features, applications, browsers, and so forth for the automation of manual tasks. To be able to run a Python script, first, they must be saved with .py format, and then they can run by typing “python script.py” in your Linux shell or PowerShell in Windows. Additionally, a Python script file can run using just ./script.py only if the file is set for execution in Linux using chmod u+x your script.py.

1. Fundamentals

1.1 Loops in Python

1.1.1 while Statement

You can make a block of code to execute time and time again using a while statement. The code in a while section will be executed so long as the while statement’s condition is true.

i = 1

while i < 6:

print(i)

if i == 5:

break

i += 1

1.1.2 for Statement

A for loop is used for an interaction over a sequence that can be a collection of either a list, a set, or a string.

for i in <collection>

    <loop body>

1.1.3 Examples of different forms of for loop:

for x in “banana”:

            print x

city = [“Brisbane”, “Sydney”, “Adelaid”]

for x in city

            print x

for x in (0, 1, 2, 3, 4, 5) # a series of numbers from 0 up to 5

for x in range(10): # a series of numbers from 0 up to 10

for x in range(2, 10): # a series of numbers from 2 up to 10

for x in range(1, 20, 2): # a series of numbers from 1 up to 20 with 2 as the stride

print(x)

if x == “10”:

break

else:

print(“Finally reached 20!”)

2. Data management automation

The following example shows you how to open a file, read/filter its content based on a condition and write the new data to another file.

#! python3

file = open(‘location/text.txt’, ‘r’)  # open a file with read-only permission

newfile = open(‘newfile.txt’, ‘w’) # create another file filtered data

the-rest = open(‘the-rest.txt’, ‘w’) # create another file for the rest of the data

For line in file:

            line_split = line.split()

            If line_split[3] == ‘string’: # a condition of the third field of each line

                        newfile.write(line) # write the filtered data to the new file

            else:

                        rest.write(line)

file.close()

newfile.close()

the-rest.close()

3. Web automation

Python benefits us from a variety of libraries for web automation and web scraping. In this article, two popular ones including Selenium and Beautiful Soap are briefly introduced and then explained how to be used with examples.

3.1 Selenium

3.2 What is Selenium?

Selenium is a Python library that suites for automation testing of web applications as it supports all popular web browsers. Selenium WebDriver is a web framework that permits you to execute cross-browser tests. This tool is used for automating web-based application testing to verify that it performs expectedly.

3.3 Basic Steps in a Selenium WebDriver Script

Install selenium

Create a WebDriver instance.

Navigate to a webpage.

Locate a web element on the webpage via locators in selenium.

Perform one or more user actions on the element.

Preload the expected output/browser response to the action.

Run test.

3.4 How Selenium WebDriver Works?

A simple script that opens Youtube and then open a channel after searching it.

#! python3

from selenium import webdriver # Import the webdriver library from Selenuim module

Driver = webdriver.chrome() # creat an instance of Chrome

Driver.get(‘https://youtube.com’) # this command navigate to a page given by an URL

Search-box = driver.find_element_by_xpath(‘//*[@id=”search”]’)

# this command finds an element in a web page. The given code must be also derived from the source code of the search box (input box) in YouTube using inspect option and copy Xpath.

Search-box.send_keys(‘YouTube channel name’)   # Importing/entering this name to the search box

Search-icon = driver.find_element_by_xpath(‘//*[id=”search-icon-legacy”]’)

# The given code must be also derived from the source code of the search button ( next to the input box) in YouTube using inspect option and copy Xpath.

Search-icon.click() # automatically click on the search button

3.2 Beautiful Soap

Beautiful soap is a Python library used for pulling data from HTML and XML with a help of an HTML parser that isolates/separates each element of a web page’s HTML depending on the name of the elements in a web page.

Firstly, after installing this library, it is necessary to import its library into our Python file.

#! python3

from bs4 import BeautifulSoup

import requests # Request library used for retrieving the HTML code of a website into Python

Secondly, using the requests library, we retrieve the HTML code of a web page to a variable.

URL = ‘https://www.easillyy.com’

page = requests.get(URL)

Thirdly, we need to introduce a parser to Beautiful soap. Although by default, it supports the HTML parser included in Python, another third-party parse such as lxml can be installed and passed to this library to extract the HTML code of our link to a variable.

soup = BeautifulSoup(page.content, ‘html.parser’)

Now the parser converts the HTML code of the given URL to a nexted structure that can be manipulated. For example:

soup.title # shows the <title> easillyy </title> of the web page

soup.title.string # shows the title of the element alone that is easilly

soup.p # shows the <p element

soup.a # shows all the <a element of the webpage

soup.find_all(‘a’) # shows all the <a element of the webpage

Each element and sub-element of a web page can be found by a class name like <span> abd <div>.

soup.find_all(‘span’, class_=’a’)

Examples:

1. Extracting all the URLs found within the <a> tag of page:

for link in soup.find_all(‘a’):

    print(link.get(‘href’))

2. Extracting the text of an element in a webpage:

text = Soup.find_all(‘span/div’, class_=’text’)

tags = Soup.find_all(‘span/div’, class_=’tags’)

            Print (text)

Print (tags)

4. Operating system control

4.1 We go through this section by giving two practical examples with respective explanations line by line. The following script creates a directory and validates the existence of it.

#! python3

import os # OS module provides functions allowing us to interact and get an OS’s information

dirname = input(“Please enter your directory name:\n”)

directory = input(“Please enter an absolute location:\n”)

path = os.path.join(directory, dirname) # join both location and the given name

os.mkdir(‘path) # create a directory in the given address

print(“Directory ‘%s’ created” %dirname)

os.listdir(‘directory’) # listing the parent directory to check the existence of the directory

4.2 The next script is set to backs up a directory with its content. It does so by generating an incremental ZIP file as a backup file with the format of string_1.zip of a given directory. Each important part is explained to show how to write an actual script for yourself. This script is also bright to you from Automate the Boring Stuff with Python

   #! python3

   import zipfile, os # importing the ZIP module and os module for directory management

   def backupToZip(folder): # Create a function with one

       folder = os.path.abspath(folder)   # Reading a folder with an absolute address

     # A loop for creating the name of a backup file with an incremental number and checking the existence of the previous backup file
      number = 1
      while True:
           zipFilename = os.path.basename(folder) + ‘_’ + str(number) + ‘.zip’
           if not os.path.exists(zipFilename):
               break
           number = number + 1

    # Create a ZIP file with ZipFile ()
    print(f’Creating {zipFilename}… ‘)
    backupZip = zipfile.ZipFile(zipFilename, ‘w’)

     # A loop for checking the entire folder tree and compress all the files in each folder using os module
    for foldername, subfolders, filenames in os.walk(folder):
         print(f’Adding files in {foldername}…’)
        backupZip.write(foldername) # Add the current folder to the ZIP file

         # A loop for adding all the files in this folder to the ZIP file.
        for filename in filenames:
            newBase = os.path.basename(folder) + ‘_’
            if filename.startswith(newBase) and filename.endswith(‘.zip’): # An exclusion for the backup file itself
                continue 
             backupZip.write(os.path.join(foldername, filename))
     backupZip.close()
     print(‘Done.’)

backupToZip(‘C:\\string’) # The directory as the source for backup process

As the main goal of this paper is to familiarize users with the basics of automation, popular tools, and ways to automate, it is not elaborately written to teach us how to write and automate complex scripts. This article, as a straightforward reference, attempts to shed some light on and scratch the surface of the different ways of automation and will be improved, elaborated further, and rectified need be. With this, I do thank you if you share your knowledge and suggestion about this article and automation as a whole with us.