Monday, November 30, 2020

Different types of Queue's in IBM MQ and its usage

 A queue is a container for messages. Business applications that are connected to the queue manager that hosts the queue can retrieve messages from the queue or can put messages on the queue.Let see the MQ objects and some theory about it .

Queues:

  • Broadly queues can be categorized under 2 names, 
  1. Predefined queue
  2. Dynamic Queues

Predefined queues

  • These are created by an administrator using the appropriate WebSphere MQ script (MQSC) commands or GUI tools. 
  • Predefined queues are permanent.
  • They exist independently of the applications that use them and survive WebSphere MQ restarts. 

Local queue (QLOCAL or QL)

  • Local queues are the only type of queue object within a queue manager that can hold messages.
  • Messages placed on the local queue, will occupy space on file system specified for it.
  • An Application can place messages in local queue and can receive messages from it. 
  • Triggering can be enabled for this type of queue.
  • Messages in queue local won’t be passed on to another queue manager. So, to move messages from one local queue to another one needs to use other support packs and third party Utilities.

Alias queue object (QALIAS or QA)

  • Alias queue objects provide a reference to another queue object with a different name. 
  • The target queue object can be a local queue, a remote queue object, or a queue shared within the queue manager cluster.
Notes:
  • Alias queue doesn’t store any messages in it. 
  • An application can place a message and consume messages from Alias Queue.

Few practical scenarios for using Alias Queues,

Satisfying Naming conventions

  • Each company/client will have their own naming standards for identifying objects or coding structures.
  • To satisfy different naming standards one can create any number of Alias queues pointing to same base queue.

Providing Different level of access

  • Allowing each application to access a base queue via an Alias queue, will facilitate us to restrict any particular application/program from accessing that queue without interrupting others.

Hiding the queue from other clients etc

  • Sometimes, internal programs have to exchange information with external parties (clients).
  • Keeping security reasons in mind, we can create alias queue pointing to the actual queue and will inform the other end team about Alias Queue only, not the base queue.

Model queue object (QMODEL or QM)

  • Model queue object is nothing but a template of queue attributes referring which, application can create dynamic queues as needed. 
Notes:
  • The attributes of the model queue object determine the attributes of the dynamic queue created.
  • Model queue doesn’t store any messages in it. 

There are 2 types of dynamic queues, 

  • Temporary dynamic queues
  • Permanent Dynamic queues
  • Type of dynamic queue to be created depends on one of model queue attribute value *DefinitionType*

Remote queue object (QREMOTE or QR)

  • Remote Queue is local definitions of remote destination queue.
  • This will provide destination Queue and Queue manager information to the message placed by Application. 
  • Imagine this with the working of postal box operation for ease of understanding.
  • A message passing through remote queue will get the address attached to it saying where it has to go.
Syntax of remote queue definition,

Define Qremote(Remote queue name) 
RNAME(destination queue name) 
RQMNAME(Destination Queue manager name) 
Xmitq(transmission queue)
  • Remote queue name:
    • It’s the name with which the local queue manager knows it.
  • Rname:
    • This attribute value is destination queue name, to which messages should be reached.
  • RQMNAME:
    • Remote queue manager name in with Rname queue exists.
Notes:
  • Remote queue doesn’t store any messages. 
  • An application can’t pull/get a message from remote queue but can put a message in it. 
  • Remote queue doesn’t have any curdepth attribute as it doesn’t store anything.

Queue manager Alias:

  • Just like, an alias queue is a false name to a actual queue, Queue manager Alias is a false name to represent another queue manager name. its definition can be given as below,
  • Define Qremote(QMGR Alias Name) RNAME(‘’) RQMNAME(NAME of the remote queue)
  • It is same as a remote queue definition but excluding the RNAME parameter.
  • This is used in multi-hopping of messages from one queue manager to another queue manager

Special local queues:

There are some local queues that have special purposes in WebSphere MQ.

 Dead letter queue

  • It is considered to be a backup queue for a queue manager.
  • If queue manager fails to place a message on destination queue then a message will be placed in dead letter queue with reason for failure. 
  • A dead letter queue is special type of local queue. 
  • MQ administrators may analyze the messages available in dead letter queue, and take a decision on to weather to move the messages to application or discard.
  • It is not mandatory for a queue manager to have a dead letter queue, but it is strongly recommended.
  • One can assign a dead letter queue to queue manager by, runmqsc command. Alter qmgr deadq(Name of dead letter queue)

 Initiation queue

  • It holds the trigger messages generated during triggering process .
  • Trigger monitor, which continuously monitors this queue will consume the trigger message and takes appropriate action.

Transmission queue

  • Transmission queue is related to remote queue defined in that queue manager.
  • A message placed in remote will pass on to transmission queue before guided to destination queue manager.
    • The Usage attribute indicates that a local queue is used as a transmission queue.
    • Transmission Queue is generally used for channel triggering.
    • A transmission queue works with message channels to enable queue manager-to-queue manager communication. 

Command queue

  • It receives WebSphere MQ commands from an MQ administration tool (MQ explorer & other tools or commands) running locally or remotely.
  • If this queue is PUT disabled (not available for placing messages), then no Administration tool can work with that queue manager.

Event queue

  • When a queue manager detects an instrumentation event (which is nothing but some significant occurrence in a queue manager such as an error or a warning), it puts an event message describing the event on an event queue. 
  • An event queue can be monitored by a system management application that can get the event message and take appropriate action.
    • For example:
      • SYSTEM.ADMIN.PERFM.EVENT
      • SYSTEM.ADMIN.CHANNEL.EVENT

Default queues

  • Identify the default values of attributes of any new queue that is created. 
  • There is one default queue for each of the four types of queues: local, alias, remote, and model. 
  • Thus, you only need to include in the definition of a queue those attributes whose values are different from the default values.
  • We can alter the default attribute values 
    • For example, 
    • SYSTEM.DEFAULT.LOCAL.QUEUE: holds the default attribute values for any new local queue created.
    • SYSTEM.DEFAULT.REMOTE.QUEUE: holds the default attribute values for any remote queue created.
  • Try looking for default attribute by typing
    • Dis q(SYSTEM*) in runmqsc mode and will show all the object queues.

Dynamic Queues:

  • When an application program issues an MQOPEN call to open a model queue, the queue manager dynamically creates an instance of a local queue with the same attributes as the model queue. This is called a Dynamic Queue.
  • Use a dynamic queue when you do not need the queue after your application ends.

An intro about both types of Dynamic Queues,

  • Temporary Dynamic Queues:
    • They hold non-persistent messages only. 
    • They are non-recoverable. 
    • They are deleted when the queue manager is started. 
    • They are deleted when the application that issued the MQOPEN call that created the queue closes the queue or terminates. 
  • Permanent Dynamic Queues:
    • They hold persistent or non-persistent messages. 
    • They are recoverable in the event of system failures. 
    • They are deleted when an application (not necessarily the one that issued the MQOPEN call that created the queue) successfully closes the queue using the MQCO_DELETE or MQCO_DELETE_PURGE option. 
    • They can be deleted in the same way as a normal queue. 

Listeners:

  • Listener objects are used to accept incoming network requests from remote queue managers, or client applications
  • It’s a continuously running process at a particular port of the machine.
  • It listens to the incoming messages and informs the receiving MCA or server MCA about the network requests.
  • Listener can be controlled by queue manager automatically, i.e, It can start automatically with queue manager startup. For this ensure, the control attribute of listener should be as “QMGR”

Namelists:

  • A namelist is a WebSphere MQ object that contains a list of other WebSphere MQ objects. 
  • Namelists are used by applications such as trigger monitors, where they are used to identify a group of queues. 
  • The advantage of using a namelist is that it is maintained independently of applications; i.e, it can be updated without stopping any of the applications that use it. Also, if one application fails, the namelist is not affected and other applications can continue using it.
  • Namelists are also used with queue manager clusters so that you can maintain a list of clusters referenced by more than one WebSphere MQ object.

Monday, November 9, 2020

How to Encrypt JBOSS Passwords using VAULT

 Here we will see how to encrypt the password used in JBOSS like Keystore password ,key Pass phrase ,Data Source password or any other passwords that need to encrypt can be done with the VAULT and follow steps.

Encrypt Password in JBOSS
------------------------------------
1. Copy jks keystore to /hom/jboss in both Master and Slave
Bring down all controllers
2. 06-Oct-14@14:05:29-jboss@hostname1a:/rh/jboss/app1a/bin>./vault.sh
=====================================================================

JBoss Vault

JBOSS_HOME: /rh/jboss/app1a

JAVA: /usr/IBM/WebSphere/AppServer/java/bin/java

=====================================================================

**********************************
**** JBoss Vault ***************
**********************************
Please enter a Digit:: 0: Start Interactive Session 1: Remove Interactive Session 2: Exit
1

1
Removing the current interactive session
Please enter a Digit:: 0: Start Interactive Session 1: Remove Interactive Session 2: Exit
Removing the current interactive session
Please enter a Digit:: 0: Start Interactive Session 1: Remove Interactive Session 2: Exit
0
Starting an interactive session
Enter directory to store encrypted files:/home/jboss
Enter Keystore URL:/home/jboss/hostname1a.jks
Enter Keystore password:
Enter Keystore password again:
Values match
Enter 8 character salt:12345678
Enter iteration count as a number (Eg: 44):44
Enter Keystore Alias:hostname1a
Initializing Vault
Oct 6, 2014 2:06:33 PM org.picketbox.plugins.vault.PicketBoxSecurityVault init
INFO: PBOX000361: Default Security Vault Implementation Initialized and Ready
Vault Configuration in AS7 config file:
********************************************
...
</extensions>
<vault>
<vault-option name="KEYSTORE_URL" value="/home/jboss/hostname1a.jks"/>
<vault-option name="KEYSTORE_PASSWORD" value="MASK-2exADfZEVkq4nkGflMRrtM"/>
<vault-option name="KEYSTORE_ALIAS" value="hostname1a"/>
<vault-option name="SALT" value="12345678"/>
<vault-option name="ITERATION_COUNT" value="44"/>
<vault-option name="ENC_FILE_DIR" value="/home/jboss/"/>
</vault><management> ...
********************************************
Vault is initialized and ready for use
Handshake with Vault complete
Please enter a Digit:: 0: Store a secured attribute 1: Check whether a secured attribute exists 2: Exit

0
Task: Store a secured attribute
Please enter secured attribute value (such as password):
Please enter secured attribute value (such as password) again:
Values match
Enter Vault Block:db2ds
Enter Attribute Name:db2ds
Secured attribute value has been stored in vault.
Please make note of the following:
********************************************
Vault Block:db2ds
Attribute Name:db2ds
Configuration should be done as follows:
VAULT::db2ds::db2ds::1
********************************************
Please enter a Digit:: 0: Store a secured attribute 1: Check whether a secured attribute exists 2: Exit
1
Task: Verify whether a secured attribute exists
Enter Vault Block:db2ds
Enter Attribute Name:db2ds
A value exists for (db2ds, db2ds)
Please enter a Digit:: 0: Store a secured attribute 1: Check whether a secured attribute exists 2: Exit
[2] + Stopped (SIGTSTP) ./vault.sh
You have mail in /usr/spool/mail/jboss
06-Oct-14@14:14:46-jboss@hostname1a:/rh/jboss/app1a/bin>


3. Add below Vault in both Domain and Host controller
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
FROM CLI

[domain@10.91.74.96:39999 /] /host=/host=master/core-service=vault:add(vault-options=[("KEYSTORE_URL" => "/home/jboss/a01sribapp3a.jks"),("KEYSTORE_PASSWORD" => "MASK-2exADfZEVkq4nkGflMRrtM"), ("KEYSTORE_ALIAS" => "a01sribapp3a"), ("SALT" => "12345678"), ("ITERATION_COUNT" => "44"), ("ENC_FILE_DIR" => "/home/jboss/")])

Manually

<vault>
<vault-option name="KEYSTORE_URL" value="/home/jboss/vaultks.jks"/>
<vault-option name="KEYSTORE_PASSWORD" value="MASK-2exADfZEVkq4nkGflMRrtM"/>
<vault-option name="KEYSTORE_ALIAS" value="vaultks"/>
<vault-option name="SALT" value="12345678"/>
<vault-option name="ITERATION_COUNT" value="44"/>
<vault-option name="ENC_FILE_DIR" value="/home/jboss/"/>
</vault><management> ...

4. Edit domain.xml and in place of password give ${VAULT::db2ds::db2ds::1}

5. Start Domain ,host controllers and servers test the connectivity


Note: There is no way we can decrypt the password that is encrypted using VALUE . We can only check the key value exist or not and update the new password .

Wednesday, October 28, 2020

How to create SAN Certificate and its usage

 SAN Stands for Subject alternative name . Where the same certificate with multiple names used for multiple domains . This will save cost .Example we can generate one certificate and add other domain names in the subject Alternative Names can use for multiple sites .

Follow below steps for generation SAN certificate to configure in Apache WebServer .

create san.conf file with the below sample content
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
prompt= no
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
organizationName = Organization Name (eg, company)
commonName = Common Name (e.g. server FQDN or YOUR name)
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = abc.com
DNS.2 = def.com
DNS.3 = fgh.com

Updated san.conf looks like below for multiple common names

This image has an empty alt attribute; its file name is SAN.png
Generate key with csr file using below openssl command
openssl req -out sslcert.csr -newkey rsa:2048 -sha256 -nodes -keyout private.key -config san.conf
This image has an empty alt attribute; its file name is KeyGeneration-1024x118.png
This image has an empty alt attribute; its file name is files.png

once CSR file is generated you can verify the content on the https://www.entrust.net/ssl-technical/csr-viewer.cfm
copy content of sslcert.csr into the above UR or use below
openssl req -noout -text -in sslcert.csr | grep DNS

This image has an empty alt attribute; its file name is csrver1-1024x625.png
This image has an empty alt attribute; its file name is csrver2-1024x615.png

You can verify CSR with openssl

This image has an empty alt attribute; its file name is DNS.png
Once CSR is verified .Get this signed with Third pary vendor line Entrust ,Symatic --etc
Down load the CSR and root , Intermediate certs in PEM format .

for Apache SSL configuration we need 3 files .
Rename given CER to httpd.cer for SSLCertificateFile . private.key to httpd.key . Create SSLCACertificateFile by appending Intermediate with Root.cer files.

cat intermediate.cer >httpd_ca.crt && cat root.cer >>httpd_ca.crt

SSLCertificateFile /web/apache/WEB1A/ssl/httpd.cer
SSLCertificateKeyFile /web/apache/WEB1A/ssl/httpd.key
SSLCACertificateFile /web/apache/WEB1A/ssl/httpd_ca.crt

Wednesday, October 21, 2020

JBOSS EAP Patching and Rollback steps

Refer below JBOSS patching and reversion steps 

JBOSS Patching

Download Patch from https://access.redhat.com/downloads/.
Before running any jboss commands please make sure JAVA_HOME is set
export JAVA_HOME=/app/java8_64
Go to JAVA_HOME/bin and run jboss-cli.sh then run below to patch
patch apply /path/to/downloaded-patch.zip

[standalone@localhost:9999/] patch apply /tmp/jboss-eap-6.4.2.zip
"outcome" : "success",
"response-headers" : {
"operation-requires-restart" : true,
"process-state" : "restart-required"
}
}
updated modules available in JBOSS_HOME/modules/system/layer/base/.overlay
Once patch is successfull then restart the services

Patch Rollback

[standalone@localhost:9999/] patch rollback --patch-id=jboss-eap-6.4.2.CP --reset-configuration=true
{
"outcome" : "success",
"response-headers" : {
"operation-requires-restart" : true,
"process-state" : "restart-required"
}
}

Restart the services after successfull rollback

-->During Jboss patching if you get any error says /tmp is full . By default jboss used /tmp as temperary directory .
Can change it by -Djava.io.tmpdir=<new path>
export JAVA_OPTS="-Djava.io.tmpdir=<new path>"
--> To preseve any config file use --preserve=[bin/jboss-cli.xml]

Monday, October 12, 2020

configure Custom Identity and Custom Trust store in Weblogic

 


In this we will see how to generate self-signed certificate and configure to Weblogic Admin server .Before configuration we have demo certificate

Generated Self-Signed certificate with the below command . We can generate CSR from it and get it signed from third party . SSL configuration steps remain same .

/apps/java11_64/bin/keytool -genkey -v -alias weblogic -keyalg RSA -keysize 2048 -sigalg SHA256withRSA \
-dname "CN=www.abc.com, OU=abc, O=ABC Corp, C=IN, ST=Banglore, L=India" \
--keypass password -storepass password -validity 365 -keystore keystore.jks

Configuring Custom Keystore and truststore will see now .

Login to WLS Admin console
--> Go to Enviroment /Servers and click on the server for which ssl need to enable
--> Go Keystores tab change keystore to Custom Identity and Custom Trust
--> Enter keystore and truststore details then save
--> GO to SSL Tab then enter the certificate to use and keypassword and save

Restart the Admin sever since we did the SSL configuration to Admin server . We need to follow same for each Managed instance on Weblogic for SSL

Now admin logs are loding cert and trust certs from /apps/ssl/keystore.jks 
Here we used same keystore for both key and trust certs . We can configure seperate trust store if need.

<Dec 25, 2020, 12:24:07,261 PM Singapore Standard Time> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias weblogic from the jks keystore file /apps/ssl/keystore.jks.> 
<Dec 25, 2020, 12:24:07,565 PM Singapore Standard Time> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file /apps/ssl/keystore.jks.> 
<Dec 25, 2020, 12:24:07,929 PM Singapore Standard Time> <Warning> <Security> <BEA-090172> <No trusted certificates have been loaded. Server will not trust to any certificate it receives.> 
Expot same cert and import back to same keystore as trust cert. because self signed is signed by it self so same cert is need in trust as well.
Export certificate
keytool -exportcert -alias weblogic -file wls.cer -keystore keystore.jks

Add as trust 
[wlsuser@localhost ssl]$ /apps/java11_64/bin/keytool -import -alias root -file wls.cer -keystore keystore.jks 

Enter keystore password:  
Certificate already exists in keystore under alias <weblogic>
Do you still want to add it? [no]:  yes
Certificate was added to keystore
[wlsuser@localhost ssl]$ /apps/java11_64/bin/keytool -list -keystore keystore.jks 
Enter keystore password:  
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

root, Dec 25, 2020, trustedCertEntry, 
Certificate fingerprint (SHA-256): B9:CE:10:5C:7C:9D:D9:6B:DF:DC:62:25:DF:FE:86:E2:EF:A0:58:A0:C3:EC:7A:51:93:8E:CC:4D:5E:09:AE:E2
weblogic, Dec 25, 2020, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): B9:CE:10:5C:7C:9D:D9:6B:DF:DC:62:25:DF:FE:86:E2:EF:A0:58:A0:C3:EC:7A:51:93:8E:CC:4D:5E:09:AE:E2
[wlsuser@localhost ssl]$

If we dont import same cert as trust certificate then the server will start but cannot able to load the page .

After successful SSL configuration we can see new certificate in Weblogic console .

Monday, October 5, 2020

How to know the validity of a certificate installed on the server

 

There are multiple ways to get the validity of the certificate installed on the server .

1.If we have access to server login and open the keystore and verify using keytool

keytool -details -alias mycert -keystore abc.jks -password **** -v

2.use one of the client to browse the URL using https://ip:port:/ that will present the certificate in the browser after successful connection .

3.Use openssl to get the certificate and verify the validity .

openssl s_client -connect ip:port 2>/dev/null |openssl x509 -noout -dates

Tuesday, September 15, 2020

How to fix Weblogic startup slowness issues

 There might be multiple reasons for the startup slowness . To troubleshoot the issue fist remove the application targeted from the application server for this JVM and then try to start the server alone . If we still get the issue then it is something not relevant to Application .Refer the below in some cases this solve the issue where the application is not using random number generator . before implementing the change get the confirmation once with App team and finalise the change .

The library used for random number generation in Oracle’s JVM relies on /dev/random by default for UNIX platforms. This can potentially block the WebLogic Server process because on some operating systems /dev/random waits for a certain amount of “noise” to be generated on the host machine before returning a result.

Although /dev/random is more secure, it’s recommended to use /dev/urandom if the default JVM configuration delays WebLogic Server startup. To determine if your operating system exhibits this behavior, try displaying a portion of the file from a shell prompt: head -n 1 /dev/random

If the command returns immediately, you can use /dev/random as the default generator for JVM. If the command does not return immediately, use these steps to configure the JVM to use /dev/urandom:

Open the $JAVA_HOME/jre/lib/security/java.security file in a text editor.
Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
Save your change and exit the text editor.
And because there’s a bug in JDK when you use /dev/urandom you have to set it up as /dev/./urandom

You can also set up system property “java.security.egd” which will override the securerandom.source setting.
-Djava.security.egd=file:/dev/./urandom

Take backup of java.security and update the value of securerandom.source=file:/dev/./urandom and restart the server . 

Monday, August 24, 2020

Shell Scripting for Beginners

 A shell script is a file containing series of commands .The shell reads this file and run each command as if like it entered on the command line.On a Linux Operating System there are multiple Interpreters which will execute the commands that we pass to them . Default is Bash and this is widely available on various operating systems . Other 's are KSH,SH . Some commands may different on there interpreters but majority will support by all.

1.What is shell Scripting
2.Variables
3.File Manipulations
4.Common Iterations/Loops
5.Useful System Variables
6.Test Operators
7.Numeric Tests
8.String tests
9.Logical Tests
10.Argument Variables
11.Some Advance commands ,utilities ,network and file handling commands

1.What is shell Scripting

  In short and simple a shell script is a file containing series of commands .The shell reads this file and run each command as if like it entered on the command line.
  The benefit of this is to simplify the day to day mundane activities . Shell is a scripting language interpreter.
  Scripts unlock the power of our Linux machine. imagine you have 2 files on with 100 words and another with 100000 words and you need to check first file words on second file and print the word existence and no of times present .
  This is where the power of scripting comes .Once you write a simple script then start using it and enhance further to fit into day to day activities .
#!/bin/bash
echo "Hello World"

First line #!/bin/bash is the interpreter which will execute the commands . Next to that are the commands . echo will print the values in the screen .

2.Variables

 Variable is a keyword to store some values . We have some System predefined variable and user can also define variables in the scripts to store the results of commands .
 Example :
 PATH is the OS define variable .run echo $PATH to print the values 
 VALUE=`date` && echo $VALUE --> here VALUE is user defined variable and it exists till the script finishes .Once finished VALUE variable no more exist.

3.File Manipulations

Below table gives the basic file manipulation operation help full while handling the files.

File CommandsExplaination
> fileCreate/Overwrite file
>> fileAppend to file
>file 2>&1redirect both output and error to file
< fileread from file
file1 | file1pipe output of file1 as input to file2

4.Common Iterations/Loops

Below given blocks are very important because in the scrips for each command that we use we need to check something before executing and store the results some where and iterate through some repetitive steps .

read text file line by line . in many scenarios we need to read file line by line and apply the logic .In the below we are reading and printing with echo.

while read line 
do 
  echo "Line is $line
done < file 

Find matching lines

grep foo file --> print the lines that has foo matching word
egrep 'foo|bar' file -->find multiple word and print the matching lines
grep -i FOO file --> same as above but i for ignore case
grep -v foo file --> -v used to print not matching lines 

Get the output of command to a variable

FILELIST=`ls`
COUNTOFFILES=`ls -lrt |wc -l`

Case is a good way to Iterate avoiding multiple if/elif blocks

case.sh
#!/bin/bash
# case example
 $1 means first argument in the script
# Usage ./case.sh argument
case $1 in
	start)
		echo starting
	;;
	stop)
		echo stoping
	;;
	restart)
		echo restarting
	;;
	*)
		echo don\'t know
	;;
esac

Function declaration and calling ...

multiply() {
	expr $1 \*2
	}
multiply 3 

For loop iterates the in the list of give values and run the logic

for i in 1 2 3 4 5 
do
	echo "In $i loop"
	echo "Do some logic here "
done	

5.Useful System Variables

$? --> what the shell returned or previous command return status
0 means success
other value means failure
This $? verification is very useful to check the status of previous command to take next decession
$* --> all arguements
./abc.sh 1 2 --> in this $* is 3
$# --> No of the variable values . where $0 is name of script , $1 is 1st arguement ,$2 is 2nd arguement

6.Test Operators

# Compare 2 variable and do something 
if [ "$x" -lt "$y" ] ; then
 # Do something
 fi

7.Numeric Tests

Numeric Test operationsExplaination
ltless than
gtgreate than
eqequal
nenot equal
gegreate or equal
leless than or equal

8.String tests

File Test operationsExplaination
ntnewer than
dis a directory
fis a file
rreadable
wwritable
xexecutable

9.Logical & String Tests

Logical & String TestsExplaination
=equal to
zzero lenth
nnot zero length
&&Logical AND
||logical OR
!logical Not

10.Argument Variables

Argument VariablesExplaination
$0Program name or script name
$11st argument
$22nd arguments
$99th argument
$*all aguements
$#No of Arguments

11.Some Advance commands ,utilities ,network and file handling commands

below are some of the commands that we daily use for the day-to-day activities.

Linux CommandsUsage
command1 ||Command2run command1; if it fails, run Command2
command1 && Command2run command1; if it works, run Command2
command1 ; Command2keeping 2 commands on the same line
ls -lStlist files biggest last
ls -lrtlist files newest last
ls -alshow all fils including hidden
sort -nsort numarically
wget URLdownload url
read xread some value from user /keyboard
touch filecreate empty file
cmd |tee file.txtcommand output to stdout also to file.txt
ifconfig -alist all network interfaces .can see ip here
netstat -rshow routers
netstat -tnpl |grep -I listenshows all listening ports on the server
ssh u@hostlogin to host as user u
scp file.txt u@host:\tmpcopy file.txt to host /tmp/ wit u user
alias l='ls -lrt'creating alias to a command
df -hshow dis mount points
find . -type f -name a.txtfind a.txt in current directory
find . -type f -name *.txt -printfind all the txt files
find /foo -type d -lslist all directories under foo
awk -F":" '{ print $1 " " $NF}'print file value in each line of the file delimted with : and NF for last value
tar -cvf abc.tar a.txt b.txtcreate acrchive file
tar -tvf abc.tarcheck the list of file in abc.tar
tar -rvf abc.tar c.txtadd c.txt to existing abc.tar
tar -xvf abc.tarextract abc.tar file
tar -zcvf my_archive.tar.gz *create zip and then tar
tar -zxvf my_archive.tar.gz *extract zip tar
zcatview the file without decompressing it like cat
ps -ef |grep keywordgrep some process running
kill -9 pidkill process with pid
pwdpresent working dir
cdGo to user home
cd ..Go to previous directory
hosnamehostname of the server

Hope the blog gives some useful information for starting writing scripts ...

Monday, July 6, 2020

Linux find and replace

 

There is a requirement to find and replace a string from file's in the in the directory name across all the files and directories in the given directory.

We already know some basic commands like mv,sed/perl & find commands . We will use these basic commands to write a script and get the expected result .

basic commands 

mv
mv ABC DEF  -> to rename a directory
find
find /dir -type f -exec grep -l "ABC" {} \; | xargs perl -pi -e 's/ABC/DEF/g' 
Above command is used find and replace ABC in all the files with DEF on /dir 
sed
sed 's/ABC/DEF/g' -->To replace a string in a word 
perl
perl -pi -e 's#ABC#DEF#g' file --> this searches and replaces ABC to DEF .

We combine this in to 2 scenarison .

  1. To find and replace the string in given file .
  2. To find and replace the string in the directy recursivley along with matched sub directory string .

On the below we have ABC directory with abc.txt and ACBC directory . Test results for both scenarios 1 and 2 given below .

Scenario 1 :

Scenario2 :

1.sh

#!/bin/bash
echo "Enter file Name"
read fileName
echo "Enter findstring "
read findstring
echo "Enter replacestring "
read replacestring
if [ -f $fileName ] ; then
 perl -pi -e 's#'$findstring'#'$replacestring'#g' $fileName
 echo "$findstring replace with $replacestring "
 else
 echo "Given file is not available "
fi

2.sh

#!/bin/bash
echo "Enter Dir Name"
read dir 
echo "Enter findstring "
read findstring
echo "Enter replacestring "
read replacestring
if [ -d $dir ] ; then
 find $dir -type f -exec grep -l "$findstring" {} \; | xargs perl -pi -e 's/'$findstring'/'$replacestring'/g' 
 echo " Word $findstring replaced with $replacestring successfully ..."
 for directory in `find $dir -type d -name "$findstring"` 
  do
    newDirName=`echo $directory |sed 's/'$findstring'/'$replacestring'/g'`
	mv $directory $newDirName
	echo " $directory name changed to $newDirName "
  done
 else
  echo "Directory not found : $dir "
fi

set -x and set +x are for debugging . Hope this article helps in detail for find and replace . ..

Featured

Weblogic Domain Migration

 In this blog we will see domain re-configuration which will be done as part of Weblogic migration from lower version to higher version [ Ex...