Wednesday, February 26, 2020

awk utility for shell script

 

awk works on programs that contain rules comprised of patterns and actions. The action is executed on the text that matches the pattern. Patterns are enclosed in curly braces ({}). Together, a pattern and an action form a rule. The entire awk program is enclosed in single quotes (').

Here is the output of -h

Perhaps i don’t need all the output I want only mounted file systems to print and it size let say.

df -h |grep -v Filesystem |awk '{ print $2 " " $NF }'

In the above we are ignoring first description line using grep –v. Then printing required values .Infact we can print as many values as we want . NF holds last values .

$0 represents entire line 
$1 represents the first field 
$2 represent second field
$7 represent 7th field
$NF represents the last record.

In the previous command if we don’t specify delimiter then space is the default .  so df –h delimited with space then we are printing the values.Also we used “ “ space when printing .If we use , then space used while printing.

df -h |grep -v Filesystem |awk '{ print $2 , $NF }'

Let’s see different delimiter “:” with simple echo passing to awk

echo “ABC:DEF:XYZ” |awk –F”:” ‘{ print $1,$2,$NF }’

In this both NF and $3 are having same value .

Another example just want to print the date in Mon DD YYYY format

date | awk '{print $2,$3,$6}'

OFS --> Output field Separator

date | awk 'OFS="-" {print$2,$3,$6}'

The BEGIN and END Rules

A BEGIN rule is executed once before any text processing starts. In fact, it’s executed before awk even reads any text. An END rule is executed after all processing has completed. You can have multiple BEGIN and END rules, and they’ll execute in order.

awk 'BEGIN {print "File Systems"} {print $NF}' /tmp/dfout.txt

Adding pattern’s or Conditions

We can also add patterns before printing with AWK . In the below example we will check if the 3rd value greater or equal 1000 the print those rows or fields on those rows .

Combining both BEGIN and patterns on awk

In case if we want to develop our own logic using awk follow below

The first line of the script tells the shell which executable to use

#!/usr/bin/awk -f

BEGIN {
  # set the input and output field separators
  FS=":"
  OFS=":"
  # zero the accounts counter
  accounts=0
}
{
  # set field 2 to nothing
  $2=""
  # print the entire line
  print $0
  # count another account
  accounts++
}
END {
  # print the results
  print accounts " accounts.\n"
}

Hope this article help to use awk for data processing with advance utility ...

Tuesday, February 25, 2020

Keytool commands for Certificate provision and management

 

Follow below steps to get SSL certificate for an JAVA based Application server .In the below steps we will use keytool which is availale in JAVA_HOME/bin/

Create a new keystore keystore.jks for managing your public/private key pairs and certificates.

Note : -v option is for detailed output

Keytool help for the commands 
[wlsuser@localhost tmp]$ keytool
Key and Certificate Management Tool

Commands:

 -certreq            Generates a certificate request
 -changealias        Changes an entry's alias
 -delete             Deletes an entry
 -exportcert         Exports certificate
 -genkeypair         Generates a key pair
 -genseckey          Generates a secret key
 -gencert            Generates certificate from a certificate request
 -importcert         Imports a certificate or a certificate chain
 -importpass         Imports a password
 -importkeystore     Imports one or all entries from another keystore
 -keypasswd          Changes the key password of an entry
 -list               Lists entries in a keystore
 -printcert          Prints the content of a certificate
 -printcertreq       Prints the content of a certificate request
 -printcrl           Prints the content of a CRL file
 -storepasswd        Changes the store password of a keystore

Use "keytool -command_name -help" for usage of command_name
[wlsuser@localhost tmp]$ 
Generate key

-genkey
keytool -genkey -v -alias mycert -keyalg RSA -keysize 2048 -sigalg SHA256withRSA \
-dname "CN=www.abc.com, OU=abc, O=ABC Corp, C=IN, ST=Banglore, L=India" \
--keypass pkpassword -storepass storepassword -validity 365 -keystore keystore.jks
Generate a CSR in the file carequest.csr for submission to a CA. The CA signs and returns a certificate or a certificate chain that authenticates your public key.
CSR

keytool -certreq -v -alias mycert -file carequest.csr -keystore keystore.jks -storepass storepassword

Send or upload the csr file to the third pary site like and get it signed . Then download root ,intermediate certificates and carequest.cer file .

Print the contents of a certificate file in a human-readable form.
keytool -printcert -v -file carequest.cer
Import Root
keytool -importcert -alias root -file root.cer -keystore keystore.jks -storepass storepassword
Import Intermediate 
keytool -importcert -alias inter -file intermediate.cer -keystore keystore.jks -storepass storepassword
Import sever Certificate 
keytool -importcert -alias mycert -file carequest.cer -keystore keystore.jks -storepass storepassword
verify keystore
keytool -list -v -alias mycert -keystore keystore.jks -storepass storepassword
change keystore password
keytool -storepasswd -keystore keystore.jks
Change key password
keytool -keypasswd -alias mycert -keystore keystore.jks
Delete the certificate with the alias aliasname from the keystore keystore.jks.
keytool -delete -alias aliasname -keystore keystore.jks -storepass storepassword
print the certificate 
keytool -printcert -f mycert.cer

Now the keystore is ready. we can use this keystore to configure in Application server's like Jboss,Weblogic,Websphere,tomcat ..etc

In a certificate if both Owner and Issuer are same then it is Self signed certificate . We can use self signed cert for the server but it is not secured and not recommended.
Every root certificate of the Third party provider are self signed it self and used to sign other certificates .

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

 If we have any SSL handshake issue first we need to understand the details steps that take place during SSL handshake then only we will be able to address it easily . Find the below some of the details .

There are multiple reasons for SSL handshake . When we get this Error on the logs . First of all we need to verify server side whether the SSL configured is 2 Ways SSL or 1 Way SSL .

SSL Handshake Issue troubleshooting :

Make sure the Public keys [ Trusted certificates (root & Intermediate )] are imported in the client truststore .

For 2 Ways SSL both side Signer certificate should be imported .

If Server side enforce the certificate then server certificate also need to import at the client trust store .

Both side should have at-least a common Allowed Protocol . Ex TLS1.2 and a cipher

If something is not satisfied in the above then make the changes accordingly .Like importing the signers in to truststore

keytool -import -file /tmp/root.cer -alias root -keysoore /pathToSSL/***.jks then enter and provide the password .

In some case we can not figure out what is the issue .Then enable SSL debug using below parameter in startup script or in the server JVM arguement .then restart the JVM

-Djavax.net.debug=ssl:handshake:verbose

once restarted test the connectivity . In the logs we can see full debug statements . and also we can see what trusted certs are loaded .There are certain steps in SSL handshake all are printed . Refer below table for SSL Handshake to get better idea .

Exception:

*** ClientHello, TLSv1.2
..
...
Compression Methods: { 0 }
Extension elliptic_curves, curve names: {secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1}
Extension ec_point_formats, formats: [uncompressed]
Extension signature_algorithms, signature_algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA256withDSA, SHA1withECDSA, SHA1withRSA, SHA1withDSA
Extension extended_master_secret
Extension renegotiation_info, renegotiated_connection: b7:c5:d2:43:3b:dd:24:c8:33:41:15:8b

***
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1.2 Handshake, length = 288
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1.2 Application Data, length = 384
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', WRITE: TLSv1.2 Application Data, length = 1808
...
...
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', RECV TLSv1.2 ALERT: fatal, handshake_failure
%% Invalidated: [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', called closeSocket()
[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)', handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

In the above error handshake failed because Server side rejectClientNegotiation is set to true . and client tried to negotiate for multiple call's renegotiation_info that is where it failed .

Now you can match every step in the SSL handshake debug log with the attached screen to see at what step the failure happened .

Featured

Weblogic Domain Migration

 In this blog we will see domain re-configuration which will be done as part of Weblogic migration from lower version to higher version [ Ex...