In this blog we will see domain re-configuration which will be done as part of Weblogic migration from lower version to higher version [ Ex Weblogic 12 to Weblogic 14 or Weblogic 10 to Weblogic 12 or weblogic 10 to 14 version]
/aps for Domains and /sw mount points available software installations in our case
We will bave /sw/bea12 installed and have a domain /aps/tst/tstDomain
Domain Reconfiguration 1) Backup Entire tstDomain before reconfiguring the domain
Note: wlst.sh should be executed from the new version 2) execute /sw/bea14/oracle_common/common/bin/wlst.sh 3) in offline WLST execute in sequence a) readDomainForUpgrade('/aps/tst/tstDomain') b) updateDomain() c) closeDomain() d) exit() 4) Update setDomainEnv.sh with the new location of commEnv.sh at ${WL_HOME}/../oracle_common/common/bin/commEnv.sh 5) Rename WLS_DIAGNOSTICSxxxxxx.DAT for each instance in {instance folder}/data/store/diagnostics 6) Start the services and verify .
Now verify the Admin console and logs it will reflect latest version of Weblogic .
Error's that might happen during domain re-configuration
Caused by:com.oracle.cie.domain.script.ScriptException:64254:Error occurred in "Backup & Initialization" phase execution 64254:Encountered error:60971: Tha domain is not reconfigurable.
Upgrading WebLogic domain from WLS 12.1.3 to 12.2.1.4 using WLST fails with:
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Error: readDomainForUpgrade() failed. Do dumpStack() to see details. Error: runCmd() failed. Do dumpStack() to see details. Problem invoking WLST - Traceback (innermost last): File "/shared/weblogic/WLSbin/binaries_Install/WLS12214_binaries/DomainUpgrade.py", line 2, in ? File "/tmp/WLSTOfflineIni4018200302849410640.py", line 269, in readDomainForUpgrade File "/tmp/WLSTOfflineIni4018200302849410640.py", line 19, in command 64254: Encountered error: 60971: The domain is not reconfigurable. 60971: The domain did not contain information necessary to determine how to reconfigure the domain. 60971: Refer to the upgrade documentation on how to proceed. 64254: Check log for more detail. null at com.oracle.cie.domain.script.jython.CommandExceptionHandler.handleException(CommandExceptionHandler.java:69) at com.oracle.cie.domain.script.jython.WLScriptContext.handleException(WLScriptContext.java:3085) at com.oracle.cie.domain.script.jython.WLScriptContext.runCmd(WLScriptContext.java:738) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) at java.lang.reflect.Method.invoke(Method.java:508)
com.oracle.cie.domain.script.jython.WLSTException: com.oracle.cie.domain.script.jython.WLSTException: 64254: Error occurred in "Backup & Initialization" phase execution 64254: Encountered error: 60971: The domain is not reconfigurable. 60971: The domain did not contain information necessary to determine how to reconfigure the domain. 60971: Refer to the upgrade documentation on how to proceed. 64254: Check log for more detail.
CAUSE
The cause behind the issue is with corrupted config.xml file for the upgrading domain.
SOLUTION
Use the following to resolve:
1. Restore correct config.xml from backup to $DOMAIN_HOME/config/config.xml file for upgrading WebLogic domain. 2. To re-configure / Upgrade a WebLogic Domain Using WebLogic Scripting Tool (WLST)
a. Navigate to <12.2.1.4_ORACLE_HOME>/oracle_common/common/bin Invoke WLST using ./wlst.sh or java weblogic.WLST
In A Server environment we frequently listen about a web server and app server . Once we know what is it and the purpose of it it is easier to differentiate and understand .
1. It is not mandatory for every application to have a Web Server in it Architecture
2. If the Application is internet facing then it is recommended to have a Web Server for landing page . So all the PEN Test issues related to security will be easily addressed here .
3.If some one is using an App Server software from ABC vendor then Web Server is generally free of License .Example if We use IBM WebSPhere Application server then IBM Https server is free to use behind the Application server .
4. For cross Vendor use need to purchase the License .
In general for a 3 tire architecture Request flow like below
Browser --> SLB --> WebServer's --> APp Server's --> Data Base
No of WebServer and AppServer can be 2 or More for a High Availability Environment .
What Is a Web Server?
A web server is usually a kind of software program that is designed to handle all of your web requests. The server is created to accept only the static contents such as HTML pages, video files, images, and much more in the form of incoming requests. After getting requested, the web server responds to the requests over the HTTP protocol with other optional data contents.
The web server is serving all the contents of the World Wide Web and make them accessible to its end users. This refers to a kind of system that consists of both software and hardware web contents as per the end-users.
In simple terms, the webserver is a computer that delivers web pages when they are requested for accessing the content by the end-users. Apache fulfills this approach in current days as it is the most popularly and widely used open-source web server developed and fully maintained by the Apache Software Foundation.
What Is an Application Server?
The application server is frequently mistaken for the webserver and that as though they offer same administrations however in all actuality, it utilizes different techniques to process the majority of the applications which are started by the webserver.
This permits to have and convey top of the line business applications that are comprising of a 3-level application Graphical User Interface(GUI) which incorporates exchange server, a database, and an application server.
The primary purpose of the application server is to serve the right business logic to the mobile app programs using variant protocols that include HTTP. Be sure that it is not restricted to do the same. Being a software framework that facilitates both web-based and enterprise-based applications, it does not only serve the static content but also suited for serving the dynamic content and application transfer from one device to another.
We will see some of the main differences on the below table .
No.
Key
Web Server
Application Server
1
Purpose
Web Server contains Web container only.
Application Server contains Web Container plus EJB Container.
2
Useful
A web server is good in case of static contents like static html pages.
Applcation server is relevant in case of dynamic contents like bank websites.
3
Resource Consumption
Web server consumes less resources like CPU, Memory etc. as compared to application server.
Application server utilizes more resources.
4
Target Environment
Web Server provides the runtime environment for web applications.
Application server provides the runtime environment for enterprise applications.
5
Multithreading support
Multithreading is not supported.
Multithreading is supported.
6
Protocol(s) supported
Web Server supports HTTP Protocol.
Application Server supports HTTP as well as RPC/RMI protocols.
7
Load Balancer
It can act as Load balancer for all the back end Application server
Application server itself will not have that feature .
8
HTTP Methods
Majority of HTTP methods are directly controlled from Web Server configuration like TRACE
Will not have controls on all types of Application server
9
Example
AOL Server Apache HTTP IIS Nginx Boa Tux/kHTTPd IBM HTTP HFS Oracle I-Planet
Weblogic Websphere Glass Fish JBoss eap Oracle Application server Boreland Tomcat Cold Fusion
Runtime provisioning is a feature introduced in WebSphere Application Server V7. It focuses on improving application server startup time by providing intelligent analysis of application set and server configuration to determine the needed subset of components to be loaded. This feature does not load an entire runtime library and thus decreases the memory footprint and shortens the start time.
There is no need for administrators and application developers to modify any processes to take advantage of the runtime provisioning. To turn this feature on, complete the following steps:
Click Servers > Server Types > WebSphere application servers, as shown below:WebSphere application servers window
In the Application servers view, select your desired application server, as shown below: WebSphere application server view server selection
Select Start component as needed to enable runtime provisioning in this server, as shown below:Application servers configuration window
Configuring the JVM
As part of configuring an application server, you might define settings that enhance the way your operating system uses of the Java virtual machine (JVM).
The JVM is an interpretive computing engine that is responsible for running the byte codes in a compiled Java program. The JVM translates the Java byte codes into the native instructions of the host machine. The application server, being a Java process, requires a JVM to run and to support the Java applications running on it. JVM settings are part of an application server configuration.
To view and change the JVM configuration for an application server process, use the Java virtual machine page of the administrative console or use wsadmin scripts to change the configuration.
In the administrative console, click Servers > Server Types > WebSphere application servers > server_name.
Under Server Infrastructure, click Java and Process Management > Process definition.
Select Java Virtual Machine.
Specify values for the JVM settings in Generic JVM arguments text area as needed, and click OK.
Click Save in the messages section of the administrative console panel to save the changes to the master configuration.
Restart the application server.
Garbage collection (GC) is an integral part of the Java Virtual Machine (JVM) as it collects unused Java heap memory so that the application can continue allocating new objects. The effectiveness and performance of the GC play an important role in application performance and determinism. The IBM JVM provided with IBM WebSphere Application Server V8 (on supported platforms) provides four different GC policy algorithms:
-Xgcpolicy:optthruput
-Xgcpolicy:optavgpause
-Xgcpolicy:gencon (default in WAS 8)
-Xgcpolicy:balanced
Each of these algorithms provides different performance and deterministic qualities. In addition, the default policy in WebSphere Application Server V8 has changed from -Xgcpolicy:optthruput to the -Xgcpolicy:gencon policy.
Configuring connection pooling properties
Performance of an application that connects to a database can be greatly affected by the availability of connections to the database and how those connections affect the performance of the database itself. There are no simple rules that tell you how to configure the connection pool properties. Your configuration is highly dependent on application, network, and database characteristics. You need to coordinate the values that you specify in WebSphere closely with the database administrator.
Complete the following steps to access the connection pool properties:
Navigate to Resources > JDBC > Data sources and click the data source name.
Click Connection pool properties in the Additional Properties section.
The window shown in figure below opens
Specify the following information:
Connection TimeoutSpecify the interval, in seconds, after which a connection request times out and a ConnectionWaitTimeoutException is thrown. This action can occur when the pool is at its maximum (Maximum Connections) and all of the connections are in use by other applications for the duration of the wait.For example, if Connection Timeout is set to 300 and the maximum number of connections is reached, the Pool Manager waits for 300 seconds for an available physical connection. If a physical connection is not available within this time, the Pool Manager throws a ConnectionWaitTimeoutException.Tip: If Connection Timeout is set to 0, the pool manager waits as long as necessary until a connection is allocated.
Maximum ConnectionsSpecify the maximum number of physical connections that can be created in this pool.These connections are the physical connections to the back-end database. After this number is reached, no new physical connections are created and the requester waits until a physical connection that is currently in use is returned to the pool, or a ConnectionWaitTimeoutException is thrown.For example, if Maximum Connections is set to 5, and there are five physical connections in use, the Pool Manager waits for the amount of time specified in Connection Timeout for a physical connection to become free. If, after that time, there are still no free connections, the Pool Manager throws a ConnectionWaitTimeoutException to the application.
Minimum ConnectionsSpecify the minimum number of physical connections to be maintained. Until this number is reached, the pool maintenance thread does not discard any physical connections. However, no attempt is made to bring the number of connections up to this number.For example, if Minimum Connections is set to 3, and one physical connection is created, that connection is not discarded by the Unused Timeout thread. By the same token, the thread does not automatically create two additional physical connections to reach the Minimum Connections setting.
Reap TimeSpecify the interval, in seconds, between runs of the pool maintenance thread.For example, if Reap Time is set to 60, the pool maintenance thread runs every 60 seconds. The Reap Time interval affects the accuracy of the Unused Timeout and Aged Timeout settings. The smaller the interval you set, the greater the accuracy. When the pool maintenance thread runs, it discards any connections that have been unused for longer than the time value specified in Unused Timeout, until it reaches the number of connections specified in Minimum Connections. The pool maintenance thread also discards any connections that remain active longer than the time value specified in Aged Timeout.Tip: If the pool maintenance thread is enabled, set the Reap Time value less than the values of Unused Timeout and Aged Timeout.The Reap Time interval also affects performance. Smaller intervals mean that the pool maintenance thread runs more often and degrades performance.
Unused TimeoutSpecify the interval in seconds after which an unused or idle connection is discarded.Tips:
Set the Unused Timeout value higher than the Reap Timeout value for optimal performance. Unused physical connections are only discarded if the current number of connections not in use exceeds the Minimum Connections setting.
Make sure that the database server's timeout for connections exceeds the Unused timeout property specified here. Long lived connections are normal and desirable for performance.For example, if the unused timeout value is set to 120, and the pool maintenance thread is enabled (Reap Time is not 0), any physical connection that remains unused for two minutes is discarded. Note that accuracy of this timeout, as well as performance, is affected by the Reap Time value. See the Reap Time bullet for more information.
Aged TimeoutSpecify the interval in seconds before a physical connection is discarded, regardless of recent usage activity.Setting Aged Timeout to 0 allows active physical connections to remain in the pool indefinitely. For example, if the Aged Timeout value is set to 1200, and the Reap Time value is not 0, any physical connection that remains in existence for 1200 seconds (20 minutes) is discarded from the pool. Note that accuracy of this timeout, as well as performance, is affected by the Reap Time value. See Reap Time for more information.Tip: Set the Aged Timeout value higher than the Reap Timeout value for optimal performance.
Purge PolicySpecify how to purge connections when a stale connection or fatal connection error is detected.Valid values are EntirePool and FailingConnectionOnly. If you choose EntirePool, all physical connections in the pool are destroyed when a stale connection is detected. If you choose FailingConnectionOnly, the pool attempts to destroy only the stale connection. The other connections remain in the pool. Final destruction of connections that are in use at the time of the error might be delayed. However, those connections are never returned to the pool.Tip: Many applications do not handle a StaleConnectionException in the code. Test and ensure that your applications can handle them.
WebSphere Application Server data source properties
You can set the properties that apply to the WebSphere Application Server connection, rather than to the database connection. To access the connection pool properties, navigate to Resources > JDBC > Data sources and click the data source name.
Click WebSphere Application Server data source properties in the Additional Properties section.
Clicking the link opens the window shown in figure below.
Specify the following information:
Statement cache sizeSpecify the number of prepared statements that are cached per connection. A prepared statement is a precompiled SQL statement that is stored in a prepared statement object. This object is used to execute the given SQL statement multiple times. The WebSphere Application Server data source optimizes the processing of prepared statements.In general, the more statements your application has, the larger the cache should be. For example, if the application has five SQL statements, set the statement cache size to 5, so that each connection has five statements.Tip: This setting is vital to performance of the database and will most likely require tuning to suit the specific application. In general, the default is not high enough for best performance.
Enable multithreaded access detectionIf you enable this feature, the application server detects the existence of access by multiple threads.
Enable database reauthenticationConnection pool searches do not include the user name and password. If you enable this feature, a connection can still be retrieved from the pool, but you must extend the DataStoreHelper class to provide implementation of the doConnectionSetupPerTransaction() method where the reauthentication takes place.Connection reauthentication can help improve performance by reducing the impact of opening and closing connections, particularly for applications that always request connections with different user names and passwords.
Enable JMS one-phase optimization supportActivating this support enables the Java Message Service (JMS) to get optimized connections from the data source. Activating this support also prevents JDBC applications from obtaining connections from the data source.
Log missing transaction contextSpecifies whether the container issues an entry to the activity log when an application obtains a connection without a transaction context.
Non-transactional data sourceSetting the flag to true will cause the Application Server to never enlist the connections from the data source in global or local transactions. Applications must explicitly call setAutoCommit(false) on the connection if they want to start a local transaction on the connection, and they must commit or roll back the transaction that they started. This property should rarely be set to true.
Error detection modelThe error detection model has been expanded and the data source has a configuration option that you can use to select the exception mapping model or the exception checking model for error detection.
Connection validation propertiesThere are two properties, and you can choose both. If you select the Validate new connections check box, the application server tries to connect to the database. If you select this property, you can specify how often, in seconds (interval).If you select the Validate existing pooled connections check box, the application server retries to make a connection. If you select this property, you can specify the retry interval for the server reroute. The pretest SQL string is sent to the database to test the connection.Tip: Connection validation by SQL query is deprecated in WebSphere Application Server V8.0. You can use validation by the JDBC Driver instead. If you use the property of validation by JDBC driver, you need JDBC 4.0 or later. If you do not have JDBC 4.0, you have to update the JDBC driver first.
According to this configuration, the client must present a valid certificate, otherwise, the handshake fails. However, both WLS instances are located in the same machine and configured with the same identity and trusted keystore files.
There is no apparent reason to fail, so to understand the SSL interactions between client and server, enabling SSL debug flags is required.
For this example, SSL debug flags were enabled on client end, represented by the WLS AdminServer. These lines were added at the end of the bin/setDomainEnv.sh file:
if [ "${SERVER_NAME}" != "AdminServer" ] ; then JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.enableJSSE=true -Dweblogic.StdoutDebugEnabled=true -Dweblogic.log.StdoutSeverity=Debug -Dssl.debug=true -Djavax.net.debug=ssl" export JAVA_OPTIONS fi
The AdminServer was restarted and the issue was replicated collecting from the log files, the following sequence:<Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.> *** ClientHello, TLSv1.2 *** ServerHello, TLSv1.2 *** Certificate chain chain [0] = [ Subject: CN=<FQDNHERE>, OU=IT, O=FICTIONALCOMPANY, L=Havant, ST=Hampshire, C=UK Issuer: CN=<FQDNHERE>, OU=IT, O=FICTIONALCOMPANY, L=Havant, ST=Hampshire, C=UK *** Found trusted certificate: Subject: CN=<FQDNHERE>, OU=IT, O=FICTIONALCOMPANY, L=Havant, ST=Hampshire, C=UK Issuer: CN=<FQDNHERE>, OU=IT, O=FICTIONALCOMPANY, L=Havant, ST=Hampshire, C=UK *** CertificateRequest Cert Types: RSA, DSS, ECDSA Supported Signature Algorithms: SHA512withECDSA, SHA512withRSA, SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA, SHA224withECDSA, SHA224withRSA, SHA1withECDSA, SHA1withRSA, SHA256withDSA, SHA1withDSA Cert Authorities: ... <CN=<FQDNHERE>, OU=IT, O=FICTIONALCOMPANY, L=Havant, ST=Hampshire, C=UK> *** ServerHelloDone Warning: no suitable certificate found - continuing without client authentication *** Certificate chain *** Finished [ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)', RECV TLSv1.2 ALERT: fatal, handshake_failure
Somehow the client wasn't presenting its certificate, hence handshake failed.
concluding that 2-way SSL handshake fails if the channel is not outbound enabled, and server default channels are not outbound enabled.
Enable outbound connections for the servers requiring 2-way SSL using a WLST script:
connect(<USER>, <PASSWORD>, <URL>)
edit()
cd ("Servers/SERVER_NAME_HERE")
cmo.setOutboundEnabled(true)
cmo.setOutboundPrivateKeyEnabled(true)
save()
activate()
Restart the client and verify 2-way ssl handshake .
Uniform Cluster is introduced in 9.1.X CD and in 9.2 LTS releases .It support Application connection balancing to form ACTIVE/ACTIVE Queue Manager setup.
A MQ Uniform cluster allows a set of Queue Managers to work together intelligently to work as a single unit to distribute the load evenly across. This provides a horizontal scaled small collection of Queue Managers . These QMGR's are almost identically configured with same queues , Channels and Authentication records The Uniform cluster make sure the traffic is evenly distributed across the available queue managers for better load balancing the connection .
Advantage of a uniform cluster, each application should also be scaled into multiple matching instances, preferably with at least as many instances as there are queue managers, if not many more.An IBM MQ cluster, of whatever size, provides multiple capabilities:
A directory of all clustering resources, discoverable by any member in a cluster
Automatic channel creation and connectivity
Horizontal scaling across multiple matching queues, using message workload balancing
Dynamic message routing, based on availability
Uniform clusters use IBM MQ clustering for communication between queue managers, and balancing of workload between queues. However they differ from typical IBM MQ clusters in the following ways:
Uniform clusters typically have a smaller number of queue managers in the cluster. You should not create a uniform cluster with greater than 10 queue managers.
Every member of the cluster has near-identical configuration.
The cluster is typically used by a single application, or group of related applications.
The number of application instances connecting to the cluster should be greater than, or equal to, the number of queue managers.
Unified Cluster Example
Let us try to create a Unifrom cluster with 3 Queue Managers .
Create a file UniCluster.ini that describes how you want the cluster itself to look in terms of full repositories.As for any cluster, two full repositories act as central stores of information about the cluster.Specifically, you need to describe the names and connection names for the two full repositories in this cluster. Here we use QMGR1 and QMGR2 as full repositories.
The RepositoryNConname fields are used as the conname attribute for other cluster members to define cluster senders (CLUSSDR) to them
Create a sample configuration file, UniCluster.mqsc which contains the MQSC definitions you want to be applied to all cluster members.There is one mandatory line needed in this file, which is a definition of a cluster receiver channel (CLUSRCVR), with a CLUSTER attribute of the automatic cluster name (usually through the +AUTOCL+ insert) and a channel name which includes the +QMNAME+ insert.This describes how other members of the uniform cluster connect to each queue manager and is used as a template of how to connect to the other queue managers as well.
+AUTOCL+The automatic cluster name
+QMNAME+The name of the queue manager being created+
+VARIABLE+Any <variable name> defined during queue manager creation or in the Variables qm.ini stanza, for example +CONNAME+
Make these two files available on each machine that will host a uniform cluster member. In our Scenario We are using single machine .
UniCluster.mqsc
Simplify the demo system by disabling channel and connection authentication ALTER QMGR CHLAUTH(DISABLED) CONNAUTH(' ') REFRESH SECURITY TYPE(CONNAUTH)
The only definition required to join a Uniform CLuster when using AutoCluster is to define a cluster receiver channel
This will use the cluster name from the AutoCluster ini file setting and the connection name from the crtmqm command DEFINE CHANNEL(UNICLUSTER.+QMNAME+) CHLTYPE(CLUSRCVR) CLUSTER(+AUTOCL+) CONNAME(+CONNAME+) SHORTRTY(120) SHORTTMR(5)
Every queue manager needs to accept client connections DEFINE CHANNEL(SVRCONN.CHANNEL) CHLTYPE(SVRCONN)
Messaging resources like queues need to be defined on every member of the uniform cluster DEFINE QLOCAL(Q1) CLUSTER(UNICLUSTER) DEFPSIST(YES) DEFBIND(NOTFIXED)
Note: Replace . with * in the above content . * will not be executed in runmqsc it is for comment .
On the command line, supply:
A request to start a listener, on the expected port
A request for automatic INI configuration (-ii) pointing to the automatic cluster setup file (uniclus.ini)
A request for automatic MQSC configuration (-ic) pointing to the MQSC configuration file which includes a CLUSRCVR definition for the uniform cluster.
A variable for the CONNAME for this queue manager
# Create QMGR1 and listen on port 1414 crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1414)" -p 1414 QMGR1
# Create QMGR2 and listen on port 1415 crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1415)" -p 1415 QMGR2
# Create QMGR3 and listen on port 1416 crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1416)" -p 1416 QMGR3
mqm@ip-172-31-14-154 ~]$ crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1414)" -p 1414 QMGR1 IBM MQ queue manager created. Directory '/var/mqm/qmgrs/QMGR1' created. The queue manager is associated with installation 'Installation1'. Creating or replacing default objects for queue manager 'QMGR1'. Default objects statistics : 84 created. 0 replaced. 0 failed. Completing setup. Setup completed. [mqm@ip-172-31-14-154 ~]$ crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1415)" -p 1415 QMGR2 IBM MQ queue manager created. Directory '/var/mqm/qmgrs/QMGR2' created. The queue manager is associated with installation 'Installation1'. Creating or replacing default objects for queue manager 'QMGR2'. Default objects statistics : 84 created. 0 replaced. 0 failed. Completing setup. Setup completed. [mqm@ip-172-31-14-154 ~]$ crtmqm -ii UniCluster.ini -ic UniCluster.mqsc -iv CONNAME="127.0.0.1(1416)" -p 1416 QMGR3 IBM MQ queue manager created. Directory '/var/mqm/qmgrs/QMGR3' created. The queue manager is associated with installation 'Installation1'. Creating or replacing default objects for queue manager 'QMGR3'. Default objects statistics : 84 created. 0 replaced. 0 failed. Completing setup. Setup completed. [mqm@ip-172-31-14-154 ~]$
Each queue manager in the uniform cluster is created with an almost identical command line - all the differences between full and partial repository are handled automatically for a uniform cluster.
Start the QMGR;s using below
strmqm QMGR1
strmqm QMGR2
strmqm QMGR3
mqm@ip-172-31-14-154 ~]$ strmqm QMGR1 ; strmqm QMGR2 ;strmqm QMGR3
The system resource RLIMIT_NOFILE is set at an unusually low level for IBM MQ.
The system resource RLIMIT_NPROC is set at an unusually low level for IBM MQ.
Successfully applied automatic configuration INI definitions.
IBM MQ queue manager 'QMGR1' starting.
The queue manager is associated with installation 'Installation1'.
5 log records accessed on queue manager 'QMGR1' during the log replay phase.
Log replay for queue manager 'QMGR1' complete.
Transaction manager state recovered for queue manager 'QMGR1'.
IBM MQ queue manager 'QMGR1' started using V9.2.0.1.
The system resource RLIMIT_NOFILE is set at an unusually low level for IBM MQ.
The system resource RLIMIT_NPROC is set at an unusually low level for IBM MQ.
Successfully applied automatic configuration INI definitions.
IBM MQ queue manager 'QMGR2' starting.
The queue manager is associated with installation 'Installation1'.
5 log records accessed on queue manager 'QMGR2' during the log replay phase.
Log replay for queue manager 'QMGR2' complete.
Transaction manager state recovered for queue manager 'QMGR2'.
IBM MQ queue manager 'QMGR2' started using V9.2.0.1.
The system resource RLIMIT_NOFILE is set at an unusually low level for IBM MQ.
The system resource RLIMIT_NPROC is set at an unusually low level for IBM MQ.
Successfully applied automatic configuration INI definitions.
IBM MQ queue manager 'QMGR3' starting.
The queue manager is associated with installation 'Installation1'.
5 log records accessed on queue manager 'QMGR3' during the log replay phase.
Log replay for queue manager 'QMGR3' complete.
Transaction manager state recovered for queue manager 'QMGR3'.
IBM MQ queue manager 'QMGR3' started using V9.2.0.1.
[mqm@ip-172-31-14-154 ~]$
++++++++++++++++
[mqm@ip-172-31-14-154 ~]$ dspmq
QMNAME(QMGR1) STATUS(Running)
QMNAME(QMGR2) STATUS(Running)
QMNAME(QMGR3) STATUS(Running)
[mqm@ip-172-31-14-154 ~]$ ps -ef |grep lsr
mqm 18731 18686 0 Apr04 ? 00:00:04 /opt/mqm/bin/runmqlsr -r -m QMGR1 -t TCP -p 1414
mqm 18895 18839 0 Apr04 ? 00:00:03 /opt/mqm/bin/runmqlsr -r -m QMGR2 -t TCP -p 1415
mqm 19070 19012 0 Apr04 ? 00:00:03 /opt/mqm/bin/runmqlsr -r -m QMGR3 -t TCP -p 1416
mqm 21594 21557 0 12:39 pts/0 00:00:00 grep --color=auto lsr
[mqm@ip-172-31-14-154 ~]$
QMGR's are created with Auto config file and Unifom.mqsc definitions . Let us see the cluster information
The amqsghac, amqsphac, and amqsmhac programs are started from the command line, and can be used in combination to demonstrate reconnection after the failure of one instance of a multi-instance queue manager. We will use amqsphac in our example
MQ_INSTALLATION_PATH=/opt/mqm
export MQCHLLIB=/home/mqm
export MQCHLTAB=CCDT.JSON
# Check that the CCDT.JSON file exists before starting the applications
if [ -f "$MQCHLLIB/$MQCHLTAB" ]; then
# Start multiple instances of the sample application
for (( i=0; i<10; ++i)); do
$MQ_INSTALLATION_PATH/samp/bin/amqsghac Q1 *ANY_QM &
done
else
echo "$MQCHLLIB/$MQCHLTAB not found"
fi
RunClient.sh file with the below CCDT file . We used 10 you can change to any values to create n number of connections
There is no default JSON CCDT, and IBM MQ does not supply any tooling to create or edit CCDTs in JSON format. You do not need to be using IBM MQ for Multiplatforms to create and edit a JSON CCDT file. Using the JSON format, you can define duplicate channel definitions of the same name. When you deploy IBM MQ on the cloud, you can use this to make your deployment scalable and highly available. The JSON file is human readable, which can simplify queue manager configuration. You need no specialist tooling to maintain the CCDT file. The file is smaller. This format provides backwards and forwards compatibility. Locally by setting the MQCHLLIB and MQCHLTAB environment variables. Run the command runmqsc -n Run the DISPLAY CHANNEL command. For example, run DISPLAY CHANNEL(*). as shown in the below
Note: When a JSON CCDT is used it is possible to have multiple channels with the same name. If multiple channels with the same name exist, and they have CLNTWGHT(0) then the channels will be selected in the order that they are defined in the JSON CCDT.
sh RunClient.sh -- run this command in background using & to generate connections
[mqm@ip-172-31-14-154 ~]$ 14:02:24 : EVENT : Connection Broken MQGET ended with reason code 2009 MQCLOSE ended with reason code 2009 14:02:24 : EVENT : Connection Broken MQGET ended with reason code 2009 MQCLOSE ended with reason code 2009 14:02:24 : EVENT : Connection Broken MQGET ended with reason code 2009 MQCLOSE ended with reason code 2009 14:02:24 : EVENT : Connection Broken MQGET ended with reason code 2009 MQCLOSE ended with reason code 2009 MQDISC ended with reason code 2009 Sample AMQSGHAC end MQDISC ended with reason code 2009 Sample AMQSGHAC end MQDISC ended with reason code 2009 Sample AMQSGHAC end MQDISC ended with reason code 2009 Sample AMQSGHAC end endmqm QMGR2 AMQ8146E: IBM MQ queue manager not available. [mqm@ip-172-31-14-154 ~]$ [mqm@ip-172-31-14-154 ~]$
2009 is the error code for Connection broken .
Connection rebalenced with in couple of seconds Bring down QMGR3 . Stopping and starting a queue manager will show connections being moved to alternative queue managers and then connections being rebalanced once the queue manager is available again.
Note: that you'll probably want to end the queue manager using endmqm -r QMGR, otherwise the connected applications will terminate their connections rather than move them.
start QMGR3 Again and try with -r [ endmqm -r QMGR3 ]
When there are enough consuming application instances, there is always an instance of the application processing messages.
When you stop a queue manager, any connected application instances are evenly distributed across the remaining queue managers in the cluster.
When you start a queue manager, any application instances connected to other queue managers in the cluster are automatically rebalanced to include the newly started queue manager.
This means that the uniform cluster continually ensures that applications are optimally distributed, maximising message processing, even in the event of planned and unplanned outages.
Sample script is given below . Where we get the list of QMGR name using dspmq and awk command combination then used runmqsc . Here one thing to note is any commands that we want to run on QMGR can be passed between <<!!! and !!!. If there are no commands then it just connect to QMGR runmqsc and exit . with return code we will identify the QMGR status .$? store the return value from the previous command if it is 0 then success otherwise failure
[mqm@ip-172-31-14-154 ~]$ cat test.sh
for QM in `dspmq |awk -F"(" '{ print $2 }' |awk -F")" '{ print $1 }'`; do
runmqsc $QM > /dev/null <<!!!
!!!
if [ $? -eq 0 ] ; then
echo "$QM QMGR is running "
else
echo "$QM QMGR is not running "
fi
done
[mqm@ip-172-31-14-154 ~]$
Let us run the above test.sh and see the status .
mqm@ip-172-31-14-154 ~]$ sh test.sh
QM1 QMGR is running
QM2 QMGR is running
PR1 QMGR is not running
PR2 QMGR is not running
FR1 QMGR is not running
FR2 QMGR is not running
GWQM QMGR is not running
APPQM1 QMGR is not running
TEST.QM1 QMGR is not running
TEST.QM2 QMGR is not running
TEST.QM3 QMGR is not running
TEST.QM4 QMGR is not running
[mqm@ip-172-31-14-154 ~]$
Perfect .It gives us the correct QMGR which is up and running .
Let's imagine a situation where we want to run the script and see specific SDR Channel is not running then we need to reset and start it.
[mqm@ip-172-31-14-154 errors]$ vi AMQERR01.LOG
[1]+ Stopped vi AMQERR01.LOG
[mqm@ip-172-31-14-154 errors]$ cd
[mqm@ip-172-31-14-154 ~]$ cat chstest.sh
QMGR=$1
CHL=$2
echo " dis chs($CHL) status" |runmqsc $QMGR | grep "(RUNNING)"
if [ $? -eq 0 ] ; then
echo " $CHL is running "
exit 0
fi
runmqsc $QMGR > /dev/null <<!!!
reset chl($CHL)
start chl($CHL)
!!!
if [ $? -eq 0 ] ; then
echo "$QMGR $CHL is started "
else
echo "$QMGR $CHL is not able to strart .Pls check "
fi
[mqm@ip-172-31-14-154 ~]$
Channel looks not running and the result looks like below
[mqm@ip-172-31-14-154 ~]$ sh chstest.sh QM1 TO.QM2
QM1 TO.QM2 is started
[mqm@ip-172-31-14-154 ~]$
let see the logs to see if the channel triggered to start
----- amqrccca.c : 1121 -------------------------------------------------------
04/03/2021 01:43:27 AM - Process(1463.18) User(mqm) Program(amqzlaa0)
Host(ip-172-31-14-154.us-east-2.compute.internal) Installation(Installation1)
VRMF(9.2.0.1) QMgr(QM1)
Time(2021-04-03T01:43:27.142Z)
ArithInsert1(1)
CommentInsert1(TO.QM2)
AMQ8023I: IBM MQ channel reset.
EXPLANATION:
Channel 'TO.QM2' has been reset, the new sequence number of the channel is 1.
ACTION:
None.
----- amqkchla.c : 2929 -------------------------------------------------------
04/03/2021 01:43:27 AM - Process(15224.1) User(mqm) Program(runmqchl)
Host(ip-172-31-14-154.us-east-2.compute.internal) Installation(Installation1)
VRMF(9.2.0.1) QMgr(QM1)
Time(2021-04-03T01:43:27.174Z)
CommentInsert1(TO.QM2)
AMQ9002I: Channel 'TO.QM2' is starting.
EXPLANATION:
Channel 'TO.QM2' is starting.
ACTION:
None.
----- amqrccca.c : 469 --------------------------------------------------------
04/03/2021 01:43:27 AM - Process(15224.1) User(mqm) Program(runmqchl)
Host(ip-172-31-14-154.us-east-2.compute.internal) Installation(Installation1)
VRMF(9.2.0.1) QMgr(QM1)
Time(2021-04-03T01:43:27.347Z)
CommentInsert1(TO.QM2)
AMQ9299I: Channel 'TO.QM2' has started.
EXPLANATION:
The channel 'TO.QM2' has finished starting.
ACTION:
No action required.
----- amqrccca.c : 969 --------------------------------------------------------
Looks fine . Let us try to stop the channel and run the script once again .
[mqm@ip-172-31-14-154 ~]$ sh chstest.sh QM1 TO.QM2
QM1 TO.QM2 is started
[mqm@ip-172-31-14-154 ~]$
[mqm@ip-172-31-14-154 ~]$
[mqm@ip-172-31-14-154 ~]$ runmqsc QM1
5724-H72 (C) Copyright IBM Corp. 1994, 2020.
Starting MQSC for queue manager QM1.
dis chs(TO.QM2)
1 : dis chs(TO.QM2)
AMQ8417I: Display Channel Status details.
CHANNEL(TO.QM2) CHLTYPE(SDR)
CONNAME(172.31.14.154(1415)) CURRENT
RQMNAME(QM2) STATUS(RUNNING)
SUBSTATE(MQGET) XMITQ(QM1.XMITQ)
end
2 : end
One MQSC command read.
No commands have a syntax error.
All valid MQSC commands were processed.
[mqm@ip-172-31-14-154 ~]$
Run the script again to see the result
[mqm@ip-172-31-14-154 ~]$ sh chstest.sh QM1 TO.QM2
RQMNAME(QM2) STATUS(RUNNING)
TO.QM2 is running
[mqm@ip-172-31-14-154 ~]$
As the QMGR channel TO.QM2 is up and running fine . script just displayed information .
Now this script is required to configure in cron to run every 2 min to check the status and start the channel if it is down for some reason .
IBM MQ 9.2.0 releases with new features and enhancements on the top of 9.1.0 . The features that have access is based on the entitlement that organizaton purchanges .
Majority of the features are available in CD release of 9.1 to 9.1.5 . along with that some new are introduced in 9.2.0 LTS ( Long Term support )
what each purchase entitles you to install, see IBM MQ license information. [https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.2.0/com.ibm.mq.pro.doc/q126140_.html?view=kc#q126140_]
One of the main features of the LTS releases is that they are primarily made up of features previous released in a CD release.
That is certainly the case with 9.2, having new features available in previous CD releases gave customers the opportunity to try them out as well as for IBM to tweak those features a bit before releasing them in the LTS release. Not all features that are in 9.2 were in a CD release but the list of what is completely new, is small and specific.
This type of release give more flexibility and stability with LTS . but if some one use CD will need to plan their patch management due to multiple releases but worth of using it CD release.
On Multiplatforms, that is, all platforms (Lin Win AIX ..etc )the following features and enhancements are available with both base product entitlement, and with Advanced entitlement. Below are some of the hi-lighted new features with 9.2 .
Automatic balancing of a pool of connected applications across a set of available queue managers Uniform cluster. A key benefit of the uniform cluster is that the applications can connect to any of the queue managers within it. This is supported by a consistent set of definitions (which has always been possible), but the key is the changes to the infrastructure that allow client applications to connect to the best queue manager
Increased level of control to determine how IBM MQ uses available storage Queue storage. You can now limit the size of the queue not just the number of messages
TLS 1.3. Extended connectivity security support Updated IBM MQ Console look and feel
REST messaging. Quick and easy access to MQ for applications without the need to use the traditional client API
Java 11 clients. Support for java 11 for clients which is important because many users are moving off java 8
For example, before IBM MQ 9.2.0 the MQSC command could be sent to the /admin/action/qmgr/{qmgrName}/mqsc resource in the following format:
Disaster recovery for high availability replicated data queue managers RDQM. A very powerful failover option for Linux environments that is more flexible than existing multi-instance or do it yourself options.
IBM MQ Internet Pass-Thru (MQIPT) is a utility that can be used to implement messaging solutions between remote sites across the internet. In IBM MQ 9.2.0, MQIPT is a fully-supported optional component of IBM MQ that you can download from IBM Fix Central for IBM MQ. MQIPT has previously been available as support pack MS81.
Container improvements. Simplified deployment options as the evolution of Kubernetes and OpenShift evolve .zip images of IBM MQ are available for use with development and production containers And many more
Refer below for more details from IBM official documentation . https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.2.0/com.ibm.mq.pro.doc/q134120_.htm
IBM MQ has started providing mqweb .using which one can be able to administer the MQ on the server suing MQWEB . It can be managed using API or MQ Web Console .
Here we will set how to start MQ Web console and some configuration which will help to begin with .
Starting from MQ 9.X onwards by default with MQ installation mqweb is getting installed to start there are some configuration and startup service needed . Let's see it .
On a linux system by default MQ Web configuration are available in the below path
[mqm@ip-172-31-14-154 mqweb]$ ls -lrt
total 24
-rw-r-----. 1 mqm mqm 589 Dec 29 15:56 server.xml
-rw-rw-r--. 1 mqm mqm 858 Dec 29 15:56 jvm.options
-rw-rw----. 1 mqm mqm 891 Mar 28 05:05 mqwebuser.xml_bkp
drwxrwsr-x. 3 mqm mqm 22 Mar 28 05:07 resources
-rw-rw----. 1 mqm mqm 8763 Mar 28 05:14 mqwebuser.xml
drwxrwsr-x. 5 mqm mqm 151 Mar 28 05:14 workarea
drwxrws---. 3 mqm mqm 134 Mar 28 05:14 logs
[mqm@ip-172-31-14-154 mqweb]$
mqwebuser.xml is the configuration file that need to modify . Let's take backup of the file and copy basic_registry.xml as mqwebuser.xml then start mqweb using strmqweb command utility .
[mqm@ip-172-31-14-154 mqweb]$ dspmqweb
MQWB1125I: Server 'mqweb' is not running.
[mqm@ip-172-31-14-154 mqweb]$ strmqweb
Starting server mqweb.
Server mqweb started with process ID 1895.
[mqm@ip-172-31-14-154 mqweb]$ dspmqweb
MQWB1124I: Server 'mqweb' is running.
URLS:
https://localhost:9443/ibmmq/rest/
https://localhost:9443/ibmmq/console/
[mqm@ip-172-31-14-154 mqweb]$
Once mqweb is started successfully then display the status using dspmqweb . if it is running then it will show 2 url's . API and a console url .
if you looks at the url's they are running on loop back address . to startup using the serverip or any interface on the system update mqwebuser.txt file with httpHost
<!--
<variable name="httpHost" value="*"/>
-->
<basicRegistry id="basic" realm="defaultRealm">
<!--
This sample defines two users with unencoded passwords
and a group, these are used by the role mappings above.
-->
<user name="mqadmin" password="mqadmin"/>
<user name="mqreader" password="mqreader"/>
<user name="mftadmin" password="mftadmin"/>
<user name="mftreader" password="mftreader"/>
<group name="MQWebAdminGroup">
<member name="mqadmin"/>
</group>
</basicRegistry>
By default it is commented . Uncomment it and restart mqweb.
endmqweb and update httpHost and start using strmqweb
mqm@ip-172-31-14-154 mqweb]$ endmqweb
Stopping server mqweb.
Server mqweb stopped.
[mqm@ip-172-31-14-154 mqweb]$
<variable name="httpHost" value="*"/>
MQWB1125I: Server 'mqweb' is not running.
[mqm@ip-172-31-14-154 mqweb]$ strmqweb
Starting server mqweb.
Server mqweb started with process ID 2721.
[mqm@ip-172-31-14-154 mqweb]$ dspmqweb
MQWB1124I: Server 'mqweb' is running.
URLS:
https://ec2-18-218-83-33.us-east-2.compute.amazonaws.com:9443/ibmmq/rest/
https://ec2-18-218-83-33.us-east-2.compute.amazonaws.com:9443/ibmmq/console/
[mqm@ip-172-31-14-154 mqweb]$
Now try to access the web console using the URL and 9443 is the default port for this to access .
mqweb login page
Login with the default id that was configured in mqwebuser.xml file .mqadmin/mqadmin in our case .
We can see all the QMGR running on the server and its status .
We can create configure administer monitor all the MQ . all the operation sthat we can do from runmqsc will be able to perform from the MQ web console .
It is not mandatory to use mqweb or start the process . but it is one of the option given int the latest mq .if you wish that it cat help you can start using it and play around .
User's are unable to login to Websphere admin console where LDAP is integrated . From the logs we could see below error
00000048 SSLHandshakeE E SSLC0008E: Unable to initialize SSL connection. Unauthorized access was denied or security settings have expired. Exception is javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at com.ibm.jsse2.b.a(b.java:33) at com.ibm.jsse2.nc.a(nc.java:456) at com.ibm.jsse2.nc.unwrap(nc.java:373) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:26) at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.readyInbound(SSLConnectionLink.java:534) at com.ibm.ws.ssl.channel.impl.SSLConnectionLink.ready(SSLConnectionLink.java:294) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214) at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
Since we are unable to login to Websphere admin console .Lets try to disable the security manually
To disable security manually follow below steps
1.Take backup of security.xml file located in <PROFILE_HOME>/config/cells/CellName/security.xml
2.update enabled="true" to enabled="false" on the first occurrence in security.xml file then kill the admin process that is running as we cannot stop since we don't have working userid and password .
hi-lighted string decide the security of admin console .It will act as a switch to on/off
3.start the dmgr or AppNode Now we should be able to login to Admin without userid & password
Once logged in, proceed to "Security -> SSL certificate and key management".
From there, select "Key stores and certificates -> NodeDefaultKeyStore -> Personal certificates".
Once the configured certificates are displayed, you have to check the expiration of the certificate.
If the certificate's are still valid then issue might be due to certificate validation add below JVM argument and restart the server
-Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true to avoid possible "No subject alternative names"
Follow below steps to add JVM argument in WebSphere Application Server
-->Login to Admin console .In the left pane, expand the Servers node.
-->Click Application Servers. Then click the name of the server that you want to configure.
-->Select Process Definition ► Java Virtual Machine.
-->Provide the JVM options in the Generic JVM Arguments text box.
Example : -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true
Click Apply. Then click Save to save the changes to the master configuration.
-->Restart the server so that the new settings are used.
You can verify in the ps -ef |grep -i WebSphere output for the newly added JVM argument .
4. Then go security.xml file and change back to enabled="true" . Kill the admin process then start it try to login to admin console with LDAP id and password .
Issue might be due to Websphere patch update where JAVA is updated and started validating subject alternative names . Issue resolved with above JVM argument .
Some times, upon rebooting your computer, Websphere (Network Deployment) will start but the node agent won't. The node agent is responsible for spawning and killing server processes and also responsible for configuration synchronization between the Deployment Manager and the Node. None of the distributed products would work if the node agent isn't up. You will not be able to start the node agent from the Websphere Admin Console GUI. You will have to start it from the command line utility.
To start Node agent for a specific App node we should navigate to that profile bin .From there only we will be able to start it
Steps to start Node Agent on Windows server
Open up a Command Prompt as an administrator (right-click -> Run as administrator)
Navigate to you Websphere server bin directory: ie C:/Program Files/IBM/Websphere/profile2/profiles/AppSrv01/bin
Run startNode.bat
You should see the confirmation saying "node agent is open for e-business".
Now log back into the WAS Admin Console. Click on "System Administration" -> node agents -> refresh the status. You should see a green arrow upon refresh.
Now you can start your cluster or server by clicking on "Servers" on the left panel from the DMGR Admin console .
On Linux/AIX server use startNode.sh .remaining all the steps are same .
For each App Node there will be only one nodeAgent associated that required to start to manage any services under it from DMGR Admin console .
In Websphere there is a possiblility to disable Admin security and login to Admin console without userid and password . This feature is not available in JBOSS EAP, Weblogic Server etc
Below steps are applicable for both DMGR and Standalone websphere nodes .
Generally we don't require this .but there might a situation where one the following reasons
Someone changed the password or the password is locked if it is LDAP Account .
we may forget the credential . There is no way to get the working credentials .
Messed up the SSL/Security configuration or security.xml
To disable security manually follow below steps
1.Take backup of security.xml file located in <PROFILE_HOME>/config/cells/CellName/security.xml
2.update enabled="true" to enabled="false" on the first occurrence in security.xml file then kill the admin process that is running as we cannot stop since we don't have working userid and password .
hi-lighted string decide the security of admin console .It will act as a switch to on/off
3.start the dmgr or AppNode Now we should be able to login to Admin without userid & password
4.Do what ever changes we need to reset the password and other configuration's if need to do then go security.xml file and change back to enabled="true"
5.kill the Admin process and start Admin now we should be able to login to Admin console successfully with new password .
Other way
We can also disable security by using wsadmin commands
1.connect to wsadmin
wsadmin>securityoff
LOCAL OS security is off now but you need to restart server1 to make it affected.
wsadmin>
2.kill the admin process
3.start Admin process
4.Login to Admin console and do the necessary changes
5.connect to wsadmin and run securityon to enable security back
wsadmin>securityoff
6.Restart DMGR or Admin Node and login with new password .
