Using Google Chrome?

Download my
free Chrome Extension, Power Notes Searcher, to make searching for and evaluating SAP notes, much easier.

Recent Posts

Sunday, July 17, 2011

VMWare ESXi 4.1 Test Rig - UPDATE

This is an update to my previous post here.

The VMware install onto the HP XW6200 hardware was a snip:
VMware ESXi 4.1 Update 1 Install Guide
VMware ESXi 4.1 Update 1 Release Notes

I decided to use a spare 8GB USB pen (check requirements in the installation guide relating to swap space) to install the VMware hypervisor on.  This enables me to use the entire size of the internal SATA disks as VMware datastores for hosting guest operating systems.  It also means I can create a backup copy of the USB pen when trying out a VMware upgrade.

Total time to install:
- Download boot CD ISO      = 10mins.
- Burn boot CD from ISO      = 5 mins.
- Install from CD to USB pen = 10 mins.
- Download vSphere Client    = 5 mins.
                                TOTAL = 30mins.

Once installed, I then downloaded the VMware standalone converter to my laptop.
I had two VirtualBox guest machines on my laptop, so I used the VMware converter to simply pull them off (there is no pun here!) onto the VMware virtual infrastructure.
My LAN is not the fastest so total time to v2v conversion was about 10 hours to do both guest VMs.

The next problem was the Oracle Linux 6.1 install.
This seems to come in two separate parts; the boot ISO and the main DVD ISO.
Downloading the main DVD ISO took about 3 hours over my simple ADSL connection.

Once downloaded, my problems started.
I attached the Oracle Linux 6.1 boot ISO to the virtual CDROM of a new 64bit virtual machine and click "Power On".
Then, bam!    No longmode enabled:



What!
Well, it turns out that the history of the Intel XEON processor is a rather tarnished story of rivalry between Intel and AMD.
Although the XEON processor I have *is* 64bit capable, and does support EM64t with Hyperthreading and all that, it just doesn't have the VT (virtualization) capability.
The spec of the XEON processors in the HP XW6200 workstations can be seen on the Intel page here http://ark.intel.com/products/27086/64-bit-Intel-Xeon-Processor-3_40E-GHz-2M-Cache-800-MHz-FSB.

Looking at the Intel page is even more confusing, since it states it doesn't have Hyperthreading, but I have just enabled it in the BIOS.  Does it?  Doesn't it?    Arghh!

In summary, you're better off getting any one of the Core2Duo processors as these will undoubtedly support EM64t and VT.
Only problem is, most don't support 8GB of RAM due to the system board limitations (more than likely a soft limitation).  Oh dear indeed.

So my story doesn't end there, I'm now in the process of downloading and installing 32bit Oracle Linux 6.1.  Not quite what I wanted, but it will suffice because I have limited memory so will not be needing the use of 64bit memory addressing capabilities - yet.
My other task is to download the 32bit version of the Oracle 11g XE.  But wait, Oracle only provide a Linux version of the 11g XE in 64bit mode.  Scuppered!  I'll have to use the 32bit version for Windows only.

As you can see, we are in this awkward transition period where software is both 32bit and 64bit but hardware is 32bit or 64bit or 64bit with flags.
We are trying our best to virtualise, but for the small guy it's an expensive project.
You either buy a server or you buy the latest and greatest desktops that support more than 4GB of RAM.
It's a very tricky thing to get right.
For me, it's a case of trying to keep skills current.  I think I can afford £166 for now, then re-invest another £166 at a later stage to make the jump to a "proper" 64bit (with VT) capable system.

One other thing, once the free version of VMware ESXi expires (you get 60-days free use of all the features), you'll need to spend some time faffing with the license to register it (downgrade to free only features) so you can continue to use it. I did this up-front so I don't have a situation where I can't use the VMs because I forgot the expiry date.
See instructions here: http://planetmediocrity.com/2010/09/vmware-vsphere-evaluation-downgrade/.

Friday, July 15, 2011

VMWare ESXi 4.1 Test Rig - Up up and away!

This weekend I will be focussing my attention on getting my new (old) HP XW6200 workstation transformed into a VMWare ESXi 4.1 test rig.
Bought on eBay for £166.00.

I did take a brief look at the Oracle VM 2.2 offering, but it required about twice the amount of storage space to install it compared to the ~200MB required by VMWare ESXi 4.1 and it didn't have the nice GUI for simplicity or the excellent VMWare converter for transfering my existing VirtualBox guest VMs to the virtual infrastructure.

The plan is to install VMWare onto a USB pen device, then use the 500GB SATA disk and 1TB SATA disk as virtual disk storage where I can host all manner of Virtual Servers.
I'm planning to use OpenFiler inside a virtual appliance, to provide a shared chunk of disk via iSCSI so that I can get ASM working in a cluster.
I'm hoping to get an Oracle Unbreakable Linux server running and download the beta Oracle 11gR2 XE RDBMS.
For those of you in the know, Oracle don't officially support VMWare (I wonder why...) but since I'm not running a production database, this is not a problem.  I know for sure that it *will* work fine.

The XW6200 machine spec:
2 x Intel Xeon 3.4gGhz with HT and EM64t.
256MB graphics card.
4GB RAM (for the moment, but will support 8GB via 4x2GB DIMMS).
500GB SATA
1TB SATA
CDROM
Floppy (really!)
USB sockets galore.

It's a little power sucker, but it supports wake-on-lan so I can have it shutdown until required.

Monday, July 11, 2011

When Should I Pay For Oracle Diagnostics/Tuning Pack in 10G/11G


If you are unsure about what action will result in you having to pay for a license for the Oracle Diagnostic Pack or Tuning Pack (or other options come to think of it), then you could use the query below to determine the check conditions:

SELECT NAME, DESCRIPTION FROM WRI$_DBU_FEATURE_METADATA;

This will output the name of the feature and the description of the check that is performed.
You could also see the check "code" if you include the USG_DET_LOGIC column in the SELECT.

In 10G it is immediately possible to see that the AWR is only classed as used if "At least one Workload Repository Report has been created by the user".
In 11G, AWR is classed as used if "At least one AWR Baseline has been created by the user" OR "At least one AWR Baseline Template has been created by the user".

The AWR is licenseable (http://www.orafaq.com/wiki/AWR).

So does this mean you can select from those views? Well Oracle do ask you to in some support situations. So maybe it is reasonable to assume so.
The Oracle 11G Diagnostics Pack doc http://www.oracle.com/technetwork/oem/grid-control/overview/diagnostics-pack-11gr2-datasheet-134502.pdf states otherwise: "These features may be accessible through Oracle Enterprise Manager , Oracle SQL Developer and APIs provided with Oracle Database software. The use of these and other features described in the product licensing documentation requires licensing of the Oracle Diagnostic Pack regardless of the access mechanism.".

The Oracle 11G Tuning Pack *does not* say the same: http://www.oracle.com/technetwork/oem/grid-control/overview/tuning-pack-11gr2-datasheet-134450.pdf
But, the tuning pack APIs use features included in the Diagnostic Pack.  Chicken and egg.

I guess it's open to interpretation. Better stay safe and ensure that you don't hit any of the checks.

Saturday, July 09, 2011

Analysing SAP Java Memory Usage in Windows

Running SAP systems on a Windows operating system provided me with some difficulties when it came to understanding memory usage. I was used to Unix based systems and using terms like "swap file", "virtual memory" etc.
Hopefully this article will help you diagnose SAP Java system memory issues on Windows, especially “Out Of Memory” (OOM) error situations in the SAP Java stack.
It has been written based on Windows Server 2003 64bit with SAP Netweaver 7.0 (2004s) and Oracle 10g (10.2.x).

REFERENCES:

Understanding memory usage in Windows 2000.
SAP Note 723909  v33 - Java VM Settings For J2EE 6.40/7.0
SAP Note 879377 - Adjusting SDM Java Memory Usage
SAP Note 716604  v56 - Sun J2SE Recommended Options
SAP Note 313347 - Windows Editions Memory Support
Netweaver 7.0 SR2 Java Windows Oracle Installation Guide.

WINDOWS MEMORY TERMINOLOGY

Before we go into the memory details of an SAP system on Windows, it is useful to understand the memory terminology of the Windows operating system.
This is useful if you come from a UNIX background.

RAM – Random Access Memory, physical installed memory in the server.

Page File – An area of hard disk (or other storage medium) used for temporary storage of memory blocks. Generally this should be manually set according to the installation guide for the SAP system.

Virtual Memory – The logical memory accessed by user space program (calculator or any other app) this is roughly the summary of physical RAM plus page file size.

Kernel Memory – This is the area of virtual memory and RAM used by the operating system for device driver memory (non-user space programs).

Page Faults – When a process tries to access a portion of virtual memory that has been swapped to disk, a page fault occurs and the swapped portion is read back into RAM from disk potentially displacing an existing memory page).

Page – A block of memory usually around 4K in size.

DETERMINE OS MEMORY

We need to first determine how much memory is actively being used at the operating system level and how much memory is available to us.
On the Windows server, open Windows Task Manager and click on the “Performance” tab:



This will show the current memory usage of the entire server.
Ignore the line graphs and histograms as the bottom section of the pane is more important.
The following sections are described:

Physical Memory - Total Total amount of RAM installed in server.
Physical Memory - Available RAM available for CPU processes.
Physical Memory - System Cache RAM being used by the file cache.
Commit Charge - Total Size of virtual memory in use.
Commit Charge - Limit Max amount of virtual memory.
Commit Charge - Peak Virtual memory used high water mark.
Kernel Memory - Total Paged and non-paged mem used by the operating system's kernel e.g. device drivers etc.
Kernel Memory - Paged Virtual memory set aside for the kernel.
Kernel Memory - Nonpaged Physical RAM set aside for the kernel.

The most important to watch when measuring memory usage of an SAP system are the following:

Commit Charge – Peak: This is the high water mark for memory usage on the server. If this is close to the maximum available virtual memory (Commit Charge – Total) then you could be suffering performance issues caused by excessive paging to/from hard disk.

Physical Memory – System Cache: This is the file system cache that stores data accessed from the hard disk in RAM for more efficient data retrieval.
This value is maintained by the operating system but can be influenced by setting the performance options in “My Computer -> Properties -> Advanced -> Settings (Performance) -> Advanced” shown below:



Which is also related to (or rather overridden by) the setting of “Maximize data throughput for network applications” in the “File and Printer Sharing for Microsoft Networks” of the local network connections in control panel:




MANAGE OS MEMORY

Before you try and tune the memory in the SAP system, you should first explore all avenues of optimising the amount of RAM available in the operating system.
We can see that there are three factors that affect the amount of RAM available for use with programs:

1, Other programs running.
2, The file system cache.
3, Device drivers loaded in the system (kernel).

Reducing the number of running programs and services is a good way of freeing up memory.
The amount of memory used by a specific program or service can be seen in Task Manager on the “Processes” tab:


Only the needed applications for the application and its management should be running.
The svhost.exe binary signifies services that are visible in the services applet in the Administrative Tools applet in Windows Control Panel.
Disabling redundant services will improve available RAM.

You should seek to reduce the file system cache as much as possible.
Since the SAP system is not a file server, any file system level buffering of data is only wasting resources. This is especially true for the Oracle database which buffers read data in memory anyway, which is then potentially buffered again in SAP.

You can reduce the system cache by setting the “Maximize throughput for network applications” as discussed in the previous section. This should be done on all servers within the SAP landscape (apart from specific cache servers, subject to SAP recommendations).
This is generally recommended by SAP in the installation guide for the Netweaver system.

Lastly, disable any redundant devices from the device manager.
These devices require kernel memory, some of which is taken directly from RAM.
You can see the memory ranges allocated to devices in the device manager accessed from the “Computer Management -> Device Manager” option from the Administrative Tools applet in Windows Control Panel.
Simply select “View -> Resources by Connection” in the Computer Management panel:



Expand the “Memory” item from the tree on the right and the devices will be listed with their memory address ranges:


The address ranges are in hexadecimal notation and in the form [RegionLetter][HEX Values] e.g. A0000000 – AFFFFFFF. The value in hex is bytes and is calculated as follows: FFFFFFF = 4294967295 bytes in decimal. Divide the decimal by 1024 accordingly to get a value in Kilobytes, Megabytes or Gigabytes.

Devices which are not required can be disabled (right click and choose “Disable”).

DETERMINE SAP JAVA SYSTEM MEMORY

The memory allocated to an SAP system comprises of three parts:

1, The Database instance memory.
If the database is on the same server as the SAP system, then some of the operating system memory will be taken up by the database itself.

2, The ABAP system memory.
If the SAP system is a dual stack system with an ABAP instance also on the same server then some memory will also be taken up by the ABAP instance.

3, The Java stack memory.
The Java stack comprises of mainly three (sometimes four) running Java Runtime Environments (Server, Dispatcher & SDM).

The main memory of the server must be divvied up between these 3 main components of the SAP system.

Since we are concentrating on the Java stack only, we will assume that SAP recommendations were followed correctly for the setup of the database and ABAP components.

Using Windows Task Manager it is possible to determine the amount of memory in use by the SAP Java system.
The jlaunch.exe processes represent the Java Runtime Environments of the Java stack and therefore the memory usage for each process equates to the current virtual memory usage for the SAP processes:



In the Windows Task Manager you must change the view to be able to see the process ID of a process:


Ensure that you select the following columns:

PID,
Memory Usage,
Peak Memory Usage,
Page Faults
Virtual Memory Size,
Paged Pool,
Non-paged Pool


With these columns selected, we can get a far better overview of the memory usage for the SAP components:


You can now see that the actual virtual memory usage (VM Size) of the PID 5144 (what we think is the SAP Java server process) is ~1.7GB in size (1,678,188 K).

The summary of the VM Size column should roughly add up to the size of the “Commit Charge – Total” value displayed on the “Performance Tab”.

Generally you can guess that the jlaunch processes with the highest memory usage will be the SAP Java server processes (since they have the largest memory settings out of the box), followed by the dispatcher and finally the SDM (with the smallest default settings).

For confirmation, it is possible to find the exact process id using the jcmon.exe program.
Use option 30 (Shared Memory Menu), then select option 2 (Display Process Data):


The running components will be displayed and we can confirm that PID 5144 is in fact the server0 process:



DETERMINE SET MEMORY VALUES

We have determined what virtual memory the SAP Java processes are actually using but we need to see the configuration settings of the Java processes so we know what to expect at startup and maximum use.

There are a number of different areas of memory for a Java Runtime Environment.
The following is a very brief overview:

Heap Size The total amount of memory stack available to a Java Runtime Environment process (excludes the resident memory for the binary). The minimum heap memory is set using parameter “–Xms”.

New Size Also known as the “young generation”, is the amount of heap memory allocated by default for new java classes created within the JRE. The new size is set using parameter “–NewSize”.

Perm Size Also know as the “old generation” or “tenured generation” is the amount of heap memory allocated by default for java classes that have survived a number of garbage collections and are deemed “permanent” in the JRE. The perm size is set using parameter “–PermSize”.

Any java classes that are no longer referenced are cleared from memory during the garbage collection cycles which are either time-triggered or based on a percentage of free heap size.

The sections of heap memory and the heap memory total size itself can be set within bounds using the “-MaxXXXXSize” arguments or “-Xmx” (for the heap itself) to the JRE.
The JRE will not break these bounds.
If the amount of Heap allocated is not enough for the running JRE it will constantly try to garbage collect more aggressively. This will be seen in the developer trace files e.g. dev_server0, dev_dispatcher and dev_sdm.

If the JRE is still unable to reference enough memory, then a Java.Lang.OutOfMemoryError will be thrown. If this is visible in the dev_server0 trace file, then this is the JRE that threw the error and was suffering the problem (more than likely).

If no maximum heap size is set using the –Xmx parameter, then the JRE will attempt to expand the heap dynamically when required (it may not shrink back).
If the Windows virtual memory is exhausted, then the JRE will eventually throw a Java.Lang.OutOfMemoryError and Windows may throw its own error.

Now we understand roughly how the JRE memory is utilised, we can see the settings for the respective SAP JREs for the server, dispatcher and SDM.

The JRE server0 memory allocation can be seen either through ConfigTool by selecting “Cluster-Data -> Instance ID XXXXXX” -> Server” and checking the parameters on the right hand side:


Or by looking in the jlaunch process trace file:


This concludes the analysis of memory usage for the Java stack.
Using this information you should now be able to determine if this meets the recommendations by SAP.

SAP JAVA STACK MEMORY RECOMMENDATIONS

During the creation of this article a number of SAP notes became of significance.
Generally SAP try to recommend certain settings during installation of the SAP software, and then at a later date, they will release a note that details changes to the installation settings.

The base install of SAP Netweaver 7.0 Java (SR3) seems to install with the following settings:

Server:
-Xms 1024m
-NewSize 171m
-MaxNewSize 171m
-PermSize 256m
-MaxPermSize 256m

Dispatcher:
-Xms 170m
-NewSize 57m
-MaxNewSize 57m

SDM:
-Xms 256m

The installation documentation specifically states that the sizing of the Windows page file is essential for correct operation of the SAP system.
The document states in the section “3.1.2 Requirements Checklist for a Central System”:

Minimum RAM:
1.5 GB
To check RAM, in the Windows Explorer choose Help About Windows.
Paging File Size: 1 times RAM plus 8 GB


This would infer that on a system with 10GB of RAM, the page file should be 18GB in size. This would give a total virtual memory size of 28GB.

According to note 723909, SAP’s latest recommendations for sizing the JRE memory is as follows:

For 32 bit platforms we recommend to start the server nodes with 1GB
whereby the initial and the maximal heap size should be equal: -Xmx1024m
-Xms1024m
Higher values may cause troubles, for example see notes 736462 and 664607
about Windows DLLs preventing big java heap.
We recommend using additional server nodes instead of huge heaps.

For 64 bit platforms (that means not only your OS but also your JDK
supports 64 bit) we recommend using 2GB: -Xmx2048m -Xms2048m
Take into account while planning your productive landscape: for NW 7.0
(04s) there is a general recommendation to use 64bit systems, see note
996600.
2.2 For dispatcher nodes it's sufficient to set the heap size to 171m on 32
bit and to 256m on 64 bit platforms. There is no need to apply the
parameters mentioned under 4-9 below.
Default size of the young generation (one third of the heap) is ok for
dispatchers.
If the instance contains more than three server nodes we recommend to add
50MB of dispatcher heap for each additional server node.
2.3 The max heap size (plus max perm size) of all Java server nodes
must fit completely into the physical memory in order to avoid heavy
paging on OS level. You must also consider the space needed for OS
and other processes.
For each JavaVM on the server, all Java memory must fit completely
into physical memory (no paging)!
Therefore the rule of thumb for number of server nodes for SUN and HP
JavaVM with 1 GB Heap each will be
#ServerNodes = (AvailableMemory / 1.5 GB)
and factor 2.5 should be used instead of 1.5 for 64 bit with 2GB heap size.


The additional note 716604 recommends some specific settings for the SUN JVM:

For 6.40/7.00:
For the heapsize and the size of the new generation please take care of the
recommendations of note 723909.
For Windows (ia64 and x86 (64Bit)) and Linux/ia64 (Itanium) platforms set
the stack size to at least 2MB for each serverID, the dispatcher and the
Java based tools:
-Xss2m
Additionally, increase the stack size of the JIT compiler threads at least
on the server nodes of the J2EE Engine to 4MB on these platforms with
following parameter (effective on Windows starting with 1.4.2_17 b12 resp.
1.4.2_18):
-XX:CompilerThreadStackSize=4096


Any modifications to the Java stack should be proven in the DEV and QA environments before application to production.

Friday, July 01, 2011

Basic Performance Tuning Guide - SAP NetWeaver 7.0 - Part II - Reading Single Record Statistics

This is Part II. You may like to see Part I.

In Part I, we showed you how to trace an ABAP program using STAD to show the single record statistics of the response times for a transaction/program.
In this part, we will go through and decipher the statistics to help you decide where the problem may lie in your slow performing program.

Following Part I, we saw the STAD screen output.  Below is a snapshot of the right hand side of the screen showing the "standard" view fields:

STAD output fields

The "Response time (ms)" field shows the total response time for the dialog step (or summary of steps if you have chosen not to expand the left tree).

As a reminder, SAP says (http://help.sap.com/saphelp_nw70ehp1/helpdata/en/21/2c8f38c7215428e10000009b38f8cf/content.htm
):
"Response time = wait time + execution time
where: execution time =
Generation time during the run
+ Load time for programs, screens, and graphical user interfaces
+ Roll times for rolling in the work data
+ ABAP processing time
+ Database time
+ Enqueue time for logical SAP lock processes
+ CPIC/RFC time
+ Roll wait time (excluding task types RFC/CPIC/ALE).
"

So if we take a look at our initial screen of values, the example below shows that the "DB req time (ms)" is one of the larger values (4929):

SAP STAD DB request time of response time

The total database request time (time taken for database work to be done for this step) is more than 92% of the total response time for the step.
If the database time IS NOT a significant percentage of total response time, then you should check the other parts that make up the total response time, for clues.

At this point, it's better to change the screen columns so that you can see all the timings.
The best view is the "Time Analysis" view.  Switch views by clicking the "Sel. fields" button:

STAD select fields

Click "Time Analysis":

STAD time analysis

You will now see more timing fields

STAD timing fields, response time
STAD timing fields, response time

The fields highlighted are responsible for the total response time calculation.

In our example, TOTAL RESPONSE TIME = Wait Time + DB Req Time + Processing Time + Load Time + Generating Time + Roll In Time + Roll Wait Time + Enqueue Time.
=> 5330 = 0 + 4929 + 122 + 17 + 0 + 262 + 0
(5330 / 1000 = 5.3 seconds total response time)

You should check each of the fields, looking for disproportionate response times.

The "Wait time (ms)" indicates the amount of time the dialog step was waiting to be processed by a dialog work process.  High values indicate that the overall system performance is degraded.  Maybe a lack of memory, CPU or I/O performance at the O/S layer could be causing issues.  Check the dispatcher queue.

The "Time in WPs (ms)" indicates the time that the dialog step was being actively serviced by a work process and not by any other type of process (BTC, UPD, UPD2 etc). Check "SM51 -> Goto -> Server Name -> Information -> Queue Information" and SM66 to see how busy the system is.

The "Processing time (ms)" indicates the amount of time the individual components spent processing (working) and not idle or sleeping or waiting.  Check SM50 and SM66 to see how busy the system is.
Use ST06 (O/S monitor) to check CPU usage and system load.

The "Load time (ms)" indicates the amount of time taken to load the program code into memory buffers.
A problem here could indicate the program buffer is not large enough (see ST02).  You should check the CPU time if the load time value is large.

Values for "Generating time (ms)" show the time spent compiling the ABAP source code.  You will see a value here if the code has not already been compiled through SGEN.  Any large values here for programs already generated could mean a problem in the database with the code storage tables.

Large values for "Roll (i+w) time (ms)" indicates that the Roll-In and Roll-Wait times are being affected.
Roll-In is the process of moving the users session context into the work process so that it can be used to execute the dialog step.  High values could indicate that the roll buffer is too small (see ST02) or that the user has a very large set of roles and profiles and is consuming too much memory.

Values for "Enqueue time (ms)" show the time waited for a logical database lock to be performed.
A high value here could indicate excessive contention (someone else doing the same work) in the SAP system or the underlying database. Maybe check SM13.


If we look at a CPU time example, a high value here would indicate a large amount of overall processing causing the WP to use more CPU.
Maybe a rogue loop, or a large internal table operation.
The example below shows a high CPU time for a background job performing a lot of RFC calls.
Notice how the processing time (just off the screen shot, sorry) has increased too (some component has to do the work):

STAD CPU time, response time

Could be that a lot of data/work was transfered by RFC call.

Summary:
If you see high database time in your analysis, Part III will show you how to perform an SQL trace to see what the database is doing for so long.

If you have high CPU/processing time, then in Part IV you will be shown how to find where exactly in the ABAP program is causing the most response time.