SAP Process Integration Starter Guide for Basis Administrators

This is a starter guide for Basis Administrators to understand the architecture and various nuances of SAP PI Administration. This guide will be updated time to time, so I recommend you to bookmark and keep revisiting this page as you see fit.

SAP Process Integration Starter Guide for Basis Administrators

Enterprises run different software solutions to support their business processes. Not all of them are built and supported by SAP. These solutions are integrated one-to-one using various middleware solutions or customized programs or scripts.

When there is a change in one system or business process, multiple point-to-point interfaces are affected and are often expensive to change.

Although these point-to-point interfaces are using different technologies, there are recurring themes. Although there are recurring themes, code is not reusable because they were implemented using different technologies.

In 2003, SAP introduced NetWeaver, which helped them focus on providing a platform for integrating various aspects of business, including business processes (by using Exchange Infrastructure or XI). Later in 2005, XI was renamed to Process Integration (SAP PI).

SAP NetWeaver as a platform for enterprise integration

SAP Process Integration Architecture

Let us start with understanding various components of SAP PI.

SAP Process Integration Architecture

Enterprise Service Repository contains system landscape independent design objects. These objects determine how the data fetched from a source system (aka Sender System) is changed before being presented to the target system (aka Receiver System). Whether the integration is in development, QA or production system these objects do not change.

Integration Directory contains system landscape dependent configuration objects. These objects determine the actual systems that will be exchanging data with PI, and configure the details of the data exchange (security, transport protocol, etc.). These objects need to be adjusted when transported across the landscape.

Enterprise Service Builder and Integration Builder contain editors and wizards to help create, modify and delete design objects and configuration objects respectively.

The above discussed four components ESR, ID, ESB and IB are part of design-time environment of PI. These components are used for developing interfaces to meet the integration needs.

Certain type of interfaces (HTTP, SOAP, XI and WS adapter based) can be published as services (Service Interfaces). These are listed in UDDI compliant registry--Service Registry (not shown in this diagram)

Integration Server is made up of three main runtime components: Integration Engine, Adapter Engine and Business Process Engine.

Integration Engine processes the messages using PI pipelines; which are a set of services performed in a fixed sequence on a message. The most important aspects of the pipeline steps are determining the receiver and adapting the message to the structure (not native data format, just the structure in XML) required by the receiver. Integration Engine understands data in SOAP-XML format (and IDocs as it is proprietary to SAP). It does not understand third-party data formats.

Business Process Engine correlates different messages, verifies logical dependencies and processes messages based on conditions. It allows PI to send and receive multiple messages relating to a single document. For instance, line items for a purchase order may be gathered from several sources. Correlations let you relate the incoming documents using a unique key, for instance a PO number, so that incoming documents are processed only with the master document to which they belong.

The primary function of Adapter Engine (or Advanced Adapter Engine) is to translate third party data formats to format understood by Integration Engine and vice-versa. This is achieved by various Resource Adapters. The adapters are delivered by SAP and extended by SAP and/or its partners. The adapters are “plugged in” to the Adapter Framework, which handles message processing tasks. There is an included module processor that lets you extend the basic functionality of adapters by calling additional modules for processing messages in the adapter.

System Landscape Directory contains the landscape (Technical and Business Systems) and software catalog (product and software versions). This information is used both at design time (by ESR and ID) and runtime (by IS).

An optional additional adapter engine (called as Non-central Adapter Engine) which relies on Integration Engine and Business Process Engine for pipeline processing/cross-component process management, and Central Adapter Engine for design time, can be installed depending on the requirements. The non-central adapter engine has a separate SAP SID and may be used for security or geographic proximity reasons.

The runtime environment caches the active version of integration objects from ESR/ID, and SLD data. Repeated fetching of information from ESR, ID and SLD is avoided by keeping this information in runtime components' proximity.

When data is changed in ESR/ID it is updated in runtime cache by a mechanism called as Cache Update. A manual cache update can be formed by an administrator, by pulling information from design time using Cache Refresh.

The design time caches SLD data in its proximity and there by avoids multiple look ups to information in SLD.

Sender, Receiver, Outbound and Inbound

The flow of XI messages (the exchanged data) is:

Sender System to SAP PI to Receiver System.

Sender, Receiver, Outbound and Inbound Interfaces

SAP PI connects with the Sender System using Sender Communication Channel to receive PI messages. PI processes the messages and sends to Receiver System using Receiver Communication Channel. The sender and receiver systems are also referred to as Business Systems.

The interface between Sender System and PI is called Outbound Interface and the interface between PI and Receiver System is called Inbound Interface.

This is bit confusing, but the guideline is to think from the perspective of the Business Systems instead of PI.

Integration Engine

The Integration Engine processes the incoming SOAP-XML message through a series of pipeline services.


In the first step, the XML is validated for its syntax and completeness. The next two steps constitute logical routing where all the receivers of the message and interfaces (operations) are determined.

The receiver and interface determination is done in order to get the structure of message required by the receiver system. The means to connect to receiver system are not required at this point and therefore not determined.

If the message has n receivers, the message is split (actually copied) into n messages so that each interface can then operate on individual copy.

The XML structure is transformed into the receiver structure. For example sender system may have maintained First Name and Last Name of a person separately, but the receiver system maintains Name as an individual field. Message Mapping combines First Name and Last Name to Name as required by receiver system. The Integration Engine requests for mapping; it does not run the mapping program. The mapping program is run in the Adapter Engine.

If the sender structure and receiver structures match, there is no need of message mapping.

The pipeline then determines the technical (routing) method of contacting the receiver system. At this stage the communication channel details for connecting with the receiver are determined.

The message is finally sent to the receiver using Integration Engine adapters (IDoc, HTTP or WS Adapters) or adapters of Adapter Engine. In some cases, depending on integration process, the message is sent to Business Processing Engine. This final step in Integration Engine (of sending message to receiver or other engines) is called Call Adapter.

The pipeline steps are executed using RFC calls. For asynchronous processing the pipeline steps are executed by using aRFC queues of NetWeaver; monitored using SMQ2 and SMQ1 transactions.

An illustration of the PI message flow using pipeline steps is shown below.

PI message flow using pipeline steps

The pipeline services are activated, as part of post-installation steps, by activating the engine service in SICF.

Activation of Pipeline Services in SICF

Queues

You can check the queues used by Integration Engine to asynchronously process the pipeline steps from SXMB_ADMIN > Administration > Manage Queues.

Integration Engine Queues

If these queues are not registered, asynchronous processing  of PI messages will fail.

Quality of Service

SAP PI uses three quality-of-services (QoS). QoS means if the messages are processed synchronously or asynchronously.

Best Effort (BE) messages do not use queues. The messages are processed synchronously. While the messages are being processed by PI, the sender system waits for a response. Synchronous messages are not saved in PI database for performance reasons, unless there is a message failure. These messages cannot be reprocessed from PI after failure (what’s the point, sender stopped waiting after failure)

Exactly Once (EO) messages use the XBT* queues. The sender system does not keep its resources waiting to receive a response. The response (if requested) is sent asynchronously to sender after the processing completes or fails. These messages are stored in PI database and if there is a failure, the messages may be reprocessed from PI.

Exactly Once In Order (EOIO) messages are processed asynchronously in first-in-first-out basis. For each interface, a single queue (XBQ*) is used to ensure sequence. A single failure holds up the next messages in the sequence, paralyzing the interface. These messages are stored in PI database and if there is a failure, the messages may be reprocessed from PI.

Majority of the interfaces are built to use EO as the QoS.

IDoc Adapter

The IDoc Adapter is a part of the Integration Engine. It converts IDocs sent via tRFC to IDoc-XML and vice-versa.

IDoc Adapter (Sender direction) in Integration Engine

The conversion is done by using the IDoc metadata of the sender/receiver systems. The IDoc metadata (definition of various segments in the IDoc of certain type) is copied from the Sender and/or Receiver system and stored in the PI system. This metadata can be checked using the transaction IDX2.

IDoc metadata of Business Systems in PI

The RFC destination (port and client) details for reading the IDoc metadata from Sender or Receiver system can be maintained or checked using the transaction IDX1.

Ports for IDoc Metadata

If the metadata is missing in IDX2, it will be fetched automatically by the IDoc adapter when the first message of such type is processed for the given business system. Once fetched it stays there to avoid repeated fetching.

If the metadata of the IDoc is changed on the Sender/Receiver system, the older version of metadata must be deleted from IDX2 to fetch the latest version. If this is not done, the messages will fail in PI.

You can schedule the report IDX_CHECK_METADATA to run once a day and check if metadata is current.

If none of the interfaces in the Integration Engine use IDoc-XML, you can avoid an unnecessary conversion from and to XML. This means all the IDocs in the PI system will be processed in their native format. The conversion can be controlled using SXMB_ADM > Integration Engine Configuration > Specific Configuration > Category IDOC > parameter > XML_CONVERSION (0 = no conversion).

When IDocs are sent, the following acknowledgements are sent back to the Sender, by default:
  • System error acknowledgment (if there was an error in message processing in PI)
  • Application acknowledgment (if receiver application processed the IDoc)
  • Application error acknowledgment (if receiver application failed to process the IDoc)
IDocs only return acknowledgments if the receiver is configured for using ALE audit (only possible for IDocs of type logical system)

The acknowledgements can be switched off (for example when receiver application does not support acknowledgement) per message type and port/client for all types of acknowledgements using the report IDX_NOALE.

If you have the IDoc number of the Sender/Receiver system and you want to check the corresponding PI message, you can use the transaction IDX5. This is helpful when administrators of the sender system complain about a certain IDoc not in final state (for example) and if the corresponding PI message has issues.

Finding message ID from IDoc number

Business Process Engine

Before a message is sent to Business Process Engine, it is processed by Integration Engine. The BPE correlates various messages from the Integration Engine. When a predetermined number of messages are correlated or when the BPE has waited for a predetermined amount of time, the correlation completes and the processing of these messages is performed through various workflows to execute the underlying business process.


Business Process Engine may execute workflow across multiple systems.

The message (merged by correlation of various messages) is returned to IE. The IE processes this message out to the receiver system.

BPE uses queues XBQO$PE_WS* or XBPE_WS* to process the messages. BPE is inherently asynchronous. It uses sync/async bridge (which can be monitored from SXMB_MONI) to deal with synchronous messages while the process is executed asynchronously.

The Integration Processes designed in ES/Integration Builder generates the workflow in AS ABAP.


The infrastructure in AS ABAP required by BPE is configured using the transaction SWF_XI_CUSTOMIZING as part of post-installation steps.


You can automatically generate the configuration by choosing Perform Automatic Workflow Customizing and start Verification Workflow to trigger a test workflow. One of the most important aspects is "Configure RFC Destination". Workflows will not be executed if the RFC destination is not working (indicated by red traffic light in the transaction). If the workflows have deadline, RSWWDHEX report should run to detect deadlines. You have to Maintain Prefix Numbers manually. All workflows (and workflow components) created will start with the number maintained here.

If you are bringing down SAP PI system, remember to stop the Business Process Engine using the transaction SWF_XI_ADM_BPE. You have to start it after SAP PI system is up.

Starting and Stopping BPE should be part of your maintenance procedure

Adapter Engine

Integration Engine can only understand XML data. The sender and receiver systems may not always have the capability to convert their data into the format that Integration Engine can understand. This is where Adapter Engine comes in.

The Adapter Engine fetches or reads data sent in sender/receiver format and converts that data into SOAP-XML. The conversion is done using the Java Connector Architecture compliant adapters. These adapters are plugged in to the Adapter Framework.

The communication of the XML between Adapters and Integration Engine is handled by Messaging Service and Module Processor. Messaging Service uses four different queues (Send, Recv, Call and Rqst) for each adapter type to handle the messages.

Components and message flow in Adapter Framework

When a message is sent by Integration Engine, messaging service receives it and based on the receiver details, a relevant chain of modules in the module processor are used for further processing of the message.

There are two default (generic) module chains: one for the sender direction and one for the receiver direction. If message processing is to be executed entirely within the adapter, you can limit yourself to using the default module chains for your adapter. You can see these modules from Communication Channel configuration.

The module processor controls the steps in the module chain by calling generic and adapter-specific modules (if they are defined in the adapter to extend its functionality).

The last module forwards the message to the adapter. The adapter transfers the message to the receiver system.

Message processing in the sender direction proceeds in a similar way, but in the opposite direction.

Adapters

SAP delivers a set of adapters by default:

Default SAP Adapters in PI

The receiver communication channels always push data to the receiver systems (as opposed to PI waiting for receiver system to actively come and pick the data).

The sender communication channels are of two behavioral types – those that pick data from the sender (polling channels ex: JDBC, Mail and File Sender channels) and those that listen to data sent from the sender system (listening channels ex: RFC and SOAP Sender channels). The time interval for polling is defined in the communication channel configuration and the polling is scheduled by AFW scheduler.

If the communication channels are not polling for any apparent reason, you can check the scheduler http://<host>:<j2ee-http-port>/AdapterFramework/scheduler/scheduler.jsp or see if there are any old locks on the communication channel (from Locking Adapter service in NetWeaver AS). Familiarize yourself with the locks in J2EE engine. Some of the locks are set at J2EE start up and remain until it is stopped.

External drivers

JDBC and JMS adapters require client drivers to connect with the JDBC and JMS sender/receiver systems. These drivers are provided by the external software vendors of the JDBC and JMS server software.

For example: If JDBC Adapter has to connect with an Oracle database, Oracle client software (ojdbc.jar file) has to be deployed in PI. The jar file has to be added to aii_af_jmsproviderlib.sda (PI 7.0x) or com.sap.aii.adapter.lib.sda (PI 7.1x and higher).

The client jar file must be compatible with the SAP JVM version.

SAP provides SDA Maker Tool to prepare sda file carrying the client libraries (jar files) which can be deployed using the standard sda deployment methods (telnet/SDM/SUM/NWDS)

You can check the currently deployed client libraries from Java Class Loader Viewer (NWA > Configuration > Infrastructure)

How to check the external drivers used in PI

com.sap.aii.adapter.lib_api.jar is available by default. Presence of any additional jar file indicates manually added client libraries. In the example shown above, sqljdbc4.jar file is deployed to communicate with Microsoft SQL Server.

System Landscape Directory

System Landscape Directory is an encyclopedia of the enterprise systems. It is an important component for an integration system as it contains enterprise system details, their locations, roles and software components installed. PI system also finds out its role from its business system details in SLD.

SLD manages the physical host details (Technical Infrastructure), Business Systems (Applications), installed software components and the transport paths Dev > QA > Prod (for example). This information is used by both design time and runtime components of SAP PI system.

PI system is registered as a Technical System as shown below.

Make sure PI system is registered to same domain

For several application processes it is required that the PI components all belong to the same ‘domain’. This domain and the associations of this domain are set during start-up of the system. However, when changing the hostname of the Integration Server, the associations may become inconsistent and it is necessary to check or recreate them. Incorrect or missing entries can lead to message or design issues as SLD is used by both environments. SAP Note 764176 guides you to fix these errors.

The SLD may be hosted on SAP PI itself or it can be maintained on a separate NetWeaver AS Java system. The under linked step-by-step guide for SLD migration lists out important parameters for defining SLD connectivity used by PI. You can access SLD used by SAP PI by calling transaction SLDCHECK.

SLDCHECK performs important access checks

Integration Engine to SLD connection is established via RFC destination SAPSLDAPI and JCo RFC Connection in J2EE (hosting SLD) with the program ID maintained in SAPSLDAPI.

The HTTP destination details and user credentials are maintained using the transaction SLDAPICUST.

SLD also maintains transport targets, which is required for correctly importing Integration Directory objects in QA and Production PI systems.

Transport targets are required for importing PI objects properly

In the above example, when a ID TPZ file containing objects from PI DEV are imported into PI QA or PI PRD, the Business System references to BS_DEV are replaced with BS_QA and BS_PRD respectively, based on the transport targets defined in SLD.

Exchange Profile

The Exchange Profile is an XML document that contains internal parameters of the PI system, stored in the database (AII_PROFILES table) of the Integration Server.

Most of these parameters are initialized automatically during the installation phase, but in some cases the administrator may need to maintain them.

Exchange Profile

The Exchange Profile is accessed by http://<host>:<port>/exchangeProfile or http://<host:port>/webdynpro/dispatcher/sap.com/com.sap.xi.exprofui/XIProfileApp

A list of important how-to documents and issue resolutions can be found here.

Subscribe to comments of this post to get an email asI update the post.

Comments

Popular posts from this blog

OS/DB Migration - CMD. STR, TOC, EXT, R3load, DDLDBS.TPL and more

Fixing Inconsistent Table - Table activation fails due to inconsistency between DD and DB

301 Redirect Using SAP Web Dispatcher