Objective

By the end of this blog, you will have a clear understanding and a hands on with below topics:

  1. Anatomy of a Camel Message
  2. Working with XML Payloads
  3. Working with JSON Payloads
  4. Writing UDFs in Groovy

Table of Contents

  1. Objective
  2. Table of Contents
  3. Anatomy of a Camel Message
    1. Cloud Integration Context
    2. Accessing Message and Exchange properties Within Groovy Scripts
  4. Working with XML Payloads
    1. Reader vs String for loading payload
    2. XMLSlurper vs XMLParser for parsing XMLs
      1. Reading the XML documents
      2. Writing the XML Documents
  5. Working with JSON Payloads
  6. Writing UDFs in Groovy
  7. UDFs for all values of a context
  8. Whats next?

Anatomy of a Camel Message

Apache Camel, a widely-used integration framework, facilitates message routing and processing between different systems. The concept of a “Camel Message” is central to its functionality. Here’s a more concise overview of the Camel Message’s anatomy:

  1. Message Body:
    The core part of a Camel Message, containing the actual data being transferred. It can be of various data types, including simple strings, binary content, or complex objects.
  2. Message Headers:
    These are key-value pairs carrying metadata about the message, useful for routing, filtering, and carrying additional information needed during processing.
  3. Attachments (Optional):
    Used mainly in multipart/form data scenarios, like emails with attachments. Not all Camel components support them.
  4. Exchange:
    A Camel Message is part of an Exchange, the larger context representing the entire message exchange process. The Exchange contains In and Out Messages for handling request-response patterns.
  5. Exchange Properties:
    Key-value pairs that store information relevant to the entire Exchange, different from message headers as they remain constant throughout the Exchange’s lifecycle.
  6. Exception Handling:
    In case of processing errors, exceptions are stored in the Exchange, facilitating error handling mechanisms in Camel routes.

Note: Exchange is just a container of the Camel Message which is passed around within a process.

In Camel routes, processors and components interact with these elements of a message, modifying the body, headers, or attachments, and using this information for making routing decisions. Understanding these components is crucial for effectively leveraging Apache Camel’s integration capabilities.

Cloud Integration Context

In the context of Cloud Integration, mostly we will be interacting with 3 parts of the message and its container (exchange). I am assuming you are already familiar with these simple expressions through your work with iFlows.

Component Accessibility Scope of utilization
Message Headers ${header.<headerName>} Complete process
Exchange property ${property.<propertyName>} Only within Integration process i.e. can’t leave your integration process
Message body ${in.body} Complete process

Camel Message and Exchange parameters dealt in this blog series

Accessing Message and Exchange properties Within Groovy Scripts

To be able to fetch and work with different part of camel message and exchange you need to be familiar with below simple concepts.

Within Cloud Integration, the Script API uses a message object derived from the interface com.sap.gateway.ip.core.customdev.util.Message which is a subset of org.apache.camel.Message. So all capabilities and methods of Apache Camel Message cannot be utilized with the Script API Message object.

Below is a boilerplate code to realize how to use different methods to perform different operations on the message. When you create a fresh script similar methods are visible there too.

import com.sap.gateway.ip.core.customdev.util.Message

def Message processData(Message message) {
    //setting a string a body of the message
    message.setBody('Hello World')
    //using reader object to parse a message body by using getBody method
    Reader reader = message.getBody(Reader)
    //fetching exchange property of a message
    def propertyValue = message.getProperty('OrderId')
    //setting exchnage property of a message
    message.setProperty('OrderNumber', 1002233)
    //setting a new header for the message
    message.setHeader("MyHeader", "HeaderValue")
    //fetching the headerValue by its name
    def headerValue = message.getHeader("MyHeader", String.class)
    //return modified message object from previous steps
    return message
}

As we are going to use Groovy to modify payload mostly, we will likely deal with body of the message mostly. Header and exchange methods can be used on specific case basis.

Working with XML Payloads

Reader vs String for loading payload

As a best practice we will use Reader object, not String object to store the message body.

//use this to store message payloads
Reader reader = message.getBody(Reader)
//don't use String objects to store message paylaods
String strPayload = message.getBody().toString()

Reason: A Reader processes data in a stream, handling large data sets more efficiently without loading everything into memory, unlike a String. Also, Reader allows sequential and on-demand data processing, which is useful for parsing or scanning large files.

XMLSlurper vs XMLParser for parsing XMLs

As a best practice we will use XMLSlurper library methods to parse XML payloads, not XMLParser. While there are many advantage of XMLSlurper over XMLParser, the key differentiator is “Lazy Parsing”.

Lazy Parsing - XMLSlurper uses lazy parsing, meaning it doesn’t hold entire XML document, but rather return a GPathResults object which is a ‘means’ to navigate XML documents on demand. When you request a particular part of XML, it only parse that part of XML document. Thus, making it more memory efficient and a performance booster, especially in case of large XML payloads.

Based on above insight, below code should be part of your regular parsing scripts.

Reader reader = message.getBody(Reader)
//use XML slurper to parse XML documents.
def rootNode = new XmlSlurper().parse(reader)

Whever we are dealing with XMLs payloads, we are doing two steps:

  1. Parse the XML: First, you need to parse the XML content. Groovy provides convenient ways to do this, such as using the XmlSlurper class.
  2. Query the XML: Once the XML is parsed, you can use GPath expressions to query and extract data from it.

We will understand each of step in next sections.

Now that we have best practices sorted, lets take a look at the code for XML parsing.

Reading the XML documents

We use XMLSlurper to read the XML Documents. For a sample XML paylaod sent in message body as seen below:

<Books>
    <Book>
        <Title>To Kill a Mockingbird</Title>
        <Author>Harper Lee</Author>
        <Genre>Classic Fiction</Genre>
        <PublicationYear>1960</PublicationYear>
    </Book>
    <Book>
        <Title>1984</Title>
        <Author>George Orwell</Author>
        <Genre>Dystopian Fiction</Genre>
        <PublicationYear>1949</PublicationYear>
    </Book>
    <Book>
        <Title>The Great Gatsby</Title>
        <Author>F. Scott Fitzgerald</Author>
        <Genre>Classic Fiction</Genre>
        <PublicationYear>1925</PublicationYear>
    </Book>
</Books>

We can use below code to fetch the Author of 3rd book as below:

Reader reader = message.getBody(Reader)
//use XML slurper to parse XML documents.
GPathResult rootNode = new XmlSlurper().parse(reader)
//fetching Author of 3rd Book >> F. Scott Fitzgerald
def thirdBookAuthor = rootNode.Book\[2\].Author.text()

Above functionality is helpful if you are trying to fetch value(s) from a large XML and need to do some operations on it to determine which nodes to allow and which ones to ignore.

Writing the XML Documents

We will use Writer and Builder objects to build an XML payload. Writing XML documents in conjuction with Reading part, can help you achieve Java mapping functionality from SAP PI, where you used to perform XML transformations within the Java code to build an XML tree.

For below sample XML.

<Books>
    <Header>
        <BookCategory>Fiction Books</BookCategory>
        <Language>English</Language>
        <Location>Public Library</Location>
    </Header>
    <Book>
        <Title>To Kill a Mockingbird</Title>
        <Author>Harper Lee</Author>
        <Genre>Classic Fiction</Genre>
        <PublicationYear>1960</PublicationYear>
        <Available>Yes</Available>
    </Book>
    <Book>
        <Title>1984</Title>
        <Author>George Orwell</Author>
        <Genre>Dystopian Fiction</Genre>
        <PublicationYear>1949</PublicationYear>
        <Available>No</Available>
    </Book>
    <Book>
        <Title>The Great Gatsby</Title>
        <Author>F. Scott Fitzgerald</Author>
        <Genre>Classic Fiction</Genre>
        <PublicationYear>1925</PublicationYear>
        <Available>Yes</Available>
    </Book>
</Books>

We will build below XML from it.

<AvailableBooks>
    <header>
        <Category>Fiction Books</Category>
        <Language>English</Language>
    </header>
    <Book>
        <Writer>Harper Lee</Writer>
        <BookName>To Kill a Mockingbird</BookName>
    </Book>
    <Book>
        <Writer>F. Scott Fitzgerald</Writer>
        <BookName>The Great Gatsby</BookName>
    </Book>
</AvailableBooks>

Here is the code to Generate above XML document from original payload. Comments within the code explain each step.

import com.sap.gateway.ip.core.customdev.util.Message
import groovy.xml.MarkupBuilder

def Message processData(Message message) {
    Reader reader = message.getBody(Reader)
//use XML slurper to parse XML documents.
    def rootNode = new XmlSlurper().parse(reader)
//instantiating Writer object
    Writer writer = new StringWriter()
//setting up indentation
    def indentPrinter = new IndentPrinter(writer, '    ')
//instantiating builder object
    def builder = new MarkupBuilder(indentPrinter)
//building the XML through GPathResult 'rootNode' navigation
    builder.AvailableBooks {
        'header'{
            'Category' rootNode.Header.BookCategory.text()
            'Language' rootNode.Header.Language.text()
        }
        def availableBooks = rootNode.Book.findAll { it.Available.text() == 'Yes'}
        availableBooks.each { item ->
            'Book' {
                'Writer' item.Author.text()
                'BookName' item.Title.text()
            }
        }
    }
    //Setting message body with writer object
    message.setBody(writer.toString())
    return message
}

I hope you can appreciate how easy is it to navigate the XML with a GpathResult object.

With that we can conclude working with XML part. Now lets move on JSON reading and building, which uses similar concepts as XML.

Working with JSON Payloads

Similar to how we have XMLSlurper for parsing XML documents, we have JSONSlurper for JSON documents.

There is one key difference between the objects which store an XML object i.e. GPathResult and which store a JSON object i.e. Map (LazyMap to be precise). When we use JSONSlurper to parse a JSON documents it stores the JSON payload in a map of key value pairs. e.g. A book object will be parsed as:

Book={Title=To Kill a Mockingbird, Author=Harper Lee, Genre=Classic Fiction, PublicationYear=1960}

This results into one important observation a GPathResult Object store everything under Root Node of XML it is storing. A map object which is storing JSON object, stores complete JSON including super parent node of the parsed JSON.

We will convert below JSON payload:

{
  "Header": {
    "DocumentType": "BookList",
    "CreatedDate": "2024-01-25",
    "Author": "LibrarySystem"
  },
  "Books": [
    {
      "Title": "To Kill a Mockingbird",
      "Author": "Harper Lee",
      "Genre": "Classic Fiction",
      "PublicationYear": 1960
    },
    {
      "Title": "1984",
      "Author": "George Orwell",
      "Genre": "Dystopian Fiction",
      "PublicationYear": 1949
    },
    {
      "Title": "The Great Gatsby",
      "Author": "F. Scott Fitzgerald",
      "Genre": "Classic Fiction",
      "PublicationYear": 1925
    }
  ]
}

Into below JSON, i.e. collecting all the books published before 1950.

{
    "AvailableBooks": {
        "header": {
            "DocType": "BookList",
            "CreatedOn": "2024-01-25"
        },
        "Book": [
            {
                "BookName": "1984",
                "Writer": "George Orwell",
                "PublishedYear": 1949
            },
            {
                "BookName": "The Great Gatsby",
                "Writer": "F. Scott Fitzgerald",
                "PublishedYear": 1925
            }
        ]
    }
}

Below code does this transformation, the comments explain each step below, which is somewhat similar to how you build XML.

import com.sap.gateway.ip.core.customdev.util.Message
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper

def Message processData(Message message) {
Reader reader = message.getBody(Reader)
//using JSONSlurper to create map of JSON
def rootNode = new JsonSlurper().parseText(json)
//Initializing Writer object
Writer writer = new StringWriter()
def indentWriter = new IndentPrinter(writer, '    ')
//Initializing Builder object
def builder = new JsonBuilder(indentWriter)
builder.AvailableBooks {
    'header' {
        'DocType' rootNode.Header.DocumentType
        'CreatedOn' rootNode.Header.CreatedDate
    }
    //filtering out books published before 1950
    def items = rootNode.Books.findAll { book -> book.PublicationYear < 1950}
    //using closure to collect all valid Book items
    'Book' items.collect { item ->
        \[
            'BookName': item.Title,
            'Writer': item.Author,
            'PublishedYear': item.PublicationYear
        \]
    }
}
//setting a pretty printed JSON to message body
message.setBody(builder.toPrettyString())
return message
}

You can see above that, how a JSON document was read and parsed and How we built a JSON document using Groovy functions and closures, to select items of interest out of whole JSON payload.

Lets move on to writing UDFs now.

Writing UDFs in Groovy

UDFs (User Defined Fucntions) are written to execute a functionality which can’t be achieved with the standard/built-in functions provided in the message mapping in Cloud Integration. If you have done this in SAP PI ESR, the translation from Java to Groovy is somewhat similar as we deal with the similar concepts. If not, sweat not, you will learn that in this section now.

First, quickly lets go over how to use UDFs in message mapping, for this example we will use a simple email address validation function, which will also address the first category of UDFs for us i.e. single value arguments.

Give the function a name

Replace the pre-existing code with below code, which is a basic Groovy function taking in emailAddress as a single value argument and return a String object back.

import com.sap.it.api.mapping.*;

def String customFunc(String emailAddress){
    def pattern = /^\[a-zA-Z0-9.\_%+-\]+@\[a-zA-Z0-9.-\]+\\.\[a-zA-Z\]{2,}$/
    //return email address only if it matches the pattern
	return (emailAddress ==~ pattern ? emailAddress : "Invalid Email Address") 
}

This function checks the email address against a regex pattern. It returns email address if it passes the validation, else it returns the string “Invalid Email Address”.

Now we need to utilize this function in the mapping:

Simulate the mapping. First, lets give it a positive test case with valid email address.

Now lets try with negative test case:

UDFs for all values of a context

Earlier, we used a single “Customer” item in our simulation, in turn only one value for “Email”. However, what if you have more than one customer and you need to run the validation for each customer. Not only that, what if you need to pull the information from other part of your XML to generate a field value at the target XML. It will not be happening with only single value UDFs. So, we will now write the UDFs which will help us do the same.

Lets see what we will need to accomplish this:

  1. We will need a function which takes one or more arguments. In our case, we are validating only “Email”. This argument will be an array of email Strings, one email per customer.
  2. We will need to a validation loop to run through each email field value.
  3. We will need an Output object which will hold the value inside our MappingContext. This output is the output of the function which we will map to the field of our choice which needs hold this value. In this case, target field “EmailAddress”

Based on these three requirements, lets go over below code:

import com.sap.it.api.mapping.\*;

def void validateEmailAddresses(String\[\] emailAddresses, Output output, MappingContext context) {
        //loop over the emailAddress values of all context
        emailAddresses.each { emailAddress ->
            def emailRegex = /^\[a-zA-Z0-9.\_%+-\]+@\[a-zA-Z0-9.-\]+\\.\[a-zA-Z\]{2,}$/
            def isValid = emailAddress ==~ emailRegex
            //return email address if valid else the string 'Invalid Email Address'
            output.addValue(isValid ? emailAddress : 'Invalid Email Address')
        }
}

In this code we are achieving desired functionality in two step:

  1. Loops through each email address:
    • Defines a regular expression to match valid email patterns.
    • Checks if the current email address matches the pattern.
  2. Adds results to output:
    • If valid, adds the email address to the output.
    • If invalid, adds “Invalid Email Address” to the output.

When we use above code in our script and use multiple customer XML paylaod to validate, this is how it looks:

Note output.addValue method is similar to how we have ResultList.addValue() in SAP PI ESR mapping for multiple values of a context.

Whats next?

With this blog we have covered a lot of ground now. Hopefully with a recurring practice you will feel comfortable with these topics and functions. This should also help you expand on topics branching out from these basics.

In next blog we will cover the most important thing now that we are done with Groovy Development, its “The testing”. Not your plain testing where you bounce between tools with copy paste, but in organized manner. I will also introduce you to spock framework for testing groovy scripts. Which will also make you appreciate the TDD (Test Driven Development), to refine our scripts to make them fail safe.

See you in next one!