In this article, I’ll show how you can achieve all the described things with the GitOps (Argo CD) approach and several tools around OpenShift/Kubernetes like Kyverno or Red Hat Service Interconnect (open-source Skupper project). We will also use Advanced Cluster Management for Kubernetes (ACM) to create and handle “ephemeral” OpenShift clusters. If you need an introduction to the GitOps approach in a multicluster OpenShift environment read the following article. It is also to familiarize with the idea behind multicluster communication through the Skupper project. In order to do that you can read the article about multicluster load balancing with Skupper on my blog.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. It contains several YAML manifests that allow us to manage OpenShift clusters in a GitOps way. For that exercise, we will use the manifests under the clusterpool directory. There are two subdirectories there: hub and managed. The manifests inside the hub directory should be applied to the management cluster, while the manifests inside the managed directory to the managed cluster. In our traffic bursts scenario, a single OpenShift acts as a hub and managed cluster, and it creates another managed (ephemeral) cluster.

Prerequisites

In order to start the exercise, we need a running Openshift that acts as a management cluster. It will create and configure the ephemeral cluster on AWS used to handle traffic volume peaks. In the first step, we need to install two operators on the management cluster: “Openshift GitOps” and “Advanced Cluster Management for Kubernetes”.

After that, we have to create the MultiClusterHub object, which runs and configures ACM:

kind: MultiClusterHub
apiVersion: operator.open-cluster-management.io/v1
metadata:
  name: multiclusterhub
  namespace: open-cluster-management
spec: {}

We also need to install Kyverno. Since there is no official operator for it, we have to leverage the Helm chart. Firstly, let’s add the following Helm repository:

$ helm repo add kyverno https://kyverno.github.io/kyverno/

Then, we can install the latest version of Kyverno in the kyverno namespace using the following command:

$ helm install my-kyverno kyverno/kyverno -n kyverno --create-namespace

Using OpenShift Cluster Pool

With ACM we can create a pool of Openshift clusters. That pool contains running or hibernated clusters. While a running cluster is just ready to work, a hibernated cluster needs to be resumed by ACM. We are defining a pool size and the number of running clusters inside that pool. Once we create the ClusterPool object ACM starts to provision new clusters on AWS. In our case, the pool size is 1, but the number of running clusters is 0. The object declaration also contains all things required to create a new cluster like the installation template (the aws-install-config Secret) or AWS account credentials reference (the aws-aws-creds Secret). Each cluster within that pool is automatically assigned to the interconnect ManagedClusterSet. The cluster set approach allows us to group multiple OpenShift clusters.

apiVersion: hive.openshift.io/v1
kind: ClusterPool
metadata:
  name: aws
  namespace: aws
  labels:
    cloud: AWS
    cluster.open-cluster-management.io/clusterset: interconnect
    region: us-east-1
    vendor: OpenShift
spec:
  baseDomain: sandbox449.opentlc.com
  imageSetRef:
    name: img4.12.36-multi-appsub
  installConfigSecretTemplateRef:
    name: aws-install-config
  platform:
    aws:
      credentialsSecretRef:
        name: aws-aws-creds
      region: us-east-1
  pullSecretRef:
    name: aws-pull-secret
  size: 1

So, as a result, there is only one cluster in the pool. ACM keeps that cluster in the hibernated state. It means that all the VMs with master and worker nodes are stopped. In order to resume the hibernated cluster we need to create the ClusterClaim object that refers to the ClusterPool. It is similar to clicking the Claim cluster link visible below. However, we don’t want to create that object directly, but as a reaction to the Kubernetes event.

Before we proceed, let’s just take a look at a list of virtual machines on AWS related to our cluster. As you see they are not running.

Claim Cluster From the Pool on Scaling Event

Now, the question is – what kind of event should result in getting a cluster from the pool? A single app could rely on the scaling event. So once the number of deployment pods exceeds the assumed threshold we will resume a hibernated cluster and run the app there. With Kyverno we can react to such scaling events by creating the ClusterPolicy object. As you see our policy monitors the Deployment/scale resource. The assumed maximum allowed pod for our app on the main cluster is 4. We need to put such a value in the preconditions together with the Deployment name. Once all the conditions are met we may generate a new Kubernetes resource. That resource is the ClusterClaim which refers to the ClusterPool we created in the previous section. It will result in getting a hibernated cluster from the pool and resuming it.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: aws
spec:
  background: true
  generateExisting: true
  rules:
    - generate:
        apiVersion: hive.openshift.io/v1
        data:
          spec:
            clusterPoolName: aws
        kind: ClusterClaim
        name: aws
        namespace: aws
        synchronize: true
      match:
        any:
          - resources:
              kinds:
                - Deployment/scale
      preconditions:
        all:
          - key: '{{request.object.spec.replicas}}'
            operator: Equals
            value: 4
          - key: '{{request.object.metadata.name}}'
            operator: Equals
            value: sample-kotlin-spring
  validationFailureAction: Audit

Kyverno requires additional permission to create the ClusterClaim object. We can easily achieve this by creating a properly annotated ClusterRole:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kyverno:create-claim
  labels:
    app.kubernetes.io/component: background-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
rules:
  - verbs:
      - create
      - patch
      - update
      - delete
    apiGroups:
      - hive.openshift.io
    resources:
      - clusterclaims

Once the cluster is ready we are going to assign it to the interconnect group represented by the ManagedClusterSet object. This group of clusters is managed by our instance of Argo CD from the openshift-gitops namespace. In order to achieve it we need to apply the following objects to the management OpenShift cluster:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: interconnect
  namespace: openshift-gitops
spec:
  clusterSet: interconnect
---
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: interconnect
  namespace: openshift-gitops
spec:
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: vendor
              operator: In
              values:
                - OpenShift
---
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: argo-acm-importer
  namespace: openshift-gitops
spec:
  argoServer:
    argoNamespace: openshift-gitops
    cluster: openshift-gitops
  placementRef:
    apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: interconnect
    namespace: openshift-gitops

After applying the manifest visible above you should see that the openshift-gitops is managing the interconnect cluster group.

Automatically Sync Configuration for a New Cluster with Argo CD

In Argo CD we can define the ApplicationSet with the “Cluster Decision Resource Generator” (1). You can read more details about that type of generator here in the docs. It will create the Argo CD Application per each Openshift cluster in the interconnect group (2). Then, the newly created Argo CD Application will automatically apply manifests responsible for creating our sample Deployment. Of course, those manifests are available in the same repository inside the clusterpool/managed directory (3).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-init
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource: # (1)
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: interconnect # (2)
        requeueAfterSeconds: 180
  template:
    metadata:
      name: 'cluster-init-{{name}}'
    spec:
      ignoreDifferences:
        - group: apps
          kind: Deployment
          jsonPointers:
            - /spec/replicas
      destination:
        server: '{{server}}'
        namespace: interconnect
      project: default
      source:
        path: clusterpool/managed # (3)
        repoURL: 'https://github.com/piomin/openshift-cluster-config.git'
        targetRevision: master
      syncPolicy:
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Here’s the YAML manifest that contains the Deployment object and the Openshift Route definition. Pay attention to the three skupper.io/* annotations. We will let Skupper generate the Kubernetes Service to load balance between all running pods of our app. Finally, it will allow us to load balance between the pods spread across two Openshift clusters.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: sample-kotlin-spring
  annotations:
    skupper.io/address: sample-kotlin-spring
    skupper.io/port: '8080'
    skupper.io/proxy: http
  name: sample-kotlin-spring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
    spec:
      containers:
        - image: 'quay.io/pminkows/sample-kotlin-spring:1.4.39'
          name: sample-kotlin-spring
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 1000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 128Mi
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/component: sample-kotlin-spring
    app.kubernetes.io/instance: sample-spring-kotlin
  name: sample-kotlin-spring
spec:
  port:
    targetPort: port8080
  to:
    kind: Service
    name: sample-kotlin-spring
    weight: 100
  wildcardPolicy: None

Let’s check out how it works. I won’t simulate traffic bursts on OpenShift. However, you can easily imagine that our app is autoscaled with HPA (Horizontal Pod Autoscaler) and therefore is able to react to the traffic volume peak. I will just manually scale up the app to 4 pods:

Now, let’s switch to the All Clusters view. As you see Kyverno sent a cluster claim to the aws ClusterPool. The claim stays in the Pending status until the cluster won’t be resumed. In the meantime, ACM creates a new cluster to fill up the pool.

Once the cluster is ready you will see it in the Clusters view.

ACM automatically adds a cluster from the aws pool to the interconnect group (ManagedClusterSet). Therefore Argo CD is seeing a new cluster and adding it as a managed.

Finally, Argo CD generates the Application for a new cluster to automatically install all required Kubernetes objects.

Using Red Hat Service Interconnect

In order to enable Skupper for our apps we first need to install the Red Hat Service Interconnect operator. We can also do it in the GitOps way. We need to define the Subscription object as shown below (1). The operator has to be installed on both hub and managed clusters. Once we install the operator we need to enable Skupper in the particular namespace. In order to do that we need to define the ConfigMap there with the skupper-site name (2). Those manifests are also applied by the Argo CD Application described in the previous section.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: skupper-operator
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: skupper-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: skupper-site

Here’s the result of synchronization for the managed cluster.

We can switch to the OpenShift Console of the new cluster. The Red Hat Service Interconnect operator is ready.

Finally, we are at the final phase of our exercise. Both our clusters are running. We have already installed our sample app and the Skupper operator on both of them. Now, we need to link the apps running on different clusters into a single Skupper network. In order to do that, we need to let Skupper generate a connection token. Here’s the Secret object responsible for that. It doesn’t contain any data – just the skupper.io/type label with the connection-token-request value. Argo CD has already applied it to the management cluster in the interconnect namespace.

apiVersion: v1
kind: Secret
metadata:
  labels:
    skupper.io/type: connection-token-request
  name: token-req
  namespace: interconnect

As a result, Skupper fills the Secret object with certificates and a private key. It also overrides the value of the skupper.io/type label.

So, now our goal is to copy that Secret to the managed cluster. We won’t do that in the GitOps way directly, since the object was dynamically generated on OpenShift. However, we may use the SelectorSyncSet object provided by ACM. It can copy the secrets between the hub and managed clusters.

apiVersion: hive.openshift.io/v1
kind: SelectorSyncSet
metadata:
  name: skupper-token-sync
spec:
  clusterDeploymentSelector:
    matchLabels:
      cluster.open-cluster-management.io/clusterset: interconnect
  secretMappings:
    - sourceRef:
        name: token-req
        namespace: interconnect
      targetRef:
        name: token-req
        namespace: interconnect

Once the token is copied into the managed cluster, it should connect to the Skupper network existing on the main cluster. We can verify that everything works fine with the skupper CLI command. The following command prints all the pods from the Skupper network. As you see, we have 4 pods on the main (local) cluster and 2 pods on the managed (linked) cluster.

Let’s display the route of our service:

$ oc get route sample-kotlin-spring

Now, we can make a final test. Here’s the siege request for my route and cluster domain. It will send 10k requests via the Route. After running it, you can verify the logs to see if the traffic comes to all six pods spread across our two clusters.

$ siege -r 1000 -c 10  http://sample-kotlin-spring-interconnect.apps.jaipxwuhcp.eastus.aroapp.io/persons

Final Thoughts

Handling traffic bursts is one of the more interesting scenarios for a hybrid-cloud environment with OpenShift. With the approach described in that article, we can dynamically provision clusters and redirect traffic from on-prem to the cloud. We can do it in a fully automated, GitOps-based way. The features and tools around OpenShift allow us to cut down the cloud costs and speed up cluster startup. Therefore it reduces system downtime in case of any failures or unexpected situations.

The post Handle Traffic Bursts with Ephemeral OpenShift Clusters appeared first on Piotr's TechBlog.

Let’s begin from our sample application. For our test purposes I designed simple system which grants some bonuses basing on incoming orders. First, we are invoking service which put order record into DynamoDB table. Basing on insert event which triggers Lambda function we are processing this event and perform transaction on customer account which id is stored in another DynamoDB table. Afterwards we are sending message to the topic with order information. This topic is created using Amazon SNS service and there are three Lambda functions listening for incoming messages. Each of them grants a bonus that recharges customer account basing on different input data. System architecture is visualized on the figure below. Sample application source code is available on GitHub.

Every AWS Lambda function needs to implement RequestHandler interface. For more details about basic rules, deployment process and usable tools go to my first article about that subject Serverless on AWS Lambda. Coming back to our sample below you can see implementation of first lambda function PostOrder. It does nothing more saving incoming Order object in DynamoDB table. For storing data in DynamoDB we can use ORM mechanism available inside AWS Java libraries. How to use basic DynamoDB annotations you can also read in my first article about serverless.

[code language=”java”]
public class PostOrder implements RequestHandler<Order, Order> {

private DynamoDBMapper mapper;

public PostOrder() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
mapper = new DynamoDBMapper(client);
}

@Override
public Order handleRequest(Order o, Context ctx) {
LambdaLogger logger = ctx.getLogger();
mapper.save(o);
Order r = o;
logger.log("Order saved: " + r.getId());
return r;
}

}
[/code]

Assuming we have our first function implemented and deployed on AWS we should configure API gateway which expose it outside. To achieve it go to Lambda Management Console on AWS, select PostOrder function. Then go to Triggers section and select API Gateway as a trigger for calling your function.

Unfortunately it’s not all we need to have our API gateway redirecting requests to Lambda function. Go to API Gateway section and select OrderService. We should remove ANY method and configure POST invoking our function as you see on the picture below.

Then you should see diagram visible below where all the steps of calling lambda function from API Gateway are visualized.

What’s worth doing is to create a model object in Model section. For Order class it should look like in the picture below, which is with compatible JSON schema notation. After creating model definition set it as a request body inside Method Request panel and response body inside Method Response panel.

Finally, deploy the resource using Deploy API action and try to call it on your which can be checked in Stages section.

Let’s see the second implementation of lambda function – ProcessOrderFunction. It is triggered by insert event received from DynamoDB order table. This function is responsible for reading data from incoming event, then create and send message to the target topic. DynamodbEvent stores data as a map, where the key is column name in order table. To get value from map we have to pass data type, for example string is collected using getS method and integer using getN method. The message send to SNS topic is serialized to JSON string with Jackson library.

[code language=”java”]
public class ProcessOrder implements RequestHandler<DynamodbEvent, String> {

private AmazonSNSClient client;
private ObjectMapper jsonMapper;

public ProcessOrder() {
client = new AmazonSNSClient();
jsonMapper = new ObjectMapper();
}

public String handleRequest(DynamodbEvent event, Context ctx) {
LambdaLogger logger = ctx.getLogger();
final List<DynamodbStreamRecord> records = event.getRecords();

for (DynamodbStreamRecord record : records) {
try {
logger.log(String.format("DynamoEvent: %s, %s", record.getEventName(), record.getDynamodb().getNewImage().values().toString()));
Map<String, AttributeValue> m = record.getDynamodb().getNewImage();
Order order = new Order(m.get("id").getS(), m.get("accountId").getS(), Integer.parseInt(m.get("amount").getN()));
String msg = jsonMapper.writeValueAsString(order);
logger.log(String.format("SNS message: %s", msg));
PublishRequest req = new PublishRequest("arn:aws:sns:us-east-1:658226682183:order", jsonMapper.writeValueAsString(new OrderMessage(msg)), "Order");
req.setMessageStructure("json");
PublishResult res = client.publish(req);
logger.log(String.format("SNS message sent: %s", res.getMessageId()));
} catch (JsonProcessingException e) {
logger.log(e.getMessage());
}
}

return "OK";
}
}
[/code]

Same as for PostOrder function we also should add trigger for ProcessOrder – but this time the trigger is DynamoDB table.

In the Simple Notification Service section create order topic. Amazon SNS client uses ARN address for identifying the right topic. As you can see on the picture below there is also topic for DynamoDB which was created with database trigger.

The last implementation step in the sample is to create lambda functions which are listening on SNS topic for incoming order messages. Here’s OrderAmountHandler function code. The logic is simple. After message receive it needs to perform deserialization from JSON, then check order amount and modify balance value in account table using accountId from order object.

[code language=”java”]
public class OrderAmountHandler implements RequestHandler<SNSEvent, Object> {

private final static int AMOUNT_THRESHOLD = 1500;
private final static int AMOUNT_BONUS_PERCENTAGE = 10;

private DynamoDBMapper mapper;
private ObjectMapper jsonMapper;

public OrderAmountHandler() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
mapper = new DynamoDBMapper(client);
jsonMapper = new ObjectMapper();
}

@Override
public Object handleRequest(SNSEvent event, Context context) {
final LambdaLogger logger = context.getLogger();
final List<SNSRecord> records = event.getRecords();

for (SNSRecord record : records) {
logger.log(String.format("SNSEvent: %s, %s", record.getSNS().getMessageId(), record.getSNS().getMessage()));
try {
Order o = jsonMapper.readValue(record.getSNS().getMessage(), Order.class);
if (o.getAmount() >= AMOUNT_THRESHOLD) {
logger.log(String.format("Order allowed: id=%s, amount=%d", o.getId(), o.getAmount()));
Account a = mapper.load(Account.class, o.getId());
a.setBalance(a.getBalance() + o.getAmount() * AMOUNT_BONUS_PERCENTAGE);
mapper.save(a);
logger.log(String.format("Account balande update: id=%s, amount=%d", a.getId(), a.getBalance()));
}
} catch (IOException e) {
logger.log(e.getMessage());
}
}

return "OK";
}

}
[/code]

After creating and deploying all our three functions we have to subscribe them into the order topic.

All logs from your lambda functions can be inspected with CloudWatch service.

Don’t forget about permissions in My Security Credentials section. For my example I had to attach the following policies to my default execution role: AmazonAPIGatewayInvokeFullAccess, AmazonDynamoDBFullAccess, AWSLambdaDynamoDBExecutionRole and AmazonSNSFullAccess.

The post Serverless on AWS with DynamoDB, SNS and CloudWatch appeared first on Piotr's TechBlog.

Serverless is now one of the hottest trend in IT world. A more accurate name for it is Function as a Service (FaaS). Have any of you ever tried to share your APIs deployed in the cloud? Before serverless, I had to create a virtual machine with Linux on the cloud provider’s infrastructure, and then deploy and run that application implemented in, for example, nodejs or Java. With serverless, you do not have to write any commands in Linux.

What serverless is different from another very popular topic – Microservices? To illustrate the difference serverless is often referred to as nano services. For example, if we would like to create a microservice that provides API for CRUD operations on a database table, then our APIs had several endpoints for searching (GET/{id}), updating (PUT), removing (DELETE), inserting (POST) and maybe a few more for searching using different input criteria. According to serverless architecture, all of those endpoints would be independent functions created and deployed separately. While microservice can be built on an on-premise architecture, for example with Spring Boot, serverless is closely related to the cloud infrastructure.

Custom function implementation based on the cloud provider’s tools is really quick and easy. I’ll try to show it on sample functions deployed on AWS Amazon using AWS Lambda. Sample application source code for AWS serverless is available on GitHub.

How AWS serverless works

Here’s the serverless AWS Lambda solution description from the Amazon site.

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service – all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

AWS Lambda is a computing platform for many application scenarios. It supports applications written in Node.js, Java, C#, and Python. On the platform there are also some services available like DynamoDB – NoSQL database, Kinesis – streaming service, CloudWatch – provides monitoring and logs, Redshift – data warehouse solution, S3 – cloud storage and API Gateway. Every event coming to those services can trigger the calling of your Lambda function. You can also interact with those services using AWS Lambda SDK.

AWS serverless preparation

Let’s finish with the theory, all of us the most like concretes First of all, we need to set up AWS account. AWS has web management console available here, but there is also command line client called AWS CLI, which can be downloaded here. There are also some other tools through which we can share our functions on AWS. I will tell you about them later. To be able to use them, including the command-line client, we need to generate an access key. Go to web console and select My Security Credentials on your profile, then select Continue to Security Credentials and expand Access Keys. Create your new access key and save it on the disc. There are to fields Access Key ID and Secret Access Key. If you would like to use AWS CLI first type aws configure and then you should provide those keys, default region, and format (for example JSON or text).

You can use AWS CLI or even a web console to deploy your Lambda Function on the cloud. However, I will present you with others (in my opinion better :)) solutions. If you are using Eclipse for your development the best option is to download the AWS Toolkit plugin. Now, I’m able to upload my function to AWS Lambda or even create or modify a table on Amazon DynamoDB. After downloading Eclipse plugin you need to provide Access Key ID and Secret Access Key. You have AWS Management perspective available, where you can see all AWS staff including lambda function, DynamoDB tables, identity management, or other services like S3, SNS, or SQS. You can create a special AWS Java Project or work with a standard maven project. Just display project menu by clicking right button on the project and then select Amazon Web Services and Upload function to AWS Lambda…

After selecting Upload function to AWS Lambda… you should window visible below. You can choose the region for your deployment (us-east-1 by default), IAM role, and what is most important – the name of your lambda function. We can create a new function or update the existing one.

Another interesting possibility for uploading function into AWS Lambda serverless is a Maven plugin. With lambda-maven-plugin we can define security credentials and all definitions of our functions in JSON format. Here’s plugin declaration in pom.xml. The plugin can be invoked during maven project build mvn clean install lambda:deploy-lambda. Dependencies should be attached to the output JAR file – that’s why maven-shade-plugin is used during the build.

<plugin>
<groupId>com.github.seanroy</groupId>
<artifactId>lambda-maven-plugin</artifactId>
<version>2.2.1</version>
<configuration>
<accessKey>${aws.accessKey}</accessKey>
<secretKey>${aws.secretKey}</secretKey>
<functionCode>${project.build.directory}/${project.build.finalName}.jar</functionCode>
<version>${project.version}</version>
<lambdaRoleArn>arn:aws:iam::436521214155:role/lambda_basic_execution</lambdaRoleArn>
<s3Bucket>lambda-function-bucket-us-east-1-1498055423860</s3Bucket>
<publish>true</publish>
<forceUpdate>true</forceUpdate>
<lambdaFunctionsJSON>
[
{
"functionName": "PostAccountFunction",
"description": "POST account",
"handler": "pl.piomin.services.aws.account.add.PostAccount",
"timeout": 30,
"memorySize": 256,
"keepAlive": 10
},
{
"functionName": "GetAccountFunction",
"description": "GET account",
"handler": "pl.piomin.services.aws.account.find.GetAccount",
"timeout": 30,
"memorySize": 256,
"keepAlive": 30
},
{
"functionName": "GetAccountsByCustomerIdFunction",
"description": "GET accountsCustomerId",
"handler": "pl.piomin.services.aws.account.find.GetAccountsByCustomerId",
"timeout": 30,
"memorySize": 256,
"keepAlive": 30
}
]
</lambdaFunctionsJSON>
</configuration>
</plugin>

AWS Lambda functions implementation

I implemented sample AWS Lambda functions in Java. Here’s list of dependencies inside pom.xml.

<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-log4j</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.152</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-lambda</artifactId>
<version>1.11.152</version>
</dependency>
</dependencies>

Every function is connected to Amazon DynamoDB. There are two tables created for that sample: account and customer. One customer could have more than one account and this assignment is realized through the customerId field in the account table. AWS library for DynamoDB has ORM mapping mechanisms. Here’s Account entity definition. By using annotations we can declare table name, hash key, index and table attributes.

@DynamoDBTable(tableName = "account")
public class Account implements Serializable {

private static final long serialVersionUID = 8331074361667921244L;
private String id;
private String number;
private String customerId;

public Account() {

}

public Account(String id, String number, String customerId) {
this.id = id;
this.number = number;
this.customerId = customerId;
}

@DynamoDBHashKey(attributeName = "id")
@DynamoDBAutoGeneratedKey
public String getId() {
return id;
}

public void setId(String id) {
this.id = id;
}

@DynamoDBAttribute(attributeName = "number")
public String getNumber() {
return number;
}

public void setNumber(String number) {
this.number = number;
}

@DynamoDBIndexHashKey(attributeName = "customerId", globalSecondaryIndexName = "Customer-Index")
public String getCustomerId() {
return customerId;
}

public void setCustomerId(String customerId) {
this.customerId = customerId;
}

}

In the described sample application there are five lambda functions:
PostAccountFunction – it receives Account object from request and insert it into the table
GetAccountFunction – find account by hash key id attribute
GetAccountsByCustomerId – find list of accounts by input customerId
PostCustomerFunction – it receives Customer object from request and insert it into the table
GetCustomerFunction – find customer by hash key id attribute

Every AWS Lambda function handler needs to implement RequestHandler interface with one method handleRequest. Here’s a PostAccount handler class. It connects to DynamoDB using Amazon client and creates an ORM mapper DynamoDBMapper, which saves input entity in database.

public class PostAccount implements RequestHandler<Account, Account> {

private DynamoDBMapper mapper;

public PostAccount() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
client.setRegion(Region.getRegion(Regions.US_EAST_1));
mapper = new DynamoDBMapper(client);
}

@Override
public Account handleRequest(Account a, Context ctx) {
LambdaLogger logger = ctx.getLogger();
mapper.save(a);
Account r = a;
logger.log("Account: " + r.getId());
return r;
}

}

GetCustomer function not only interacts with DynamoDB, but also invokes GetAccountsByCustomerId function. Maybe this may not be the best example of the need to call another function, because it could directly retrieve data from the account table directly. But I wanted to separate the data layer from the function logic and jut show how invoking of another function works in AWS Lambda cloud.

public class GetCustomer implements RequestHandler<Customer, Customer> {

private DynamoDBMapper mapper;
private AccountService accountService;

public GetCustomer() {
AmazonDynamoDBClient client = new AmazonDynamoDBClient();
client.setRegion(Region.getRegion(Regions.US_EAST_1));
mapper = new DynamoDBMapper(client);

accountService = LambdaInvokerFactory.builder() .lambdaClient(AWSLambdaClientBuilder.defaultClient())
.build(AccountService.class);
}

@Override
public Customer handleRequest(Customer customer, Context ctx) {
LambdaLogger logger = ctx.getLogger();
logger.log("Account: " + customer.getId());
customer = mapper.load(Customer.class, customer.getId());
List<Account> aa = accountService.getAccountsByCustomerId(new Account(customer.getId()));
customer.setAccounts(aa);
return customer;
}
}

AccountService is an interface. It uses @LambdaFunction annotation to declare the name of invoked function in the cloud.

public interface AccountService {
@LambdaFunction(functionName = "GetAccountsByCustomerIdFunction")
List<Account> getAccountsByCustomerId(Account account);
}

API Configuration

I assume that you have already uploaded your Lambda functions. Now, you can go to AWS Web Console and see the full list of them in the AWS Lambda section. Every function can be tested by selecting an item in the functions list and calling Test function action.

If you didn’t configure role permissions you probably got an error while trying to call your lambda function. I attached AmazonDynamoDBFullAccess policy to the main lambda_basic_execution role for Amazon DynamoDB connection. Then I created a new inline policy to enable invoking GetAccountsByCustomerIdFunction from another lambda function as you can see in the figure below. If you retry your tests now everything works fine.

Well, now we are able to test our functions from AWS Lambda Web Test Console. But our main goal is to invoke them from outside clients, for example a REST client. Fortunately, there is a component called API Gateway which can be configured to proxy our HTTP requests from gateway to Lambda functions. Here’s figure with our API configuration, for example POST /customer is mapped to PostCustomerFunction, GET /customer/{id} is mapped to GetCustomerFunction etc.

You can configure model definitions and set them as input or output types for API.

{
"title": "Account",
"type": "object",
"properties": {
"id": {
"type": "string"
},
"number": {
"type": "string"
},
"customerId": {
"type": "string"
}
}
}

For GET request configuration is a little more complicated. We have to set mapping from path parameter into JSON object which is an input in Lambda functions. Select Integration Request element and then go to Body Mapping Templates section.

Our API can also be exported as a Swagger JSON definition. If you are not familiar with take a look at my previous article Microservices API Documentation with Swagger2.

Final words on AWS serverless

In my article, I described the next steps illustrating how to create an API based on the AWS Lambda serverless solution. I showed the obvious advantages of this solution, such as no need for self-management of servers, the ability to easily deploy applications in the cloud, configuration, and monitoring fully based on the solutions provided by the AWS Web Console. You can easily extend my sample with some other services, for example with Kinesis to enable data stream processing. In my opinion, AWS serverless is the perfect solution for exposing simple APIs in the cloud.

The post Serverless on AWS Lambda appeared first on Piotr's TechBlog.