MQ Load Testing

A step by step procedure to do performance testing with AMQ using AWS Batch. Reports are summarized and compiled into XML. S3 buckets are used to store properties and report files.

The following is a step by step procedure to do AMQ performance testing on a large scale using ActiveMQ Maven Performance Test Plugin. This is my personal notes on how to run a fully reproducible test. This notes does not include any specific test results.

Docker

Install Docker

sudo yum update
sudo yum install docker 
sudo amazon-linux-extras install docker
sudo usermod -a -G docker ec2-user
sudo service docker start

Prerequisites

mkdir amq-perf
cd amq-perf
sudo yum install -y subversion
wget https://d3pxv6yz143wms.cloudfront.net/8.222.10.1/java-1.8.0-amazon-corretto-devel-1.8.0_
222.b10-1.x86_64.rpm
wget http://mirror.cc.columbia.edu/pub/software/apache/maven/maven-3/3.6.1/binaries/apache-ma
ven-3.6.1-bin.tar.gz
tar xzvf apache-maven-3.6.1-bin.tar.gz
rm apache-maven-3.6.1-bin.tar.gz
svn checkout http://svn.apache.org/repos/asf/activemq/sandbox/activemq-perftest/

Docker run command for producer

startProducer.sh

#!/bin/sh
td=`date +'%Y-%m-%d %H:%M:%S'`
aws s3 cp s3://$REQUEST_BUCKET/openwire-producer.properties /root/activemq-perftest/openwire-produce
r.properties
sed -i "s/JmsProducer/$HOSTNAME/g" /root/activemq-perftest/openwire-producer.properties
mvn activemq-perf:producer -DsysTest.propsConfigFile=/root/activemq-perftest/openwire-producer.prope
rties
for f in ./reports/*
do
  fn=$(basename $f)
  s3obj=${td}/producer/$HOSTNAME/${fn}
  aws s3 cp $f s3://$REPORT_BUCKET/$s3obj
done

Docker run command for consumer

startConsumer.sh

#!/bin/sh
td=`date +'%Y-%m-%d'`
aws s3 cp s3://$REQUEST_BUCKET/openwire-consumer.properties /root/activemq-perftest/openwire-consume
r.properties
sed -i "s/JmsConsumer/$HOSTNAME/g" /root/activemq-perftest/openwire-consumer.properties
mvn activemq-perf:consumer -DsysTest.propsConfigFile=/root/activemq-perftest/openwire-consumer.prope
rties
for f in ./reports/*
do
  fn=$(basename $f)
  s3obj=${td}/consumer/$HOSTNAME/${fn}
  aws s3 cp $f s3://$REPORT_BUCKET/$s3obj
done

Dockerfile

Dockerfile

FROM amazonlinux:latest
ARG AWS_KEY=''
ARG AWS_SECRET_KEY=''
ARG AWS_REGION='us-east-1'
RUN yum -y update && yum install -y python python-dev python-pip python-setuptools groff less \
    && pip install --upgrade awscli 
COPY java-1.8.0-amazon-corretto-devel-1.8.0_222.b10-1.x86_64.rpm /root
RUN rpm -ivh /root/java-1.8.0-amazon-corretto-devel-1.8.0_222.b10-1.x86_64.rpm
RUN aws configure set aws_access_key_id ${AWS_KEY} && aws configure set aws_secret_access_key ${AWS_
SECRET_KEY} && aws configure set region ${AWS_REGION}
ADD apache-maven-3.6.1 /root/apache-maven-3.6.1
ADD activemq-perftest /root/activemq-perftest
COPY startProducer.sh /root/activemq-perftest
COPY startConsumer.sh /root/activemq-perftest
ENV PATH="/root/apache-maven-3.6.1/bin:${PATH}"
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0-amazon-corretto/jre/
RUN cd /root/activemq-perftest && mvn clean install && mkdir reports

Build docker image and upload to ECR

build,.sh

#!/bin/bash
docker build -t amq-perf:latest .
$(aws ecr get-login --no-include-email --region us-east-1)
docker tag amq-perf:latest 1234567890.dkr.ecr.us-east-1.amazonaws.com/amq-perf:latest
docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/amq-perf:latest

Fargate cluster

Install ecs-cli

sudo curl -o /usr/local/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-linux-amd64-latest
echo "$(curl -s https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-linux-amd64-latest.md5) /usr/local/bin/ecs-cli" | md5sum -c -
ecs-cli --version
sudo chmod +x /usr/local/bin/ecs-cli

Task execution role

task-execution-assume-role.json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Create role

aws iam --region us-east-1 create-role --role-name ecsTaskExecutionRole --assume-role-policy-document file://task-execution-assume-role.json

Attach ECS and S3 Policies

aws iam --region us-east-1 attach-role-policy --role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam --region us-east-1 attach-role-policy --role-name ecsTaskExecutionRole --policy-arn arn:aws:iam::aws:policy/service-role/AmazonS3FullAccess

Create cluster config

ecs-cli configure --cluster amq-perf --default-launch-type FARGATE --config-name amq-perf --region us-east-1

Create a esc-cli profile with access key and secret key

ecs-cli configure profile --access-key AWS_ACCESS_KEY_ID --secret-key AWS_SECRET_ACCESS_KEY --profile-name amq-perf-profile

Create a cluster with existing VPC, subnets and security group

ecs-cli up --cluster-config amq-perf --ecs-profile amq-perf-profile --vpc VPC_ID --subnets SUBNET_1, SUBNET_2 --security-group SEC_GRP

Task definitions for producers and consumers

Run task

aws ecs run-task --cluster amq-perf --launch-type FARGATE --task-definition amq-perf-producer:6 --network-configuration 'awsvpcConfiguration={subnets=["subnet-1","subnet-2", "subnet-3"],securityGroups=["sg-1"],assignPublicIp=ENABLED}'

Test Parameters

See 2. in references below for different test parameters that can be specified.

openwire-producer.properties

factory.brokerURL=BROKER_URL
factory.userName=BROKER_USER
factory.password=BROKER_PASSWORD

sysTest.reportDir=reports/
sysTest.samplers=tp
sysTest.spiClass=org.apache.activemq.tool.spi.ActiveMQReflectionSPI
sysTest.clientPrefix=JmsProducer

producer.destName=queue://PERF.TEST
producer.deliveryMode=persistent

openwire-consumer.properties

factory.brokerURL=BROKER_URL
factory.userName=BROKER_USER
factory.password=BROKER_PASSWORD

sysTest.reportDir=reports/
sysTest.samplers=tp
sysTest.spiClass=org.apache.activemq.tool.spi.ActiveMQReflectionSPI
sysTest.destDistro=equal
sysTest.clientPrefix=JmsConsumer

consumer.destName=queue://PERF.TEST
consumer.deliveryMode=persistent

TODO :

1. Collect and add CloudWatch Metrics to the above report

References

1. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html

2. https://activemq.apache.org/activemq-performance-module-users-manual

3. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html

4. https://medium.com/@hengfeng/create-a-scheduled-cron-like-task-in-aws-fargate-e4c025ec474a