also known as: "Squeezing performance of a Lucee application using FusionReactor"

CFML: Performance and Stability
Tales from the front line

About me, briefly

CFML: Performance and Stability

CFML: Performance and Stability

Reminder #1

Be good to yourselves

CFML: Performance and Stability

What we'll try to cover

  • Basics of performance debugging (briefly)
  • Tooling
    • Monitoring
    • Load testing
    • Diagnostics
  • Java memory - oh joy
  • Locking - all the joy
  • Concurrency
  • Some CFML code gotchas that might be news

CFML: Performance and Stability

What we'll won't cover

  • Frontend performance
  • Database performance

CFML: Performance and Stability

Performance Debugging 101

#1 Use the Lucee/ColdFusion debug logging

CFML: Performance and Stability

Performance Debugging 101

#2 Server logs

  1. Know where they are (and what)
  2. System out (e.g. /opt/lucee/tomcat/logs/catalina.out)
  3. Lucee/ACF specific logs such as application log, exception log, thread log, mail log, scheduler log, etc.
  4. Your web server logs (IIS/Apache/Nginx)
  5. DB Logs -> Slow query logging, error logging, etc.

CFML: Performance and Stability

Monitoring tools

#1 Wholistic monitoring: e.g. Zabbix

#!/bin/bash
ZABBIX_REPORT_FILE=/tmp/tcpzabbixreport.txt
SENDER=`which zabbix_sender`
rm -f $ZABBIX_REPORT_FILE

TOTAL=`netstat --tcp --numeric-ports -n | wc -l`
ELASTIC=`netstat --tcp --numeric-ports -n | grep ":9200" | wc -l`
TOMCAT=`netstat --tcp --numeric-ports -n | grep ":8888" | wc -l`
MYSQL=`netstat --tcp --numeric-ports -n | grep ":3306" | wc -l`
HTTP=`netstat --tcp --numeric-ports -n | grep ":80" | wc -l`
HTTPSSL=`netstat --tcp --numeric-ports -n | grep ":443" | wc -l`
ZABBIXAGENT=`netstat --tcp --numeric-ports -n | grep ":10050" | wc -l`
TOTAL_LISTENING=`netstat --tcp --numeric-ports --listen -n | wc -l`
MYSQL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":3306" | wc -l`
ELASTIC_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":9200" | wc -l`
TOMCAT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":8888" | wc -l`
HTTP_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":80" | wc -l`
HTTPSSL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":443" | wc -l`
ZABBIXAGENT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":10050" | wc -l`
ENTROPY=`cat /proc/sys/kernel/random/entropy_avail`

echo "- netstat[tcp.total] $TOTAL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es] $ELASTIC" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat] $TOMCAT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql] $MYSQL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http] $HTTP" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl] $HTTPSSL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix] $ZABBIXAGENT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.total.listening] $TOTAL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es.listening] $ELASTIC_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat.listening] $TOMCAT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql.listening] $MYSQL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http.listening] $HTTP_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl.listening] $HTTPSSL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix.listening] $ZABBIXAGENT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- entropy[bytes.available] $ENTROPY" >> $ZABBIX_REPORT_FILE

$SENDER -vv -c /etc/zabbix/zabbix_agentd.conf -i $ZABBIX_REPORT_FILE

CFML: Performance and Stability

Monitoring tools

#2 Detailed error logging, e.g. Sentry

CFML: Performance and Stability

Monitoring tools

#3 Live JMX Monitoring

VisualVM

JConsole

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9010"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.local.only=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=monitoring.hostname.com"

CFML: Performance and Stability

Monitoring tools

#4 Live JMX Monitoring and then some

CFML: Performance and Stability

Performance Problems

Common Problems and where to find them

CFML: Performance and Stability

Performance Problems

#1 Are you waiting on externals?

  • CFHTTP, CFLDAP, etc. - Always put timeouts on them
  • Database queries - find slow queries and deal with them
<cfhttp ... timeout=sensibleTimeout>

CFML: Performance and Stability

Performance Problems

#2 Memory issues

"Java manages memory all by itself for you. So you don't have to care."

End of slide.

CFML: Performance and Stability

Performance Problems

Java Memory

Heap

-Xms 102400m -Xmx 102400m

Meta

-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M

Stack

CFML: Performance and Stability

Performance Problems

Java Memory

Meta

-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M

CFML: Performance and Stability

Performance Problems

Java Memory

Heap

-Xms 102400m -Xmx 102400m

CFML: Performance and Stability

Performance Problems

Java Memory

New Gen

Old Gen

Eden

S1

S2

XX:NewRatio=2 -XX:SurvivorRatio=6

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

PermGen Space

UPGRADE JAVA!!

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Investigate...

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Get a heap dump

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/tmp/heapdump.hprof

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Analyze your heap dump

CFML: Performance and Stability

Performance Problems

Java Memory

Application goes slow, eventually "stops"

CFML: Performance and Stability

Performance Problems

Java Memory

Application goes slow, eventually "stops"

CONCURRENCY!!!

CFML: Performance and Stability

Performance Problems

Concurrency

ab -c 100 -n 10000000 http://someurl.com/

Apache2 Utils

CFML: Performance and Stability

Performance Problems

Concurrency

Concurrency != Requests per second

MAXTHREADS - Your friend

<Service name="Catalina">
    <Connector port="8888" maxThreads="50" ... />

CFML: Performance and Stability

Performance Problems

Concurrency

Concurrency + locking

More locks = less concurrency

Take thread dumps

https://fastthread.io

CFML: Performance and Stability

Performance Problems

Summarize

Monitor:

  • Logs
  • Errors (Sentry, etc.)
  • OS Perf (Zabbix, etc.)
  • All the java (FusionReactor)

Test

  • Load (Gatling, JMeter, Apache AB)
  • CONCURRENCY CONCURRENCY

Analyze

  • FusionReactor
  • fastthread.io
  • Eclipse MAT

Know

  • Your application limits

Be

  • Good to yourself!

Thanks for listening

https://www.preside.org
https://www.preside.org/signup
https://www.preside.org/slack

https://presidecms.atlassian.net

@dom_watson

@dom_watson

CTO @ Pixl8 Group
Preside lead developer

Improving performance and stability of CFML applications with FusionReactor (and more)

By Dominic Watson

Improving performance and stability of CFML applications with FusionReactor (and more)

It is early 2019, some of our complex web applications show signs of instability at relatively unpredictable times. Our extensive resource monitoring shows no particular pattern, web traffic logs again prove fruitless. Enter FusionReactor and some determination to get to the bottom of things. Dominic will take you through in detail how he discovered seemingly minor changes in Pixl8’s core platform stack that made some drastic improvements to performance. Come along if you’re interested in: * Micro optimization * Getting to the bottom of hard to debug problems * Improving overall problems in your applications

  • 1,021