also known as: "Squeezing performance of a Lucee application using FusionReactor"

CFML: Performance and Stability
Tales from the front line

Photo by Mikita Karasiou on Unsplash

About me, briefly

CTO @ The Pixl8 Group
(Preside Platform)

@dom_watson

@dom_watson

CFML: Performance and Stability

@dom_watson

CFML: Performance and Stability

Reminder #1

Be good to yourselves

@dom_watson

CFML: Performance and Stability

What we'll try to cover

  • Basics of performance debugging (briefly)
  • Tooling
    • Monitoring
    • Load testing
    • Diagnostics
  • Java memory - oh joy
  • Locking - all the joy
  • Concurrency
  • Some CFML code gotchas that might be news

@dom_watson

CFML: Performance and Stability

What we'll won't cover

  • Frontend performance
  • Database performance

@dom_watson

CFML: Performance and Stability

Performance Debugging 101

#1 Use the Lucee/ColdFusion debug logging

@dom_watson

CFML: Performance and Stability

Performance Debugging 101

#2 Server logs

  1. Know where they are (and what)
  2. System out (e.g. /opt/lucee/tomcat/logs/catalina.out)
  3. Lucee/ACF specific logs such as application log, exception log, thread log, mail log, scheduler log, etc.
  4. Your web server logs (IIS/Apache/Nginx)
  5. DB Logs -> Slow query logging, error logging, etc.

@dom_watson

CFML: Performance and Stability

Monitoring tools

#1 Wholistic monitoring: e.g. Zabbix

#!/bin/bash
ZABBIX_REPORT_FILE=/tmp/tcpzabbixreport.txt
SENDER=`which zabbix_sender`
rm -f $ZABBIX_REPORT_FILE

TOTAL=`netstat --tcp --numeric-ports -n | wc -l`
ELASTIC=`netstat --tcp --numeric-ports -n | grep ":9200" | wc -l`
TOMCAT=`netstat --tcp --numeric-ports -n | grep ":8888" | wc -l`
MYSQL=`netstat --tcp --numeric-ports -n | grep ":3306" | wc -l`
HTTP=`netstat --tcp --numeric-ports -n | grep ":80" | wc -l`
HTTPSSL=`netstat --tcp --numeric-ports -n | grep ":443" | wc -l`
ZABBIXAGENT=`netstat --tcp --numeric-ports -n | grep ":10050" | wc -l`
TOTAL_LISTENING=`netstat --tcp --numeric-ports --listen -n | wc -l`
MYSQL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":3306" | wc -l`
ELASTIC_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":9200" | wc -l`
TOMCAT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":8888" | wc -l`
HTTP_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":80" | wc -l`
HTTPSSL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":443" | wc -l`
ZABBIXAGENT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":10050" | wc -l`
ENTROPY=`cat /proc/sys/kernel/random/entropy_avail`

echo "- netstat[tcp.total] $TOTAL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es] $ELASTIC" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat] $TOMCAT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql] $MYSQL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http] $HTTP" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl] $HTTPSSL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix] $ZABBIXAGENT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.total.listening] $TOTAL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es.listening] $ELASTIC_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat.listening] $TOMCAT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql.listening] $MYSQL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http.listening] $HTTP_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl.listening] $HTTPSSL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix.listening] $ZABBIXAGENT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- entropy[bytes.available] $ENTROPY" >> $ZABBIX_REPORT_FILE

$SENDER -vv -c /etc/zabbix/zabbix_agentd.conf -i $ZABBIX_REPORT_FILE

@dom_watson

CFML: Performance and Stability

Monitoring tools

#2 Detailed error logging, e.g. Sentry

@dom_watson

CFML: Performance and Stability

Monitoring tools

#3 Live JMX Monitoring

VisualVM

JConsole

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9010"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.local.only=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=monitoring.hostname.com"

@dom_watson

CFML: Performance and Stability

Monitoring tools

#4 Live JMX Monitoring and then some

@dom_watson

CFML: Performance and Stability

Performance Problems

Common Problems and where to find them

@dom_watson

CFML: Performance and Stability

Performance Problems

#1 Are you waiting on externals?

  • CFHTTP, CFLDAP, etc. - Always put timeouts on them
  • Database queries - find slow queries and deal with them
<cfhttp ... timeout=sensibleTimeout>

@dom_watson

CFML: Performance and Stability

Performance Problems

#2 Memory issues

"Java manages memory all by itself for you. So you don't have to care."

End of slide.

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

Heap

-Xms 102400m -Xmx 102400m

Meta

-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M

Stack

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

Meta

-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

Heap

-Xms 102400m -Xmx 102400m

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

New Gen

Old Gen

Eden

S1

S2

XX:NewRatio=2 -XX:SurvivorRatio=6

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

PermGen Space

UPGRADE JAVA!!

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Investigate...

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Get a heap dump

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/tmp/heapdump.hprof

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

java.lang.OutOfMemory

Heap Space

Analyze your heap dump

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

Application goes slow, eventually "stops"

@dom_watson

CFML: Performance and Stability

Performance Problems

Java Memory

Application goes slow, eventually "stops"

CONCURRENCY!!!

@dom_watson

CFML: Performance and Stability

Performance Problems

Concurrency

ab -c 100 -n 10000000 http://someurl.com/

Apache2 Utils

@dom_watson

CFML: Performance and Stability

Performance Problems

Concurrency

Concurrency != Requests per second

MAXTHREADS - Your friend

<Service name="Catalina">
    <Connector port="8888" maxThreads="50" ... />

@dom_watson

CFML: Performance and Stability

Performance Problems

Concurrency

Concurrency + locking

More locks = less concurrency

Take thread dumps

https://fastthread.io

@dom_watson

CFML: Performance and Stability

Performance Problems

Summarize

Monitor:

  • Logs
  • Errors (Sentry, etc.)
  • OS Perf (Zabbix, etc.)
  • All the java (FusionReactor)

Test

  • Load (Gatling, JMeter, Apache AB)
  • CONCURRENCY CONCURRENCY

Analyze

  • FusionReactor
  • fastthread.io
  • Eclipse MAT

Know

  • Your application limits

Be

  • Good to yourself!

Thanks for listening

https://www.preside.org
https://www.preside.org/signup
https://www.preside.org/slack

https://presidecms.atlassian.net

@dom_watson

@dom_watson

CTO @ Pixl8 Group
Preside lead developer

Made with Slides.com