also known as: "Squeezing performance of a Lucee application using FusionReactor"
CFML: Performance and Stability
Tales from the front line
Photo by Mikita Karasiou on Unsplash
About me, briefly
CTO @ The Pixl8 Group
(Preside Platform)
CFML: Performance and Stability
CFML: Performance and Stability
Reminder #1
Be good to yourselves
CFML: Performance and Stability
What we'll try to cover
- Basics of performance debugging (briefly)
-
Tooling
- Monitoring
- Load testing
- Diagnostics
- Java memory - oh joy
- Locking - all the joy
- Concurrency
- Some CFML code gotchas that might be news
CFML: Performance and Stability
What we'll won't cover
- Frontend performance
- Database performance
CFML: Performance and Stability
Performance Debugging 101
#1 Use the Lucee/ColdFusion debug logging
CFML: Performance and Stability
Performance Debugging 101
#2 Server logs
- Know where they are (and what)
- System out (e.g. /opt/lucee/tomcat/logs/catalina.out)
- Lucee/ACF specific logs such as application log, exception log, thread log, mail log, scheduler log, etc.
- Your web server logs (IIS/Apache/Nginx)
- DB Logs -> Slow query logging, error logging, etc.
CFML: Performance and Stability
Monitoring tools
#1 Wholistic monitoring: e.g. Zabbix
#!/bin/bash
ZABBIX_REPORT_FILE=/tmp/tcpzabbixreport.txt
SENDER=`which zabbix_sender`
rm -f $ZABBIX_REPORT_FILE
TOTAL=`netstat --tcp --numeric-ports -n | wc -l`
ELASTIC=`netstat --tcp --numeric-ports -n | grep ":9200" | wc -l`
TOMCAT=`netstat --tcp --numeric-ports -n | grep ":8888" | wc -l`
MYSQL=`netstat --tcp --numeric-ports -n | grep ":3306" | wc -l`
HTTP=`netstat --tcp --numeric-ports -n | grep ":80" | wc -l`
HTTPSSL=`netstat --tcp --numeric-ports -n | grep ":443" | wc -l`
ZABBIXAGENT=`netstat --tcp --numeric-ports -n | grep ":10050" | wc -l`
TOTAL_LISTENING=`netstat --tcp --numeric-ports --listen -n | wc -l`
MYSQL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":3306" | wc -l`
ELASTIC_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":9200" | wc -l`
TOMCAT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":8888" | wc -l`
HTTP_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":80" | wc -l`
HTTPSSL_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":443" | wc -l`
ZABBIXAGENT_LISTENING=`netstat --tcp --numeric-ports --listen -n | grep ":10050" | wc -l`
ENTROPY=`cat /proc/sys/kernel/random/entropy_avail`
echo "- netstat[tcp.total] $TOTAL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es] $ELASTIC" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat] $TOMCAT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql] $MYSQL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http] $HTTP" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl] $HTTPSSL" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix] $ZABBIXAGENT" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.total.listening] $TOTAL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.es.listening] $ELASTIC_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.tomcat.listening] $TOMCAT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.mysql.listening] $MYSQL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.http.listening] $HTTP_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.httpssl.listening] $HTTPSSL_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- netstat[tcp.zabbix.listening] $ZABBIXAGENT_LISTENING" >> $ZABBIX_REPORT_FILE
echo "- entropy[bytes.available] $ENTROPY" >> $ZABBIX_REPORT_FILE
$SENDER -vv -c /etc/zabbix/zabbix_agentd.conf -i $ZABBIX_REPORT_FILECFML: Performance and Stability
Monitoring tools
#2 Detailed error logging, e.g. Sentry
CFML: Performance and Stability
Monitoring tools
#3 Live JMX Monitoring
VisualVM
JConsole
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9010"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.local.only=false"
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="$JAVA_OPTS -Djava.rmi.server.hostname=monitoring.hostname.com"CFML: Performance and Stability
Monitoring tools
#4 Live JMX Monitoring and then some
CFML: Performance and Stability
Performance Problems
Common Problems and where to find them
CFML: Performance and Stability
Performance Problems
#1 Are you waiting on externals?
- CFHTTP, CFLDAP, etc. - Always put timeouts on them
- Database queries - find slow queries and deal with them
<cfhttp ... timeout=sensibleTimeout>
CFML: Performance and Stability
Performance Problems
#2 Memory issues
"Java manages memory all by itself for you. So you don't have to care."
End of slide.
CFML: Performance and Stability
Performance Problems
Java Memory
Heap
-Xms 102400m -Xmx 102400m
Meta
-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M
Stack
CFML: Performance and Stability
Performance Problems
Java Memory
Meta
-XX:MaxMetaspaceSize=512M
-XX:MetaspaceSize=256M
CFML: Performance and Stability
Performance Problems
Java Memory
Heap
-Xms 102400m -Xmx 102400m
CFML: Performance and Stability
Performance Problems
Java Memory
New Gen
Old Gen
Eden
S1
S2
XX:NewRatio=2 -XX:SurvivorRatio=6
CFML: Performance and Stability
Performance Problems
Java Memory
java.lang.OutOfMemory
PermGen Space
UPGRADE JAVA!!
CFML: Performance and Stability
Performance Problems
Java Memory
java.lang.OutOfMemory
Heap Space
Investigate...
CFML: Performance and Stability
Performance Problems
Java Memory
java.lang.OutOfMemory
Heap Space
CFML: Performance and Stability
Performance Problems
Java Memory
java.lang.OutOfMemory
Heap Space
Get a heap dump
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/tmp/heapdump.hprof
CFML: Performance and Stability
Performance Problems
Java Memory
java.lang.OutOfMemory
Heap Space
Analyze your heap dump
CFML: Performance and Stability
Performance Problems
Java Memory
Application goes slow, eventually "stops"
CFML: Performance and Stability
Performance Problems
Java Memory
Application goes slow, eventually "stops"
CONCURRENCY!!!
CFML: Performance and Stability
Performance Problems
Concurrency
ab -c 100 -n 10000000 http://someurl.com/
Apache2 Utils
CFML: Performance and Stability
Performance Problems
Concurrency
Concurrency != Requests per second
MAXTHREADS - Your friend
<Service name="Catalina">
<Connector port="8888" maxThreads="50" ... />CFML: Performance and Stability
Performance Problems
Concurrency
Concurrency + locking
More locks = less concurrency
Take thread dumps
https://fastthread.io
CFML: Performance and Stability
Performance Problems
Summarize
Monitor:
- Logs
- Errors (Sentry, etc.)
- OS Perf (Zabbix, etc.)
- All the java (FusionReactor)
Test
- Load (Gatling, JMeter, Apache AB)
- CONCURRENCY CONCURRENCY
Analyze
- FusionReactor
- fastthread.io
- Eclipse MAT
Know
- Your application limits
Be
- Good to yourself!
Thanks for listening
https://www.preside.org
https://www.preside.org/signup
https://www.preside.org/slack
https://presidecms.atlassian.net
@dom_watson
@dom_watson
CTO @ Pixl8 Group
Preside lead developer
Improving performance and stability of CFML applications with FusionReactor (and more)
By Dominic Watson
Improving performance and stability of CFML applications with FusionReactor (and more)
It is early 2019, some of our complex web applications show signs of instability at relatively unpredictable times. Our extensive resource monitoring shows no particular pattern, web traffic logs again prove fruitless. Enter FusionReactor and some determination to get to the bottom of things. Dominic will take you through in detail how he discovered seemingly minor changes in Pixl8’s core platform stack that made some drastic improvements to performance. Come along if you’re interested in: * Micro optimization * Getting to the bottom of hard to debug problems * Improving overall problems in your applications
