Preface
It is important to realize that each and every environment has its own "recipe" for best performance.
This document is intended as a base to come up with a setup from which further enhancing / analyzing should be done.
As new insights are realized they will be added to this document.
Scope
In order to achieve better performance it is recommended to tune the Kopano server settings as well as other components such as MYSQL, LDAP and Apache
Audience
This document is intended to be used by experienced system administrators and engineers who have in-depth knowledge of both Linux and MySQL environments.
Disclaimer
Use these suggestions at your own risk and only if you know what you are doing.
These instructions are intended as pointers and not as a definitive configuration, offering the best performance for each and every environment.
Recommendations
These recommendations are valid for any Kopano server being used regardless if they function as a home server or archive server.
The so called zero user server sometimes used in conjunction with hosts running WebApp on a separate machine may need a different tuning.
We strongly recommend validating all parameters in a testing environment before deploying these in a production environment.
Kopano configuration
# requires to set the thread_stack to 512K in my.cnf enable_sql_procedures = yes # 8 threads is mostly sufficient for up to 500 users # Other rule of thumb is to have 4 threads per cpu core threads = 8
Things you can disable in the default configuration and instead run less frequently with a cronjob.
sync_gab_realtime = no softdelete_lifetime = 0
Periodically run from a cronjob:
- kopano-admin --sync (recommendation every 30 minutes, or manually when you just have made changes in LDAP)
- kopano-admin --purge-softdelete 30 (recommendation daily outside of production hours)
The rest of the tunable parameters
Kopano's cache system is optimized for short sessions. If you encounter that your system is constantly swapping and you cannot put more ram in your server it might be better to reduce these caches, than to have ram of kopano-server and mysql put into swap.
# Should be set as high as you can afford to set it. As it is very important to the overall performance of Kopano. As a rule of thumb it should at least be set to 25% of your ram. cache_cell_size = 2048M # caches all objects and their folder hierarchy cache_object_size = 100kb / user # contains all unique id's of objects cache_indexed_object_size = 512kb / user # Up to 500 users per 1M cache_quota_size = 1M # 60 seconds is more than sufficient cache_quota_lifetime = 1 # Up to 500 users per 1M cache_acl_size = 1M # Up to 500 users per 1M cache_store_size = 1M # 25M per 500 users cache_userdetails_size = 25M
New proposed defaults
cache_object_size = 16+M
cache_indexedobject_size = 32+M
Mysql configuration
In general we recommend to use mysql 5.6 or higher for performance reasons, and sugges to have the databases stored on either EXT4 or XFS filesystems.
Depending on the mysql version some of the settings below might not be available.
Warning: Changing settings like innodb_data_file_path and innodb_log_file_size require either a dump and restore or might require you to delete logfiles. Failure to do so will result in a non-working Mysql setup.
[mysqld] # This can be optionally uncommented. (check if supported on your mysql version) transaction-isolation = READ-COMMITTED # Careful this might also needs binlog_format=ROW if you are using a binlog. #verify# if binlog_format=statement. # Should be the minimum amount of kopano(server.cfg) threads(default 8) + thread_limit(default 40) + 9. max_connections = 57 # Best value is 32M, for smaller environments 16M is ok too. max_allowed_packet = 32M # should not be set higher than this, since it has to be allocated before being used tmp_table_size = 64M max_heap_table_size = 64M # max size 2M (regardless of size of environment) sort_buffer_size = 2M join_buffer_size = 2M # Good value is mumber of threads configured + 1 # nice writeup about actual scaling of thread_cache_size http://anothermysqldba.blogspot.de/2013/09/mysql-optimization-tip-threadcachesize.html # #verify# number of threads to be dermineds thread_cache_size = 4 # Only available on mysql 5.5 and higher as above. Good figure the amount is cores/2, depending of usage give read more threads. innodb_write_io_threads = 4 innodb_read_io_threads = 4 # (Kopano tables + max mysql connections) * 10, for example (26 + 24) * 10 # #verify# can be checked for example with "mytop". when you have a low "key efficiency" table_cache has to be increased # table_open_cache was known as table_cache in MySQL 5.1.2 and earlier. table_open_cache = 500 # DO NOT SET IT TOO HIGH! https://www.percona.com/blog/2009/11/16/table_cache-negative-scalability/ # In order to use enable_sql_procedures = yes in Kopano server.cfg 512K is needed at least. thread_stack = 512K # RHEL6 do no use any query cache size !! kills performance query_cache_size = 64M query_cache_limit = 2M # On a shared system (running both Kopano 30% and Mysql 50% and 20% for system Apache etc.) only use 30% of total ram. innodb_buffer_pool_size = 2G # On slow storage 100M is recommended # syntax is file_name:file_size[:autoextend[:max:max_file_size]] innodb_data_file_path = ibdata1:50G:autoextend # On fast like SSD storage or multipath innodb_data_file_path = ibdata1:100G;ibdata2:100G:autoextend # increment ibdata with 1000MB each time (default is 4MB) innodb_autoextend_increment = 1000 innodb_file_per_table = OFF # Be careful 1 second of log (high performance = 2 , normal = 1) innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 8M # The threshold compared to innodb ( 25% of buffer pool size at max 1 GB) innodb_log_file_size = 100M innodb_log_files_in_group = 3 innodb_lock_wait_timeout = 120 # in case of remote storage innodb_flush_method = O_DIRECT # otherwise innodb_flush_method = O_DSYNC
kopano-cachestat.py
It is recommended to run kopano-cachestat.py only on a system that has been running for a few hours at least so every cache has been filled (warmed up).
Kopano-cachestat is shipped with Kopano Core
Example output;
Kopano Cache Statistics
Server start time: Di Jan 6 12:37:40 2016
Current time : Fri Jan 30 09:49:52 2016
Cache Hit ratio Mem usage ratio
uquota ( 609624/ 625300) (97%) ( 528/ 1048576) ( 0%)
obj ( 525421997/ 525531113) (99%) ( 5933808/ 67108864) ( 8%)
abinfo ( 11857088/ 11857152) (99%) ( 6888/ 26214400) ( 0%)
userid ( 17280384/ 18554192) (93%) ( 3608/ 1048576) ( 0%)
quota ( 1256545/ 1377896) (91%) ( 440/ 1048576) ( 0%)
server ( 0/ 0) (N/A) ( 0/ 1048576) ( 0%)
cell ( 930698599/ 932916096) (99%) ( 127201976/ 2621440000) ( 4%)
index2 ( 65447307/ 66177951) (98%) ( 27128446/ 134217728) (20%)
extern ( 5987446/ 6506413) (92%) ( 5104/ 1048576) ( 0%)
index1 ( 297211484/ 297487612) (99%) ( 25520646/ 134217728) (19%)
store ( 331842593/ 448856038) (73%) ( 1010640/ 1048576) (96%)
acl ( 121527/ 121858) (99%) ( 19536/ 1048576) ( 1%)
Explanation of the fields
uquota = cache_quota_size obj = cache_object_size abinfo = cache_userdetails_size userid / extern = cache_user_size quota = cache_quota_size server = cache_server_size cell = cache_cell_size index1 / index2 = cache_indexedobject_size extern = cache_user_size acl = cache_acl_size
Apache
It is recommended to only have the Apache modules installed which are required.
At the time of writing we recommend to enable to following modules for WebApp;
mod_expires
mod_headers
mod_setenvif
mod_deflate
PHP
You can use a PHP accelerator like php-apc, Xcache or OPCache to speed up WebApp and Z-Push.
Z-Push
# Make sure it is enabled KeepAlive On
# Should be higher or equal to the ping time as configured in the z-push/config.php (define('PING_INTERVAL', 30);)
KeepAliveTimeout 45
# This should be higher than the amount of concurrent users you expect.
MaxKeepAliveRequests 500
# Should be sufficiently high for all your users to connect.
MaxClients 500
When experiencing a high server load both values could be increased, as the 'push' will take place less often.
OpenLDAP optimization
(Debian / Ubuntu commands given and paths used in examples)
Attribute Indexing
Create a file called optimize-index.ldif containing:
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: cn eq
olcDbIndex: gidNumber eq
olcDbIndex: mail eq
olcDbIndex: memberUid eq
olcDbIndex: ou eq
olcDbIndex: uid eq
olcDbIndex: uidNumber eq
olcDbIndex: uniqueMember eq
olcDbIndex: kopanoAccount eq
olcDbIndex: kopanoAliases eq
olcDbIndex: kopanoViewPrivilege eq
Add the ldif to add the new indexes.
# cat optimize-index.ldif | ldapmodify -Y EXTERNAL -H ldapi:///
Check if our new olcDbIndex keys have been added.
# slapcat -b cn=config | grep olcDbIndex:
The output should be similar to below
olcDbIndex: objectClass eq olcDbIndex: cn eq olcDbIndex: gidNumber eq olcDbIndex: mail eq olcDbIndex: memberUid eq olcDbIndex: ou eq olcDbIndex: uid eq olcDbIndex: uidNumber eq olcDbIndex: uniqueMember eq olcDbIndex: kopanoAccount eq olcDbIndex: kopanoAliases eq olcDbIndex: kopanoViewPrivilege eq
You could check your slapd logging for suggestion of additional candidates for indexation.
# cat /var/log/syslog |grep bdb_equality_candidates
Configuring OpenLDAP Database
Check if the DB_CONFIG settings in /var/lib/ldap/DB_CONFIG are sufficient for the amount of records in your OpenLDAP.
set_cachesize 0 2097152 0 set_lk_max_objects 1500 set_lk_max_locks 1500 set_lk_max_lockers 1500
These values should be increased to suit the size of your Ldap.
See the two paragraphs below for more information.
OpenLDAP Database Caching
Checking the current cache statistics
2MB 520KB Total cache size 1 Number of caches 1 Maximum number of caches 2MB 520KB Pool individual cache size 0 Maximum memory-mapped file size 0 Maximum open file descriptors 0 Maximum sequential buffer writes 0 Sleep after writing maximum sequential buffers 0 Requested pages mapped into the process' address space 359827 Requested pages found in the cache (99%) 24 Requested pages not found in the cache 0 Pages created in the cache 24 Pages read into the cache 0 Pages written from the cache to the backing file 0 Clean pages forced from the cache 0 Dirty pages forced from the cache
OpenLDAP Database Locking
Checking the current locking statistics
84 Last allocated locker ID 0x7fffffff Current maximum unused locker ID 9 Number of lock modes 1500 Maximum number of locks possible 1500 Maximum number of lockers possible 1500 Maximum number of lock objects possible 40 Number of lock object partitions 7 Number of current locks 46 Maximum number of locks at any one time 3 Maximum number of locks in any one bucket 0 Maximum number of locks stolen by for an empty partition 0 Maximum number of locks stolen for any one partition 12 Number of current lockers 23 Maximum number of lockers at any one time 7 Number of current lock objects 39 Maximum number of lock objects at any one time
Verifying tuning
The following tests can be done to analyze the the tuning results.
They should be performed on adequately warmed up systems compare the output of the the following commands before and after the tuning.
Kopano
kopano-cachestat before and after the tuning.
kopano-stats --top check the rtt and qlen value
Here generally the values should be lower and also the duration of peaks should be shorter.
LDAP
Use the script provided here https://github.com/zarafagroupware/zarafa-tools/blob/master/helpers/ldap/ldap_export.sh (requires ldapsearch to be installed).
This would mimic opening op the GAB 50 as an user does.
Use the wrapper below to have it run a couple of times to get the average of that.
#!/bin/bash
count=0
times=50
while [ $count -lt 50 ]
do
./ldap_export.sh > /dev/null
count=$[$count+1]
done
Adjust the times variable to a value which makes it easier to compare later.
# time ./time.sh
Mysql
Compare the output ot the mysql tuning script being used before and after.
Use either one MySQLTuner-perl or mysql-tuning-primer scripts.
Always keep in mind there will always be recommendations as tuning is an ongoing process.
Additional Tools / Information
These tools can be used to get some additional tuning recommendations, please only run this tools when the database(is warmed up) has been up and running in production for some time days rather than hours.
https://github.com/major/MySQLTuner-perl
https://launchpad.net/mysql-tuning-primer/
Mysql's temp files on ramdisk using tmpfs, or on a ssd disk partition
It might be useful to run /tmp on tmpfs or instruct mysql to run its temp files somewhere on a tmpfs mountpoint, or an ssd disk partition (YMMV).
Be very careful with this !!
[mysqld] tmpdir = /some/mountpoint
See the links below for more information.
http://everythingmysql.ning.com/profiles/blogs/using-tmpfs-for-mysqls-tmpdir
More background information
How to calculate a more suitable innodb_log_file_size http://www.percona.com/blog/2008/11/21/how-to-calculate-a-good-innodb-log-file-size/
Information regarding binlog_format=ROW is needed when using transaction-isolation = READ-COMMITTED http://bugs.mysql.com/bug.php?id=66619
Nice writeup about actual scaling of thread_cache_size http://anothermysqldba.blogspot.de/2013/09/mysql-optimization-tip-threadcachesize.html
Kopano administrator manual
kopano core administrator manual
Todo: topic to take a look at use/switch to syslog
Notes on using virtualization
Do's and don'ts
Do
- Use paravirtualized drivers when available for both disk and network interfaces.
- Plan your virtualization needs and size accordingly.
Don't
- Use any of the following features of your virtualization platform as they adversely impact Kopano, MySQL or any program which actively use a lot of memory.
- Ballooning
- Scorecard
- Compression
Page Sharing or Memory De-Duplication
Overcommit Memory, do not give out more memory as the host machine has, as this will cause swapping in the host system
- Do not use virtualization just for the sake of it.