SELECT UUID() as data1, UUID() as data2, UUID() as data3, UUID() as data4, 
UUID() as data5, UUID() as data6, UUID() as data7, UUID() as data8, 
UUID() as data9, UUID() as data10, UUID() as data11, UUID() as data12, 
UUID() as data13, UUID() as data14, UUID() as data15, UUID() as data16,
UUID() as data17, UUID() as data18, UUID() as data19, UUID() as data20,
UUID() as data21, UUID() as data22, UUID() as data23, UUID() as data24,
UUID() as data25, UUID() as data26, UUID() as data27, UUID() as data28,
UUID() as data29, UUID() as data30, UUID() as data31, UUID() as data32

It has 32 fileds, and every one of them is 36 bytes long, which makes the whole record 1152 bytes.

MySQL can hardly utilize  all the cpu and network traffic, so I started four MySQL instances on one machine and four clients remotely. After warm-up period, I captured realtime network traffic with ‘sar -n DEV’:
QPS for each instance are 12823, 12936, 13153 and 13123. Basicly 2370 byte per query.

We all know its original size is just 1152 bytes, so that’s a surprisingly expensive query! Let’s dig a little deeper to see what the hell happens inside. Read chapter ”Client/Server Communication” of “Understanding MySQL Internals” carefully, the knowledge there is still updated (although it’s 6 years old already). Usually ppl are too lazy to read it by themselves, so I summarized it a little.

There are four kinds of packets sent from server to client:

1. OK Packet
2. Error Packet
3. EOF Packet
4. Result Set Packets (NOTICE: plural)

Result Set Packets contain field definition sequence of packets and data packets.

After server receives a query, it responses client with result Set packets as following sequence:

1. A packet with the body consisting of the standard field-length specifier sequence
2. A group of field description packets, one for each field, in the field order of the result set
3. EOF packet
4. Row data packets, one packet per row
5. EOF packet

Before finding out what’s inside every packet, we need to learn an important definition called “Data Field” first. Basicly, every field in MySQL packet is abstracted as ”Data Field”, and the length of “Data Field” is like the following table:

Length of Actual Data (N Byte)	Length specifier sequence	Length of Data Field
N < 251 Byte	1 Byte	(1 + N) Byte
N < 65536 Byte	2 Byte	(2 + N) Byte
N < 16777216 Byte	3 Byte	(3 + N) Byte
N >= 16777216 Byte	8 Byte	(8 + N) Byte

According to the table we define a function called “length_of_data_filed”:

Function length_of_data_filed(data)
{
  If (length(data) < 251)
    return (length(data)+1);
  else if (length(data) < 65536)
    return (length(data)+2);
  else if (length(data) < 16777216)
    return (length(data)+3);
  else
    return (length(data)+4);
}

Here comes the length of packets:

1. Standard Field-length Specifier Sequence Packet, 1 byte
2. Field Description Packet, 17 + length_of_data_filed(db_name) + length_of_data_filed(table_name) + length_of_data_filed(alias_table_name) + length_of_data_filed(column_name) + length_of_data_filed(alias_column_name) + length_of_data_filed(default_value)
3. Row Data Packet, length_of_data_filed(field1) + length_of_data_filed(field2) + … + length_of_data_filed(fieldn)
4. EOF Packet, 5 bytes

To verify if the knowledge in  ”Understanding MySQL Internals” is updated, I used Wireshark to analyze packets captured by tcpdump on Linux server. The whole analysis is shown as below:

Ethernet II protocol costs 14 byte, IP V4 costs 20 bytes, and TCP costs 20 bytes too

One TCP packet can contain several MySQL packets, but it depends on MySQL packet size. In my kv query case, one TCP packet is big enough to hold many MySQL packets. Usually it costs 4 bytes to delimit MySQL packets inside TCP packet. One byte is used to describe the sequence of MySQL packet, and the rest 3 bytes are used for delimiter.

Standard Field-length Specifier Sequence Packet

Field Description Packet

Row Data Packet

EOF Packet

Honestly speaking, it’s very hard to analyze network traffic precisely, because TCP  behavior varies greatly along with the change of MySQL packet’s size. However it’s easy if we assume that only one record will be returned for one query, which means kv query. Assuming the length of record is k bytes,  no alias name is used and no default vaule is set either, we get the formula of network traffic caused by kv query :

Field definition sequence of packets + Data packets
= (21 + length_of_data_filed(db_name) + length_of_data_filed(table_name) + 1 + length_of_data_filed(column_name) + 1) * field_number + 68 + 4 + k + field_number + 63
= (24 + length_of_data_filed(db_name) + length_of_data_filed(table_name) + length_of_data_filed(column_name)) * field_number + k + 135

That’s all, ^_^.
But wait! why did I show you the experiment in the beginning of this post? Well, I left a little homework for ppl who want to know if the formula matches the network traffic in real life. So have fun!

I am sorry to tell you, that’s not possible without limitation. So the problem becomes how can we generate server_id  most uniquely.
server_id is a 32-bit int in MySQL, and IP address is represented as a 32-bit int in C too. To record port information into a 32-bit int, we need to ignore some bits from the head of IP address which are always the same to us. Here is my way to do it in Linux:

#!/bin/bash

hostip=`hostname -i`
a=`echo $hostip|cut -d\. -f1`
b=`echo $hostip|cut -d\. -f2`
c=`echo $hostip|cut -d\. -f3`
d=`echo $hostip|cut -d\. -f4`
port=65 #Change this in your code

echo "${a}.${b}.${c}.${d}"
serverid=`expr \( ${a} \* 256 \* 256 \* 256 + ${b} \* 256 \* 256 + ${c} \* 256 + ${d} \)`
echo ${serverid}

serverid=$((${serverid} << 6))
serverid=`expr ${serverid} % 4294967296`
echo ${serverid}

serverid=`expr ${serverid} + \( ${port} % 64 \)`
echo ${serverid} #Yeah, here we go

Here  is a  configuration example for  multi MySQL instances running on Redhat or CentOS 6:

[mysqld_safe]
pid-file=/mysql/myPORT/run/mysqld.pid
[mysql]
port=PORT
prompt=\\u@\\d \\r:\\m:\\s>
default-character-set=gbk

[client]
port=PORT
socket=/mysql/myPORT/run/mysql.sock

[mysqld]
#dir
basedir=/mysql/myPORT
datadir=/mysql/myPORT/data
tmpdir=/mysql/myPORT/tmp
lc_messages_dir=/usr/share
log-error=/mysql/myPORT/log/alert.log
slow_query_log_file=/mysql/myPORT/log/slow.log
socket=/mysql/myPORT/run/mysql.sock

#innodb
innodb_data_home_dir=/mysql/myPORT/log/iblog
innodb_log_group_home_dir=/mysql/myPORT/log/iblog
innodb_buffer_pool_size=INNODB_BUFFER_POOL_SIZE
innodb_buffer_pool_instances=8
innodb_log_files_in_group=4
innodb_log_file_size=1G
innodb_log_buffer_size=64M
innodb_flush_log_at_trx_commit=1
innodb_additional_mem_pool_size=20M
innodb_max_dirty_pages_pct=60
innodb_io_capacity=INNODB_IO_CAPACITY
innodb_thread_concurrency=THREADBY4
innodb_read_io_threads=THREAD
innodb_write_io_threads=THREAD
innodb_open_files=60000
innodb_file_format=Barracuda
innodb_file_per_table=1
innodb_flush_method=O_DIRECT
innodb_flush_neighbor_pages=0
innodb_change_buffering=inserts
innodb_adaptive_flushing=1
innodb_adaptive_flushing_method=keep_average
innodb_adaptive_hash_index_partitions=8
innodb_old_blocks_time=1000
innodb_fast_checksum=1
innodb_stats_on_metadata=0
innodb_lazy_drop_table=1
innodb_read_ahead="linear"
innodb_use_native_aio=1
innodb_lock_wait_timeout=5
innodb_rollback_on_timeout=0
#innodb_strict_mode=1
transaction-isolation=READ-COMMITTED

#myisam
key_buffer=64M
myisam_sort_buffer_size=64M
concurrent_insert=2
delayed_insert_timeout=300

#replication
master-info-file=/mysql/myPORT/log/master.info
relay-log=/mysql/myPORT/log/relaylog
relay_log_info_file=/mysql/myPORT/log/relay-log.info
slave_load_tmpdir=/mysql/myPORT/tmp
slave_type_conversions="ALL_NON_LOSSY"
slave_net_timeout=4
skip-slave-start

#binlog
log-bin=/mysql/myPORT/log/binlog
server_id=1153668
binlog_cache_size=32K
max_binlog_cache_size=2G
max_binlog_size=500M
binlog-format=ROW
sync_binlog=1
log-slave-updates
expire_logs_days=7

#server
default-storage-engine=INNODB
character-set-server=gbk
lower_case_table_names=1
skip-external-locking
open_files_limit=65536
safe-user-create
local-infile=0
#sqlmod="STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE"
performance_schema=0
log_slow_admin_statements=1
log_slow_verbosity=full
log_warnings=1
long_query_time=1
slow_query_log=1
query_cache_type=0
query_cache_limit=1M
query_cache_min_res_unit=1K
table_definition_cache=65536
table_cache=65536
thread_stack=512K
thread_cache_size=256
read_rnd_buffer_size=128K
sort_buffer_size=256K
join_buffer_size=128K
read_buffer_size=128K
port=PORT
skip-name-resolve
skip-ssl
max_connections=8192
max_user_connections=8000
max_connect_errors=65536
max_allowed_packet=128M
max_long_data_size=128M
connect_timeout=8
net_read_timeout=30
net_write_timeout=60
back_log=1024

可怜的妈妈早上九点多就被推到产房，下午出来整个人都虚脱了。

出生的第一天，毛豆同学只张开了一只眼睛，样子看起来好挫哦！不过第二天一大早洗了澡之后，就变得稍微可爱了一些。
奇怪的是，怎么看起来像个男胖子呢？

大家都说毛都长得像粑粑，但是为什么越看越像范伟呢…………………………

奶奶老是觉得毛豆很冷，于是穿上了麻麻小时候的花棉袄，一股乡土气息扑面而来…………
为什么看起来这么像村中一霸？！

可怜的毛豆，穿得太多结果得了湿疹，于是又换上了gap的战袍~（还是一个小胖子！）

虽然只有两个星期大，但已经是一个赤果果的表情帝了~

 18天，双眼皮已经很明显了。随访的阿姨说体重涨到了8斤。

再来两张！脸上肉多的挂不住了！

再过两天就满三个月啦，天天要妈妈竖着抱，天天咿咿呀呀的小胖子。

妈妈最喜欢乖宝宝

每次醒来发现旁边没人，毛豆就会哭得很可怜。

四个半月啦，越来越像爸爸小时候了。

五个月啦！

1. MySQL error code 167: Failed to set row auto increment value
This error is due to a HandlerSocket bug. If you can understand Chinese, please read this post written by Ruohui Huang.
The solution here is very easy: set innodb_autoinc_lock_mode to 0.

2. HNDSOCK failed to lock tables
If you try to use HandlerSocket API to update a read-only InnoDB table, this message will appear in the error log.

3. DDL timeout
It is very hard to do DDL on HandlerSocket enabled tables beacause it keeps tables opened for reuse. Even HandlerSocket in Percona server would close tables when traffics become small, there is definitely no parameter to control this behaviour. If you are suffering from this DDL problem, maybe you can try HandlerSocket in Spider server. There is a parameter named handlersocket_close_table_interval, hope it’s helpful to you.

But actually I fount that you cannot use /dev/sd[f-i] and /dev/sda1 only, and all of the EBS attached will be renamed by Redhat internally.
So the conclusion is, we can attach 26-3+26*15-1+26+26*15=828 volumes!

But wait, even attaching many volumes will provide a better performance, do we need to attach so many volumes?
Of course not! mdadm, the soft raid used in Ec2 instance, cannot deal with too many disks at the same time. Maybe mdadm does not support mutil-thread very well, or there is some other bottle necks in the soft RAID, the best performance will be gained if we use 16 or less volumes under soft RAID.

This series of tests were conducted on m2.xlarge instance by SysBench.

I set up a MySQL replication several months ago, and we don’t use these production servers yet. So you can imagine how surprised I was when I saw  the CPU usage. Load is minor but MySQL uses 100% CPU in top.

top - 08:50:55 up 114 days,  5:46,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 140 total,   1 running, 139 sleeping,   0 stopped,   0 zombie
Cpu(s): 13.7%us,  0.5%sy,  0.0%ni, 85.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  35128056k total,  6323556k used, 28804500k free,   351320k buffers
Swap:        0k total,        0k used,        0k free,  2927496k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                       
28879 mysql     20   0 29.9g 2.2g 5732 S 105.0  6.5 117807:11 mysqld

I tried to restart one of the MySQL, but it didn’t help. After I restarted one of the servers, things changed.

top - 08:55:10 up  7:09,  2 users,  load average: 0.00, 0.00, 0.00
Tasks: 142 total,   1 running, 141 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.4%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  35128056k total,  3169936k used, 31958120k free,    86580k buffers
Swap:        0k total,        0k used,        0k free,   297368k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                       
 3382 mysql     20   0 29.0g 2.1g 5188 S  0.0  6.4   0:22.37 mysqld

pstack usually will tell us what’s going on in such a situation, but I fount that the normal and abnormal instances have similar stack:

Thread 19 (Thread 0x7f34c4048700 (LWP 28887)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f2dde25d700 (LWP 28888)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f2ddd85c700 (LWP 28889)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f2ddce5b700 (LWP 28890)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f2ddc45a700 (LWP 28891)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f2ddba59700 (LWP 28892)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f2ddb058700 (LWP 28893)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f2dda657700 (LWP 28894)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f2dd9c56700 (LWP 28895)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f2dd9255700 (LWP 28896)):
#0  0x0000003a5ca00614 in ?? () from /lib64/libaio.so.1
#1  0x00000000008fe503 in os_aio_linux_handle ()
#2  0x00000000008b2d78 in fil_aio_wait ()
#3  0x0000000000833940 in io_handler_thread ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f2dd8854700 (LWP 28904)):
#0  0x000000000082eeb4 in srv_lock_timeout_thread ()
#1  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#2  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f2dd7e53700 (LWP 28905)):
#0  0x0000003a5d60b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000008ffd53 in os_event_wait_time_low ()
#2  0x000000000082ea15 in srv_error_monitor_thread ()
#3  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f2dd7452700 (LWP 28906)):
#0  0x0000003a5d60b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000008ffd53 in os_event_wait_time_low ()
#2  0x000000000083188b in srv_monitor_thread ()
#3  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f2dd6a51700 (LWP 28907)):
#0  0x0000003a5cede2f3 in select () from /lib64/libc.so.6
#1  0x0000000000900baf in os_thread_sleep ()
#2  0x000000000082c7fa in srv_LRU_dump_restore_thread ()
#3  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f2dd6050700 (LWP 28908)):
#0  0x0000003a5d60b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000008ffea1 in os_event_wait_low ()
#2  0x000000000082f4b1 in srv_master_thread ()
#3  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2dd564f700 (LWP 28911)):
#0  0x0000003a5d60f245 in sigwait () from /lib64/libpthread.so.0
#1  0x0000000000513bab in signal_hand ()
#2  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f2dd560e700 (LWP 28912)):
#0  0x0000003a5d60e4ed in read () from /lib64/libpthread.so.0
#1  0x0000000000989402 in vio_read_buff ()
#2  0x000000000051c852 in my_real_read(st_net*, unsigned long*) ()
#3  0x000000000051cb5c in my_net_read ()
#4  0x00000000006637aa in cli_safe_read ()
#5  0x0000000000530d11 in handle_slave_io ()
#6  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#7  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f2dd55cd700 (LWP 28913)):
#0  0x0000003a5d60b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000716e43 in MYSQL_BIN_LOG::wait_for_update_relay_log(THD*) ()
#2  0x000000000052de67 in exec_relay_log_event(THD*, Relay_log_info*) ()
#3  0x000000000052f059 in handle_slave_sql ()
#4  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003a5cee570d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f34c404a700 (LWP 28879)):
#0  0x0000003a5cedc053 in poll () from /lib64/libc.so.6
#1  0x0000000000514cf4 in handle_connections_sockets() ()
#2  0x000000000051ad38 in mysqld_main(int, char**) ()
#3  0x0000003a5ce1ecdd in __libc_start_main () from /lib64/libc.so.6
#4  0x0000000000510099 in _start ()

We can locate that thread 28905 uses the most CPU in this instance.

top - 09:07:55 up 114 days,  6:03,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 166 total,   2 running, 164 sleeping,   0 stopped,   0 zombie
Cpu(s): 14.9%us,  0.3%sy,  0.0%ni, 84.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:  35128056k total,  6323424k used, 28804632k free,   351320k buffers
Swap:        0k total,        0k used,        0k free,  2927504k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                       
28905 mysql     20   0 29.9g 2.2g 5732 S 62.4  6.5  5124100h mysqld

What is this thread doing? Check pstack again, it seems to be a sleeping thread:

Thread 8 (Thread 0x7f2dd7e53700 (LWP 28905)):
#0  0x0000003a5d60b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000008ffd53 in os_event_wait_time_low ()
#2  0x000000000082ea15 in srv_error_monitor_thread ()
#3  0x0000003a5d6077f1 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a5cee570d in clone () from /lib64/libc.so.6

WTF? How come a sleeping thread eat up so much CPU? My previous colleague suggested me to use PMP. It is very convenient for me because we installed Percona toolkit.

# pt-pmp -p 28879 -s 5
Wed Sep 19 09:36:15 GMT 2012
     10 libaio::??,os_aio_linux_collect,os_aio_linux_handle,fil_aio_wait,io_handler_thread,start_thread,clone
      1 srv_lock_timeout_thread,start_thread,clone
      1 sigwait,signal_hand,start_thread,clone
      1 select,os_thread_sleep,srv_LRU_dump_restore_thread,start_thread,clone
      1 read,vio_read_buff,my_real_read,my_net_read,cli_safe_read,read_event,handle_slave_io,start_thread,clone
      1 pthread_cond_wait,os_cond_wait,os_event_wait_low,srv_master_thread,start_thread,clone
      1 pthread_cond_wait,inline_mysql_cond_wait,MYSQL_BIN_LOG::wait_for_update_relay_log,next_event,exec_relay_log_event,handle_slave_sql,start_thread,clone
      1 pthread_cond_timedwait,os_cond_wait_timed,os_event_wait_time_low,srv_monitor_thread,start_thread,clone
      1 pthread_cond_timedwait,os_cond_wait_timed,os_event_wait_time_low,srv_error_monitor_thread,start_thread,clone
      1 poll,handle_connections_sockets,mysqld_main,__libc_start_main,_start

Here is a question that PMP’s result differs from top -H. That is because PMP order the result based on the times of function appeared in stack, when top -H only order the result by CPU usage. In our case, top -H seems more helpful. All of the information we know now indicate that Redhat 6.3 and EC2 instance are suspicious. Currently I am working on systemtap to make suer, and I will update this post if anything new discovered.

——————-UPDATE——————-
This problem was solved!
Check the post here: http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

就拿最自豪的小学成绩来说，我只拿过三年级的一次期终考全校第一。更多时候我需要和同班的毛霄雯班长、邵迪和张儒展开拉锯战，企图保住前三的宝座。而我在心里面认为，张儒肯定比我要强。至于课外的数学奥林匹克就更不用说了，全年级高手如云，我始终也只是中上水平。

很悲催的，老妈对我的课外投资都打了水漂。毛笔练得狗咬吕洞宾不说，数奥最后只得了个全国三等奖，搞得六中的老师打电话过来询问是否要买个名额的时候，语气仿佛是在买过夜的猪肉一般蔑视。老妈的梦想是送我上好的大学，而好的大学依赖于好的高中，而好的高中依赖于好的初中。当她含泪咬牙切齿问我要不要掏钱进六中的时候，年幼的我感到了莫大的压力，可耻地逃到了分配的十二中。

到了十二中后，大家依然疯狂地竞争着，而我依然游离在成功之外。数奥就不说了，像篮球这种纯爷们的运动，天天打也就只能凑个最佳第六人。更关键的是，这么青春的岁月里，我长得一点都不帅。实在是离成功非常遥远。不过感谢倒数第二次模拟考，考得太烂受到了刺激，终于每天晚上狂喝雀巢咖啡，最后一次模拟考获得了继小学三年级之后的又一次全校第一。不过真正的升学考试时，掉到了全校第六，不过美滋滋地挤进了全市最好的高中温州中学。

本以为到了温州中学，生活就会美好起来，事实证明这才是地狱。论天才程度，本班的虞冠驰同学无与伦比。论帅气，又非郑衡莫属。各门科目就别说了，各科猛人无数。这个情况还只限于我所在的六班是全校倒数。隔壁班的某个妹子是我家旁边面店老板的女儿，最烦的就是她老妈时不时过来问问考试成绩。虽然我没有loser的自觉，但是老爸老妈总会发发脾气。

等到文理分班，六班因为太烂被拆掉之后，我很荣幸地被分到了三班。更荣幸的是，这边有更加天才而且努力的张立中、耳凌宇。当时全年级600个人，排名下来我也只能排到100出头。到了高考前夕，突然有阵子就是什么都不想学。最后考试掉到了全年级200名，感觉真是人生最灰暗的时候。老妈天天叨叨上浙大，实际情况却是根本上不了了，只能背井离乡去了西电。

大学生活很快乐。不过尽管我比一般人努力那么一丝丝，也仍旧是比不过那些学霸们。我仍旧记得胡宁杭、葛小村、陈科等各科猛男。即使是我比较喜欢的编程，前面也有刘宇康等硕大的阴影笼罩。每次评奖学金，总是排在三等的前列，勉强成为三等残废中的领军人物。学校里的社团就更不用说了，拍马屁的功夫不到位，脸皮也没人那么厚，上不上下不下整个一苏打夹心饼干。

最后想考gre出国留学，累死累活看看太傻就泄气，那些学霸上来就说自己考了多少多少有几个论文，也不管潜水群众们受不受得了，末了还来一句自己觉得没戏。家里老妈要发疯，成天唧唧歪歪外国的美好生活，搞得留学了就能呼风唤雨年薪百万一样。作为一个loser，被密西根州立的教授抛弃也算是意料之中，获得IIT一个坑钱的doctor offer也属正常，比较幸运的是我选择留在淘宝工作。

2008年的淘宝并不拥挤，即使是没什么想法，没什么智商的我，依旧可以干得有滋有味。但是说起成功，连铭比我聪明，竹庄比我勤奋；而均量，则比我高。一年又一年过去，工资在涨，压力也在变大。新来的同事们更加勤奋和聪明，而此时的我，依旧离所谓的成功有四个p级那么远。有时候想想不免气馁，为什么自己就是没有想象中那么牛气。

如今我更是身在新加坡，过着每天被日本人监控的日子，住着豪华的佣人房，离成功已经需要用光年来计算。然而新加坡同事偶然的一句话让我感觉到自己其实还蛮幸福。他羡慕地说：＂30岁不到就娶到了老婆，你真行。＂

突然之间，我想起了很多事情。
小学每天能吃到的烧饼、锅贴、糯米饭，初中每天半小时的篮球3v3，高中周三下午的闲暇时间，大学周末的钓鱼，工作后给爷爷的压岁钱，结婚后和老婆出去兜兜风，等等。很多平时容易遗忘的小幸福，在不成功的时候我也照样享受到了。如果说知足常乐的话，我想离成功远一些，也许会更加幸福。阿Q一点来说，反正成功追也追不到，保持那么点距离，貌似蛮好！

After several hours’ searching on Google, finally I found a clear definition in “A Gentle Introduction to symfony”.
Symfony provides three alternative types of intelligent code fragments to replace includes:
• If the logic is lightweight, you will just want to include a template file to some data you pass to it. For that, you will use a partial.
• If the logic is heavier (for instance, if you need to access the data modify the content according to the session), you will prefer to presentation from the logic. For that, you will use a component.
• If the fragment is meant to replace a specific part of the layout, for content may already exist, you will use a slot.

And the most important reason that we use a component instead of an action is, we cannot include an action in the template!
Wish this will end the discussions on Stack Overflow. But I have another question for all the viewers: which layer should component belong to?

According to “A Gentle Introduction to symfony”, it should belong to view layer. However it is very weird to me that there is complicated business logic in view layer.
Crazy to think about those questions… Maybe we should leave those questions behind, and start to focus on coding?