Today I analyzed disk reads and writes on a server with iotop and strace and found some interesting possible optimizations.
With iotop you can check which processes are reading and writing from the disks. I always press the o, p and a keys in iotop so that it only shows me processes doing I/O and so that it will show accumulated I/O instead of the bandwidth. With the left and right arrows I select on which columns to sort the list.
Once you have identified the processes wich are doing much I/O, you can check what they are reading or writing with strace, for example
# strace -f -p $PID -e trace=open,read,write
(you can leave out read and/or write if this gives too much noise)
This way I identified some locations where processes do lots of read and write operations on temporary files.
For nagios I placed /var/lib/nagios3/spool and /var/cache/nagios3 on a tmpfs, for Amavis /var/lib/amavis/tmp and for PostgreSQL /var/lib/postgresql/8.4/main/pg_stat_tm.
Other candidates you might want to consider: /tmp, /var/tmp and /var/lib/php5. There are probably many others, depending on which services you use.