Solaris Performance Techniques, Chicago Feb 2008 ---------------------------------------------------------------------- focus on problem (problem description), not on metrixs (how busy X is) Q=N/R Q=throughput N=amount of X (i.e. threads) R=rate instrumentation levels/trigger - IT depends ---------------------------------------------------------------------- fsstat fsstat ufs 1 #replaces iostat in some areas nicstat dtrace toolkit cputrack abitrace lockstat -k psrinfo -p #find exceptions prstat -m prstat -Lmp PID kstat -c net #get full/half, speed plockstat dumpstabs #show compiled options for binary in Studio detail slides: medicast.sun.com - Linda Katlie oracle - use tracewait (oracle tool) and not dtrace errinfo -c (dtrace toolkit) dtrace - give zone priv within a zone pmap -sx PID zvmstat (zone vmstat) fstat -> dtrace fsinfo prstat -nL #num threads used & locks plockstat look for *stat* in /usr/sbin, /sbin dtrace examples installed in /usr/demo/dtrace (get dtrace manual/doc) dtrace -l | grep pattern ---------------------------------------------------------------------- disk & network: are more IOPS (latency/responsetime) issue & not throughput laden cpu utilization is almost always useless because of virtualization, threads, cpu cores prstat - too many cpus to track (512 core systems on the way) mpstat - wt (wait) field is always 0 - cpus don't wait, threads do use dtrace to quickly drill down intrstat (can be broken in pre update4 release - either 0 or very large) device interrupts are (default) roundrobin across cpus, use prstat if you have Gbe & large traffic load ---------------------------------------------------------------------- sunstudio toolkit: collect, debugger & tools & compilers (free!!!) always use latest s/w (compilers, apps) and patches gcc - sun has optimized gcc for sparc, see cool tools in bigadmin compile with "-fast" or "-0x3" (min) debug - cc -S, use mdb to disassemble dumpstabs #show compiled options for binary in Studio ---------------------------------------------------------------------- performance issues: mindset, perceptions, skills, management views, pressures look everywhere test, control, track, tune: from slowest to fastest (slowest) network, I/O, cpu, memory, kernel, apps (fastest) #my note: usually the app is where 80% of the tuning gain is though tune stack use latest s/w (compilers, apps) and patches interactive - tune sys => app ...loop... app: dprofile (memory), dtrace (threads, processes) tunefs - if I/O bad => look @ FS issues Java 1.5b11 - has autotuner builtin, don't tune via arguments cpu bound = 2.5 runqueue (w/ no blocks) times the number of cpu cores high %sys: intrstat (of high sys time, but not high % cpu in prstat) trapstat shortlived.d #shortlived processes that prstat misses 1sec -vs- 1 nanosec ---------------------------------------------------------------------- ZFS let ZFS see both mirrors for healing, don't give just the RAID single high end LUN(s) 5sec write flush blocksize/IOsize = 128K ZIL = ZFS Intent log cache - set nfs_nocache_flush can change block size in ZFS ZFS not good in random writes (yet) ZFS has no direct I?O not cpu effecient (yet) wants lots of cpu not good for busy OLTP load RAID-Z Keep ZFS pool % used < 90% (cow) like UFS, needs free space great @ streaming preformance opensolaris.org/os/communities/zfs solarisinternals.com/zfs #Evil tuning guide ---------------------------------------------------------------------- mdb -k ::memstat (cpu expensive) #don't need memtool with this ::memlist mdb -k Loading modules: [ unix krtld genunix specfs dtrace ufs pcisch ssd fcp fctl qlc md ip sctp usba s1394 nca lofs zfs random isp sd audiosup nfs logindmux ptm cpc sppp crypto wrsmd fcip ipc ] > $M MACRO NATIVE EQUIVALENT bufctl ::bufctl bufctl_audit ::bufctl -v cpu ::cpuinfo -v cpun ::cpuinfo -v cpus ::walk cpu |::cpuinfo -v devinfo ::print struct dev_info devinfo.minor ::minornodes devinfo.next ::walk devi_next |::devinfo -s devinfo.parent ::walk devinfo_parents |::devinfo -s devinfo.prop ::devinfo devinfo.sibling ::walk devinfo_siblings |::devinfo -s devinfo_brief ::devinfo -s devinfo_major ::devbindings -s devnames_major ::devnames -m devt ::devt devt2snode ::dev2snode findthreads ::walk thread |::thread major2snode ::major2snode mblk ::mblk -v modctl.brief ::modctl modules ::modinfo mount ::fsinfo msgbuf ::msgbuf mutex ::mutex panicbuf ::panicinfo pid2proc ::pid2proc |::print proc_t proc2u ::print proc_t p_user procargs ::print proc_t p_user.u_psargs queue ::queue -v sema ::print sema_impl_t stackregs ::stackregs stacktrace ::stackregs systemdump 0>pc;0>npc;nopanicdebug/W 1;:c thread ::print kthread_t threadlist ::threadlist -v u ::print user_t utsname utsname::print cpu$