Chapter 7. Going Beyond the Basics

Introduction

Haveyoueverwonderedwhatmodificationsawebormail

administratormakestoherservers?Maybeyou'recurious

aboutwhatpoliciesotheradministratorsusetoimplement

bandwidthcontrol?Howdobusyadministratorsmanagethelog

datafromaserverfarm?

Perhapsyou'vecontemplatedusingtheExpectscripting

language.However,there'sagoodchanceyou'veneverthought

ofusingeesh,atotallyundocumentedbutusefulscripting

utility.

Thischapteralsoincludestwohacksontheemergencyrepair

process,asmanyusersprefertohopethatthey'llneverneed

anemergencyrepairkit.Instead,learntoovercomeyourfear

oftheinevitableandmastertheartofrepairingbeforethe

emergency.



Hack69TuneFreeBSDforDifferentApplications



KnowhowtotuneandwhattotuneonyourFreeBSD

system

Asanadministrator,youwanttotuneyourserversystemsso

theyworkatpeakefficiency.Howdoyouknowwhattotune?

Theanswerdependsheavilyuponthesystem'sfunction.Will

thesystemperformalotofsmallnetworktransactions?Willit

performasmallnumberoflargetransactions?Howwilldisk

operationsfactorin?

Howyouanswertheseandotherquestionsdetermineswhat

youneedtodotoimprovetheperformanceofyoursystems.

Thishackstartswithgeneraloptimizationsandthenlooksat

function-specifictunables.



7.2.1OptimizingSoftwareCompiling

Agoodplacetostartiswithsoftwarecompiling,asyouwantto

compilesoftwareandupdatesasefficientlyaspossible.

Wheneveryoucompile,yourcompilermakesassumptions

aboutyourhardwareinordertocreatebinaries.Ifyouhavean

x86-compliantCPU,forexample,yourcompilerwillcreate

binariesthatcanrunonanyCPUfroma386onward.Whilethis

allowsportability,itwon'ttakeadvantageofanynewabilitiesof

yourCPU,suchastheextendedMMX,SSE,SSE2,or3DNow!

instructionsets.Thisisalsowhyusingprecompiledbinarieson

yoursystemisasurefirewaytoreduceyouroverall

performance.

Toensurethatsoftwarewillbecompiledefficiently,updateyour



compilerflagsin/etc/make.conf.Thisfiledoesnotexiston

newsystems,butyoucancopyitfrom

/usr/share/examples/etc/defaults/make.conf.

StartbyeditingtheCPUTYPE=linetoreflectyourCPUtype;

you'llfindsupportedtypeslistedascommentsjustbeforethis

line.WhilethiswilltakeadvantageofyourCPU'sfeatures,the

disadvantageisthatyourcompiledbinariesmaynotrunon

differentCPUtypes.However,ifallofyoursystemsrunthe

sameCPUplatform,anyoptimizationsyoumaketoshared

binarieswillaffectallofyoursystemsequallywell.

Next,changetheCFLAGSlinetoCFLAGS=-O2-pipefunroll-loops.The-pipeoptioncansignificantlydecreasethe

amountoftimeittakestocompilesoftware,byusingpipesto

communicatebetweencompilerprocessesinsteadoftemporary

files,butattheexpenseofusingslightlymorememory.Thefunroll-loopssavesoneCPUregisterthatwouldotherwisebe

tiedupintrackingtheiterationoftheloop,butattheexpense

ofmakingaslightlylargerbinary.



Themake.conffilealsocontainsalineforCXXFLAGS.Theseoptionsare

similartotheCFLAGSoptionsbutapplytoC++code.



7.2.2KernelOptimizations

Inyourkernelconfiguration,addthefollowinglineafterthe

machinei386line:



makeoptionsCOPTFLAGS="-O2-pipe-funroll-loops-ffast-math"



ThisissimilartotheCLAGSoptionin/etc/make.conf,except

thatitoptimizeskernelcompilation.



See[Hack#54]forinstructionsonhowtostripandcompileakernel.



Youcanalsoaddthisline:

TOP_TABLE_SIZE=number



wherenumberisaprimenumberthatisatleasttwicethe

numberoflinesin/etc/passwd.Thisstatementsetsthesizeof

thehashthattopuses.

SetthefollowingoptionifyouhaveanAMDK5/K6/K6-2or

Cyrix6x86chip.ItenablescachewriteallocationfortheL1

cache,whichisdisabledbydefaultforthesechips.

optionsCPU_WT_ALLOC



ThisoptionwilldisableNFSservercode,soincludeitwhenyou

knowthatyouwillnotbeactingasanNFSserver:

optionsNFS_NOSERVER



Anotherwayofsavingkernelmemoryistodefinethemaximum

numberofswapdevices,asshowninthenextexample.Your

kernelneedstoallocateafixedamountofbitmappedmemory



sothatitcaninterleaveswapdevices.Isetthenumberto1on

myworkstationand2onmyservers.IfIneedtoaddmoretoa

server,Icaneasilycreateanotherpartition.

optionsNSWAPDEV=number



Ifyouplanoncompilingallyourrequisitesintothekernel(NIC

driver,IPF/IPFW,etc.)andwon'tbeloadinganyoftheseoptions

asmodules,youcanincludethislinetoskipmodulecompiling.

Thissavessignificantlyonthetimetakentocompileakernel

(sometimesreducingitbytwo-thirds).

makeoptionsMODULES_OVERRIDE=""



Bydefault,allkerneloptionsarecompiledasmodules.This

allowsyoutousekldloadtoloadamoduleeventhoughitisn't

specifiedinyourkernelconfigurationfile.

TheadvantageofMODULES_OVERRIDEisthedecreaseinkernel

compilationtime.Thedisadvantageisthatyou'llneedto

recompileyourkernelifyoueverneedtoaddadditional

functionality,sinceyouwillhavelosttheabilitytoloadthe

kernelmoduleseparately.



7.2.3OptimizingNetworkPerformance

Mostmodernnetworkcardsandswitchessupporttheabilityto

auto-negotiatethecommunicationspeed.Whilethisreduces

administration,itcomesatthecostofnetworkthroughput.If

yourswitch,server,orworkstationissettouseautonegotiation,itwillstoptransferringnetworktrafficeveryfew

momentstorenegotiateitsspeed.



Ifyournetworkdriversupportsit,youcansetnetworkspeed

withifconfigatruntimeorin/etc/rc.confatboottime.Hereis

anexample:

%grepfxp0/etc/rc.conf



ifconfig_fxp0="inetx.x.x.xnetmaskx.x.x.xmedia100BaseTXmed

full-duplex"



ReadthemanpageforyourNICdrivertoseewhetheritsupports

mediaopt.Forexample,ifyourNICisrl0,readman4rl.



Next,youcanenableDEVICE_POLLINGinyourkernel,which

changesthemethodbywhichdatatravelsfromyournetwork

cardtothekernel.Withoutthissetting,frequentinterruptcalls

mayneverfreethekernel.Thisisknownaslivelockandcan

leaveyourmachineunresponsive.Thoseofusunfortunate

enoughtobeonthewrongsideofcertaindenial-of-service

attacksknowaboutthis.

TheDEVICE_POLLINGoptioncausesthekerneltopollthe

networkcardatcertainpredefinedtimes,duringidleloops,or

onclockinterrupts.Thisallowsthekerneltodecidewhenitis

mostefficienttopolladeviceforupdatesandforhowlong,and

ultimatelyresultsinasignificantincreaseinperformance.

TotakeadvantageofDEVICE_POLLING,youneedtocompiletwo

optionsintoyourkernel:optionsDEVICE_POLLINGand

optionsHZ=1000.Thelatteroptionslowstheclockinterrupts

to1,000timespersecond,whichpreventsthekernelfrom



pollingtoooften.

Onceyou'verecompiledyourkernel,you'llstillneedtoenable

thefeature.Addthislineto/etc/sysctl.conf:

kern.polling.enable=1



TheDEVICE_POLLINGoptiondoesnotworkwithSMP-enabled

kernelsbydefault.IfyouarecompilinganSMPkernelwith

DEVICE_POLLING,firstremovethefollowinglinesfrom

/usr/src/sys/kern/kern_poll.c:

#ifdefSMP

#include"opt_lint.h"

#ifndefCOMPILING_LINT

#errorDEVICE_POLLINGisnotcompatiblewithSMP

#endif

#endif



7.2.4OptimizingMailServers

Mailserverstypicallyhaveaverylargenumberofnetwork

connections,duringwhichtheytransferasmallamountofdata

forashortperiodoftimebeforeclosingtheconnection.Inthis

case,itisusefultohavealargenumberofsmallnetwork

buffers.



Networkconnectionshavetwobuffers,oneforsendingandone

forreceiving.Thesizeofthebufferdictateshowquicklydata

willfunnelthroughthenetworkand,intheeventofanetwork

delay,howmuchdatacanbackuptheserverforthat

connectionbeforethereisaproblem.Havinganetworkbuffer

thatistoosmallwillcauseadatabacklogastheCPUwaitsfor

thenetworktoclear,whichcausesgreaterCPUoverhead.

Havinganetworkbufferthatistoolargewastesmemoryby

usingthebufferinefficiently.Findingabalanceisthekeyto

tuning.

Ifindthatmultiplyingthenumberofestablishedconnectionsby

32leavesmewithroomtobreatheintheeventthatIseean

abnormallyhighsurgeoftraffic.I'vecometothisnumberover

timethroughtrialanderror.So,ifyouexpecttohaveapeakof

128serverssendingyoumail,having8,192networkbuffer

clusterswouldbegood(1282perconnection32).Also,

rememberthatconnectionscantakeuptotwofullminutesor

moretoclosecompletely.Ifyouexpectmorethan128emails

inanygiventwo-minuteperiod,increasethenumber

accordingly.

Anotherimportantvaluetocontrolisthemaximumnumberof

sockets.Startwiththesamenumberofsocketsasthereare

networkbuffers,andthentuneasappropriate.

Youcanfindouthowmanynetworkbufferclustersareinuse

withthecommandnetstat-m.Youcanspecifythevaluesyou

wantin/boot/loader.conf.Forexample:

kern.ipc.nmbclusters=8192

kern.ipc.maxsockets=8192



Aswithanyperformancetuning,monitoryoursystemafter

makingchanges.Didyougooverboardorunderestimatewhat



youwouldneed?Alwayscheckandadjustaccordingly.



7.2.5OptimizingFileServers

Fileserversgenerallyhavelonger-livedandlessfrequent

networkconnectionsthanthoseonmailservers.Theyusually

transferlargerfiles.

Todeterminetheoptimalnumberofnetworkbufferclusters,

considerhowmanyclientsyouhave.Multiplyingthenumberof

networkbuffersbytwoisgoodpractice,thoughsomeadmins

prefertomultiplybyfourtoaccommodatemultiplefile

transfers.Ifyouhave128clientsconnectingtothefileserver,

setthenumberofnetworkbufferclustersto1,024(1282per

connection4).



7.2.6OptimizingWebServers

Ifyouhavemorethanoneelementonyourwebpage(for

example,multipleimagesorframes),expectwebbrowsersto

makemultipleconnectionstoyourwebserver.It'scommonto

seefourconnectionsperpageserved.Alsocountanydatabase

ornetworkconnectionsmadeinserver-sidescripting.

Webserversgothroughperiodsofhighsandlows.Whileyou

mightserve100pagesperminuteonaverage,atyourlowyou

mightserve10pagesperminuteandatpeakover1,000pages

perminute.Atapeakof1,000pagesperminute,yourclusters

andsocketsshouldbearound16,384(1,000pages2per

connection4connections2forgrowth).



7.2.7SeeAlso



mantuning

mangcc(theGCCmanpage,whichexplainsCPUcompiling

optimizations)

manifconfig

"TuningFreeBSDfordifferentapplications"

(http://silverwraith.com/papers/freebsd-tuning.php)

"OptimizingFreeBSDanditskernel"

(http://silverwraith.com/papers/freebsd-kernel.php)

NotesontuningApacheserversat

http://www.bolthole.com/uuala/webtuning.txt



Hack70TrafficShapingonFreeBSD



Allocatebandwidthforcrucialservices.

Ifyou'refamiliarwithyournetworktraffic,youknowthatit's

possibleforsomesystemsorservicestousemorethantheir

fairshareofbandwidth,whichcanleadtonetworkcongestion.

Afterall,youhaveonlysomuchbandwidthtoworkwith.

FreeBSD'sdummynetmayprovideaviablemethodofgettingthe

mostoutofyournetwork,bysharingbandwidthbetween

departmentsorusersorbypreventingsomeservicesfrom

usingupallyourbandwidth.Itdoessobylimitingthespeedof

certaintransfersonyournetworkalsocalledtrafficshaping.



7.3.1ConfiguringYourKernelforTrafficShaping

Totakeadvantageofthetrafficshapingfunctionalityofyour

FreeBSDsystem,youneedakernelwiththefollowingoptions:

optionsIPFIREWALL

optionsDUMMYNET

optionsHZ=1000



dummynetdoesnotrequiretheHZoption,butitsmanpage

stronglyrecommendsit.See[Hack#69]formoreaboutHZ

and[Hack#54]fordetailedinstructionsaboutcompilinga

customkernel.