We are commonly advised to install new strings before setup; I don't especially agree*, as long as the strings are in good condition. If you are using new strings, be sure they are well settled in, particularly on classical guitars - the strings will continue to stretch. Any additional string length that is subsequently pulled through the nut will decrease both the mass and diameter of the remaining playing portion of the string, effecting compensation. I find that steel strings tend to break at the tuning machine when repeatedly tightened and loosened, so if you have a lot of adjusting to do, it's good to do the work before installing new strings.
One big difference in setup procedure when preparing for a compensated nut is to compensate the saddle for the 12th or 14th fret to be in tune with the 2nd fret – not with the open string! This method results in the saddle being well adjusted to all the positions of the fretboard, rather than to a likely-out-of-tune open string. Open string out of tune? Yes - think about it - once the saddle is adjusted to the fretboard, it makes more sense to think of the open string, not the fretted notes, as being out of tune. (Later, we will correct the open string to first fret, with nut compensation.)
I very strongly recommend the use of feeler gauges for setting the nut height, and an appropriate scale or gauge for setting the action on each string at the 12th or 14th fret. I know that there are some experienced luthiers that can make these settings by eye and feel, but amateurs should always use gauges; it will save you a lot of grief and provide a much better result.
Before attempting to compensate the nut, the other elements should be adjusted as well as possible. Adjustments should be made, or checked, in the sequence that they are presented below, but keep in mind that if adjustments are significant, an iterative process may be used, making partial adjustments all around, and then returning for more refined adjustments.
This is the sequence of setup adjustments:
The truss should be adjusted as well as possible before making any other adjustments, because it greatly alters the action height. If you were to leave too much relief (truss too loose) and then adjust the saddle for correct action height, you would take too much material off the saddle, and perhaps, off the nut slots. Subsequent tightening of the truss to correct the relief adjustment would result in the action being too low.
Looking down the neck to see the curvature may be ok for preliminary assessment of a guitar, but it is not an accurate measurement of relief. You should, however, look for and note, any irregularities in the curve. Also, observe carefully what happens to the curve at the point where the neck is joins the body, and beyond.
Imagine the arc of a plucked open string, where the maximum swing will be at the 12th fret, somewhat matching the relief of the neck. But, don’t forget to also picture the result of playing on the 4th or higher fret. In that case, the maximum swing will occur 12 frets above where you are playing - ask yourself whether the shape of the neck at that point could be a problem.
A good way to measure the relief is to place a capo on the first fret, press the string down at about the 12th or 14th fret, and with an automotive feeler gauge, measure the distance from the top of the 7th or 8th fret to the bottom of the string.
If you want to use a scale (ruler) instead, to check neck relief, be sure the scale does not flex. (For instance, a stainless steel printer’s scale is too flexable for this.)
Check the height and fret number recommendations from your guitar maker and from trusted luthiers.
Stewart Macdonald’s Dan Erlewine recommends a nearly flat (.002” relief) adjustment, and to increase that if needed to eliminate buzz. This works well if your neck is really straight, but not so well if you have irregularities. Relief settings of about .005” to .012” are more common. I recommend no more than .010".
Check for string buzz. If you think there could be portions of the neck with no relief, or negative relief, use a short scale to check it out. Check for frets not seated properly. If there is still a problem, you may have to loosen the truss for more relief, or consider some fret dressing.
Subsequent lowering of the action will slightly decrease the tension on the truss, producing a little more relief, so check again after action adjustments.
The purpose of this section is to set the nut action well enough that the saddle action can then be accurately set. If the action at the nut has already been set fairly low, then you can proceed to the saddle.
There are two ways to set the nut height before (or after) having set the bridge saddle height. I prefer this first method, which sets the nut string height only slightly above the plane of the fret tops, providing clean, close action.
Press the string down at the first fret and measure the string gap above the second fret. Use this measurement to set the string height above the first, adding .001” - .002” to allow for wear. If you raise the saddle later, it will appropriately raise the nut action by a tiny amount. Likewise, if you lower the saddle height, it will appropriately lower the nut action, leaving it slightly above the plane of the fret tops.
This second method of setting nut action is not affected by the saddle height, and is used by those who prefer higher nut action, and to prevent back buzz on acoustic guitars.
To set the nut height, press the string down at the 2nd fret, then observe the height of the string over the first fret. The gap should be enough so that the string will not vibrate against the first fret. File the nut slots to achieve this height. If the nut action is already at least that low, then proceed to saddle adjustment. Otherwise, if there is much gap over the 1st fret, then file the fret slot until the gap is tiny, or to your preferred height.
Be sure to file the nut slots with a downward angle, so the string is held firmly to the slot, otherwise buzzing will result. I file at a straight angle a bit less than the headstock angle, and then round it in the back, to become parallel to the headstock.
Nut slot files are a must for professionals, but they’re quite expensive for a player to be used on just a few instruments. I started out decades ago on my first $20 Mexican guitar with an old paring knife, razor blade, and sandpaper, but you can do much better than that! See Tools, Tips, and Tricks.
Saddle String height and compensation
This is so easy on most electric guitars! Acoustic guitars are much more difficult, because you are concerned with both height and compensation adjustment at the same time, on one piece of material. Not so difficult for a professional, but for a player, it takes a lot of patience and trial-and-error! My suggestion is to get your compensation adjustment close while you still have a little extra string height to work with. Check for buzz as you go.
As mentioned before, the objective is to compensate the saddle to work with the fretboard – not with the uncompensated open string. The way to do it is to compare the note at the 2nd fret to the octave at the 14th. See Measuring the Intonation Difference Between Two Notes.
Unless you are a professional with a proper setup, hold the instrument in playing position when making the pitch measurements, in order to insure that the neck is not stressed, and that you are fretting with your normal force.
Note: Unless the curve looks really good, I advise not testing at a fret above where the neck joins the body, because the fretboard may deviate from the curve of the neck. Therefore, a guitar with 12 frets to the body probably should be tested comparing fret 2 to fret 12. It doesn’t need to be an octave - just adjust until the notes are in tune with each other. For the same reason, the action measurement would be best at the 12th fret.
For electric players that use the whole fretboard, it may be best to intonate the treble saddles using fret 5 - 17 or thereabouts. It would think it a good idea to do the basses differently from the trebles; how high up do you normally play on the bases?