We are commonly advised to install new strings before setup; I don't especially agree*, as long as the strings are in good condition. If you are using new strings, be sure they are well settled in, particularly on classical guitars - the strings will continue to stretch. Any additional string length that is subsequently pulled through the nut will decrease both the mass and diameter of the remaining playing portion of the string, effecting compensation. I find that steel strings tend to break at the tuning machine when repeatedly tightened and loosened, so if you have a lot of adjusting to do, it's good to do the work before installing new strings.
One big difference in setup procedure when preparing for a compensated nut is to compensate the saddle for the 12th or 14th fret to be in tune with the 2nd fret – not with the open string! This method results in the saddle being well adjusted to all the positions of the fretboard, rather than to a likely-out-of-tune open string. Open string out of tune? Yes - think about it - once the saddle is adjusted to the fretboard, it makes more sense to think of the open string, not the fretted notes, as being out of tune. (Later, we will correct the open string to first fret, with nut compensation.)
I very strongly recommend the use of feeler gauges for setting the nut height, and an appropriate scale or gauge for setting the action on each string at the 12th or 14th fret. I know that there are some experienced luthiers that can make these settings by eye and feel, but I’ve seen others who think they can, and can’t. Amateurs should always use gauges; it will save you a lot of grief and provide a much better result.
* See luthier Anders Sterner's comments regarding intonation with old strings: https://www.sternercapo.se/Compensation/Read-Eng/old_strings.htm.
Before attempting to compensate the nut, the other elements should be adjusted as well as possible. Adjustments should be made, or checked, in the sequence that they are presented below, but keep in mind that if adjustments are significant, an iterative process may be used, making partial adjustments all around, and then returning for more refined adjustments.
This is the sequence of setup adjustments:
The truss should be adjusted as well as possible before making any other adjustments, because it greatly alters the action height. If you were to leave too much relief (truss too loose) and then adjust the saddle for correct action height, you would take too much material off the saddle, and perhaps, off the nut slots. Subsequent tightening of the truss to correct the relief adjustment would result in the action being too low.
Looking down the neck to see the curvature may be ok for preliminary assessment of a guitar, but it is not an accurate measurement of relief. You should, however, look for and note, any irregularities in the curve. Also, observe carefully what happens to the curve at the point where the neck is joins the body, and beyond.
Imagine the arc of a plucked open string, where the maximum swing will be at the 12th fret, somewhat matching the relief of the neck. But, don’t forget to also picture the result of playing on the 5th or higher fret. In that case, the maximum swing will occur 12 frets above where you are playing - ask yourself whether the shape of the neck at that point could be a problem.
A good way to measure the relief is to place a capo on the first fret, press the string down at about the 14th fret, and with an automotive feeler gauge, measure the distance from the top of the 7th or 8th fret to the bottom of the string.
If you want to use a scale instead, to check neck relief, be sure the scale does not flex. (For instance, a stainless steel printer’s scale is no good for this.)
Check the height and fret number recommendations from your guitar maker and from trusted luthiers.
Stewart Macdonald’s Dan Erlewine recommends a nearly flat (.002” relief) adjustment, and to increase that if needed to eliminate buzz. This works well if your neck is really straight, but not so well if you have irregularities. Relief settings of about .008” to .012” are more common. Check your manufacturers recommendations.
Check for string buzz. If you think there could be portions of the neck with no relief, or negative relief, use a short scale to check it out. Check for frets not seated properly. If there is still a problem, you may have to loosen the truss for more relief, or consider some fret dressing.
Subsequent lowering of the action will slightly decrease the tension on the truss, producing a little more relief, so check again after action adjustments.
The purpose of this section is set the nut action well enough that the saddle action can then be accurately set. If the action at the nut has already been set fairly low, then you can proceed to the saddle.
The following method of setting nut action is not affected by the saddle height. (It can also be used to prevent back buzz on acoustic guitars.)
To set the nut height, press the string down at the 2nd fret, then observe the height of the string over the first fret. The gap should be very tiny – just enough so that the string will not vibrate against the fret. File the nut slots to achieve this height. If the nut action is already at least that low, then proceed to saddle adjustment.
Be sure to file the nut slots with a downward angle, so the string is held firmly to the slot, otherwise buzzing will result. I file at a straight angle a bit less than the headstock angle, and then round it in the back, to become parallel to the headstock.
Nut slot files are a must for professionals, but they’re quite expensive for a player to be used on just a few instruments. I started out decades ago on my first $20 Mexican guitar with an old paring knife and a razor blade, but you can do much better than that! See Tools, Tips, and Tricks.
Saddle String height and compensation
This is so easy on most electric guitars! Acoustic guitars are much more difficult, because you are concerned with both height and compensation adjustment at the same time, on one piece of material. Not so difficult for a professional, but for a player, it takes a lot of patience and trial-and-error! My suggestion is to get your compensation adjustment close while you still have a little extra string height to work with. Check for buzz as you go.
As mentioned before, the objective is to compensate the saddle to work with the fretboard – not with the uncompensated open string. The way to do it is to compare the note at the 2nd fret to the octave at the 14th. See Measuring the Intonation Difference Between Two Notes.
Unless you are a professional with a proper setup, hold the instrument in playing position when making the pitch measurements, in order to insure that the neck is not stressed, and that you are fretting with your normal force.
Note: Unless the curve looks really good, I advise not testing at a fret above where the neck joins the body, because the fretboard may deviate from the curve of the neck. Therefore, a guitar with 12 frets to the body probably should be tested comparing fret 2 to fret 12. It doesn’t need to be an octave - just adjust until the notes are in tune with each other. For the same reason, the action measurement would be best at the 12th fret.
For electric players that use the whole fretboard, it may be best to intonate the treble saddles using fret 5 - 17 or thereabouts. It would think it a good idea to do the basses differently from the trebles; how high do you normally play on the bases?