Measurement Fundamentals & Hand Tools
A measurement is only as good as the instrument and the person taking it. This page covers the core concepts that separate a "rough check" from a valid inspection result, plus the hand tools every machinist uses daily — and how to use them correctly.
Accuracy vs Precision — The Classic Distinction
These two words are often confused, but they mean different things. A measurement system can be accurate but not precise, precise but not accurate, both, or neither.
| Concept | Definition | Question it answers |
|---|---|---|
| Accuracy | How close a measured value is to the true (reference) value | "Is the reading correct?" |
| Precision | How repeatable the measurement is when taken multiple times | "Do I get the same number every time?" |
| Resolution | Smallest increment the instrument can display | "Can I read 0.01 or 0.001?" |
| Repeatability | Same person, same instrument, same part, short time span — variation | "Can I reproduce my own measurement?" |
| Reproducibility | Different person, same instrument, same part — variation | "Does another inspector get the same number?" |
The 10:1 rule: your measuring instrument should be at least 10× more accurate than the tolerance being measured. If you're checking a ⌀25 ±0.01 mm feature (total tolerance 0.02), use an instrument with resolution of 0.002 mm or better. A caliper (0.01 mm resolution) is NOT accurate enough for ±0.01 work — use a micrometer.
Measurement Uncertainty
No measurement is exact. Every result has some uncertainty from the instrument, the environment, the operator, and the part itself. A properly reported measurement includes its uncertainty: 25.000 ±0.003 mm (k=2).
| Source of uncertainty | Typical cause |
|---|---|
| Instrument calibration | Gauge out of calibration, worn tips |
| Operator technique | Too much/too little pressure, inconsistent contact |
| Thermal expansion | Part at 25°C vs 20°C reference temperature — steel moves ~11.7 ppm/°C |
| Part surface | Rough surface gives different readings than polished |
| Contact geometry | Flat vs rounded tips on a curved surface |
Thermal growth matters: a 100 mm steel bar at 30°C (hand-warm) is about 0.012 mm longer than at 20°C. For tight tolerances, let parts cool to room temperature before measuring. Never handle precision parts with bare hands — use gloves or tweezers.
Calibration and Traceability
Calibration compares your instrument to a known reference (gauge block) and adjusts or records any error. Traceability means the reference itself is calibrated against a national standard (NIST in the US, PTB in Germany, etc.) — creating an unbroken chain back to a physical standard.
| Instrument | Typical calibration interval | Reference standard |
|---|---|---|
| Vernier caliper (150 mm) | 6–12 months | Cercian gauge blocks, grade 0 |
| Outside micrometer (0–25 mm) | 6–12 months | Gauge blocks, grade K |
| Dial indicator | 6 months | Criterion check / gauge block |
| Bore gauge | 6 months | Setting ring / master ring |
| CMM | Annual (plus daily check) | Laser interferometer, ball bar |
Hand Tools — When to Use Which
Vernier / Digital Caliper
The most common shop tool. Measures outside, inside, and depth. Resolution: 0.01 mm (0.0005 in). Best for: rough dimensions, non-critical features, stock size checks.
| Can measure | Accuracy | Don't use for |
|---|---|---|
| Outside length/width | ±0.02–0.03 mm | Tighter than ±0.03 mm |
| Inside diameter | ±0.03–0.05 mm | Fine bores |
| Depth | ±0.03 mm | Deep holes (flexes) |
Caliper technique: let the jaws close with light pressure — don't squeeze. Wipe the jaws clean. Zero the caliper before each use. Never measure a moving part. Never use a caliper as a pry bar or marking tool.
Outside Micrometer
Higher accuracy than a caliper. Measures outside dimensions only. Resolution: 0.01 mm (with vernier 0.001 mm). Best for: precision OD, shaft diameters, checking tight tolerances.
| Range | Typical accuracy | Use case |
|---|---|---|
| 0–25 mm | ±0.002 mm | Small shafts, pins |
| 25–50 mm | ±0.002 mm | Medium shafts |
| 50–75 mm | ±0.003 mm | Larger parts |
| 75–100 mm | ±0.003 mm | Large diameters |
Micrometer mistakes: (1) don't over-tighten — use the ratchet/ thimble slip, it clicks at the correct pressure. (2) Always zero at the closing point — 0–25 mm closes on the anvil, 25–50 mm uses the 25 mm standard. (3) Wipe part and anvil — a chip between faces gives a false reading.
Dial / Digital Indicator
Measures deviation from a reference, not absolute size. Mounted on a magnetic base or height stand. Resolution: 0.01 mm (digital) or 0.001 mm (dial). Best for: runout, flatness, alignment, comparing parts to a master, checking deflection.
Bore Gauge / Inside Micrometer
For measuring internal diameters accurately. Telescopic gauges + micrometer (transfer method), dial bore gauges (direct reading), or inside micrometer. Resolution: 0.001–0.01 mm. Best for: precision bores, hydraulic cylinders, bearing seats.
Height Gauge
Measures height from a surface plate. Digital height gauges resolve to 0.001 mm. Best for: checking hole positions, step heights, laying out part coordinates on a CMM or surface plate.
Gauge Blocks (Jo Blocks)
The reference standard. Precision-ground steel or ceramic blocks of known thickness, stacked to create any desired dimension. Grade 0 (highest accuracy) is used in the calibration lab; grade 2 for the shop. These are what your calipers and micrometers are checked against.
Which Tool for Which Tolerance?
| Tolerance | Recommended tool | Don't use |
|---|---|---|
| ±0.1 mm (rough) | Caliper, steel rule | — |
| ±0.05 mm | Caliper, outside micrometer | Steel rule |
| ±0.02 mm | Outside micrometer, bore gauge | Caliper (too loose) |
| ±0.01 mm | Outside micrometer with vernier, dial bore gauge | Caliper |
| ±0.005 mm or tighter | CMM, inside micrometer, length machine | Any hand tool |
Related: planning which instrument to use for a given feature — Measurement Planning Guide. Surface roughness measurement: Surface Finish Conversion.