This is a sanding problem. A conditionaer/wash/sealer might help a bit
but sanding them out is the only real solution.

You can use a rag with mineral spirits to wipe the raw boards and see
what they'll look like finished, always a good step before staining. As
you sand out the bumps, you can check your progress by wiping again.