I pulled down some thick old wallpaper off the walls in the livingroom.
What I was left with was a papery surface with some spots where it
tore off with the wallpaper, some kind of wall/joint compound
underneath, and then the actual drywall under that which is intact. Is
this papery substance adhesive residue or is it something that peeled
off the back of the thick wallpaper and stayed as a souvenier???

I'm not quite sure how to handle this paper substance that's left.
I've gone over it with sandpaper and putty knives so all the parts that
were loose have flaked off. Does it all need to be removed or can I
seal it all with something like a oil based primer or Gardz and then
skim coat over that? Or should I just skim over it as is and then seal
ontop of the skim coat?

I'm thinking about the following...

Patch holes and let dry
oil based primer or Gardz
skim coat
Gardz
primer
paint

How does that sound?