If you have all of those pencils, go back to the room that is a backwards L that's immediately to the left of the room that you found the sword pencil in. Go down and to the left and you'll find a door that needs your key pencil to unlock and go from there!