Uncertainty about the bookseller's future and lagging Nook sales are keeping investors on edge