How do they know the title of each book in the citadel library in Game of Thrones if the book's back faces the wall?

Having books with the spine facing out is a relatively recent innovation.

For the record, when you tuck a book away with the title-bearing spine pointing out, you’re carrying on a tradition that began roughly 480 years ago. “The first spine with printing dates from 1535, and it was then that books began to spin into the position we’re familiar with,” says Mari.

But before book, there were scrolls, and that’s where Mari’s story starts.

---

<sub>- Libraries Used to Chain Their Books to Shelves, With the Spines Hidden Away
- Smithsonian</sub>

Since Game of thrones is a fantasy and seems to be (comparatively) set in what would have been something close to Earth's "Middle Ages" with knights and kings all warring constantly this seems to be a pretty accurate depiction of a library.

As scrolls gave way to books, new shelves and a new organizational system were in order.

For the next fourteen hundred or so years, books, as Henry Petroski, a professor of civic engineering and history at Duke, writes in The Book on the Bookshelf, were shelved every which way but straight up, spine out. Engravings of private studies show books piled horizontally, standing on the edge opposite their spine (their fore edge), as well as turned fore edge out.

Before the printing press books were ornate constructions, and in comparison to what came after they were both highly valuable and in short supply.

In the Middle Ages, when monasteries were the closest equivalent to a public library, monks kept works in their carrels. To increase circulation, these works were eventually chained to inclined desks, or lecterns, thus giving ownership of a work to a particular lectern rather than a particular monk.

When space got tight the monks moved their books to shelves, but they stacked them with the spines hidden. Which, as you can imagine, would have been quite confusing. The solution, Mari says: “Sometimes an identifying design was drawn across the thick of the pages.”

---

<sub>- Libraries Used to Chain Their Books to Shelves, With the Spines Hidden Away
- Smithsonian</sub>

So, to answer your question about how they were identified: there were sometimes identifying markings on "The thick of the pages".