In order to tell what books are currently downloaded and available for offline listening, first go to the library section of the app. Then click the filter display below the selected book.
This will display the filter menu. At the bottom of the menu, there is a toggle to display either all books or only currently downloaded content. When toggled to the right (green), the library will only display content that has been downloaded to the device. Keep in mind that this will apply no matter which filter you have selected (All Books, Unplayed Books, Unfinished Books), so make sure to turn it back off when you wish to browse your entire library.