You can find the "Custom" option on the /settings/theme page for your books, e.g. here:
https://leanpub.com/YOUR_BOOK/settings/theme
...making sure to replace YOUR_BOOK with your book's unique web address.
Please note, however, that the theme options available for a book depend on the author plan of the Primary Author of the book. We're sorry that's not clearer.
So, you will not see the Custom theme option, if the Primary Author of the book does not have a Standard or Pro account.
The Primary Author can be set here:
https://leanpub.com/YOUR_BOOK/authors
...but it can only be changed by the Primary Author themselves.
Please note that there are implications in our Terms of Service for who the Primary Author is. If you search for the term Primary Author in our Terms of Service, you can find the details: