We recommend that for images in your book (as opposed to the cover image) you use images set to a resolution of 72 pixels per inch (PPI) and that you use the JPG format set to 80% compression. These settings should help keep the size of your book files smaller.
The maximum image size for your book depends on the book style you have chosen. You can check your book style and the page size by clicking on "Writing" in your book tools and selecting the "Formatting" option. Here is a table setting out the maximum image sizes for the different page sizes, based on an image resolution of 72 PPI: here. For information about the the three default book stles, please follow this link.
For images that are drawings, like line diagrams, PNG will look better than JPEG, and we recommend you set the image resolution to 150 PPI or 300 PPI.
Note that we have a whole section in each sample book about images. See the sample content for each Leanpub book type at these links for business, fiction and technical books.