To display images in Phoca Gallery the same way like on this page: www.phoca.cz/joomla3demo/ follow this guide:
Install Phoca Gallery and set thumbnails size in Options of Phoca Gallery (Options - Thumbnails):
Large Image Width: 640
Large Image Height: 480
Medium Image Width: 224
Medium Image Height: 150
Small Image Width: 112
Small Image Height: 75
Detail Window Width: 680
Detail Window Height: 560
Of course, these values are set on demo page and can be different for your site (set by your needs).
Set this parameter (Options - Thumbnails):
Additional Thumbnails: Yes
Add images to gallery and thumbnails will be automatically created.
Categories View
Mosaic Effect (Categories View): Yes
Category View
Display Name: Display
Display Detail Icon: Hide
Display Image Description In Box: Yes (HTML)
Different Thubmnail Height: Yes (Masonry Effect)