The format WinHelp likes best is WMF, which can contain both vector and bitmap elements. For screenshots, use the same resolution at which they were taken, 96DPI; more or less will result in unreadable text.
For non-Windows WinHelp, created via Altura on Macintosh or Bristol HyperHelp on UNIX, only BMP is supported; see §17.2.4 Accommodating platform differences.