For WinHelp, to make a topic pop up, assign property PopOver to the hotspot format:
[HelpStyles] PopCharFmt = PopOver
See §17.8 Creating jumps and pop-ups for WinHelp.
If your HTML-based Help system has a DITA2Go-generated browse sequence, to avoid including pop-up topics in the browse sequence you must declare these topics to be extracts instead of splits; see §27.4 Extracting files.
For HTML Help, all you get is plain-text pop-ups, unless you use a third-party tool. To create a pop-up link in HTML Help, put the entire pop-up content (plain text only) in a DITA hypertext alert PI marker embedded in the hotspot.
See §18.5 Creating pop-ups for HTML Help.
OmniHelp, JavaHelp, Oracle Help for java
For OmniHelp, JavaHelp, and Oracle Help for Java, you can use any HTML in pop-up topics, including graphics and jumps. To make a topic pop up, assign reserved window name popup to the hotspot format:
[SecWindows] PopCharFmt = popup