Specifies the formatting of a BibTeX (.BIB) document; referenced by the LaTeX document and used by the BibTeX program "bibtex" when generating items for the bibliography; may also be used to create HTML output from a BibTeX file.
Custom BST files can be created using the latex makebst command.
Program(s) that open bst files
Mac: MacTeX; LyX/Mac Windows: proTeXt; XEmTeX; LaTeXEditor
Unix: LaTeX (often included with Unix); teTeX
Information from FileInfo.com - The File Extensions Resource.
http://www.fileinfo.com/extension/bst