The best replacement for bibtex is biblatex.

The best alternatives to bibtex are rated as following:
★★★★★  biblatex
★★★  endnote
★★  biber
★★  jabref
★★  bibtex8
★  natbib
  bibdesk

Your alternative for bibtex is not listed? Suggest!
Speaking about bibtex, one probably shouldn't forget about latex.

Other search terms related to bibtex and their ratings are:
★★★★★ latex
★★★★ texshop
★★★★ beamer
★★★★ pdflatex
★★★★ lyx
★★★ texmaker
★★ miktex
★★ cite
★★ zotero
★ mendeley
★ revtex
You have in mind a companion for bibtex and it is not yet here? Propose!



Google Trends

Scale is based on the average worldwide traffic of bibtex in all years. Click on graph for more.
bibtex
biblatex
endnote