I am creating a tracking spreadsheet for vacation time (earned every
paycheck). Our company allows for negative balances. Below is the formula
that I am using:

=TEXT(IF((E9+C11-D11)A11,A11,( E9+C11-D11)),"[H]:MM")

It errors on the next paycheck if the result is negative.

Negative numbers cause display errors when formatted as times.

You can obtain the textual rendition that you want of negative times by
using a formula like

=IF(A12<0,"-","")&TEXT(ABS(A12),"[h]:mm")

assuming your result is in A12.