It is not only courtesy, it is the law.
http://www.oldbridgefirstaid.com/BlueLight.html
I'd look up the actual law based on the statute provided, but I'm lazy.
Ok, I found it.
"NJSA 39:3-54.12
39:3-54.12 Rights of motor vehicle with emergency lights in operation.
6. Nothing contained herein is intended to grant to any member of a volunteer fire company, a volunteer first aid or rescue squad or a volunteer Office of Emergency Management any privileges or exemptions denied to the drivers of other vehicles, and such members operating emergency warning lights shall drive with due regard for the safety of all persons and shall obey all the traffic laws of this State including R.S.39:4-81, provided, however,
that the drivers of non-emergency vehicles upon any highway shall yield the right of way to the vehicle of any member of a volunteer fire company, a volunteer first aid or rescue squad or a volunteer Office of Emergency Management operating emergency warning lights in the same manner as is provided for authorized emergency vehicles pursuant to R.S.39:4-92."