The derelict building was last used as a Methodist chapel in 2015, developer plans say.
The wife of the victim was in another car and witnessed the crash in her rear-view mirror.
The Environment Agency says Adam Ivory ignored its letters about burning plastics and other items.
