I am wondering when then site gets slammed with Panda algorithm and traffic drops.Is it possible to rebuild authority of the site and gain old traffic back?
It is possible indeed. Get backlinks from good pr websites though relevant. Instead of no. go with quality. Even the no follow links are useful, if relevant.