What is the ruling for granting a monetary prize and an award of merit to a person who insults and vilifies prophet Mohammed [pbuh] and Islam? In his published books he: 

- describes Islam as a lie;
- portrays prophet Mohammed as a fabricator;
- claims that Abu Talib fabricated the revelation and Mohammed's prophethood to seize the leadership of Quraish and Mecca from the Umayyads;
- claims that Abu Talib sought help from the Jews to propagate Mohammed's prophethood.

Is it permissible for the [prize award] committee to honor such a person by offering him an award of merit and a monetary prize from the Muslim funds which will ultimately elevate his status and promulgate his ideas among the masses? And if this is impermissible, who is to pay back the prize money that was squandered from public funds?
[Please note that] the committee is aware of the contents of his books.

It is a fact in Islamic law that venerating the Shari'ah [En. Islamic law], religion and the status of the Prophet [PBUH] are among the greatest pillars of Islam and one of its most important duties.

The following verses from the Qur'an illustrate this:
Whoever honors the symbols of Allah - indeed, it is from the piety of hearts [Al-Hajj, 32].

Indeed, We have sent you as a witness, a bringer of good tidings and a warner. That you [people] may believe in Allah and His Messenger and honor him, respect the Prophet and exalt Allah morning and afternoon [Al-Fath, 8-9].
Indeed, those who abuse Allah and His Messenger - Allah has cursed them in this world and the Hereafter and prepared for them a humiliating punishment [Al-Ahzab, 57].

[People], do not call upon the Messenger's as you call one another—God is well aware of those of you who steal away surreptiously—and those who go against His order should beware lest a trial afflict them or they receive a painful punishment [Al-Nur,63].

There is a consensus among Muslims that defaming Islam and insulting the prophet puts the perpetrator beyond the pale of Islam and the community. Such a person is blameworthy in this life and deserves punishment in the Hereafter.

Article 98 (w) of the penal law criminalizes slandering heavenly religions or any of their sects and harming national unity or social peace.

In reference to the question, the texts mentioned in the question are blasphemous and put their writer beyond the pale of Islam if he is a Muslim. If they are confirmed, these erroneous and vile claims are worthy of being criminalized and not honored. It is obligatory to take all the legal and penal measures against such a person to prevent his evil from spreading. [It is important] to set him up as an example for whoever is lured by Satan into wrongdoing.

Allah the Almighty said:
Say, [O Muhammad], "Shall we inform you of the greatest losers as to [their] deeds?

[They are] those whose effort is lost in worldly life, while they think that they are doing well in work [Al-Kahf, 103-104].
If the prize award committee is aware of the contents of his books, then it must give back the value of the prize which taken from Muslim funds.

And Allah knows best.