Bismillah, Alhamdulillah,
Zahhaad, Allah (subhanehu wata'alaa) requires us to be just and carefull when we Judge - and ignoring this, to judge based on "What I seem to remember hearing is" is very disturbing. If you disgree with his Aqidah, that is one thing but to invoke apostacy based on hearsay is very dangarous. Allah (subhanehu wa ta'alaa) warns us:
"follow not that whereof thou hast no knowledge. Lo! the HEARING and the sight and the heart of each of these it will be asked." - Surah Al-Isra, 36.
"O ye who believe ! Be ye staunch in justice, witnesses for Allah, even though it be against yourselves or (your) parents or (your) kindred, whether (the case be of) a rich man or a poor man, for Allah is nearer unto both (than ye are) . So follow not passion lest ye lapse ( from truth ) and if ye lapse or fall away, then lo! Allah is ever Informed of what ye do."- Surah An-Nisa' 135.
Cotrary to what you believe Al-Hafidh Ibn Hajar is aware of the accusations and mentions them in most of his books including Fath Al Bari; and since he is master of jarh Wa Ta'adeel who is relied upon, and would not take things lightly (as he is not known for Tasahul) as you assume:
Ibn Hajar said, "those of his stances that were rejected from him were not said by him due to mere whims and desires and neither did he obstinately and deliberately persist in them after the evidence was established against him. Her e are his works overflowing with refutations of those who held to tajseem yet despite this he is a man who makes mistakes and is also correct. So that which he is correct in and that is the majority is to benefited from and Allaahs Mercy should be sought for him due to it, and that which he is incorrect in should not be blindly followed. Indeed he is excused for his mistakes because he is one of the Imaams of his time and it has been witnessed that he fulfilled the conditions of ijtihaad
From the astonishing qualities of this man was that he was the strongest amongst men against the People of Innovation, the Rawaafidah, and the Hululiyyah, and the Ittihaadiyyah, and his works on this are many and famous, and his fata awaa on them cannot be counted, so how the eyes of these innovators must have found joy when they heard those who declared him to be a kaafir! And how delighted they must have been when they saw those who did not declare him to be a kaafir in turn being labelled kaafir! It is obligatory upon the one who has donned the robe of knowledge and possesses intelligence that he consider the words of a man based upon his well-known books or from the tongues of those who are trusted to accurately convey his words then to isolate from all of this what is rejected and warn from them with the intention of giving sincere advice and to praise him for his excellent qualities and for what he was correct in as is the way of the scholars.
If there were no virtues of Shaykh Taqi ad-Deen except for his famous student Shaykh Shams ad-Deen ibn Qayyim al-Jawziyyah, writer of many works, from which both his opponents and supporters benefited from then this would be a suffic ient indication of his (ibn Taymiyyahs) great position. And how could it be otherwise when the Shaafi`ee Imaams and others, not to speak of the Hanbalees, of his time testified to his prominence in the (Islamic) sciences" - From Al-Hafih Ibn Hajar Al-Asqalani's endorsement of Ibn Nasurideen Ad-Dimashqi As-Sahafi'i's book Radd al-Waafir in defnce of Shaikh Al-Islam Ibn Taymiyya.
As for position of Shaikh Al Islam Taqi-ud-Deen As-Subki who was a contemporary of Shaikh Al Islam Ibn Taymiyah, he can't be used as evidence as it is clear that the disferences based on Madhab disqualifies testimony against one another. This is evident from the statements of the two Masters of Jarh wa Ta'adeel, Al Hafidh Ad-Dhahabi and Al Hafidh Ibn Hajar that the statements of contamporaries against ane another is not accepted, even more so if it is due to enmity or (diffrences in) Madhab, as no ane can be safe from Hasad except whom Allah pretects. [see Mizaan Al I'itidaal V. 1 p. 111; Siyar A'alaam An-Nubala' V. 5 P. 399]
If you need an example of this sectarianism, as-Sakhaawi endorsed the following words about Taajudeen As-Subki, the son of Shaikh Al Islam Taqi-ud-Deen As-Subki, following his statement: "did any of the Hanbalis raise their heads":
"This is from the strangest of things, and the most partisan of attitudes, and this is why the Qaadi of our time, and Shaikh of the madhab al-Izz al-Kanaani wrote under this statement, and likewise Allaah did not raise the heads of the Mu'attila and then he said about Taajudeen As-Subki, 'he is a man having little manners, lack of scholarly integrity, ignorant of Ahl as-Sunnah and their ranks.- " - see al-I'laan bi-Tawbeekh liman Dhamma at-Ta areekh by of as-Sakhaawi.
From my extensive research, no one justified the extreem position taken against Shaikh al Islam Ib Taymiyya, even his opponent, Taajudeen As-Subki, used to pray for Him let alone consider him a Kafir here on TAI:
"Yaghfirullahu Li Ibn Taymiyya Walaa Haramahu Wasilata an-Nabiy sallalhu 'alaihi wasallam" tabaqat As-Sahafi'iya v. 10, p. 149.
Taajudeen As-Subki (rahimahullah) also said that Al Hafidh Abal Hajjaj Al Mizzi did not write in his hand-writing the words (Shaikh Al Islam) except for two: for His father and Ibn Taymiyya" - see Tabaqat As-Shafi'iyah v. 10 page 195.
The overwhelming proof indicates that Shaikh Al Islam Ibn Taymiyah one of the greatest Imams after the Imams of the Madhabs. And the accusations against him are baseless- as summarized by AlHafidh Ibn Kathir:
"He was, may Allaah have mercy upon him, from the greatest of scholars, also from those who err and are correct. However his errors with respect to his correct rulings were like a drop in a huge ocean and they are for given him as is authentically reported by Bukhaaree, when a ruler makes a ruling, and he is correct then he has two rewards, and if he has erred then he has one reward."
If you read Al-Bidaya wa An-Nihaya, Al Hafidh Ibn Kathir makes it clear that when the scholars of his time gathered for a sitting with ibn Taymiyyah to discuss his work Aqueedah al-Hamawiyyah that his replies to their accusations could not be refuted. see al-Bidaayah wan Nihaayah V.14 pages 5, 48 - 50. Here is an excerpt:
"They reconvened on Jum'ah, after the Prayer, on the 8th of the same month (Rajab). Shaikh Safiyudeen al-Hindi was present and he had a long discussion with Shaikh Taqiyudeen Ibn Taymiyya), but he was less than a match (Saqiyatuhu latamat Bahran = his water-wheel hit an ocean). Then they agreed that Shaikh Kamaludeen bin Zamalkani to be the one to litigate him (Ibn Taymiyya) [...] Then the situation ended with the council accepting the Aqeedah (Al-Wasityah). The Shaikh (Ibn Taymiyya) returned to his home glorious and honored."
To provide you a hint that most of the accusations against Shaikh Al Islam Ibn Taymiyah are baseless, here is what Al Hafidh Ibn Kathir says, about Shaikh Al Islam Taqi-ud-Deen As-Subki's accusation related to visiting graves:
" Now look at this distortion of Shaikh al-Islam (Ibn Taymiyya) whose reply does not contain the prohibition of visiting the graves of Prophets and righteous. Rather he mentioned two views about embarking on a journey or setting out specifically for visiting graves. The visiting of graves with out setting out is one issue, setting out specifically for visiting graves is another issue. The Shaikh (Ibn Taymiyya) does not prohibit the visitation that does not involve setting out. In fact he considers it commendable and urges to it. His books (and his writings) on rites and pilgrimage is a proof to that effect. He does not oppose visitation or say, it is a sin, and he did not narrate i'jma' related to its prohibition. And he is not ignorant about the words of the Prophet, sallalahu 'alaihi wasallam: "Visit the graves, it is a reminder of the last day" - see Albidaya wa-annihaya Volume 14, Page 124.
Shaikh al-Islam Ibn Taymiyya's position about Visting of Graves can be verified from his books. The position Imam al-Haramain Al-Juwayni concerning "setting out to visiting graves" is similar to that of Shaikh Al Islam Ibn Taymiyya:
Imam An-Nawawi in Sharh Sahih Muslim says:
" The scholars disagree concerning setting out to other than the three mosques, such as going to the graves of the righteous, to special places which are similar. Shaikh Abu Muhammad Al-Juwayni, one of our companions said, 'it is Haraam' " - See Sharh Sahih Muslim Volume 9, page 106.
But Imam an-Nawawi does not support the above position and prefers Al-Qadi 'Iyad's, 'it is not haraam'.
Salaam,