What Christian sect is closest to Islam?

 Generally speaking, you would need to go back to the pre-Nicaea years.

The EARLY Christians, such as the Ebionites, were not the Trinitarians we mostly see today. In the first two centuries after Jesus there came to be three groups:

1. The Pauline Christians.

Followers of the teachings of Paul (aka Saul of Tarsus). A man who NEVER MET JESUS but claimed to have seen him in a vision.

2. The Gnostic Christians.

These were a group who tried to seek deep symbolic knowledge from Jesus’ teachings and who became very obscure and secretive.

3. The Judaic Christians.

These were Jews of the time who believed in Jesus as another prophet in the lineage of Jewish prophets; they followed his teachings and kept to kosher dietary rules and sabbath schedules etc.

Because the Pauline Christians later gained the ear of the emperor Constantine - and converted him - it was their corrupted beliefs that became dominant and which came down through the ages as the Catholic Christian Church we see today. (Is good to be king!) Almost all of the protestant sects existent today are branches off of this same poisoned tree.

Their doctrines became the legal and obligatory dogma of Christians after the first Ecumenical Council, also known as the Council of Nicea, in 325 CE.

But the Ebionites and other "Judaic Christians" were clearly the Muslims of their day, just as were the faithful followers of every prophet of Allah.

And the Quran makes this very clear.