When exactly did God take both Israel and Judah back?
When Judah returned from the Babylonian captivity, they did not come alone. "So all Israel was enrolled by genealogies; and behold, they are written in the Book of the Kings of Israel. And Judah was carried away into exile to Babylon for their unfaithfulness. Now the first who lived in their possessions in their cities were Israel, the priests, the Levites and the temple servants. Some of the sons of Judah, of the sons of Benjamin and of the sons of Ephraim and Manasseh lived in Jerusalem" (I Chronicles 9:1-3). Notice in the roll taken on the return by Ezra are people of the tribes of Ephraim and Manasseh, two of the ten northern tribes. When they returned, twelve priests carried tribute from all of Israel (Ezra 8:24-25) On their arrival in Jerusalem, twelve bulls were offered for all Israel (Ezra 8:35).
Later, Ezra rebukes the people for marrying foreign wives. Judah and Benjamin were listed as being guilty (Ezra 10:9-10). The Levites, as well, are listed among those pledged to put away their wives (Ezra 10:18-19), but notice that those of Israel are also listed (Ezra 10:25).
All of this makes sense because God said that both kingdoms would return from captivity together. "In those days the house of Judah will walk with the house of Israel, and they will come together from the land of the north to the land that I gave your fathers as an inheritance" (Jeremiah 3:18).
Jeremiah 3:8 - God divorced Israel and gave her a bill of divorcement.
Malachi 2:11 - Judah married strange gods.
Deuteronomy 24:1-4 - If a husband gives his wife a bill of divorcement and she goes and marries another and defile herself then the former husband can't take her back.
How could God take Israel back after she defiled herself?
The core problem is your citation of Malachi 2:11.
"Do we not all have one father? Has not one God created us? Why do we deal treacherously each against his brother so as to profane the covenant of our fathers? Judah has dealt treacherously, and an abomination has been committed in Israel and in Jerusalem; for Judah has profaned the sanctuary of the LORD which He loves and has married the daughter of a foreign god. As for the man who does this, may the LORD cut off from the tents of Jacob everyone who awakes and answers, or who presents an offering to the LORD of hosts" (Malachi 2:11).
Typically when God talked about His covenant with Israel or Judah, He refers to Israel and Judah as females to draw an analogy to a marriage covenant, such as Jeremiah 3:6-10. However, here Judah is referred to as a male who marries the daughter of a foreign god (not the god itself). "Judah" being used as a figure of speech where the whole is being personified as a single man. The problem is the same one mentioned in Ezra 10 and Nehemiah 13. The people of Israel were marrying people of other nations who worship idols. It is not saying that all of Israel or all Judah had entered into a covenant with another so-called god.
Therefore, Deuteronomy 24:1-4 doesn't apply. Israel and Judah did commit spiritual adultery with idolatry, but they did not enter into another covenant with them. God was within His rights to forgive and bring His people back, just as He illustrated with Hosea's marriage to Gomer.
In Jeremiah 3:1, the prophet points out the law in Deuteronomy 24. Why was Jeremiah warning them of this law if it doesn't apply?
God lays out a case against the idolatry if Israel and Judah that actually starts in Jeremiah 2 and continues into Jeremiah 3. "For of old I have broken your yoke and burst your bonds; and you said, 'I will not transgress,' when on every high hill and under every green tree you lay down, playing the harlot" (Jeremiah 2:20).
"Now this came about because the sons of Israel had sinned against the LORD their God, who had brought them up from the land of Egypt from under the hand of Pharaoh, king of Egypt, and they had feared other gods and walked in the customs of the nations whom the LORD had driven out before the sons of Israel, and in the customs of the kings of Israel which they had introduced. The sons of Israel did things secretly which were not right against the LORD their God. Moreover, they built for themselves high places in all their towns, from watchtower to fortified city. They set for themselves sacred pillars and Asherim on every high hill and under every green tree, and there they burned incense on all the high places as the nations did which the LORD had carried away to exile before them; and they did evil things provoking the LORD. They served idols, concerning which the LORD had said to them, "You shall not do this thing."
"Yet the LORD warned Israel and Judah through all His prophets and every seer, saying, "Turn from your evil ways and keep My commandments, My statutes according to all the law which I commanded your fathers, and which I sent to you through My servants the prophets."
"However, they did not listen, but stiffened their neck like their fathers, who did not believe in the LORD their God. They rejected His statutes and His covenant which He made with their fathers and His warnings with which He warned them. And they followed vanity and became vain, and went after the nations which surrounded them, concerning which the LORD had commanded them not to do like them. They forsook all the commandments of the LORD their God and made for themselves molten images, even two calves, and made an Asherah and worshiped all the host of heaven and served Baal. Then they made their sons and their daughters pass through the fire, and practiced divination and enchantments, and sold themselves to do evil in the sight of the LORD, provoking Him.
"So the LORD was very angry with Israel and removed them from His sight; none was left except the tribe of Judah. Also Judah did not keep the commandments of the LORD their God, but walked in the customs which Israel had introduced. The LORD rejected all the descendants of Israel and afflicted them and gave them into the hand of plunderers, until He had cast them out of His sight. When He had torn Israel from the house of David, they made Jeroboam the son of Nebat king. Then Jeroboam drove Israel away from following the LORD and made them commit a great sin. The sons of Israel walked in all the sins of Jeroboam which he did; they did not depart from them until the LORD removed Israel from His sight, as He spoke through all His servants the prophets. So Israel was carried away into exile from their own land to Assyria until this day" (II Kings 17:7-23).
God asked, "Why do you contend with Me? You have all transgressed against Me," declares the LORD. "In vain I have struck your sons; they accepted no chastening. Your sword has devoured your prophets like a destroying lion" (Jeremiah 2:29-30).
Israel and Judah had reached the point that they thought it did not matter what they did. Much like the modern-day belief that "once saved, always saved," the people of Israel thought that all they had to do is call on God and He would accept them back. But God points out that He is within His rights to never take them back. "God says, "If a husband divorces his wife and she goes from him and belongs to another man, will he still return to her? Will not that land be completely polluted? But you are a harlot with many lovers; yet you turn to Me," declares the LORD" (Jeremiah 3:1). Even though they would not do this in their own marriages, they expect God to accept them despite their idolatry. "Have you not just now called to Me, 'My Father, You are the friend of my youth? Will He be angry forever? Will He be indignant to the end?' Behold, you have spoken and have done evil things, and you have had your way" (Jeremiah 3:4-5).
The start of Jeremiah 3 is a warning to Judah that they were on the brink of no return. If they bound themselves to the idols, God would reject them permanently. God had already "divorced" northern Israel and Judah still thought it would not happen to them.
"Then the LORD said to me in the days of Josiah the king, "Have you seen what faithless Israel did? She went up on every high hill and under every green tree, and she was a harlot there. I thought, 'After she has done all these things she will return to Me'; but she did not return, and her treacherous sister Judah saw it. And I saw that for all the adulteries of faithless Israel, I had sent her away and given her a writ of divorce, yet her treacherous sister Judah did not fear; but she went and was a harlot also. Because of the lightness of her harlotry, she polluted the land and committed adultery with stones and trees. Yet in spite of all this her treacherous sister Judah did not return to Me with all her heart, but rather in deception," declares the LORD" (Jeremiah 3:6-10).
Yet, despite their excessive idolatry, God is offering them one more chance to return to Him but it will require that they no longer worship the idols.
"Go and proclaim these words toward the north and say, 'Return, faithless Israel,' declares the LORD; 'I will not look upon you in anger. For I am gracious,' declares the LORD; 'I will not be angry forever. Only acknowledge your iniquity, that you have transgressed against the LORD your God and have scattered your favors to the strangers under every green tree, and you have not obeyed My voice,' declares the LORD" (Jeremiah 3:12-13).
"If you will return, O Israel," declares the LORD, "Then you should return to Me. And if you will put away your detested things from My presence, and will not waver, and you will swear, 'As the LORD lives,' in truth, in justice and in righteousness; then the nations will bless themselves in Him, and in Him they will glory" (Jeremiah 4:1-2).
Israel and Judah had a choice to make. They could join themselves to idolatry permanently or they could return to God. God has already rejected them and if they persist, He will never take them back. He would be just if He left them discarded. But in His great mercy, He is offering to receive them back, but only if they give up their idols permanently. They didn't deserve this chance, but God is offering it to them.