The prisoner release is part of a push by US secretary of state John Kerry to bring the two sides back to the table. Sunday's 13-7 vote, with two abstentions, marks his first visible achievement after six months of shuttle diplomacy.
As a next step, Israeli and Palestinian negotiators are set to hold preliminary talks in Washington on Tuesday, to be followed by up to nine months of negotiations on a peace deal.
Saeb Erekat, the chief Palestinian negotiator, welcomed the Cabinet vote as "a step toward peace."
The fate of Palestinian prisoners is emotionally charged for both sides. Palestinians tend to view prisoners as heroes who made sacrifices in the struggle for independence. Most Israelis view them as cold-blooded terrorists.
The Cabinet approved the release in principle of the 104 prisoners, said a government official who spoke on condition of anonymity in line with briefing regulations. Under the deal brokered by Kerry, the prisoners would be freed in four stages over several months. Each step would be linked to progress in negotiations.
According to a list provided by the Palestinians, the prisoners have served between 19 and 30 years for involvement in deadly attacks on Israelis. Their release would go a long way toward giving Palestinian President Mahmoud Abbas a popular mandate to resume negotiations with Israel, despite widespread skepticism on both sides after 20 years of intermittent talks that produced no results.
On the Israeli side, Prime Minister Benjamin Netanyahu faced stiff opposition from within his governing coalition, including his own Likud Party. Two Likud ministers and those from the pro-settler Jewish Home Party voted against a prisoner release.
Outside the government complex, hundreds of Israelis who lost loved ones in Palestinian attacks demonstrated against the release.
Netanyahu told the Cabinet releasing prisoners involved in deadly attacks was difficult for him.
"This is not an easy moment for me, and is not easy for the ministers in the government and is especially difficult for the bereaved families," Netanyahu said. "But there are moments where I need to make tough decisions for the good of the country and this is one of those moments."
"I believe that resuming the political process at this time is important for Israel," he said, noting that any peace deal would have to be approved in a national referendum.
Along with the prisoner release, ministers also authorized the resumption of talks with the Palestinians and agreed that a team led by the prime minister would oversee negotiations.
They approved the draft of an amended bill that would require a referendum on any partition deal with the Palestinians.
A resumption of peace talks is not yet assured, though.
Abbas has said he won't go back to talks unless Netanyahu accepts the pre-1967 war lines as a starting line for border talks. That refers to all of the West Bank, Gaza Strip and east Jerusalem. The Palestinians want to establish their state those territories, captured by Israel in 1967, though they have said they are ready for minor adjustments.
Abbas told reporters Friday that the invitation to talks to be issued by Kerry will state that the basis for negotiations is the 1967 line. It's not clear whether Netanyahu has dropped his long-standing objection to the idea.
In his comments Sunday, Erekat referred to the touchy issue.
"I hope that we can use this opportunity that the U.S. has provided for us to resume negotiations, in order to achieve peace with a state of Palestine that can live in peace and security next to the state of Israeli on the 1967 lines," he said.
Before the Cabinet vote, hundreds of relatives of those killed by the prisoners protested the release outside parliament. Families held pictures of slain loved ones and chanted, "Terrorists must not be freed."
One of Netanyahu's major coalition partners, Naftali Bennett of the Jewish Home Party, took part in the rally. "Whoever demands the killers of women and children to be freed are not worthy to be called a partner," he said, referring to Abbas. "We have nothing to negotiate with he who praises the killers of women and children," Bennett said.
Bennett has threatened to pull his party out of the government if the prime minister agrees to other Palestinian demands, such as recognizing the 1967 line as a base for talks.
Other Israeli leaders said freeing the prisoners is a necessary move in order to promote peace talks.
"This is a painful decision ... but it's a step that will strengthen Israel strategically," opposition leader Shelly Yachimovich of the centrist Labor party said. She called on Netanyahu to listen to an Israeli majority in favor of a peace deal and "not to be led by extremists in his government."
Israel has a history of releasing Palestinian prisoners, including those involved in attacks. In 2011, it exchanged some 1,000 Palestinians for a single Israeli soldier held by Gaza militants.