Jonathan Lu poses for the media in Hangzhou Zhejiang Province. Credit: Reuters/Aly Song
His appointment reflects the company's desire to ensure it is helmed by an executive with deep knowledge of the e-commerce business as it gears up for change.
Barclays said recently Alibaba could be worth $55 billion.
"He has a very strong reputation and he has a very strong background in the e-commerce space," said analyst Elinor Leung at Hong Kong-based brokerage CLSA.
"Ideally, you always want the founder to lead the IPO, but if the founder decides not to do it, Jonathan has a lot of experience," Leung said.
Lu was hired in 2000 to build out Alibaba's South China sales region. He has also led a team to establish Alibaba's e-payment unit, Alipay, and served as president of the unit. In 2008, he moved to shopping website Taobao and was appointed as its head in 2010.
Both units were successful under his leadership and were key to making Alibaba Group dominant in a country where 16 percent of the population shops online.
In 2011, Lu was chosen by Ma to lead the group's business-to-business e-commerce platform, Alibaba.com, after the unit's chief executive stepped down following a string of fraud cases.
Lu is known to be more low-key than Ma, who worked as an English teacher before building an e-commerce juggernaut. Ma is known for his quips and brash style, and his ease in speaking both English and Mandarin, while Lu is low profile and more comfortable speaking Mandarin.
Billionaire Ma is the 11th richest man in China according to Forbes and one of China's best known corporate leaders. Ma announced his decision to step down as CEO in January, citing his desire to pass baton to a younger generation of leaders. He will remain chairman of the group.
Alibaba is facing competition from e-commerce platforms including 360buy, E-commerce China Dangdang Inc, Amazon.com Inc and Tencent Holdings.
Last year, Alibaba bought back a partial share of the stake Yahoo Inc owned in it and may be preparing for an IPO. Yahoo still owns 24 percent of Alibaba Group. Although Alibaba has said it has no timeline for an IPO, industry sources said the likely Hong Kong listing is among the big IPOs bankers are preparing for late this year or early next year.
In the agreement inked between Yahoo and Alibaba last year, Yahoo built in financial incentives for Alibaba if it lists by 2015.