The poll by Forsa for broadcaster RTL and Stern magazine found that 53% of respondents did not think the conservative bloc, made up of Merz's Christian Democrats (CDU) and their Bavarian sister party, the Christian Social Union (CSU), would gain more support under a different chancellor. Some 41% said they believed the alliance would benefit from a change in leadership, the DPA reported.
Even among CDU/CSU supporters, 42% said the conservatives would have better prospects with another chancellor from their ranks, while 56% disagreed.
The survey of 1,008 people was conducted on May 28 and 29.
The findings come after German media reported discussions within conservative circles about whether Merz could eventually be replaced following a difficult first year in office. Hendrik Wüst, the premier of Germany's most populous state, North Rhine-Westphalia, has been mentioned as a possible successor.
The speculation comes against a backdrop of weak polling for the conservatives and rising support for the far-right Alternative for Germany (AfD).
Merz's allies have strongly rejected the speculation. Conservative parliamentary leader Jens Spahn described reports of a possible leadership challenge as "nonsense" over the weekend, while Wüst also said recent speculation was "simply nonsense" and pledged his full support for the chancellor.
Merz's conservatives and their coalition partners in the Social Democratic Party (SPD) are currently locked in tough negotiations over major reforms to pensions and health care, while the country's economy has taken a severe hit from the US-Israeli war against Iran.