Transportation Secretary Pete Buttigieg said Sunday that Senate Democrats and Republicans must establish a clear direction on infrastructure negotiations when Congress returns to Washington after the Memorial Day break, signaling that the White House is losing patience with bipartisan talks.

“By the time that they return, which is June 7 just a week from tomorrow, we need a clear direction,” Buttigieg said during an interview on CNN’s “State of the Union.” “The President keeps saying, ‘inaction is not an option’ and time is not unlimited here. The American people expect us to do something.”

Senate Democrats plan to move forward with crafting a sweeping infrastructure package next month with or without Republican support in order to pass a bill this summer.

The two parties are in ongoing talks but are not close to an agreement on what the plan would include and how the government would pay for the much-needed investments.

Buttigieg said he believes the White House is “getting pretty close to a fish or cut bait moment” on bipartisan negotiations.

“This can’t go on in terms of the condition of our infrastructure, therefore, the negotiations can’t go on forever either,” he said.

Republicans on Thursday provided President Joe Biden with a $928 billion counteroffer on infrastructure, amounting to roughly half of the $1.7 trillion proposal the administration previously offered. The White House originally put forward a $2.3 trillion infrastructure proposal.

Democrats and Republicans have disagreed on what constitutes infrastructure and how best to pay for the plan.

Democrats have rejected a GOP offer to fund the plan through user fees, arguing that doing so could cause a tax hike for middle-class Americans who drive. Republicans have opposed the Democrats’ proposal to raise the corporate tax rate to at least 25% to pay for the plan.

IDemocrats could ultimately pass the legislation without GOP support through the process of budget reconciliation, which would require a simple majority vote in the Senate.