The cash was taken in 2022 and 2023 as the country grappled with soaring costs and energy bills.
Farmers have been urging the Scottish Government to return the funding, with the call repeated on Wednesday at the AgriScot conference in Edinburgh by NFU Scotland president Martin Kennedy.
Speaking at an event also attended by John Swinney and Scottish agriculture minister Jim Fairlie, Mr Kennedy urged the First Minster to use his Government's December 4 draft Budget announcement to return the cash next year, as well as provide a multi-year fund for farmers.
Mr Swinney said: "The £46 million that the Government has utilised for short-term purposes to ensure that we could deal with the effects of inflation in other areas of the budget will be returned to this portfolio for use in the agricultural sector.
"I give that cast-iron commitment on behalf of the Scottish Government."
But Mr Swinney told attendees the Government will struggle to provide multi-annual funding due to a lack of clarity over future block grants coming from Westminster.
"However much I would like to deliver a multi-annual settlement, we only have financial information for one financial year, and given the point I have just made about uncertainty about the implications in our budget, that will constrain the ability of the Scottish Government to set out the multi-annual certainty that everyone in the industry would like us to do," he said.
Mr Swinney also stressed the importance of working with the sector, as he reflected on the impact of the past few years.
"The combination of Covid, the effect of inflation, energy prices and all that's gone with it has had the feeling of being a bit of a long, dark winter," he said.
"After nearly two decades of economic volatility, there is one thing that we know - the climate and the economy are not going to remedy themselves.
"Therefore, what I appeal today for is a spirit of working together between Government and the industry to prepare the ground for what I hope is a spring that can emerge from this difficult and challenging, dark winter."
But he said changes to inheritance tax brought forward by the UK Government in Chancellor Rachel Reeves's first Budget last month are having a "chilling effect" on farming.
Under the plans, the tax will be charged at 20% on agricultural assets above £1 million, although Ms Reeves has said in some cases the threshold could in practice be about £3 million.
The First Minister called on the UK Government to "urgently" commit to carrying out impact assessments of how the decision will affect farmers and crofters in Scotland.
"Fundamentally, my Government will want to see a tax system that supports rather than hinders orderly succession planning and the transfer of land to the next generation of custodians," he said.
Mr Swinney said he was "disappointed" the Scottish Government was not consulted on the "arbitrary" decision before it was announced, but he committed to working with the UK Government to push for changes, owing to the better relationship with Sir Keir Starmer's administration than the previous Tory incumbent.
A UK Government spokesperson said: "This Budget took the difficult decisions to fix the public services crumbling across the United Kingdom, including Scotland, and address the £22 billion fiscal black hole inherited from the previous government.
"With 40% of agricultural property relief going to the 7% of wealthiest claimants, we made the decision to ensure the relief is fiscally sustainable.
"Around 500 claims each year will be impacted and farm-owning couples can pass on up to £3 million without paying any inheritance tax - this is a fair and balanced approach."