While some big names such as Wal-Mart or department stores have historically tried to pass on rising costs to suppliers, the border tax is such a big change it will be hard to make it up that way, retail consultant and CNBC contributor Jan Kniffen told "Power Lunch."
"This basically, on the average retailer, cuts the profitability in half," he said.
He believes they would have to raise prices, on average, 10 percent to get back to even.
"If you've got great pricing power, this all works. If you don't have pricing power, you are going to lose a lot of business or a lot of profitability or both," Kniffen noted.
The border adjustment tax is part of the Republicans' overall tax reform plan. President Donald Trump, who recently met with retail executives, has yet to definitively weigh in on the issue.
On Thursday, Treasury Secretary Steve Mnuchin told "Squawk Box" the administration is listening to an array of businesses that are concerned about the impact of the tax on their profits.
"We understand the concerns, we understand where people are, and we're going to have a plan that addresses these concerns," Mnuchin said.
— CNBC's Elizabeth Gurdus contributed to this report.