Certain types of driverless vehicles may not be safe, peers have warned.
Over-reliance on technology could mean drivers react slowly to taking back control of a semi-autonomous vehicle in an emergency, they said.
However, the Lords Science and Technology Committee noted that some technology could reduce accidents caused by human error.
The Department for Transport said driverless cars "have the potential to transform the way we travel."
Intel makes $15bn bet on driverless cars
California mulls driver-free car tests
Vehicles can be split into different levels of automation, according to industry body the Society of Motor Manufacturers and Traders (SMMT).
Level 0 is a vehicle with no automation, while level 5 is fully automated.
There is a "very dangerous" problem that lies with vehicles on the midway point of this scale, peers on the Lords Science and Technology Committee said.
Vehicles at level 3 still need a driver, but can shift critical operations to the car, and back again.
The risk that the vehicle may need to hand back control to an unprepared driver in an emergency may be "too great to tolerate", the Lords Science and Technology Committee said.Media captionHow the public reacted to a driverless pod in Milton Keynes
Professor Neville Stanton of the University of Southampton told the committee: "As vehicles become fully autonomous, even the most observant human driver's attention will begin to wane.
"Their mind will wander… This is particularly true if they are engaging in other activities such as reading, answering emails, engaged in conversations with passengers, watching movies or surfing the internet."
In general, research suggested drivers of automated vehicles were not as effective at dealing with emergencies as drivers of manual cars.
For example, they had been found to take six times longer to respond to emergency braking by other vehicles.Image copyright PA Image caption The Chancellor promised more money to support the development of driverless cars in his Spring Budget
Complacency could also negatively affect other road users, such as pedestrians and cyclists, the Lords committee said.
The government should give priority to research into people's behaviour around autonomous vehicles, it added.
The Department for Transport said: "Automated vehicles could make our roads even safer and easier to use, as well as promising new mobility for those who cannot drive.
"We have committed over £200m to research and development and are also changing regulations to unblock barriers to this exciting new technology."