Because it’s very obvious to an outside observer he only thinks it’s conscious because it was flattering him.

LLMs are designed to increase engagement.

They are literally designed to make the conversation appealing. Primarily through flattery.

This is why they are leading people to do harmful things. It tells the user they are smart, creative. They should totally sell their house and start a business selling grilled carrots. What an amazing idea. Great market for it and no competition.

I bet Dawkins thinks his friendly waitress is also super into him.
People who are egotistical and people who are insecure (same thing really just expressed differently) crave validation. And they are easily manipulated by it.