Password functions
The Server Initialization Packet and the Client Authentication Packet both have an 8-byte field, scramble_buff. The value in this field is used for password authentication. It works thus:
The server sends a random string to the client, in scramble_buff.
The client encrypts the scramble_buff value using the password that the user
enters. This happens in sql/password.c:scramble() function.
The client sends the encrypted scramble_buff value to the server.
The server encrypts the original random string using a value in the mysql
database, mysql.user.Password.
The server compares its encrypted random string to what the client sent
in scramble_buff.
If they are the same, the password is okay.
Relevant MySQL Source Code:
libmysql/password.c comments at start of file.