A transaction in a database management system refers to a logical unit of work that consists of one or more database operations. It represents a sequence of operations that should be executed as a single, indivisible unit. A transaction can involve reading or modifying data stored in a database.