The slang term “rat's nest” is used to describe a particularly messy or disordered environment, referencing the habits of many species of rat. The term can also be used metaphorically, to describe a messy situation, as in “a rat's nest of problems was uncovered when the company was audited.” As a general rule, this term is used in a deprecating way, under the assumption that most people prefer a state of general cleanliness to a mess.
Rats are famous for having messy nests because they build their nests from a wide assortment of items, and they tend to hoard objects of interest or curiosity. A rat's nest might include scraps of paper and textiles, twigs, grasses, and leaves, but it will also hold things like pens, metal, and any other objects which a rat might find intriguing. As young rats are raised in the nest, they soil it until they reach an age where they can strike out on their own, rendering the nest spectacularly messy.
Humans and rats have been living close to each other for centuries, so it is perhaps not surprising that a messy human's living space or office would be compared to a rat's nest. The implication is that the space is totally chaotic and filled with a wide assemblage of hoarded objects, stored in no particular order and for no apparent reason. A rat's nest might also have various food items squirreled away, causing a distinctive odor to hang over the space.
Some people prefer messy environments, and in fact some would argue that although their spaces look messy, they have a very particular and set order, even if it is understood only by the occupant. Much like the rats they are likened to, people who inhabit or work in rat's nests may actually have a very sound logic which governs the arrangement and contents of their space, although more clean-minded people might have trouble comprehending it.
In extreme cases, a rat's nest can become a hazard, as a collection of items may pose a fire hazard, or potentially serve as an incubator for mold, mildew, and animal pests who take advantage of discarded food. In the sense of a generally messy situation, a rat's nest is an entangled, complicated situation which may require a great deal of work to understand or clean up.