Densityscout looks for files that have been packed, that is compressed or encrypted. Most Windows executables are not, so malware tends to stand out.
Run as follows (on linux):
densityscout -s cpl,exe,dll -p 0.1 -o densityscout-output.txt -r /path/to/Windows/System32/
This will print to standard-out the ones that show evidence of packing, and output the level of packing for all executables to the output file.
For more details, see https://www.cert.at/downloads/software/densityscout_en.html