I'd try to keep external libraries out of the main source directory or as part of separate git submodule hierarchy if using stuff like dub is not an option. They don't belong to project sources.