Make sure your file's visibility in Google Drive is set to "Anyone with the link". If it's set to "Restricted" then only people who are logged in to Google and have been granted access to the file will be able to open the direct link (which probably isn't what you want).
This only works for uploaded files, not documents created in Google Docs/Sheets/Slides.
If your file is very large, the direct link may first open a page saying that the file is too large for Google to scan for viruses. That page will have a button to download the file.