Automating Downloads

Earthdata URS

ESDIS, our parent organization, mandates tracking file downloads and restricting access to certain data for authorized users only, as required by the MODIS and VIIRS Science teams. To fulfill these requirements, ESDIS has implemented Earthdata Profile (URS), a profile manager that records user activity. Previously, users authenticated via a username and password dialog on this site. Now, users are redirected to the URS login screen and returned to this site upon successful authentication.

Deprecation of FTP

File Transfer Protocol (FTP) has been widely used for many years to transfer files via FTP clients and various scripting languages. However, it is now considered a security risk by many cybersecurity experts. As a result, NASA's Goddard Space Flight Center has blocked all requests to public-facing FTP servers, including LAADS DAAC and LANCE NRT.

Downloading via HTTP

Hypertext Transfer Protocol (HTTP) drives most website internet traffic today, and its secure variant, HTTPS, has been chosen to replace FTP. HTTPS encrypts all transactions between client and server, making it extremely difficult for third parties to intercept the data being transferred. All HTTPS downloads on this site require either an active login session or a Download Token passed via an Authorization HTTP header..

Earthdata Profile

Each user must create an Earthdata URS profile in order to download files. Each URS profile is tracked by URS usernames.

Download Tokens

Users wishing to download via a browser simply need to log in and click links within the Archive. Scripted downloads will need to use Earth Data Login (EDL) Tokens in order to be properly authorized.

Requesting a Download Token

Any user that does not already have a Download Token will need to create them following the steps below:

1. Log in by going to Profile -> Login
2. Navigate to Profile -> Download Tokens
3. Create your new Download Token by clicking the "Generate Download Token" button

Retrieving a Download Token

Any forgotten or lost Download Tokens can be replaced by simply logging in and generating a new token.

How to use a Download Token

This example uses the curl command to make a request for the file on our web server at the URL https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/PATH_TO_MY_FILE

curl -v -H "Authorization: Bearer MY_APP_KEY" "-C - https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/PATH_TO_MY_FILE" > result

The example above passes your Download Token via the Authorization HTTP header while utilizing the Bearer schema. When finished, the resulting download will be written to a file called “result“ in whatever directory (folder) you run the command from.

The Download Token located in the header is how this site identifies users. If the Download Token is invalid, missing, or an -h is used instead of -H, the curl command will not work. The -v parameter tells curl to be verbose so extra information about the request and response will be printed out. If this extra information is not needed the -v parameter can be omitted.

Automation

If all you need is one file and you know which file it is, it is much easier to go to the data archive and click to download as needed. If you need many files you might prefer to rely on scripts. We have samples for wget, an open source batch downloader.

Code Samples

wget

wget is an open source utility that can recursively download entire directories of files with just one command. The only path that is required is the root directory. wget will automatically traverse the directory and download any files it locates.

wget is free and available for Linux, macOS, and Windows.

Installation

1. Linux
   1. Launch a command-line terminal
   2. Type yum install wget -y
2. macOS
   1. Install Homebrew (admin privileges required)
   2. Launch Applications > Utilites > Terminal
   3. Type brew install wget
3. Windows
   1. Download the latest 32-bit or 64-bit binary (.exe) for your system
   2. Move it to C:\Windows\System32 (admin privileges will be required)
   3. Click the Windows Menu > Run
   4. Type cmd and hit Enter
   5. In the Windows Command Prompt, type wget -h to verify the binary is being executed successfully
   6. If any errors appear, the wget.exe binary may be not be located in correct directory or you may need to switch from 32-bit <-> 64-bit

Command-Line/Terminal usage:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://nrt3.modaps.eosdis.nasa.gov/archive/PATH_TO_DATA_DIRECTORY" --header "Authorization: Bearer INSERT_YOUR_TOKEN_HERE" -P TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM

macOS/Linux example:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://nrt3.modaps.eosdis.nasa.gov/archive/allData/61/MOD14T/2025/081/" --header "Authorization: Bearer ABCDEFGH-12345-IJKL-6789-MNOPQRST" -P /Users/jdoe/data

Windows example:

wget -e robots=off -m -np -R .html,.tmp -nH --cut-dirs=3 "https://nrt3.modaps.eosdis.nasa.gov/archive/allData/61/MOD14T/2025/081/" --header "Authorization: Bearer ABCDEFGH-12345-IJKL-6789-MNOPQRST" -P C:\Users\jdoe\data

Be sure to replace the following:

• PATH_TO_DATA_DIRECTORY: location of source directory in the Archive such as archive/allData/61/MOD14T/2025/081/
• INSERT_YOUR_TOKEN_HERE: Your token
• TARGET_DIRECTORY_ON_YOUR_FILE_SYSTEM: Where you would like to download the files. Examples include /Users/jdoe/data for macOS and Linux or C:\Users\jdoe\data for Windows