Recover tons of deleted files in your Dropbox
It all started a couple of days before a paper submission deadline - when I really needed my data accessible, and my code available. I store all my academic stuff in my Dropbox folder, so it is backed up to the cloud. Lo and behold, Murphy entered the game and caused my Mac to halt, and when I forced a reboot it apparently resulted in some inconsistent state of the Dropbox client. This drove Dropbox into deleting tens of thousands of files from my account, causing files to be deleted remotely and locally, under the tips of my typing fingers. The only thing I had is previous remote revisions of the files. I don't have Dropbox Pro and therefore don't have PackRat, which anyway I'm not sure would help in this case. Besides, I didn't actually care about indefinitely long history, but about quick and convenient recovery. I've decided to take a look at the Dropbox API. I found it quite easy, using the supplied Python Dropbox SDK, to write several tools to help me search and recover my deleted files. The Dropbox Python SDK can be either downloaded and installed using setup.py or using
$ pip install dropbox
In order to access your Dropbox you need to create a Dropbox application that's linked to your account. It can be done through the Dropbox application console (https://www.dropbox.com/developers/apps) and gets you an application key and application secret which are later used by the Python scripts to obtain a token that enables accessing your Dropbox storage. I store the application key and secret in a JSON configuration file, and written a tiny helper script to generate this JSON (make_config.sh):
echo "{
"app_key" : "$1",
"app_secret" : "$2"
}"
The code to obtain an access token for all further actions looks like that
import dropbox
def get_token():
config = open(CONFIG_FILE, 'r')
config_data = json.load(config)
app_key = config_data['app_key']
app_secret = config_data['app_secret']
flow = dropbox.client.DropboxOAuth2FlowNoRedirect(app_key, app_secret)
authorize_url = flow.start()
# Have the user sign in and authorize this token
authorize_url = flow.start()
print '1. Go to: ' + authorize_url
print '2. Click "Allow" (you might have to log in first)'
print '3. Copy the authorization code.'
code = raw_input("Enter the authorization code here: ").strip()
# This will fail if the user enters an invalid authorization code
access_token, user_id = flow.finish(code)
return access_token
The access token is stored in a cookie file for later use.
def create_client():
try:
cookie_content = open(COOKIE_FILE, 'r').read()
access_token = json.loads(cookie_content)
client = dropbox.client.DropboxClient(access_token)
except Exception:
access_token = get_token()
client = dropbox.client.DropboxClient(access_token)
cookie_content = json.dumps(access_token)
open(COOKIE_FILE, 'w').write(cookie_content)
Once we have initialized a client we have access to all kind of methods such as metadata, revisions, restore, file_delete, etc. There are several utilities provided:
find_deleted - Finds deleted file under a given path (optionally by date)
ls - Lists files (optionally by date)
restore_file - Restores files. Paths are provided through standard input.
delete_files - Deletes files. Paths are provided through standard input.
Using these utilities I was able to restore all my accidentally deleted files in a matter of hours. One small technical detail I had to apply is to handle rate limiting imposed by Dropbox. Once in a while an operation would result in an exception if too many actions were attempted in a short amount of time. I had to catch this, and retry the operation.
The code is available under https://github.com/ymcrcat/DropboxRecoveryTools.
The project is maintained by Yan Michalevsky (@ymcrcat). If you have suggestions you can contact him using the contact form on http://www.stanford.edu/~yanm2/contact.html