Introducing Magic Uploads via Ubuntu One Rest API.

One of the features recently introduced in Ubuntu One and hidden beneath the covers is what we refer to as Magic Uploads. Recently this feature was also enabled in the REST API so I would like to demonstrate how this works.

But first What is a MAGIC Upload? a magic upload is the ability to save time when uploading a file you have uploaded in the past. This is done by including the hash of the file in the upload, if Ubuntu One detects you have previously uploaded that hash, it will not require an upload and just create the file. Magic hashing is an additional way to do this for files uploaded to Ubuntu One by anyone. This is done by including the hash of the file along with a magic hash which is calculated on the hash of the file. This may sound sophisticated, but it’s not, here’s a function that returns the hash of the file along with the so-called magic hash:

def get_u1_hashes(filename):
    """Calculate the file hashes used to update a file."""
    hash_object = sha1()
    magic_hash_object = sha1('Ubuntu One')
    with open(filename, 'rb') as f:
    content = f.read()
    hash_object.update(content)
    magic_hash_object.update(content)
    hash = 'sha1:' + hash_object.hexdigest()
    magic_hash = 'magic_hash:' + magic_hash_object.hexdigest()
    return hash, magic_hash

As you can see, there’s no real magic. But in order to calculate the hash, you will need the content of the file. Note this is no trade secret I’m revealing here, it’s right there in the source for the Ubuntu One Storage Protocol used by the client. So how can you use this in the REST API?

Recently hash and magic_hash parameters were added to the PUT file API although as of this writing they are not documented in the public documentation. Note that this is different than the Content URI to put file content and upload a file. Here’s how the documentation should eventually include:

PUT /api/file_storage/v1/~/path/to/volume/path/to/node

Create, or change a file with existing content that was already uploaded.

 {
    "kind": "file",
    "hash": "<sha1 hash of file>",
    "magic_hash": "<magic hash>", 
 }

And that’s it. Note that this feature has already been added to my Restful Ubuntu One client. So how can you take advantage of this? See my next post.

Upload a directory using restful-u1 library

Ever wonder how to upload an entire directory tree and files to Ubuntu One? Here’s 7 lines that will do it:

import os
from u1rest import get_files_user
user = get_files_user()
for p, d, files in os.walk('/home/john/Pictures'):
    for n in files:
        path = os.path.join(p,n)
        user.upload_file(path, '~/Ubuntu One' + path)

Using the restful-u1 library with Ubuntu One

This post will demonstrate how the restful-u1 library wraps all of the details of making the REST API calls into a simple-to-use interface. Please read my previous post about how to get the restful-u1 python library.

I’m going to use an ipython console to demonstrate:

In [1]: from u1rest import get_files_user
In [2]: user = get_files_user(use_file_keystore=True)

Note that I am using the flag use_file_keystore=True in get_files_user to use the file created earlier when we authorize the code to access our Ubuntu One account. If I didn’t use this flag, it would default to use an OAuth token from your keyring.

The primary way to us the u1rest library is to get a user and then make all the calls using this object. The User object is itself a resource (in Restful terms) and has attributes. Most of them are obvious, but I want to focus on 2 of them for now.

In [3]: user.root_node_path
Out[3]: u'/~/Ubuntu One'

In [4]: user.user_node_paths
Out[4]: [u'/~/SyncMusic/Music', u'/~/.ubuntuone/Purchased from Ubuntu One']

The root_node_path is the path of your default root directory, this is always going to be ‘~/Ubuntu One’.

In [5]: root = user.get_node(user.root_node_path)
In [8]: root.kind, root.path, root.resource_path, root.when_created
Out[8]: (u'directory', u'/', u'/~/Ubuntu One', u'2009-01-06T22:40:59Z')

Additionally, user_node_paths is a list of root directories for User Defined Folders. As you can see I have two of these. One is the special folder where music I purchase in the Ubuntu One music store is placed the other one is a UDF I created on one of my clients. Like the root_node_path these are root directories where all of your files exist. I could describe it, but I would rather demonstrate.

In [11]: d1 = user.make_directory("~/Ubuntu One/a/b/c/d")
In [12]: d1.resource_path
Out[12]: u'/~/Ubuntu One/a/b/c/d'
In [13]: d1.path
Out[13]: u'/a/b/c/d'

Here we use a method right off the user object to create a directory. Notice that we included the path “~/Ubuntu One” this tells Ubuntu One that you’re creating this in your default root directory. In this example, the directory created has the resource_path of ~/Ubuntu One/a/b/c/d all resources in the Ubuntu One Rest api have a  resource_path this is the unique path of the resource within the API. Note also that the directory has a path attribute. This is the path of the node under the root.

Of course Ubuntu One isn’t really about directories, it’s about files. The most common thing to do is upload files. So here’s how it’s done with the u1rest library.

In [15]: f1 = user.upload_file('/home/john/Screenshot-1.png', "~/Ubuntu One/I/uploaded/this/Screenshot.png")
In [16]: f1.resource_path
Out[16]: u'/~/Ubuntu One/I/uploaded/this/Screenshot.png'
In [17]: f1.content_path
Out[17]: u'/content/~/Ubuntu One/I/uploaded/this/Screenshot.png'
In [18]: f1.path
Out[18]: u'/I/uploaded/this/Screenshot.png'

As you can see, uploading a file is very simple. What’s really nice is that this created the folder structure needed under Ubuntu One along with the upload. So now the path ‘/I/uploaded/this’ was created without an extra step. So now I have a file uploaded to Ubuntu One, what else can I do? How about making the file public?

In [19]: f1.set_public()
In [20]: f1.public_url
Out[20]: u'http://ubuntuone.com/p/uRK/'

There! now we have a public file.

That’s all for now, If you want to see more, checkout the example.txt doctest in the source. In the next post I’ll talk about volumes.

A python library for Ubuntu One’s REST API.

During UDS in Budapest a few weeks ago a new feature of Ubuntu One was quietly announced on Wednesday. A set of pages documenting REST APIs was made available here including a new Rest API for accessing files and directories synchronized with Ubuntu One.

As we developed the API, I also threw developed a simple library for accessing this API mostly so I could test the server side to make sure we have something usable. The main goal for this client was to make it easy to use and provide a library which could be used to create much more robust U1 clients.

I’ll be blogging about this more, but if you want to tinker with it you’ll need to do the following:

In order to use the library:

• Sorry, this currently only works on Ubuntu until I get a generic OAuth library.
• Install bzr (sudo apt-get install bzr)
• Grab the branch lp:restful-u1
• If you’re not already using Ubuntu One get a free account https://one.ubuntu.com

For the examples we won’t be relying on the keyring for storing credentials. So run the following:

PYTHONPATH=. ./u1rest/createfilekeystore.py --email <your username> \
                                            --password <your password> \
                                            --name RESTfulClient

Now you should a file credentialsfile.txt. Guard this file as it contains the OAuth tokens to access Ubuntu One.
Here’s a simple example of using the client in a python console session:

from u1rest import get_files_user
user = get_files_user(use_file_keystore=True)
f = user.upload_file(, "~/Ubuntu One/")

Look easy? Check out the doctest example.txt for a full demo of what it can do.
I’ll be back with more on using this library.

Happy Hacking!