Rackspace Cloud Files: Upload Files Using Linux / Unix FTP Command

by on October 29, 2012 · 4 comments· LAST UPDATED November 4, 2012

in ,


I'm using cloud files from Rackspace that provides online storage and CDN for javascript, css, images and video files. However, I found that their online upload is not designed for bulk upload. How do I upload files to Rackspace cloud files storage using standard Linux or Unix ftp client in recursive mode?

Tutorial details
DifficultyIntermediate (rss)
Root privilegesYes/ No
RequirementsRackspace cloud files account and pip.
Estimated completion timeN/A

Rackspace cloud files is an online storage and it can be accessed using web service interface RESTful API or online control panel. This is built with OpenStack technology so that following tutorial should work with any OpenStack powered cloud solution.

Requirements

  1. ftp-cloudfs - Python based FTP interface to Rackspace Cloud Files and OpenStack Swift. This act as a proxy to Rackspace cloud. By default the ftp-cloudfs will listen on port 2021 and you can connect to this port using any standard ftp client to upload / download files or create containers.
  2. pip command - Install ftp-cloudfs.
  3. Rackspace API username and password.

Install ftp-cloudfs

Type the following command ( see how to setup and install pip ):
$ pip install ftp-cloudfs
Sample outputs:

Downloading/unpacking ftp-cloudfs
  Downloading ftp-cloudfs-0.18.tar.gz
  Running setup.py egg_info for package ftp-cloudfs
Downloading/unpacking pyftpdlib (from ftp-cloudfs)
  Downloading pyftpdlib-0.7.0.tar.gz (94kB): 94kB downloaded
  Running setup.py egg_info for package pyftpdlib
 
    You might want to install pysendfile module to speedup transfers:
 
http://code.google.com/p/pysendfile/
 
Downloading/unpacking python-cloudfiles (from ftp-cloudfs)
  Downloading python-cloudfiles-1.7.10.tar.gz (397kB): 397kB downloaded
  Running setup.py egg_info for package python-cloudfiles
    warning: no previously-included files found matching 'python_cloudfiles.egg-info'
Downloading/unpacking python-daemon (from ftp-cloudfs)
  Downloading python-daemon-1.6.tar.gz (41kB): 41kB downloaded
  Running setup.py egg_info for package python-daemon
Downloading/unpacking python-memcached (from ftp-cloudfs)
  Downloading python-memcached-1.48.tar.gz (40kB): 40kB downloaded
  Running setup.py egg_info for package python-memcached
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./py_virtual/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg (from python-daemon->ftp-cloudfs)
Downloading/unpacking lockfile>=0.9 (from python-daemon->ftp-cloudfs)
  Downloading lockfile-0.9.1.tar.gz
  Running setup.py egg_info for package lockfile
Installing collected packages: ftp-cloudfs, pyftpdlib, python-cloudfiles, python-daemon, python-memcached, lockfile
  Running setup.py install for ftp-cloudfs
    changing mode of build/scripts-2.6/ftpcloudfs from 644 to 755
    changing mode of /home/vivek/py_virtual/bin/ftpcloudfs to 755
  Running setup.py install for pyftpdlib
 
    You might want to install pysendfile module to speedup transfers:
 
http://code.google.com/p/pysendfile/
 
  Running setup.py install for python-cloudfiles
    warning: no previously-included files found matching 'python_cloudfiles.egg-info'
  Running setup.py install for python-daemon
  Running setup.py install for python-memcached
  Running setup.py install for lockfile
Successfully installed ftp-cloudfs pyftpdlib python-cloudfiles python-daemon python-memcached lockfile
Cleaning up...
 

You also need to install a Python interface to sendfile to speed up file operation:
$ pip install py-sendfile pysendfile
Sample outputs:

Downloading/unpacking py-sendfile
  Downloading py-sendfile-1.2.3.tar.gz
  Running setup.py egg_info for package py-sendfile
Downloading/unpacking pysendfile
  Downloading pysendfile-2.0.0.tar.gz
  Running setup.py egg_info for package pysendfile
Installing collected packages: py-sendfile, pysendfile
  Running setup.py install for py-sendfile
    building 'sendfile' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c sendfilemodule.c -o build/temp.linux-x86_64-2.6/sendfilemodule.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/sendfilemodule.o -o build/lib.linux-x86_64-2.6/sendfile.so
  Running setup.py install for pysendfile
    building 'sendfile' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c sendfilemodule.c -o build/temp.linux-x86_64-2.6/sendfilemodule.o
    gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-2.6/sendfilemodule.o -o build/lib.linux-x86_64-2.6/sendfile.so
Successfully installed py-sendfile pysendfile
Cleaning up...
 

How do I start ftp-cloudfs proxy server?

Simply type the following command:
$ ftpcloudfs
By default ftpcloudfs will listen on 127.0.0.1:2021. You can verify this with the following commands. Make sure ftpcloudfs is started and you can view it with the ps command:
$ ps aux | grep ftpcloudfs
Sample outputs:

vivek    32314  0.2  0.1  43992  9408 ?        S    22:28   0:00 /home/vivek/py_virtual/bin/python /home/vivek/py_virtual/bin/ftpcloudfs
vivek    32315  0.0  0.1  52188  9268 ?        S    22:28   0:00 /home/vivek/py_virtual/bin/python /home/vivek/py_virtual/bin/ftpcloudfs
vivek    32322  0.0  0.1  43988  8840 ?        S    22:28   0:00 /home/vivek/py_virtual/bin/python /home/vivek/py_virtual/bin/ftpcloudfs
vivek    32333  0.0  0.0   8996   836 pts/8    S+   22:28   0:00 grep --color ftpcloudfs

Use the netstat command to verify that port 2021 is opened by ftpcloudfs, run:
$ netstat -tulpn | grep :2021
Sample outputs:

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:2021          0.0.0.0:*               LISTEN      32314/python

How do I use ftp client to connect to the Rackspace cloud via ftpcloudfs proxy?

You need to connect to the ftp proxy IP 127.0.0.1 at 2021 port:

+-------------+       +------------------+           +-----------------------+
|  ftp client | --->  | ftpcloudfs proxy |   <---->  | Rackspace cloud files |
+-------------+       +------------------+           +-----------------------+
                      Proxy IP: 127.0.0.1
                      Proxy Port: 2021

Type the ftp command as follows:

 
ftp 127.0.0.1 2021
 

You can also use lftp command as follows:

 
lftp -u USERNAME -p 2021 127.0.0.1
lftp -u USERNAME,PASSWORD -p 2021 127.0.0.1
lftp -e 'ftp-command1; ftp-command2;exit;' -u USERNAME,PASSWORD -p 2021 127.0.0.1
lftp -e 'ls;exit;' -u USERNAME,PASSWORD -p 2021 127.0.0.1
 

Example - Sample session

Linux and Unix like operating systems comes with ftp client which allows a user to transfer files to and from a remote network site:

 
ftp 127.0.0.1 2021
 

Sample outputs:

Connected to 127.0.0.1.
220 Rackspace Cloud Files 0.18 using pyftpdlib 0.7.0 ready.
Name (127.0.0.1:vivek): nixcraft
500 Command "AUTH" not understood.
500 Command "AUTH" not understood.
SSL not available
331 Username ok, send password.
Password: HIIDENPASSWORD
230 Welcome nixcraft
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
200 Active data connection established.
125 Data connection already open. Transfer starting.
drwxr-xr-x 745 nixcraft nixcraft 42698582 Oct 29 17:08 images.cyberciti.biz
drwxr-xr-x 100 nixcraft nixcraft   497143 Oct 29 17:08 static
drwxr-xr-x 555 nixcraft nixcraft 22378320 Oct 29 17:08 pdfs
drwxr-xr-x   1 nixcraft nixcraft        0 Oct 29 17:08 videos
drwxr-xr-x   1 nixcraft nixcraft        0 Oct 29 17:08 wpuploads
drwxr-xr-x 1346 nixcraft nixcraft 67375928 Oct 29 17:08 images
226 Transfer complete.

See how to use ftp command to upload / download files for more information about ftp commands.

A note about GUI ftp clients

If you do not want to type the commands, than use a browser such as Firefox and type the following at url bar:

ftp://127.0.0.1:2021/
ftp://USERNAME:PASSWORD@127.0.0.1:2021/

You can also use any standard GUI based FTP client.

A note about ftp

You may want to use sftp-cloudfs which provides SFTP (secure ftp) interface to Rackspace Cloud Files and Open Stack Object Storage (Swift) as described here. Stay tuned for sftp-cloudfs tutorial.

How do I download files from cloud files?

You can use URL to access files in this container via the public internet via cdn url such as follows:

http://8173579aa04ea33f3314-2bd3e6c481b4a486fea8b805a638363d.r25.cf2.rackcdn.com/file.txt

Login to your rackspace cdn account to see CDN enabled url:

Rackspace Cloud Files Akamai CDN URL

Fig.01: Finding Cloud Files Akamai CDN URL via control panel

Check out related media

This tutorial is also available in a video format:


(Video.01: ftp-cloudfs Rackspace demo )

References
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 4 comments… read them below or add one }

1 Raj October 30, 2012 at 5:36 am

Great tutorial. Can’t wait for sftp related tutorial.

Reply

2 Pete October 30, 2012 at 12:40 pm

Nice job! Also check out cloudfuse for some local mounting of the cloud files. It uses fuse hotness to mount the cloud files container in user space.

Reply

3 nixCraft November 1, 2012 at 10:06 am

Thanks for the comment and link. I will check it out.

Reply

4 Hoekstraman March 24, 2014 at 7:05 pm

Any way to do this for hubic? ftp-cloudfs is looking for an authurl. I have no idea what to fill in there.

Reply

Leave a Comment

Tagged as: , , , , , , , , ,

Previous Faq:

Next Faq: