History and Future of Cloud Federation


Federated Cloud Sharing - Connect self-hosted, decentralized clouds

Federated Cloud Sharing – Connect self-hosted, decentralized clouds

I’m now working for about two years on something called Federated Cloud Sharing. It started on June, 23er 2014 with the release of ownCloud 7.0. Back then it was simply called “Server to Server sharing”. During all this years I never wrote about the broader ideas behind this technology, why we do it, what we achieved and where we are going.

Motivation

The Internet started as a decentralized network, meant to be resilient to disruptions, both due to accidents or malicious activity. This was one of the key factors which made the Internet successful. From the World Wide Web, over IRC, news groups, e-mail to XMPP. Everything was designed as decentralized networks, which is why if you are on the Google servers you can email people at Yahoo. Everybody can set up his own web server, e-mail or chat server and communicate with everyone else. Individuals up to large organisations could easily join the network, participate and build business without barriers. People could experiment with new innovative ideas and nobody had the power to stop them or to slow them down. This was only possible because all underlying technology and protocols were build on both Open Standards and Free Software.

This changed dramatically over the last ten years. Open and inclusive networks were replaced by large centralized services operated by large companies. In order to present yourself or your business in the public it was no longer enough to have your own website, you had to have a page on one or two key platforms. For communication it was no longer enough to have a e-mail address, or be on one of the many IRC or XMPP servers. Instead people expected that you have a account on one of the major communication platforms. This created huge centralized networks, with many problems for privacy, security and self-determination. To talk to everybody, you have to have an account on Facebook, at Google, Skype, Whatsapp, Signal and so on. The centralization also made it quite easy to censor people or manipulate their view by determining the content presented to them. The algorithms behind the Facebook news feed or the “what you missed” in Twitter are very clever — or so we assume, as we don’t know how they work or determine what is important.

The last few years many initiatives started to solve this problem in various ways, for example by developing distributed social networks. I work in the area of liberating people who share and sync all sort of data. We saw the rise of successfully projects such as ownCloud, Pydio and now of course Nextcloud. They all have in common that they built Free Software platforms based to a large extend on Open Standards to allow people to host, edit and share their data without giving up control and privacy. This was a huge step in creating more competition and restoring decentralized structures. But it also had one big drawback. It created many small islands. You could only collaborate with people on the same server, but not with others who run their own server. This leads us to the concept of federated cloud sharing.

Server to Server Sharing

The first version of this ideas was implemented in ownCloud 7.0 as “Server to Server Sharing”. ownCloud already knew the concept of sharing anonymous links with people outside of the server. And, as ownCloud offered both a WebDAV interface and could mount external WebDAV shares, it was possible to manually hook a ownCloud into another ownCloud server. Therefore the first obvious step was to add a “Add to your ownCloud” button to this link shares, allowing people to connect such public links with their cloud by mounting it as a external WebDAV resource.

Federated Cloud Sharing

Server to server sharing already helped a lot to establish some bridges between many small islands created by the ability to self-host your cloud solution. But it was still not the kind of integration people where used to from the large centralized services and it only worked for ownCloud, not across various open source file sync and share solutions.

federated-cloud-id

The next iteration of this concept introduced what we called a “federated cloud ID”, which looks similar to a e-mail address and, like email, refers to a user on a specific server. This ID could then be used in the normal share dialog to share files with people on a different server!

share dialog - federated cloud id

The way servers communicate with each other in order to share a file with a user on a different server was publicly documented with the goal to create a standardized protocol. To further the protocol and to invite others to implement it we started the Open Cloud Mesh project together with GÉANT, an European research collaboration initiative. Today the protocol is already implemented by ownCloud, Pydio and now Nextcloud. This enables people to seamlessly share and collaborate, no matter if everyone is on the same server or if people run their own cloud server based on one of the three supporting servers.

Trusted Servers

In order to make it easier to find people on other servers we introduced the concept of “trusted servers” as one of our last steps. This allows administrator to define other servers they trust. If two servers trust each other they will sync their user lists. This way the share dialogue can auto-complete not only local users but also users on other trusted servers. The administrator can decide to define the lists of trusted servers manually or allow the server to auto add every other server to which at least one federated share was successfully created. This way it is possible to let your cloud server learn about more and more other servers over time, connect with them and increase the network of trusted servers.

federation

Open Challenges: where we’re taking Federated Cloud Sharing

Of course there are still many areas to improve. For example the way you can discover users on different server to share with them, for which we’re working on a global, shared address book solution. Another point is that at the moment this is limited to sharing files. A logical next step would be to extend this to many other areas like address books, calendars and to real-time text, voice and video communication and we are, of course, planning for that. I will write about this in greater detail in on of my next blogs but if you’re interested in getting involved, you are invited to check out what we’re up to on GitHub and of course, you can contact me any time.

Road Ahead


Road ahead

CC BY 2.0 by Nicholas A. Tonelli

I just realized that at June, 1 it is exactly four years since I joined ownCloud Inc. That’s a perfect opportunity to look back and to tell you about some upcoming changes. I will never forget how all this get started. It was FOSDEM 2012 when I met Frank, we already knew each other from various Free Software activities. I told him that I was looking for new job opportunities and he told me about ownCloud Inc. The new company around the ownCloud initiative which he just started together with the help of others. I was directly sold to the idea of ownCloud and a few months later I was employee number six at ownCloud Inc.

This was a huge step for me. Before joining ownCloud I worked as a researcher at the University of Stuttgart, so this was the first time I was working as a full-time software engineer on a real-world project. I also didn’t write any noteworthy PHP code before. But thanks to a awesome community I got really fast into all the new stuff and could speed up my contributions. During the following years I worked on many different aspects of ownCloud, from sharing, over files versions to the deleted files app up to a complete re-design of the server-side encryption. I’m especially happy that I could contribute substantial parts to a feature called “Federated Cloud Sharing”, from my point of view one of the most important feature to move ownCloud to the next level. Today it is not only possible to share files across various ownCloud servers but also between other cloud solutions like Pydio.

But the technical part is only a small subset of the great experience I had over the last four years. Working with a great community is just amazing. It is important to note that with community I mean everyone, from co-workers and students to people who contributed great stuff to ownCloud in their spare time. We are all ownCloud, there should be no distinction! We not only worked together in a virtual environment but meet regularly in person at Hackathons, various conferences and at the annual ownCloud conference. I met many great people during this time which I can truly call friends today. I think this explains why ownCloud was never just a random job to me and why I spend substantial parts of my spare time going to conferences, giving talks or helping at booths. ownCloud combined all the important parts for me: People, Free Software, Open Standards and Innovation.

Today I have to announce that I will move on. May, 25 was my last working day at the ownCloud company. This is a goodbye and thank you to ownCloud Inc. for all the opportunities the company provided to me. But it is in no way a goodbye to all the people and to ownCloud as a project. I’m sure we will stay in contact! That’s one of many great aspects of Free Software. If it is done right a initiative is much more than any company which might be involved. Leaving a company doesn’t mean that you have to leave the people and the project behind.

Of course I will continue to work on Free Software and with great communities, especially I have no plans to leave the ownCloud community. Actually I hope that I can even re-adjust my Free Software and community focus in the future… Stay tuned.

Federated Sharing – What’s new in ownCloud 9.0


Privacy, control and freedom was always one of the main reasons to run your own cloud instead of storing your data on a proprietary and centralized service. Only if you run your own cloud service you know exactly where your data is stored and who can access it. You are in control of your data. But this also introduces a new challenge. If everyone runs his own cloud service it become inevitable harder to share pictures with your friends or to work together on a document. That’s the reason why we at ownCloud are working at a feature called Federated Cloud Sharing. The aim of Federated Cloud Sharing is to close this gap by allowing people to connect their clouds and easily share data across different ownCloud installations. For the user it should make no difference whether the recipient is on the same server or not.

What we already had

The first implementation of Federated Cloud Sharing was introduced with ownCloud 8.0. Back then it was mainly a extension of the already existing feature to share a file or folder with a public link. People can create a link and share it with their friends or colleagues. Once they open the link in a browser they will see a button called “Add to your ownCloud” which enables them to mount the share as a WebDAV resource to their own cloud.

add-to-your-owncloud

With ownCloud 8.1 we moved on and added the Federated Cloud ID as a additional way to initiate a remote share. The nice thing is that it basically works like a email address. Every ownCloud user automatically gets a ID which looks similiar to john@myOwnCloud.org. Since ownCloud 8.2 the users Federated Cloud ID is shown in the personal settings.

federated-cloud-id

To share a file with a user on a different ownCloud you just need to know his Federated Cloud ID and enter it to the ownCloud share dialog. The next time the recipient log-in to his ownCloud he will get a notification that he received a new share. The user can now decide if he wants to accept or decline the remote share. In order to make it easier to remember the users Federated Cloud ID the Contacts App allows you to add the ID to your contacts. The share dialog will automatically search the address books to auto-complete the Federated Cloud IDs.

What’s new in ownCloud 9.0

With ownCloud 9.0 we made it even easier to exchange the Federated Cloud IDs. Below you can see the administrator setting for the new Federation App, which will be enabled by default.

federation

The option “Add server automatically once a federated share was created successfully” is enabled by default. This means, that as soon as a user creates a federated share with another ownCloud, either as a recipient or as a sender, ownCloud will add the remote server to the list of trusted ownClouds. Additionally you can predefined a list of trusted ownClouds. While technically it is possible to use plain http I want to point out that I really recommend to use https for all federated share operations to secure your users and their data.

What does it mean that two ownClouds trust each other? ownCloud 9.0 automatically creates a internal address book which contains all users accounts. If two ownClouds trust each other they will start to synchronize their system address books. In order to synchronize the system address books and to keep them up-to-date we use the well known and widespread CardDAV protocol. After the synchronization was successful ownCloud will know all users from the trusted remote servers, including their Federated Cloud ID and their display name. The share dialog will use this information for auto-completion. This allows you to share files across friendly ownClouds without knowing more than the users name. ownCloud will automatically find the corresponding Federated Cloud ID and will suggest the user as a recipient of your share.

The screen-shot of the new Federation App shows a status indicator for each server with three different states: green, yellow and red. Green means that both servers are connected and the address book was synced at least once. In this state auto-completion should work. Yellow means that the initial synchronization is still in progress. Creating a secure connection between two ownCloud servers and syncing the users happens in the background. This can take same time, depending on the background job settings of your ownCloud and the settings of the remote server. If the indicator turns red something went wrong in a way that it can’t be fixed automatically. ownCloud will not try to reestablish a connection to the given server. To reconnect to the remote server you have to remove the server and add it again.

If the auto-add option is enabled, the network of known and trusted ownClouds will expand every time a user on your server establish a new federated share. The boundaries between local users and remote users will blur. Each user will stay in control of his data, stored on his personal cloud but from a collaborative point of view everything will work as smooth as if all users would be on the same server.

What will come next? Of course we don’t want to stop here. We will continue to make it as easy as possible to stay in control of your data and at the same time share your files with all the other users and clouds out there. Therefor we work hard to document and standardize our protocols and invite other cloud initiatives to join us to create a Federation of Clouds, not only across different ownCloud servers but also across otherwise complete different cloud solutions.

The ownCloud Public Link Creator


ownCloud Share Link Creator - Context Menu

ownCloud Share Link Creator – Context Menu

Holiday season is the perfect time to work on some stuff on your personal ToDo list. ownCloud 6 introduced a public REST-style Share-API which allows you to call various share operations from external applications. Since I started working on the Share-API I thought about having a simple shell script on my file manager to automatically upload a file and generate a public link for it… Here it is!

I wrote a script which can be integrated in the Thunar file manager as a “custom action”. It is possible that the program also works with other file managers which provide similar possibilities, e.g Nautilus. But until now I tested and used it with Thunar only. If you try the script with a different file manager I would be happy to hear about your experience.

ownCloud Share Link Creator - File Upload

ownCloud Share Link Creator – File Upload

If you configure the “custom action” in Thunar, make sure to pass the paths of all selected files to the program using the “%F” parameter. The program expects the absolute path to the files. In the “Appearance and Conditions” tab you can activate all file types and directories. Once the custom action is configured you can execute the program from the right-click context menu. The program works for all file types and also for directories. Once the script gets executed it will first upload the files/directories to your ownCloud and afterwards it will generate a public link to access them. The link will be copied directly to your clipboard, additionally a dialog will inform you about the URL. If you uploaded a single file or directory than the file/directory will be created directly below your default target folder as defined in the shell script. If you selected multiple files, than the program will group them together in a directory named with the current timestamp.

This program does already almost everything I want. As already said, it can upload multiple files and even directories. One think I want to add in the future is the possibility to detect a ownCloud sync folder on the desktop. If the user selects a file in the sync folder than the script should skip the upload and create the share link directly.

Edit: In the meantime I got feedback that the script also works nicely with Dolphin, Nautilus and Nemo

Introduction to the new ownCloud Encryption App


Last weekend we released a first preview version of the new encryption app. This wouldn’t be possible without the work done by Sam Tuke and Florin Peter. Thanks a lot for all your work! Let me take the opportunity to tell you some details about the app, what it does and how it works.

The encryption app for ownCloud 5 was a complete re-write. We moved from the relatively weak blowfish algorithm to the more secure AES algorithm. The complete encryption is built on top of OpenSSL a well-known and tested encryption library. Further, the encryption app is integrated into ownCloud seamlessly. This means that the encrypt and decrypt happens transparently so that you can still use all the other features from ownCloud like sharing, different viewer apps, WebDAV access etc.

To make this possible, we decided to perform the encryption server-side. Still the architecture allows us to implement client-side encryption as an additional option later. Server-side encryption is especially interesting for users who also use the external storage app. Combining the external storage app with the encryption app allows you to use external storage without giving any 3rd-party provider access to your data.

ownCloud uses the users log-in password for encryption. This means that you should choose a strong password in order to protect your data. It is important to know that by default a user will lose access to his data if he loses his log-in password. As an additional feature the administrator can generate a recovery key which allows him to recover user data. Once this feature is activated in the administrator settings every user can enable the recovery key in his personal settings. By default the recovery key is disabled. Every user can decide for himself whether he wants this additional protection against password loss or not. Since we are using server-side encryption this feature does not reduce the security. Keep in mind that your ownCloud administrator will always be able to intercept your data because everything gets encrypted and decrypted at the server. Since ownCloud is Free Software you can choose a trustworthy administrator freely or decide to be your own administrator if you wish.

Let’s talk about some technical details and how the encryption works. The encryption is based on three different keys: every user has a private/public key-pair, every file has a file-key and to give multiple users access to a file we have share-keys.

Every user has an asymmetric 4096-bit strong key-pair which consists of a private and a public key. The private key is encrypted with the users log-in password, for the encryption AES-128 is used. Additionally there are up to two system-wide key-pairs: One for public link shares which allows ownCloud to decrypt files which are shared as public link and if enabled the recovery-key-pair.

In order to not always have to encrypt and decrypt large files we have introduced the file-keys which are 183 byte strong ASCII keys. The file-key is used to encrypt the users file symmetrically with AES-128. Than the file-key gets encrypted with the public keys from all users with access to the file. This means that if a user gets added or removed from a file we only have to re-encrypt the small file-key instead of the whole file.

Every time a file-key gets encrypted to multiple users OpenSSL generates for each user an additional share-key. Only the combination of the users private key with the corresponding share-key enables the user to decrypt the given file again.

Everybody is welcome to test the new encryption app and report issues on our mailing list or preferable directly on GitHub. But keep in mind that this is a preview version, you should always have a backup of your unencrypted data!

A new toy arrived


Thin Client

My little new toy arrived! It’s a Thin Client Tux@Home Q-Box 270 (Intel Atom) with 1GB RAM and 500GB hard disk. One of the nice things about the device is the low power consumption, only 10-15Watt. This is important because I want to use the device as a small home server. To avoid paying the “windows tax / proprietary software tax” I bought the computer at ixsoft.de, a online shop which sells hardware with GNU/Linux pre-installed. It came with Fedora GNU/Linux and now runs the Debian GNU/Linux (Squeeze) operating system which I consider more suitable for the tasks I want to use the device.

Now the more interesting part: What software / services should run on this little helper? In a first step I want to enable ssh access from outside, install an IRC bouncer and a subversion (SVN) server. The next think I would like to install is some kind of address book and calendar which can be used from any device (desktop computer, laptop, smart phone). But therefor I have to look into some solutions first. Kolab looks quite interesting. But I have to see how well it works with my software/hardware setup (Claws-Mail (Thunderbird) and Android). OwnCloud is another project I want to look at. Maybe this could be a nice solution for some file hosting.

If you have a tip for a good address book and calender solution or any other fancy idea what could be done with the Thin Client than drop me a mail or add a comment.