Cloud Federation – Getting Social


Clouds getting Social

Clouds getting Social

With Nextcloud 11 we continue to work on one of our hot topics: Cloud Federation. This time we focus on the social aspects. We want to make it as easy as possible for people to share their contact information. This enabled users to find each other and to start sharing. Therefore we extended the user profile in the personal settings. As the screenshot at the top shows, users can now add a wide range of information to their personal settings and define the visibility for each of them by clicking on the small icon next to it.

Privacy first

Change visibility of personal settings

Change visibility of personal settings

We take your privacy serious. That’s why we provide fine grained options to define the visibility of each personal setting. By default all new settings will be private and all settings which already exists before will have the same visibility as on Nextcloud 10 and earlier. This means that the users full name and avatar will only be visible to users on the same Nextcloud server, e.g. through the share dialog. If enabled by the administrator, this values, together with the users email address, will be synced with trusted servers to allow users from trusted servers to share with each other seamlessly.

As shown at the screenshot at the right we provide three levels of visibility: “Private”, “Contacts” and “Public”. Private settings will be only visible to you, even users on the same server will not have access to it. The only exceptions are the avatar and the full name because this are central data used at Nextcloud for activities, internal shares, etc. Settings which are set to “Contacts” will be shared with users on the same server and trusted servers, defined by the administrator of the Nextcloud server. Public data will be synced to a global and public address book.

Introducing the global address book

The best real world equivalent to the global address book is a telephone directory. For a new phone number people can chose to publish their phone number together with their name and address to a public telephone directory to enable other people to find them. The global address book follows the same pattern. By default nothing gets published to the global address book. Only if the user sets at least one value in their personal settings to “Public”. In this case all the public data will be synced to the global address book together with the users Federated Cloud ID. Users can remove their data at any time again by simply setting their personal data back to “Contacts” or “Private”.

In order to use the global address book as a source to find new people, this lookup needs to be enabled explicitly in the “Federated Cloud Sharing” settings by the administrator. For privacy reasons this is disabled by default. If enabled the share dialog of Nextcloud will query the global address book every time a user wants to share a file or folder, and suggest people found in the global address book. In the future there might be a dedicated button to access the global address book, both for performance reasons and to make the feature more discoverable.

Future work

The global address book can return many results for a given name. How do we know that we share with the right person? Therefore we want to add the possibility to verify the users email address, website and Twitter handle in Nextcloud 12. As soon as this feature is implemented the global address book will only return users where at least one personal setting is verified and also visualize the verified data so that the user can use this information to pick the right person.

Further, I want to extend the meaning of “Contacts” in one of the next versions. The idea is that “Contacts” should not be limited to trusted servers but include the users personal contacts. For example the data set to “Contacts” could be shared with every person to which the user already established at least one federated share successfully, or to all contacts with a Federated Cloud ID in the users personal address book. This way we will move slowly in the direction of some kind of decentralized and federated social network based on the users address book. This will also enable users to easily push their new phone number or other personal data to all their friends and colleagues, things for which most people use centralized and proprietary services like so called “business networks” these days.

Another interesting possibility, made possible by the global address book is to move complete user accounts from one server to another. Given that the user published at least some basic information on the global address book, they could use it to announce their move to another server. Other Nextcloud servers could find this information and make sure that existing federated shares continue to work.

Transfer Public Links to Federated Shares


Transform Public Links to Federated Shares

Transform a public link to a federated share

Creating public links and sending them to your friends is a widely used feature of Nextcloud. If the recipient of a public link also has a Nextcloud or ownCloud account he can use the “Add to your Nextcloud” button to mount the content over WebDAV to his server. On a technical level all mounted public links use the same token, the one of the public link, to reference the shared file. This means that as soon as the owner removes the public link all mounts will disappear as well. Additionally, the permissions for public links are limited compared to normal shares, public links can only be shared read-only or read-write. This was the first generation of federated sharing which we introduced back in 2014.

A year later we introduced the possibility to create federated shares directly from the share dialog. This way the owner can control all federated shares individually and use the same permission set as for internal shares. Both from a user perspective and from a technical point of view this lead to two different ways to create and to handle federated shares. With Nextcloud 10 we finally bring them together.

Improvements for the owner

Public Link Converted to a Federated Share

Public link converted to a federated share for bjoern@myNextcloud.net

From Nextcloud 10 on every mounted link share will be converted to a federated share, as long as the recipient also runs Nextcloud 10 or newer. This means that the owner of the file will see all the users who mounted his public link. He can remove the share for individual users or adjust the permissions. For each share the whole set of permissions can be used like “edit”, “re-share” and in case of folder additionally “create” and “delete”. If the owner removes the original public link or if it expires all federated shares, created by the public link will still continue to work. For older installations of Nextcloud and for all ownCloud versions the server will fall-back to the old behavior.

Improvements for the user who mounts a public link

After opening a public link the user can convert a public link to a federated share by adding his Federated Cloud ID or his Nextcloud URL

After opening a public link the user can convert it to a federated share by adding his Federated Cloud ID or his Nextcloud URL

Users who receive a public link and want to mount it to their own Nextcloud have two options. They can use this feature as before and enter the URL to their Nextcloud to the “Add to your Nextcloud” field. In this case they will be re-directed to their Nextcloud, have to login and confirm the mount request. The owners Nextcloud will then send the user a federated share which he has to accept. It can happen that the user needs to refresh his browser window to see the notification.
Additionally there is a new and faster way to add a public link to your Nextcloud. Instead of entering the URL to the “Add to your Nextcloud” field you can directly enter your federated cloud ID. This way the owners Nextcloud will send the federated share directly to you and redirect you to your server. You will see a notification about the new incoming share and can accept it. Now the user also benefit from the new possibilities of the owner. The owner can give him more fine grained permissions and from the users point of view even more important, he will not lose his mount if the public link gets removed or expires.

Nextcloud 10 introduces another improvement in the federation area: If you re-share a federated share to a third server, a direct connection between the first and the third server will be created now so that the owner of the files can see and control the share. This also improves performance and the potential error rate significantly, avoiding having to go through multiple servers in between.

History and Future of Cloud Federation


Federated Cloud Sharing - Connect self-hosted, decentralized clouds

Federated Cloud Sharing – Connect self-hosted, decentralized clouds

I’m now working for about two years on something called Federated Cloud Sharing. It started on June, 23er 2014 with the release of ownCloud 7.0. Back then it was simply called “Server to Server sharing”. During all this years I never wrote about the broader ideas behind this technology, why we do it, what we achieved and where we are going.

Motivation

The Internet started as a decentralized network, meant to be resilient to disruptions, both due to accidents or malicious activity. This was one of the key factors which made the Internet successful. From the World Wide Web, over IRC, news groups, e-mail to XMPP. Everything was designed as decentralized networks, which is why if you are on the Google servers you can email people at Yahoo. Everybody can set up his own web server, e-mail or chat server and communicate with everyone else. Individuals up to large organisations could easily join the network, participate and build business without barriers. People could experiment with new innovative ideas and nobody had the power to stop them or to slow them down. This was only possible because all underlying technology and protocols were build on both Open Standards and Free Software.

This changed dramatically over the last ten years. Open and inclusive networks were replaced by large centralized services operated by large companies. In order to present yourself or your business in the public it was no longer enough to have your own website, you had to have a page on one or two key platforms. For communication it was no longer enough to have a e-mail address, or be on one of the many IRC or XMPP servers. Instead people expected that you have a account on one of the major communication platforms. This created huge centralized networks, with many problems for privacy, security and self-determination. To talk to everybody, you have to have an account on Facebook, at Google, Skype, Whatsapp, Signal and so on. The centralization also made it quite easy to censor people or manipulate their view by determining the content presented to them. The algorithms behind the Facebook news feed or the “what you missed” in Twitter are very clever — or so we assume, as we don’t know how they work or determine what is important.

The last few years many initiatives started to solve this problem in various ways, for example by developing distributed social networks. I work in the area of liberating people who share and sync all sort of data. We saw the rise of successfully projects such as ownCloud, Pydio and now of course Nextcloud. They all have in common that they built Free Software platforms based to a large extend on Open Standards to allow people to host, edit and share their data without giving up control and privacy. This was a huge step in creating more competition and restoring decentralized structures. But it also had one big drawback. It created many small islands. You could only collaborate with people on the same server, but not with others who run their own server. This leads us to the concept of federated cloud sharing.

Server to Server Sharing

The first version of this ideas was implemented in ownCloud 7.0 as “Server to Server Sharing”. ownCloud already knew the concept of sharing anonymous links with people outside of the server. And, as ownCloud offered both a WebDAV interface and could mount external WebDAV shares, it was possible to manually hook a ownCloud into another ownCloud server. Therefore the first obvious step was to add a “Add to your ownCloud” button to this link shares, allowing people to connect such public links with their cloud by mounting it as a external WebDAV resource.

Federated Cloud Sharing

Server to server sharing already helped a lot to establish some bridges between many small islands created by the ability to self-host your cloud solution. But it was still not the kind of integration people where used to from the large centralized services and it only worked for ownCloud, not across various open source file sync and share solutions.

federated-cloud-id

The next iteration of this concept introduced what we called a “federated cloud ID”, which looks similar to a e-mail address and, like email, refers to a user on a specific server. This ID could then be used in the normal share dialog to share files with people on a different server!

share dialog - federated cloud id

The way servers communicate with each other in order to share a file with a user on a different server was publicly documented with the goal to create a standardized protocol. To further the protocol and to invite others to implement it we started the Open Cloud Mesh project together with GÉANT, an European research collaboration initiative. Today the protocol is already implemented by ownCloud, Pydio and now Nextcloud. This enables people to seamlessly share and collaborate, no matter if everyone is on the same server or if people run their own cloud server based on one of the three supporting servers.

Trusted Servers

In order to make it easier to find people on other servers we introduced the concept of “trusted servers” as one of our last steps. This allows administrator to define other servers they trust. If two servers trust each other they will sync their user lists. This way the share dialogue can auto-complete not only local users but also users on other trusted servers. The administrator can decide to define the lists of trusted servers manually or allow the server to auto add every other server to which at least one federated share was successfully created. This way it is possible to let your cloud server learn about more and more other servers over time, connect with them and increase the network of trusted servers.

federation

Open Challenges: where we’re taking Federated Cloud Sharing

Of course there are still many areas to improve. For example the way you can discover users on different server to share with them, for which we’re working on a global, shared address book solution. Another point is that at the moment this is limited to sharing files. A logical next step would be to extend this to many other areas like address books, calendars and to real-time text, voice and video communication and we are, of course, planning for that. I will write about this in greater detail in on of my next blogs but if you’re interested in getting involved, you are invited to check out what we’re up to on GitHub and of course, you can contact me any time.

Federated Sharing – What’s new in ownCloud 9.0


Privacy, control and freedom was always one of the main reasons to run your own cloud instead of storing your data on a proprietary and centralized service. Only if you run your own cloud service you know exactly where your data is stored and who can access it. You are in control of your data. But this also introduces a new challenge. If everyone runs his own cloud service it become inevitable harder to share pictures with your friends or to work together on a document. That’s the reason why we at ownCloud are working at a feature called Federated Cloud Sharing. The aim of Federated Cloud Sharing is to close this gap by allowing people to connect their clouds and easily share data across different ownCloud installations. For the user it should make no difference whether the recipient is on the same server or not.

What we already had

The first implementation of Federated Cloud Sharing was introduced with ownCloud 8.0. Back then it was mainly a extension of the already existing feature to share a file or folder with a public link. People can create a link and share it with their friends or colleagues. Once they open the link in a browser they will see a button called “Add to your ownCloud” which enables them to mount the share as a WebDAV resource to their own cloud.

add-to-your-owncloud

With ownCloud 8.1 we moved on and added the Federated Cloud ID as a additional way to initiate a remote share. The nice thing is that it basically works like a email address. Every ownCloud user automatically gets a ID which looks similiar to john@myOwnCloud.org. Since ownCloud 8.2 the users Federated Cloud ID is shown in the personal settings.

federated-cloud-id

To share a file with a user on a different ownCloud you just need to know his Federated Cloud ID and enter it to the ownCloud share dialog. The next time the recipient log-in to his ownCloud he will get a notification that he received a new share. The user can now decide if he wants to accept or decline the remote share. In order to make it easier to remember the users Federated Cloud ID the Contacts App allows you to add the ID to your contacts. The share dialog will automatically search the address books to auto-complete the Federated Cloud IDs.

What’s new in ownCloud 9.0

With ownCloud 9.0 we made it even easier to exchange the Federated Cloud IDs. Below you can see the administrator setting for the new Federation App, which will be enabled by default.

federation

The option “Add server automatically once a federated share was created successfully” is enabled by default. This means, that as soon as a user creates a federated share with another ownCloud, either as a recipient or as a sender, ownCloud will add the remote server to the list of trusted ownClouds. Additionally you can predefined a list of trusted ownClouds. While technically it is possible to use plain http I want to point out that I really recommend to use https for all federated share operations to secure your users and their data.

What does it mean that two ownClouds trust each other? ownCloud 9.0 automatically creates a internal address book which contains all users accounts. If two ownClouds trust each other they will start to synchronize their system address books. In order to synchronize the system address books and to keep them up-to-date we use the well known and widespread CardDAV protocol. After the synchronization was successful ownCloud will know all users from the trusted remote servers, including their Federated Cloud ID and their display name. The share dialog will use this information for auto-completion. This allows you to share files across friendly ownClouds without knowing more than the users name. ownCloud will automatically find the corresponding Federated Cloud ID and will suggest the user as a recipient of your share.

The screen-shot of the new Federation App shows a status indicator for each server with three different states: green, yellow and red. Green means that both servers are connected and the address book was synced at least once. In this state auto-completion should work. Yellow means that the initial synchronization is still in progress. Creating a secure connection between two ownCloud servers and syncing the users happens in the background. This can take same time, depending on the background job settings of your ownCloud and the settings of the remote server. If the indicator turns red something went wrong in a way that it can’t be fixed automatically. ownCloud will not try to reestablish a connection to the given server. To reconnect to the remote server you have to remove the server and add it again.

If the auto-add option is enabled, the network of known and trusted ownClouds will expand every time a user on your server establish a new federated share. The boundaries between local users and remote users will blur. Each user will stay in control of his data, stored on his personal cloud but from a collaborative point of view everything will work as smooth as if all users would be on the same server.

What will come next? Of course we don’t want to stop here. We will continue to make it as easy as possible to stay in control of your data and at the same time share your files with all the other users and clouds out there. Therefor we work hard to document and standardize our protocols and invite other cloud initiatives to join us to create a Federation of Clouds, not only across different ownCloud servers but also across otherwise complete different cloud solutions.

The next Generation of Code Hosting Platforms


Source Code

CC BY-SA 2.0 by
Christiaan Colen

The last few weeks there has been a lot of rumors about GitHub. GitHub is a code hosting platform which tries to make it as easy as possible to develop software and collaborate with people. The main achievement from GitHub is probably to moved the social part of software development to a complete new level. As more and more Free Software initiatives started using GitHub it became really easy to contribute a bug fix or a new feature to the 3rd party library or application you use. With a few clicks you can create a fork, add your changes and send them back to the original project as a pull request. You don’t need to create a new account, don’t need to learn the tools used by the project, etc. Everybody is on the same platform and you can contribute immediately. In many cases this improves the collaboration between projects a lot. Also the ability to mention the developer of other projects easily in your pull request or issue improved the social interactions between developers and makes collaboration across different projects the default.

That’s the good parts of GitHub, but there are also bad parts. GitHub is completely proprietary which makes it impossible to fix or improve stuff by yourself or run it by your own. Benjamin Mako Hill already argued 2010 why this is a problem and why Free Software needs free tools. More and more people seems to realize that this can create serious problems and a large group of active and influential GitHub users sent a letter to GitHub which ends with:

“Hopefully none of these are a surprise to you as we’ve told you them before. We’ve waited years now for progress on any of them. If GitHub were open source itself, we would be implementing these things ourselves as a community — we’re very good at that!”

I can’t stress this argument enough. The Free Software community is a community of people who are used to do stuff and don’t just consume it. If we use a third party library and find a bug or need a feature we don’t just complain, instead we look at the code, try to fix it and provide a patch to upstream. We could do the same for the tools we use. But we need to be able to do it. It has to be Free Software.

Now a lot of rumors and discussion evolved around the news that GitHub is undergoing a full-blown overhaul as execs and employees depart. Some people even predict that this will be the end of GitHub.

Wait for it. Three months from now, GitHub introduces "features" no-one wants or needs. 12 months from now, the exodus.

— Pieter Hintjens (@hintjens) February 7, 2016

It seems that many people underestimated the lock-in effect of the new hosting platforms such as GitHub for a long time. Now they start to realize that it might be easy to export the git repository but what about the issue tracker, the wiki, CI integration, all the social interaction and collaboration between the projects, all the useful scripts written for the GitHub-API? You can’t clone all this stuff easily and move on.

I don’t want to go deeper into the discussion about what’s going on at GitHub and what will happen next. There are plenty of articles and discussions about it, you can read some of them if you follow the links in this blog.

At the moment the ESLint initiative discusses the option to move away from GitHub and by reading the comments you can get a idea about the lock-in effect I’m talking about. With the growing dissatisfaction and with people realizing that they are sitting in a “golden cage” I have the feeling that we might have a opportunity to think about the next generation of code hosting platforms and how they should look like.

Some of you may remember how Git come into existence, the tool which is used as the underlying technology of GitHub. Ironically, Git was born because of quite similar reasons for which the next generation source code hosting platforms might arise. Before Git, the Linux-Kernel developer community used BitKeeper. BitKeeper is a proprietary source control management system. The developer decided to use it because from a technical point of view BitKeeper was so much better than what we had until then, mainly SVN and CVS. The developer enjoyed the tool and didn’t thought about the problems such a dependency could create. At some point the copyright holder of BitKeeper had withdrawn gratis use of the product after claiming that Andrew Tridgell had reverse-engineered the BitKeeper protocols. The Linux-Kernel community had to move on and Linus Torvalds wrote Git.

Back to the next generation of source code hosting and collaboration platforms. It is easy to find Free Software to run your own git repository, a issue tracker and a wiki. But in 2016 I think that this is no longer enough. As described before, the crucial part is to connect software initiatives and developer to make the interaction between them as easy as possible. That’s why traditional code hosting platforms like for example Savannah are no longer a real option for many projects. I think the next generation code hosting platform needs to work in a decentralized way. Every project should be able to either host its own platform or chose a provider freely without loosing the connection to other software initiatives and developers. This development, from proprietary and centralized solutions to centralized Free Software solutions to federated Free Software solutions is something we already saw in the area of social networks and cloud services. Maybe it is worth looking at what they already achieved and how they did it.

To make the same transition happen for code hosting platforms we need implementations based on Free Software, Open Standards and protocols which enabled this kind of federation. The good news is that we already have most of them. Git by itself is already a distributed revision control system and doesn’t need a central server for collaboration. What’s missing is a nice web interfaces to glue all this parts together: a issue tracker, a wiki, good integration in Free Software CI tools, good APIs and of course Git. This will enable us to fork projects across servers, send pull requests, interact with the other developers and comment on issues no matter if they are on the same server or not. Chances are high that we will already find a suitable protocol by looking at the large amount of federated social networks. By choosing a exiting protocol of a established federated social network we could even provide a tight integration in traditional social networks which could provide additional benefits beyond what we already have. The hard part will be to pull all this together. Will it happen? I don’t know. But I hope that after we have seen the raise and fall of SourceForge, Google Code and maybe at some point GitHub we will move on to create something more sustainable instead of building the next data silo and wait until it fails again.