From Kerrighed


[edit] Latest news

[edit] Test kDFS with VMs

Looking for contributors, don't hesitate to contact us !

[edit] kernel/kerrighed Distributed File System

kDFS aims at providing an integrated cluster file system for High Performance Computing.

Based on several concepts suggested in KerFS (Kerrighed 1.02), the kernel Distributed File system has been develop from scratch. One of the main idea consists in developing a distributed file system pluggable under the VFS and only based on the KDDM component of Kerrighed. The KDDM features are used to build a cooperative cache for both data and meta-data using all available memory in the cluster.

  • Further informations is available in the following pdf presentation:

kDFS Overview: current state and main objectives (june 2007)

kDFS, Alpha version (october 2007)

A research report addressing design and current implementation of kDFS is available there (december 2007). Feel free to contact us if you are interested by a longer version or any further information.

Contact : Adrien Lebre / Pierre Riteau

[edit] Developers

[edit] Adrien Lèbre

Debugging/Finalizing the current code. Working on I/O scheduler strategies and dynamical RAID placement. Since september 2008, his activities have been focusing on the integration issues between kDFS and the other cluster services (by exploiting aspect oriented programming approaches). Currently, he's working on adding striping mechanisms in the core of kDFS.

[edit] Pierre Riteau

During His Computer Science Master, He worked on adding file checkpointing capabilities to kDFS, using Copy-On-Write and data distribution over different nodes. In mid-term, this will be used by the checkpoint/restart services of Kerrighed to improve fault tolerance. During summer 2008 Pierre worked on kDFS to port it to the kddm-standalone framework, debug/finalize the code, add diskless support, etc. Currently, Pierre maintains the kDFS branch for the XtreemOS Consortium and he's involved in the development of the striping mechanism.

[edit] Past Contributors

[edit] Marko Obrovac

He worked on the kDFS superblock management during summer 2008. At the same time, He studied several disk models. Based on this analysis, several I/O probes will be implemented as part of kDFS in order to provide better global load-balancing scheduling. This approach should help the scheduler to improve global cluster performances (contact Marko Obrovac).

[edit] Latest news

[edit] June 2009: Striping in kDFS

We are currently implementing the striping policy in kDFS. First results shoud be available by July 2009.

[edit] May 2009: kDFS file checkpointing presented at CCGrid 2009

Our paper on the file checkpointing features implemented in kDFS (Handling Persistent States in Process Checkpoint/Restart Mechanisms for HPC Systems) has been selected for the CCGrid 2009 conference and will be presented there by Pierre. You can read a technical report covering these features here.

[edit] November 2008: kDFS at SuperComputing 2008 - booth 2203

Pierre will present a complete demo of kDFS and will describe the latest kDFS developments. Don't hesitate to visit the INRIA booth.

[edit] August 2008: kDFS is available as a standalone kernel module

kDFS was recently ported to work on the kDDM standalone framework. This means kDFS can now be used without Kerrighed. More information is available by downloading the kdfs-standalone branch in the Kerrighed repository: svn checkout svn://scm.gforge.inria.fr/svn/kerrighed/branches/kdfs-standalone

[edit] May 1 2008: kDFS design will be presented at the Europar Conference

The paper "Reducing Development Complexity in Distributed Environments" has been accepted to the Europar Conference 2008. This paper introduces the design of kDFS based on the kDDM layer. This paper will be officially available by the end of august 2008. In the meantime, you can have a look at our internal research report !

[edit] Apr 29 2008: Looking for a new contributor

We are looking for a new contributor to port kDFS from the Kerrighed framework to the kDDM standalone module under Linux 2.6.25. Even if the main developments will still focus on the kDFS integration within Kerrighed, we would like to provide kDFS as an independent module. C programmers are welcome, don't hesitate to contact us !

[edit] Apr 28 2008: kDFS passes the NTFS-3G POSIX File System Test Suite

kDFS is able to successfully complete the NTFS-3G POSIX File System Test Suite. 

All tests successful.
Files=184, Tests=1950, 161 wallclock secs ( 9.47 cusr + 82.34 csys = 91.81 CPU)

[edit] Virtual machines

If you want to play with kDFS, we provide fully configured VMWare & QEMU virtual machines. The first virtual machine is a DHCP/NFS server that allows to netboot four other virtual machines. These four virtual machines can be used to create Kerrighed clusters running kDFS.

The IPs 192.168.0.1, 192.168.0.2, 192.168.0.3, 192.168.0.4 and 192.168.0.16 are used by the virtual machines, be careful if you have the same IPs in your network.

WARNING:

The QEMU virtual machines are rather slow, so you're encouraged to play with the VMWare machines.

The kDFS code provided in the different VMs is outdated (feb 2008), please check the README file to update it from our SVN!

[edit] VMWare virtual machines

You can use these virtual machines with VMWare Fusion on Mac OS X. On Linux you can use VMWare Player. Please note that the given images do not run with VMWare 1.0.4 (VMWare 2 is required). Before starting the VMWare virtual machines, please read this readme file.

Links to the files:

[edit] QEMU virtual machines

In order to run the QEMU virtual machines, you need to get QEMU. There's also a Mac OS X port, but you should be aware that it hasn't been tested. If you successfully ran all the kDFS configuration on Mac OS X, please let us know. Before starting the QEMU virtual machines, please read this readme file.

Links to the files:

Retrieved from "http://www.kerrighed.org/wiki/index.php/KernelDevelKdFS"