Hasher app serves for checksum computing and verifying. By using Hasher app can be checksums calculated and stored into so called extended attributes. Furthermore is possible to use those checksums to check file integrity, whether your files would not been changed during transfer between your local PC and Cesnet Data Storage or even during the stay in Cesnet Data Storage. The system automatically calculate checksums until 24 hours after you uploaded your files and save them into extended attributes.
User app Hasher is intended for Cesnet Data Storage (DS) users. App Hasher is intended for checksums calculation. Those checksums allow user to check, whether his files were corrupted, or not.
Hasher app provides standard checksums algorithm MD5, SHA1, RMD160, SHA256, SHA384, SHA512 a SHA224. Default checksum algorithm is MD5.
If you will use command hasher without any parameter, you shall see following list of parameters.
The system automatically calculate checksums until 24 hours after you uploaded your files and save them into extended attributes. Furthermore you can use them to check the integrity of your files using some of following methods.
Verifying checksum for my files onto DS
Verifying own checksums (e.g. SHA256) for my files onto DS
Using Hasher app to verify my data after transferring onto DS
At first you have to make Hasher app installation from DS repository to use hasher app on your local PC. Further you can follow the guide bellow. This use case is required especially in cases, when you use any other transfer protocol except rsync.
Usage of Hasher app to check your data after transferring from DS
Autonomous system automatically calculate MD5 checksums until 24 hours after you uploaded your data onto DS.
1. Verify checksum (default MD5) on particular file
hasher my_file Processing my_file... Actual hash of my_file is MD5: 92f7e6dfc18a25981a59a9f74298dd7 User checksum type MD5 verified OK on my_file
2. Verify checksum (default MD5) in a directory (recursion)
hasher -r my_dir Walking down my_dir/ Adding my_dir//my_file1 Adding my_dir//my_file2 Adding my_dir//my_file3 Processing my_dir//my_file3... Actual hash of my_dir//my_file3 is MD5: b26795c69684a7c9cc4a94ff70a95b71 User checksum type MD5 verified OK on my_dir//my_file3 Processing my_dir//my_file2... Actual hash of my_dir//my_file2 is MD5: 9448a1bf333fadd2a57965ec38487b89 User checksum type MD5 verified OK on my_dir//my_file2 Processing my_dir//my_file1... Actual hash of my_dir//my_file1 is MD5: e5828c564f71fea3a12dde8bd5d27063 User checksum type MD5 verified OK on my_dir//my_file1
1. Calculating and storing checksum for particular file using SHA256 algorithm
hasher -s -d SHA256 my_file Processing my_file... Actual hash of my_file is SHA256: 5e8cd142e6fa81bfd5ffea2516aaa69c837a33c9bcc954d2617229b626c17f
2. Calculating and storing SHA256 checksum for files in any directory (recursion)
hasher -r -s -d SHA256 my_dir Walking down my_dir Adding my_dir/my_file1 Adding my_dir/my_file2 Adding my_dir/my_file3 Processing my_dir/my_file3... Actual hash of my_dir/my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd Processing my_dir/my_file2... Actual hash of my_dir/my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 Processing my_dir/my_file1... Actual hash of my_dir/my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772
3. Verify SHA256 checksum on particular file
hasher -d SHA256 my_file Processing my_file... Actual hash of my_file is SHA256: 5e8cd142e6fa81bfd5ffea2516aaa69c837a33c9bcc954d2617229b626c17f User checksum type SHA256 verified OK on my_file
4. Verify SHA256 checksum in any directory (recursion)
hasher -r -d SHA256 my_dir Walking down my_dir Adding my_dir/my_file1 Adding my_dir/my_file2 Adding my_dir/my_file3 Processing my_dir/my_file3... Actual hash of my_dir/my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd User checksum type SHA256 verified OK on my_dir/my_file3 Processing my_dir/my_file2... Actual hash of my_dir/my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 User checksum type SHA256 verified OK on my_dir/my_file2 Processing my_dir/my_file1... Actual hash of my_dir/my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 User checksum type SHA256 verified OK on my_dir/my_file1
1. First is necessary to perform an export of checksums into external file. Checksum export must be performed on your local PC using the guide described in export section.
2. Further transfer desired files from your PC onto DS. Do not forget to transfer file checksums.txt. See export section.
3. Further we will import checksums using the method described in the previous section, i.e. import section, which will perform checksum verification step as well.
That use case should be used preferably, if you use transfer protocol, which does not allow to transfer extended attributes (i.e. scp, ftp, cifs, nfs). The key stone is in that you can export calculated checksums into external file (*.txt) and then you can transfer all of your files including *.txt file onto your local PC. Afterward you can use Hasher app using the guide below and verify checksums contained in *.txt file on your local PC.
Now you can continue as follows:
1. Export checksums of selected files into external file
1a) Export of default MD5 checksums
hasher -e my_file1 my_file2 my_file3 > checksums.txt cat kontrolni_soucty.txt MD5 e5828c564f71fea3a12dde8bd5d27063 my_file1 MD5 9448a1bf333fadd2a57965ec38487b89 my_file2 MD5 b26795c69684a7c9cc4a94ff70a95b71 my_file3
You can use following command for all files in particular directory:
hasher -e * > checksums.txt
1b) Export of user defined checksums, here an example SHA256
hasher -e -d SHA256 my_file1 my_file2 my_file3 > checksums.txt cat checksums.txt SHA256 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 my_file1 SHA256 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 my_file2 SHA256 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd my_file3
1c) Export of checksums in particular directory, here an example SHA256
hasher -e -r -d SHA256 my_dir > checksums.txt cat checksums.txt SHA256 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd my_dir/my_file3 SHA256 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 my_dir/my_file2 SHA256 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 my_dir/my_file1
2. Now transfer desired files from DS onto your local Do not forget to transfer file checksums.txt too.
3. Now you can run on your local PC checksums import and verification. Here we can use basic command hasher , which will perform verification of all files, original files even newly downloaded, which checksums are located in the checksums.txt file.
3a) Import and verification of default MD5 checksums
hasher -p checksums.txt; hasher * Processing checksums.txt... Actual hash of checksums.txt is MD5: 53d2d77158854967d3dde13a3425dd User checksum not set on 'checksums.txt' nothing to verify. Processing my_file1... Actual hash of my_file1 is MD5: e5828c564f71fea3a12dde8bd5d27063 User checksum type MD5 verified OK on my_file1 Processing my_file2... Actual hash of my_file2 is MD5: 9448a1bf333fadd2a57965ec38487b89 User checksum type MD5 verified OK on my_file2 Processing my_file3... Actual hash of my_file3 is MD5: b26795c69684a7c9cc4a94ff70a95b71 User checksum type MD5 verified OK on my_file3
3b) Import and verification user defined checksums, here SHA256 (file checksums.txt must be exported in accordance with 1b)
hasher -p checksums.txt; hasher -d SHA256 * Processing checksums.txt... Actual hash of checksums.txt is SHA256: 885e818d42b1d94d247b65f91e50ea097c87cfb2a9b471ab07039371ddd User checksum not set on 'checksums.txt' nothing to verify. Processing my_file1... Actual hash of my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 User checksum type SHA256 verified OK on my_file1 Processing my_file2... Actual hash of my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 User checksum type SHA256 verified OK on my_file2 Processing my_file3... Actual hash of my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd User checksum type SHA256 verified OK on my_file3
The easiest method how to install Hasher app is by using Cesnet DS repository. Repository location for various linux distribution and the the method of their adding can be found below. In case you operate unsupported distribution you have to perform own compilation
Ubuntu14.04-17.04 (i386/amd64)
wget ftp://homeproj.cesnet.cz/apt/du/du_squeeze.list -O /etc/apt/sources.list.d/cesnet-du.list wget ftp://homeproj.cesnet.cz/apt/du/APT-GPG-KEY-du -O - | apt-key add - apt-get update apt-get install hasher.x86_64
Debian7-9 (i386/amd64)
wget ftp://homeproj.cesnet.cz/apt/du/du_jessie.list -O /etc/apt/sources.list.d/cesnet-du.list wget ftp://homeproj.cesnet.cz/apt/du/APT-GPG-KEY-du -O - | apt-key add - apt-get update apt-get install hasher.x86_64
openSUSE13.2 & 42.2 (pouze x86_64)
zypper addrepo ftp://homeproj.cesnet.cz/rpm/du-opensuse/stable/x86_64/ zypper install --no-recommends hasher.x86_64
CentOS6 (pouze x86_64)
wget https://du.cesnet.cz/_media/cs/navody/nfs/cesnet-du.repo -O /etc/yum.repos.d/cesnet-du.repo yum install hasher.x86_64
CentOS7 (pouze x86_64)
wget https://du.cesnet.cz/_media/cs/navody/nfs/cesnet-du7.repo -O /etc/yum.repos.d/cesnet-du.repo yum install hasher.x86_64
For the installation itself it is necessary to install packages countaing required libraries for following compilation. For Ubuntu and CentOS are following packages. Pro samotnou instalaci je nejdříve nezbytné doinstalovat balíky obsahující potřebné knihovny pro následnou kompilaci. V případě Ubuntu a
Ubuntu 16.10
sudo apt-get install libgnutls28-dev libattr1-dev libglib2.0-dev libgcrypt20 libgcrypt11-dev
CENTOS 7
yum install spice-glib-devel.x86_64 libattr-devel.x86_64 libgcrypt-devel.x86_64 libgcrypt.x86_64 gnutls-devel.x86_64
Now we can continue as follows:
1. Source files of the Hasher app can be found at this link.
2. Go to ../gnutls/hasher directory
3. Now it is necessary to perform compilation of hasher.c file on your local PC using following command.
gcc -g -o hasher hasher.c ``pkg-config --cflags --libs glib-2.0`` -Wall -I. -lgcrypt -lgnutls
4. Now you can move hasher file into /usr/local/bin directory. Now you should be able to use common command hasher in command line, which is described in the previous sections.