Hasher application - file integrity check

Hasher app serves for checksum computing and verifying. By using Hasher app can be checksums calculated and stored into so called extended attributes. Furthermore is possible to use those checksums to check file integrity, whether your files would not been changed during transfer between your local PC and Cesnet Data Storage or even during the stay in Cesnet Data Storage. The system automatically calculate checksums until 24 hours after you uploaded your files and save them into extended attributes.

Checksums in Linux environment - Hasher app

User app Hasher is intended for Cesnet Data Storage (DS) users. App Hasher is intended for checksums calculation. Those checksums allow user to check, whether his files were corrupted, or not.

Hasher app provides standard checksums algorithm MD5, SHA1, RMD160, SHA256, SHA384, SHA512 a SHA224. Default checksum algorithm is MD5.

Hasher app stored calculated checksums into so called extended attributes. So if you wish to preserve extended attributes after you transfer your data to DS, it is necessary to use transfer protocol, which allows you to preserve file extended attributes

In case you use for transferring your files from DS some protocol, which does not allow you to preserve extended attributes, you can export extended attributes into text file. Afterward you can download it and once more calculate checksums and verify calculated checksums with those ones which you exported and downloaded from DS. Please see further.

Control of Hasher app

If you will use command hasher without any parameter, you shall see following list of parameters.

-r recursive recursive checksum calculation
-d digest (type -d help) specify type of checksum algorithm (default MD5)
-s set digest saves calculated checksum into extended attributes
-e export digest exports calculated checksums into extended attributes
-p import digest imports checksums for defined files
-v verify digest calculates checksum and verify it with checksum stored into extended attributes
-h print this help help
-l follow symlinks follows symbolic links
-q quiet hide dumps of performed operation
-i list computed hashes dumps the list of the stored checksums

Using Hasher app onto DS

The system automatically calculate checksums until 24 hours after you uploaded your files and save them into extended attributes. Furthermore you can use them to check the integrity of your files using some of following methods.

Verifying checksum for my files onto DS
Verifying own checksums (e.g. SHA256) for my files onto DS
Using Hasher app to verify my data after transferring onto DS

Using Hasher app on your local PC

At first you have to make Hasher app installation from DS repository to use hasher app on your local PC. Further you can follow the guide bellow. This use case is required especially in cases, when you use any other transfer protocol except rsync.

Usage of Hasher app to check your data after transferring from DS

Verifying my file checksums on DS

Autonomous system automatically calculate MD5 checksums until 24 hours after you uploaded your data onto DS.

1. Verify checksum (default MD5) on particular file

hasher my_file
Processing my_file...
Actual hash of my_file is MD5: 92f7e6dfc18a25981a59a9f74298dd7
User checksum type MD5 verified OK on my_file

2. Verify checksum (default MD5) in a directory (recursion)

hasher -r my_dir
Walking down my_dir/
Adding my_dir//my_file1
Adding my_dir//my_file2
Adding my_dir//my_file3
Processing my_dir//my_file3...
Actual hash of my_dir//my_file3 is MD5: b26795c69684a7c9cc4a94ff70a95b71
User checksum type MD5 verified OK on my_dir//my_file3
Processing my_dir//my_file2...
Actual hash of my_dir//my_file2 is MD5: 9448a1bf333fadd2a57965ec38487b89
User checksum type MD5 verified OK on my_dir//my_file2
Processing my_dir//my_file1...
Actual hash of my_dir//my_file1 is MD5: e5828c564f71fea3a12dde8bd5d27063
User checksum type MD5 verified OK on my_dir//my_file1

Verify user defined checksums (e.g. SHA256) for my files on DS

1. Calculating and storing checksum for particular file using SHA256 algorithm

hasher -s -d SHA256 my_file
Processing my_file...
Actual hash of my_file is SHA256: 5e8cd142e6fa81bfd5ffea2516aaa69c837a33c9bcc954d2617229b626c17f

2. Calculating and storing SHA256 checksum for files in any directory (recursion)

hasher -r -s -d SHA256 my_dir
Walking down my_dir
Adding my_dir/my_file1
Adding my_dir/my_file2
Adding my_dir/my_file3
Processing my_dir/my_file3...
Actual hash of my_dir/my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd
Processing my_dir/my_file2...
Actual hash of my_dir/my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48
Processing my_dir/my_file1...
Actual hash of my_dir/my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772

3. Verify SHA256 checksum on particular file

hasher -d SHA256 my_file
Processing my_file...
Actual hash of my_file is SHA256: 5e8cd142e6fa81bfd5ffea2516aaa69c837a33c9bcc954d2617229b626c17f
User checksum type SHA256 verified OK on my_file

4. Verify SHA256 checksum in any directory (recursion)

hasher -r -d SHA256 my_dir
Walking down my_dir
Adding my_dir/my_file1
Adding my_dir/my_file2
Adding my_dir/my_file3
Processing my_dir/my_file3...
Actual hash of my_dir/my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd
User checksum type SHA256 verified OK on my_dir/my_file3
Processing my_dir/my_file2...
Actual hash of my_dir/my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48
User checksum type SHA256 verified OK on my_dir/my_file2
Processing my_dir/my_file1...
Actual hash of my_dir/my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772
User checksum type SHA256 verified OK on my_dir/my_file1

In case you decided to use other checksum algorithm than default MD5, it is necessary during each action (i.e. -s or -v) specify by using -d, which checksum algorithm supposed to be used (calculated/verified). List of available checksum algorithms can be found by using -d help parameter.

Using Hasher app to check the file integrity after upload onto DS

1. First is necessary to perform an export of checksums into external file. Checksum export must be performed on your local PC using the guide described in export section.

Verification of checksums must be performed at the same directory level, where the export was done. In case you export checksums using 1c export section you must make following verification at the same directory level, not inside of particular directory.

2. Further transfer desired files from your PC onto DS. Do not forget to transfer file checksums.txt. See export section.

3. Further we will import checksums using the method described in the previous section, i.e. import section, which will perform checksum verification step as well.

Using Hasher app for file verification after download from DS

That use case should be used preferably, if you use transfer protocol, which does not allow to transfer extended attributes (i.e. scp, ftp, cifs, nfs). The key stone is in that you can export calculated checksums into external file (*.txt) and then you can transfer all of your files including *.txt file onto your local PC. Afterward you can use Hasher app using the guide below and verify checksums contained in *.txt file on your local PC.

Now you can continue as follows:

1. Export checksums of selected files into external file

1a) Export of default MD5 checksums

hasher -e my_file1 my_file2 my_file3 > checksums.txt
cat kontrolni_soucty.txt 
MD5 e5828c564f71fea3a12dde8bd5d27063 my_file1
MD5 9448a1bf333fadd2a57965ec38487b89 my_file2
MD5 b26795c69684a7c9cc4a94ff70a95b71 my_file3

You can use following command for all files in particular directory:

hasher -e * > checksums.txt

1b) Export of user defined checksums, here an example SHA256

hasher -e -d SHA256 my_file1 my_file2 my_file3 > checksums.txt
cat checksums.txt 
SHA256 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 my_file1
SHA256 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 my_file2
SHA256 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd my_file3

1c) Export of checksums in particular directory, here an example SHA256

hasher -e -r -d SHA256 my_dir > checksums.txt
cat checksums.txt 
SHA256 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd my_dir/my_file3
SHA256 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48 my_dir/my_file2
SHA256 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772 my_dir/my_file1

2. Now transfer desired files from DS onto your local Do not forget to transfer file checksums.txt too.

To continue it is necessary to perform installation of Hasher app on your local PC, which can be done using this guide, in case you did not do it during any of previous steps.

3. Now you can run on your local PC checksums import and verification. Here we can use basic command hasher , which will perform verification of all files, original files even newly downloaded, which checksums are located in the checksums.txt file.

Verification of checksums must be performed at the same directory level, where the export was done. In case you export checksums using 1c export section you must make following verification at the same directory level, not inside of particular directory.

3a) Import and verification of default MD5 checksums

hasher -p checksums.txt; hasher *
Processing checksums.txt...
Actual hash of checksums.txt is MD5: 53d2d77158854967d3dde13a3425dd
User checksum not set on 'checksums.txt' nothing to verify.
Processing my_file1...
Actual hash of my_file1 is MD5: e5828c564f71fea3a12dde8bd5d27063
User checksum type MD5 verified OK on my_file1
Processing my_file2...
Actual hash of my_file2 is MD5: 9448a1bf333fadd2a57965ec38487b89
User checksum type MD5 verified OK on my_file2
Processing my_file3...
Actual hash of my_file3 is MD5: b26795c69684a7c9cc4a94ff70a95b71
User checksum type MD5 verified OK on my_file3

3b) Import and verification user defined checksums, here SHA256 (file checksums.txt must be exported in accordance with 1b)

hasher -p checksums.txt; hasher -d SHA256 *
Processing checksums.txt...
Actual hash of checksums.txt is SHA256: 885e818d42b1d94d247b65f91e50ea097c87cfb2a9b471ab07039371ddd
User checksum not set on 'checksums.txt' nothing to verify.
Processing my_file1...
Actual hash of my_file1 is SHA256: 11a77c3d96c06974b53d7f40a577e6813739eb5c811b2a86f59038ea90add772
User checksum type SHA256 verified OK on my_file1
Processing my_file2...
Actual hash of my_file2 is SHA256: 4551db5fd4d56e27be71a8a9437cfaa4342b8e96a326e2d6427b3aaa5a48
User checksum type SHA256 verified OK on my_file2
Processing my_file3...
Actual hash of my_file3 is SHA256: 2f878d89f48b51fc327951d28d71ffb2b8758c1741feb46d247f1c038c67bdd
User checksum type SHA256 verified OK on my_file3

Installation of the Hasher app on OS Linux

The easiest method how to install Hasher app is by using Cesnet DS repository. Repository location for various linux distribution and the the method of their adding can be found below. In case you operate unsupported distribution you have to perform own compilation

Ubuntu14.04-17.04 (i386/amd64)

wget ftp://homeproj.cesnet.cz/apt/du/du_squeeze.list -O /etc/apt/sources.list.d/cesnet-du.list
wget ftp://homeproj.cesnet.cz/apt/du/APT-GPG-KEY-du -O - | apt-key add -
apt-get update
apt-get install hasher.x86_64

Debian7-9 (i386/amd64)

wget ftp://homeproj.cesnet.cz/apt/du/du_jessie.list -O /etc/apt/sources.list.d/cesnet-du.list
wget ftp://homeproj.cesnet.cz/apt/du/APT-GPG-KEY-du -O - | apt-key add -
apt-get update
apt-get install hasher.x86_64

openSUSE13.2 & 42.2 (pouze x86_64)

zypper addrepo ftp://homeproj.cesnet.cz/rpm/du-opensuse/stable/x86_64/
zypper install --no-recommends hasher.x86_64

CentOS6 (pouze x86_64)

wget https://du.cesnet.cz/_media/cs/navody/nfs/cesnet-du.repo -O /etc/yum.repos.d/cesnet-du.repo
yum install hasher.x86_64

CentOS7 (pouze x86_64)

wget https://du.cesnet.cz/_media/cs/navody/nfs/cesnet-du7.repo -O /etc/yum.repos.d/cesnet-du.repo
yum install hasher.x86_64

Compilation of the Hasher app for unsupported distributions

For the installation itself it is necessary to install packages countaing required libraries for following compilation. For Ubuntu and CentOS are following packages. Pro samotnou instalaci je nejdříve nezbytné doinstalovat balíky obsahující potřebné knihovny pro následnou kompilaci. V případě Ubuntu a

Ubuntu 16.10

sudo apt-get install libgnutls28-dev libattr1-dev libglib2.0-dev libgcrypt20 libgcrypt11-dev

CENTOS 7

yum install spice-glib-devel.x86_64 libattr-devel.x86_64 libgcrypt-devel.x86_64 libgcrypt.x86_64 gnutls-devel.x86_64

Now we can continue as follows:

1. Source files of the Hasher app can be found at this link.
2. Go to ../gnutls/hasher directory 3. Now it is necessary to perform compilation of hasher.c file on your local PC using following command.

gcc -g -o hasher hasher.c ``pkg-config --cflags --libs glib-2.0`` -Wall -I. -lgcrypt -lgnutls

4. Now you can move hasher file into /usr/local/bin directory. Now you should be able to use common command hasher in command line, which is described in the previous sections.

Now you can continue with file verification for download or upload.

Oddělení datových úložišť CESNET

Sidebar