File size distribution: Difference between revisions

 

;Task:

 

 

My suggestion is to sort by logarithmn of file size, since a few bytes here or there, or even a factor of two or three, may not be that significant.

 

 

Is your file system predominantly devoted to a large number of smaller files, or a smaller number of huge files?

<br><br>

 

=={{header|C}}==

The platform independent way to get the file size in C involves opening every file and reading the size. The implementation below works for Windows and utilizes command scripts to get size information quickly even for a large number of files, recursively traversing a large number of directories. Both textual and graphical ( ASCII ) outputs are shown. The same can be done for Linux by a combination of the find, ls and stat commands and my plan was to make it work on both OS types, but I don't have access to a Linux system right now. This would also mean either abandoning scaling the graphical output in order to fit the console buffer or porting that as well, thus including windows.h selectively.

===Windows===

#include<windows.h>

#include<string.h>

double scale;

FILE* fp;

if(argC==1)

printf("Usage : %s <followed by directory to start search from(. for current dir), followed by \n optional parameters (T or G) to show text or graph output>",argV[0]);

sprintf(commandString,"forfiles /p %s /s /c \"cmd /c echo @fsize\" 2>&1",startPath);

}

else if(strlen(argV[1])==1 && argV[1][0]=='.')

strcpy(commandString,"forfiles /s /c \"cmd /c echo @fsize\" 2>&1");

else

sprintf(commandString,"forfiles /p %s /s /c \"cmd /c echo @fsize\" 2>&1",argV[1]);

fileSizeLog[strlen(str)]++;

}

if(argC==2 || (argC==3 && (argV[2][0]=='t'||argV[2][0]=='T'))){

for(i=0;i<MAXORDER;i++){

}

}

else if(argC==3 && (argV[2][0]=='g'||argV[2][0]=='G')){

CONSOLE_SCREEN_BUFFER_INFO csbi;

 

max = fileSizeLog[0];

for(i=1;i<MAXORDER;i++)

(fileSizeLog[i]>max)?max=fileSizeLog[i]:max;

(max < csbi.dwSize.X)?(scale=1):(scale=(1.0*(csbi.dwSize.X-50))/max);

for(i=0;i<MAXORDER;i++){

printf("\nSize Order < 10^%2d bytes |",i);

}

}

}

return 0;

}

}

Invocation and textual output :

<pre>

{{libheader|POSIX}}

This works on macOS 10.15. It should be OK for Linux as well.

#include <locale.h>

#include <stdint.h>

total_size += file_size;

size_t index = 0;

++count[index];

} else if (flag == FTW_DNR) {

printf("Total file size: %'lu\n", total_size);

return EXIT_SUCCESS;

 

{{out}}

 

=={{header|C++}}==

#include <array>

#include <filesystem>

}

return EXIT_SUCCESS;

 

{{out}}

Number of files: 7,874

Total file size: 11,963,566,673 bytes

</pre>

 

=={{header|Go}}==

{{trans|Kotlin}}

 

import (

func main() {

fileSizeDistribution("./")

 

{{out}}

</pre>

=={{header|Haskell}}==

import Control.Concurrent.Chan (Chan, newChan, readChan,

writeChan, writeList2Chan)

import GHC.Conc (getNumProcessors)

import System.Directory (doesDirectoryExist, doesFileExist,

pathIsSymbolicLink)

import System.Environment (getArgs)

 

fileSizes :: [Item] -> [Integer]

 

folders :: [Item] -> [FilePath]

 

totalBytes :: [Item] -> Integer

Folder _ -> (a, succ b)) (0, 0)

 

frequencyGroups :: Int -- ^ Desired number of frequency groups.

-> [Integer] -- ^ List of collected file sizes. Must be sorted.

frequencyGroups _ [] = newFrequencyGroups

frequencyGroups totalGroups xs

| otherwise = foldr placeGroups newFrequencyGroups xs `using` parTraversable rseq

where

placeGroups n fgMap = case findGroup n groupMinMax of

Just k -> Map.alter incrementCount k fgMap

 

expandGroups :: Int -- ^ Desired number of frequency groups.

-> FrequencyGroups -- ^ Expanded 'FrequencyGroups'

expandGroups gsize fileSizes groupThreshold

where

loop 0 gs = gs -- break out in case we can't go below threshold

 

expand :: FrequencyGroups -> FrequencyGroups

expand = foldr f . withStrategy (parTraversable rseq) <*>

where

f :: Maybe (FGKey, FrequencyGroups) -- ^ expanded frequency group

-> FrequencyGroups -- ^ accumulator

:: FGKey -- ^ Group Key

-> Integer -- ^ Count

groupsFromGroup (min, max) count

| length range > 1 = Just ((min, max), frequencyGroups gsize range)

| otherwise = Nothing

where

 

displaySize :: Integer -> String

| n >= 2^30 && n <= 2^40 = display (2^30) "GB"

| n >= 2^40 && n <= 2^50 = display (2^40) "TB"

where

display :: Double -> String -> String

display b = printf "%7.2f%s " (realToFrac n / b)

 

displayFrequency :: Integer -> FrequencyGroup -> IO ()

percentage :: Double

percentage = (realToFrac count / realToFrac filesCount) * 100

 

 

parallelItemCollector :: FilePath -> IO [Item]

loop :: Chan FilePath -> Chan [Item] -> [Item] -> IO [Item]

loop folderChan resultItemsChan xs = do

if null regularFolders then pure []

else do

result <- mapM (loop folderChan resultItemsChan) childItems

pure (join childItems <> join result)

 

main :: IO ()

Left errorMessage -> hPutStrLn stderr errorMessage

Right (path, groupSize) -> do

items <- parallelItemCollector path

let (fileCount, folderCount) = counts items

printf "Total size: %s\n" $ displaySize $ totalBytes items

mapM_ (displayFrequency fileCount) $ Map.assocs results

where

{{out}}

<pre style="height: 50rem;">$ filedist ~/Music

Total files: 688

Total folders: 663

 

Distribution:

 

81B <-> 161B = 74 10.756%: ███████████

162B <-> 242B = 112 16.279%: ████████████████

243B <-> 323B = 99 14.390%: ██████████████

54.20KB <-> 108.41KB = 99 14.390%: ██████████████

709.39KB <-> 945.85KB = 44 6.395%: ██████

6.67MB <-> 13.33MB = 72 10.465%: ██████████

 

$ filedist ~/Music 10

Total files: 688

Total folders: 663

 

Distribution:

 

89B <-> 177B = 75 10.901%: ███████████

178B <-> 266B = 156 22.674%: ███████████████████████

267B <-> 355B = 57 8.285%: ████████

94.59KB <-> 189.17KB = 42 6.105%: ██████

851.28KB <-> 945.87KB = 44 6.395%: ██████

5.33MB <-> 8.00MB = 41 5.959%: ██████

8.00MB <-> 10.67MB = 35 5.087%: █████

</pre>

 

{{works with|Julia|0.6}}

 

 

function sizelist(path::AbstractString)

end

 

 

{{out}}

- between 0.0 B and 1.0 B bytes: 0

- between 1.0 B and 10.0 B bytes: 1

 

=={{header|Kotlin}}==

 

import java.io.File

fun main(args: Array<String>) {

fileSizeDistribution("./") // current directory

 

{{out}}

Number of inaccessible files : 0

</pre>

 

=={{header|Perl}}==

{{trans|Raku}}

use List::Util qw(max);

 

 

sub fsize { $fsize{ log10( (lstat($_))[7] ) }++ }

{{out}}

<pre>File size distribution in bytes for directory: .

=={{header|Phix}}==

Works on Windows and Linux. Uses "proper" sizes, ie 1MB==1024KB. Can be quite slow at first, but is pretty fast on the second and subsequent runs, that is once the OS has cached its (low-level) directory reads.

{{out}}

<pre>

The distribution is stored in a '''collections.Counter''' object (like a dictionary with automatic 0 value when a key is not found, useful when incrementing). Anything could be done with this object, here the number of files is printed for increasing sizes. No check is made during the directory walk: usually, safeguards would be needed or the program will fail on any unreadable file or directory (depending on rights, or too deep paths, for instance). Here links are skipped, so it should avoid cycles.

 

from collections import Counter

 

for dir in arg:

dodir(dir)

s = n = 0

for k, v in sorted(h.items()):

print("Total %d bytes for %d files" % (s, n))

 

 

=={{header|Racket}}==

 

 

(define (file-size-distribution (d (current-directory)) #:size-group-function (sgf values))

(module+ test

(call-with-values (λ () (file-size-distribution #:size-group-function log10-or-so))

 

{{out}}

By default, process the current and all readable sub-directories, or, pass in a directory path at the command line.

 

sub log10 (Int $s) { $s ?? $s.log(10).Int !! 0 }

my %fsize;

my ($end, $bar) = $scaled.polymod(8);

(@blocks[8] x $bar * 8) ~ (@blocks[$end] if $end) ~ "\n"

 

{{out}}

 

=={{header|REXX}}==

<br>older versions of Windows that used suffixes to express big numbers &nbsp; (the size of a file), &nbsp; and also versions

 

Also, some Windows versions of the &nbsp; '''dir''' &nbsp; command insert commas into numbers, so code was added to elide them.

numeric digits 30 /*ensure enough decimal digits for a #.*/

parse arg ds . /*obtain optional argument from the CL.*/

exit /*stick a fork in it, we're all done. */

/*──────────────────────────────────────────────────────────────────────────────────────*/

 

The &nbsp; '''LINESIZE.REX''' &nbsp; REXX program is included here &nbsp; ──► &nbsp; [[LINESIZE.REX]].<br>

{{libheader|walkdir}}

{{works with|Rust|2018}}

use std::error::Error;

use std::marker::PhantomData;

}

}

{{out}}

<pre>

 

=={{header|Sidef}}==

dir.open(\var dir_h) || return nil

 

}

 

{{out}}

<pre>

log10(size) ~~ 8 -> 2 files

Total: 370026462 bytes in 2650 files

</pre>

 

=={{header|zkl}}==

// hoover all files in tree, don't return directories

fcn(pipe,dir){ File.globular(dir,"*",True,8,pipe); }

println("%15s : %s".fmt(szchrs[idx,*], "*"*(scale*cnt).round().toInt()));

idx-=1 + comma();

{{out}}

<pre>

Found 4320 files, 67,627,849,052 bytes, 15,654,594 mean.

File size Number of files (* = 69.84)

n,nnn : *

n,nnn,nnn : *

nn,nnn,nnn : **************************************************