Delete Duplicate Files With Bash

Posted on January 25, 2012 by shigs

I needed to delete all but 1 duplicate files from a bunch of experiments. I didn't care about file names. Here's the bash - it works.

#!/bin/bash

declare -a list=( `find ./DIRECTORY_CONTAINING_FILES f` );
declare -a sums;
cnt=${#list[@]}

echo "creating md5sum list"
for ((x = 0; x < $cnt -1; x++))
do
    sums[$x]=`md5sum ${list[$x]} | cut -d ' ' -f 1`
    #echo ${sums[$x]}
    progress=$(echo "scale=2;($x/$cnt)*100" | bc)
    echo -ne "progress $progress %\r"
done

echo "doing compare"
for ((x = 0; x < $cnt -1; x++))
do
    for ((y = x+1; y < $cnt; y++))
        do
            if [ "${sums[$x]}" == "${sums[$y]}" ];then
                #echo ${sums[$x]} " - "  ${sums[$y]}
                if [ ${list[$x]} != ${list[$y]} ]; then
                    #remove '#' and combine next 2 lines to enable
                    echo "Delete file\n${list[$y]}\n${list[$x]}"
                    # && rm -f ${list[$y]}
                fi
            fi
    done
done
Posted in Programming, Simulation, Social Data Analysis, useful | Leave a comment

Counting distinct sets in a dictionary of sets in Python

Posted on November 26, 2011 by shigs

If like me you suck at Python you may want to do convoluted things like count the distinct sets in some data. Here's one way you can do it.

In this example sets are read into a defaultdict and in this case set 1 is identical to set 3. So out of the 4 sets read into the dictionary there are 3 distinct sets.

import collections

def getDistinctSets(dupedList):
	new_set = set()
	for k,v in m.iteritems():
		new_set.add(frozenset(v))
	return new_set

m = collections.defaultdict(set)
m[0].add("ham")
m[1].add("ham")
m[1].add("cheese")
m[2].add("ham")
m[2].add("cheese")
m[2].add("pineapple")
m[3].add("cheese")
m[3].add("ham")

print getDistinctSets(m)

This returns a set of all the distinct sets, minus the duplicates:

set([
	frozenset(['cheese', 'ham']),
	frozenset(['ham']),
	frozenset(['cheese', 'ham', 'pineapple'])
])
Posted in Programming, Python | Leave a comment

Simple Distributed Community Detection for The One Simulator

Posted on November 8, 2011 by shigs

In the implementation of the Simple algorithm [Distributed Community Detection in Delay Tolerant Networks Pan Hui, Eiko Yoneki, Shu-yan Chan and Jon Crowcroft Sigcomm Workshop MobiArch '07, August, Kyoto, Japan] by PJ Dillon for The One Simulator, updates to communities are only recalculated when connections between devices end. Furthermore devices are always added to a local community whenever the familiar threshold is satisfied and community similarity is always checked regardless of encounter time. We believe this implementation would not work well in cases where nodes are connected for long periods of time and have therefore created a new version available to download here:

The new version does not work with the Decision Engine Router either , for simplification of updating nodes every simulator tick. The code is quick and dirty and as a result has many old comments and debug methods left in but if you are familiar with Simple you should be able to work it out.

  1. SimpleRouter.java
  2. SimpleRouterPromote.java Simple plus devices being added to local communities who are "most familiar".


 

Posted in ONE Simulator, Simulation, Social Data Analysis | Leave a comment

Why Eat 1 Breakaway Bar When You Can Eat 9 Breakaway Bars

Posted on November 8, 2011 by shigs

After the rather successful exposé Why Eat 1 Tunnock’s Caramel Wafer When You Can Eat 8 Tunnock’s Caramel Wafers I've decided to pen another report. This time my adoring fan, I present to you Not 8 but 9 Nestle Breakaway Bars! So without further ado let the comfort eating for the painfully alone commence. There's 50% extra free waiting to be eaten and an adoring public waiting to hear about it.

9 Breakaway Bars.

9 Breakaway Bars.

Breakaway Bar Number 1: There's Something about multi-packs which makes the chocolates smaller and less satisfying. It's probably caused by the subconscious reaction to having 9 chocolate bars. The reaction causing you not to chew and dribble crumbs all over your keyboard because you think nobody is watching. The first goes down easier than a university coursework submission system.

Breakaway Bar Number 2: Seriously what gives? Each bar is only 99 calories and that must all be in the golden biscuit because the thin layer of chocolate has absolutely no sophistication or taste like the other chocolate that I like. Nestle should have used proper chocolate like Cadbury's. That chocolate shaped like a penis probably put them off. Remember If it's stamped on the back with Cadbury's — it came from their factory. What a prick.

Breakaway Bar Number 3: "100% Wholemeal and Oat Flour. A source of fibre", they're not kidding. Pooing like a rabbit and I've only had 3. Going to have to put the nutritional information under closer scrutiny.

Breakaway Bar Number 4: Sorry no news on nutritional information just yet. I've found myself staring blankly at the ® symbol next to "It's good to know ®" written on the back of the packet. It's good to know ® Nestle think some people can't print the words It's good to know ® in that order without consulting them first. Yep, It's good to know ®.

It's good to know ®

It's good to know ®

Breakaway Bar Number 5: I'm sure these used to have tinfoil wrapped around them. As soon as i remembered that my face shriveled up like a prune. Fillings groaning at the very thought of finding a bit of foil stuck to melted chocolate.

Breakaway Bar Number 6: If like me you thought that eating 9 99 calorie buscuits was easy, then consider this. I've just eaten my 6th Breakaway bar in the space of as many hours and that's 99 calories x 6 = 594 calories! That's 100 more than a Big Mac! But unlike a Big Mac these taste like air. Deceiving little bastards.

Breakaway Bar Number 7,8 and 9: There's no fucking way I'm eating another 3 of these things today.  I'm going to eat an orange. What was the point of this post? It's good to know ®.

Day 2

Breakaway Bar Number 7: Ive just woken from my sugar induced coma in a blind panic. There'll be no rest whilst there's biscuits left, so turn up woman's hour all the way to 11 - here we go!

Breakaway Bar Number 8: There's nothing quite like coffee and cake for breakfast, this counts right? I have a very loose definition of what a cake is unlike you Jaffa Snobs.

Breakaway Bar Number 9: Good morning sports fans! I've knocked the 9th out of the park, hit it for 6, got all the way to 4th base with this dirty golden slab. Now I'm going back to bed.

Posted in whyhaveonewhenyoucanhaveeight | 1 Comment

BonnMotion 2.0 released

Posted on November 7, 2011 by shigs

BonnMotion 2.0 released. BonnMotion is a mobility scenario generation and analysis tool, which also supports output for The ONE. I've not looked at it yet but the 3D support looks nice for more complexed scenarios.

Some of the new features:

  • 3D support
  • New models
  • SLAW
  • SteadyStateRandomWaypoint
  • RandomWaypoint3D
  • Export of DisasterArea movements to standard NS-2/NS-3
  • Import of GPX traces
  • Code validation tool

More information (including changelog and source code) can be found at: http://bonnmotion.net.cs.uni-bonn.de/

Posted in Simulation | Leave a comment