deedle_dataIn this article I want to briefly present how F# can easily compete with language that is commonly found as a go to tool for data analysis - Python.

Some time ago I’ve completed Computational Investing course @ Coursera. It’s an 8 weeks long tutorial that helps you familiarize yourself with how modern electronic markets work, why stock prices change in the ways they do, and how computation can help you understand price changes. This course also gives you a basic overview of how hedge funds and other investment companies operate, how to build simple market simulator, study market events and perform technical analysis.

Course assignments were based on a Python with Numpy and Pandas libraries and were mostly oriented around manipulating large amounts of equities prices. Data Frame based libraries like Pandas are a joy to work with for time based series but for a while F# didn’t have a similar product. Now thanks to Tomas Petricek and Howard Mansell (and other active members of F# open source community) we do – meet Deedle. Also Coursera course used QuantSoftware Toolkit (QSTK) – Python “open source software framework designed to support portfolio construction and management”. F# does not have direct counterpart but thanks to Type Providers and rich .NET platform replacing some features of this library was simple and easy.

I’ve decided to translate a QSTK_Tutorial_1 to F#. This tutorial covers some basics of QSTK and Pandas. Full script in F# can be found here.

Getting stock quote data from Yahoo Finance

Thanks to F# Type Providers getting data from Yahoo Finance is as easy as:

type StockData = CsvProvider<"http://ichart.finance.yahoo.com/table.csv?s=SPX", InferRows=5>
// helper function to generate yahoos api url
let urlFor symbol (startDate:System.DateTime) (endDate:System.DateTime) = 
    sprintf "http://ichart.finance.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i" 
        symbol
        (startDate.Month - 1) startDate.Day startDate.Year 
        (endDate.Month - 1) endDate.Day endDate.Year 

Now let’s define a function to load stock data and create Data Frame out of it. At first we load stock data for each symbol and convert it into time series. As a last step we convert a list of series into Data Frame (Frame.orderRows orders series by DateTime). Simple.

let loadStocks symbols startDate endDate =
    [ for symbol in symbols ->
        symbol => 
            let data = StockData.Load(urlFor symbol startDate endDate).Data
            Series.ofObservations [ for d in data -> (d.Date, d.Close)]]
    |> Frame.ofColumns
    |> Frame.orderRows

Let’s load up the data into Data Frame.

let startDate = new DateTime(2011, 01, 01)
let endDate = DateTime.Now
let symbols = ["AAPL"; "MSFT"; "GOOG"; "XOM"; "SPX"; "GLD"]
let stocks = loadStocks symbols startDate endDate

Working with Data Frames

Let’s normalize this data so they can be compared in one chart. We define a function that maps each value per column (so per stock type) and creates new Data Frame with normalized values.

let normalized = 
    stocks 
    |> Frame.mapColValues (fun os -> 
        let osAsFloat = os.As<float>()
        let firstItem = osAsFloat.GetAt(0)
        osAsFloat / firstItem)

F# has simple yet powerful charting library

Chart.Combine
    [ for s in normalized.GetAllSeries() 
        -> Chart.Line(s.Value |> Series.observations, Name = s.Key) ]
|> Chart.WithLegend (Docking = ChartTypes.Docking.Top)
fsharp_chart

In the same way we can calculate daily returns.

let dailyReturns =
    stocks 
    |> Frame.mapColValues (fun os -> 
        os.As<float>() 
        |> Series.pairwiseWith (fun k (v1, v2) -> v2 / v1 - 1.0))

And display correlation charts.

let correlationChart index1 index2 =
    let values1 = dailyReturns.GetSeries<float>(index1) |> Series.values
    let values2 = dailyReturns.GetSeries<float>(index2) |> Series.values
    let correlationValue = Correlation.Pearson(values1, values2)    
    Chart.Point (Seq.zip values1 values2, 
        Name = sprintf "Correlation coefficient between %s and %s is %f" 
                       index1 index2 correlationValue)
    |> Chart.WithLegend (Docking = ChartTypes.Docking.Top)    
    |> Chart.WithXAxis (Enabled = false) 
    |> Chart.WithYAxis (Enabled = false)
// Exxon Mobil and S&P 500 Index are pretty correlated
correlationChart "SPX" "XOM"

high_correlation
// Gold and S&P 500 Index are basically not correlated
correlationChart "SPX" "GLD"

not_correlated

In closing

I hope this short blog post convinces you that F# has a potential of being a great tool for data analysis. To sum up F# (and .NET platform) provides you with:

If this is not enough, one can always use R provider, Matlab provider (work in progress) or in the future even Python provider.

So there, F# is awesome.

Over a month ago or so I decided to stop thinking that it would be great to learn F# and actually do it instead. This is a summary of my experiences and thoughts of what worked for me the most.

During this month I managed to raise my skills set to a level where I’m pretty comfortable with reading and understanding of F# code and where I’m able to write simple F# programs without much struggle. There is of course a still long road ahead of me as I haven’t yet touched some of the important features of this language + my code probably looks still bit C# like.

What has worked

Try F# – is a set of interactive tutorials helping tremendously to “taste” F# as a language and what possible problems can it help to solve, good starting point and it is pretty well done.

F# Koans – is a GitHub project started by Chris Marinos aiming to teach you F# through testing. You are presented with a set of challenges showing different aspects of the language with gradually increasing difficulty. Great thing about these approach compared to Try F# is that it actually engages you and is quite fun. I personally found it a great way to learn.

F#unctional Londoners Meetup Group – I’ve been to a few meetups organized by this group and as a result I got some inspiration where to go next with my learning and had fun with hands on sessions like Generative Art presented by Phillip Trelford.

What has worked less

F# project – I decided to write GUI application for Windows 8 with F# backend but that didn’t quite taught me as much F# as I originally planned. I spent most of my time dabbling with XAML rather than F# code that in the end turned out to be pretty trivial. I learned quite a bit of Windows 8 development because it turned out to be not as close to WPF as I originally thought. Developing a simple game would probably be more fun and would made me to use more F# features. So it’s totally my fault to pick the wrong project type and I still believe that deep dive into a code can bring great results in terms of learning optimization.

Project Euler – once I’ve run out of F# Koans I wanted more so I went to solve few Euler problems. Soon I’ve found myself fighting more with math than programming language so I quickly gave up (although math was fun but it wasn’t a goal at the moment).

Books – they undoubtedly have advantage over blog posts because somebody actually made an effort to put all the information as a logical whole. Nowadays though I’m finding them mostly useful as a reference source rather than learning tool. Classic books used to have exercises at the end of each chapter and that worked great to engage and challenge user – a feature rarely found in modern books. But again I believe that there are great books about F# that I haven’t encountered yet and I certainly do not want to state that books are completely useless – they have their place and role to organize knowledge and to give a deeper insight to the reader – they are just not the best for rapid learning in my opinion.

Other ideas and thoughts

Kit Eason was learning F# (with great results) by doing actuarial mathematics – learning business and language at the same time. I want to do the same about FX Options as FX markets is an area where I work at the moment. Tomas Petricek during DDDEastAnglia presented how to build DSLs with F# around Options so that should give me a good start.

I also want to slowly introduce F# to my current place of employment – my idea is to give few brown bag presentations and start using F# to write specifications. That should be gentle enough to slowly boil the frog.

In closing

If you want to get up to speed with F# I propose that you go through Try F# and then F# Koans – they undoubtedly gave me the biggest gain for the smallest effort.

F# is a great language and if you are a C# developer that wants to join fast growing functionally thinking crowd – the time is now!

Tl;Dr F# is awesome and join me for a 4 week challenge to learn it.

4 week challenge ?

Some time ago during my friend’s birthday party the idea was born - let’s challenge ourselves to learn new skills. By skills we defined things like foreign language, dancing, drawing or hey! - something challenging technically.

General plan is to dedicate yourself to learn one particular skill in a predefined time frame (4 weeks) with a measurable goal. Also one should try to achieve it in a most efficient manner and share his path and results with the world (hence this post). Public announcement will mean there is no return and our documented experiences will help hopefully to develop even better learning techniques.

You can read more about challenge by itself on Jakub’s blog (and it’s rules).

My learning techniques and strategies are going to be based on a book Pragmatic Thinking and Learning but general idea is to try do something practical as a first step, fail fast and build upon it. It seems that for many this way of learning is more cost efficient than pointless feeding of one’s brain with disconnected theory.

To put it into more formal frame:

Goal

Learn enough F# so I can be comfortable enough to start my new projects using it.

I understand that my F# code will still probably look like C# without curly braces but I hope that accomplishing this goal will give me a solid enough foundation to build upon and become proficient by the end of this year.

Motivation

I have been developing using C# for more than 6 years and I can say that I feel more than comfortable in its world. For some time though I had an urge to move forward. C# is a powerful, general purpose language but the more software I develop the more I’m finding it to be too verbose and not as expressive as I would like it to be. By comparing samples of C# and F# code I’ve quickly fallen in love with the latter. That might be very superficial love - as I have only relied on few external features like prettiness of the code structure, terseness or overall fun factor (type inference, type providers and many more FTW). So this goal will also verify potential marriage material although expectations are high! :-)

Path

As a part of my monthly trial I want to experiment with different learning techniques. Generally I want to follow two (somehow similar) rules:

  • practice beats knowledge - try something and fail fast and then build knowledge upon your failure

  • don’t try go perfect with the first attempt - you will learn faster by being roughly right - time to master that skills will come later

In practice I see it as a form of iterative (Agile) approach where practice goes first, then knowledge, than more practice followed by even more knowledge etc. One might ask - hold on cowboy - how can you actually practice something without knowing anything ? This and more details will be explained in my next blog post.

I want to start my monthly challenge on Monday 01.07.2013.

How about you also give it a try and learn something that you always wished but never managed to convince yourself ? If interested please check more details and ideas on Jakub’s blog.

This post is about my experiences with reading technical books and perceived effectiveness of such. I look into a proven method that helps me retain and put into practice large amount of knowledge that I acquired by reading.

I was inspired to write this post after rereading great book titled "Pragmatic Thinking and Learning: Refactor your wetware" by Andy Hunt, author of Pragmatic Programmer (another great book!). His book is indirectly addressed to a software developers and it’s ideas resonated with me for a long time after I’ve finished it reading for the first time.

The good

My adventures with reading technical books went through certain stages. At the very beginning of my programming endeavors I accidentally got it right (or more like circumstances made it).

It was somewhere in the 90s and I decided to teach myself Assembly language (I really do not remember why…) but because I only had access to this one (fortunately good) book (and heck – no internet!) and I didn’t know much about programming at that time I literally consumed it in it’s entirety. I remember going through it time after time, trying things, going back, rereading certain parts, coding again etc. I literally knew each page of this book and it end up looking like a consumed up by the dog (and I only had cat). Information contained in this book was enough for me to create my first program that was a clone of Norton Commander (but faster because it was pure Assembly! ;-D). I can say without a doubt that I’ve taken most out of this book.

The bad

Than it went a little bit sideways, books that I mostly got into my hands were boring step by step guides, internet came around, I got the first job and days got shorter. I’ve started to have an approach of figuring things out on the way. It worked, sort of.

I was getting my things done and I was getting some sort of experience – I thought that that’s the way things should be. Problem was – I was quite often struggling with and cursing at certain technologies. It hasn’t occurred to me immediately that I was missing understanding of core underlying concepts. Reading of blog posts / articles found on internet was just helping me to overcome current obstacles but wasn’t really giving me any deeper foundation of knowledge (similar to popular nowadays Stackoverflow Driven Development).

I’ve realized that I need to read books again – so I started but my approach to this task was not ideal. At my first job I’ve unfortunately gained few bad habits (and I’ve learned hell lot of how not to write software…) and one of them was to be in a constant rush and impatient. That was also my way of reading books, quick skimming through content, rarely giving them a deeper thought. But this approach was flawed for many reasons – and as a effect my reading was highly inefficient. I can barely recall names of the books I’ve read back than not to mention concepts that they were presenting.

The solution

After I read Andy’s Hunt work I slowly got my mindset tuned into getting most out of the books again.

Andy proposes a process of deliberate reading as a way of making it more effective. In particular he describes a process called SQ3R that is an acronym for the steps you need to take:

  • Survey - scan the table of contents and chapter summaries for an overview.
  • Question - note any questions you have.
  • Read - read in its entirety.
  • Recite - summarize, take notes, and put in your own words.
  • Review - reread, expand notes, and discuss with colleagues.

It's all about making reading a conscious process as opposed to reading random books and forgetting about them immediately.

For example let's say you are Java developer and want to read a book about C#. You could:

  • start by flipping through the table of contents and familiarizing yourself with a book
  • become curious about content, ask yourself questions like: "what's are differences between Java an C#", "what is this LINQ I heard so much about", "what are conventions of standard library" etc.
  • now read this book in it's entirety, parts that are not interesting to you are worth at least skimming through - so you can know what you do not know :-)
  • than important - try to use information from this book, try to write a program in this language from scratch - preferably different from book exercises. During that processes you will get back to different parts of this book, you can make some extra notes or highlight important parts. It's all about turning this book into practical knowledge instead of pale memory.
  • at last you can talk through content of this book with some friends or on some internet forum, check what other people think.

It depends

Obviously you are reading particular book with a certain goal in mind. If your goal is to use this language / framework / tool at hand in your next project or to get a job than technique I've described above is the way to go.

On the other hand if you just want to familiarize yourself with some concepts without going to deeply into details I believe quick reading through it and taking some general notes is a good enough solution. When need arises at least you know what you do not know and can easily catch up on your knowledge. For example I've done so with "REST in practice" (great book BTW) as at the moment I didn't need to create any REST services but after reading it I'm at least well aware of concepts, practices and how to put pieces together if opportunity arises (and I can understand what are all this flame wars on twitter all about ;-).

Conclusion is simple – read books with goal in mind and take care how you do it -  otherwise it’s just a waste of time.

This blog post is a summary of my experience how to configure NServiceBus with Castle Windsor and RavenDB.

NServiceBus has many extensibility points and one of them is an ability to define our own Unit Of Work implementation. Here is a blog post why and when we should / could implement it.

Andreas gave an example how to configure it using StructureMap. General implementations between different DI frameworks that are supported by NServiceBus do not vary greatly – yet there are few quirks that not advanced user of particular framework might not be aware of.

So in my case on current projects that I’m working on I’m using Castle Windsor. Castle Windsor since version 3.0 has new Lifestyle option called “Scoped”. According to project wiki:

Scoped lifestyle - allows objects to be reused within any arbitrarily selected logical scope. The default scope is denoted by BeginScope extension method, but any custom scopes can be used by implementing new IScopeAccessor interface.

NServiceBus behind the curtains is opening a new scope for each Handler execution. Thanks to that we can reuse the same IDocumentSession both in our UnitOfWork implementation and Handler object.

Sample CastleWindsor Installer:

public class RavenDBInstaller : IWindsorInstaller
{
    IWindsorContainer _container;

    public void Install(IWindsorContainer container, 
                        IConfigurationStore store)
    {
        _container = container;
        container.Register(Component.For<IDocumentStore>()
                                .LifestyleSingleton()
                                .UsingFactoryMethod(initDocumentStore),
                            Component.For<IDocumentSession>()
                                .LifestyleScoped()
                                .UsingFactoryMethod(openSession),
                            Component.For<IManageUnitsOfWork>()
                                .LifestyleTransient()
                                .ImplementedBy<RavenUnitOfWork>());
    }

    static IDocumentStore initDocumentStore()
    {
        return new DocumentStore
        {
            ConnectionStringName = "RavenConnectionString"
        }.Initialize();
    }

    IDocumentSession openSession()
    {
        return _container.Resolve<IDocumentStore>().OpenSession();
    }
}

And for completeness RavenUnitOfWork implementation

public class RavenUnitOfWork : IManageUnitsOfWork
{
    readonly IDocumentSession _session;

    public RavenUnitOfWork(IDocumentSession session)
    {
        _session = session;
    }

    public void Begin()
    {
    }

    public void End(Exception ex)
    {
        if (ex == null)
            _session.SaveChanges();
    }
}
And that’s it! Hope it’s going to be useful.

This article explains how to unit test asynchronous controllers with ASP.NET MVC 3. I’ve written simple extension method to AsyncController that makes it easier to focus on a behaviour side of a test rather than on thread ‘noise’.

Here goes our simple asynchronous controller:

public class SampleAsyncController : AsyncController
{
    public void SquareOfAsync(int number)
    {
        AsyncManager.OutstandingOperations.Increment();
 
        // asynchronous operation
        new Thread(() =>
        {
            Thread.Sleep(100);
 
            // do some async long operation like ... 
            // calculate square number
            AsyncManager.Parameters["result"] = number * number;
                
            // decrementing OutstandingOperations to value 0 
            // will execute Finished EventHandler on AsyncManager
            AsyncManager.OutstandingOperations.Decrement();
        }).Start();
    }
 
    public JsonResult SquareOfCompleted(int result)
    {
        return Json(result);
    }
}

Here instead of some asynchronous operation I’m simply spawning new thread.

If you are not familiar with AsyncController you might be puzzled by AsyncManager.OutstandingOperations.Increment() and Decrement() methods. They are simple way of signalling AsyncManager when asynchronous operation has finished executing and when actionCompleted method can be executed.

Testing of asynchronous controller is a little bit more complicated than synchronous counterpart. In order to test whole path (so actionAsync and actionCompleted) we need to register manually actionCompleted to Finished event handler on AsyncManager. That way when actionAsync has finished processing it will call actionCompleted. Please note that actionAsync does not finish processing when code reaches end of method but rather when AsyncManager.OutstandingOperations is reduced to 0 by calling Decrement() method (hence it is so important!).

AutoResetEvent is used to signal when asynchronous operation has really finished processing – in this case it will happen after actionCompleted has finished. Trigger.WaitOne() will hold it till trigger.Set() is called.

static class AsyncControllerExtensions
{
    public static void ExecuteAsync(this AsyncController asyncController, 
                                         Action actionAsync, 
                                         Action actionCompleted)
    {
        var trigger = new AutoResetEvent(false);
        asyncController.AsyncManager.Finished += (sender, ev) =>
        {
            actionCompleted();
            trigger.Set();
        };
        actionAsync();
        trigger.WaitOne();
    }
}

To describe behaviour of my controller I’m using Machine.Specifications framework, but of course there is no problem to use other “old school” TDD approach.

public class When_calling_square_of : WithSubject<SampleAsyncController>
{
    It should_return_square_number_of_input_value = () =>
        ((int)Result.Data).ShouldEqual(TestValue * TestValue);
 
    Because of = () =>
        Subject.ExecuteAsync(
            () => Subject.SquareOfAsync(TestValue),
            () => Result = 
                Subject.SquareOfCompleted(
                    (int)Subject.AsyncManager.Parameters["result"]));
 
    static JsonResult Result;
    const int TestValue = 5;
}

Using traditional AAA nomenclature my “Act” is a call on Subject (that is SampleAsyncController) that specifies two parts of an action to execute. Please note that actionCompleted is taking Subject.AsyncManager.Parameters["result"] parameter – during normal execution it is simply hydrated by framework but here we have to take care of it ourselves.

Hope that helps!

PS This code is available on github

DSCN0728
Sometimes you might have a need to get some specific information from tag soup of Internet. You have two options here – either to use some existing solution or simply write your own. Generally first option seems more reasonable (why to invent another wheel), but your requirements might be very specific or … it seems like a lot of fun! Since I’ve chosen this second path I’m going to share some of my insights useful particularly for .NET developers.

Important: if you need scalable, large, google like spider then you should really consider looking at existing solutions. My thoughts are relevant for scanner able to process few hundreds pages per minute on a single machine, while provided a list of root nodes to scan.

When I mentioned at Software Craftsman meeting that now I’m writing software scanner guy sitting next to me looked at me with disbelief and said “Dude, 1995 is over”. There is always a question on whether to create something from scratch to perfectly suit your needs in opposite to using ready made solution. There is gold sentence that every consultant is overusing and it is "that depends”. In past I was a part of a team who in two years successfully crafted and implemented ERP solution. Previous implementation of ready made software failed miserably. Ok, enough of philosophy, let’s get down to business.

Getting a web page.

Is easy like this:

    var webRequest = (HttpWebRequest)WebRequest.Create(url);
    webRequest.Method = "GET";            
    webRequest.UserAgent = "Mozilla/5.0 (compatible; mycrawler/1.0)";
    response = webRequest.GetResponse();

Then you just need to GetResponseStream on your response object, handle all exceptions, check content type (we prefer something that contains text/html) and we can parse our text. But…

Regular Expressions are not best solution.

It’s usually first thought you get when approaching this problem. Looking for href seems like piece of cake. But it’s not – I’m not going to dig into this topic – I think best answer on StackOverflow is going to convince you, if not check Jeff Atwood post.

So what is our quick and lazy alternative ? Mine was HtmlAgilityPack. What is it?

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

Getting all anchor tags from a webpage is as easy as:

    document.DocumentNode.SelectNodes("//a");


Looking for links.

Not only <A /> tags are links. <AREA /> as well (inside <MAP />). You will have to follow frames and pages with instant redirection through META refresh tag. In all these cases HtmlAgilityPack is very helpful:

    document.DocumentNode
            .SelectNodes(@"//a|//area")
            .Select(x => x.Attributes["href"]);
    
    document.DocumentNode
            .SelectNodes(@"//frame|//iframe")
            .Select(x => x.Attributes["src"]);

*code above is missing null checks etc.

Pages contain both absolute and relative URLs, we really don’t want to analyze strings to deal with that kind of stuff. System.Uri comes to the rescue. Some useful code snippets utilizing this class:

    // Is our text an absolute url ?
    Uri url;
    if (Uri.TryCreate(urlCandidate, UriKind.Absolute, out url)) return url;
    
    // If not - is it a relative url ? 
    // Try to crate url on a base of url from this page 
    if (Uri.TryCreate(baseUrl, urlCandidate, out url)) return url;

Breadth first crawling ?

For simple scanner your might prefer breadth first crawling. Probably you will need two lists – one list containing pages to visit, and other list containing already visited pages (we don’t want to get into loop). Even for simple solutions though we need some degree of parallelism – downloading page by page can be very slow task.

When to stop ?

It depends on your needs – but if you haven’t decided to create a "better google” than probably for example you want to scan one specific domain. Here again class Uri and especially Uri.IsBaseOf() method is very useful, we can check every link before adding it to “to visit” list. But we should definitely create some other stop criteria – like visit counter of specific URLs (calendar controls are sometimes a trap for crawler), time spent inside specific domain etc.

Getting information out of HTML soup.

Usually you are building your scanner to get something specific, looking for specific phrase, for specific link, element, to download images etc. And again HtmlAgilityPack is a bless here (seriously, why would you do this with Regular Expressions?). For example:

    // getting all alt nodes of images
    document.DocumentNode.SelectNodes("//img[@alt]");

Should I do it ?

Well again, it depends (I’m a good material for consultant :-). If you want to crawl and parse tens of thousands of pages in a minute, then probably not. Parallel computing is note an easy piece of cake, and it was well thought and developed by many very smart people – and you can get results of their work as an open source (not in C# though). But if you want to scan reasonable amount of pages, you want to get specific information and robots.txt on your site of destination is not denying it, then go ahead. And good luck.

programming

And I mean full, I don’t have any hard data to support my statement but my one month experience here and comparison with my home city (Poznan) that is same size as Oslo. And even if % of people who are typing lines of code here is not really higher at least they are way more active and passionate of what they do.

.NET Users, Windows Phone Developer Community, The Oslo Lean Meetup, Extreme Programming Group, Java Developers, heck! - there is even a Delphi users group. No to mention that local software houses, consulting firms, etc. are really interested in growth of their employees and they actively invite speakers outside of Norway.

I have visited Test-Driven Development with JavaScript and node.js organized in an office of one of Norway’s software houses (60 people attended, and right after there was an XP meeting with more then 100 registered – quite impressive). First speaker was Norwegian, but fortunately after a short survey whether there are or not non Norwegian people at the public they decided to have this talk in English (it turned out that only for me – thank you :-)!). BTW Norway is truly amazing country where everyone, and I mean everyone can speak English (yes, even 80 years old people and drug addicts). I met some really nice people there, very passionate about what they do.

This week however I experienced something completely new. I attended a meeting that was a form of Coding Dojo. According to Coding Dojo website it is:

.. a meeting where a bunch of coders get together to work on a programming challenge. They are there have fun and to engage in DeliberatePractice in order to improve their skills.

Host of this meeting was Jon Jagger, a truly passionate guy who created http://www.cyber-dojo.com/. As he states on this webiste:

I built CyberDojo so you can practice and learn about:

  • coding and test driven development
  • team dynamics and collaboration

Now please try to imagine, around 8 notebooks, pair programming, 5 minutes of time between group change, C#, C, Java, Python, Ruby and one Roman Numbers problem :-) It might seem like a chaos at the beginning, and yes, it was hard to get to the working solution, but for me lesson of importance of red, green development, leaving your code in a state that work can be continued right after you left, getting closer to the solution in a small steps was really valuable.

If you can ever have chance to experience such a thing, I do strongly encourage you to do this, you are going to learn a lot about your team working skills.

razor

While thinking about Toyota Way it is hard not realize that many of principles that driven this company success are not really something new. They were “alive” for many centuries - since mankind ever tried to conduct any bigger project. One of them is simplicity – in Toyota there is no place for bureaucratic, overblown mindset.

Recently I stumbled upon Occam’s Razor law, also known as Latin expression Lex parsimoniae. It is one of those rules of thumbs that driven many outstanding minds of few last centuries (including Albert Einstein, Leonhard Euler, Isaac Newton).

The Principle of Plurality - Plurality should not be posited without necessity
The Principle of Parsimony - It is pointless to do with more what is done with less

Although they seem obvious, The Principle of Parsimony is something that can be really taken into hearts of many software developers. Creating overblown solutions just to shoot “the fly” is a common problem – and if you have a problem first step is to admit it ;-) overblown

First sin is to create architecture way to complex for current project. Do we really need DDD in simple web app ? Maybe Unit Of Work pattern with conjunction of Linq To Sql is enough ?

Another problem is adding features that seem “necessary”, but in the end our client is never going to use them because from his perspective they are useless. This subject is mostly covered with Agile movement, but still it is a quite common behavior among software developers.

I truly believe that this law is really accurate in almost every aspect of developer’s life – e.g. documentation should be concise, and should cover generalities without getting lost in details.

stop

There are times we do know that we should stop - we are in a middle of creative process, things are going great, suddenly something nasty is showing it’s face – but we do not stop to perform a plastic surgery, we merely cover it with makeup and we do go on chased by deadlines, “I will do it later” or “That’s not such a big deal” thoughts. I’m not talking about bugs here, we have to fix them in order for code to work – I think about ugly pieces of code, badly structured, copy – paste solutions etc. We all have been there right ? How many times we actually did went back to fix those problems ? How many of those stabbed us in a back just at worst time possible ?

While reading “The Toyota Way” I was fascinated by one principle and how much does it relates to software development profession – “Build a culture of stopping to fix problems, to get quality right the first time”.

Quality takes precedence (Jidoka). Any employee in the Toyota Production System has the authority to stop the process to signal a quality issue.”

That’s right, one person can stop whole production line at a first sight of a problem. This is one of the milestones of Toyota success and at the first moment it might seem ridiculous. So what happens next ? Problem is solved as quickly as possible by using all strengths of production group.

In our software development world stopping our coding process and gathering all team members seems so easy and cheap – yet we do still have this “I will do it later” mind set. But it’s time to face the truth - this actually doesn’t work, we are never going to get back to this wrong piece of code unless it’s going to hurt us badly. So maybe we should stop cheating ourselves and instead adopt some good principles from others ?

About
Dawid Kowalski Dawid Kowalski is software developer, traveler and strong advocate of "let's go do it".


Recent Posts
Recent Comments
Twitter
Syndication