Adventures in coding

C# Web Scanner

2011-06-12T20:46:02Z

Sometimes you might have a need to get some specific information from tag soup of Internet. You have two options here – either to use some existing solution or simply write your own. Generally first option seems more reasonable (why to invent another wheel), but your requirements might be very specific or … it seems like a lot of fun! Since I’ve chosen this second path I’m going to share some of my insights useful particularly for .NET developers.

Important: if you need scalable, large, google like spider then you should really consider looking at existing solutions. My thoughts are relevant for scanner able to process few hundreds pages per minute on a single machine, while provided a list of root nodes to scan.

When I mentioned at Software Craftsman meeting that now I’m writing software scanner guy sitting next to me looked at me with disbelief and said “Dude, 1995 is over”. There is always a question on whether to create something from scratch to perfectly suit your needs in opposite to using ready made solution. There is gold sentence that every consultant is overusing and it is "that depends”. In past I was a part of a team who in two years successfully crafted and implemented ERP solution. Previous implementation of ready made software failed miserably. Ok, enough of philosophy, let’s get down to business.

Getting a web page.

Is easy like this:

    var webRequest = (HttpWebRequest)WebRequest.Create(url);
    webRequest.Method = "GET";            
    webRequest.UserAgent = "Mozilla/5.0 (compatible; mycrawler/1.0)";
    response = webRequest.GetResponse();

Then you just need to GetResponseStream on your response object, handle all exceptions, check content type (we prefer something that contains text/html) and we can parse our text. But…

Regular Expressions are not best solution.

It’s usually first thought you get when approaching this problem. Looking for href seems like piece of cake. But it’s not – I’m not going to dig into this topic – I think best answer on StackOverflow is going to convince you, if not check Jeff Atwood post.

So what is our quick and lazy alternative ? Mine was HtmlAgilityPack. What is it?

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

Getting all anchor tags from a webpage is as easy as:

    document.DocumentNode.SelectNodes("//a");

Looking for links.

Not only <A /> tags are links. <AREA /> as well (inside <MAP />). You will have to follow frames and pages with instant redirection through META refresh tag. In all these cases HtmlAgilityPack is very helpful:

    document.DocumentNode
            .SelectNodes(@"//a|//area")
            .Select(x => x.Attributes["href"]);
    
    document.DocumentNode
            .SelectNodes(@"//frame|//iframe")
            .Select(x => x.Attributes["src"]);

*code above is missing null checks etc.

Pages contain both absolute and relative URLs, we really don’t want to analyze strings to deal with that kind of stuff. System.Uri comes to the rescue. Some useful code snippets utilizing this class:

    // Is our text an absolute url ?
    Uri url;
    if (Uri.TryCreate(urlCandidate, UriKind.Absolute, out url)) return url;
    
    // If not - is it a relative url ? 
    // Try to crate url on a base of url from this page 
    if (Uri.TryCreate(baseUrl, urlCandidate, out url)) return url;

Breadth first crawling ?

For simple scanner your might prefer breadth first crawling. Probably you will need two lists – one list containing pages to visit, and other list containing already visited pages (we don’t want to get into loop). Even for simple solutions though we need some degree of parallelism – downloading page by page can be very slow task.

When to stop ?

It depends on your needs – but if you haven’t decided to create a "better google” than probably for example you want to scan one specific domain. Here again class Uri and especially Uri.IsBaseOf() method is very useful, we can check every link before adding it to “to visit” list. But we should definitely create some other stop criteria – like visit counter of specific URLs (calendar controls are sometimes a trap for crawler), time spent inside specific domain etc.

Getting information out of HTML soup.

Usually you are building your scanner to get something specific, looking for specific phrase, for specific link, element, to download images etc. And again HtmlAgilityPack is a bless here (seriously, why would you do this with Regular Expressions?). For example:

    // getting all alt nodes of images
    document.DocumentNode.SelectNodes("//img[@alt]");

Should I do it ?

Well again, it depends (I’m a good material for consultant :-). If you want to crawl and parse tens of thousands of pages in a minute, then probably not. Parallel computing is note an easy piece of cake, and it was well thought and developed by many very smart people – and you can get results of their work as an open source (not in C# though). But if you want to scan reasonable amount of pages, you want to get specific information and robots.txt on your site of destination is not denying it, then go ahead. And good luck.

Oslo is full of developers

2011-02-19T20:08:12Z

And I mean full, I don’t have any hard data to support my statement but my one month experience here and comparison with my home city (Poznan) that is same size as Oslo. And even if % of people who are typing lines of code here is not really higher at least they are way more active and passionate of what they do.

.NET Users, Windows Phone Developer Community, The Oslo Lean Meetup, Extreme Programming Group, Java Developers, heck! - there is even a Delphi users group. No to mention that local software houses, consulting firms, etc. are really interested in growth of their employees and they actively invite speakers outside of Norway.

I have visited Test-Driven Development with JavaScript and node.js organized in an office of one of Norway’s software houses (60 people attended, and right after there was an XP meeting with more then 100 registered – quite impressive). First speaker was Norwegian, but fortunately after a short survey whether there are or not non Norwegian people at the public they decided to have this talk in English (it turned out that only for me – thank you :-)!). BTW Norway is truly amazing country where everyone, and I mean everyone can speak English (yes, even 80 years old people and drug addicts). I met some really nice people there, very passionate about what they do.

This week however I experienced something completely new. I attended a meeting that was a form of Coding Dojo. According to Coding Dojo website it is:

.. a meeting where a bunch of coders get together to work on a programming challenge. They are there have fun and to engage in DeliberatePractice in order to improve their skills.

Host of this meeting was Jon Jagger, a truly passionate guy who created http://www.cyber-dojo.com/. As he states on this webiste:

I built CyberDojo so you can practice and learn about:

coding and test driven development

team dynamics and collaboration

Now please try to imagine, around 8 notebooks, pair programming, 5 minutes of time between group change, C#, C, Java, Python, Ruby and one Roman Numbers problem :-) It might seem like a chaos at the beginning, and yes, it was hard to get to the working solution, but for me lesson of importance of red, green development, leaving your code in a state that work can be continued right after you left, getting closer to the solution in a small steps was really valuable.

If you can ever have chance to experience such a thing, I do strongly encourage you to do this, you are going to learn a lot about your team working skills.

Occam’s Razor & computer programming

2011-02-03T14:36:10Z

While thinking about Toyota Way it is hard not realize that many of principles that driven this company success are not really something new. They were “alive” for many centuries - since mankind ever tried to conduct any bigger project. One of them is simplicity – in Toyota there is no place for bureaucratic, overblown mindset.

Recently I stumbled upon Occam’s Razor law, also known as Latin expression Lex parsimoniae. It is one of those rules of thumbs that driven many outstanding minds of few last centuries (including Albert Einstein, Leonhard Euler, Isaac Newton).

The Principle of Plurality - Plurality should not be posited without necessity
The Principle of Parsimony - It is pointless to do with more what is done with less

Although they seem obvious, The Principle of Parsimony is something that can be really taken into hearts of many software developers. Creating overblown solutions just to shoot “the fly” is a common problem – and if you have a problem first step is to admit it ;-)

First sin is to create architecture way to complex for current project. Do we really need DDD in simple web app ? Maybe Unit Of Work pattern with conjunction of Linq To Sql is enough ?

Another problem is adding features that seem “necessary”, but in the end our client is never going to use them because from his perspective they are useless. This subject is mostly covered with Agile movement, but still it is a quite common behavior among software developers.

I truly believe that this law is really accurate in almost every aspect of developer’s life – e.g. documentation should be concise, and should cover generalities without getting lost in details.

The Toyota Way vs Computer Programming, part 2

2010-11-15T23:13:53Z

There are times we do know that we should stop - we are in a middle of creative process, things are going great, suddenly something nasty is showing it’s face – but we do not stop to perform a plastic surgery, we merely cover it with makeup and we do go on chased by deadlines, “I will do it later” or “That’s not such a big deal” thoughts. I’m not talking about bugs here, we have to fix them in order for code to work – I think about ugly pieces of code, badly structured, copy – paste solutions etc. We all have been there right ? How many times we actually did went back to fix those problems ? How many of those stabbed us in a back just at worst time possible ?

While reading “The Toyota Way” I was fascinated by one principle and how much does it relates to software development profession – “Build a culture of stopping to fix problems, to get quality right the first time”.

“Quality takes precedence (Jidoka). Any employee in the Toyota Production System has the authority to stop the process to signal a quality issue.”

That’s right, one person can stop whole production line at a first sight of a problem. This is one of the milestones of Toyota success and at the first moment it might seem ridiculous. So what happens next ? Problem is solved as quickly as possible by using all strengths of production group.

In our software development world stopping our coding process and gathering all team members seems so easy and cheap – yet we do still have this “I will do it later” mind set. But it’s time to face the truth - this actually doesn’t work, we are never going to get back to this wrong piece of code unless it’s going to hurt us badly. So maybe we should stop cheating ourselves and instead adopt some good principles from others ?

The Toyota Way vs Computer Programming, round 1

2010-10-30T00:32:55Z

One of the side effects of aging and getting wiser (except of getting fatter and grumpier) is gaining ability of discovering parallels and deeper meanings. Notion of giving and receiving, how every single small thing done by someone shows how he/she does everything else or even how starting your day in bad mood affects our whole day. But I’m not here to write about some funky New Age stuff, the thing I wanted to share today is my discovery of great knowledge that was coined before first line of code was written (although I’m not actually sure whether it wasn’t punched around that time).

We software developers have tendency to put our failures on shoulders of “not enough time” / “undecided customers” / “bad specs” / “someone before me made a mess here”, but as a core of this problems we do point our fingers at bridges! Yes we do, we are bravely stating that those things whose main purpose is making our daily commute to other side of a river easier, were actually build for plenty of centuries already, whole generations of architects were thinking about every single detail of those constructions, and most importantly it was written and coined how to build them. And we poor software developers are coding for a few decades at most, it is still wild wild west, nothing is straight or settled and we basically have to walk with our eyes blindfolded to finish our jobs.

But maybe, just maybe, there is knowledge who applies to software creation process pretty well, maybe there are already core values that we can use, maybe there is a parallel world of programming ? Well while reading “The Toyota Way” by Jeffrey K. Liker I had exactly those thoughts. I was fascinated by similarities between our profession and the way Toyota organized their company to become world no. 1 car manufacturer. At first I was skeptical – how on earth ideas how to create a repeatable product can apply to something that was closer to prototyping one single vehicle. After immersing myself into this subject I found out they actually can – and they were partially for example in Agile Movement.

I hope you liked this teaser post, more in next part of this small series…

Autonomous positioning system in a confined space based on MPEG-7 descriptors and physical markers (ARTags)

2010-09-01T00:54:55Z

I was inspired (by really really great person) to contribute my master thesis to global www heap. If it may actually help even one person then it is more then worth it. My work was finished two years ago, send to a drawer, but maybe someone can use this in process of creating something valuable? I do regret only one thing, it was written in Polish, so it might be an obstacle, but I’m willing to answer any question if someone gets really interested.

Main idea of my master thesis was to answer a question how to help vision impaired people in environments that haven’t been thought about them in a first place. Since I was always more practically, then theoretically, oriented I decided to create an actual software prototype that can be run in embedded devices (my platform of choice was signal processor Blackfin from Analog Devices). System consisted of camera to gather live data, signal processor to process it and software to decided weather something important for blind person is in front of him. Warnings or information could be send to this person in form of voice commands.

I decided to check two tracks – first one was using designed for Augmented Reality tags – called ARTags (if you haven’t seen what Augmented Reality is – I do truly encourage you to see some demos on youtube – possibilities of this idea are just mind bubbling). Main goal was to put ARTags on some important obstacles (like doors, near stairs etc), and make my software recognize them in real time. As signal processors are designed in terms of power efficiency my software was using library specifically written for small devices (like PDAs, smartphones etc). It is called ARToolkitPlus, and I have ported it for Analog Devices environment needs. Source codes on request.

My second idea wasn’t really explored before (or at least Google said so). I was thinking how do describe an image in a way that we can actually tell that in front of us is some specific shape or obstacle. My weapon of choice were Visual Descriptors from MPEG-7 standard. I checked two - Color Layout and Edge Histogram. As first was really very tightly tied to specific location (different buildings – different colors), and prone to poor light conditions, the latter gave surprisingly quite good results. Edge Histogram can extract edge distribution in an image, it means it can actually describe shapes of some specific locations. Stairs happened to be very good subject, as their recognition level was very high. As a matter of fact they are a big threat for vision impaired people and I think universal warning system could find a real life usage. How it worked ? Well at first I made pictures of different stairs locations and “described” them using Visual Descriptors. By that we gained a set of attributes with whom we can compare live pictures gathered by camera working as a blind man eyes. Every image captured by signal processors is described using this same descriptors and then compared with database of previously tagged shapes.

As I mentioned at the beginning of this article I would be very happy if this information can be useful to somebody, instead of laying and catching dust on my hard drive. Maybe you should share your work with world as well, who know what combined knowledge of us all can create ?

Presentation about my master thesis

My master thesis

C2C Conference

2010-03-16T18:51:15Z

This years Communities 2 Communities conference is going to be held 17th of April 2010 at Warsaw. At a glance this is a Microsoft community conference, focused on Microsoft products. It’s divided into 3 paths, .NET, SQL Server and IT professionals. I’m really looking forward to see Hadi Hariri speech “Deep Dive into ASP.NET MVC” as I’m really into this subject recently. Of course there are a lot of others great speakers but for details please visit conference site.

This year I was involved in it’s organization, so I think I should strongly encourage you to visit it :-)

ReadOnly WPF containers

2010-02-04T18:20:35Z

One thing has been really annoying me for a long time – when I was setting IsEnabled property on some WPF container to false all its contained controls automatically became disabled as well. Ok, that really wasn’t a source of my annoyance – it was rather expected. Really frustrating thing is that you are unable to scroll a DataGrid, to copy and paste a text from TextBox etc. One can say – but there is a ReadOnly property – and he’s going to be right, but unfortunately it’s available on only few of these controls. TextBox for instance has an IsReadOnly property and DataGrid control that I’m currently using has a ReadOnly property.

After a while of googling and finding nothing I decided to do it my way. At first I thought that it will be a great idea to have an attached property on one of container controls. That control could recursively look through all its children’s and decide whether it should be set to ReadOnly or Enabled. Few whiles later I had my code ready but something wasn’t right. When I was trying to iterate through my controls collection it turned out that my container is empty ? But it wasn’t. Ok, in fact it was at that moment, apparently the visual/logical tree is empty until the Window undergoes layout at least once. Now that was a pity. I had an option to do it by utilizing OnContentRendered(EventArgs e) event method but it would rather make my code ugly, dependent and evil.

Finally I decided do create some extension methods and set this property from code-behind. It’s still something that I’m not proud of, but well, it works. If someone has a better idea please don’t hesitate to write a comment or contact me.

Some code:

public static class ControlsExtensions
{
    public static void IsReadOnly(this GroupBox gb, bool isReadOnly)
    {
        RecurseTree(gb, isReadOnly);
    }

    // more extension methods for other containers...

    private static void RecurseTree(FrameworkElement control, 
                                    bool isReadOnly)
    {
        foreach (var c in LogicalTreeHelper.GetChildren(control))
        {                
            var frameworkElement = c as FrameworkElement;
            if (frameworkElement == null) continue;

            var tb = c as TextBox;
            if (tb != null)
            {
                tb.IsReadOnly = isReadOnly;
                continue;
            }

            var dg = c as SomeDataGrid;
            if (dg != null)
            {
                dg.ReadOnly = isReadOnly;
                continue;
            }

            if (c is GroupBox || c is DockPanel || c is Grid 
                || c is StackPanel || c is TabControl)
            {
                RecurseTree(frameworkElement, isReadOnly);
            }
            else
            {
                frameworkElement.IsEnabled = !isReadOnly;
            }
        }
    }
}

You might notice that I don’t repeat recursion for not container types – that is intentional in terms of my project, and it saves a few processor cycles.

I do think that somewhere out there is a more elegant solution, if you know one please let me know.

Technorati Tagi: WPF

25th PG.NET meeting

2010-01-31T18:11:37Z

On 25th PG.NET meeting I’ve had presentation concerning S#arp Architecture. I do really like Steve Jobs style of presentation. I mean, I’m far from his abilities but I just believe in two things – 1. tell a story 2. don’t put to much information on a single slide. Unfortunately there is a downside of that kind of presentation - there is really nothing to show on internet unless you are going to record it live.

So here it is, almost no text (except the who-can-help.me slide – It was intentionally crowded to rise a question – are there too many third party frameworks and where is it going?). Oh and I’ve changed S#arp Architecture logo color (as a matter of fact I’ve inverted colors) to better match my background color :-)

Notes to remember:

while presenting your code live everything can happen, maybe it is better to record it first ? On the other hand doing real live coding is cool :-)
don’t forget what your next thing to show is while you are doing live code demonstration (again – recording first ?)
make more questions to audience, involve them (it works great to set their focus)

So far that’s all, I do really like being a speaker and I’m thinking already about next presentation.

Technorati Tagi: S#arp Architecture,speaking

How to create a successful product

2010-01-06T23:21:55Z

Well it turns out that’s not so hard, Leon Bambrick @ secretgeek.net came out with a great idea! Just check it out, it’s worth it - The Ultimate Agent of WERF Destruction.

It took me a while to recover after that shocking idea :-) (back to normal = to get up from the floor)