Archive for Programming

Reusing PHPBB’s Authentication System

Question:

What’s the easiest way to let a custom application re-use accounts from a PHPBB installation?

Answer:

The first thing that occurred to me was that I’d have to look over the PHPBB user tables, connect to mysql, and read the user’s password (which I assume is hashed) to do the authentication. The second idea I had was to write a proxy service in PHP that I could call from my custom app.

Both of those options seemed like an awful pain in the butt. Then I had an epiphany, if you take a look at the ideals behind REST services, one could reason that PHPBB already exposes a REST service for logging in.

What if I simply used the already exposed HTML form as a sort of pseudo web service?

Making HTTP posts with C# is pretty easy, but I didn’t want to try to remember what the syntax was for the API; So I did a quick search online and found a nice little class that makes it really easy:
http://geekswithblogs.net/rakker/archive/2006/04/21/76044.aspx

With this class in hand, I used FireBug while logging in to the PHPBB and inspected the post variables. Easy, there were only four variables posted:

  • autologin=on
  • login=Login
  • username=[My Username]
  • password=[My Password]

I grabbed the URL it was posting to and added the variables (see the example below). Once I get the result back, a simple regular expression parses through the HTML and figures out if the login was successful or not. The result worked surprisingly well … the method below takes a username, password, and url. The URL will look something like this:

http://mywebsite.com/forum/ucp.php?mode=login&sid=9cd8b8da2649060b9d22d297f27a1dc7

private static bool Authenticate(string username, string password, string url)
{
    PostSubmitter post = new PostSubmitter();
    post.Url = url;

    post.PostItems.Add("autologin", "on");
    post.PostItems.Add("login", "Login");
    post.PostItems.Add("username", username);
    post.PostItems.Add("password", password);
    post.Type = PostSubmitter.PostTypeEnum.Post;
    string result = post.Post();

    string loggedinstring = string.Format("Logout \\[ {0} \\]", username);
    Regex r = new Regex(loggedinstring, RegexOptions.IgnoreCase);
    var match = r.Match(result);

    return match.Success;
}

Disclaimer: I’ve only tried this code with one instance of phpbb, not sure if it will work with other versions. The point of this was that we can trivially reuse existing user stores that are already exposed on the web via simple html forms.

Comments (1)

Custom Transactions

If you haven’t used TransactionScope from the System.Transactions namespace, you don’t know what you’re missing.  This system, introduced with .NET 2.0 provides a flexible mechanism for allowing your code to take part in transactions.  Many of the built-in subsystems such as ADO.NET automatically enlist in these transactions, but the real power comes from the fact that you can allow your own custom code to also take part in ambient transactions.

Recently at work, I wanted to perform a transaction across several unrelated modules with a custom resource. Of course TransactionScope was the first solution brought up. Unfortunately, after some analysis, I realized that this wasn’t going to work for us; We have existing database code in those modules that have different conditions in which the transaction would be rolled back.

We needed a way to have multiple transaction scopes, each with different conditions of success or failure. I started thinking about how to accomplish this, and decided to write my own implementation which mimics the TransactionScope, but lets me control things a bit closer.  So I came up with a class which can be used like this:

using (Txn scope = Txn.New<MyScope>())
{
    // ... do work

    scope.Commit();
}

Notice that the usage pattern is nearly identical to the TransactionScope API … put the scope in a using statement, and call .Commit if the work was completed.  In the example above, the MyScope class is defined quite simply as:

public class MyScope : Txn
{
    protected override void OnStart()
    {
        Console.WriteLine("\tstarting");
    }

    protected override void OnCommit()
    {
        Console.WriteLine("\tcommitting");
    }

    protected override void OnRollback()
    {
        Console.WriteLine("\trolling back");
    }
}

All you have to do is inherit from the Txn class, and implement three methods: OnStart, which occurs when the transaction is first beginning; OnCommit, which is invoked only when the top-most scope exits and all sub transactions were committed successfully; And OnRollback, which as you might imagine is only called if the transaction (or a subtransaction) was not committed successfully.

One difference between this API and the regular TransactionScope is that only one instance of “MyScope” will be created when the top-most transaction is first created. As I’ve alluded to, you can nest transactions just as you can with TransactionScope. And each scope must be committed if the entire transaction is to be completed.

The Txn class can be found below:

public abstract class Txn : IDisposable
{
    private Queue<bool> committed = new Queue<bool>();

    public Txn()
    {
    }

    [ThreadStatic]
    public static bool committable = true;

    [ThreadStatic]
    public static int depth = 0;

    [ThreadStatic]
    public static Txn current;

    protected abstract void OnStart();
    protected abstract void OnCommit();
    protected abstract void OnRollback();

    public void Commit()
    {
        this.committed.Enqueue(true);
    }

    void IDisposable.Dispose()
    {
        if (committed.Count == 0 || !committed.Dequeue())
        {
            committable = false;
        }

        depth--;
        if (depth == 0)
        {
            if (committable)
            {
                this.OnCommit();
            }
            else
            {
                this.OnRollback();
            }

            current = null;
        }
    }

    public static T New<T>() where T : Txn, new()
    {
        depth++;

        if (current == null)
        {
            current = new T();
        }

        if (depth == 1)
        {
            // first transaction, assume committable
            committable = true;
            current.OnStart();
        }

        return current as T;
    }

    #region IDisposable Members

    #endregion
}

There is one caveat to mention. My requirement was to run this code in a windows service. The entire scope of the transaction would be single threaded, but there would be multiple ongoing transactions at once. To support this scenario, notice that some of the internal state of the Txn class uses the [ThreadStatic] attribute. This means that the API can be used from multiple threads at once and each thread would have its own state.

Of course, this might be a problem if you want to use this in an ASP.NET project. I’ve written about this issue before. There is probably a way to make this work using the techniques I outlined in that article, but I haven’t given it a lot of thought (because I didn’t need to). But I thought I’d share the work I did in case it is useful for anyone else.

Comments

Resource Pool in F#

Erik Schulz, who has written a bunch of articles on how to make XNA games with F# has ported the Resource Pool class I wrote about in F#. Cool!

type FPool<'a>(newRoutine, capacity : int) =
    let queue = Queue<'a> capacity

    member this.Count = queue.Count

    member this.New() =
        if queue.Count > 0 then queue.Dequeue() else newRoutine()

    member this.Return(item) =
        queue.Enqueue(item)

I’ve been toying around with F# recently, it’s good to see an example that you can easily compare and contrast with the C# version. Has anyone else tried out F#?

Comments

The Next Decade in Software

The scientists are already hard at work at coming up with predictions for the next 10 years. I thought it would be interesting to extract from the list, the predictions that are directly related to software … along with a few recommendations on how to get started today.

  • Augmented Reality
    This one is almost already a reality. A number of iPhone apps use the compass, gps, and camera as a viewport to overlay information on top of the world. A few years ago, Johnny Lee posted some amazing videos on how to use cheap consumer hardware to meld real and virtual worlds. It is clear that 3D graphics programming will be very helpful in this field … why not start learning now?
  • True artificially intelligent computer programs
    This one seems to be the most far fetched. Not because the field is not progressing, because it is, but because it’s been promised for so long. The biggest problem I think is lack of a clear goal; you can say you want something that learns, but learns what? There needs to be a catalyst that makes it clear what people really want/need from an intelligence. In my opinion, the game industry is pushing the state of the art here.
  • Cell phone apps that will act as a health monitor … Sort of an OnStar system for the body
    Since the iPhone proved to everyone that you can have a powerful computer in your pocket, new devices have started appearing that try to piggyback off of the success. I’m looking forward to more people making symbiotic peripherals for phones like the iphone, android based devices, and hopefully WinMo if they ever get their act together.  Mobile programming will surely be a skill to watch in the next decade.
  • Cloud computing
    I was a little surprised to see a reference to cloud computing in a science article. But it makes sense when you think about it, because scientists need to make increasingly complex models to prove their theories and solve problems, they will need more and more processing power to do it.

    I don’t think that it will be worth it for the average “you and me” to invest time into learning how to make cloud computing infrastructures. The big 3 (Microsoft, Google, and Amazon) will commoditize these infrastructures and make them easily available … for a price of course. Learning how to make programs that thrive in the clouds however, will prove to be useful. I’ve written some about this topic recently. You can start programming in functional languages like F#, or learn how to program shaders using HLSL to teach your brain how to think in highly parallelizable terms.

Time is always the ultimate author of history, so we will have to wait and see how many of these predictions turn out to be true.  But one thing is clear, if you learn how to make games, there is a good chance you will be successful in the next decade ;-)

Comments (1)

Cleaning up after MSTest

For those of you that use MSTest, you’ve no doubt noticed a folder called “TestResults” which can get quite large.  This is because each test run copies all of the project output into that folder to store the results of the run.

Well, I updated my little open source tool to take care of it: http://vsclean.codeplex.com

As it’s traversing the directories, it will delete any of those “TestResults” folders that it comes across.  Download now!

Comments

Semi-Literate Programming with C#

Recently, I’ve been reading the book Coders at Work, where author Peter Seibel interviews lots of well known developers.  One of the questions that he often asks is whether they have tried Literate Programming, an idea introduced by Donald Knuth in the 70s.  Although most of them say no, some of them have tried it briefly.  They usually come to the conclusion that it’s an interesting idea, but a lot of the tooling doesn’t really make it a realistic solution.

The author’s insistence at asking them about this got me thinking about some of the problems that I’ve encountered during my career.  Could literate programming be applied in this modern day and age to help solve some of the problems that we all face?

A Use Case

I started thinking about the types of code where these questions tend to come up with. It’s usually around code where business analysts make up some business rules.  Let’s use an example.  Say you work at a company that needs to creating billing statements for clients.  When you process the statements, there are a series of fees which you must attach to the bill based on certain conditions.

The rules may have been explained to the developer in a face to face meeting like this:

  • Apply a 7% tax on the principal when the client is in Florida
  • A $50 flat fee will be applied when a New York based client maintains a principal of more than $1000

So I, as a developer will scuttle away and create the program per specifications. The application goes to production and everyone is happy. Two years pass by, most of the original staff that was working when the system was originally deployed has turned over, and the new staff has a question about why some fees are being charged for a given client.

In an ideal world, the business will refer back to documentation that they created when wanting to know about the behavior of some system that you programmed. However, in the real world, a more likely scenario is that they will end up asking you over the phone about some obscure section of the code and you end up having to crack open the source to figure out what the code is doing in that piece.

The Solution?

I want a solution that lets me write code, and without manual intervention, allow other developers and the end user to understand what the business logic is doing.  Of course, most people will point to the XML documentation feature of C# along with auto-documentation products like SandCastle, and suggest that this is enough.  However, maintaining XML comments violates the “without manual intervention” part of my own requirements. It also creates output that is not really consumable by end users.

There is also another issue that most people probably don’t really think about. A lot of the code in today’s projects is not really … useful, to document. Serialization code, parsing code, data access code … most of that is pretty standard.  Developers will easily understand it assuming they already know how to use the APIs like ADO.NET, and WCF.  And users won’t care about it.  So that really just leaves the fundamental logic that is the raison d’être for your application in the first place. This is what I am interested in making easily available for a human to read.

For the solution, I wrote a simple Rule class:

public class Rule<T>
{
    private Expression<Action<T>> expression;
    private Expression<Func<T, bool>> evalExpression;
    private Action<T> compiled;
    private Func<T, bool> evalCompiled;

    public void Execute(T context)
    {
        if (this.evalCompiled(context))
        {
            this.compiled(context);
        }
    }

    public Expression<Func<T, bool>> Evaluation
    {
        get { return this.evalExpression; }
        set
        {
            this.evalExpression = value;
            this.evalCompiled = value.Compile();
        }
    }

    public Expression<Action<T>> Action
    {
        get { return this.expression; }
        set
        {
            this.expression = value;
            this.compiled = value.Compile();
        }
    }

    public override string ToString()
    {
        return string.Format("if {0} then {1}",
                                    this.Evaluation.Body,
                                    this.Action.Body);
    }
}

This class takes, as a generic parameter a context which represents one item that needs to be processed.  You will set two lambda expressions: the Evaluation, and the Action.  The evaluation will return true if the action is to be applied.  An example can be seen below:

List<Rule<BizContext>> rules = new List<Rule<BizContext>>();

rules.Add(new Rule<BizContext>()
{
    Evaluation = c => c.State == Florida,
    Action = c => c.Fees.Add(c.Principal * .07M)
});

rules.Add(new Rule<BizContext>()
{
    Evaluation = c => c.State == NewYork && c.Principal > 1000.00M,
    Action = c => c.Fees.Add(50.00M)
});

“BizContext” in the above code can contain anything that pertains to the item that needs to be processed.  In our case, the analyst’s rules say that we need to operate based on the principal and client’s state, and add fees.  So those are the properties that the context contains. Because the rules were added to a list, you can iterate through the list and call the rule class’ “Execute” method.

foreach (var rule in rules)
{
    rule.Execute(context);
}

So far, there’s nothing groundbreaking about the Rule class. I’m sure many of you have written something similar time and time again. But here’s where the literate programming comes into play. Because the “Evaluation” and “Action” methods are actually Expressions … we have access to the textual representations of the code, in addition to having the ability to execute it.

The overridden .ToString method on the class will output an easy to understand string of the business rules using actual code that will execute when it’s run. So for the two rules defined above, you can get a printout like this:

if (c.State = Florida) then c.Fees.Add((c.Principal * 0.32))
if ((c.State = NewYork) && (c.Principal > 1000.00)) then c.Fees.Add(50.00)

The end user gets a realistic printout of the actual business logic in the system on-demand; And the developer doesn’t have to do anything to update this when the business logic changes.

So there you have it; I wouldn’t exactly call it full fledged literate programming with C# in the way that it was described originally.  But I think that it embodies the qualities of literate programming, where documentation and code are one and the same. It’s a compromise, and it would be interesting to see if this approach can be implemented in a real-world scenario.

Any takers? :-)

Comments (1)

Qizmt: MapReduce Framework in C#

I was recently surprised to find that MySpace had open sourced a distributed “MapReduce Framework” called Qizmt (http://qizmt.myspace.com/).  From the site’s description:

MySpace Qizmt [kiz-mit] is a mapreduce framework for both developing and executing distributed computing applications on large clusters of Windows servers.

This has been a topic that I’ve been interested in for a while so I’m glad to see that someone has been making progress in this space for the .NET world.  Dryad remains an interesting prospect which is apparently even seen production use in Microsoft’s ad service, however it’s clear that it hasn’t been “productionized” yet.

One interesting aspect of Qizmt is that it seems they paid a lot of attention to making it easy to deploy.  In my opinion, the ability for a developer to easily get started with a one machine install is a key enabler for the uptake of any new technology.  Hopefully with more competition for technologies such as this, we will see some cool options become available to us as developers for high performance computing.

Comments

Executing PowerShell Scripts via C#

When Dave asked me for some help with a little side project of his that he was researching, I jumped at the chance.  The requirement was to execute a powershell script programmatically and pass in some parameters that were gathered from a simple form.

I had been wanting to learn more about powershell since it came out (the original codename was called monad) and this was the perfect opportunity.  The end result is a simple little static class that you can use to execute a powershell script and pass in some parameters in a strongly typed fashion.

Here’s a sample usage:

string result = PowerShell.Execute(
    @"c:\users\joel\dev\script.ps1",
    () => new
    {
        server = formserver,
        fname = formname
    });

Although there are quite a few little nuances involved in the execution from a command line, once you figure them out it’s quite easy.  The class basically just does a Process.Execute on the PowerShell.exe command and passes in some command line arguments that executes the ps1 file.

I opted to do this instead of the cleaner API that is available via hosting the powershell runtime because that has an additional requirement on System.Management.Automation.dll which must be installed with the windows sdk.  I didn’t want to introduce this dependency for the project, so the command line method was preferrable.

Below is the class … you’ll obviously have to include a few extra using statements at the top of your file, but you can find those easily.  Enjoy!

public static class PowerShell
{
    public static string Execute(string scriptPath, Expression<Func<object>> parameters)
    {
        string shellPath = "powershell.exe";
        StringBuilder sb = new StringBuilder();
        sb.AppendFormat("\"& '{0}'\" ", scriptPath);

        NewExpression n = parameters.Body as NewExpression;

        for (int i = 0; i < n.Members.Count; i++)
        {
            var member = n.Members[i];
            var value = n.Arguments[i];
            string paramValue;
            if (value is MemberExpression)
            {
                paramValue = Expression.Lambda(value).Compile().DynamicInvoke().ToString();
            }
            else
            {
                paramValue = value.ToString().Replace("\"", string.Empty);
            }
            sb.AppendFormat(" -{0} {1}", member.Name.Replace("get_", ""), paramValue);
        }

        string result = ExecuteCommand(shellPath, sb.ToString());
        return result;
    }

    private static string ExecuteCommand(string shellPath, string arguments)
    {
        arguments = "-noprofile " + arguments;
        var process = new Process();
        var info = process.StartInfo;

        process.StartInfo.UseShellExecute = false;
        process.StartInfo.FileName = shellPath;
        process.StartInfo.Arguments = arguments;
        process.StartInfo.RedirectStandardError = true;
        process.StartInfo.RedirectStandardOutput = true;

        process.Start();

        var output = process.StandardOutput;
        var error = process.StandardError;

        string result = output.ReadToEnd();
        process.WaitForExit();
        return result;
    }
}

Comments (2)

Static Access to Request-Specific Data

I wrote a post over on the nGenSoft Blog talking about how to gain Static Access to Request-Specific Data:

As we have all come to learn in the last decade plus of web development, web applications are inherently stateless.  Unlike their native client cousins, every request must be treated as if it was done in isolation from any other user action.  This can tend to complicate application level concerns.  More often than not people just end up polluting their application by mixing code that is related to servicing the http request, with their business logic.

We wanted a way to maintain application related plumbing such as database connections neatly maintained, without having to always worry about the stateless nature of http requests.  We noticed that ASP.NET has a really nice pattern that works really well in the HttpContext.Current property.  This is a static property that contains information about only the current request … at first I couldn’t figure out how this works because ASP.NET is by nature a multi-threaded environment.  How was it segregating the information, which is accessed statically, to each individual requests?


Update: repost of original text can now be found below

As we have all come to learn in the last decade plus of web development, web applications are inherently stateless. Unlike their native client cousins, every request must be treated as if it was done in isolation from any other user action. This can tend to complicate application level concerns. More often than not people just end up polluting their application by mixing code that is related to servicing the http request, with their business logic.

We wanted a way to maintain application related plumbing such as database connections neatly maintained, without having to always worry about the stateless nature of http requests. We noticed that ASP.NET has a really nice pattern that works really well in the HttpContext.Current property. This is a static property that contains information about only the current request … at first I couldn’t figure out how this works because ASP.NET is by nature a multi-threaded environment. How was it segregating the information, which is accessed statically, to each individual requests?

After doing some research online, I finally figured out a great way to maintain request level state across different components (ie. http module –> http handler –> mvc action filter –> etc.). I did a good bit of searching, but found it was succinctly put in a blog post by hanselman:

http://www.hanselman.com/blog/ATaleOfTwoTechniquesTheThreadStaticAttributeAndSystemWebHttpContextCurrentItems.aspx

I started off by looking at (ie. Reflectoring) how the enigmatic HttpContext.Current works. Turns out there’s a lot of magic going on under the hood there with the web hosting framework and further .net remoting. In the end, looks like there are two simple ways to solve this problem:

  • [ThreadStatic] attribute lets you have an instance of your static *per* thread.
  • HttpContext.Current.Items, only usable in the context of asp.net obviously, but correctly manages your scope for the lifetime of the request.

As hanselman puts it:

Today’s lesson learned:the [ThreadStatic] attribute is only useful when YOU control the ThreadPool (and the lifecycle of the threads).

So it seems that in order to solve the problem we need to adapt our strategy. If our app is running in a local client (ie. stateful), we can either use the threadstatic attribute, or nothing at all if we don’t plan on doing complex multithreading. However, if we are executing our application’s code in an asp.net app, we need to use HttpContext.Current.Items. Armed with this knowledge, we could have a small initialization step that lets you set up the strategy for how to manage session information. So in the app_start method of the global asax, we can do something like:

AppContext.SetEnvironment(new AppEnvironment());

Thus, in ASP.NET you have an implementation that can know how to provide the proper scoping for that hosting environment. AppContext is defined as:

public interface IAppEnvironment
{
    public AppContext Current { get; set; }
}
public class AppContext
{
    // instance data
    public IDatabase Database { get; set; }

    // static lifecycl
    private static IAppEnvironment environment;

    public static void SetEnvironment(IAppEnvironment env) { environment = env; }

    public static AppContext Current
    {
        get { return environment.Current; }
        set { environment.Current = value; }
    }
}

The instance data can be whatever you want … in the case of a data-driven app, it can maintain a request level database connection and whatever other information we need to refer to (which you can easily do by just saying “AppContext.Current.Database”). The static “Current” property that everyone would use simply defers to the environment implementation. Below are two implementations of the IAppEnvironment that you can use from ASP.NET and a custom one that you can use in a console app, or unit test.

public class WebEnvironment : IAppEnvironment
{
    public AppContext Current
    {
        get { return HttpContext.Current.Items["appcontext"] as AppContext; }
        set { HttpContext.Current.Items["appcontext"] = value; }
    }
}

public class CustomEnvironment : IAppEnvironment
{
    [ThreadStatic]
    private static AppContext context;

    public AppContext Current
    {
        get { return context; }
        set { context = value; }
    }
}

The CustomEnvironment implementation above just uses the simple thread static attribute since it’s assuming that you will be managing the hosting environment (threading and all) … where in the WebEnvironment, you can defer to the httpcontext stuff since that is handled for you.

Techniques such as these let you focus on your application, while limiting the amount of time that you have to spend worrying complexities of adapting your application to run in a web application.

Comments

Simple Pipeline Event model with C#

After declaring my love for extension methods in the last post, it only seemed appropriate that it would come up again in an answer I gave to a stackoverflow question.  The question stated:

In ASP.NET Web Apps , events are fired in particluar order :

for simplicity Load => validation =>postback =>rendering

Suppose I want to develop such pipeline -styled event

Example :

Event 1 [ "Audiance are gathering" ,Guys{ Event 2 and Event 3 Please wait until i signal }]

after Event 1 finished it task

Event 2 [ { Event 2, Event 3 "Audiance gathered! My task is over } ]

Event 2 is taking over the control to perform its task

Event 2 [ " Audiance are Logging in " Event 3 please wait until i signal ]

after Event 2 finished it task

…..

Event 3 [ "Presentation By Jon skeet is Over :) "]

With very basic example can anybody explain ,how can i design this ?

My answer again leveraged an extension method to simplify the notification of events to each individual handler:

public abstract class Handler
{
  public abstract void Handle(string event);
}

public static class HandlerExtensions
{
  public static void RaiseEvent(this IEnumerable<Handler> handlers, string event)
  {
     foreach(var handler in handlers) { handler.Handle(event); }    
  }
}

...

List<Handler> handlers = new List<Handler>();
handlers.Add(new Handler1());
handlers.Add(new Handler2());

handlers.RaiseEvent("event 1");
handlers.RaiseEvent("event 2");
handlers.RaiseEvent("event 3");

Comments