Enumerating Over a Custom Collection

Hello,

In this post I would like to do something really simple and show you how to implement an IEnumerable on a custom collection. Some of you may ask why would I need to implement this interface on my custom collection? Usually the answer has to do with using the foreach loop, which, without the IEnumerable interface won’t be available to the collection users.

Ok, so, let’s say you have a simple Loan class with the following definition:

internal sealed class Loan
    {
        public Loan(decimal rate, DateTime maturity, decimal amount)
        {
            if (rate > 0 && maturity > DateTime.Now && amount > 0)
            {
                this.rate = rate;
                this.maturity = maturity;
                this.amount = amount;
            }
        }

        public decimal Rate { get { return rate; } }
        public DateTime Maturity { get { return maturity; } }
        public decimal Amount { get { return amount; } }

        private decimal rate;
        private DateTime maturity;
        private decimal amount;
    }

You created the above Loan class to be able to see the total amount you need to repay by some future date. There is really nothing wrong with simply having an array of Loans (e.g. one for your car, another for your house) and using the foreach on the array. In other words, something like this:

class EntryPoint
    {
        static void Main(string[] args)
        {
            decimal totalToRepay = 0;
            Loan [] loans =  
            { 
                new Loan((decimal)0.03, DateTime.Parse("19/04/2016"), (decimal)15000),
                new Loan((decimal)0.018, DateTime.Parse("19/04/2016"), (decimal)260000)
            };

            foreach (Loan l in loans)
                totalToRepay += ((l.Maturity - DateTime.Now).Days / 365) * l.Rate * l.Amount + l.Amount;

            Console.WriteLine("{0} {1,2:C}", "You will need to repay an amount of ", totalToRepay);
            Console.ReadLine();

        }
    }

The above works for simple single-threaded use. However, as soon as you introduce a possibility of one thread updating the loans array while another thread is iterating over it, you need to implement some kind of synchronization mechanism. The loans array inherits the ICollection implementation from the Collections class. This interface has a SyncRoot property which allows to lock the variable. At this point you begin to realise that a simple array of loans may need to evolve into a custom collection with added synchronization benefit. You go ahead and implement a collection of loans class:

internal sealed class LoanCollection: IEnumerable
    {
        public LoanCollection(Loan[] loanArray)
        {
            loanCol = new Loan[loanArray.Length];
            for (int i=0; i< loanArray.Length; i++)
            {
                loanCol[i] = loanArray[i];
            }
        }

       public IEnumerator GetEnumerator()
       {
           lock (loanCol.SyncRoot)
           {
               for (int i = 0; i < loanCol.Length; i++)
                   yield return loanCol[i];
           }
       }

       private readonly Loan[] loanCol;
    }

Note how simple it is to synchronize with the lock statement. C# compiler will be busy expanding this statement into a proper Monitor.Enter and Monitor.Exit block. Also note that for a non-generic IEnumerable we only need to implement the GetEnumerator method. So, are we done here? Not quite, as I always like to point out alternative and sometimes simpler methods for doing the same things. Instead of separating the enumeration and the synchronization, we could attempt to get both of these features in one go. To do this, we can inherit from a thread-safe System.Collections.Concurrent.ConcurrentBag and simplify the LoanCollection class a little bit. The ConcurrentBag is a generic collection, so we will restrict the type to Loan class only:

internal sealed class LoanCollectionSync<T> : System.Collections.Concurrent.ConcurrentBag<T> where T : Loan
    {
        public LoanCollectionSync(T[] loanArray)
        {
            foreach (T l in loanArray)
            {
                Add(l);
            }
        }
    }

How cool is that? Not only we get the IEnumerable from the ConcurrentBag, but the collection is now thread-safe as well! There is a special TryPeek method in the ConcurrentBag that can be used to iterate over the collections. The built-in enumerator is also thread-safe (i.e. represents a snapshot of a collection in the moment in time before any updates), so, we can still use the same iteration approach as before:

LoanCollectionSync<Loan> syncLoans = new LoanCollectionSync<Loan>(loans);

   foreach (Loan l in syncLoans)
      totalToRepay += ((l.Maturity - DateTime.Now).Days / 365) * l.Rate * l.Amount + l.Amount;
   
   Console.WriteLine("{0} {1}", "You will need to repay an amount of ", totalToRepay.ToString("C"));
   Console.ReadLine();

Thanks for reading and have a nice day!

Advertisements

Using Boost Math Library in C#

Hello,

In this post I will share with you a lazy way to expose boost.math library to a C# project using C++/CLI. It is lazy and very wrong to do it this way, but I would like to share it with you anyway. I am doing this because of the following reasons:

  • Mixing different languages and frameworks is very difficult and error-prone. I have looked through tons of sites on ways to expose unmanaged functionality to the managed code, and apart from PInvoke, I have not found any stable and simple solution. Hopefully the step-by-step solution I am sharing may turn out to be helpful to someone one day.
  • My approach is by far not the best way to expose the Boost library to C#. But it is dead easy. I am aware of this, and if you know a better way – please let me know by contributing to this post.
  • There are times when all you need is a quick hack to get something done. This solution is just that, and it should not, ideally, be a part of a production system used by many people. Keep this in mind.

Now that the disclaimer part is over, here is the step-by-step solution.

Limitations

The limitations of this solution are huge. Firstly, it cannot be used it a multi-threaded environment. Boost library has an atomic component that is not CLR compatible. Inside Boost.Math (v. 1.57.0), the Bernoulli distribution class uses atomic data type, I believe for concurrency. My solution does not work with atomic and therefore won’t let you utilize the Bernoulli distribution functions.

A second important limitation has to do with the overall approach. I am not a C++/CLI expert, and my approach is limited to what I know about the language. However, the C++/CLI dll is the bridge between Boost and C# I am about to show. In my solution it requires turning everything into a function that returns some basic value type. The function can then be used by a C# program, but the return type always has to be a basic value type. For example, imagine you need to create an instance of a normal distribution with some known mean m and standard deviation s. You are then planning to use this instance to calculate quantiles and cdf. C++/CLI won’t let you to return an instance of unmanaged boost::math::normal class to C#. So, you need to keep the class inside the C++/CLI code, and instead expose the functions that return some probability from cdf or a quantile as doubles.

Solution Summary

After downloading the Boost library,  create a Visual C++ CLR class library. Add the boost library location as the additional include directory to the project.  Add a new C++/CLI ref class exposing the required boost functionality to the solution’s header file. This is done as function calls accepting basic value type arguments and returning basic value type parameters. Modify the atomic file by commenting out the error definition for managed code, and compile the solution as a dll. This dll can be referenced by a C# project and the underlying methods that expose the Boost.Math library functions can be called from your C# code.

Step-by-Step Solution

1. The latest stable boost library can be downloaded from http://www.boost.org. I am using version 1.57.0. Unzip its contents to a local folder (e.g. C:\Program Files (x86)\boost).

2. Using Visual Studio, create a new C++ CLR Class Library project. I am using VS 2013, and the location of a class library project for me is Visual C++->CLR->Class Library. Give it whatever name you want (e.g. TestClassLibrary). The project solution will contain header files, resource files and source files. You will only need to modify the header file, i.e. the TestClassLibrary.h.

3. Right-click on the project name and go to Properties. On the C/C++ properties, add the location of your boost library to the ‘Additional Include Directories‘. If you want, you may also add the same location to the additional #using directories. I don’t do this because I fully qualify the boost namespaces.

4. Let’s say that for this example we would like to expose two functions from Boost.Math: calculate the inverse of the incomplete beta function and calculate some quantile of a normal distribution with custom mean m and standard deviation s. To do this, add the following code to the solution’s header file:

// TestClassLibrary.h

#pragma once
#include <boost\math\special_functions\beta.hpp>
#include <boost\math\distributions\normal.hpp>

using namespace System;

namespace TestClassLibrary {

	public ref class BoostMathExpose
	{
	public:
		double static InverseIncompleteBeta(double a, double b, double x)
		{
			return boost::math::ibeta_inv(a, b, x);
		}
		
		double static NormalDistribution(double m, double s, double quantile)
		{
			boost::math::normal sol = boost::math::normal::normal_distribution(m, s);
			return boost::math::quantile(sol, quantile);
		}

	};
}

5. If you try to compile this code, you are going to get a compilation error about atomic class not being compatible with /clr or /clr:pure switch. To fix this, I resolved to commenting out the three lines that cause this error. You can find the relevant atomic file in C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include directory. You need to be logged in as an administrator to change this file. Because it is marked as read-only, make a copy of the modified file somewhere else (e.g. Desktop folder) and then copy and paste the modified file replacing the original. As I mentioned before, because you are disabling this warning, bad things can happen if you attempt to use these methods in a multi-threaded environment. The lines to comment out are:

#ifdef _M_CEE
#error is not supported when compiling with /clr or /clr:pure.
#endif /* _M_CEE */

6.Once you make the above change, the C++/CLI solution will compile and produce a dll. Create a C# project from which you are planning to call the dll functions. In the project solution explorer add a reference to the dll file (e.g. TestClassLibrary.dll). Once done, you will be able to see two methods InverseIncompleteBeta and NormalDistribution. Here is an example C# code that calls these methods:

using System;

namespace TestUsingCLI
{
    class EntryPoint
    {
        static void Main(string[] args)
        {
            double a = 2;
            double b = 3;
            double x = 0.4;
            double answer = TestClassLibrary.BoostMathExpose.InverseIncompleteBeta(a, b, x);

            double mean = 70;
            double std = 4;
            double quantile = 0.90;
            double answer2 = TestClassLibrary.BoostMathExpose.NormalDistribution(mean, std, quantile);

            Console.WriteLine(answer.ToString()+" "+answer2.ToString());
            Console.ReadLine();
        }
    }
}

That is all. Thanks for reading.

There is a String Class, and then there is a SecureString Class

Greetings to my blog readers!

Have you ever worked on an application that required accepting a password or other sensitive data from a user? If yes, then how did you handle the password retrieval and storage? Since most passwords, or other sensitive data like national insurance number, can contain alphanumeric values, it is sensible to treat them as strings. In C#, string or String is an immutable reference type with value-like semantics. This means that passing a string as a parameter to a method that locally modifies its value results in locally changed copy, with the original string unchanged. One approach to handling passwords can be to accept user’s input as a string, encrypt or hash it, and then store away the encrypted value. The unencrypted string would be collected by the Garbage Collector (GC). The problem with this approach is that until the GC gets around to reclaim the managed memory, the password exists in its unencrypted form in the process’s heap. This may present a security gap.

C# offers a very useful class for sensitive data processing. The class is SecureString. It allows one to allocate a block of unmanaged memory and encrypt its content at initialization. The class has no built-in methods for decrypting the string back, which makes it more secure. Let’s look at an example:

using System;
using System.Security;

namespace MySecureString
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Please enter your password:");

            // SecureString implements IDisposable
            using (SecureString password = new SecureString())
            { 
                ConsoleKeyInfo nextKey = Console.ReadKey(true);
                while (nextKey.Key != ConsoleKey.Enter)
                {
                    password.AppendChar(nextKey.KeyChar);
                    Console.Write("*");
                    nextKey = Console.ReadKey(true);
                }

                // makes the password immutable
                password.MakeReadOnly();
                // ...
            }
        }
    }
}

In the example code above we are creating an instance of SecureString and initializing it with the user’s input. Note that I wrapped the example in the using block to avoid unmanaged memory leak. A call to MakeReadOnly ensures that the encrypted password cannot be modified, which SecureString class achieves by maintaining an internal boolean flag.

Ok, so how does this actually work? The data encryption is a process of several stages. SecureString class makes a call to the native Win32 SystemFunction040() method passing it the pointer to the memory buffer with user’s input, its length and the CRYPTPROTECTMEMORY_SAME_PROCESS flag. This method encrypts the memory in place, meaning that the memory buffer is overridden with encrypted content. The flag sets the rules on which process is allowed to decrypt this memory content (e.g. the current process only). The encryption is done according to the Windows Data Protection API (DPAPI), which offers password-based data protection service. In short, the data protections is done using a MasterKey, derived with the help of user’s system logon password, and a session key. An additional ‘secret’ word can be supplied to strengthen security. Only the MasterKey is stored in the user’s profile, to allow for later decryption of the data. By default, MasterKey is stored in the user’s profile folder, which is %AppData%/Microsoft/Protect. Also, by default, the key expires every three months, after which it gets re-generated.

Since SecureString class does not expose any methods for decryption, a call to the Windows native SecureStringToBSTR method has to be made. Here is an example of how to decrypt the password:

IntPtr bstr = IntPtr.Zero;
// it is important to free the memory pointed to by IntPtr
// therefor we use try/finally block
try
{
    bstr = System.Runtime.InteropServices.Marshal.SecureStringToBSTR(password);
    Console.WriteLine("Original password is: " + System.Runtime.InteropServices.Marshal.PtrToStringUni(bstr));
}
finally
{
    System.Runtime.InteropServices.Marshal.ZeroFreeBSTR(bstr);
}

Hope this post has at least triggered your curiosity about DPAPI. Thank you for reading!