Tesseract OCR for Xamarin (part 1)

If you’re thinking about getting image recognition into a Xamarin app check out this open source Tesseract OCR port I’ve put together for Xamarin. Few weeks ago this question was asked on StackOverflow and I was surprised to find that there is no free OCR for Xamarin (neither for iOS nor for Android). What developer does when he founds that there is nice and open-source library for other platforms but not for the platform he is working with? Of course he migrates the library. So I created wrappers for Tesseract OCR for Xamarin.iOS and Xamarin.Android and also for Xamarin.Forms. In this post I will give some details about the library, will show how to use it and will tell about some issues and workarounds.

Some history

The Tesseract engine was originally developed at HP in 1985 and was then released as open source in 2005. It has been sponsored by Google since 2006. This wrapper is base on two open-source builds – tess-two by Robert Theis and Tesseract OCR iOS by G8Production.

How to use

First of all you can use either the NuGet package or the source code. This is not the “final stable” version, just one of the first ones. But I’m working on adding features and fixing bugs so if something is broken try getting the last available version. If it’s still broken don’t hesitate to post an issue on the GitHub.

TL;DR

You can find few samples here.

Long story

One of the main ideas I kept in mind is that whichever platform (Android/iOS/Forms, sorry no Windows Phone this time) you are working with, you can use same API and have access to all Tesseract engine features.

Let’s do it step by step.
1. Import your tessdata folder (you can download one or any number of languages from https://code.google.com/p/tesseract-ocr/downloads/list) into your application’s resources folder. For Android tessdata folder should sit under the Assets folder and all files should have Build action set to AndroidAsset. For iOS folder with files marked as BundleResource should be in the Resources folder.
2. Instantiate the API.

//for iOS
ITesseractApi api = new TesseractApi ();
//for Android
ITesseractApi api = new TesseractApi (context);

I personally prefer to use XLabs.IoC.Autofac though. This is the only step that can be done only in platform-specific assemblies. All other steps are platform-agnostic and can be done in a PCL or in a shared  projects.
3. Initialise Tesseract engine.

bool initialised = await api.Init ("eng");

You can use more than one language at a time. Just import all respective files to your tessdata folder and pass list of languages to the Init method (e.g. “eng+rus+esp”).
4. Recognise an image.

bool success = await api.SetImage (source);
if (success)
{
    string textResult = api.Text;
}

You can use System.IO.Stream, byte[] or image path as a source. On Android keep in mind that you need the image should be a valid Bitmap and not a YUV image (here you can find how to convert YUV to a Bitmap).
5. Read results. Apart from getting all the text as a single string you can also get a list of blocks, paragraphs,text lines, words or symbols.

bool success = await api.SetImage (source);
if (success)
{
    List<Result> words = api.Results (PageIteratorLevel.Word);
    List<Result> symbols = api.Results (PageIteratorLevel.Symbol);
    List<Result> blocks = api.Results (PageIteratorLevel.Block);
    List<Result> paragraphs = api.Results (PageIteratorLevel.Paragraph);
    List<Result> lines = api.Results (PageIteratorLevel.Textline);
}

Each Result contains text, it’s coordinates and result confidence so you can decide if you trust it or not (from my experience if confidence is less than 70% this result has nothing to do with the actual text).
6. Magic.
This text is perfect for the Tesseract engine.

Sample 1

The quick brown fox
jumped over the 5
lazy dogs!

Surprisingly this text is not bad as well.

ABCDE FGHI JKLHN OPQR STUVVJXYZ

ABCDE FGHI
JKLHN OPQR
STUVVJXYZ

This text is good, too.

the quick brown fox jumps over the lazy dog- THE QUICK BROlLIN FOX JUMPS OVER THE LAZY DOG.

the quick brown fox
jumps over the lazy dog
THE QUICK BROlLIN FOX
JUMPS OVER THE LAZY DOG.

7. No magic.
This is an open-source OCR library not a magical box converting any image to text. Here are some sample images and outputs to give you an idea of that you can get by using Tesseract engine.

Good font for the OCR mfizufrfomfiv rfie DC’R m m“ {mu mom Good font size for ocn

Good font for the OCR
mfizufrfomfiv rfie DCR
m m“ {mu mom
Good font size for ocn

You can see that same image with the different contrast can give absolutely different result. Hence you should prepare you image before processing it.

1

1

POSTAG E PAI D AU STRALI A

POSTAG E
PAI D
AU STRALI A

You can see that Tesseract is a powerful and open-source OCR engine and now you can go ahead and easily include it in your Xamarin app. In the next post I’ll tell you about TesseractApi settings and give some tips and tricks to improve recognition quality.

Advertisements

41 thoughts on “Tesseract OCR for Xamarin (part 1)

  1. Pingback: Tesseract OCR for Xamarin (part 2) | Artur Shamsutdinov's blog

  2. Mario

    Hi,

    Is there any mobile hardware limitations to use it?
    Like camera ou memory minimum requirement.

    Thanks.

    Like

    Reply
    1. Artur Shamsutdinov Post author

      No, there is no limitations. Tesseract works with images not with camera. Memory requirements are based on the size of the image you are working with but as I know Tesseract is pretty slim and requres a small ammout of memory for it’s own needs. Just keep in mind that image will be extracted in memory and you 1 Mb JPEG can become 10 Mb in memory.

      Like

      Reply
  3. Marius

    Hi Artur,
    I ‘v tried to launch your test project, but on android platform the app crashes on
    var initialised = await _tesseract.Init (“eng”) in MainActivity.

    Do you have same behavior?

    StackTrace:

    Java.Lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at — End of managed exception stack trace —
    at java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
    at Caused by: java.lang.reflect.InvocationTargetException
    at at java.lang.reflect.Method.invoke(Native Method)
    at at java.lang.reflect.Method.invoke(Method.java:372)
    at at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
    at … 1 more
    at Caused by: md52ce486a14f4bcd95899665e9d932190b.JavaProxyThrowable: System.ArgumentNullException: Argument cannot be null.
    ….

    Thank you!!

    Liked by 1 person

    Reply
    1. Artur Shamsutdinov Post author

      Hi Marius,

      You were lucky/unlucky to get the version I pushed couple of hours ago and in which I forgot to fix the DI initialisation. Sorry, my bad. Please check out the last version from master. It should be fine.

      Cheers,
      Artur.

      Like

      Reply
  4. Pingback: Taking the best of two worlds or learning to Grunt | Artur Shamsutdinov's blog

    1. Artur Shamsutdinov Post author

      It’s almost impossible to recognise image received from phone camera without preprocessing (like contrast, b/w, noise removing etc). Image preprocessing is not this lib’s job so you will need to use something like OpenCV.

      Like

      Reply
  5. Haider Rasool

    In my xamarin.forms project “await api.Init (“eng”);” allways returns 0. I am testing on android and i have added language files in Assets folder aswell.

    Like

    Reply
  6. faruk

    hi artur,
    Firstly I’ve to thank you because that’s what I definitly look for it. I use xamarin for just 3 months.
    I tried to do step by step whatever you write but I couldn’t. I get failure.
    “System.InvalidOperationException: Call Init first”.
    Could you share your poject please. Wait your answer asap.
    Thanks again

    Like

    Reply
  7. Adeena Mignogna

    Apologize that these are probably newbie questions, but I’m new to VS and Xamarin and developing mobile apps (I’m old school C/C++ developer). I really committed to figuring out how to get this to work. 🙂 I’ve re-read this post and looked at your code samples over and over and I’m stuck on 1) I don’t understand where you’re instantiating the API (step 2) and 2) Why are the namespaces throughout your sample code not the same? and 3) Do I need to NuGet the actual Tesseract library? Thanks in advance!!

    Like

    Reply
      1. Adeena Mignogna

        Hi Artur, Thanks! Yeah, I figured out #2 and #3 later in the day. 🙂 As for #1… where in the code does that happen? In MainPage.xaml.cs? The example code has a line: “var initialised = await _tesseract.Init(“eng”);” in Task Recognize. Is that the API initialization?

        Later in the day, I ran in to different issues… the code I had built, but then when I tried to debug, I got a Autofac.Core.DependencyResolutionException … (I’m *really* new to all the Ioc stuff.)… and I wound up trying a few different things… that led me to updating my Xamarin.Forms package and others and playing around with build targets… and now it won’t build at all… (lots of CS0117 errors). Later today I’m going to start over again and see where it all went wrong. I’m going to try and match the exact package versions in your example. I’m not sure what else to do.

        I’m really motivated to get this to work… There’s an app I want to make. I’m targeting Droid first.

        Like

      2. Artur Shamsutdinov Post author

        >>The example code has a line: “var initialised = await _tesseract.Init(“eng”);” in Task Recognize. Is that the API initialization?
        Yes!

        >>Later in the day, I ran in to different issues
        Can you share your code somewhere? Looks like you something is missing in your IoC configuration.

        Like

      3. Adeena Mignogna

        So my plan to rebuild using the exact library versions specified in your example packages.config worked. It built and I downloaded a copy of the sample images to my phone. It deploys to the phone, but when I click “load image” and choose one of the samples, it spins forever (or at least 11 min… that’s when I gave up). Any suggestions?

        Like

      1. Adeena Mignogna

        Not at the moment. 🙂 After testing it on the sample files and on photos, I see what you mean about requiring pre-processing on the photos. From a performance standpoint, I get a result fairly quickly, just not a good one. lol I’m familiar with and have used OpenCV, but have never integrated it with Xamarin/VS. So that’s my next task. 😀

        Like

  8. rexonance

    I am relatively new to Xamarin but I will like to use the OCR function for a project I am working on. Please where will this section of the code sit in the iOS/Android codes? Kindly confirm.
    //for iOS
    ITesseractApi api = new TesseractApi ();
    //for Android
    ITesseractApi api = new TesseractApi (context);

    Like

    Reply
  9. lee

    i try to make a program for OCR on xamarin but it’s not works . i think tesseractAPI using XLabs plugin, but that’s maintain any more. do you have any idea? I hope to get your reply.

    Like

    Reply
  10. Richard Massé

    Hi,

    Did you have any sample code for xamarin forms (android and ios) without autofac. Can’t get it to work.

    What did I have to add to mainactivity.cs and appdelegate.cs. And how to call api from shared code.

    Thanks.

    Like

    Reply
  11. Nguyễn Đình Nghĩa

    Hi,
    I have an issue, when run your example https://github.com/halkar/Tesseract.Xamarin/tree/master/Tesseract.Nuget.Test
    Unhandled Exception:

    Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor ‘Void .ctor(Context)’ on type ‘TesseractApi’. —> dalvik.system.PathClassLoader[DexPathList[[zip file “/data/app/com.tesseract.nuget.test-1/base.apk”],nativeLibraryDirectories=[/data/app/com.tesseract.nuget.test-1/lib/arm64, /data/app/com.tesseract.nuget.test-1/base.apk!/lib/arm64-v8a, /vendor/lib64, /system/lib64]]] couldn’t find “liblept.so” (See inner exception for details.)

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s