Tesseract OCR for Xamarin (part 1)

59 Replies

If you’re thinking about getting image recognition into a Xamarin app check out this open source Tesseract OCR port I’ve put together for Xamarin. Few weeks ago this question was asked on StackOverflow and I was surprised to find that there is no free OCR for Xamarin (neither for iOS nor for Android). What developer does when he founds that there is nice and open-source library for other platforms but not for the platform he is working with? Of course he migrates the library. So I created wrappers for Tesseract OCR for Xamarin.iOS and Xamarin.Android and also for Xamarin.Forms. In this post I will give some details about the library, will show how to use it and will tell about some issues and workarounds.

Some history

The Tesseract engine was originally developed at HP in 1985 and was then released as open source in 2005. It has been sponsored by Google since 2006. This wrapper is base on two open-source builds – tess-two by Robert Theis and Tesseract OCR iOS by G8Production.

How to use

First of all you can use either the NuGet package or the source code. This is not the “final stable” version, just one of the first ones. But I’m working on adding features and fixing bugs so if something is broken try getting the last available version. If it’s still broken don’t hesitate to post an issue on the GitHub.

TL;DR

You can find few samples here.

Long story

One of the main ideas I kept in mind is that whichever platform (Android/iOS/Forms, sorry no Windows Phone this time) you are working with, you can use same API and have access to all Tesseract engine features.

Let’s do it step by step.
1. Import your tessdata folder (you can download one or any number of languages from https://code.google.com/p/tesseract-ocr/downloads/list) into your application’s resources folder. For Android tessdata folder should sit under the Assets folder and all files should have Build action set to AndroidAsset. For iOS folder with files marked as BundleResource should be in the Resources folder.
2. Instantiate the API.

//for iOS
ITesseractApi api = new TesseractApi ();
//for Android
ITesseractApi api = new TesseractApi (context);

I personally prefer to use XLabs.IoC.Autofac though. This is the only step that can be done only in platform-specific assemblies. All other steps are platform-agnostic and can be done in a PCL or in a shared  projects.
3. Initialise Tesseract engine.

bool initialised = await api.Init ("eng");

You can use more than one language at a time. Just import all respective files to your tessdata folder and pass list of languages to the Init method (e.g. “eng+rus+esp”).
4. Recognise an image.

bool success = await api.SetImage (source);
if (success)
{
    string textResult = api.Text;
}

You can use System.IO.Stream, byte[] or image path as a source. On Android keep in mind that you need the image should be a valid Bitmap and not a YUV image (here you can find how to convert YUV to a Bitmap).
5. Read results. Apart from getting all the text as a single string you can also get a list of blocks, paragraphs,text lines, words or symbols.

bool success = await api.SetImage (source);
if (success)
{
    List<Result> words = api.Results (PageIteratorLevel.Word);
    List<Result> symbols = api.Results (PageIteratorLevel.Symbol);
    List<Result> blocks = api.Results (PageIteratorLevel.Block);
    List<Result> paragraphs = api.Results (PageIteratorLevel.Paragraph);
    List<Result> lines = api.Results (PageIteratorLevel.Textline);
}

Each Result contains text, it’s coordinates and result confidence so you can decide if you trust it or not (from my experience if confidence is less than 70% this result has nothing to do with the actual text).
6. Magic.
This text is perfect for the Tesseract engine.

Sample 1

The quick brown fox
jumped over the 5
lazy dogs!

Surprisingly this text is not bad as well.

ABCDE FGHI JKLHN OPQR STUVVJXYZ

ABCDE FGHI
JKLHN OPQR
STUVVJXYZ

This text is good, too.

the quick brown fox jumps over the lazy dog- THE QUICK BROlLIN FOX JUMPS OVER THE LAZY DOG.

the quick brown fox
jumps over the lazy dog–
THE QUICK BROlLIN FOX
JUMPS OVER THE LAZY DOG.

7. No magic.
This is an open-source OCR library not a magical box converting any image to text. Here are some sample images and outputs to give you an idea of that you can get by using Tesseract engine.

Good font for the OCR mfizufrfomfiv rfie DC’R m m“ {mu mom Good font size for ocn

Good font for the OCR
mfizufrfomfiv rfie DC’R
m m“ {mu mom
Good font size for ocn

You can see that same image with the different contrast can give absolutely different result. Hence you should prepare you image before processing it.

1

1

POSTAG E PAI D AU STRALI A

POSTAG E
PAI D
AU STRALI A

You can see that Tesseract is a powerful and open-source OCR engine and now you can go ahead and easily include it in your Xamarin app. In the next post I’ll tell you about TesseractApi settings and give some tips and tricks to improve recognition quality.

59 thoughts on “Tesseract OCR for Xamarin (part 1)”

  2. Pingback: Tesseract OCR for Xamarin (part 2) | Artur Shamsutdinov's blog

  3. Mario

    Hi,

    Is there any mobile hardware limitations to use it?
    Like camera ou memory minimum requirement.

    Thanks.

    Like

    Reply ↓
    1. Artur Shamsutdinov Post author

      No, there is no limitations. Tesseract works with images not with camera. Memory requirements are based on the size of the image you are working with but as I know Tesseract is pretty slim and requres a small ammout of memory for it’s own needs. Just keep in mind that image will be extracted in memory and you 1 Mb JPEG can become 10 Mb in memory.

      Like

      Reply ↓
  5. Marius

    Hi Artur,
    I ‘v tried to launch your test project, but on android platform the app crashes on
    var initialised = await _tesseract.Init (“eng”) in MainActivity.

    Do you have same behavior?

    StackTrace:

    Java.Lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at — End of managed exception stack trace —
    at java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
    at Caused by: java.lang.reflect.InvocationTargetException
    at at java.lang.reflect.Method.invoke(Native Method)
    at at java.lang.reflect.Method.invoke(Method.java:372)
    at at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
    at … 1 more
    at Caused by: md52ce486a14f4bcd95899665e9d932190b.JavaProxyThrowable: System.ArgumentNullException: Argument cannot be null.
    ….

    Thank you!!

    Liked by 1 person

    Reply ↓
    1. Artur Shamsutdinov Post author

      Hi Marius,

      You were lucky/unlucky to get the version I pushed couple of hours ago and in which I forgot to fix the DI initialisation. Sorry, my bad. Please check out the last version from master. It should be fine.

      Cheers,
      Artur.

      Like

      Reply ↓

  6. Pingback: Taking the best of two worlds or learning to Grunt | Artur Shamsutdinov's blog

    1. Artur Shamsutdinov Post author

      It’s almost impossible to recognise image received from phone camera without preprocessing (like contrast, b/w, noise removing etc). Image preprocessing is not this lib’s job so you will need to use something like OpenCV.

      Like

      Reply ↓
  8. Haider Rasool

    In my xamarin.forms project “await api.Init (“eng”);” allways returns 0. I am testing on android and i have added language files in Assets folder aswell.

    Like

    Reply ↓
  9. faruk

    hi artur,
    Firstly I’ve to thank you because that’s what I definitly look for it. I use xamarin for just 3 months.
    I tried to do step by step whatever you write but I couldn’t. I get failure.
    “System.InvalidOperationException: Call Init first”.
    Could you share your poject please. Wait your answer asap.
    Thanks again

    Like

    Reply ↓
  10. Adeena Mignogna

    Apologize that these are probably newbie questions, but I’m new to VS and Xamarin and developing mobile apps (I’m old school C/C++ developer). I really committed to figuring out how to get this to work. 🙂 I’ve re-read this post and looked at your code samples over and over and I’m stuck on 1) I don’t understand where you’re instantiating the API (step 2) and 2) Why are the namespaces throughout your sample code not the same? and 3) Do I need to NuGet the actual Tesseract library? Thanks in advance!!

    Like

    Reply ↓
      1. Adeena Mignogna

        Hi Artur, Thanks! Yeah, I figured out #2 and #3 later in the day. 🙂 As for #1… where in the code does that happen? In MainPage.xaml.cs? The example code has a line: “var initialised = await _tesseract.Init(“eng”);” in Task Recognize. Is that the API initialization?

        Later in the day, I ran in to different issues… the code I had built, but then when I tried to debug, I got a Autofac.Core.DependencyResolutionException … (I’m *really* new to all the Ioc stuff.)… and I wound up trying a few different things… that led me to updating my Xamarin.Forms package and others and playing around with build targets… and now it won’t build at all… (lots of CS0117 errors). Later today I’m going to start over again and see where it all went wrong. I’m going to try and match the exact package versions in your example. I’m not sure what else to do.

        I’m really motivated to get this to work… There’s an app I want to make. I’m targeting Droid first.

        Like

      2. Artur Shamsutdinov Post author

        >>The example code has a line: “var initialised = await _tesseract.Init(“eng”);” in Task Recognize. Is that the API initialization?
        Yes!

        >>Later in the day, I ran in to different issues
        Can you share your code somewhere? Looks like you something is missing in your IoC configuration.

        Like

      3. Adeena Mignogna

        So my plan to rebuild using the exact library versions specified in your example packages.config worked. It built and I downloaded a copy of the sample images to my phone. It deploys to the phone, but when I click “load image” and choose one of the samples, it spins forever (or at least 11 min… that’s when I gave up). Any suggestions?

        Like

      3. Adeena Mignogna

        Not at the moment. 🙂 After testing it on the sample files and on photos, I see what you mean about requiring pre-processing on the photos. From a performance standpoint, I get a result fairly quickly, just not a good one. lol I’m familiar with and have used OpenCV, but have never integrated it with Xamarin/VS. So that’s my next task. 😀

        Like

  11. rexonance

    I am relatively new to Xamarin but I will like to use the OCR function for a project I am working on. Please where will this section of the code sit in the iOS/Android codes? Kindly confirm.
    //for iOS
    ITesseractApi api = new TesseractApi ();
    //for Android
    ITesseractApi api = new TesseractApi (context);

    Like

    Reply ↓
  14. lee

    i try to make a program for OCR on xamarin but it’s not works . i think tesseractAPI using XLabs plugin, but that’s maintain any more. do you have any idea? I hope to get your reply.

    Like

    Reply ↓
  15. Richard Massé

    Hi,

    Did you have any sample code for xamarin forms (android and ios) without autofac. Can’t get it to work.

    What did I have to add to mainactivity.cs and appdelegate.cs. And how to call api from shared code.

    Thanks.

    Like

    Reply ↓
  16. Nguyễn Đình Nghĩa

    Hi,
    I have an issue, when run your example https://github.com/halkar/Tesseract.Xamarin/tree/master/Tesseract.Nuget.Test
    Unhandled Exception:

    Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor ‘Void .ctor(Context)’ on type ‘TesseractApi’. —> dalvik.system.PathClassLoader[DexPathList[[zip file “/data/app/com.tesseract.nuget.test-1/base.apk”],nativeLibraryDirectories=[/data/app/com.tesseract.nuget.test-1/lib/arm64, /data/app/com.tesseract.nuget.test-1/base.apk!/lib/arm64-v8a, /vendor/lib64, /system/lib64]]] couldn’t find “liblept.so” (See inner exception for details.)

    Like

    Reply ↓
  18. abhijeet

    Hi

    Thanks for putting Tesseract together for Xamarin.

    I am using Prism.Forms & able to add this library to the project.

    able to get it initialized for eng using test data you have provided in your test project, I am running iOS.

    but this line tesseractApi.SetImage(file.GetStream()) always returns with false for me.

    What do you think that I might be missing?

    Like

    Reply ↓
  19. Tomas

    Thank you for this wrapper! Works perfectly. However there are still some issues for languages with diacritics on Android devices. Will be there an update for Android and Tess 4?

    Like

    Reply ↓
    1. Artur Shamsutdinov Post author

      Hi Tomas. Tess 4 is not yet in production and `tess-two`, Android project which I use doesn’t support it. This project is just a wrapper for existing Android and iOS libs.

      Like

      Reply ↓
  20. Albrecht Schänzel

    Dear Artur,
    I’m currently visiting an iOS course. This contains a “project of approval”
    I’m German so my English is not the best, I’m really sorry about that.
    My idea for that project is:

    I can’t use a real iPhone or IPad. Only the simulator is allowed
    I would scan a document for the scan
    I’m really confused what wrapper you to linked to download

    My development environmemt is:
    iMac, Visual Studio 2017 Community
    Target of the project is: iOS native, with Xamarin

    I would like to scan a picture (because the simulator barrier). After that I would like to call a text to speach webservice. After that I would like to store both data in a database.

    I think, that’s enough for a week for a iOS beginner.

    Please tell me what I have to download and to import in my Solution.

    Thank you very much in advance for your help

    Best regards,
    Albrecht

    Like

    Reply ↓
  22. Chris Lamont

    Hi Artur,

    I am having trouble with `await _tesseractApi.Init(“eng”);`

    I have tried everything but this silently kills the app.

    In the Application output there is only one line that may reference the issue: `actual_tessdata_num_entries_ <= TESSDATA_NUM_ENTRIES:Error:Assert failed:in file tessdatamanager.cpp, line 53`

    Please let me know if this makes any sense to you.

    Cheers, Chris

    Like

    Reply ↓
    1. Chris Lamont

      I think I have a lead on the issue. In your blog you don’t mention that you need to download multiple files. I have just downloaded just the `eng.traineddata` file, but I see from another example that there are multiple “eng” files you need to download.

      Like

      Reply ↓
    1. Victor Arce (@victorarce)

      Yes, have you seen anyone implementing this? Kinda of model number scanner, where there’s a particular “scannable” area and it automatically keeps scanning the camera image area until it reads a model number text.

      Like

      Reply ↓
  25. Sharun

    When I type the lines:
    ITesseractApi api = new TesseractApi ();
    ITesseractApi api = new TesseractApi (context);
    I get an error saying ITesseractApi and TesseractApi don’t exist. I downloaded the nuget plugin Xamarin.Tesseract, and did step 1 of the setup. What am I missing that causes step 2 not to work? And should I put the above lines into Project.iOS/Main.cs and Project.Android/MainActivity.cs, or put both into MainPage.xaml.cs?

    Like

    Reply ↓
    1. Sharun

      I was just being dumb, I got the iOS one to work by looking at rexonance’s comment above, but the android one still isn’t working.

      It says that the new TesseractApi requires (Context,AssetsDeployment), not (context), which is different from what the original post says. I put it in the MainActivity.cs, but I don’t know what to do for the two fields above.

      Like

      Reply ↓
      1. Sharun

        Where do I do step 3? I did step 2 in AppDelegate.cs(iOS) and MainActivity.cs(Android), but I don’t know where to initialize the api. Can I only initialize it once? I don’t want to put step 3 in both the iOS and Android code.

        Like

Leave a comment