10 Replies Latest reply: Mar 18, 2014 6:44 AM by Daniel Jankovic RSS

Experiences with InfiniDB

Richard Leeke

I've just been experimenting with InfiniDB and have managed to get it going without too much trouble (following a tip from Joe who has also been having some success with it).

 

I had been using Infobright a bit lately, as per this thread, and following Joe's comments I thought I'd have a look at InfiniDB, too, to see how they compare.  The only other comments I'd seen about InfiniDB on these forums were Michal Klos's experiences The specified item was not found., which made it sound like a non-starter, so I thought I'd just post a few quick notes on what I've found.

 

I'm using version 2.1 of the InfiniDB community edition, which has just been released.  I'm guessing that has just added enough capability to get past the issues reported by Michal.  I'm running the Win32 version (because the 64 bit version isn't out yet) on my laptop.

 

 

All I have done so far is create and load a single table which I also have in Infobright and convert a copy of the Tableau workbook for Infobright to run against the InfiniDB version.

 

 

The table loader for InfiniDB has a pretty clunky interface, but handles data errors much better than JET (which quietly ignores them) or Infobright (which aborts the load on the first error - or even quite often when there arent errors and also lies about where the errors are).  It is also seriously fast - InfiniDB loaded my 102 million rows in 16 minutes at the first attempt, and that was with both the input file and the database on my laptop's hard drive.  That compares with about 25 minutes for Infobright and quite a long time for the Data Engine (I haven't timed it, but based on past tests it would be a few hours).

 

The resulting database was significantly larger than with Infobright (18 GB with InfiniDB, 1.1 GB with Infobright).  This is because InfiniDB community edition doesn't offer compression, unlike Infobright community edition.

 

 

I repointed a workbook from Infobright to InfiniDB by simply changing the MySQL port number to the InfiniDB one.  All 20 or so worksheets displayed correctly.  Initially there was one error reported on a query to do with displaying quick filters, but this turned out to be because I still had the datasource customisations for MySQL connections needed for Infobright in place.  Removing CAP_QUERY_HAVING_REQUIRES_GROUP_BY fixed that.

 

 

I haven't done any timed comparisons yet, but my impression is that sheets take quite a lot longer to display with InfiniDB than Infobright.  I suspect the lack of compression may just mean there is a lot more data to read from disk - so I expect it might be a completely different story with the enterprise edition.

 

I'm also fairly sure that both are a lot slower than the data engine for quite a lot of my analysis - particularly where I have calculated dimensions.  So there really are pros and cons for all the options at the moment.  It will be very interesting to see how the balance shifts with the upcoming Tableau release which various of the Tableau folk have hinted is going to improve some of the Data Engine capabilities in this space.

  • 1. Re: Experiences with InfiniDB
    Robert Morton

    This is great work Richard, thanks for sharing your analysis!

    -Robert

  • 2. Re: Experiences with InfiniDB
    Naveen Arumugam

    Richard

     

    Good Analysis to the point. Btw, did you try latest infinidb/infobright community editions? What's your take? Is there restrictions in community editions? can they handle multi-terabyte dwh?

  • 3. Re: Experiences with InfiniDB
    Richard Leeke

    No, I haven't done anything with either Infobright or InfiniDB since Tableau 6.1 came out, so can't comment on any of that, sorry.

  • 4. Re: Experiences with InfiniDB
    Naveen Arumugam

    Thanks Richard.

  • 5. Re: Experiences with InfiniDB
    Jim Tommaney

    Hi Naveen, Richard,

     

    Just as an FYI, InfiniDB 4 is now available - the community edition has moved to version 4 as well, and we have significantly altered our OSS offering:  We are now GPL v2 with no restrictions on syntax, scale or performance.  And yes, compression is included as well.

     

    InfiniDB also added another platform beyond local disk, EBS, and GlusterFS, specifically HDFS.  This is InfiniDB installed on an existing CDH or HDP Hadoop distribution.

  • 6. Re: Re: Experiences with InfiniDB
    Richard Leeke

    Thanks for the update.

     

    > We are not GPL v2 with no restrictions on

     

    "Not" or "now"?

  • 7. Re: Experiences with InfiniDB
    Jim Tommaney

    Hi Richard,

    Corrected the "not" to "now" .  Ooops.

    Correct statement is: We are now GPL v2 with no restrictions on syntax, scale or performance.

    Cheers,

    Jim

  • 8. Re: Experiences with InfiniDB
    Jim Tommaney

    I think I just fixed…  That is about as unclear as possible.   

     

    Should be “now”.

  • 9. Re: Experiences with InfiniDB
    Joshua Magaña

    I have been using the Calpont InfiniDB DB for about a year now.  Originally as a POC on 3.5 and then upgraded to 3.6 and we now are using 4.0.  Calpont used to have a Community version and an Enterprise version, but now they are full open source with a subscription model for support.  I really like the product and the support team has been very responsive.  As with any software product, it is not a panacea.  But for the price you get pretty close.  I would not recommend going it alone.  There is not enough external sources to navigate through the technical weeds that you can get yourself into, but they will work with you 100%.  I also compare them to Infobright, and I found the bulk loader "cpimport" to be much faster than Inforbrights and column mapping better since my query times were more responsive.  Depending on how you structure tables you can have loads with millions of rows getting loaded in seconds.  We have some tables that require lots of deletes, which I don't think any columnar DB does well, so we truncate and load which can take a few minutes but smaller loads under 10 million can run in a few seconds.  So far the product has been satisfactory.

  • 10. Re: Experiences with InfiniDB
    Daniel Jankovic

    Hi all,

     

    I'm starting with implementation of InfiniDB as data source and I noticed that InfiniDB doesn't support all features/functionalities Tableau requires/uses. Namely, when I tried using MySql connector, I'm often getting errors about "null safe equals" operator (which isn't supported by InfiniDB). On the other hand, when I tried my luck with ODBC connector, several different errors started appearing (mostly related to casting different values to integers).

     

    I'm interested if anyone managed to find a good fix for this or people just accept this situation as-is and go without using problematic functionalities.

     

    I'm really in need of using InfiniDB (or at least I think so) because I'm dealing with large amounts of data which made MySql too slow (and exports are not an option since I also need to have up-to-date data).

     

    Thanks in advance for your suggestions.

     

    Cheers!