michaelsilver.us Software, programming and whatever.

January 16, 2018

Huge Speed Increase with Sqlite Write Ahead Logging

Filed under: C#,Database,SQLite — Michael Silver @ 12:39 am

Recently, I started work on a small, simple work queue written in C#, called NWorkQueue. Durability/persistence is one of the core requirements for the project.  Where better to find durability with one of the most widely distributed embedded databases on the planet, SQLite?  It’s rock solid journaling is famous for keeping corrupted database files to a minimum.

While performing some insert tests of 10,000 records, I noticed a HUGE increase in performance when all the inserts were wrapped in a single transaction.  Within one transaction it took about 300+ ms.  Not earth shattering, but not bad. With each insert in it’s own transaction, performance dropped off a cliff to 63 seconds. Can that be right?!

Sadly, many of my inserts will probably not be grouped together in a single transaction.  I was now worried I was giving up performance for durability by using SQLite.

I began a search for SQLite performance tuning and stumbled upon numerous PRAGMA statements for SQLite.  I found three that seemed to have an impact.  The first two,


Temp_Store is a no brainer, as there is little disadvantage to moving temp storage into memory.  SYNCHRONOUS is a little more tricky.  By reducing it from FULL, I dropped the 63 seconds to 53 seconds and it appears I gave up very little in resiliency, although it is slightly less safe.  Paraphrasing from the SQLite documentation, if you get a corrupt database file while in FULL mode, your disk is also probably non-operable, with the take-away from that being, that it’s hard to corrupt a SQLite database in FULL mode.

So, that’s a small victory, but a very small one.  I’m down to 53 seconds, and that’s still terrible.  On to more tuning…

There are some interesting options for the JOURNAL_MODE PRAGMA, but unfortunately they greatly reduce the resiliency, except for one:


In the documentation for SYNCHRONOUS, it mentions that FULL mode is a common setting when in WAL mode. WAL mode?  What’s this new feature?

As of 2010, SQLite added an alternative method for tracking transactions instead of journaling: Write Ahead Logging.  It’s turned off by default, since SQLite puts high value on backwards compatibility.  Apparently, this is a much more advanced method and works especially well with frequent writes, the same scenario I’m expecting.

So switching on WAL caused my tests to run so fast, I thought they were broken.  It dropped the single transaction from 300ms to between 75 and 100ms.  What about the 10,000 transactions?  It dropped from 53 seconds to 622ms.  Wow!  That’s an 85x improvement in speed.

If you need not only resiliency, but speed as well, I strongly encourage you to look at using WAL for journaling.  There are some caveats and it may not be perfect for every situation, but the docs do a great job of explaining those.

As a bonus, with the introduction of Windows 10, Microsoft started including the SQLite DLL with windows.  Keep in mind, if you are using .NET Core, you will need to make changes to accommodate the  lack of a SQLite DLL in non-windows systems, but it’s wonderful to see Microsoft embrace such a great product as SQLite.

October 22, 2015

Rome Tennis Ladder

Filed under: Uncategorized — Michael Silver @ 1:27 pm

To enhance competitive playing opportunities, we are starting a free tennis ladder. Here are the basic rules:

  • Point based system:
    • If winning team is ranked below or tied, they get losing teams points plus 15
    • If winning team is ranked above, they receive 15 points
    • Losing team receives point for each game won in the first two sets.
    • Forfeiting team gets no points, other team receives 13.
    • Either team can cancel match up to 2 hours beforehand or based on CVTA weather rules, otherwise the opposing team can take a forfeit.
    • All teams start with zero points.
  • Standings will be published on website.
  • Initially, scores will be sent to me, but eventually entered in a website.
    • It’s critical that scores be sent immediately, as this could affect standings and the amount of points won.
  • Will only be open men’s and ladies division, but may split up based on NTRP ratings depending on participation.
  • Flex scheduling
  • A team consists of you and your partner.
  • You can play as many matches as you want, but can’t schedule more than one at a time since the ladder ranking may change between matches.
  • Third set tiebreaker in lieu of full set, unless both parties agree to a full set.

Interested in playing?  Send me an email with you and your partners name, NTRP ratings, email and phone number.

If you want to challenge another team:

  • You must be within 5 ranking positions of the opposing team at the time of the challenge.  You may have multiple challenges scheduled at once.
  • You can’t have played them in the last 2 weeks.  The other team may accept a challenge earlier, but are under no obligation to accept.
  • Challenged team must provide new can of balls (winner keeps balls)
  • If declined, you can try challenging same opponents again in a week or pick another team.

We’ve been challenged:

  • Must always accept first challenge received (unless rejecting all challenges).
  • You must provide the can of balls (winner keeps balls)
  • Inform your opponent of your home court and schedule a time.
  • If you are no longer playing in the ladder, let challengers and me know.  We will flag you as inactive.
  • Can decline matches for any reason, but you must decline for all challenges (i.e. injured or out of town).  You can’t pick and choose who you play.  Remember, the more you play, the more points you get, so not playing will drop you in the ladder.
  • You can also decline if you have played the challenger in the last 2 weeks or already have another match scheduled.

Remember, USTA and CVTA leagues take precedence on all courts. There are no fees to participate and you can play on any courts in Rome or neighboring communities.  Challengers should let the defending players pick location, unless there are fees.  Both parties must agree on courts with fees.  Since this is not a league and no league fees are collected, you must pay to play downtown at the RTC.

One last note: All of these rules are subject to change to ensure congenial and competitive matches.  

Frequently Asked Questions:

Q: Can we challenge teams below us?

A: Yes, but it will be counted as any other match, i.e., you can only win a maximum number of 15 points and will drop in the rankings if you lose.  You can only challenge 5 positions below you.

Q: Can I be on multiple teams?

A: Yes, but for practicality, we are limiting it to two active teams at a time.

Q: What if I am busy with league matches, and can’t play a ladder match this week.

A: The ladder has no requirement that you play every week.  If you are too busy with league matches or other activities, just decline all challenges until you are available.

Q: What if the ladder is unbalanced with high rated players playing against low rated ones?

A: Once the ladder is in full swing, you’ll be playing against people of similar ability, even though higher or lower rated players are participating.  Eventually, with enough participation, we may split the ladder by NTRP rating.

Q: I’m a lady.  Can I play in the men’s ladder?

A: Yes, but men cannot play in the women’s ladder.

July 24, 2010

Facebook Etiquette 101

Filed under: Internet,Proposals — Michael Silver @ 2:27 am

Miss Manners in the White House

Facebook rocks.  Well, maybe it doesn’t rock, but it does fill a niche on the Internet that no other site or service fills as well. The type of interaction that Facebook allows is new to all of us, after all Facebook is a baby.  As with kids when they learn to eat at a fancy restaurant or have conversations at a dinner parties, they make faux paus.  They are unaware of proper etiquette.  Such interactions are new to them, just as Facebook is to us.

In many cases, etiquette exists for good reason: to avoid offending, inconveniencing or putting others in a difficult or uncomfortable position.

Enter Facebook.

We are all kids learning how to interact via this new medium and it desperately needs a common etiquette to help remove the cruft that often ends up in our friend feed.  Michael to the rescue.  Here are some guidelines to help improve our communication via Facebook via good etiquette:

Faux Pas #1: Bait Post

This is unbelievable common.  Ever browsing your friend feed and come across a post like this? “OMG, that was awesome!!!”  That’s all.  No context.  No explanation.  This is a Bait Post.  It begs the reader to respond “Hey, what was so awesome?”.  Then the original poster has to explain what should have been in the original post.

Sample Bait Post

If the poster doesn’t want their friends to know what they are talking about, why are they even posting to Facebook?  Send a text message or an email.  If it’s private, keep it private.  Don’t tease.

If it’s not private, why do this to your friends? Is this how you carry on conversations in real life? Begging for attention? Facebook already brings out the narcissist in us, don’t encourage it.

In all fairness, I have been guilty of this in the past and a recent Bait Post of mine prompted me write this entry to help remind me:  Friends don’t Bait Post friends.

Faux Pas #2: Bait and Ignore Post

What could be worse than the dreaded Bait Post?  Well, the Bait and Ignore Post.  Not only does the poster bait you, they ignore your pleas to know what’s going on.

Faux Pas #3: Bait and Non-Of-Your-Business Post

Yes, it gets worse.  This is less common, thank god, but still happens.  The poster baits their friends and then after the obligatory “What’s going on?”, they post to the conversation: “I’ll email you,” essentially telling the rest of their friends, that it’s non of their business.   Hopefully, if you are one of the lucky “What’s going on?” posts, you will get the explanation email.  I never do, because I never fall for the bait post in the first place.  You shouldn’t either.

Hopefully, if we all stop responding to Bait Posts, they will slowly die.  But doubtful, so feel free to send offending friends to this blog entry

Faux Pas #4: App Spam

Everyone is quite familiar with this one.  Facebook applications posting into your friend feed:  “Joe Blow needs one more diamonds is his mining adventure.”, “Milard Jones just joined the Italian Mafia gang”.  This takes annoying to new heights.  The information is useless, no one cares, everyone hides it, so why do it?  If the application doesn’t give you the option to not automatically post to your feed, you shouldn’t play it.  It’s spam, plain and simple.  Kudo’s to Facebook for allowing us to hide such posts.

Sometimes there’s a fine line.  Some applications, such as Foursquare, actually do post interesting information.  Foursquare posts about your current whereabouts to your Facebook feed.  At least it has meaning.  I’ll find it interesting if you are cable hang gliding in Trevallyn State Recreation Area.  Knowing that you scored 100,000 points in Bejeweled, not so much.

Faux Pas #5: Anonymous Friends

I consider Facebook a place to communicate with my friends.  If I don’t know you, I can’t really consider you a friend. Sorry, it’s the nature of relationships.  With that understanding, I won’t approve you as my friend.  You should presume all Facebookers will reject unknown friend requests and here is where etiquette comes into play:

If you ask to be friends with someone and you barely or don’t know them, add a note to the friend request, either jogging their memory or explaining why you should become Facebook pals.  It’s courteous and respectful and if you don’t do it, expect to be rejected.

That’s it.  That’s all I have.  See how painless it was?  Now you know how simple proper etiquette on Facebook can be.  With these faux pas clearly spelled out you will recognize them everyday and realize just how inconvenient they are. Just because it’s a new medium doesn’t mean we can’t apply common sense to it.

Did I miss one?  Disagree?  Let me know.

July 23, 2010

Android App: ADW.Launcher

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 6:57 pm


Courtesy of AndroLib

ADW.Launcher replaces the existing Android drawer.  The drawer is the icon at the bottom of the screen you click to see all your apps.  ADW.Launcher is a great piece of software.  There are tons of options and rarely a bug.  It’s wildly popular in the Android community, especially among developers.

It allows you to have up to 9 home pages, 4 shortcuts on the drawer (a ‘secret’ dock-bar can hold even more shortcuts).  With version 1.0, there are even some great looking themes, some of which cost a $1 (ADW.Launcher is free).  Well worth it.  Highly recommended.

Some manufacturers already customize the drawer heavily so this application may not make sense in those scenarios.

Download ADW.Launcher


Android App: Facebook (official app)

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 2:23 pm


Courtesy of AndroLib

Facebook’s Android app does exactly what you expect.  You can browser your Facebook friend feed and also post your status as well as check email.  The app’s biggest asset isn’t even the actual app, it’s the Android integration that comes along with it.  For example, after taking a picture, you can ‘share’ it right to Facebook.  It also provides synchronization between your Android contacts and those in Facebook.  For example, when calling my mom, whom I don’t have a picture for, I see your facebook photo pop up instead.  It’s also helped me find phone numbers (only when the Facebook user has shared their phone number).

Some Android manufacturers offer their own Facebook integration, so this app may not benefit you.

Another ‘must have’ app.

Download Facebook


Android App: ColorNote

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 2:07 pm


Courtesy of http://socialnmobile.blogspot.com

ColorNote stands above over note taking applications due to it’s checklist functionality and ease of use.  For example, in doing some heavy house renovations, I created several notes and placed them on a separate home screen.  This gave me easy access to the lists from the home page.  At the beginning of each day I would view my tasks to  be done and at the end of the day I would update a list of supplies that I would need to buy.  It kept my chaos remarkably organized.  Highly recommended.

Download ColorNote

Don’t worry if pink isn’t your color.  As the name suggests, you can customize the color of each note.  The default is yellow.


Android App: Dialer One

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 2:06 pm

Dialer One

courtesy of AndroLib

Dialer One is another app that should be included with Android.  Dialer One replaces the simple phone dialer that ships with Android with one that can perform T9 searches on your phonebook.  Even many basic non-smartphones provide this functionality, so I’m not sure why Google omitted it.  Many phone manufacturers (HTC, Samsung and I think Sony) include their own dialer,  so in those cases, this app would be redundant.

Download Dialer One

Dialer One is a ‘must have’ application, but has problems.  Contact pictures sometimes do not display, due to restricted access to Facebook profile pictures.  It’s also slow on my HTC Magic running Android 2.1.  On faster phones and Android 2.2, I would expect it to run without sluggishness.


Android App: Quick Settings

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 2:00 pm

Quick Settings

courtesy of AndroLib

Quick Settings put all the mostly used settings in one place for easy access instead of having to dig through the settings menu.  I use it almost everyday, especially to adjust the screen brightness or turn on WiFi.  It even has a flashlight app, which I have yet to use.  It is highly customizable, allowing you to hide or replace settings you might or might not use.  Place it on the desktop or dock bar or any place you can access it easily.  A ‘must have’ app.

The developer doesn’t have a dedicated home page, but does have an active twitter feed.


Android App: Barcode Scanner

Filed under: Android,Recommended,Reviews,Software — Michael Silver @ 1:59 pm

Barcode Scanner

Barcode Scanner should ship with Android.  It allows your device to scan most barcodes.  Many applications actually rely on this software, for example, Key Ring.  Barcodes?  Who cares?  I thought that at first too.  What makes Barcode Scanner so useful is the proliferation of QR Codes, which are fancy square barcodes that allow encoding of many different type of data.  For example, I have included a QR Codes in all my reviews.  The QR Code contains an address to the Android Market allowing you to download the app within seconds.  Just scan the code in (right from your monitor) and you will be prompted to download the app.  No searching, no typing.

Download Barcode Scanner

QR Codes also allow encoding of contact information, latitude and longitude, plain text, etc.  My next set of business cards will have a QR Code containing my contact information allowing people to add me to their address books with the scan of a code.  I also plan on using them in my geocaching activities.

Want to generate your own QR Code?  Have at it.



Killer Android Apps You Must Have

Filed under: Recommended,Reviews,Software — Michael Silver @ 12:58 am

I’ve been running Android on my Roger’s Magic for several months.  As time passes, I have collected a selection of app that I use often and highly recommend to others.  I’ve included the QR code (keep reading if that means nothing to you), the AndroLib web site and sometimes a review.  Without further delay:

Recommended Android Apps

Older Posts »

Powered by WordPress