110GB+ worth of photos/videos on a microSD memory card #2

In my previous post, I talked about wanting to write a script to see how far I can squeeze 110GB worth of media (photos and videos).

I wrote the script the quick-and-dirty way, in PHP. When people think of PHP, they think of web pages, but PHP can equally be used for writing your everyday scripts, just like Perl or Python.

The script resizes all photos to a maximum width/height of 1024. I found this provides maximum size reduction for the photos without sacrificing the viewing quality in full photo mode on my tablet (screen resolution: 1280x800). Obviously it will affect the quality when zooming in, but I rarely do that when flipping through the photos. Since I haven't got a tablet with retina display yet, I am not sure how that will fare, but for this exercise, I am sacrificing quality for size.

For videos, I use handbrake-cli to transcode all of them to MP4 with a maximum width of 640. Again, this reduces the size drastically, improves compatibility (since all Android tablets/smartphones I have come across so far can play MP4 with no problems) and looks OK quality-wise on my tablet.

The script takes a source directory and replicates the folder structure in a target directory. It transcodes all files with certain extensions (configurable in config.php) and places them in the corresponding target subdirectories. If the file has already been transcoded, it will not be transcoded again, so it will be relatively quick to update the target directory after adding more files to the source directory.

The script took 1.5 days to transcode my 110GB collection (bear in mind this is a older, slower AMD Turion II laptop). The result was only 29GB and fits easily into a 32GB microSD card.

I popped the microSD card into my tablet and viewed the files using QuickPic. The result was awesome.Check it out in the video below:

Note that this video was produced with the microSD card inserted into an al-cheapo single-core tablet bought for the kids during a sale. I was pleasantly surprised by how quick and smooth it was. Needless to say, on my own 10" quad-core tablet, the result was even better.

QuickPic is a better app than Gallery for this purpose because:

1. It allows you to set "Include Folders" to include only the media files from the microSD card

2. It lets you set the 'Explorer View" to browse through the folders/subfolders in a hierarchical manner.

3. It has "Stack", "Grid" and "List" views for different visual summaries of the folder content.

4. It is quick and has tons of other options.

The result of the transcoding operation exceeded all my expectations. I have a snapshot of 110GB worth of media on a tiny memory card, ready to be popped into any tablet or PC to view. At only 29GB, there's lots of room for growth, especially on a 64GB microSD card (which no doubt all tablets will support in the future, even cheapo ones).

If you think you might find this useful, you can download the script below. It includes the PHP and Handbrake-CLI binaries so it should be ready to run under Windows. It shouldn't be too difficult to adapt the script to work under other platforms as well.

After download, unzip into another directory and edit config.php. You only need to change $srcdir and $tgtdir. Then bring up a command prompt and run transcode.

Once you verify that everything works correctly, you may want to schedule it to run periodically via the Task Scheduler.

Download: Batch Media Transcoder V1.0 (ZIP 25.2MB)

Download: Batch Media Transcoder V1.0 (7-ZIP 17.1MB)

Read More

110GB+ worth of photos/videos on a microSD memory card #1

We have a family NAS where we keep all kinds of stuff. Well, actually it's an old laptop that functions as a server, with 3 x 3TB external USB HDDs attached. I run all the standard mirroring and backup on the laptop, and one of the HDD is shared over the network for everyone to dump their stuff into.

Like most family these days, we take photos and videos on a number of different devices. The smart phones, digital camera, DSLR etc. Even the kids have their own cheap digital compacts. And the devices have evolved over the years. The photos have mostly stayed JPEG, with the resolution increasing steadily. The videos, however, are a different story. Starting from the V8 (which we converted to XVID AVIs) to a Sony (which produced MPEG-1 files) to a Casio (which gave us H264/MOV files) to the current-gen Sony (MTS format), there's an incredible number of video formats we have to deal with. On top of that, we do scans of old photos and slides, the kids' artwork etc. These are all dumped into the family NAS.

To avoid being locked into a particular platform, we resisted moving to any software that relies on an underlying database for organizing the photos. Instead we created folders (2010, 2011, 2012 etc) and subfolders (eg. 20100825 Adam's Birthday Party) within the underlying filesystem to organize the photos. With the subfolders, the photos and videos are renamed to "YYYYMMDD-hhmmss" format. jhead is a great help batch-renaming of photos. For video files, the strategy varies. And not all filles can be renamed this way (eg. the scanned photos). Though crude, this has mostly worked. It is also quite flexible when it comes to backup and batch processing.

For viewing the photos, we use XnView.

IrfanView works as well, but it requires more tinkering to get it working with the different video file formats that we have. XnView works pretty much out of the box with the video files.

These two apps have a number of advantages going for them:

1. They are both freeware with lots of plugin support.

2. They work at the filesystem level, and we can easily expand/collapse the folders/subfolders to view our media archive. Some apps require you to import the media that you want to view.

3. They display both photos and videos, both at the thumbnail and individual media level. Most media viewing apps support only photos.

4. They are portable, meaning they can be easily copied/moved from machine to machine without re-installation and executed with their settings intact.

The downside is that both apps are not particularly sexy, but they do get the work done quickly and effectively.

One problem I face is that the photo/video archive on the NAS is standing at 110GB and growing steadily. I want to be able to view this archive on my Android tablet, which is a very natural medium for perusing all the photos and videos that we have accumulated. But fitting 110GB of files into the tablet is next to impossible currently (the largest microSD card is 64GB), and even if I could, the tablet won't be able to deal with all the video formats in the archive (the 1080p MTS files are particularly problematic).

So I decided to write a script to transcode all the files in the archive to a smaller size to see how much I can squeeze it down without ruining the viewing experience on the tablet. This will be the topic of my next post.

Read More

Coding the Liturgical Calendar

For a long time, I've wanted to integrate the Roman Breviary into a couple of projects (nothing public) but have never been able to find a suitable calendar, breviary or psalter implementation.  So I wrote one in python.  It requires the `dateutil` package, but other than that its standard python.  My principle use of this is going to be to create a JSON interface to get the current office, season, feast-day and reading from the 1962 Liturgical calendar.  I am working on a "modern" liturgical calendar, its just not my priority right now. 

Read More

Fast Diet - Hunger Pangs

When I am doing the Fast Diet, I frequently experience long periods of calm after a brief period of hunger pangs. It is as if the body has gone into hibernation, and I am left well alone to carry on with my daily routine. This is most apparent the morning after the diet day, when I could usually skip breakfast and have a late lunch.

I am reminded of what my diving instructor told me during training many years ago, that feelings of suffocation after a brief period under water is not due to lack of oxygen, but due to the buildup of carbon dioxide in the blood stream. This means you are not going to die (yet), and with knowledge you should  try to suppress the panic, remain calm and find a way out of the current situation to surface.

Hunger is mostly the same thing. It makes us uncomfortable and compels us to eat to rid ourselves of the discomfort. Worse, we have come to associate a lot of other feelings (both positive and negative) with eating. So we eat to make ourselves feel better, not necessarily when we are hungry. And this slowly builds up over the years in the world we live in today, where food is available in abundence 24x7.

I think occasional calories restriction (which is essentially what the Fast Diet is about) serves to remind ourselves that it is OK to feel hungry, that you don't have to eat at the first sign of hunger. It also serves to calibrate the "gauges" in our bodies and I think that's where the health benefits are derived.

Read More

Looking for the perfect car MP3 player/FM transmitter

I have bought quite a number of car MP3 player/FM transmitters from DealExtreme and other similar websites.

A car MP3 player/FM transmitter is a device that takes a SDHC memory card or a USB memory stick and plugs into the car cigarette lighter socket. It plays the MP3 files stored on the memory card/stick and transmits that as FM signal to be picked up and played by the car stereo.

It is a nifty and convenient device because not all car stereo can play MP3 files, or be able to accept a memory card or stick. Also such a device is very portable. You can simply plug it into any vehicle eg. a company car and it will play all your favorite music.

Other alternatives exist eg. bluetooth, but they are simply not as straightforward and universal as this solution. After all, you'd be hard-pressed to find a car without a cigarette lighter plug.

The perfect car MP3 player/FM transmitter should have the following characteristics/features:

1. Emits strong FM signal
2. Remembers last FM frequency used
3. Remembers last MP3 file played
4. Remembers position of last MP3 file played
5. Supports Unicode font on LCD display
6. Body of memory card should not stick out when plugged in

1 and 3 are pretty much essential for the device to be useful. Unfortunately I have had to throw at least .a few devices away right upon arrival because they were lacking one or both of these features.

2 and 4 are good-to-have features (some may even consider them essential). 2 can be overcome by setting the car stereo to the default frequency, but sometimes you need to change the default frequency on the device because of interference, and having to do that during every cold start gets old after a while.

4 should be quite easy to do (just a little extra memory to store the last play position), but surprisingly I have not seen any device that has this feature! Either there is some unknown technical barrier to implementing this feature, or Chinese manufacturers are .too stingy to spend a few more cents to do so.

5 is a nice-to-have, since none of my Mandarin songs have their ID3 displayed correctly. On the other hand, this is understandable because it will probably add a lot to the production cost to embed a Unicode font into such a tiny device.

6 is simply a design issue. Half of the devices have it right, others have the memory card sticking out so badly I wonder what the designers were thinking.

I have yet to find my perfect car MP3 player/FM transmitter. I have bought at least a dozen such devices now, and none of them have all of 1,2,3 implemented, not to mention the others. The local Chinese OEM devices sold in Oz stores are not .much better.

Anyway, if you know of the perfect car MP3 player/FM transmitter, please leave me a comment. I am willing to pay a premium for such a device, but I am tired of buying one device after another and being disappointed over and over again...

Read More

Children - The ultimate test in anger management

Having just spent two full holiday weeks with my kids, it occurred to me this morning that raising young children is the ultimate test in anger management.

You know the lines. Address the situation, not the other person. Bring change to yourself first, not to the other person. Easy on paper, very difficult in practice.

With adults, it is often easy to blame it on the other person. After all, he/she is a full-grown adult, why can't he/she be more mature, rational, have a good grounding in the facts etc.?

With children, you can't pin the blame on them anymore! They are what they are at this age. You can't expect them to be mature or rational! But we still lose our cool, and that's a perfect reflection of our own emotional management skills.

Therein lies the ultimate test in anger management. We have to change ourselves to address the situation, and that ain't always easy,  because being human means we resist change with all the might we could muster.

Failure to manage our anger eventually leads to child abuse. The male gender is more frequently associated with such behaviour, but it seems the female gender is sometimes not spare such emotional rampage too.

Our children grow up with us, but we grow up with our children too. Good luck to all the parents traveling on this amazing journey together!

Read More

Fast Diet - Incredible Body Hack

In terms of body hacks, the Fast Diet probably ranks all the way at the top for me.

I first read about it in the papers and found the technique rather intriguing. Could something that simple actually work? Besides the book, it didn't really have anything fancy to sell. No secret formula, patent-pending ingredient, celebrity endorsement etc.

A few weeks later, at a whim, I started to try the diet on myself. At this point, I have not read the book yet. I just started restricting my diet two days a week to 600 calories. And I wasn't too particular with counting the calories either. I simply looked it up on the smartphone and kept a rough count through the day. Sometimes I prepared the food specifically for myself. Other times, I simply ate a smaller portion of whatever the wife had cooked. One or twice, I actually broke fast midday as unexpected commitments cropped up.

I also wasn't particularly keeping a keen eye on my weight during this time. Ironically, I didn't really think the diet would work. It just seems too free-and-easy. Not enough suffering! Then about 8 weeks later, I weighed myself in the bathroom and found that I had lost 6kg. To say I was surprised was an understatement! I started weighing myself more regularly from then on, now out of curiosity then anything. I found myself steadily losing weight at a rate of about 0.5kg per week. Another 8 weeks on, I have lost another 3kg or so. I haven't seen this weight for a long long time, not since my university days!

And I was still eating all my usual junk on the off days: cakes, chips, chocolate, fries, fizzy drinks. Nothing "health conscious" has crept into my mindset so far that I was deliberately avoiding them. I was just eating normally as before.

I did pick up the book to see if I coud learn anything new, but I didn't. I am quite familiar with scientific studies of how severe calorie restriction can prolong life, but most of us can't live that way. Even ADF (Alternate Day Fasting) seems too harsh to me. I think the breakthrough for the book is to introduce the idea that even a moderate amount of fasting will do us plenty of good. Other than that, I am not sure one has to read the book in order to get started on the diet. I don't even find the recipes in the book particularly useful. As I mentioned, I did quite alright with eating whatever my wife cooks, but in smaller quantities.

The book did mention that once you reach the desired weight, you can go into "maintenance mode" and reduce the fasting to once a week. For me, I plan to carry on fasting twice a week just to see how far this diet can hack my body weight. I am pretty sure my weight will stabilize at a certain point and will not reduce indefinitely! I will blog about my progress in times to come.

Read More

ComboFix - Malware Removal Tool

Again, I am writing this down more for my own future reference, with the hope that it will benefit others Googling for this info as well.

ComboFix is a relatively unknown but surprisingly good anti-malware tool. I had in the past received a number of computers from relatives/friends that have been infected with malware. The general characteristics are that the machines boots OK into Windows, but runs erratically eg. displays fake error messages, prevents you from running certain executables, logs you off after awhile etc.

In my experience, it was impossible to get the existing antivirus software to perform a proper scan, or to install a new antivirus software because the system ran so erratically. Offline antivirus software that runs off a CD didn't help much either. I have a collection of them and none of them had helped on the computers I was trying to fix.

Enter ComboFix, which was able to get the computers back into good enough working condition so that I can reinstall the antivirus software and perform a proper scan. I am not sure what's the exact magic behind the program. It seems to "implement a collection of pre-made fixes for large amount of known malware and hunts down all files associated with it". Sounds good to me!

Windows Repair (All In One) is also helpful after running ComboFix. Some malware messes up all kinds of stuff, and functions like "Set Windows Services To Default Startup" is invaluable for straightening the system.

Of course, some people will tell you it's best to wipe the machine when it has been compromised. From a technical point of view, I would totally agree and will do it with my own machines. But in the real world with real people, that's not always the ideal solution for them. So I am glad ComboFix and Windows Repair AIO has enabled me to help some people along the way.

Read More

Do not upgrade to CM10.1 if you use Bluetooth A2DP extensively

I flashed my Nexus S to CM10.1 recently and found that the A2DP streaming stutters so badly to the point of being unusable. Some googling reveals that from Android 4.1 to 4.2, Google swapped the Bluez stack that was used in Android since 2.x and replaced it with a stack contributed by Broadcom. This replacement has apparently created more issues than it has solved.

Anyway, I am back to CM10 (based on Android 4.1.2 with the Bluez stack), and all is well again. So if you use A2DP as much as I do (I listen to podcasts, 1~2 hours a day), you have been warned!

Read More

Samba Server on Android Media Player

I recently bought an Android-based media player: the Minix Neo X5. It came with the original firmware, which had issues playing the MTS video files that my digital camera produces, so I promptly upgrade to the latest ver 011 of the firmware. That solved the MTS playback issue, but the firmware itself was not very stable and hang quite frequently. So I promptly installed Clockmod Recovery, followed by the CWM version of Finless V1.2b. Now everything is fast and stable!

Next, I needed an SMB server on the media player so that I can manage its content from other PCs in the house. Surprisingly this turned out to be harder than I thought given the hackish nature of Android.

SambaDroid was quite easy to install and work with. Unfortunately, it does not support files > 2GB in size. This is quite a major limitation given than many media files these days are > 2GB in size.

Samba Server turned out to be a dud. Despite having root access, I couldn't set it up so that it is accessible by other Windows PCs. I even performed some registry hacks to enable legacy NTLM without any success.

I had partial success with Samba Filesharing (V1.0.6) and it supports files > 2GB, but I could only get it to work occasionally after a lot of coxing. Turns out it does not yet support Jelly Bean, so the app crashes the other 99% of the time, even with the samba-rc hack.

With no other candidates to try, I tried hacking into Samba Filesharing to see if I can make it work better with JB. Armed with adb shell,  I found that the Android version of smbd and nmbd are found under /data/data/com.funkyfresh.samba/files and can be started by running:

> sh ./samba-rc restart

But the app itself interfered with the operation of the daemons. Perhaps it is trying to control them and ended up crashing/killing the daemons. Anyway, the conclusion is that there is a fully working version of Samba included with the app. It's the app itself that is incompatible with JB. The app is a wrapper that receives events so that it can start smbd and nmbd when the network is up, and kill the daemons upon user request or when the network goes down.

So what I ended doing was:

1. Install and configure Samba Filesharing. This properly configures samba-rc and smb.conf.

2. Perform adb pull /data/data/com.funkyfresh.samba/files. Then update samba-rc by adding a single line.

3. Uninstall Samba Filesharing. This gets rid of the app itself so that it does not interfere with the daemons.

4. Manually create /data/data/com.funkyfresh.samba/files via adb shell.

5. Zip up the files and upload via adb push. The zipping is necessary because adb push can only handle individual files. Then adb shell and unzip.

6. Perform chmod 0777 samba-rc smbd nmbd.

Then I try to manually execute sh ./samba-rc restart. This started the smbd and nmbd daemons and I was able to connect to the shares from another machine.

Now this left the problem of running the script when the machine starts up, after the network interface is brought up. The simplest way I found was to use Script Manager. There are probably other ways (eg. via DHCP hooks), but I am lazy and went with the simplest method that worked.

I am sure Samba Filesharing will eventually be updated to support JB fully. There is a very active thread here on xda-developers for this purpose. But in the mean time, if Samba Filesharing does not work directly on your device yet, I hope my experience here is of some help to get it working on yours.

Read More

General (lossy) method for normalizing volume of any video file

I have an MKV video file (H264 video, AC3 audio) whose volume I need to normalize. Unfortunately, there doesn't seem to be any lossless way of normalizing the AC3 audio track. So I found out the general (lossy) way of normalizing the audio track of any video file is:

> ffmpeg -i input.mkv -vol nnn -vcodec copy output.mkv

nnn = 256 (no change)
nnn = 128 (half the volume)
nnn = 512 (double the volume)
etc.

This should work for any video file format that ffmpeg recognizes.

In addition, I can use the -map commands to remove any unwanted subtitles.

> ffmpeg -i input.mkv -map 0:0 -map 0:1 -ac 2 -vol nnn -vcodec copy output.mkv

This tells ffmpeg to only include track 0 (video) and 1 (audio) in the output file and ignore the rest.

Read More